GRPREL computes the interrater reliability statistics for the specified data set. The data set used in the computations can be restricted to the set of raters and items desired by using appropriate group selection and item selection masks. In addition, use of certain run time specifiers will cause GRPREL to eliminate deviant raters and/or items from the data set, and then perform its computations on the new restricted data set. When this occurs, the values computed during each iteration of GRPREL will be reported. The values reported will be based on the raw data values, or if requested, the raw values will be adjusted prior to the computations.
GRPREL will also optionally produce new task factors if an output Task Factor file is specified. These factors include a new membership selection mask if any raters were removed and a factor containing the number of raters which rated each item. Two other factors are also produced. The first contains the mean rating value for each item. The second contains the corresponding standard deviations. These last two factors are computed using the adjusted ratings if the ratings were adjusted for the interrater agreement computations. A final option may be employed which will standardize the mean and standard deviation factors to a specified mean and standard deviation.
Program Invocation
The specific syntax for program invocation varies between operating systems. This document diplays the code for the AIX version.
opt: E Explode, compute reliability for each module or duty after
computing values for entire data set
N Nogo run, scan control cards for errors
ssss: The study number
CDf: Input Case Data file
TT: Input Task Title file or Module Title file
TFf: Input/output Task Factor file, required if either a membershipr
and/or task selection mask is used or the generated mean and
standard deviation factors are to be saved.
GRPREL: cc 01-06 The literal 'GRPREL' study: cc 08-11 The study number for this run maxit: cc 13-14 Maximum number of iterations (default=3) r11: cc 16-21 The desired value for the R(1,1). If the computed R(1,1) meets or exceeds this value and the computed R(k,k) meets or exceeds the value specified in the rkk field of this card GRPREL will stop, regardless of the maximum number of iterations. (default = 0.999) rkk: cc 23-28 The desired value for the R(k,k). If the computed R(k,k) meets or exceeds this value and the computed R(1,1) meets or exceeds the value specified in the r11 field of this card GRPREL will stop regardless of the maximum number of iterations. (default = 0.999) nitem: cc 30-35 The minimum acceptable number of ratings for each task. All tasks that have fewer ratings than specified will be deleted prior to the performance of any computations. (default = 0) mbr: cc 37-42 The group ID for the set of raters to be used. If blank, all members will be used (i.e. GP0001). fac: cc 44-49 The factor ID for the task factor for task selection. If present, the factor type must also be specified. All tasks will be used which have a valid, positive, non-zero task factor value. (default = all tasks) typ: cc 51-53 The type of the task factor for task selection.
Task Report Note Cards (optional)
These cards, if used, will appear on the listing of task means and standard deviations(SDs). They are started by the literal 'NOTE', and if used, must be terminated with an @EOF.
NOTE ...any number note cards... @EOF
Title Card (optional)
This card is not required. If used, the title supplied will be printed at the top of each page on the output report. If the title field of this card is blank or if this card is not used the title for the study found on the input file will be used and printed at the top of each page of the report.
TITLE: cc 01-05 The literal 'TITLE' title: cc 07-66 Any report title information desired
Rater Control Card (Optional)
This card is required only if one or more of the default values specified in the field descriptions below is to be changed.
RATER: cc 01-05 The literal 'RATER' amean: cc 07-12 If blank, ratings are left in their raw form. If a decimal value, the values for each rater are adjusted such that the rater mean is the specified value. If the word 'SAMPLE', each rater's ratings are adjusted to a mean equal to the sample mean for items rated in common. asd: cc 14-19 If blank and adjustment is performed, the standard deviation is adjusted to 1.0 for each rater. If a decimal value, the standard deviation for each rater is adjusted to this value. Use of the word 'RATER' prevents adjustment of the standard deviation. rcut: cc 21-26 The minimum correlation required to keep a rater in the next iteration. (default = 0.20) pcut: cc 28-33 The minimum probability required to keep a rater in the next iteration. (default = 0.95) minuse: cc 35-40 The minimum number of raters. This value is calculated after deciding which raters would not be used in the next iteration. If this minimum value will not be met, the run will be stopped. (default=2) maxdel: cc 42-47 The maximum percentage of raters to delete. This is calculated after deciding which raters would not be used in the next iteration. If this maximum value is exceeded, the run is stopped. (default = 20.0 %)
Value Control Card (optional)
This card is required only if one or more of the default values specified in the field descriptions below is to be changed.
VALUE: cc 01-05 The literal 'VALUE' std: cc 07 'S' = Standardize the output mean and standard deviation factors. (Default = do not standardize) smn: cc 09-14 The desired mean for the standardized task means (default = 5.0). Not used if the std field does not contain a 'S'. ssd: cc 16-21 The desired standard deviation for the standardized tasks means. Not used if the smn field is ignored. (default=1.0) sct: cc 23-28 The maximum standard deviation for a task. If the s.d. exceeds this value, the task is removed AFTER the current iteration. If blank, no tasks are deleted based on their s.d.. Values should reflect scale transformations produced by the previous two fields. zct: cc 30-35 The maximum z-score for task standard deviation. If the z-score exceeds this value, the task is removed after this iteration. If blank, no tasks are deleted because of a high z-score. minu: cc 37-42 The minimum number of tasks. If the number of tasks to use in the next iteration is less than this value, the run is stopped. (default = 2) maxd: cc 44-49 The maximum percentage of tasks to delete. If this value is exceeded the run is halted. (default=100.0 %) mint: cc 51-56 The minimum task value (default=original minimum from AUDITR) maxt: cc 58-63 The maximum task value (default=original maximum from AUDITR)
Sorts Card (optional)
Normally the listing of tasks will be printed three times - inventory order, sorted on task mean, and sorted on task standard deviation. If the user wants to restrict the number of task listings, the SORTS card must be used.
SORTS: cc 01-05 The literal 'SORTS' lit: cc 07-09 The literal(s) for specifying the sort(s). I - Sort tasks in inventory order M - Sort tasks on task mean S - Sort tasks on task standard deviation N - Sort tasks by number of raters rating If all four letters are specified, four listings will be printed. If no literal is specified, no task listings are generated.
Rater Correlation Table Note Cards (optional)
The cards in this section are textual notes that are to be printed on the rater correlation table note cards. They are used in the same way as the task report note cards. They are started by the literal 'NOTE', and if used, must be terminated with an @EOF.
NOTE ...any number of note cards.. @eof
The control cards are terminated by an end-of-file condition or an '@eof' image.
Examples
@codap grprel - ssss cd200 tt001 grprel ssss @eof
This is an example of the simplest GRPREL run possible. The interrater reliability will be computed for all raters (cd200 here implies Task Difficulty ratings) and all items in the data set. No adjustment of ratings will be made. A printed report will be produced, and no factors will be saved.
@codap grprel - ssss cd200 tt001 grprel ssss NOTE @ADD <filespec for a file containing note cards> @eof TITLE Main title line for this grprel RATER SAMPLE VALUE S SORTS M NOTE These note cards will be printed on the rater correlation table only @eof
This GRPREL run would produce the interrater reliability values for all raters and items in the data set. The values would be based on adjusted task values where the rater's mean was adjusted to match the sample mean for the set of items rated in common. Each rater's standard deviation would be adjusted to 1.0. This is the most typical method used for adjusting rater values. The use of the 'S' on the VALUE card would cause the item means and standard deviations to be standardized. The factor of mean values would be standardized to a mean of 5.0 and a standard deviation of 1.0. The factor of standard deviations would be rescaled to correspond to the new mean values factor. Both of these factors would be saved on the specified Task Factor file. The task listing would be sorted on task mean. This listing would use note cards from the file specified, while the rater correlation uses other note cards.