ERIC Number: ED411303
Record Type: Non-Journal
Publication Date: 1997-Mar
Reference Count: N/A
Rater Severity in Large-Scale Assessment: Is It Invariant?
McQueen, Joy; Congdon, Peter J.
A study was conducted to investigate the stability of rater severity over an extended rating period. Multifaceted Rasch analysis was applied to ratings of writing performances of 8,285 primary school (elementary) students. Each performance was rated on two performance dimensions by two trained raters over a period of 7 rating days. Performances rated on the first day were rerated at the end of the rating period. Statistically significant differences between raters were found within each day and in all days combined. Daily measures of the relative severity of individual rates were found not to be invariant when compared to single, on-average measures for the whole rating period. For more than half of the raters, severity measures on the last day were significantly different from measures on the first day. These findings must cast doubt on the practice of using a single calibration of rater severity as the basis for adjustment of person measures. (Contains 8 figures, 12 tables, and 31 references.) (Author/SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A