NotesFAQContact Us
Search Tips
ERIC Number: ED431032
Record Type: Non-Journal
Publication Date: 1999-Apr
Pages: 37
Abstractor: N/A
Reference Count: N/A
The Assignment of Raters to Items: Controlling for Rater Effects.
Sykes, Robert C.; Heidorn, Mark; Lee, Guemin
A study was conducted to evaluate the effect of different modes (modalities) of assigning raters to test items. The impact on total constructed response (c.r.) score, and subsequently on total test score, of assigning a single versus multiple raters to an examination reading of a student's set of c.r. responses was evaluated for several mixed-item format tests. Samples of approximately 2,000 students were obtained from a state mathematics field test at each of grades 5, 8, and 10 and from a reading field test at each of grades 4, 8, and 10. Item responses for c.r. items for each selected student in the six samples were allocated to raters three different ways: (1) single rater reading of all responses (SM1); (2) assignment of each c.r. response to a different rater (SM2); and (3) splitting the c.r. items into thirds, with a different rater for each portion (SM3). SSM1 readings produced average total c.r. scores that were greater than the average total of c.r. scores produced in the SM2 condition. Average total c.r. scores when students' responses were allocated to three different raters (SM3) were similar to those of the SM2 condition. Results suggest that for tests with relatively large numbers of c.r. items, the use of as few as three raters to score a student's examination could produce scores that were similar in magnitude and scale to those obtained by assigning a different rater to each item. (Contains 7 tables and 11 references.) (SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A