NotesFAQContact Us
Search Tips
ERIC Number: ED523734
Record Type: Non-Journal
Publication Date: 2011-Feb
Pages: 31
Abstractor: As Provided
Reference Count: 16
An Examination of the Link between Rater Calibration Performance and Subsequent Scoring Accuracy in Graduate Record Examinations[R] (GRE[R]) Writing. Research Report. ETS RR-11-03
Ricker-Pedley, Kathryn L.
Educational Testing Service
A pseudo-experimental study was conducted to examine the link between rater accuracy calibration performances and subsequent accuracy during operational scoring. The study asked 45 raters to score a 75-response calibration set and then a 100-response (operational) set of responses from a retired Graduate Record Examinations[R] (GRE[R]) writing prompt that had been vetted to determine correct scores prior to the study. The study found a positive relationship between calibration accuracy and scoring accuracy. Results suggest that a longer calibration test is more reliable and has a stronger correlation with operational scoring exact agreement; however, even tests as short as 10 responses had a reasonable correlation with operational accuracy, and the shorter test classified raters accurately 87% of the time when compared to the 75-response criterion. The additional requirement of no discrepant scores during calibration had better correlation with exact agreement, but much poorer classification accuracy in the shorter tests. This research suggests that calibration is useful for screening raters in its current form, but that further investigation of passing standards and of less controlled operational scoring conditions is required. (Contains 16 tables, 3 figures and 2 notes.)
Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site:
Publication Type: Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Educational Testing Service
Identifiers - Assessments and Surveys: Graduate Record Examinations