ERIC Number: ED383722
Record Type: RIE
Publication Date: 1993-Oct
Reliability of Essay Rating and Score Adjustment. Program Statistics Research Technical Report No. 93-36.
Longford, Nicholas T.
A model-based approach to rater reliability for essays read by multiple readers is presented. Variation of rater severity (between-rater variation) and rater inconsistency (within-rater variation) is considered in the presence of between-examinee variation. An additive variance component model is posited and the method of moments for its estimation described. The model involves no distributional assumptions. Minimum mean squared error estimators of examinees' true scores and readers' severities are derived. Model diagnostic procedures are an integral component of the approach. The methods are illustrated on data from standardized educational tests. Six tables and seven figures are included. (Contains eight references.) (Author)
Publication Type: Reports - Evaluative
Education Level: N/A
Authoring Institution: Educational Testing Service, Princeton, NJ.