ERIC Number: EJ864667
Record Type: Journal
Publication Date: 2009
Abstractor: As Provided
Reference Count: 22
Monitoring Rater Performance over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use
Myford, Carol M.; Wolfe, Edward W.
Journal of Educational Measurement, v46 n4 p371-389 Win 2009
In this study, we describe a framework for monitoring rater performance over time. We present several statistical indices to identify raters whose standards drift and explain how to use those indices operationally. To illustrate the use of the framework, we analyzed rating data from the 2002 Advanced Placement English Literature and Composition examination, employing a multifaceted Rasch approach to determine whether raters exhibited evidence of two types of differential rater functioning over time (i.e., changes in levels of accuracy or scale category use). Some raters showed statistically significant changes in their levels of accuracy as the scoring progressed, while other raters displayed evidence of differential scale category use over time.
Descriptors: English Literature, Advanced Placement, Measures (Individuals), Writing (Composition), Advanced Placement Programs, Secondary Education, Educational Testing, Student Evaluation, Evaluation Methods, Educational Assessment, Measurement, Evaluation Research, Scoring, Testing Problems, Evaluation Problems
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: firstname.lastname@example.org; Web site: http://www.wiley.com/WileyCDA/
Publication Type: Journal Articles; Reports - Research
Education Level: Elementary Secondary Education; Secondary Education
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Advanced Placement Examinations (CEEB)