ERIC Number: EJ1073383
Record Type: Journal
Publication Date: 2015
Pages: 18
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0022-0655
EISSN: N/A
Comparing the Effectiveness of Self-Paced and Collaborative Frame-of-Reference Training on Rater Accuracy in a Large-Scale Writing Assessment
Raczynski, Kevin R.; Cohen, Allan S.; Engelhard, George, Jr.; Lu, Zhenqiu
Journal of Educational Measurement, v52 n3 p301-318 Fall 2015
There is a large body of research on the effectiveness of rater training methods in the industrial and organizational psychology literature. Less has been reported in the measurement literature on large-scale writing assessments. This study compared the effectiveness of two widely used rater training methods--self-paced and collaborative frame-of-reference training--in the context of a large-scale writing assessment. Sixty-six raters were randomly assigned to the training methods. After training, all raters scored the same 50 representative essays prescored by a group of expert raters. A series of generalized linear mixed models were then fitted to the rating data. Results suggested that the self-paced method was equivalent in effectiveness to the more time-intensive and expensive collaborative method. Implications for large-scale writing assessments and suggestions for further research are discussed.
Descriptors: Interrater Reliability, Writing Evaluation, Training Methods, Pacing, Scores, Essays, Data Analysis, Instructional Effectiveness, Intermode Differences
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A