ERIC Number: EJ1059779
Record Type: Journal
Publication Date: 2015
Abstractor: As Provided
Reference Count: 26
The Effects of Rater Training on Inter-Rater Agreement
Pufpaff, Lisa A.; Clarke, Laura; Jones, Ruth E.
Mid-Western Educational Researcher, v27 n2 p117-141 2015
This paper addresses the effects of rater training on the rubric-based scoring of three preservice teacher candidate performance assessments. This project sought to evaluate the consistency of ratings assigned to student learning outcome measures being used for program accreditation and to explore the need for rater training in order to increase rater agreement. There were three phases during this project: (1) authentic student work was rated by department faculty members in the absence of rubric training; (2) faculty were then trained to administer rubric scoring guides; and (3) additional student work was rated by faculty after training. Inter-rater agreement was calculated pre- and post-rater training, using side-by-side comparisons. Little to no improvement in rater agreement was seen post-training. Implications and future research needs for rater training in the application of rubrics are discussed.
Descriptors: Evaluators, Interrater Reliability, Preservice Teachers, Scoring Rubrics, Outcomes of Education, Performance Based Assessment, College Faculty, Teacher Education, Accountability, Training, Undergraduate Students, Special Education Teachers, Case Studies, Portfolios (Background Materials), Internet, Surveys, Feedback (Response)
Mid-Western Educational Research Association. P.O. Box 34421, Chicago, IL 60634-0421. Tel: 419-372-7401; Fax: 419-372-2828; e-mail: firstname.lastname@example.org; Web site: http://www.mwera.org
Publication Type: Journal Articles; Reports - Research; Tests/Questionnaires
Education Level: Higher Education; Postsecondary Education
Authoring Institution: N/A