Download full text
Download full text
ERIC Number: EJ1165794
Record Type: Journal
Publication Date: 2017-Dec
Abstractor: As Provided
The Miscalculation of Interrater Reliability: A Case Study Involving the AAC&U VALUE Rubrics
Szafran, Robert F.
Practical Assessment, Research & Evaluation, v22 n11 Dec 2017
Institutional assessment of student learning objectives has become a fact-of-life in American higher education and the Association of American Colleges and Universities' (AAC&U) VALUE Rubrics have become a widely adopted evaluation and scoring tool for student work. As faculty from a variety of disciplines, some less familiar with the psychometric literature, are drawn into assessment roles, it is important to point out two easily made but serious errors in what might appear to be one of the more straightforward assessments of measurement quality--interrater reliability. The first error which can occur when a third rater is brought in to adjudicate a discrepancy in the scores reported by an initial two raters has been well-documented in the literature but never before illustrated with AAC&U rubrics. The second error is to cease training before the raters have demonstrated a satisfactory level of interrater reliability. This research note describes an actual case study in which the interrater reliability of the AAC&U rubrics was incorrectly reported and when correctly reported found to be inadequate. The note concludes with recommendations for the correct measurement of interrater reliability.
Descriptors: Interrater Reliability, Case Studies, Scoring Rubrics, Behavioral Objectives, Error of Measurement, Evaluation Problems, Correlation, Test Reliability, Higher Education, Writing Assignments
Center for Educational Assessment. 813 North Pleasant Street, Amherst, MA 01002. e-mail: firstname.lastname@example.org; Tel: 413-577-2180; Web site: https://scholarworks.umass.edu/pare
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Authoring Institution: N/A
Grant or Contract Numbers: N/A