ERIC Number: ED458239
Record Type: Non-Journal
Publication Date: 2001
Reference Count: N/A
Inter-Rater Reliability on Performance Criteria: Theoretical Issues.
A hypothetical case study provides examples of the inter-rater reliability issues involved in complex performance assessment, focusing on the Baldrige model. A hypothetical team of five evaluators was asked to rate a Baldrige model performance assessment along the seven defined criteria or performance dimensions that comprise the Baldrige model and arrive at a consensus score. The procedures discussed in this paper point to some of the issues that face those who design, administer, and score authentic assessments in the private or public sector using scoring rubrics or criteria, particularly when more than one rater is involved. It is also easy to see they it is advisable to test the reliability of scores assigned by having more than one rater score the same respondent and by having the same person rescore a respondent previously scored to a rubric. It is not uncommon for raters to discover that they have assigned the same paper different scores on independent blind observations. The hypothetical study also shows the importance of having raters retrained to the rubrics on a regular basis prior to assigning consensus scores. An appendix lists the categories and point values of the Baldrige criteria. (SLD)
Descriptors: Case Studies, Criteria, Evaluators, Interrater Reliability, Performance Based Assessment, Performance Factors
Tennessee Valley Educators for Excellence, P.O. Box 2882, Muscle Shoals, AL 35662. Tel: 256-446-5278; Fax: 256-446-5278; Web site: http://www.dpo.uab.edu/~tnmarie.
Publication Type: Reports - Research
Education Level: N/A
Authoring Institution: Tennessee Valley Educators for Excellence, Muscle Shoals, AL.