ERIC Number: ED414319
Record Type: RIE
Publication Date: 1997-Oct
Clarifying the Blurred Image: Estimating the Inter-Rater Reliability of Performance Assessments.
Moore, Alan D.; Young, Suzanne
As schools move toward performance assessment, there is increasing discussion of using these assessments for accountability purposes. When used for making decisions, performance assessments must meet high standards of validity and reliability. One major source of unreliability in performance assessments is interrater disagreement. In this paper, the literature on interrater reliability is reviewed, and a useful and understandable summary of methods is presented for estimating interrater reliability that can be used in performance assessments. Methods of quantifying the degree of interrater reliability are classified into three categories: (1) methods based on bivariate correlation, such as the Pearson product-moment correlation and Spearman's rank correlation coefficient; (2) methods based on the percent of interrater agreement; and (3) methods based on intraclass correlation or by treating raters as a facet in a generalizability study. Examples illustrate use of these methods and issues related to their use. (Contains 5 tables and 17 references.) (Author/SLD)
Publication Type: Information Analyses; Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Note: Paper presented at the Annual Meeting of the Northern Rocky Mountain Educational Research Association (Jackson, WY, October 1997).