ERIC Number: ED243962
Record Type: RIE
Publication Date: 1984-Apr
Reference Count: 0
Inter-Judge Reliability: Is Complete Agreement among Judges the Ideal?
Constable, Elizabeth; Andrich, David
In circumstances where judges are required to make ratings of performance, it is usually required to have two or more raters who are trained to agree on independent ratings of the same performance. It is suggested that such a requirement may produce the paradox of attenuation associated with item analysis, in which too high a correlation between items, while enhancing reliability, decreases validity. With an empirical example of ratings of levels of social understanding it is shown how information is lost as judges agree more and more over a series of ratings which follow discussions of the previous set of ratings. Pursuing the item analysis analogy, it is demonstrated how it might be more appropriate to handle systematic differences among judges using a person-item response theory approach. (Author)
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers: Attenuation Paradox
Note: Paper presented at the Annual Meeting of the National Council on Measurement in Education (New Orleans, LA, April 24-26, 1984).