NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1175504
Record Type: Journal
Publication Date: 2018-Apr
Pages: 16
Abstractor: As Provided
ISSN: EISSN-1531-7714
Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability
Wilhelm, Anne Garrison; Gillespie Rouse, Amy; Jones, Francesca
Practical Assessment, Research & Evaluation, v23 n4 Apr 2018
Although inter-rater reliability is an important aspect of using observational instruments, it has received little theoretical attention. In this article, we offer some guidance for practitioners and consumers of classroom observations so that they can make decisions about inter-rater reliability, both for study design and in the reporting of data and results. We reviewed articles in two major journals in the fields of reading and mathematics to understand how researchers have measured and reported inter-rater reliability in a recent decade. We found that researchers have tended to report measures of inter-rater agreement above the 0.80 threshold with little attention to the magnitude of score differences between raters. Then, we conducted simulations to understand both how different indices for classroom observation reliability are related to each other and the impact of reliability decisions on study results. Results from the simulation studies suggest that mean correlations with an outcome are slightly lower at lower levels of percentage of exact agreement but that the magnitude of score differences has a more dramatic effect on correlations. Therefore, adhering to strict thresholds for inter-rater agreement is less helpful than reporting exact point estimates and also examining measures of rater consistency.
Center for Educational Assessment. 813 North Pleasant Street, Amherst, MA 01002. e-mail:; Tel: 413-577-2180; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A