NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1096518
Record Type: Journal
Publication Date: 2016-Apr
Pages: 13
Abstractor: As Provided
ISSN: ISSN-1863-9690
Uncovering Predictors of Disagreement: Ensuring the Quality of Expert Ratings
Hoth, Jessica; Schwarz, Björn; Kaiser, Gabriele; Busse, Andreas; König, Johannes; Blömeke, Sigrid
ZDM: The International Journal on Mathematics Education, v48 n1-2 p83-95 Apr 2016
Rating scales are a popular item format used in many types of assessments. Yet, defining which rating is correct often represents a challenge. Using expert ratings as benchmarks is one approach to ensuring the quality of a rating instrument. In this paper, such expert ratings are analyzed in detail taking a video-based test instrument of teachers' professional competencies from a follow-up study to TEDS-M (the so-called TEDS-FU study) as an example. The paper focuses on those items that did NOT reach sufficient consensus among the experts and analyzes in depth their features by coding the experts' comments on those items and additionally considering their rating outcome. The results revealed that the item-wording and the composition of the group of experts strengthened or weakened agreement among the experts.
Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A