ERIC Number: EJ1083299
Record Type: Journal
Publication Date: 2015
Abstractor: As Provided
Non-Numeric Intrajudge Consistency Feedback in an Angoff Procedure
Harrison, George M.
Journal of Educational Measurement, v52 n4 p399-418 Win 2015
The credibility of standard-setting cut scores depends in part on two sources of consistency evidence: intrajudge and interjudge consistency. Although intrajudge consistency feedback has often been provided to Angoff judges in practice, more evidence is needed to determine whether it achieves its intended effect. In this randomized experiment with 36 judges, non-numeric item-level intrajudge consistency feedback was provided to treatment-group judges after the first and second rounds of Angoff ratings. Compared to the judges in the control condition, those receiving the feedback significantly improved their intrajudge consistency, with the effect being stronger after the first round than after the second round. To examine whether this feedback has deleterious effects on between-judge consistency, I also examined interjudge consistency at the cut score level and the item level using generalizability theory. The results showed that without the feedback, cut score variability worsened; with the feedback, idiosyncratic item-level variability improved. These results suggest that non-numeric intrajudge consistency feedback achieves its intended effect and potentially improves interjudge consistency. The findings contribute to standard-setting feedback research and provide empirical evidence for practitioners planning Angoff procedures.
Descriptors: Interrater Reliability, Standard Setting (Scoring), Cutting Scores, Feedback (Response), Comparative Analysis, Generalizability Theory
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: email@example.com; Web site: http://www.wiley.com/WileyCDA
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A