ERIC Number: EJ1054067
Record Type: Journal
Publication Date: 2015
Abstractor: As Provided
Consistency of Angoff-Based Standard-Setting Judgments: Are Item Judgments and Passing Scores Replicable across Different Panels of Experts?
Tannenbaum, Richard J.; Kannan, Priya
Educational Assessment, v20 n1 p66-78 2015
Angoff-based standard setting is widely used, especially for high-stakes licensure assessments. Nonetheless, some critics have claimed that the judgment task is too cognitively complex for panelists, whereas others have explicitly challenged the consistency in (replicability of) standard-setting outcomes. Evidence of consistency in item judgments and passing scores is necessary to justify using the passing scores for consequential decisions. Few studies, however, have directly evaluated consistency across different standard-setting panels. The purpose of this study was to investigate consistency of Angoff-based standard-setting judgments and passing scores across 9 different educator licensure assessments. Two independent, multistate panels of educators were formed to recommend the passing score for each assessment, with each panel engaging in 2 rounds of judgments. Multiple measures of consistency were applied to each round of judgments. The results provide positive evidence of the consistency in judgments and passing scores.
Descriptors: Standard Setting (Scoring), Reliability, Scores, Licensing Examinations (Professions), Test Items
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A