NotesFAQContact Us
Collection
Advanced
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ822972
Record Type: Journal
Publication Date: 2009-Jan
Pages: 21
Abstractor: As Provided
Reference Count: 20
ISBN: N/A
ISSN: ISSN-0895-7347
An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting Procedure
Clauser, Brian E.; Harik, Polina; Margolis, Melissa J.; McManus, I. C.; Mollon, Jennifer; Chis, Liliana; Williams, Simon
Applied Measurement in Education, v22 n1 p1-21 Jan 2009
Numerous studies have compared the Angoff standard-setting procedure to other standard-setting methods, but relatively few studies have evaluated the procedure based on internal criteria. This study uses a generalizability theory framework to evaluate the stability of the estimated cut score. To provide a measure of internal consistency, this study also compares the estimated proportion correct scores resulting from the Angoff exercise to empirical conditional proportion correct scores. In this research, judges made independent estimates of the proportion of minimally proficient candidates that would be expected to answer each item correctly; they then discussed discrepancies and revised their estimates. Discussion of discrepancies decreased the variance components associated with the judge and judge-by-item effects, indicating increased agreement between judges, but it did not improve the correspondence between the judgments and the empirical proportion correct estimates. The judges then were given examinee performance information for a subset of the items. Subsequent ratings showed a substantial increase in correspondence with the empirical conditional proportion correct estimates. Particular attention is given to examining the discrepancy between the judgments and empirical proportion correct estimates as a function of item difficulty. (Contains 8 figures and 4 tables.)
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: United Kingdom