ERIC Number: EJ1093282
Record Type: Journal
Publication Date: 2016
Abstractor: As Provided
Effect of Content Knowledge on Angoff-Style Standard Setting Judgments
Margolis, Melissa J.; Mee, Janet; Clauser, Brian E.; Winward, Marcia; Clauser, Jerome C.
Educational Measurement: Issues and Practice, v35 n1 p29-37 Spr 2016
Evidence to support the credibility of standard setting procedures is a critical part of the validity argument for decisions made based on tests that are used for classification. One area in which there has been limited empirical study is the impact of standard setting judge selection on the resulting cut score. One important issue related to judge selection is whether the extent of judges' content knowledge impacts their perceptions of the probability that a minimally proficient examinee will answer the item correctly. The present article reports on two studies conducted in the context of Angoff-style standard setting for medical licensing examinations. In the first study, content experts answered and subsequently provided Angoff judgments for a set of test items. After accounting for perceived item difficulty and judge stringency, answering the item correctly accounted for a significant (and potentially important) impact on expert judgment. The second study examined whether providing the correct answer to the judges would result in a similar effect to that associated with knowing the correct answer. The results suggested that providing the correct answer did not impact judgments. These results have important implications for the validity of standard setting outcomes in general and on judge recruitment specifically.
Descriptors: Academic Standards, Standard Setting (Scoring), Cutting Scores, Credibility, Medical Education, Item Analysis, Test Items, Accuracy, Validity, Knowledge Level, Medical Students, Student Evaluation, Expertise, Test Wiseness
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: email@example.com; Web site: http://www.wiley.com/WileyCDA
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Authoring Institution: N/A
Grant or Contract Numbers: N/A