ERIC Number: EJ864665
Record Type: Journal
Publication Date: 2009
Abstractor: As Provided
Reference Count: 20
Judges' Use of Examinee Performance Data in an Angoff Standard-Setting Exercise for a Medical Licensing Examination: An Experimental Study
Clauser, Brian E.; Mee, Janet; Baldwin, Su G.; Margolis, Melissa J.; Dillon, Gerard F.
Journal of Educational Measurement, v46 n4 p390-407 Win 2009
Although the Angoff procedure is among the most widely used standard setting procedures for tests comprising multiple-choice items, research has shown that subject matter experts have considerable difficulty accurately making the required judgments in the absence of examinee performance data. Some authors have viewed the need to provide performance data as a fatal flaw for the procedure; others have considered it appropriate for experts to integrate performance data into their judgments but have been concerned that experts may rely too heavily on the data. There have, however, been relatively few studies examining how experts use the data. This article reports on two studies that examine how experts modify their judgments after reviewing data. In both studies, data for some items were accurate and data for other items had been manipulated. Judges in both studies substantially modified their judgments whether the data were accurate or not.
Descriptors: Standard Setting (Scoring), Program Effectiveness, Expertise, Health Personnel, Licensing Examinations (Professions), Psychometrics, Educational Testing, Evaluation Methods, Educational Assessment, Measurement, Evaluation Criteria, Evaluation Research, Scoring, Testing Problems, Evaluation Problems, Data, Data Analysis
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: firstname.lastname@example.org; Web site: http://www.wiley.com/WileyCDA/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A