ERIC Number: EJ1025483
Record Type: Journal
Publication Date: 2013
Abstractor: As Provided
Reference Count: N/A
The Impact of Process Instructions on Judges' Use of Examinee Performance Data in Angoff Standard Setting Exercises
Mee, Janet; Clauser, Brian E.; Margolis, Melissa J.
Educational Measurement: Issues and Practice, v32 n3 p27-35 Fall 2013
Despite being widely used and frequently studied, the Angoff standard setting procedure has received little attention with respect to an integral part of the process: how judges incorporate examinee performance data in the decision-making process. Without performance data, subject matter experts have considerable difficulty accurately making the required judgments. Providing data introduces the very real possibility that judges will turn their content-based judgments into norm-referenced judgments. This article reports on three Angoff standard setting panels for which some items were randomly assigned to have incorrect performance data. Judges were informed that some of the items were accompanied by inaccurate data, but were not told which items they were. The purpose of the manipulation was to assess the extent to which changing the instructions given to the judges would impact the extent to which they relied on the performance data. The modified instructions resulted in the judges making less use of the performance data than judges participating in recent parallel studies. The relative extent of the change judges made did not appear to be substantially influenced by the accuracy of the data.
Descriptors: Standard Setting (Scoring), Judges, Data, Decision Making, Performance, Accuracy, Expertise
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: email@example.com; Web site: http://www.wiley.com/WileyCDA/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A