ERIC Number: EJ722980
Record Type: Journal
Publication Date: 2005-Dec
Reference Count: 23
A Closer Look at Using Judgments of Item Difficulty to Change Answers on Computerized Adaptive Tests
Vispoel, Walter P.; Clough, Sara J.; Bleiler, Timothy
Journal of Educational Measurement, v42 n4 p331-350 Dec 2005
Recent studies have shown that restricting review and answer change opportunities on computerized adaptive tests (CATs) to items within successive blocks reduces time spent in review, satisfies most examinees' desires for review, and controls against distortion in proficiency estimates resulting from intentional incorrect answering of items prior to review. However, restricting review opportunities on CATs may not prevent examinees from artificially raising proficiency estimates by using judgments of item difficulty to signal when to change previous answers. We evaluated six strategies for using item difficulty judgments to change answers on CATs and compared the results to those from examinees reviewing and changing answers in the usual manner. The strategy conditions varied in terms of when examinees were prompted to consider changing answers and in the information provided about the consistency of the item selection algorithm. We found that examinees fared best on average when they reviewed and changed answers in the usual manner. The best gaming strategy was one in which the examinees knew something about the consistency of the item selection algorithm and were prompted to change responses only when they were unsure about answer correctness and sure about their item difficulty judgments. However, even this strategy did not produce a mean gain in proficiency estimates.
Descriptors: Mathematics, Item Analysis, Adaptive Testing, Computer Assisted Testing, Test Wiseness, Guessing (Tests), Difficulty Level, Test Items
Journal Customer Services, Blackwell Publishing, 350 Main Street, Malden, MA 02148. Tel: 800-835-6770 (Toll Free); Fax: 781-388-8232; e-mail: email@example.com.
Publication Type: Journal Articles; Reports - Evaluative
Education Level: N/A
Authoring Institution: N/A