NotesFAQContact Us
Search Tips
ERIC Number: ED466638
Record Type: Non-Journal
Publication Date: 2002-Apr-2
Pages: 10
Abstractor: N/A
Reference Count: N/A
The Accuracy of Pass/Fail Decisions in Random and Difficulty-Balanced Domain-Sampling Tests.
Schnipke, Deborah L.
A common practice in some certification fields (e.g., information technology) is to draw items from an item pool randomly and apply a common passing score, regardless of the items administered. Because these tests are commonly used, it is important to determine how accurate the pass/fail decisions are for such tests and whether fairly small, simple changes can be made to such tests to improve their psychometric properties without being too large a burden on the testing program. This simulation study compared a random test with a difficulty-balanced version of the test using 4 tests simulated for each of 1,000 examinees. Ability estimates and pass/fail decisions were slightly less accurate for the random test than for the difficulty-balanced test. The difference in difficulty distributions between the two test designs was dramatic. In addition, test takers who ended up failing the random test had harder tests on average than test takers in the difficulty-balanced test design, and those who ended up passing the random test had easier tests on average than test takers in the difficulty-balanced test design. From a fairness (and legal defensibility) point of view, the random test design is very undesirable. However, balancing tests on item difficulty is relatively easy to do. And this produced tests that are of equal difficulty across test takers and ability estimates and pass/fail decisions that are more accurate. Based on these results, it is recommended that testing programs that currently use random test designs switch to a difficulty-balanced test design. (Author/SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A