ERIC Number: ED467810
Record Type: Non-Journal
Publication Date: 1999-Mar
Reference Count: N/A
A Bayesian Method for the Detection of Item Preknowledge in CAT. Law School Admission Council Computerized Testing Report. LSAC Research Report Series.
McLeod, Lori D.; Lewis, Charles; Thissen, David.
With the increased use of computerized adaptive testing, which allows for continuous testing, new concerns about test security have evolved, one being the assurance that items in an item pool are safeguarded from theft. In this paper, the risk of score inflation and procedures to detect test takers using item preknowledge are explored. When test takers use preknowledge of items, their item responses may deviate from the underlying item response theory (IRT) model, and estimated abilities may be inflated. This deviation may be detected through the use of person-fit indices. A Bayesian posterior log odd ratio index is proposed for detecting the use of item preknowledge. In this approach to person-fit, the estimated probability that each test taker has preknowledge of items is updated after each item response. These probabilities are based on IRT parameters, a model specifying the probability that each item has been memorized, and the test takers item responses. Simulations based on an operational computerized adaptive test pool were used to demonstrate the risk of item preknowledge to test security and the use of the odds ratio index. An appendix discusses the three classes of models used in the study. (Author/SLD)
Publication Type: Reports - Research
Education Level: N/A
Authoring Institution: Law School Admission Council, Princeton, NJ.
Identifiers - Assessments and Surveys: Law School Admission Test