ERIC Number: ED488734
Record Type: Non-Journal
Publication Date: 2005-Sep
Using IRT DIF Methods to Evaluate the Validity of Score Gains. CSE Technical Report 660
Koretz, Daniel M.; McCaffrey, Daniel F.
National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
Given current high-stakes uses of tests, one of the most pressing and difficult problems confronting the field of measurement is to develop better methods for distinguishing between meaningful gains in performance and score inflation. This study explores the potential usefulness of adapting differential item functioning (DIF) techniques for this purpose. It distinguishes between reactive and nonreactive changes in DIF over time and relates these to the framework for validating scores under high-stakes conditions offered by Koretz, McCaffrey, and Hamilton (2001). It contrasts score-anchored and item-anchored approaches to DIF in terms of their potential for this purpose. It explored changes in the distribution of DIF in the NAEP eighth-grade mathematics assessment between 1990 and 2000 in five low-gain and five high-gain states, in each case treating all other participating states as the reference group. It used the score-anchored method of DIF analysis implemented in BILOG-MG (Bock & Zimowski, 2003b), which allows only item difficulties to vary across groups. This exploration indicated that the approach has potential but confronts several substantial difficulties. Further exploration using data from high-stakes testing programs is recommended.
Descriptors: Item Response Theory, High Stakes Tests, Achievement Gains, Scores, Grade 8, National Competency Tests, Test Items, Mathematics Tests, Item Analysis
Center for the Study of Evaluation (CSE)/National Center for Research on Evaluation, Standards, and Student Testing (CRESST), 300 Charles E. Young Drive North, GSE&IS Bldg., 3rd Flr./Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532.
Publication Type: Reports - Research
Education Level: N/A
Sponsor: Institute of Education Sciences (ED), Washington, DC.
Authoring Institution: California Univ., Los Angeles. Center for the Study of Evaluation.