ERIC Number: ED453227
Record Type: Non-Journal
Publication Date: 2001-Apr
Reference Count: N/A
A Comparison of the Standardization and IRT Methods of Adjusting Pretest Item Statistics Using Realistic Data.
Chang, Shun-Wen; Hanson, Bradley A.; Harris, Deborah J.
The requirement of large sample sizes for calibrating items based on item response theory (IRT) models is not easily met in many practical pretesting situations. Although classical item statistics could be estimated with much smaller samples, the values may not be comparable across different groups of examinees. This study extended the authors' AERA 2000 by further exploring the standardization method and comparing its effectiveness with the one-parameter (1PL) and three-parameter (3PL) logist IRT models in adjusting pretest item statistics with small sample sizes, using more realistic data than the previous study. Based on the realistic data generated from a 50-dimensional multidimensional IRT model, the 3PL model performed better than the 1PL or standardization method in recovering both the population p-values and point biserial correlations. The standardization method outperformed the 1PL model in recovering the population point biserial, correlations but not in recovering the population p-values. The performance of the methods was also evaluated using the real pretest data of a high-stakes test. In terms of recovering the p-values and point biserial correlations for the real data, the 1PL model produced the most satisfactory results. The 3PL model performed worst in terms of recovering the p-values for the real data, and the standardization method performed worst in recovering the point biserial correlations for the real data. Due to the very limited number of conditions studied, one must be cautious about making conclusions about the standardization method relative to IRT methods based on these studies. The standardization method appears to be a viable alternative to IRT methods that may be simpler to implement, although these results do not suggest that it will produce more accurate results. (Contains 8 tables and 13 references.) (Author/SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A