ERIC Number: ED464949
Record Type: Non-Journal
Publication Date: 2002-Apr
Reference Count: N/A
Applying Hierarchical Model Calibration to Automatically Generated Items.
Williamson, David M.; Johnson, Matthew S.; Sinharay, Sandip; Bejar, Isaac I.
This study explored the application of hierarchical model calibration as a means of reducing, if not eliminating, the need for pretesting of automatically generated items from a common item model prior to operational use. Ultimately the successful development of automatic item generation (AIG) systems capable of producing items with highly similar statistical properties may permit the development and implementation of adaptive on-the-fly testing. The study applied the related siblings model to mathematics item data from an experimental administration associated with a national testing program to explore the application of the model for calibrating operational data incorporating multiple items generated both from AIG and manual item generation. The sample consisted of 3,793 examinees in grade 8, distributed among 4 test forms. Results suggest that the inclusion of AIG generated items in item families will have a tendency to have item characteristic curves that are somewhat more variable than if the family consisted of the same item under repeated administration. However, this increased variability is neither assured, nor in most cases, even particularly pronounced. (Contains 2 figures, 1 table, and 13 references.) (SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: Educational Testing Service, Princeton, NJ.