NotesFAQContact Us
Search Tips
ERIC Number: ED498534
Record Type: Non-Journal
Publication Date: 2006-Apr
Pages: 21
Abstractor: Author
Empirical vs. Expected IRT-Based Reliability Estimation in Computerized Multistage Testing (MST)
Zhang, Yanwei; Breithaupt, Krista; Tessema, Aster; Chuah, David
Online Submission, Paper presented at the Annual Conference of the National Council of Measurement in Education (San Francisco, CA, Apr 2006)
Two IRT-based procedures to estimate test reliability for a certification exam that used both adaptive (via a MST model) and non-adaptive design were considered in this study. Both procedures rely on calibrated item parameters to estimate error variance. In terms of score variance, one procedure (Method 1) uses the empirical ability distribution from a particular sample of examinees, and the other procedure (Method 2) assumes a normal distribution of ability and is sample-free. Due to the problem of sampling restriction in adaptive tests, Method 1 was modified (Method 1 extension) to "beef up" the sample and estimate reliability for each testlet in a MST panel before aggregating the estimates into an overall estimate for a test form or route. Overall, results imply that Method 1 and Method 2 tend to produce similar results for both adaptive and non-adaptive tests on the panel level and the test section level. Method 1 should not be applied to individual test forms in adaptive tests by a MST design. In the latter case, the modified or extended procedure can be used to alleviate the problem of restricted sample. The algorithms of the discussed procedures can be implemented in common statistical programming language such as SAS and SPSS as flexible alternatives to the theoretical and empirical reliability estimates computed, for example, by the BILOG-MG software. (Contains 6 tables and 1 figure.)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A