NotesFAQContact Us
Collection
Advanced
Search Tips
ERIC Number: ED421531
Record Type: Non-Journal
Publication Date: 1995-Apr-22
Pages: 43
Abstractor: N/A
Reference Count: N/A
ISBN: N/A
ISSN: N/A
Scoring with Nominal Missing-Response Parameters: Its Effects on Achievement Levels Set on the National Assessment of Educational Progress (NAEP).
Bay, Luz; Loomis, Susan Cooper; Wang, Tianyou
This study examines the effects on the National Assessment of Educational Progress (NAEP) achievement levels of using item response theory (IRT) models that have nominal missing-response parameters. It compares cutpoints based on item parameters that were fitted using two different models. The first set of cutpoints were based on parameters for the two- and three-parameter logistic model, and the second set of cutpoints were based on R. Bock's (1972) nominal model. Data are from the 1992 NAEP in mathematics and reading for grade 12. For reading, data included 1,966 responses to a block of items, and for mathematics, data included 2,192 responses. Other data were the item-by-item ratings by each panelist who participated in the Achievement Levels Setting process for NAEP 1992. For each subject, the cutpoints set using the probability curves obtained by the different models were compared. The percent of students scoring at each level were also compared. For reading, the logistic model, when fitted to the data, converged in 25 iterations and yielded a marginal reliability of 0.67 with maximum information of 4.3 at theta equals -0.5. The nominal model converged in 88 iterations and yielded a marginal reliability of 0.85 with a maximum information of 14.2 at theta equals 0. For mathematics, the logistic model converged in 184 iterations and yielded a marginal reliability of 0.61 with maximum information of 23.9 at theta equals 1.5. The nominal model, with mathematics data, converged in 46 iterations, and yielded a marginal reliability of 0.62, with maximum information of 7.3 at theta equals -0.2. When the percentages of students scoring at or above each cutpoint were compared, none scored at the Advanced level using the nominal model. These preliminary results suggest the direction of future studies, but cannot be generalized to the NAEP assessment program. An appendix contains achievement levels descriptions. (Contains three tables, six figures, and five references.) (SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: National Assessment of Educational Progress