ERIC Number: ED421533
Record Type: Non-Journal
Publication Date: 1998-Apr
Reference Count: N/A
A Demonstration of Using Person-Fit Statistics in Standard Setting.
Bay, Luz; Nering, Michael L.
The use of person-fit methods to determine the extent to which a panelist's ratings fit the item response theory (IRT) models used in the National Assessment of Educational Progress (NAEP) is demonstrated. Person-fit methods are statistical methods that allow the identification of nonfitting response vectors. To determine whether panelists' ratings fit the IRT models used in the NAEP, the l(z) statistic (F. Drasgow, M. Levine, and E. Williams, 1985) was used. Rating data from the 1994 NAEP achievement level setting process were obtained for grade 12 geography, for which 29 panelists (primarily teachers) set levels. A response vector was created for each panelist for each achievement level using each of three p-value criteria and simulated item score string estimation (ISSE) values were created. The l(z) statistic was calculated for each of the 27 response vectors associated with each of the 29 panelists. Means and standard deviations of the l(z) distributions were computed for each cell of the experimental design and are presented in table form. Typically, they indicated that the simulated ISSE ratings or response vectors underfit the model. The results of this study provide preliminary information about the use of person-fit statistics in standard setting and are the basis for additional planned studied. (Contains 3 tables, 3 figures, and 15 references.) (SLD)
Publication Type: Reports - Descriptive; Speeches/Meeting Papers
Education Level: N/A
Sponsor: National Assessment Governing Board, Washington, DC.
Authoring Institution: ACT, Inc., Iowa City, IA.
Identifiers - Assessments and Surveys: National Assessment of Educational Progress