ERIC Number: ED338653
Record Type: RIE
Publication Date: 1991-Jun-6
Reference Count: N/A
The Validity and Credibility of the Achievement Levels for the 1990 National Assessment of Educational Progress in Mathematics.
Linn, Robert L.; And Others
The statute authorizing the National Assessment of Educational Progress (NAEP) calls for the National Assessment Governing Board (NAGB) to set appropriate achievement levels in all areas and grades tested by the NAEP. These levels are intended to establish what students should know, not just what they do know. In 1990, the NAEP posited three achievement levels (advanced, proficient, and basic) at each grade level assessed (grades 4, 8, and 12). This preliminary report examines the reasonableness and technical adequacy of the achievement levels, addressing the fundamental question of whether the levels are adequate for supporting the conclusions or inferences for which they will be used. Analyses indicate that the achievement levels are seriously flawed. Robustness appears to be inadequate in two respects: (1) interrater reliability is inadequate; and (2) the coherence of rating across the three grades has not been established. The available evidence does not address the question of whether the achievement levels were actually criterion-based, although there are suggestions that they were not. It is suggested that practical post hoc adjustments will not remedy these problems. It is recommended that these achievement levels should not be used in any public reporting of state or national NAEP results. Fourteen tables and 22 figures present results of the analyses. (SLD)
Descriptors: Academic Achievement, Academic Standards, Credibility, Educational Assessment, Elementary Secondary Education, Grade 12, Grade 4, Grade 8, Interrater Reliability, Knowledge Level, Mathematics Achievement, National Programs, Robustness (Statistics), Standard Setting (Scoring), Test Results, Testing Programs, Validity
Publication Type: Reports - Evaluative
Education Level: N/A
Sponsor: National Center for Education Statistics (ED), Washington, DC.
Authoring Institution: Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
Identifiers: National Assessment of Educational Progress
Note: Paper prepared in collaboration with the University of Colorado, the NORC at the University of Chicago, and Arizona State University.