ERIC Number: ED417204
Record Type: Non-Journal
Publication Date: 1997-Nov
Reference Count: N/A
Optimal Rating Procedures and Methodology for NAEP Open- Ended Items. Working Paper Series.
Patz, Richard J.; Wilson, Mark; Hoskens, Machteld
The National Assessment of Educational Progress (NAEP) collects data in the form of repeated, discrete measures (test items) with hierarchical structure for both measures and subjects, that is complex by any standard. This complexity has been managed through a "divide and conquer" approach of isolating and evaluating sources of variability one at a time, using a sequence of relatively simple analyses. The cost of this simplicity for the NAEP has been limits on the propagation of information from one subanalysis to another. This has made some questions that would be relatively straightforward to address in ordinary circumstances, quite difficult to answer for the NAEP. This study considers NAEP's fragmented analysis of errors in the rating of open-ended responses, develops methodology for more unified analyses, and applies the methodology to the analysis of rater effects in NAEP data. How to minimize rater effects using modern imaging technology is studied, and conclusions and recommendations are drawn in light of these analyses. (Contains 15 figures, 13 tables, and 30 references.) (SLD)
Descriptors: Data Analysis, Data Collection, Elementary Secondary Education, Error Patterns, National Surveys, Research Methodology, Tables (Data), Test Items
U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, 555 New Jersey Avenue, N.W., Room 400, Washington, DC 20208-5654.
Publication Type: Numerical/Quantitative Data; Reports - Evaluative
Education Level: N/A
Sponsor: National Center for Education Statistics (ED), Washington, DC.
Authoring Institution: California Univ., Berkeley.; CTB / McGraw-Hill, Monterey, CA.
Identifiers - Assessments and Surveys: National Assessment of Educational Progress