ERIC Number: ED193307
Record Type: RIE
Publication Date: 1978-Mar
Reference Count: 0
Measures of Scorer Agreement and Patterns of Disagreement.
Johnson, Eugene G.; And Others
The scoring scheme of the National Assessment of Educational Progress (NAEP) is studied, as well as techniques for estimating its reliability. NAEP data are collected on students of age 13 in the fall, age 9 in the winter, and age 17 in the spring. The scoring task, involving hand scoring of open-ended items, may vary between 25,000 responses to 10 items and 1,000,000 responses to 400 items. The size of the scoring staff may vary between two and 20 scorers, and their work is subject to 10 percent quality control over the entire scoring period. The general scoring scheme requires that responses be divided into two overall evaluative categories: acceptable and unacceptable. Acceptable responses provide some evidence that the respondent meets the relevant educational objective; unacceptable responses are not adequate to provide evidence. A discussion of techniques for estimating reliability of the scoring scheme involves: (1) determining the percent of observed agreement among scorers and comparing it to the agreement one would expect on the basis of chance alone; and (2) examining the patterns of disagreement in the scoring of individuals. The test statistics for measuring disagreement are applied to data from multi-way tables. (RL) Primary type of information provided by report: Procedures (Scoring) (Evaluation).
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers: Interrater Reliability; National Assessment of Educational Progress
Note: Paper presented at the Annual Meeting of the American Educational Research Association (62nd, Toronto, Ontario, Canada, March 27-31, 1978).