ERIC Number: ED347186
Record Type: RIE
Publication Date: 1992-Apr
Reference Count: 0
Reliability of Professionally Scored Data: NAEP-Related Issues.
Kaplan, Bruce A.; Johnson, Eugene G.
Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are examined, including measures of the reliability between raters within a year, the reliability of groups of raters between years, and other parameters. Measures examined are: (1) percent exact agreement; (2) Cohen's kappa statistic; (3) intraclass correlation coefficient; and (4) minimum variance unbiased estimator of intraclass correlation. These issues have been important in the National Assessment of Educational Progress (NAEP), and are addressed in this paper through the use of NAEP writing data from 1984 through 1990. Eight data tables and 16 references are included. (Author/SLD)
Descriptors: Educational Assessment, Elementary Secondary Education, Estimation (Mathematics), Evaluators, Interrater Reliability, Mathematical Models, Measurement Techniques, Multiple Choice Tests, National Surveys, Scores, Scoring, Standardized Tests, Test Construction, Test Reliability, Testing Problems, Writing Tests
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Identifiers: Intraclass Correlation; Kappa Coefficient; National Assessment of Educational Progress; Subjective Evaluation
Note: Paper presented at the Annual Meeting of the American Educational Research Association (San Francisco, CA, April 20-24, 1992).