ERIC Number: ED458296
Record Type: Non-Journal
Publication Date: 2001-Apr
Reference Count: N/A
Computerized Scoring of Essays for Analytical Writing Assessments: Evaluating Score Validity.
Kelly, P. Adam
The purpose of this research was to establish, within the constraints of the methods presented, whether the computer is capable of scoring essays in much the same way that human experts rate essays. The investigation attempted to establish what was actually going on within the computer and within the mind of the rater and to describe the degree to which these processes equated. Revealing this parallelism depended on the careful assessment of the "intrinsic" aspects of validity as proposed by S. Messick (1995) of computerized essay scores. The focus was "e-rater" (TM), a computer-based essay scoring system developed by the Educational Testing Service. The study used 1,794 existing Graduate Record Examination Writing Assessment essays written and scored during recent test administration. Factor analysis and the advice of expert raters were used to guide the deconstruction of essay scoring models into subscore models corresponding to writing characteristics within the essay. The writing characteristics identified in this process were used as the basis for developing characteristic-specific scoring rubrics to be used by expert raters. Fresh essay samples were scored by expert rates, both holistically and characteristic-by-characteristic. The same essay samples were assigned both holistic scores and character-wise subscores by the computer scoring models. The degree of convergent validity of scores was evidenced by the proportion of agreement and strength of pairwise correlation among scores and subscores. The statistics derived in this study suggest that simpler e-rater models might do just as well at agreeing with the scores of expert rates, although the proportion of total variance in the expert rater scores explained by the e-rater scores might decrease from an already modest level. (Contains 7 tables and 30 references.) (Author/SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Sponsor: Educational Testing Service, Princeton, NJ. Graduate Record Examination Board Program.
Authoring Institution: N/A