NotesFAQContact Us
Search Tips
ERIC Number: ED525309
Record Type: Non-Journal
Publication Date: 2011-Aug
Pages: 23
Abstractor: As Provided
Reference Count: 10
A Differential Word Use Measure for Content Analysis in Automated Essay Scoring. Research Report. ETS RR-11-36
Attali, Yigal
Educational Testing Service
This paper proposes an alternative content measure for essay scoring, based on the "difference" in the relative frequency of a word in high-scored versus low-scored essays. The "differential word use" (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high-scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE[R] issue), "across" different prompts. Evaluation results across four GRE and TOEFL[R] tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt-specific content analysis measures of e-rater[R]. Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e-rater features. This measure provides a way to conduct a task-level content analysis that is related to the prompt-specific content analysis of e-rater but does not require prompt-specific training. Regression results for the prompt-level DWU show that it can be used as a replacement to prompt-specific content-vector analysis (CVA) features. Appended are: (1) Examples of Extreme d Values for the GRE Issue Task; and (2) Examples of Extreme d Values for the GRE Argument Task. (Contains 8 tables.)
Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site:
Publication Type: Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Educational Testing Service
Identifiers - Assessments and Surveys: Graduate Record Examinations; Test of English as a Foreign Language