NotesFAQContact Us
Search Tips
ERIC Number: ED505571
Record Type: Non-Journal
Publication Date: 2009-Jan
Pages: 41
Abstractor: As Provided
Reference Count: 18
Evaluating the Construct-Coverage of the e-rater[R] Scoring Engine. Research Report. ETS RR-09-01
Quinlan, Thomas; Higgins, Derrick; Wolff, Susanne
Educational Testing Service
This report evaluates the construct coverage of the e-rater[R[ scoring engine. The matter of construct coverage depends on whether one defines writing skill, in terms of process or product. Originally, the e-rater engine consisted of a large set of components with a proven ability to predict human holistic scores. By organizing these capabilities into features, e-rater researchers organized the e-rater engine along the lines of trait scoring, which recognizes that essay quality has several dimensions. Some traits of essay quality cut across different methods for scoring essay quality, such as the rubrics employed by the GRE[R] and TOEFL[R] assessments, as well as the 6-trait scoring model. Factor analyses conducted by Attali and Powers (2008) suggest that e-rater features capture low-level aspects of essay quality, such as sentence complexity, vocabulary, and conventions. Future e-rater development should focus on (a) deepening and expanding coverage of the construct, such as by developing measures of essay content and organization, as well as on (b) addressing accuracy issues in existing features. (Two appendices are included: (1) Scoring Guides for the Graduate Record Examinations[R] (GRE[R]), Test of English as a Foreign Language[TM] (TOEFL[R]), SAT[R]; and National Assessment of Educational Progress (NAEP); and (2) Glossary of e-rater Microfeatures. Contains 2 figures and 1 tables.)
Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site:
Publication Type: Reports - Evaluative
Education Level: Elementary Secondary Education; Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Educational Testing Service
Identifiers - Assessments and Surveys: Graduate Record Examinations; Test of English as a Foreign Language