ERIC Number: EJ1109652
Record Type: Journal
Publication Date: 2015-Jun
Pages: 16
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-2330-8516
EISSN: N/A
Automated Trait Scores for "TOEFL"® Writing Tasks. Research Report. ETS RR-15-14
Attali, Yigal; Sinharay, Sandip
ETS Research Report Series, Jun 2015
The "e-rater"® automated essay scoring system is used operationally in the scoring of "TOEFL iBT"® independent and integrated tasks. In this study we explored the psychometric added value of reporting four trait scores for each of these two tasks, beyond the total e-rater score.The four trait scores are word choice, grammatical conventions, fluency and organization, and content. Trait scores were computed on the basis of several criteria for determining feature weights: regression parameters of the trait features on human scores, reliability of trait features, and coefficients of features from a principal component analysis. In addition, augmented trait scores, based on information from other trait scores, were also analyzed.The psychometric added value of trait scores beyond total e-rater scores was evaluated by comparing the ability to predict a particular trait score on one task from the same trait score on the other task versus the e-rater score on the other task. Results supported the use of trait scores, and are discussed in terms of their contribution to the construct validity of e-rater as an alternative essay scoring method.
Descriptors: Writing Tests, Scores, Language Tests, English (Second Language), Second Language Learning, Essays, Writing Evaluation, Computer Assisted Testing, Evaluators, Evaluation Criteria, Regression (Statistics), Psychometrics, Prediction, Comparative Analysis, Construct Validity, Scoring, Correlation, Factor Analysis, Weighted Scores
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Test of English as a Foreign Language
Grant or Contract Numbers: N/A