Download full text
Download full text
ERIC Number: EJ1124758
Record Type: Journal
Publication Date: 2016-Feb
Abstractor: As Provided
Building "e-rater"® Scoring Models Using Machine Learning Methods. Research Report. ETS RR-16-04
Chen, Jing; Fife, James H.; Bejar, Isaac I.; Rupp, André A.
ETS Research Report Series, Feb 2016
The "e-rater"® automated scoring engine used at Educational Testing Service (ETS) scores the writing quality of essays. In the current practice, e-rater scores are generated via a multiple linear regression (MLR) model as a linear combination of various features evaluated for each essay and human scores as the outcome variable. This study evaluates alternative scoring models based on several additional machine learning algorithms, including support vector machines (SVM), random forests (RF), and "k"-nearest neighbor regression (k-NN). The results suggest that models based on the SVM algorithm outperform MLR models in predicting human scores. Specifically, SVM-based models yielded the highest agreement between human and e-rater scores. Furthermore, compared with MLR, SVM-based models improved the agreement between human and e-rater scores at the ends of the score scale. In addition, the high correlation between SVM-based e-rater scores with external measures such as examinee's scores on the other parts of the test provided some validity evidence for SVM-based e-rater scores. Future research is encouraged to explore the generalizability of these findings.
Descriptors: Scoring, Models, Artificial Intelligence, Automation, Multiple Regression Analysis, Correlation, Essay Tests
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A