NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1202928
Record Type: Journal
Publication Date: 2018-Dec
Pages: 33
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-2330-8516
Understanding Mean Score Differences between the "e-rater"® Automated Scoring Engine and Humans for Demographically Based Groups in the "GRE"® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12
Ramineni, Chaitanya; Williamson, David
ETS Research Report Series, Dec 2018
Notable mean score differences for the "e-rater"® automated scoring engine and for humans for essays from certain demographic groups were observed for the "GRE"® General Test in use before the major revision of 2012, called rGRE. The use of e-rater as a check-score model with discrepancy thresholds prevented an adverse impact on the examinee score at the item or test level. Despite this control, there remains a need to understand the root causes of these demographically based score differences and to identify potential mechanisms for avoiding future instances of discrepancy. In this study, we used a combination of statistical methods and human review to propose hypotheses about the root cause of score differences and whether such discrepancies reflect inadequacies of e-rater, human scoring, or both. The human rating process was found to be influenced strongly by the scale structure and did not fully correspond to the e-rater scoring mechanism. The human raters appeared to be using conditional logic and a rule-based approach to their scoring, while e-rater uses linear weighting of all the features. These analyses have implications for future research and operational policies for the scoring of the rGRE.
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: China; India; Japan; South Korea; Taiwan
Identifiers - Assessments and Surveys: Graduate Record Examinations