NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ785796
Record Type: Journal
Publication Date: 2008-Mar
Pages: 15
Abstractor: Author
Reference Count: 29
ISSN: ISSN-0969-594X
How Important Is Content in the Ratings of Essay Assessments?
Shermis, Mark D.; Shneyderman, Aleksandr; Attali, Yigal
Assessment in Education: Principles, Policy & Practice, v15 n1 p91-105 Mar 2008
This study was designed to examine the extent to which "content" accounts for variance in scores assigned in automated essay scoring protocols. Specifically it was hypothesised that certain writing genre would emphasise content more than others. Data were drawn from 1668 essays calibrated at two grade levels (6 and 8) using "e-rater[TM]", an automated essay scoring engine with established validity and reliability. "E-rater" v 2.0's scoring algorithm divides 12 variables into "content" (scores assigned to essays with similar vocabulary; similarity of vocabulary to essays with the highest scores) and "non-content" (grammar, usage, mechanics, style, and discourse structure) related components. The essays were classified by genre: persuasive, expository, and descriptive. The analysis showed that there were significant main effects due to grade, F(1,1653) = 58.71, p less than 0.001, and genre F(2, 1653) = 20.57, p less than 0.001. The interaction of grade and genre was not significant. Eighth grade students had significantly higher mean scores than sixth grade students and descriptive essays were rated significantly higher than those classified as persuasive or expository. Prompts elicited "content" according to expectations with lowest proportion of content variance in persuasive essays, followed by expository and then descriptive. Content accounted for approximately 0-6% of the overall variance when all predictor variables were used. It accounted for approximately 35-58% of the overall variance when "content" variables alone were used in the prediction equation. (Contains 9 tables, 2 figures and 2 notes.)
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site:
Publication Type: Journal Articles; Reports - Evaluative
Education Level: Grade 6; Grade 8
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A