NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ875319
Record Type: Journal
Publication Date: 2010
Pages: 21
Abstractor: As Provided
ISSN: ISSN-1543-4303
Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience
Barkaoui, Khaled
Language Assessment Quarterly, v7 n1 p54-74 2010
Various factors contribute to variability in English as a second language (ESL) essay scores and rating processes. Most previous research, however, has focused on score variability in relation to task, rater, and essay characteristics. A few studies have examined variability in essay rating processes. The current study used think-aloud protocols to examine the roles of rating scales, rater experience, and interactions between them in variability in raters' decision-making processes and the aspects of writing they attend to when reading and rating ESL essays. The study included 11 novice and 14 experienced raters, who each rated 12 ESL essays, both holistically and analytically, while thinking aloud. The findings indicated that rating scale type had larger effects on the participants' rating processes than did rater experience. With holistic scoring, raters tended to refer more often to the essay (the focus of the assessment), whereas with analytic scoring they tended to refer to the rating scale (the source of evaluation criteria) more frequently; analytic scoring drew raters' attention to all evaluation criteria in the rating scale, and novices were influenced by variation in rating scales more than were the experienced raters. The article concludes with implications for essay rating practices and research. (Contains 8 tables and 4 footnotes.)
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Adult Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A