NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ914060
Record Type: Journal
Publication Date: 2011-Jan
Pages: 25
Abstractor: As Provided
Reference Count: 43
ISSN: ISSN-0265-5322
Think-Aloud Protocols in Research on Essay Rating: An Empirical Study of Their Veridicality and Reactivity
Barkaoui, Khaled
Language Testing, v28 n1 p51-75 Jan 2011
Think-aloud protocols (TAPs) are frequently used in research on essay rating processes. However, there are very few empirical studies of the completeness of TAP data and the effects of this technique on rater performance (i.e., rating processes and outcomes). This study aims to start to address this research gap. As part of a larger study on rater decision-making behaviors, 11 novice and 14 experienced raters rated, both analytically and holistically, a sample of ESL essays silently and while thinking aloud. The raters were then interviewed about their perceptions of thinking aloud and its effects. Essay scores were submitted to "FACETS" analyses, while TAP and interview data were analyzed qualitatively. Score and qualitative data analyses provided evidence and explanations concerning the veridicality and reactivity of TAPs across rater groups (novice vs. experienced) and rating scales (holistic vs. analytic). The paper concludes with several theoretical and methodological implications and questions for future studies using TAPs to build models of and compare essay rating processes across individuals, groups and contexts. (Contains 3 notes, 2 tables, and 1 figure.)
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail:; Web site:
Publication Type: Journal Articles
Education Level: Higher Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A