NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ934579
Record Type: Journal
Publication Date: 2011
Pages: 20
Abstractor: As Provided
Reference Count: 53
ISSN: ISSN-0969-594X
Using Verbal Reports to Explore Rater Perceptual Processes in Scoring: A Mixed Methods Application to Oral Communication Assessment
Joe, Jilliam N.; Harmes, J. Christine; Hickerson, Corey A.
Assessment in Education: Principles, Policy & Practice, v18 n3 p239-258 2011
In recent years, there has been a growth in the use of rater cognitive data to inform test development and validity arguments. In this study, we examined differences in feature attention and categorisation between experienced and inexperienced raters for a college-level assessment of oral communication. The focus was two-fold: (a) rater cognition as it informs rubric development; and (b) impact of verbal protocols on data collection in a speech scoring context. The findings of the study indicate that the complexity of the scoring rubric may be as much of a threat to scoring consistency as raters' pre-existing cognitive frameworks. A next step in the development of this tool is to simplify the rubric by identifying the critical features that most clearly define the construct, and modify training to address the cognitive influences raters bring to scoring. The results did not indicate a verbal report main effect on scoring. (Contains 1 note, 1 table, and 3 figures.)
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A