ERIC Number: EJ1071217
Record Type: Journal
Publication Date: 2015
Pages: 21
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1543-4303
EISSN: N/A
Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test
Cai, Hongwen
Language Assessment Quarterly, v12 n3 p262-282 2015
This study is an attempt to classify raters according to their weighting patterns and explore systematic differences between rater types in the rating process. In the context of an EFL speaking test, 126 raters were classified into three types--form-oriented, balanced, and content-oriented--through cluster analyses of their weighting patterns derived from a holistic judgment task. In subsequent verbal protocols, a smaller sample of raters from each type rated the performance of real test takers and justified their ratings. The procedure for classifying raters yielded reliable results that agreed broadly with self-perceived and actual weights of the raters in the rating process. Verbal protocol analyses found systematic differences across rater types in the distinction between criteria and conformity to rubric-implied criteria. The form-oriented raters were found to overweigh form-related criteria and underweigh content-related criteria, which suggested unbalanced construct representation and led to more halos. Preliminary evidence also suggested that the form-oriented raters might overrate digressive content due to lower sensitivity to digression.
Descriptors: Classification, Language Tests, English (Second Language), Second Language Learning, Evaluators, Oral Language, Protocol Analysis, Evaluation Criteria, Qualitative Research, Weighted Scores, Multivariate Analysis, Coding, Regression (Statistics), Foreign Countries, Majors (Students)
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: China
Grant or Contract Numbers: N/A