ERIC Number: EJ1083348
Record Type: Journal
Publication Date: 2015
Abstractor: As Provided
Reference Count: 50
What Are We Aligning Tests to When We Report Test Alignment to the CEFR?
Harsch, Claudia; Hartig, Johannes
Language Assessment Quarterly, v12 n4 p333-362 2015
The study reported here investigates the validity of judgments made when aligning tests to the "Common European Framework of Reference" (CEFR). Listening tests operationalizing pre-defined difficulty-determining characteristics were to be aligned to CEFR levels. We employed a modified version of the item-descriptor-matching-method. Ten judges stated the CEFR descriptors they thought each item operationalizes and assigned a global CEFR level per item. We compared agreement on CEFR level judgments and CEFR descriptors quoted. Analyzing the relationship between CEFR level judgments and item ratings of difficulty-determining characteristics shed light on further criteria the judges employed. Follow-up interviews helped to triangulate the findings by examining judges' perceptions of the alignment procedure. We found that judges relied on different criteria and CEFR descriptors to a varying degree, interpreting CEFR levels differently. There seemed little comparability in what aspects judges used to form their global CEFR judgments. Therefore, if an alignment does not take into account the meaning of the CEFR levels as expressed by existing CEFR descriptors, this raises issues with alignment validity, and hence the validity of test-score interpretation and usage. Given the impact of using CEFR aligned tests for high-stakes purposes, this article aims to shed more light on what assigning a CEFR level to a test actually means.
Descriptors: Foreign Countries, Listening Comprehension Tests, Language Tests, Alignment (Education), Difficulty Level, Reading Comprehension, Reading Tests, Judges, Second Language Instruction, Statistical Analysis, Semi Structured Interviews
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research; Tests/Questionnaires
Education Level: N/A
Authoring Institution: N/A
Identifiers - Location: Germany