NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1412378
Record Type: Journal
Publication Date: 2023
Pages: 32
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0125-2488
EISSN: EISSN-2287-0024
Detecting Differential Rater Severity in a High-Stakes EFL Classroom Writing Assessment: A Many-Facets Rasch Measurement Approach
Apichat Khamboonruang
PASAA: Journal of Language Teaching and Learning in Thailand, v66 spec iss p5-36 2023
Differential rater severity (DRS), one prevalent case of differential rater functioning (aka rater bias or rater interaction) effects, manifests itself when a rater assigns unusually severe or lenient ratings, threatening the validity and fairness of rater-mediated assessment. Building on a many-facets Rasch measurement (MFRM) approach, this study aimed to detect whether teachers exercised DRS towards rating criteria and student subgroups (classroom, proficiency, and gender) in a high-stakes EFL classroom writing examination. Data were collected from three teachers who applied a four-point five-criteria analytic rubric to rate opinion essays written by 42 English-major undergraduates during the examination. Main findings revealed that the teachers were not uniform in their severity levels, with the most experienced teacher exhibiting the highest severity and the least experienced teacher exercising the lowest severity. Whilst the most experienced and most severe teacher exposed slight DRS towards student genders, the less experienced and less severe teachers exerted substantial DRS in reverse pattern towards rating criteria and student classrooms. Surprisingly, the less experienced teachers scored their own classroom students less severely but marked each other's classroom students more severely than expected. The current findings raise the attention and awareness of teachers, educators, and policymakers concerning the impact of rater effects on the validity and fairness of rater-mediated assessment in the classroom context.
Chulalongkorn University Language Institute. Prem Purachatra Building, Chulalongkom University, Phayathai Road, Pathumwan, Bangkok 10330, Thailand. Tel: +66-2-218-6092; Fax: +66-2-218-6104; e-mail: pasaa.editor@gmail.com; Web site: https://www.culi.chula.ac.th/en/pasaa/1
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Thailand
Grant or Contract Numbers: N/A