ERIC Number: EJ934593
Record Type: Journal
Publication Date: 2011
Pages: 24
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0969-594X
EISSN: N/A
The Interrelations of Features of Questions, Mark Schemes and Examinee Responses and Their Impact upon Marker Agreement
Black, Beth; Suto, Irenka; Bramley, Tom
Assessment in Education: Principles, Policy & Practice, v18 n3 p295-318 2011
In this paper we develop an evidence-based framework for considering many of the factors affecting marker agreement in GCSEs and A levels. A logical analysis of the demands of the marking task suggests a core grouping comprising: (i) question features; (ii) mark scheme features; and (iii) examinee response features. The framework synthesises several strands of previous empirical research. Additionally, we present new data, collected by observing the social dynamics of examiner meetings, to illustrate some effects of these interactions. We use our framework to argue that the two most important goals of research into marker agreement are "prediction" and "control." That is, assessment agencies need to be able not only to predict the level of agreement between a marker's mark and the "definitive" mark, but also to anticipate the effects of different possible courses of action on the level of agreement. We conclude with a list of possible courses of action, considering their effect not only on marking accuracy, but also on overall validity--the overriding concern. (Contains 3 tables and 5 figures.)
Descriptors: Interrater Reliability, Grading, Scoring, High Stakes Tests, Test Items, Prediction, Influences, Models, Evidence, Exit Examinations, Foreign Countries
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Evaluative
Education Level: Elementary Secondary Education; Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: United Kingdom
Grant or Contract Numbers: N/A