NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1147239
Record Type: Journal
Publication Date: 2017
Pages: 14
Abstractor: As Provided
ISSN: ISSN-0307-5079
And the Winner Is … : Inter-Rater Reliability among Scholarship Assessors
Johnston, Lucy; Schluter, Philip J.
Studies in Higher Education, v42 n9 p1736-1749 2017
With increasing competition for postgraduate research scholarships, awarding processes demand attention and scrutiny. We examine inter-rater reliability for two prestigious New Zealand scholarships, the Shirtcliffe Fellowship and the Gordon Watson Scholarship. For each scholarship, five assessors (three academic; two non-academic) independently evaluate all applicants over three domains: Academic Merit, Quality of Study Plans and Character/Leadership. Data from years 2009 to 2014 were extracted, comprising 12 separate assessment rounds. Good to excellent agreement was observed for each scholarship in each year. Agreement was significantly higher for the Academic Merit domain compared to the other domains. Moreover, agreement among academics was higher and less variable than non-academics for this Academic Merit domain. No such differences were noted in the other domains. While resource efficiencies could be made, reductions in committee size resulted in poorer applicant selection performance. Applicants and donors alike can be confident that the awardee for these scholarships is a top applicant.
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: New Zealand
Grant or Contract Numbers: N/A