NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1023944
Record Type: Journal
Publication Date: 2014
Pages: 25
Abstractor: As Provided
ISSN: ISSN-0162-6620
Leveraging Data Sampling and Practical Knowledge: Field Instructors' Perceptions about Inter-Rater Reliability Data
Soslau, Elizabeth; Lewis, Kandia
Action in Teacher Education, v36 n1 p20-44 2014
For accreditation and programmatic decision making, education school administrators use inter-rater reliability analyses to judge credibility of student-teacher assessments. Although weak levels of agreement between university-appointed supervisors and cooperating teachers are usually interpreted to indicate that the process is not being implemented with fidelity, it is unclear which part of the process contributes to weak agreements. Quantitative analysis of data ("N" = 230) from four semesters, and matched-pair sampling ("n" = 14), countered weaknesses found in usual analysis methods. Using "data-driven decision making," "personal practical knowledge" and "evidence-based practice" as theoretical frameworks, university-based field instructors' discussions about what accounted for varying correlations were analyzed. Qualitative data analysis (focus group/questionnaires) found that field instructors ("n" = 7) assumed divergent scores indicate weakness in evaluation processes and posited conflicting root causes. Inter-rater reliability analyses should include pair-wise sampling, so that weak and strong rates of agreement are unmasked and opportunities for meaningful data conversations are possible.
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A