NotesFAQContact Us
Collection
Advanced
Search Tips
ERIC Number: ED574287
Record Type: Non-Journal
Publication Date: 2015
Pages: 44
Abstractor: As Provided
Reference Count: 40
ISBN: N/A
ISSN: N/A
Improving Teacher Selection: The Effect of Inter-Rater Reliability in the Screening Process. CEDR Working Paper. WP #2015-7
Martinkova, Patricia; Goldhaber, Dan
Center for Education Data & Research
Inter-rater reliability, commonly assessed by intra-class correlation coefficient ICC, is an important index for describing the extent to which there is consistency amongst two or more raters in assigned measures. In organizational research, the data structure is often hierarchical and designs deviate substantially from the ideal of a balanced (fully crossed or nested) design. Also, often it is necessary to include covariates in the model, making it impossible to use traditional correlation-based or analysis of variance (ANOVA)-based methods for estimation of inter-rater reliability. We advocate the use of hierarchical (mixed effect model)-based methods, where variance components can be estimated by restricted maximum likelihood or Bayesian approaches. In this work, we use data from teacher hiring in Spokane public schools to demonstrate the usage of hierarchical (mixed effect) models to estimate inter-rater reliability and to demonstrate how reliability can be estimated with more complex data structures. We generally find low levels of inter-rater reliability, though this overall reliability varies according to whether the measure is assessed "across' or "within" schools, or even within job openings. We also find evidence that inter-rater reliability of some subcomponents of the hiring rubric varies according to the type of position to which applicants are applying, or according to whether the applicant is internal or from outside of the district. The direct effect of reliability on predictive power of the selection instrument is demonstrated and policy implications for public school hiring are discussed. [Additional support for this study was provided by the Czech Science Foundation through grant #15-15856Y.]
Center for Education Data & Research. 3876 Bridge Way North Suite 201, Seattle, WA 98103. Tel: 206-547-5585; Fax: 206-547-1641; e-mail: cedr@uw.edu; Web site: http://www.cedr.us
Publication Type: Reports - Research; Numerical/Quantitative Data
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); National Center for Analysis of Longitudinal Data in Education Research (CALDER) at American Institutes for Research
Authoring Institution: Center for Education Data & Research (CEDR)
Identifiers - Location: Washington
IES Funded: Yes
Grant or Contract Numbers: R305C130030|R305A060018