NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED562838
Record Type: Non-Journal
Publication Date: 2014
Pages: 11
Abstractor: ERIC
Reference Count: 2
Teacher Evaluation in Practice: Understanding Evaluator Reliability and Teacher Engagement in Chicago Public Schools
Sporte, Susan E.; Jiang, Jennie Y.; Luppescu, Stuart
Society for Research on Educational Effectiveness
Starting in 2012-13, researchers have worked in partnership with Chicago Public Schools (CPS) and the Chicago Teachers Union (CTU) to study implementation of Chicago's new teacher evaluation system: Recognizing Educators Advancing Chicago's Students (REACH). This paper presents findings as well as experiences from the collaboration with CPS and the CTU as they seek to identify real-time strategies for improving implementation. The paper focuses on findings from two key components for implementing a new evaluation system: the reliability of evaluators, and teacher perceptions of their evaluators and the new process. Researchers share findings from the following research questions about Chicago's new teacher evaluation system: (1) To what extent do systematic differences between evaluators affect teacher ratings?; (2) What are teacher perceptions of their evaluators?; and (3) What are teacher perceptions of the new observation process? In addition to these questions, this paper describes how the district utilized findings to guide changes to the implementation of Chicago's new evaluation system. To understand systematic differences in evaluators and their effect on teacher observation ratings, researchers focus on three aspects of evaluator behavior: (1) evaluator severity; (2) evaluator internal consistency; and (3) reliability using a generalizability theory approach. Data for this study includes CPS personnel and administrative data from the 2012-13 school year, observation ratings from external observers, and survey data from the 2012-13 and 2013-14 school year. Findings from the end of the first year of REACH implementation include: (1) In general, administrators were able to apply the rating rubric in ways that were consistent with predictions given component difficulty. About 3 percent of administrators were identified as muted and about 5 percent of administrators were erratic; (2) There is considerable variation in evaluator severity, but few are so extreme as to have a very substantial effect on teacher ratings. About 10 percent of the sample of evaluators is extremely severe or lenient. In general, most administrators and external observers were rating within 0.5 points of the average; and (3) In the first year of REACH implementation, teachers were generally positive about the accuracy of the ratings they received from evaluators and the observation process. Tables and figures are appended.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research; Tests/Questionnaires
Education Level: Secondary Education; Elementary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Illinois