NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED562857
Record Type: Non-Journal
Publication Date: 2014
Pages: 6
Abstractor: ERIC
Reference Count: 10
Using Anchoring Vignettes to Calibrate Teachers' Self-Assessment of Teaching
Yuan, Kun; Engberg, John; Kaufman, Julia; Hamilton, Laura; Hill, Heather; Umland, Kristin; McCaffrey, Daniel
Society for Research on Educational Effectiveness
High-quality measures of instructional practice are essential for research and evaluation of innovative instructional policies and programs, as well as for providing feedback to teachers and administrators. In this study, the authors examined whether using anchoring vignettes in web-based surveys improved the validity of teachers' self-assessments of their mathematics instruction. To investigate validity, they compared correlations between teachers' self-ratings and other measures of teaching including teachers' value-added scores, student surveys, and observation ratings of instruction before and after calibration to examine whether calibration improves the correlation between teachers' self-ratings and other teaching measures. Data came from 61 mathematics teachers in grades 4-9 participating in the Bill & Melinda Gates Foundation's Measures of Effective Teaching Extension project. The sample was roughly evenly distributed between elementary (grades 4-5) and middle school (grades 6-8) teachers, with only three percent teaching 9th grade. The authors used both non-parametric and parametric methods to calibrate teachers' self-ratings. Preliminary findings suggest that anchoring vignettes represent a promising innovation for measuring teachers' instruction through survey self-reports. Specifically, the authors found: (1) Teachers' survey responses that are calibrated through the use of anchoring vignettes have increased variation compared to teachers' raw survey responses, particularly for the cognitive challenge dimension; (2) Teachers' calibrated survey responses regarding mathematical vocabulary and cognitively challenging tasks are more strongly correlated with the composite measure of teacher performance compared to raw survey responses; and (3) If teachers gave their self-rating after rating the vignettes, rather than before, the entire collection of calibrated self-ratings are significantly correlated with the composite performance measure (p<0.05).
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: Elementary Education; Middle Schools; Secondary Education; Junior High Schools; Grade 4; Intermediate Grades; Grade 5; Grade 6; Grade 7; Grade 8; Grade 9; High Schools
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)