Purpose: The article reports on a study of school leader decision making that examined variation in the validity of teacher evaluation ratings in a school district that has implemented a standards-based teacher evaluation system. Research Methods: Applying mixed methods, the study used teacher evaluation ratings and value-added student achievement data to identify 23 school leaders with "more" and "less" valid results. These leaders were interviewed to learn about their attitudes on teacher evaluation, their decision-making strategies, and school contexts. Results from interviews with a subset of eight school leaders with 2 years of consistent validity scores (n = 4 more valid and n = 4 less valid) were analyzed. Findings: Substantial variation was found in the relationship of evaluators' ratings of teachers and value-added measures of the average achievement of the teachers' students. The results did not yield a simple explanation for the differences in validity of evaluators' ratings. Instead, evaluators' decisions were found to be a complex and idiosyncratic function of motivation, skill, and context. Conclusion: The results suggested why overall validity of ratings was lower than expected and highlighted challenges in research on school leader decision making and cautions for using such decisions for high stakes purposes. Recommendations are provided for improving evaluation accuracy and validity, with considerations for performance evaluation policy and future research on school leader decision making and teacher evaluation. (Contains 4 tables and 1 note.)