Classroom observation is a common approach to teacher evaluation. Yet, concerns about differences in rater judgment are widespread. Despite this concern, few researchers have examined the practical impact of such differences in rater judgments on teachers' judged effectiveness. This study fills that gap. Using data from a large-scale teacher evaluation system, we found substantial differences in principal severity that affected teachers' classification within performance categories. We then demonstrate a technique that researchers and practitioners can use to control for differences in rater severity -- thus limiting the degree to which rater severity differences can threaten the fairness of classroom observations. We discuss implications for research and practice.