Nations, states, and districts must choose among an array of different approaches to measuring school effectiveness in implementing their accountability policies, and the choice can be difficult because different approaches yield different results. This study compares two approaches to computing school effectiveness: a "beating the odds" type approach and a "value-added" approach. We analyze the approaches using both administrative data and simulated data and reveal the reasons why they produce different results. We find that differences are driven by a combination of factors related to modeling decisions as well as bias stemming from nonrandom assignment. Generally, we find that the value-added method provides a more defensible measure of school effectiveness based purely on test scores, but we note advantages and disadvantages of both approaches. This study highlights the consequences of several of the many choices facing policymakers in choosing a methodology for measuring school effectiveness.