The passage of ESSA in 2015 created a novel mandate for States to develop school accountability systems that use alternative measures of school quality beyond test scores. This created an opportunity for States to be innovative and make new forms of data available that could potentially lead to new strategies for improving schools. This study explored early experiments in alternative accountability measures in two urban districts. Drawing on interviews, documents, and observations gained through participant research, this study highlights the variable types of data the models made available to stakeholders and how those differences suggest different theories of change for improving schools. The findings highlight how different ways of specifying school quality point to different ideas about what schools should be responsible for improving.