We apply a utilization-focused evaluation [UFE] approach to an evaluation of Virginia's latest comprehensive accountability policy--the Standards of Accreditation (SOA). Our study focuses on the implementation experiences of district and high school level administrators in four urban districts. This study is significant in two important ways. First it provides an example of UFE approaches to policy analysis and evaluation. The literature on UFE supports the approach's application to policy analysis, but its use in this arena has been limited. Second, our study provides important insights for evaluation stakeholders formulating and implementing complex accountability policy, particularly in early phases of implementation. Our findings explore how and why local level administrators understand each of the new standards, what factors shape those understandings, and then how the case districts and schools respond to each of the SOA. Our findings illustrate how, along one dimension, stakeholders' understandings of the standards are related to factors that influence their sense of implementation efficacy. Along a second compliance-continuous improvement dimension, administrators' leadership experience and change-oriented mindsets appeared to determine how they responded to the new policy regime. We present conclusions and recommendations for key stakeholders and other students of policy analysis and evaluation.