Oklahoma is one of 16 states electing to use an A-F letter grade as an indicator of school quality. On the surface, letter grades are an attractive policy instrument for school improvement; they are seemingly clear, simple, and easy to interpret. Evidence, however, on the use of letter grades as an instrument to rank and improve schools is scant at best. We address the gap in the literature by using student test scores to evaluate the use of Oklahoma's A-F grades as a school quality indicator. Achievement differences between letter grades were small and in most cases not statistically significant when student and school characteristics were held constant. School grades did not reveal large achievement gaps in the highest ranked schools. Additionally, free/reduced lunch and minority students in D and F schools outperformed peers in A and B schools.