Research Findings: The Early Childhood Environment Rating Scale-Revised (ECERS-R) is widely used, often to evaluate whether preschool programs are of sufficient quality to improve children's school readiness. We examined the validity of the measure for this purpose. Item response theory (IRT) analyses revealed that many items did not fit together to measure single dimensions, particularly when rated by consultants as indicating aspects of quality relevant for multiple domains of child development. IRT results also conflicted with the scale developers' expectations in terms of whether markers that they attached to higher response categories represented higher quality empirically. When reanalyzed based on experts' ratings, IRT results also showed that relatively few indicators captured the moderate to high range of quality. Practice or Policy: Our results suggest that policymakers should carefully consider whether measures designed for specific purposes are appropriate for other high-stakes uses. We encourage continued refinement of existing quality measures, development of new measures, and the accumulation of evidence for their various uses.