One method to investigate classroom quality is for a person to observe what is happening in the classroom. However, this method raises practical and technical concerns such as how many observations to collect, when to collect these observations and who should collect these observations. The purpose of this study is to provide empirical evidence to address these concerns using a particular middle school mathematics classroom observation tool. Data included multiple ratings of mathematics classrooms from 69 middle school teachers within 11 districts that were analyzed using generalizability theory. Findings suggest that raters trained to use this particular measure required 3 observations to consistently capture habitual classroom environments. Implications for investigating classroom quality using this and other classroom observation tools should be guided by decisions about the specific purpose of the observation tool, as well as budget and practical considerations.