The usability of classroom interventions plays a significant role in the likelihood that they will reach wide scale use. The current report describes the usability testing of a software application designed for ClassWide Peer Tutoring (CWPT), a research-based classroom intervention. Initially, CWPT teachers rated the usability of the software and indicated how they used it in their classrooms. Then, teachers were observed completing seven common activities with the software. The following usability data were collected during the activities: how many teachers requested help and reported problems, what part of the software caused problems and prompted requests for help, and the time to complete activities. A part of the software was considered "difficult to use" if 40% or more teachers requested help with it or reported problems. After identifying the difficult-to-use parts of the software, the nature of the problem and how to correct it was assessed by reviewing teachers' specific questions and problems. Findings suggest that the software's usability ratings were much higher than its observed usability. The implications of usability evaluations beyond subjective ratings are discussed in terms of how to improve the scalability of educational interventions and technology. (Contains 1 table and 7 figures.)