Much research has been done on the development of an intelligent tutoring system (ITS), and small empirical studies have demonstrated the effectiveness of ITS at promoting student learning. However, large-scale implementation of ITS in school settings has not been researched thoroughly. In this paper, we describe an ongoing randomized controlled trial (RCT) to evaluate the efficacy of a web-based tutoring system--the ASSISTments--as support for homework. The program is used in 46 middle schools in the state of Maine, to provide immediate feedback to students, and to provide reports to teachers to support homework review and instruction adaptation. We describe the challenges for the RCT, approaches used to understand implementation of the system, and findings on how the system is being used. [This paper was published in: Proceedings of ITS 2014, Honolulu, HI, USA, June 5-9, 2014. p561-566.]