The field of learning analytics is rapidly developing techniques for using data captured during online learning. In this article, we develop an additional application: the use of analytics for improving implementation fidelity in a randomized controlled efficacy trial. In an efficacy trial, the goal is to determine whether an innovation has a beneficial effect under best-case implementations. Analytics is more accurate and less expensive than traditional ways of collecting and analyzing implementation fidelity data, and may allow targeted adaptations of the innovation that improve the quality of the research. We report our experience in developing and using analytics during the course of an efficacy trial that evaluated the use of ASSISTments as an online homework tool for middle school mathematics. [This paper was published in: ICLS 2014 Proceedings, Boulder, CO, USA, June 23-27, 2014. p527-534.]