Teaching mathematics has long required the use of technology due many powerful affordances. More recently, education technology has been developed to support personalized learning through the use of adaptive learning systems. Through the use of educational technology in online learning, there is great potential for improving students' mathematics achievement. In this article, we report the results of an evaluation study, where 11 online mathematics educational technology products were distributed to close to 200,000 K-12 students and their teachers in the state of Utah to supplement classroom instruction. While only ten percent of students used the products at the recommended level over the course of the 2014-15 school year, there were six products where an educationally meaningful impact on mathematics achievement was found. While teachers responded positively, a third of teachers reported lack of access to technology as a barrier. We are already seeing improved usage during the second year of the project due to modifications to the expectations for schools based on what was learned from the first year of implementation.