This study examines data from a field experiment investigating the effects of a personalized recommendation algorithm that proposes to students which videos to watch next, after they complete mini-assessments for algebra that available on the Math Nation intelligent virtual learning environment (IVLE). The end users of Math Nation are students enrolled in an Algebra 1 course in middle and high schools of the state of Florida, and the IVLE is used both during and out of school time. The objective of the developed recommendation algorithm is to increase student preparation to take the state-mandated End-of-Course (EoC) Algebra 1 assessment at the end of the school year. The algorithm is based on a Markov Decision Process framework that uses as input the students' responses to a series of mini-assessment tests. The current study randomly assigned 16,406 students to either treatment or control conditions, which were blind to both students and teachers. The results indicate that the effects of the recommendation algorithm depend on the level of usage of students, showing significant improvements on EoC test scores of students who have a moderate level of usage. However, there was no effect for low usage students. The study also shows that students practicing with the mini-assessments available on Math Nation, helps them improve by a small margin their performance on the End-of-Course test, irrespective of the usage level. Finally, the study provides insights on challenges posed for implementing personalized recommendation algorithms at a large scale, related both to student self-regulation and teacher orchestration of technology use in the classroom. [For the full proceedings, see ED615472.]