A repeated crossover experiment comparing learning among students handing in pen-and-paper homework (PPH) with students handing in web-based homework (WBH) has been conducted. The system used in the experiments, the tutor-web, has been used to deliver homework problems to thousands of students in mathematics and statistics over several years. Since 2011, experimental changes have been made regarding how the system allocates items to students, how grading is done, and the type of feedback provided. The experiment described here was conducted annually from 2011 to 2014. Approximately, 100 students in an introductory statistics course participated each year. The main goals were to determine whether the above-mentioned changes had an impact on learning as measured by test scores in addition to comparing learning among students doing PPH with students handing in WBH.