Online software systems are extensively used to give students practice on course content, especially in mathematics and physics courses. They offer instant feedback, and several of these systems are open source or very economical compared with hiring graders for traditional paper-and-pencil-based homework (PPH). In this article, the authors evaluate WeBWorK (WW), an online software tool, in an introductory course on probability over two semesters. WW is compared with PPH by measuring student perception, average time spent on a problem, collaborative work outside of classroom, resilience, self-efficacy, and exam performance. The authors find that except for working in groups on homework, students perform similarly on all the aforementioned aspects in both PPH and WW. The authors also suggest potential strategies to improve student understanding and learning while using WW and recommend the use of WW in mathematics-oriented courses.