Mathematics has been widely recognized as being challenging for most students. In this study, an online formative peer-tutoring approach was proposed to cope with this problem, and an online learning system was developed accordingly. To evaluate the effectiveness of the proposed approach, an experiment was conducted to explore its effects on students' learning behaviors as well as their learning performance and perceptions of the mathematics course. The experimental results showed that both the formative and conventional peer-tutoring approaches significantly improved the students' learning achievement as well as reduced their cognitive load in comparison with conventional online collaborative learning approaches. In addition, the learning behavior analysis results showed that the students playing the role of tutor in the formative peer-tutoring group revealed more appropriate behaviors of assisting their tutees via discussing with them and providing hints to them as well as guiding them to solve the mathematics problems, while the tutees were more willing to ask questions than those learning with the conventional peer-tutoring approach. This implies that the proposed approach could motivate both the tutors and tutees to become active learners with good learning conceptions.