Many universities are currently using Course Management Systems (CMSes) to conduct online learning, for example, by distributing course materials or submitting homework assignments. However, most CMSes do not include comprehensive activity tracking and analysis capabilities. This paper describes a method to track students' online learning activity, and provide this information to instructors to allow them to assess students' learning behavior and progress. Further, the information can be used by educational researchers to evaluate the contribution of CMSes to learning. A prototype of the proposed system, called the Moodle Watchdog, or "Moodog", has been implemented on top of the Moodle CMS. The Moodog system provides accurate tracking information with an easy-to-use interface, and offers a wide range of activity information to instructors and educational researchers. As a basis for evaluating Moodog, the system has been deployed in several courses taught at the University of California in Santa Barbara. Using the results and experiences obtained from this evaluation, Moodog has been shown to collect useful activity information for both educators and educational researchers. This information suggests different groups of students tend to have different behavior in some online learning activities, such as forum access, login sessions and course material access, but do not seem to have different behavior on lecture notes access. The information produced by Moodog and the analysis performed in this paper suggest that a more detailed analysis of the data by educational researchers would yield interesting results about the use of course management systems. (Contains 13 figures and 1 footnote.)