In this work we study how different motivations of learners relate to their engagement in a massive open online course (MOOC). The research was conducted in a cohort-based astronomy MOOC "Astronomy: Exploring Time and Space" provided through the Coursera platform. In order to discern motivations of course participants, the Science Motivation Questionnaire II was administered as a first assignment of the course. The survey data were collected from December 2016 until December 2018 and responses from 3701 learners were acquired. In order to determine the relationship of motivation to course performance, four course engagement indicators are defined: course completion, use of video lectures, participation in the discussion forums, and participation in peer-grading assignments. Self-determination and self-efficacy are identified as the main motivational factors influencing all aspects of the course engagement except for the video usage. We show that both of these decrease with age, but increase with the number of previously taken science courses. The main reasons for signing up for the course which had an effect on course engagement were the goal of getting a certificate and perceived importance of the reputation of the instructor. There was no statistically significant difference in motivations between people who dropped after the first peer graded writing assignment and those who successfully completed the course, suggesting that after the first writing assignment learners quit from other reasons than lack of motivation. Unexpectedly, learners with high career, grade, and social motivation watched statistically significantly fewer instructional videos, and social motivation did not play a role in the forum usage.