This article uses U.S. survey data (N = 15,847) to characterize the evolution of student interest in an astronomy career in the period between middle school and the beginning of college. We find that middle school students have a relatively high interest in astronomy, which sharply declines with every phase of their education. However, many of the students who leave astronomy--particularly male students--feed heavily into other STEM disciplines. Through statistical modeling, we find that students who spend extracurricular time observing stars, tinkering with mechanical or electrical devices, or reading/watching science or science fiction are significantly more likely than students who do not engage in these activities to hold an interest in pursuing an astronomy career at the end of high school. We also find that females who observe stars during extracurricular time show a greater improvement in their odds of pursuing astronomy than males do. Furthermore, we find that these out-of-school-time activities are better predictors of astronomy interest than commonly studied academic predictors. We discuss the implications of these findings on future extracurricular programming for students.