The low number of baccalaureates in science, technology, engineering, and mathematics (STEM) is often viewed as problematic for the US's economic competitiveness, leading scholars to search for explanations for STEM retention. Our analyses of the Beginning Postsecondary Students Longitudinal Study indicate that the notion of a so-called "leaky STEM pipeline" out of STEM majors overstates the problem because it neglects the substantial influx into STEM from other majors throughout college. Researchers concerned with STEM retention should focus on a broader defined group of "STEM-actives": A combination of freshman students who declared a STEM major or who take a considerable number of STEM credits. Among these students (N = 3,020) we examine the variation in the relatively lower grades that many individuals earn in STEM courses compared to their non-STEM courses. The size of an undergraduate's "STEM-grading penalty"--an individual grading disparity--in the first couple of college semesters is significantly associated with the probability of leaving STEM. The influence of this STEM-penalty on STEM graduation chances is robust to college students' variation in both general academic achievement and STEM-specific preparation, thereby eliminating a large portion of the effect due to skills, performance, and selection. Our analyses expands on previous research regarding relative grading conducted "within" STEM-fields.