Biomedical preparatory programs (pipeline programs) have been developed at colleges and universities to better prepare youth for entering science- and health-related careers, but outcomes of such programs have seldom been rigorously evaluated. We conducted a matched cohort study to evaluate the Stanford Medical Youth Science Program's Summer Residential Program (SRP), a 25-year-old university-based biomedical pipeline program that reaches out to low-income and underrepresented ethnic minority high school students. Five annual surveys were used to assess educational outcomes and science-related experience among 96 SRP participants and a comparison group of 192 youth who applied but were not selected to participate in the SRP, using ~2:1 matching on sociodemographic and academic background to control for potential confounders. SRP participants were more likely than the comparison group to enter college (100.0 vs. 84.4%, p = 0.002), and both of these matriculation rates were more than double the statewide average (40.8%). In most areas of science-related experience, SRP participants reported significantly more experience (>twofold odds) than the comparison group at 1 year of follow-up, but these differences did not persist after 2-4 years. The comparison group reported substantially more participation in science or college preparatory programs, more academic role models, and less personal adversity than SRP participants, which likely influenced these findings toward the null hypothesis. SRP applicants, irrespective of whether selected for participation, had significantly better educational outcomes than population averages. Short-term science-related experience was better among SRP participants, although longer-term outcomes were similar, most likely due to college and science-related opportunities among the comparison group. We discuss implications for future evaluations of other biomedical pipeline programs.