Background/Context: Although there has been progress in closing the test score gaps among student groups over past decades, that progress has stalled. Many researchers have speculated why the test score gaps closed between the early 1970s and the early 1990s, but only a few have been able to empirically study how changes in school factors and social background characteristics relate to that convergence. The main reason for this is the lack of data for multiple student cohorts--information necessary for the examination of such relationships. Purpose/Objective/Research Question/Focus of Study: We analyzed nationally representative data from 1972, 1982, 1992, and 2004, examining the mathematics achievement of four high school senior cohorts, and several school and family background characteristics. We examined how changes in these measures (in terms of means and coefficients) relate to the black-white and Latino-white test score gaps and to changes in school minority composition. Population/Participants/Subjects: For our analysis, we used the following nationally representative data sets, which are part of the Longitudinal Studies (LS) program at the National Center for Education Statistics (NCES): National Longitudinal Study of the High School Class of 1972 (NLS-72); High School and Beyond senior cohort of 1982 (HSB-82); National Education Longitudinal Study senior cohort of 1992 (NELS-92); and Educational Longitudinal Study senior cohort of 2004 (ELS-04). Research Design: Conducting secondary data analyses of these nationally representative data, we estimated a series of regressions for each senior cohort, entering the race dummy variables to estimate the unadjusted predicted mathematics test score difference between black and white students and between Latino and white students. Next, we estimated a series of multilevel regressions for each cohort to analyze how trends in school and social background measures are related to trends in the black-white and Latino-white mathematics test score gaps. Finally, we used the pooled coefficients in the decomposition of the difference between the predicted means of white and black test scores. Findings/Results: Our estimates revealed that between 1972 and 2004, increases in school segregation corresponded to significant increases in the black-white and Latino-white test score gaps, outweighing the positive changes in family background measures for these minority groups. Conclusions/Recommendations: Understanding how our society can address these countervailing forces--the improving socioeconomic conditions for black and Latino families on the one hand, and the increasing racial isolation of these students in schools on the other--necessitates innovative ideas and experimentation.