In Tanzania, two types of primary schools exist, based on the language of instruction used. These are English medium schools and Swahili medium schools. Students who complete their studies from both types of schools join secondary schools where the language of instruction is solely English. This study investigated the influence of students' backgrounds in the language of instruction on secondary school academic performance. The specific objectives of the study were two: First, to compare form one annual examinations results for students who had used English and those who had used Swahili as the medium of instruction at primary school level and, secondly, to compare the form two national examinations results for students who had used English and those who had used Swahili as the medium of instruction at primary school level. The study involved 524 students from eight secondary schools in Mbeya region. The data were collected through review of documents containing students' admission lists and examinations results. The analysis of the collected data was done using computer software, Statistical Package for Social Sciences, version 18, in which an independent samples t-test was conducted to test the hypotheses advanced for this study. The findings were as follows: First, students whose medium of instruction at primary school level was English, performed better in form one annual examinations than their counterparts who had used Swahili and the difference was statistically significant at p<0.05, in a two-tailed test. Secondly, students who had used English as the medium of instruction at primary school level, performed better in form two national examinations than those who had used Swahili and the difference was also statistically significant at p<0.05, in a two-tailed test. It was thus, concluded that the language of instruction used by students at primary school level influenced significantly students' academic performance at secondary school level.