Purpose: The present study examined the relationship between mathematics and language to better understand the nature of the deficit and the academic implications associated with specific language impairment (SLI) and academic implications for English language learners (ELLs). Method: School-age children (N = 61; 20 SLI, 20 ELL, 21 native monolingual English [NE]) were assessed using a norm-referenced mathematics instrument and 3 experimental computer-based mathematics games that varied in language demands. Group means were compared with analyses of variance. Results: The ELL group was less accurate than the NE group only when tasks were language heavy. In contrast, the group with SLI was less accurate than the groups with NE and ELLs on language-heavy tasks and some language-light tasks. Specifically, the group with SLI was less accurate on tasks that involved comparing numerical symbols and using visual working memory for patterns. However, there were no group differences between children with SLI and peers without SLI on language-light mathematics tasks that involved visual working memory for numerical symbols. Conclusion: Mathematical difficulties of children who are ELLs appear to be related to the language demands of mathematics tasks. In contrast, children with SLI appear to have difficulty with mathematics tasks because of linguistic as well as nonlinguistic processing constraints.