Mathematics education in India is offered in one of the 22 officially recognized state languages or in English even though there are at least 270 languages with more than 10,000 speakers each. Caste, a deep-rooted structure that stratifies Indian society, is integrally linked to shaping state languages. There is minimal research from India that looks at language and mathematics education and practically none that factors in caste. Focusing on Tamil Nadu, a state with a history of anti-caste movement on the one hand and pure Tamil movement (a movement that sought to create a Tamil language with no words from other languages) on the other, this conceptual paper seeks to explore this dimension. More specifically, by using caste as an analytical framework, and by drawing on examples from the mathematics textbooks published by the Tamil Nadu State Board of Education and the experience of a few teachers and learners, the paper seeks to make a theoretical argument that the use of pure Tamil in mathematics textbooks has negative implication for socio-culturally and economically marginalized students who are solely dependent on textbooks as the only source for learning mathematics. There is a strong need for carrying out empirical work that would highlight the nuances and complexities involved in realizing 'mother tongue' education in mathematics, particularly for those who belong to marginalized caste-class backgrounds, and we hope that such work would emerge in the future.