Background: The underperformance of South African learners in literacy and numeracy is a source of concern, especially when learners move from Grade 3 to Grade 4. Aim: This article reflects on the reading and comprehension challenges of English language learners (ELLs) in the Grade 4 2013 mathematics Annual National Assessments (ANAs). Setting: The study took place in two primary schools that served relatively less affluent sectors of the community in the Eastern Cape. Learners were IsiXhosa speakers learning mathematics in English. Methods: A sample of 26 out of 106 isiXhosa-speaking Grade 4 learners in the two schools participated in task-based interviews (focused on ANA questions) in which reading and linguistic mediation was provided. While the broader study (from which this article derives) revealed learners' challenges in reading, comprehension, transformation and process skills, here the focus is on findings related to reading and comprehension skills, which are foundational to accessing written assessment items. Results: Interview excerpts show the negative influence poor English reading and comprehension skills had on learner access to questions and their subsequent performance in the ANA. Conclusion: The article challenges the validity of assessing ELLs' mathematical competence in English ANAs and draws implications for strengthening ELLs' language and mathematical proficiency in the Foundation Phase