The purpose of this study is to investigate to what extent the use of L2 in math tests influences bilingual education learners' process of word problem solving in a mandatory secondary education school with Content and Language Integrated Learning (CLIL). The reading comprehension level of the students was analysed using a standards-based assessment and the questions used in Programme for International Student Assessment (PISA) tests. The word problems were selected according to the students' level of reading-comprehension and mathematical competence. Leaners also had to answer a questionnaire, which was used to analyse if contextual factors were affecting mathematical performance in L2. To this end, the questionnaire included some questions related to the bilingual history of the students and their perception about solving word problems in English. Data were analysed through one-way or two-way ANOVA tests to find out which factors were relevant. Results show that solving word problems is not only affected by the use of L2, but that it also depends on the mathematical difficulty, irrespective of the students' level of language proficiency. The findings, hence, imply that interaction between linguistic difficulty and mathematical complexity is at the centre of the issues affecting word problem solving.