We examined the performance of English language learners (ELLs) and non-ELLs on Grade 4 and Grade 5 mathematics content knowledge (CK) and academic language (AL) tests. CK and AL items had different semiotic loads (numbers of different types of semiotic features) and different semiotic structures (relative frequencies of different semiotic modalities). For both linguistic groups, the percentage of items correct was higher for the AL than the CK tests. However, the score gains attributable to instruction were smaller for the AL than the CK tests. CK and AL test scores correlated more highly for non-ELLs than ELLs before instruction. This suggests that, before instruction, the meaning-making system of ELLs was less consolidated than that of non-ELLs, who dealt with the interpretive demands of CK and AL items with similar effectiveness. We discuss the importance of using a semiotic perspective in test design and in interpreting ELL student performance.