The aim of this study was to investigate the interplay between test anxiety and working memory (WM) on mathematics performance in younger children. A sample of 624 grade 3 students completed a test battery consisting of a test anxiety scale, WM tasks and the Swedish national examination in mathematics for grade 3. The main effects of test anxiety and WM, and the two-way interaction between test anxiety and WM on mathematics performance, were modelled with structural equation modelling techniques. Additionally, the effects were also tested separately on tasks with high WM demands (mathematical problem-solving) versus low WM demands (basic arithmetic). As expected, WM positively predicted mathematics performance in all three models (overall mathematics performance, problem-solving tasks, and basic arithmetic). Test anxiety had a negative effect on problem-solving on the whole sample level but concerning basic arithmetic only students with lower WM were affected by the negative effects of test anxiety on performance. Thus, students with low WM are more vulnerable to the negative effects of test anxiety in low WM tasks like basic arithmetic. The results are discussed in relation to the early identification of test anxiety.