This study examined the differences between children with a low socioeconomic status [socially disadvantaged children (SDC)] and children without this disadvantage (non-SDC) on physical fitness and academic performance. In addition, this study determined the association between physical fitness and academic performance, and investigated the possible moderator effect of SDC. Data on 544 children were collected and analysed (130 SDC, 414 non-SDC, mean age = 8.0 ± 0.7). Physical fitness was measured with tests for cardiovascular and muscular fitness. Academic performance was evaluated using scores on mathematics, spelling and reading. SDC did not differ on physical fitness, compared with non-SDC, but scored significantly lower on academic performance. In the total group, multilevel analysis showed positive associations between cardiovascular fitness and mathematics (ß = 0.23), and between cardiovascular fitness and spelling (ß = 0.16), but not with reading. No associations were found between muscular fitness and academic performance. A significant interaction effect between SDC and cardiovascular fitness was found for spelling. To conclude, results showed a specific link between cardiovascular fitness and mathematics, regardless of socioeconomic status. SDC did moderate the relationship between cardiovascular fitness and spelling.