Literacy and numeracy are important skills that are typically learned during childhood, a time that coincides with considerable shifts in large-scale brain organization. However, most studies emphasize focal brain contributions to literacy and numeracy development by employing case-control designs and voxel-by-voxel statistical comparisons. This approach has been valuable, but may underestimate the contribution of overall brain network organization. The current study includes children (N = 133 children; 86 male; mean age = 9.42, SD = 1.715; age range = 5.92-13.75y) with a broad range of abilities, and uses whole-brain structural connectomics based on diffusion-weighted MRI data. The results indicate that academic attainment is associated with differences in structural brain organization, something not seen when focusing on the integrity of specific regions. Furthermore, simulated disruption of highly-connected brain regions known as hubs suggests that the role of these regions for maintaining the architecture of the network may be more important than specific aspects of processing. Our findings indicate that distributed brain systems contribute to the etiology of difficulties with academic learning, which cannot be captured using a more traditional voxel-wise statistical approach.