Neuro-imaging studies of intelligence implicate the importance of a parietal-frontal network. One unresolved issue is whether this network underlies a general factor of intelligence ("g") or other specific cognitive factors. A second unresolved issue is whether males and females use different parts of this network. Here we obtained intelligence factors (general, speed of reasoning, spatial, memory, and numerical) from a large set of tests completed by 6929 young adults, 40 of whom (21 males, 19 females) also completed DTI and fMRI during a working memory "n"-back task. Within brain areas activated during this task, correlations were computed between percent activation and scores on the intelligence factors. The main findings were: (1) individual differences in activation during the "n"-back task were correlated to the general intelligence factor ("g"), as well as to distilled estimates (removing "g") of speed of reasoning, numerical ability, and spatial ability, but not to memory, (2) the correlations were mainly bilateral for females and unilateral for males, and (3) differences in the integrity of the axonal connections were also related to the functional findings showing that integrity of interhemispheric connections was positively correlated to some intelligence factors in females but negatively correlated in males. This study illustrates the potential for identifying aspects of the neural basis of intelligence using a combination of structural and functional imaging. (Contains 3 figures and 5 tables.)