Aging-related declines occur in many different domains of cognitive function during middle and late adulthood. However, whether a global dimension underlies individual differences in changes in different domains of cognition and whether global genetic influences on cognitive changes exist is less clear. We addressed these issues by applying multivariate growth curve models to longitudinal data from 857 individuals from the Swedish Adoption/Twin Study of Aging, who had been measured on 11 cognitive variables representative of verbal, spatial, memory, and processing speed abilities up to 5 times over up to 16 years between ages 50 and 96 years. Between ages 50 and 65 years scores on different tests changed relatively independently of one another, and there was little evidence for strong underlying dimensions of change. In contrast, over the period between 65 and 96 years of age, there were strong interrelations among rates of change both within and across domains. During this age period, variability in rates of change were, on average, 52% domain-general, 8% domain-specific, and 39% test-specific. Quantitative genetic decomposition indicated that 29% of individual differences in a global domain-general dimension of cognitive changes during this age period were attributable to genetic influences, but some domain-specific genetic influences were also evident, even after accounting for domain-general contributions. These findings are consistent with a balanced global and domain-specific account of the genetics of cognitive aging.