In this study, we reanalyzed the classic Hawai'i Family Study of Cognition (HFSC) data using contemporary multilevel modeling techniques. We used the HFSC baseline data ("N" = 6,579) and reexamined the factorial structure of 16 cognitive variables using confirmatory (restricted) measurement models in an explicit sequence. These models were initially fitted using multilevel confirmatory factor analysis techniques and the invariant six-factor models with two higher order factors fit fairly well (e[subscript a < 0.08) to the total, between- and within-family data. More crucially, a model requiring metric factorial invariance proved to be a reasonable fit to the between and within matrices, and allowed the ratio of the between-family variation to total family variation (eta-squared) to be calculated separately for each common factor (?[superscript 2]: Gc = 0.27, Gf = 0.22, Gm = 0.15, Gs = 0.04, Gv = 0.30, and SP = 0.16). Higher order factors were fitted using multilevel structural equation modeling techniques and these suggested a reasonable two-factor solution with unequal family impacts. These results suggest that (a) A "G only model" does not fit the data very well, and there are many sources of individual differences in cognitive abilities; (b) the sources of the individual differences in cognition can be measured the same way between and within families; and (c) even after the unique test components are removed, "cognitive differences are larger within families than between families". We consider other general multivariate family models, and we raise questions about family influences.