Many scholars agree on the general theoretical structure of metacognition, which is what informed the development of the Metacognitive Awareness Inventory (MAI). Although self-report instruments such as the MAI suffer many threats to validity, they continue to be used in research and practice because of their convenience. With the MAI, studies have varied in the way they calculate scores and in their adherence to the intended theory. In this study, we address these shortcomings and propose modifications in calculating MAI scores. Using confirmatory factor analysis (CFA) and multidimensional random coefficients multinomial logit (MRCML) item-response modeling, we examined how well the intended functioning of the MAI matched the data from 622 undergraduate students. The results support scoring the MAI as two dimensions, knowledge and regulation of cognition, but indicate that the 52-item instrument has poor fit. Using iterative CFA and MRCML models, we tested subsets of items that represent the theory and had good fit. We followed up with tests of between-group and time invariance. The results support the use of a 19-item subset for between-group comparisons, with provisional evidence for its use in longitudinal studies.