Multiplicative reasoning is a key concept in elementary school mathematics. Item statistics reported by the National Assessment of Educational Progress (NAEP) assessment provide the best current indicator for how well elementary students across the U.S. understand this, and other concepts. However, beyond expert reviews and statistical analysis, there is relatively little evidence on the validity of NAEP items for assessing mathematical reasoning. The present study examined the convergent validity of several public-release fourth grade NAEP multiplication items. Findings indicate that most items assessed lack sufficient validity for examining children's underlying conceptual knowledge. Rather, NAEP items may be less appropriate for assessing conceptual understanding and more appropriate for assessing procedural recall. Recommendations and implications are discussed.