In a recent article, Nadelson and Southerland (2010. Development and preliminary evaluation of the Measure of Understanding of Macroevolution: Introducing the MUM. "The Journal of Experimental Education", 78, 151-190) reported on their development of a multiple-choice concept inventory intended to assess college students' understanding of macroevolutionary concepts, the "Measure of Understanding Macroevolution" (MUM). Given that the only existing evolution inventories assess understanding of natural selection, a microevolutionary concept, a valid assessment of students' understanding of macroevolution would be a welcome and necessary addition to the field of science education. Although the conceptual framework underlying Nadelson and Southerland's test is promising, we believe the test has serious shortcomings with respect to validity evidence for the construct being tested. We argue and provide evidence that these problems are serious enough that the MUM should not be used in its current form to measure students' understanding of macroevolution. (Contains 5 notes and 1 table.)