Through the examination of a collection of macro factors and explanations for racial disparities in mathematics assessment found in the literature, this article takes up these accounts and problematizes the factors by unpacking the assumptions and exposing complexities. We do this using Critical Race Theory (CRT) to reinterpret and call out important blind spots. Essential questions that guided our analysis included: what macro factors has the field identified as influencing or explaining racial disparities in mathematics assessments? What assumptions undergird the field's conversations about racial disparities in mathematics assessments? In what ways can those assumptions be challenged through CRT to highlight the story of race in the US? Our analysis reveals that the factors fall victim to a meritocratic premise that assumes all students are exposed to equivalent forms and amounts of mathematical knowledge. This assumption effectively locates the problem of assessment within students and not the institutions that are entrusted with providing access to mathematics curricular knowledge that is requisite for success. We conclude by arguing for a call to social action and continued policy measures with a renewed focus on equitable curricular access as an important method to improve the assessment outcomes for racial minority students.