This article examines the complex task of designing a large-scale assessment to align with a reform-oriented view of mathematics teaching and learning. Specifically, we address this issue through the case of the province of Ontario that has a reform-oriented curriculum and support for its implementation. The province also has a mandate to assess the mathematical understanding of students through a large-scale assessment. This article uses our evaluation of the Grade 9 assessment of mathematics to discuss the challenges in developing an assessment instrument to survey students' mathematical understanding within the context of intended reform curriculum and pedagogy. We particularly examine the coherence of the assessment with reform-oriented curriculum and consider this coherence in light of the pervasive influence of externally-mandated assessment on both assessment and instructional practice [Barnes, M., Clarke, D., & Stephens, M. (2000). Assessment: The engine of systemic curricular reform? "Journal of Curriculum Studies," 32(5), 623-650].