This article describes the process of development of assessment instruments for a three-year longitudinal comparative study that focused on evaluating American high school students' mathematics learning from two distinct approaches to content organization: curriculum built around a sequence of three full-year courses (Algebra 1, Geometry, and Algebra 2) and a sequence of integrated mathematics courses (algebra and geometry content, together with functions, data analysis, and discrete mathematics is integrated each year). The study was conducted in six school districts in five states involving over 4,000 students from schools that were using both curricular approaches but with different groups of students. In order to develop assessment instruments that were not biased towards either of the two curriculum programs (Fair Tests), an iterative process of content analyses, identification of common topics, internal and external reviews, pilot tests, and revisions was followed, resulting in five tests that were used in the three years of the study. Results indicate that these tests have solid discrimination properties and address adequately mathematics content common to both secondary curriculum programs. The corresponding scoring rubrics are highly reliable, with interrater reliability above 94% for all tests. Mathematics education researchers involved in curriculum comparison studies need to conduct content analyses of the curriculum materials under study in order to identify salient relationships between curriculum programs and student outcomes.