Over the past three decades, the number of developing countries participating in International Large-Scale Assessments (ILSAs) has increased dramatically, while developing countries' usage of data from ILSAs for informing educational policy has not been fully realised. In this paper the authors argue that for ILSAs to be useful policy tools, alignment must occur between ILSAs' measurement quality mission and their capacity development mission. They review the use of ILSAs as both "whips" and "thermometers" and discuss issues in measurement related to the range of skills assessed, their capacity to measure change over time, and the inclusion of other indicators to measure home, classroom and school factors related to student achievement. They then discuss how ILSAs have built assessment capacity in developing countries and what further changes would be needed to improve their utility for policymakers.