In this article I advocate for a new discussion in the field of international large-scale assessments; one that calls for a reexamination of international large-scale assessments (ILSAs) and their use. Expanding on the high-quality work in this special issue I focus on three inherent limitations to international large-scale assessments noted by authors in this special issue. The first limitation has to do with the quality of background questionnaires; the second, has to do with measuring trend over time; and the third emphasizes the thorny issue of making causal claims with cross-sectional ILSA data. Following this discussion, and drawing on Scriven's "higher level" Meta-evaluation Checklist, I outline how national participation in a meta-evaluation of ILSAs may assist study administrators and ILSA users in creating data that better fits the needs of a complex and diverse set of participants.