This paper reflects on the methodology used in international comparative education surveys by conducting a systematic review of the European Survey on Language Competences (ESLC). The ESLC was administered from February to March 2011, with final results released in June 2012. The survey tested approximately 55,000 students across 14 European countries with the goals of (1) providing comparative data on foreign language competence and (2) informing policy on language learning and teaching. The paper argues that there is a gap between the purpose of international comparative education surveys such as the ESLC and the methodology used, and thus also the ability of the data to feed into and inform policy. It is suggested that further methodological advances and improvements are needed if surveys are to produce data that give a clearer and more in-depth understanding of what actually goes on in classrooms and positively impact on learning and teaching.