Large-scale international achievement studies such as PISA have been widely used to study how educational inequality compares across countries. Yet the various different biases that may affect these estimates are often not considered or are poorly understood. In this paper we draw upon the total survey error framework to provide a case study of the potential biases affecting estimates of the socio-economic achievement gaps using PISA data from Germany. The results illustrate how procedural and measurement errors have a substantial impact upon estimates of socio-economic achievement gradients in Germany, including how it compares with other countries. This leads us to conclude that estimates of socio-economic achievement gaps using the PISA data for Germany do not seem to be particularly robust. More generally, we argue that better articulation and reporting of such challenges with comparing socio-economic achievement gaps using large-scale international assessment data such as PISA are needed.