To remove cultural bias is critical for the legitimacy of Programme for International Student Assessment (PISA) as an internationally reliable academic assessment. Since its inception, PISA has made extensive effort to address this issue by putting in place a range of methodological and procedural measures to ensure its test fairness. This study attempts to disrupt the clean and reassuring accounts of PISA's technical solutions to cultural bias and exposes the irrationalities, tensions and messes that get muted in the rational and scientific discourse of test fairness. To this end, this article turns to Japanese "kokugo" (national language) curriculum experts as a source of critical insights. In particular, it looks at the way they responded to PISA's notion of reading literacy and then how those who were tasked to develop test items for reading literacy in PISA 2009 made sense of their item development experience. Their experience highlights the inherent cultural bias in the PISA's framework for reading literacy itself as well as the highly messy and coercive processes of PISA item development. In conclusion, I call for broadening the sources of critical insights beyond Anglo-American and select European countries to denaturalize the underpinning premises of PISA that remain unproblematized by Anglo-European critical education policy scholars.