International large-scale assessments such as the Programme for International Student Assessment (PISA) have emerged as important tools for understanding educational systems within Asia and around the world. Although the achievement scales tend to capture the majority of the world's attention the background scales have become important tools in their own right. To that end, testing organizations such as the Paris-based Organization for Economic Cooperation and Development (OECD) are developing background scales and providing them in their datasets. However, unlike the scales created to estimate achievement, considerably less work has been done to examine the quality and comparability of these background measures. In this paper, we purposely examine four culturally similar Chinese speaking economies and test the quality of OECD reported scales. Specifically, using both factor analysis and item response theory we evaluate the level of measurement invariance of science-related disposition scales in PISA 2015. Our analysis shows that model fit of the science self-efficacy scale is acceptable, while the scales of epistemological beliefs and science activities are not statistically valid and should not be used to compare within or between economies. We conclude by discussing the need for more involvement by non-OECD members in PISA questionnaire and scale development.