ERIC Number: EJ1089565
Record Type: Journal
Publication Date: 2016
Abstractor: As Provided
Reference Count: 64
Comparing OECD PISA Reading in English to Other Languages: Identifying Potential Sources of Non-Invariance
Asil, Mustafa; Brown, Gavin T. L.
International Journal of Testing, v16 n1 p71-93 2016
The use of the Programme for International Student Assessment (PISA) across nations, cultures, and languages has been criticized. The key criticisms point to the linguistic and cultural biases potentially underlying the design of reading comprehension tests, raising doubts about the legitimacy of comparisons across economies. Our research focused on the type and magnitude of invariance or non-invariance in the PISA Reading Comprehension test by language, culture, and economic development relative to performance of the Australian English-speaking reference group used to develop the tests. Multi-Group Confirmatory Factor Analysis based on means and covariance structure (MACS) modeling was used to establish a d[subscript MACS] effect size index for each economy for the degree of non-invariance. Only three wealthy, English-speaking countries had scalar invariance with Australia. Moderate or large effects were observed in just 31% of the comparisons. PISA index of economic, social and cultural status had a moderate inverse correlation with dMACS suggesting that socioeconomic resourcing of education played a significant role in measurement invariance, while educational practice and language factors seemed to play a further small role in non-invariance. Alternative approaches to reporting PISA results consistent with non-invariance are suggested.
Descriptors: Comparative Analysis, Reading Achievement, Achievement Tests, Secondary School Students, Correlation, Foreign Countries, Culture Fair Tests, Criticism, Cultural Differences, Cross Cultural Studies, Reading Tests, Factor Analysis, Effect Size, Socioeconomic Status, Reading Comprehension, Test Construction, Error of Measurement, Educational Resources, Role, Educational Policy, Item Response Theory, Goodness of Fit
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research
Education Level: Secondary Education
Authoring Institution: N/A
Identifiers - Location: Australia
Identifiers - Assessments and Surveys: Program for International Student Assessment