This study challenges prior evidence on the impact of Korea's high-stakes school accountability policy as a tool to improve overall achievement and close achievement gaps. Applying difference-in-differences method, the study compares Korean high school students' reading and math test score trends before and after the policy based on the National Assessment of Educational Achievement (NAEA) vs the Programme for International Student Assessment (PISA). Under the "No Student Below Basic" policy, achievement gains were only evident on NAEA as high-stakes test but not on PISA as low-stake test that emphasizes real-world applications. The achievement gaps between high-performing and low-performing schools narrowed on NAEA but widened on PISA. Through cross-national comparison with the US "No Child Left Behind" policy features and outcomes, the study offers critical insights on the risks of test score inflations and educational limitations under high-stakes school accountability policy by which test results may become neither transferrable nor sustainable.