Scientific models and modeling play an important role in science, and students' understanding of scientific models is essential for their understanding of scientific concepts. The measurement instrument of "Students' Understanding of Models in Science" (SUMS), developed by Treagust, Chittleborough & Mamiala ("International Journal of Science Education," 24(4):357-368, 2002), has commonly been used to measure SUMS. SUMS was developed using the Classical Test Theory (CTT). Considering the limitations of CTT, in this study we applied a Rasch model to validate SUMS further. SUMS was given to 629 students in 18 classes of grades 9 and 10 from six high schools in China. The results present both additional evidence for the validity and reliability of SUMS and specific aspects for further improvement. This approach of validation of a published instrument by Rasch measurement can be applied to other measurement instruments developed using CTT.