The purpose of this paper is to assess the validity of multiple-choice questions in measuring fourth grade students' ability to interpret graphs related to physical science topics such as motion and temperature. We administered a test including 6 multiple-choice questions to 28 fourth grade students. Students were asked to explain their thinking in writing for each question. In addition, we interviewed all 28 students and asked them to justify their answer for each question by thinking out loud. We found that a significant number of students were not able to provide appropriate explanations for their correct answers. Interestingly, however, a significant number of students were able to provide appropriate explanations even though they initially selected an incorrect response. As a result of this study, we suggest caution in using multiple-choice questions as a single data source to assign grades or to make other important decisions about student achievement.