During the development of a test of multiple-choice items, cognitive interviews were conducted with students from lower secondary school. The purpose of these interviews was to confirm that the prospective respondents' interpretation of mathematics test items was consistent with the interpretation intended by the item writer. The conversations with the students provided unexpected and intriguing insights into the way students think and speak about mathematical concepts: ideas worthy of comment and further research. The benefits and limitations of using cognitive interviews to improve assessment items, and the recommended procedures for conducting such interviews, had been identified from earlier research studies. With these findings, a framework was developed for designing and implementing a series of interviews with ten Year 9 students in Western Australia. The items reviewed by these students were intended for an online test of the skills and understandings essential for the development of sound proportional reasoning for Year 8 students. The responses to the interview questions not only provided useful suggestions on how to improve the test items and on the suitability of the test instrument, but they also provided valuable information about the students' understanding and communication of what they had learned.