Reflection is a critical factor in solving design problems. Using good methods to observe designers' reflection is essential to inform the design of the learning environments that support the development of design problem-solving skills. In this study, we have developed and validated a novel self-reporting questionnaire as an efficient instrument to explore reflection, called Assessing Reflective Thinking in Solving Design Problems (ARTiD). This questionnaire has been developed based on the three-dimensional model for reflective thinking: the timing, the objects and the levels of reflection. A total of 294 undergraduate and graduate students in the fields of engineering, computer science and instructional design were recruited to participate in one of four iterative, formative tests through which reliability and validity analyses were performed to revise and confirm the questionnaire. Examples of participants' reflection patterns were demonstrated to show the practical value of this questionnaire. In conclusion, the final version of ARTiD was presented as a valid instrument to explore students' reflection for research and educational practice. In addition, we found that ARTiD can be used as an effective learning tool to guide student designers' reflections toward better learning and performance.