We investigated the use of adaptive comparative judgment to evaluate the middle school student learning, engagement, and experience with the design process in an open-ended problem assigned in a technology and engineering education course. Our results indicate that the adaptive comparative judgment tool effectively facilitated the grading of the students' products and improved the judges' understanding of the processes the students used to arrive at their solutions. We found the reliability and validity of the tool to be acceptable and the outcomes of our study suggest that there is justification for the use of the adaptive comparative judgment method to assess student learning outcomes at multiple levels of Science, Technology, Engineering, and Math (STEM) education.