Major educational initiatives in the world place great emphasis on fostering rich computer-based environments of assessment that make student thinking and reasoning visible. Using thinking tools engages students in a variety of critical and complex thinking, such as evaluating, analyzing, and decision making. The aim of this study was to explore patterns in student critical thinking performance and motivation in Evidence-Centered Concept Map (ECCM) mode, compared to basic notepad mode. One hundred ninety 14-year-old students from the United States, United Kingdom, Singapore, and South Africa participated in the study. Students in both modes were able to analyze a multifaceted dilemma by using similar information resources. In the ECCM mode, students used ECCM to organize their thinking; in another mode, students were provided with a basic online notepad to make records as needed. Overall, the findings showed that students assessed in ECCM mode outperformed their peers in notepad mode in critical thinking skills. Students who worked with ECCM provided more informed recommendations by using supporting evidence from the available resources and discussing alternative points of view on the topic. The findings showed a significantly positive relationship between student recommendation score and the ability to develop ECCM. In addition, the results demonstrated that it did not matter for students' motivation whether they analyzed the dilemma with or without ECCM. Directions for future research are discussed in terms of their implications for large-scale assessment programs, teaching, and learning.