How can activities in which collaborative skills of an individual are measured be standardized? In order to understand how students perform on collaborative problem solving (CPS) computer-based assessment, it is necessary to examine empirically the multi-faceted performance that may be distributed across collaboration methods. The aim of this study was to explore possible differences in student performance in human-to-agent (H-A), compared to human-to-human (H-H) CPS assessment tasks. One hundred seventy nine 14 years-old students from the United States, Singapore and Israel participated in the study. Students in both H-H and H-A modes were able to collaborate and communicate by using identical methods and resources. However, while in the H-A mode, students collaborated with a simulated computer-driven partner, and in the H-H mode students collaborated with another student to solve a problem. Overall, the findings showed that CPS with a computer agent involved significantly higher levels of shared understanding, progress monitoring, and feedback. However, no significant difference was found in a student's ability to solve the problem or in student motivation with a computer agent or a human partner. One major implication of CPS score difference in collaboration measures between the two modes is that in H-A mode one can program a wider range of interaction possibilities than would be available with a human partner. Thus, H-A approach offers more opportunities for students to demonstrate their CPS skills. This study is among the first of its kind to investigate systematically the effect of collaborative problem solving in standardized assessment settings.