The current study aimed to measure preservice teachers' skills in implementing pedagogical model of collaborative problem solving (CPS) using a more advanced and novel human-to-agent computerized assessment instrument. By doing so, a framework with three major skills in the implementation of CPS and four major individual problem-solving processes was identified. Then, an assessment with 18 items was constructed under the framework. Additionally, the performance of preservice teachers in implementing CPS was further explored and discussed with the multidimensional random coefficients multinomial logit model. To determine the reliability and validity of the assessment, the expected a posteriori/plausible values (EAP/PV) reliability coefficient, the model data fit, and the differential item functioning (DIF) were explored. The findings demonstrated that the hypothesized structure with three dimensions was validated, and no DIF across gender groups was identified. The results showed that this assessment was feasible, and its reliability and validity were satisfied.