In this paper, a new approach for the formation of four-member collaborative learning groups is presented. Group formation is presented by the mathematical optimization problem. Based on the proposed approach and the variable neighbourhood search (VNS) algorithm, the application that solves the presented problem and provides the appropriate division into groups is created. The proposed approach considers the scores of a pretest, interpersonal relationships, and prosocial behaviour/openness skill of students. In order to validate our approach, an experiment was designed with 108 first-year university students of Belgrade Business School--Higher Educational Institution for Applied Studies. Experimental and control groups were divided into four-member groups. The experimental group was divided by using the proposed method and the control group by student selection and random selection. Multilevel analysis is used to determine whether there is a significant difference in learning outcomes between the two groups. The experimental results showed that students from the experimental group achieved significantly higher success than the students from the control group. In addition, computational results obtained with the proposed VNS algorithms are compared and verified with the results obtained by random (Monte Carlo) method.