The study of socio-cognitive abilities emerged from intelligence research, and their specificity remains controversial until today. In recent years, the psychometric structure of face cognition (FC)--a basic facet of socio-cognitive abilities--was extensively studied. In this review, we summarize and discuss the divergent psychometric structures of FC in easy and difficult tasks. While accuracy in difficult tasks was consistently shown to be face-specific, the evidence for easy tasks was inconsistent. The structure of response speed in easy tasks was mostly--but not always--unitary across object categories, including faces. Here, we compare studies to identify characteristics leading to face specificity in easy tasks. The following pattern emerges: in easy tasks, face specificity is found when modeling speed in a "single task"; however, when modeling speed across "multiple," "different" easy tasks, only a unitary factor structure is reported. In difficult tasks, however, face specificity occurs in both single task approaches and task batteries. This suggests different cognitive mechanisms behind face specificity in easy and difficult tasks. In easy tasks, face specificity relies on isolated cognitive sub-processes such as face identity recognition. In difficult tasks, face-specific and task-independent cognitive processes are employed. We propose a descriptive model and argue for FC to be integrated into common taxonomies of intelligence.