We review and provide empirical evidence to show that infants categorize and process male and female faces differently, with an advantage in processing female faces. To understand this asymmetry in categorization and processing of male and female faces, we evaluate three mechanisms influencing infant categorization of male and female faces: differential experience with female and male faces; early visual preferences for female vs. male faces; and range of physical differences among category exemplars. The paper concludes with a developmental trajectory for infant acquisition of face categories proposed within a framework that reflects current knowledge and theory in the infant categorization and face processing literatures. The proposed developments have important implications for the existing infant face perception literature and infant learning about females and males.