This research examined whether infants tested longitudinally at 10, 14, and 16 months of age (N = 58) showed evidence of perceptual narrowing based on face gender (better discrimination of female than male faces) and whether changes in caregiving experience longitudinally predicted changes in infants' discrimination of male faces. To test face discrimination, infants participated in familiarization/novelty preference tasks and visual search tasks including female and male faces. At each age of participation, they were coded as having a female primary caregiver only or distributed caregiving experience (alternating experience with a female and male primary caregiver). Perceptual narrowing was evident for infants with a female primary caregiver, but only within the visual search task, which required location of a familiarized face among 3 novel distractor faces (exemplar-based discrimination); it was not evident within the familiarity/novelty preference task, which required discrimination between a familiarized and novel face (individual-based discrimination). Caregiving experience significantly explained individual changes in infants' ability to locate male faces during the visual search task after 10 months. These data are the first to demonstrate flexibility of the face processing system in relation to gender discrimination when there is a change in caregiver within the infants' natural environment after perceptual narrowing normally manifests.