Attention is shifted reflexively to where other people are looking. It has been argued by a number of investigators that this social attention effect reflects the obligatory bottom-up activation of domain-specific modules within the inferior temporal (IT) cortex that are specialized for processing face and gaze information. However, it is also the case that top-down factors may modulate the activation of IT cells. Here we examined behaviorally whether reflexive social orienting is purely automatic or sensitive to top-down modulation. Participants were shown an ambiguous stimulus that could be perceived either as representing EYES or a CAR. In Experiment 1 we demonstrated between groups that an automatic shift of attention, equivalent to that triggered by a schematic FACE, occurred only when the stimulus was referred to as possessing EYES. In Experiment 2 all participants received the EYES and CAR conditions. When the stimulus was first referred to as a CAR and then as EYES, an attentional shift was only present for the EYES condition. However, when the stimulus was first referred to as possessing EYES, and then later as a CAR, attentional shifts were observed for both conditions. These data indicate that the emergence of a reflexive social attention effect is influenced by top-down mechanisms but in an asymmetrical manner. Top-down processes appear to be effective for triggering IT involvement, that is, for perceiving a stimulus as a face, which produces the social attention effect. But top-down mechanisms are ineffective once IT involvement has been triggered. That is, once a stimulus has been seen as having eyes, it continues to be seen that way, and accordingly, the social attention effect persists.