Perceived gaze direction is an important social cue that elicits automatic orienting of spatial attention. This effect is experimentally revealed by faster response times (RTs) to targets occurring at gazed-at relative to not gazed-at locations. Although social orienting typically occurs rapidly and automatically even when eye direction is uninformative of the target location, its properties are influenced by top-down factors such as appropriate context, social status, and the familiarity of an individual. However, at present it is difficult to estimate the relative contribution of cognitive factors in social attention because past investigations have typically confounded stimulus (e.g., face or gaze direction) and cognitive factors (e.g., context). Here we investigated the role of cognitive control in social orienting by holding the stimulus factors constant and manipulating the relevance of the gaze cues by informing participants whether the depicted individual was blind or sighted. Observers studied photographs, names, and demographic information of two individuals, and were tested on the their accuracy in retaining this information. Then, these images, displaying averted eye direction, were presented as spatially non-predictive central cues in a cuing task. The results revealed that irrespective of face identity, gaze direction of individuals known to be sighted elicited typical reflexive cuing effects in contrast to gaze direction of individuals known to be blind which overall produced less robust orienting effects and failed to trigger rapid orienting of attention. Together these data suggest that social orienting is critically influenced by the extent that eye deviation is considered socially relevant.