On the “special” status of emotional faces… Comment on Yang, Hong, and Blake (2010)

Introduction

A wealth of literature suggests that emotional faces are given special status as visual objects: Cognitive models suggest that emotional stimuli, particularly threat-relevant facial expressions such as fear and anger, are prioritized in visual processing and may be identified by a subcortical “quick and dirty” pathway in the absence of awareness (Tamietto & de Gelder, 2010). Both neuroimaging studies (Williams, Morris, McGlone, Abbott, & Mattingley, 2004) and backward masking studies (Whalen, Rauch, Etcoff, McInerney, & Lee, 1998) have supported the notion of emotion processing without awareness. Recently, our own group (Adams, Gray, Garner, & Graf, 2010) showed adaptation to emotional faces that were rendered invisible using a variant of binocular rivalry: continual flash suppression (CFS, Tsuchiya & Koch, 2005). Here we (i) respond to Yang, Hong, and Blake's (2010) criticisms of our adaptation paper and (ii) provide a unified account of adaptation to facial expression, identity, and gender, under conditions of unawareness. 

Comparing adaptation to identity with adaptation to emotional expression

Moradi, Koch, and Shimojo. (2005) investigated adaptation to the identity of faces suppressed from awareness using CFS. During adaptation trials, the adapting face sometimes escaped CFS suppression and became visible. Adaptation trials were therefore separated according to the proportion of time that the adaptation face was reported as being visible. Moradi et al. found that adaptation to facial identity relied on observers' conscious awareness of the adapting stimulus; the identity aftereffect was larger when the adapting stimulus was visible for long periods. Importantly, identity adaptation was completely eliminated when the adaptation stimulus was visible for less than a second. 

Partially motivated by the “threat” literature, we used Moradi et al.'s paradigm to ask whether awareness is similarly required for adaptation to emotion. Our observers were shown emotional faces in a CFS paradigm to suppress the faces from awareness. Resultant emotional aftereffects were measured by changes in the categorization of subsequent emotional test faces. Similarly to Moradi et al., we separated our CFS adaptation trials according to how long the adaptor face was reported as being visible. However, we added a “zero dominance” category, consisting solely of trials in which observers were completely unaware of the adapting stimulus. Similarly to the previous findings with identity, adaptation to expression was substantially reduced when adaptors were suppressed from awareness. However, in contrast to identity, we found significant emotional aftereffects even after complete suppression of the adapting stimulus (Adams et al., 2010). Our results do, therefore, suggest a difference between the processing of expression and identity, under conditions of unawareness. Can we, and should we, confer a special status to emotion processing, over other types of visual processing? 

Evaluating Yang et al.'s criticisms of our study

Yang et al. (2010) dismiss our findings, proposing instead that expression aftereffects are “essentially abolished” under CFS suppression, unless spatial attention is directed toward the suppressed face. Could the apparent difference between our results for expression and Moradi et al.'s effects with identity be due to an experimental confound? Yang et al. suggest that our reported facial expression aftereffects (FEAs) may be invalid, arguing that

 

[it is] possible that FEAs produced under CFS represented residual adaptation that carried over from visible adaptation periods. This seems highly possible given that observers experienced incomplete suppression on two thirds of suppression trials, thereby increasing the periods of visible adaptation. […] In their study, on a majority of ‘suppression’ trials, observers reported intermittent periods at which the adapting face was visible, indicating that suppression during adaption was incomplete. […] FEAs may have been induced solely by periods where the adapting face was visible in dominant and suppression trials. This adaptation carry-over effect may also explain why authors found significant FEAs with briefly visible periods (500 ms) of the adapting face since these trials were intermingled with trials of prolonged visible adaptation (4 s) in their control experiment.

 

In our study, suppression was incomplete on many trials, allowing us to separate “suppression” trials according to adaptor visibility, including a “zero dominance” category. This analysis revealed a graded effect of stimulus visibility on adaptation. Could our paradigm have led to carry-over effects? We interleaved adaptation and test stimuli depicting three different adaptation emotions (happy, angry, fear). Thus, emotion-specific carry-over effects are difficult to envisage. More plausible, perhaps, would be some generalized decrease in emotion sensitivity, lowering adaptation effects across all emotion categories. However, in our paper, we explicitly guarded against either confound by comparing emotion categorization responses following (i) a relevant adaptor (adaptation emotion = test emotion), with those following (ii) an irrelevant adaptor (test emotion ≠ adapting emotion); a similar analysis was employed by Moradi et al. to evaluate their identity effects. If our aftereffects were purely the result of decreased sensitivity, or carry-over effects from previous trials (where adapting stimuli were visible), then observers' responses on “zero dominance” trials would be independent of the adapting emotion on those “zero dominance” trials. They were not independent (see Figure 2b of our paper and associated text for more detail), and we reasonably concluded that aftereffects on zero dominance trials were caused by adaptation to the invisible stimulus and not spurious carry-over effects. (A similar logic applies to our control study, where the relevant/irrelevant adaptor comparison revealed smaller but significant aftereffects following short exposure to visible adaptors that could not be attributed to carry-over effects from intermingled trials with longer adaptation durations.) 

We must find an alternative explanation, therefore, for the difference between the findings for identity and emotion, given that the two studies employed almost identical paradigms. One might argue that a specialized subcortical pathway, which is sensitive to emotion, but does not care about identity, drives our unconscious emotion effects; we noted in Adams et al. (2010) that our results are consistent with such an account. In contrast, Yang et al. argue that identity and expression aftereffects are similarly reliant on awareness and attention (see below). This suggests that neither is subcortically driven. Here we argue that expression adaptation is more robust than identity adaptation, yet this difference can be parsimoniously explained without reference to a specialized, subcortical pathway. We suggest that the signal driving identity adaptation is (on average) smaller and weaker than the signal for expression adaptation; in essence, changes in expression produce substantial image differences, which may be larger and more salient than the image-based differences across identity. In other words, there may be a quantitative rather than a qualitative difference between these tasks and effects. This is consistent with a post-hoc comparison of adaptation to identity and emotion with visible face stimuli—our control study revealed significant expression aftereffects following brief (0.5 s), visible (non-CFS) adaptors. In contrast, Moradi et al. found no identity aftereffects when collating data from all trials that induced adaptor visibility of up to a second. 

Comparing our findings with Yang et al.'s findings

Why the apparent discrepancy between Yang et al.'s (2010) findings with emotional expression adaptation and our own? Perhaps there is not really much of a discrepancy. In two experiments without an explicit attention manipulation, Yang et al. find small aftereffects (above significance in one experiment). Our results are thus in pretty good agreement—adaptation is substantially reduced in the absence of awareness, but some adaptation remains. If our measured aftereffects appear slightly more robust than Yang et al.'s, then, as noted by Yang et al., their slightly different CFS paradigm may have resulted in a greater depth of suppression. Alternatively, any differences may be attributable to attention effects; our observers were carefully and continually monitoring and reporting any awareness of the adapting stimulus—perhaps this was more engaging/demanding than the task of Yang et al.'s observers who made a single key press to terminate (rare) trials in which the adaptor became visible. 

Yang et al. suggest that spatial attention can modulate the unconscious emotion aftereffect. In their final experiment, they asked observers to detect a small decrement in the contrast of the CFS stimulus, on the same side of the visual field as the suppressed adaptor. This paradigm resulted in robust adaptation. However, Yang et al. did not compare adaptation in attended and non-attended locations, so it is difficult to conclude much regarding the role of spatial attention on unconscious adaptation. The larger resultant aftereffects (compared to their first two experiments) might be due to either (i) an overall increase in attention, rather than directed spatial attention per se, or (ii) moving their adaptation and test stimuli away from fixation. Shin, Stolte, and Chong (2009) also suggest an important role for spatial attention in face adaptation. They found that adaptation to the gender of an “invisible” female face was preserved in the attended visual field but was reduced below significance in the unattended field. This intriguing finding must be treated with caution, however, for two reasons: First, adaptors that were visible for up to 1 s were classed as “invisible,” leaving the possibility that attention boosted adaptation during short periods of visibility. Second, Shin et al.'s data may be vulnerable to carry-over effects (erroneously suggested by Yang et al. as an account of our data) as visible and invisible trials with a single adaptor were intermingled. However, the attentional modulation suggested by both Yang et al. and Shin et al. is broadly consistent with a body of evidence suggesting attentional modulation of low-level, retinotopic adaptation to simple stimuli under suppression (Bahrami, Carmel, Walsh, Rees, & Lavie, 2008; Kanai, Tsuchiya, & Verstraten, 2006). 

Interestingly, Moradi et al. (2005) showed that subsuming attention with a high-load task, all but eliminated their identity aftereffect observed with visible stimuli. This substantial modulation of identity adaptation with visible stimuli, combined with the unconscious gender effects suggested by Shin et al., leave open the possibility that Moradi et al.'s identity aftereffects might reappear with an appropriate attentional manipulation. 

Finally, Yang et al. provide indirect evidence that non-retinotopic processing may be involved in generating expression aftereffects under suppression. They argue that although contrast adaptation is reduced by CFS suppression, this is not as substantial as the reduction in emotion adaptation, leaving a role for higher level, non-retinotopic processing. This complements our more direct evidence for non-retinotopic adaptation; in contrast to Yang et al., we presented adaptation and test stimuli to opposite sides of the visual field. Thus, our measured aftereffects were entirely non-retinotopic. 

Other behavioral studies of emotional face processing under CFS

Can the CFS paradigm tell us anything else about whether unconscious emotional faces can access specialized processing pathways or mechanisms? Jiang, Costello, and He (2007) showed that upright faces emerge from CFS suppression more rapidly than inverted faces. Yang, Zald, and Blake (2007) went on to show that, in a similar CFS paradigm, fear faces were detected more rapidly than neutral or happy faces, with this “fear advantage” present for both upright and inverted faces. Yang et al.'s work left open the intriguing question of whether this apparent prioritization of fear faces relied on a threat-sensitive mechanism, or whether it can be explained by the low-level salience of the stimulus, such as local luminance and contrast modulations. Our own work (Gray, Adams, & Garner, 2010, 2011) shows that this fear-face advantage can be explained entirely by the low-level characteristics of fear faces; stimuli with the same low-level properties but whose emotional expressions are unrecognizable are similarly detected more quickly on emergence from CFS. Interestingly, we replicated an overall “face” advantage that is not attributable to low-level, image-based features. 

Conclusions

When considering behavioral evidence from CFS studies with faces, we conclude that the emotional content of face stimuli may be somewhat more salient than other aspects of faces such as identity and gender, leading to some apparent threat prioritization effects (e.g., aftereffects that are a little more robust to suppression, faster detection following CFS suppression). However, the processing of facial emotion is modulated by awareness and may also be modulated by attention, in a very similar way to other face attributes (as proposed by Yang et al., 2010). The “fear advantage” for unconsciously presented faces may be accounted for purely by basic image properties, rather than their “meaning” or emotional content per se. Does this matter, and why should we care? Importantly, all of these effects are parsimoniously explained without reference to a subcortical pathway that specializes in the detection of threat-relevant stimuli. This does not, of course, disprove the existence of such a pathway (see Pessoa & Adolphs, 2010 for related discussion). However, it might be more helpful to consider the results of Adams et al. (2010), Gray et al. (2010), and Yang et al. (2007, 2010) as consistent with the notion that our physical facial expression of fear has evolved to be salient to general-purpose visual processes. 

Acknowledgments

We thank two anonymous reviewers for their helpful feedback. 

Commercial relationships: none. 

Corresponding author: Wendy J. Adams. 

Email: [email protected]. 

Address: Highfield, Southampton, Hampshire SO17 1BJ, UK.