Psychophysics of emotion: The QUEST for Emotional Attention

Abstract

To investigate the mechanisms involved in automatic processing of facial expressions, we used the QUEST procedure to measure the display durations needed to make a gender decision on emotional faces portraying fearful, happy, or neutral facial expressions. In line with predictions of appraisal theories of emotion, our results showed greater processing priority of emotional stimuli regardless of their valence. Whereas all experimental conditions led to an averaged threshold of about 50 ms, fearful and happy facial expressions led to significantly less variability in the responses than neutral faces. Results suggest that attention may have been automatically drawn by the emotion portrayed by face targets, yielding more informative perceptions and less variable responses. The temporal resolution of the perceptual system (expressed by the thresholds) and the processing priority of the stimuli (expressed by the variability in the responses) may influence subjective and objective measures of awareness, respectively.

Introduction

Converging evidence in the literature shows that facial expressions of emotion may benefit from enhanced processing in visual perception (Palermo & Rhodes, 2007). Subsumed under the general umbrella term of “emotional attention,” these findings nevertheless raise a number of questions with regard to the nature of the automaticity of this process (e.g., rapid, non-conscious, ballistic, and capacity-free) and the underlying mechanisms (Vuilleumier, 2005). As we start to unravel the psychophysical characteristics of emotion perception in faces (Fox, Oruc, & Barton, 2008; Neth & Martinez, 2009), the psychophysical characteristics of emotional attention remain, however, largely unknown. 

We define attention as that part of awareness underlying the ability to process selected stimuli. Selective attention to emotion-laden information is studied across a wide range of paradigms that tap into very different aspects of attention, which may sometimes be difficult to reconcile. For instance, in a visual search paradigm with schematic emotional facial expressions, Horstmann and Becker (2009) showed that emotion-laden information only produced shorter reaction times when it was task-relevant. These results, and others, illustrate that attention to emotional stimuli is biased by both top-down (i.e., task-driven) and bottom-up (i.e., stimulus-driven) mechanisms that eventually yield perception (Beck & Kastner, 2009). Therefore, any successful attempt at characterizing the role of perceived emotion in visual processing necessarily involves innovative methods to disentangle voluntary from involuntary attention. 

The processes leading to awareness of emotion-laden stimuli may be investigated in experimental settings employing backward masking (Breitmeyer & Ogmen, 2000; Wiens, 2006), which rules out conscious recognition as a factor in emotional processing. In this paradigm, a brief presentation of an emotional stimulus (target) is immediately followed by a visual mask that limits the processing of the target, and often prevents its recognition. If the interval between the onsets of the two stimuli is sufficiently short, participants may not perceive the target, and report only seeing the mask. In this context, a threshold refers to the minimum display duration necessary for the conscious perception and report of the target. 

Characterizing the involvement of attention in emotion perception is, however, made difficult by the fact that an observer may perceive a significant amount of information about the target even in the absence of conscious awareness (e.g., Szczepanowski & Pessoa, 2007). Threshold measurement thus distinguishes subjective measures of threshold (i.e., based on the observer's reported perception) from objective measures of threshold (i.e., their actual performance). Arguably, attention may have a strong impact in both cases and may involve different processes, thus influencing conscious experience in very different ways. 

In the present study, we used the QUEST adaptive staircase procedure (Watson & Pelli, 1983) to evaluate the extent to which perceptual and attentional systems are sensitive to emotional stimuli. We applied this adaptive psychometric method to study covert emotion processing and measured the minimum display duration needed by participants to perform a gender decision task on emotional faces, portraying fearful, happy, and neutral facial expressions. In this two-alternative forced-choice paradigm (possible answers were either male or female), we considered the display duration that yields 75% correct gender decisions as the threshold (Klein, 2001; Watson & Pelli, 1983). The curve representing the probability of a correct discrimination response over the increasing stimulus strength (i.e., display duration) is called a psychometric function (PF). In QUEST, the algorithm assumes the observer's PF to follow a Weibull distribution and determines the next stimulus value to be presented on the basis of the participant's response to previous trials. As the experiment goes on, knowledge on the observer's PF accumulates in the form of the distribution of trials at all possible values. At the end of the block, in addition to the threshold, which is equivalent to the location of the PF, we also consider the distribution of trials over the whole block, which is equivalent to the slope of the PF. The slope is a measure of the variability of observers' judgments. Shallow slopes result from high variability and indicate that the transition between chance and 100% correct is spread out. Steep curves indicate that there is a neat transition between chance and 100% correct. The distinction between threshold and variability is important, as attention has been shown to affect both variables. There is ample evidence that accuracy (i.e., percent correct responses) improves when participants voluntarily attend a stimulus (e.g., Bashinski & Bacharach, 1980; Cheal & Lyon, 1991; Dosher & Lu, 2000), indicating that perceptual thresholds decreased. The situation is less clear for involuntary shifts of attention. Typically, involuntary shifts of attention are triggered by a peripheral flash that does not predict the target location. Recently, Prinzmetal, McCool, and Park (2005) demonstrated that involuntary attention does not affect perceptual accuracy. However, involuntary attention made judgments less variable (Prinzmetal, Nwachuku, Bodanski, Blumenfeld, & Shimizu, 1997; Prinzmetal & Wilson, 1997). Thus, improvements of variability without changes in accuracy may be considered a signature of involuntary (or automatic) attention. 

We hypothesized that the emotion portrayed by the faces will facilitate their processing and yield better performance in a gender decision task. Facilitation may reduce thresholds, variability, or both. Most likely, facilitation is mediated by the larger capacity of emotional stimuli to attract attention. 

Methods

Participants

Thirty-one participants (15 females, mean age 29.4 years, SD = 7.64) took part in this experiment, in exchange for 10 SFr. All participants were right-handed (Edinburgh Handedness Inventory; Oldfied, 1971), had normal or corrected-to-normal vision, and had no history of psychiatric or neurological diseases. 

Apparatus

Stimuli were presented on a 17-inch CRT screen, with a refresh rate of 100 Hz, and a resolution of 1024 × 768 (Wiens, 2006; Wiens et al., 2004). The screen was placed 60 cm away from the head of the participant. Participants were comfortably seated in an armchair, with their head in a chin rest to ensure the distance to the screen was held constant throughout the experiment. The experiment was administered using MatLab, and the Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997). Participants indicated the gender of the target face at the end of each trial using the two buttons of a mouse. 

Stimuli

Targets consisted of 10 faces (five females), subtending 18° × 15° visual angle. The faces portrayed 4 controlled combinations of facial action units, depicting expressions recognized as happy, fearful, and two types of neutral expressions, which were repeated 6 times per block. To control for perceptual effects, emotional facial expressions along with the faces from one of the neutral conditions showed an open mouth. The faces for the other neutral condition showed a closed mouth. 

The faces were produced using our software FACSGen (Roesch et al., in press). FACSGen is a tool that allows for the creation of unique, realistic, static or dynamic, synthetic 3D facial stimuli based on the Facial Action Coding System (Ekman, Friesen, & Hager, 2002). It provides control over the intensity of the facial action units and allows researchers to apply the same facial expression to any face produced with FaceGen Modeler 3.2 (http://www.facegen.com/; Corneille, Hugenberg, & Potter, 2007; Moradi, Koch, & Shimojo, 2005; Shimojo, Simion, Shimojo, & Scheier, 2003). As part of a separate judgment study, 37 participants rated the extent to which the following emotions could be perceived in the facial expressions used in this experiment: anger, disgust, fear, happiness, sadness, and surprise. A scale was provided, anchored in 0 (“not at all”) and 100 (“enormously”). To determine whether participants could discriminate the emotions portrayed in the faces, repeated measures analyses of variance (ANOVAs) were performed on the ratings for each target emotion. There was a significant effect for target emotions (ps < 0.001), indicating that participants reliably recognized the target emotion portrayed by the faces. 

Masks consisted of neutral FACSGen faces, for which the inside of the face was divided in 10 × 10 pixel squares, and scrambled randomly. Each face was converted to grayscale and analyzed in MatLab to extract the mean pixel luminance and contrast range. Statistical analyses confirmed that experimental conditions did not differ for luminance and contrast (Pourtois, Grandjean, Sander, & Vuilleumier, 2004). 

Procedure

Trials and blocks

The experiment consisted of three blocks of 240 trials each. Participants were asked to determine the gender of the targets. Instructions contained practice trials presenting targets (different from the ones in the experimental blocks) with decreasing durations. Between blocks, participants were forced to a resting period of at least 3 min. 

Each trial began with a fixation cross, subtending 1° × 1° visual angle, displayed for 500 ms ( Figure 1). Participants were instructed to focus on the fixation cross and to maintain fixation throughout the trials, as this was the optimal way to perceive each face as a whole, and gather a maximum amount of information for the task. 

Figure 1

 

(a) Example of a trial. (b) Examples of target faces. The faces were produced with FACSGen, see Roesch et al. (in press) for details.

A trial consisted of 4 stimuli, presented without any inter-stimulus interval. The target appeared immediately after the fixation cross. The display duration of the target varied from trial to trial on the basis of the staircase QUEST procedure (Farell & Pelli, 1999; Watson & Pelli, 1983), described later in this section. Three masks, randomly chosen from a pool of 17, followed the target. Each mask was presented for 200 ms. At the end of the trial, participants were asked to determine the gender of the face in a two-alternative forced-choice procedure. 

QUEST: A Bayesian adaptive psychometric method

The estimation of thresholds is the core of psychophysics, and several methods are available. These methods can be separated into two broad strategies (Macmillan, 2001). The first (and earlier) strategy presupposes a priori knowledge of the psychometric function studied, in which performance increases from 0, or chance level, to perfect performance. Researchers typically sample this function at several points (usually more than five), and determine what point constitutes the threshold for each observer (method of constant stimuli). The second strategy is said to be an adaptive procedure, in which stimulus values are chosen on the basis of the observer's performance in previous trials. Results of behavioral experiments, and simulations, showed that adaptive methods are more accurate, and more efficient, than the method of constant stimuli (Watson & Fitzhugh, 1990, see also the special issue on threshold estimation in Perception and Psychophysics, 2001). An alternative strategy consists of letting the observer control the level of stimulus until it becomes barely detectable (method of adjustment), but it only provides very little information about the shape of the PF. 

QUEST (Farell & Pelli, 1999; Watson & Pelli, 1983) is an adaptive staircase procedure. It has been used in studies investigating both low-level perceptual filters, like spatiotemporal integration of visual information (Melcher & Morrone, 2003) or contrast sensitivity (Burr, Morrone, & Ross, 1994), and higher level object recognition, like letters (e.g., Solomon & Pelli, 1994) or faces (Fox et al., 2008). One study used it to investigate the effect of attention and emotion on contrast sensitivity (Phelps, Ling, & Carrasco, 2006). In QUEST, the algorithm assumes the observer's PF follows a Weibull distribution (see 1). The estimated parameters of this distribution are updated after each trial on the basis of the observer's performance and a new stimulus value is chosen for the next trial. In the original algorithm, stimulus values are chosen at the best quantile of this updated function. The final estimate of the threshold is at the mean of the function. The Psychophysics Toolbox includes MatLab code that implements this procedure. 

In our experiment, we used QUEST to estimate the minimum display duration required by participants to make a gender decision on faces portraying four facial expressions. In each block, four QUESTs ran concurrently and were randomly interleaved. After each block, we recorded the mean and the standard deviation of the participant's PF for each condition. The mean of the PF corresponds to the threshold of the observer, which is the minimum display duration required for the observer to make a 75% correct decision on the gender of the face targets (male or female). The standard deviation of the PF represents the dispersion of the Weibull distribution of the trials at the end of the block. Because the total number of trials was held constant (60 trials per QUEST), more trials around the threshold resulted in smaller standard deviations. It thus indirectly corresponds to the number of trials needed by QUEST to estimate the threshold of the observer. More generally, the standard deviation of the Weibull distribution corresponds to the variability of participants' responses in a block of trials. 

Results

One male participant systematically gave incorrect answers in the first block. We therefore discarded his data from the analyses involving this block. We conducted exploratory analyses to assess inter-individual differences. We then performed targeted analyses to address a learning effect over the blocks, and the extent to which the emotion of the targets influenced performance. 

Exploratory analysis

Exploratory analysis of the data showed great inter-individual differences. Frequency analysis of the thresholds obtained after each block showed a skewed distribution (see Figure 2a, for an example). Across all emotions, the mean thresholds for the three blocks were, respectively, 81.2, 51.9, and 52.3 ms, indicating that, on average and across all emotions, participants required the targets to be displayed for at least 50 ms to be able to make a correct decision about the gender of the targets. 

Figure 2

 

(a) Histogram of the distribution of thresholds across emotions for the last block. As expected, the mean of the distribution of thresholds was about 50 ms, which corresponds to the perceptual limit often reported and used in backward masking studies. Notice, however, that some thresholds (in this block, 68 out of 124) were below 50 ms, indicating that some participants were able to make a gender decision on faces presented for a duration less than 50 ms. (b) Mean cumulative density functions of the participants' psychometric functions for each experimental condition, estimated from the thresholds and the standard deviations for the first and last blocks. Thin lines represent results of the first block. Thick lines represent results of the last block. (c) Mean thresholds for each emotion after the first and the last blocks, along with the average over all blocks. (d) Mean standard deviations of the participants' psychometric functions for each emotion, after the first and last blocks, along with the average over all the blocks. All error bars correspond to 1 SEM.

We also computed the mean psychometric functions for each emotion, graphically representing the unfolding of the probability for a correct response as the duration of the target increases. As shown in Figure 2b, the estimated cumulative Weibull distributions vary as a function of emotion. Of importance are the slopes and the points where the probability to make a correct decision becomes greater than 0.75, corresponding to the threshold. 

Targeted analysis

Univariate Analyses of Variance (ANOVAs) were performed to assess an effect of the gender of participants across all blocks and all experimental conditions (two-sided tests). As expected, the ANOVA on thresholds yielded a main effect of Gender, F(1,358) = 7.155; p < 0.01, indicating that female participants required faces to be displayed for a shorter period of time to make a correct gender decision ( M = 52.1 ms; SD = 32.01), compared to male participants ( M = 62.9 ms; SD = 43.56). The ANOVA on standard deviations did not yield a main effect of Gender, F(1,358) = 3.454; p = 0.064, even though the data followed the same trend (Female: M = 6.23 ms, SD = 7.0; Male: M = 7.79 ms, SD = 8.79). Because gender differences do not enter the scope of our research, we did not analyze these effects further and entered Gender as a covariate in subsequent analyses. 

Univariate ANOVAs were performed, treating Blocks (1, 2, and 3) and Emotion (happy, fearful, neutral with mouth open, and neutral with mouth closed) as fixed factors, Participants as a random variable, and Gender as a covariate (two-sided tests). The ANOVA on thresholds ( Figure 2c) yielded a main effect of Blocks, F(2,58) = 9.275, p < 0.001, η ² = 0.242. Contrast analyses confirmed that thresholds were significantly higher for the first block compared to the two other blocks ( p < 0.01). The analysis also yielded a main effect of Emotion, F(3,87) = 3.677, p = 0.015, η ² = 0.113, but post hoc Tukey analyses only showed a significant difference between the two neutral conditions ( p = 0.033). 

The ANOVA on standard deviations ( Figure 2d) did not yield a main effect of Blocks, F(2,58) = 0.471, p = 0.627, but it did yield a main effect of Emotion, F(3,87) = 4.31, p = 0.007, η ² = 0.129. Post hoc Tukey analyses confirmed that the number of trials needed by QUEST to estimate the threshold was significantly smaller for emotional targets than for neutral targets ( ps < 0.05). No difference was found between fearful and happy targets ( p = 0.991). 

Discussion

The present work examined the minimum display duration needed by participants to make a correct gender decision on emotional faces. We compared the perception of fearful, happy, and neutral facial expressions. We used a staircase procedure, QUEST (Farell & Pelli, 1999; Watson & Pelli, 1983), to estimate the minimum display durations (the thresholds) needed for this task, and a measure of the variability of the participants' responses (the standard deviation of the resulting distribution of trials at the end of the block). 

Thresholds for all emotions were subject to great inter-individual differences. Across blocks, the average threshold was about 50 ms, which corresponds to the archetypal perceptual limit often reported and used in backward masking studies (Esteves & Öhman, 1993; Morris, Öhman, & Dolan, 1998; Öhman, 2002; Ruys & Stapel, 2008a, 2008b; Stapel, Koomen, & Ryus, 2002; Whalen et al., 1998, just to name a few). However, quite a few thresholds were below the mean of 50 ms, indicating that, if 50 ms was an average, some participants could perform the task under this threshold. This finding is consistent with studies addressing the objective and subjective awareness of fear perception (Pessoa, Japee, & Ungerleider, 2005; Szczepanowski & Pessoa, 2007), in which participants can sometimes detect fearful faces displayed at a duration as short as 17 ms. We extended these results by showing that this was also the case for happy and neutral faces. In contrast to these studies, however, we engaged participants in a forced-choice gender decision task, orthogonal to the aims of the experiment, to shift the focus away from the emotion contained in the faces, thus allowing us to investigate covert emotion processing. Likewise, whereas these authors used faces from various data sets (Ekman set, Karolinska Directed Emotional Faces, and a set developed by Alumit Ishai at the National Institute of Mental Health), we used a set of FACSGen synthesized faces, extensively controlled and systematically manipulated to display the exact same combinations of facial action units (Roesch et al., in press). 

Our hypothesis concerned the extent to which the emotion portrayed by the faces facilitate their processing. Analysis of the variability of responses throughout the blocks revealed a significant main effect of emotion. Post hoc analyses indicated that responses were less variable for emotional targets compared to the neutral targets. 

Our procedure allowed us to separate two aspects of the processing involved in a gender decision task with emotional facial expressions. Each of these aspects can influence and limit the processing of perceived stimuli. We propose the following distinction: 

1.  
   The measure of thresholds may relate to the subjective awareness reported by the participants (Szczepanowski & Pessoa, 2007). It refers to the limit of the conscious perceptual experience. The absence of reliable differences between emotional and neutral targets is consistent with the view that conscious visual experience (which has been shown to resemble an all-or-nothing phenomenon, Sergent, Baillet, & Dehaene, 2005) is subject to the same temporal resolution regardless of whether it is emotionally salient or not.
2.  
   The variability of the participants' responses (expressed by the standard deviation of the resulting distribution of trials at the end of each block) may relate to a more objective measure of awareness that may be less easily accessed consciously by participants, and which, in other paradigms, often yields smaller thresholds than subjectively reported. The fact that the QUEST algorithm needed less trials to estimate the observers' thresholds (i.e., resulting in smaller standard deviations) for emotional targets shows that emotional targets benefited from a heightened processing priority, compared to neutral targets, which could in turn benefit an orthogonal task (gender decision). This view is coherent with the results showing that emotional material can influence higher levels of processing even if presented subliminally, without conscious awareness of the participants (Ruys & Stapel, 2008a, 2008b; Stapel et al., 2002).

This interpretation is supported by studies addressing the effect of attention on perceptual processing. In a series of experiments, Prinzmetal et al. (1997) showed that attention did not render the percept of achromatic targets more intense, but it significantly reduced the variability of the responses. They replicated this effect in several modalities, spanning from lower level features like color and spatial frequencies to higher level features like the length or the orientation of the target (Prinzmetal, Amiri, Allen, & Edwards, 1998; Prinzmetal & Wilson, 1997). They concluded that “attention reduces the variability of responses but it does not cause biases in perception. The searchlight of attention illuminates in terms of providing more information, but it does not illuminate in terms of changing contrast” (Prinzmetal et al., 1997, p. 408). 

The phenomenology of this effect can be described as the observer being more certain of the identity of the perceived targets. In other words, attention would reduce the range of possible answers, by increasing the number of information samples taken by the perceptual system, for instance. In a connectionist framework, this could be achieved by either more perceptual neurons firing, or perceptual neurons firing at a higher rate. This would explain why attention has often been shown to increase the activation of perceptual brain regions (e.g., Esterman et al., 2008; Kastner & Ungerleider, 2000; Vuilleumier, 2005; Zelano et al., 2005). In contrast, thresholds for emotional stimuli did not differ from thresholds for neutral stimuli. In light of previous research on the distinction between involuntary and voluntary attention, the pattern of results suggests that emotional stimuli elicit involuntary shifts of attention. If voluntary attention had been involved, thresholds would also be expected to decrease. This was not the case, making it unlikely that participants voluntarily chose to attend more to emotional than to neutral stimuli. 

To conclude, emotional facial expressions led to less variability in the responses to a gender decision task, compared to two types of neutral targets. We thus suggest that attention may have been automatically drawn by the emotion portrayed by the face targets, leading to more informative perceptions and less variable responses on this orthogonal task. This proposal is supported by Phelps et al. (2006) who showed that a perception of emotion led to a heightened perception of a lower level feature like a contrast gradient in a subsequent stimulus. Importantly, in our study the emotional effect was observed for both negative (fearful faces) and positive (happy faces) emotional stimuli, as predicted by the component process model of emotion (Scherer, 2001). 

Appendix A

The QUEST algorithm is a psychometric staircase procedure to estimate thresholds (Watson & Pelli, 1983). It makes use of responses to previous trials to guide the testing. More precisely, it places each (next) trial at the current most probable Bayesian estimate of threshold, assuming the observer's psychometric function (PF) follows a Weibull distribution. 

The corresponding cumulative Weibull distribution is defined by the following equation:  

$P\left(x\right)=\delta ·\gamma +(1-\delta )(1-(1-\gamma )·e(-{10}^{\left(\beta \right(x-T+error\left)\right)}\left)\right),$

(A1)

where x represents log10 contrast relative to threshold, with specified upper asymptote 1 − δ to take into account the observer's occasional errors when the intensity of the stimulus is well above threshold (Watson & Pelli, 1983). β, δ, γ are not free parameters and should be specified before the start of the experiment. The values used in our experiment were chosen after Watson and Pelli's original recommendations for a two-alternative forced-choice procedure. They are indicated inside parentheses. β controls the steepness of the PF (3.5). δ is the fraction of trials on which the observer presses at random (0.05). γ specifies the probability of a success at zero intensity (0.5). T is a prior estimate of the threshold. In a two-alternative forced-choice procedure, the aim of the procedure is to find the threshold that yields 75% correct decision. The error term is introduced so that T will be the ideal testing point. We ran our analyses on the thresholds recorded at the end of each block and the standard deviations of the resulting distributions of responses. 

Acknowledgments

This research was partly supported by HUMAINE—6th Framework Programme IST Multimodal Interfaces and the National Centre of Competence in Research (NCCR) in Affective Sciences financed by the Swiss National Foundation (No. 51NF40-104897). The authors would like to thank Gernot Horstmann and an anonymous reviewer for comments on a preliminary version of the manuscript. 

Commercial relationships: none. 

Corresponding author: Etienne B. Roesch. 

Email: [e-mail]. 

Address: Swiss Centre for Affective Science, 7 Rue des Battoirs, 1205 Genève, Switzerland.