Of toothy grins and angry snarls—Open mouth displays contribute to efficiency gains in search for emotional faces

Abstract

Abstract

Abstract:

Abstract  The emotional face-in-a-crowd effect is widely cited, but its origin remains controversial, particularly with photorealistic stimuli. Recently, it has been suggested that one factor underlying the guidance of attention by a photorealistic emotional face in visual search might be the visibility of teeth, a hypothesis, however, that has not been studied systematically to date. The present experiments manipulate the visibility of teeth experimentally and orthogonally to facial emotion. Results suggest that much of the face-in-a-crowd effect with photorealistic emotional faces is due to visible teeth, and that the visibility of teeth can create a search advantage for either a happy or an angry target face when teeth visibility and facial emotion are confounded. Further analyses clarify that the teeth visibility primarily affects the speed with which neutral crowds are scanned, shedding new light on the mechanism that evokes differences in search efficiency for different emotional expressions.

Introduction

The emotional-face-in-a-crowd effect refers to the search advantage of a particular facial expression (e.g., an angry face) over alternative expressions (e.g., a happy face), in a crowd of faces. The search advantage was initially assumed to be categorical and restricted to angry faces (Hansen & Hansen, 1988): an angry face among happy faces was supposed to be detected with a single glance, while a happy face among angry faces was assumed to require a demanding, serial examination of the faces in the crowd. Such a result seemed to fit within an evolutionary psychology framework, that stimuli that signaled threat during human evolution are detected by dedicated and highly efficient specialized modules (e.g., Öhman, Lundqvist & Esteves, 2001; Öhman & Mineka, 2001). 

Later research revealed that such categorical differences in search processes favoring angry faces are rare unless perceptual features are confounded with facial expression (Purcell, Stewart, & Skov, 1996). Without perceptual confounds, search for an emotional face more typically produces a gradual difference in search efficiency (e.g., Horstmann & Bauland, 2006; Lipp, Price, & Tellegen, 2009b; Pinkham, Griffin, Baron, Sasson, & Gur, 2010), so that one face (e.g., an angry face) is found more efficiently than another face (e.g., a friendly face), while none of the faces are found instantly, at first glance. In this later research, it was also not always the angry face that was found more efficiently. Instead, a happy face advantage has been obtained in some studies (e.g., Calvo & Nummenmaa, 2008; Juth, Lundqvist, Karlsson, & Öhman, 2005; Becker, Anderson, Mortensen, Neufeld, & Neel, 2011). 

The inconsistency of the results with respect to which of the two expressions is found faster—the happy face or the angry face—suggests that the emotional expression category cannot possibly be the only important factor determining the face-in-a-crowd effect. In fact, it is striking that angry face advantages (e.g., Hansen & Hansen, 1988; Horstmann & Bauland, 2006) prevail with the Pictures of Facial Affect developed by Ekman and Friesen (1976), whereas happy face advantages (e.g., Calvo & Nummenmaa, 2008; Juth, Lundqvist, Karlsson, & Öhman, 2005; Williams, Moss, Bradshaw, & Mattingley, 2005) prevail in the KDEF (Lundqvist, Flykt, & Öhman, 1998) or the NimStim data base (Tottenham et al., 2009). This coincidence of advantage type and stimulus set casts serious doubts on the hypothesis that the search advantage of either emotional face category type is meaningful per se and suggests that other factors that are independent from emotional facial expression category drive the effects. 

The focus of the present study is on mouth opening and visible teeth. In particular, we examine whether and to what extent the opening of the mouth and the concomitant showing of the teeth may play a decisive role in causing the face-in-a-crowd effect. Both the happy and the angry face naturally come in two variants, one with a closed mouth and concealed teeth (the smile and the compressed lips frown), and one with an open mouth and visible teeth (as seen in laughing and in snarling). 

The visibility of teeth in emotional faces can plausibly be expected to have profound effects on search performance, because it is a perceptually salient facial component. Yet, its role in the face-in-the-crowd effect has not been systematically investigated to date. Importantly, as previous studies generally did not control for the visibility of teeth in visual search for emotional faces, differences in the visibility of teeth may be able to explain the inconsistencies in the literature, in particular, that some studies reported an angry face advantage, whereas others reported a friendly face advantage (see also Lipp, Price, & Tellegen, 2009a). 

Variants of this teeth visibility hypothesis have been noted in the literature several times before (e.g., Calvo and Nummenmaa, 2008; Lipp, Price, & Tellegen, 2009a); however, supporting experimental evidence is scarce. A study by Calvo and Nummenmaa (2008) contains the best currently available evidence for the importance of visible teeth in visual search displays. These authors found a happy face advantage and demonstrated that such an advantage can also be obtained by presenting only the isolated mouth regions of the faces, suggesting that the mouth region may be solely responsible for the search advantage. They also found a larger number of teeth displays in the friendly faces than in the angry faces, and that visible teeth displays, when analyzed separately, had a large effect on search times and eye movements. On average, faces with teeth displays could be found much faster than faces without teeth displays, suggesting that the teeth displays are effective in guiding attention to the target. 

Calvo and Nummenmaa's (2008) finding, however, is limited by the fact that only one of their happy faces did not show teeth. Since this closed-mouth target face presumably differed from the other pictures in other characteristics as well, it is possible that prolonged search times for this face were due to factors other than the visibility of teeth. 

Aim of the present study

The prime aim of the present study was to obtain clear experimental evidence for the role for visible teeth in visual search for emotional faces. Toward this aim, teeth visibility was manipulated experimentally, while the same individuals contributed open-mouth and closed-mouth displays of happiness and anger, respectively, as well as neutral faces. Furthermore, we critically tested whether visible teeth might explain the empirical inconsistency with respect to happy face versus angry face advantages in visual search. In particular, we tested whether confounding facial emotion category with teeth visibility biases the direction of the face-in-a-crowd effect. That is, we tested whether any emotional expression category, whether it be a happy or an angry face, is found more efficiently in visual search when the visibility of teeth is confounded with it. 

Extending the efforts made by previous research, we additionally conducted in-depth analyses of the processes entailed in finding the emotional faces. Quite a number of accounts have been proposed for the face-in-the-crowd effect; some assume that the emotional target guides or even captures attention due to an abstract threat feature (e.g., Eastwood, Smilek & Merikle, 2001, 2003; Öhman, Lundqvist & Esteves, 2001); others assume that attention is not easily disengaged from an emotional, in particular a threatening, distractor (Fox, Russo, & Dutton, 2002); still others assume that perceptual characteristics of the presented facial stimuli either guide attention to the target (Horstmann & Ansorge, 2009; Horstmann & Becker, 2008; Horstmann, Heumann & Borgstedt, 2006) or influence the speed of checking and rejecting the distractor faces that constitute the crowd (e.g., Horstmann & Bauland, 2006; Horstmann, Becker, Bergmann & Burghaus, 2010; Horstmann, Scharlau & Ansorge, 2006). 

The present study does not target all these hypotheses; in particular we will not focus on the question whether effects are due to emotional or perceptual factors. Rather, we examined whether differences in search efficiency are due to attentional guidance by the emotional target, or to processes entailed in scanning and rejecting the neutral distractor faces that constituted the crowd in the present study. We will refer to these two processes as “guidance” and “distractor rejection,” respectively. At first glance, it may seem obvious that differences in visual search for emotional targets reflect on the ability of the emotional target to guide attention to its position, in particular when different emotional target categories are presented within a constant set of distractors (e.g., Eastwood, Smilek & Merikle, 2003). However, it is also possible that one facial category is found faster because distractor rejection proceeds faster (e.g., Horstmann, 2009), for instance, when the expected target is more dissimilar from the crowd than another target (Duncan & Humphreys, 1989; Farmer & Taylor, 1980). 

The present study was designed with particular sensibility to the various criticisms targeted against previous visual search experiments with emotional faces. In detail, the number of faces within a crowd (set size) was varied, so that differences in search efficiency could be derived from the set-size reaction-time function, which reliably measures attentional processes and is not influenced by decision-level processes following visual search (cf. Frischen, Eastwood, & Smilek, 2008, see also Becker, Anderson, Mortensen, Neufeld, & Neel, 2011, for further elaboration of this point). Furthermore, the same neutral distractors were presented in all conditions, such that none of the results could be attributed to confounds between target category and distractor category. Additionally, different targets were presented in separate blocks to secure that each of the targets was searched for with the same priority (cf. also Frischen, Eastwood, & Smilek, 2008, and Becker, Anderson, Mortensen, Neufeld, & Neel, 2011, for further discussion). In the first experiment, we moreover used color photographs of intact faces, and all crowds of faces consisted of different individual displayers, to avoid criticism about poor ecological validity (Pinkham, Griffin, Baron, Gur, & Sasson, 2010). Finally, photos of 10 different individuals served as target stimuli in order to avoid that circumstantial saliency differences between individual displayers to bias the results. 

Experiment 1: Method

Participants

Six men and six women with a mean age of 28 years (range 20–35 years) participated in the experiment. 

Design

The experiment used a 3 (set size: 2 vs. 4 vs. 9 faces) × 2 (target presence: present vs. absent) × 2 (target emotion: angry vs. happy) × 2 (target face mouth opening: open vs. closed) × 2 (target gender: male vs. female) × 5 (target face identity) design. Effects of target gender and target face identity were included for reasons of counterbalancing only. The four different target types (which resulted from combining target face emotion and target face mouth opening) were presented in different blocks of 120 trials each. Distractors were of the same type in all blocks, that is, they were drawn from the same pool of closed-mouth neutral expressions. The four blocks were presented in an order counterbalanced across participants, according to a Latin square. Targets and distractors were chosen from a set of 10 individuals. For each trial, one individual was assigned to the target role; distractors—all showing a neutral face—were randomly chosen from the nine remaining individuals. For each target present trial there was a corresponding target absent trial, with the only difference being that the target individual from the target present trials was now a nontarget (showing a neutral face). Each individual face served equally often as a target. 

Apparatus

ERTS 3.36 (Berisoft Cooperations, Frankfurt, Germany), run on an x86 computer, controlled the experiment. The computer was connected to a 19-inch color monitor (800 × 600 pixels at 84.8 hertz) for stimulus presentations, and to a keyboard to collect the manual responses. A chin-rest secured a viewing distance of 60 centimeters. The experiment was conducted in a dimly lit and quiet room. 

Stimuli

Stimuli were drawn from the NimStim stimulus base (Tottenham et al., 2009) and presented against a black background. Ten individuals were selected for the face stimuli, five women (1, 2, 3, 7, 8) and five men (20, 21, 22, 23, 24). Each individual was presented with five different expressions: (1) a neutral face with a closed mouth, (2) a happy face with a closed mouth, (3) a happy face with an open mouth, (4) an angry face with a closed mouth, and (5) an angry face with an open mouth (see Figure 1). 

Figure 1

 

Examples of stimuli presented in the experiment.

While there was only one type of open-mouthed angry faces in the NimStim stimulus base, two types of open-mouthed happy faces were available. We chose the “x-happy” type, because this type appeared to match the angry faces in implied emotion intensity. This selection was confirmed in a pilot study in which 16 participants (eight men) judged the intensities of the displayed emotional expressions on a 11-point visual analog scale ranging from 0% to 100% (Question: “How intense is the emotion experienced by this person?”; Table 1 shows the mean intensities). The instructions stated that 0% was to be interpreted as “The person experiences no emotion at all” and 100% was to be interpreted as “The person experiences an emotion as strong as conceivably possible.” The faces and the scales were presented on paper, with the faces printed in color. An ANOVA of the ratings using emotion category (angry vs. happy), mouth opening (closed vs. open), and gender of displayer (male vs. female) as variables obtained a strong effect of mouth opening, F(1, 15) = 252.9, p < 0.001, η_p² = 0.944, intensity of emotion was judged to be significantly greater in the open-mouth faces than in the closed-mouth faces, for both emotions of happiness and anger. There was no main effect of emotion category, F(1, 15) = 1.2, p = 0.292, η_p² = 0.074, confirming our assumption that the angry and the x-happy faces imply comparable degrees of emotion intensity. Additionally, there was a marginally significant main effect of gender F(1, 15) = 3.8, p = 0.070, η_p² = 0.203, due to a trend towards lower intensity ratings for the male than for the female faces (58 vs. 61), and a significant interaction between mouth opening and gender, F(1, 15) = 5.3, p = 0.037, η_p² = 0.258, reflecting that intensity of expression was similar for both genders in closed-mouth faces, t(15) = 0.35, whereas in open-mouth faces females were rated as more intense than males (72 vs. 77), t(15) = 4.04, p = 0.001. The difference, however, was small and amounted to only 5 scale points. 

Table 1

 

Subjective ratings of emotion intensity, ranging from 0% to 100%.

	Angry		Happy
	Closed	Open	Closed	Open
Male	44	74	44	71
Female	46	78	43	75

In the main experiment, individual pictures subtended 3.24° × 4.00° and were rendered as 256-color bitmaps (the same color palette was used for all stimuli). Columns between pictures were 0.48° wide. The stimuli were displayed in groups of two, four, or nine, and were presented next to one another (subtending 5.96° × 4.00°), in a regular 2 × 2 array (subtending 5.96° × 8.48°) or in a regular 3 × 3 display (subtending 10.66° × 12.94°), respectively. 

In target absent displays, two, four, or nine neutral facial expressions of two, four, or nine different individuals were presented, drawn randomly from the entire set of 10 individuals. In different blocks of the target present trials, one of the presented faces had either an angry or happy facial expression. The positions of the faces were randomly determined. 

Procedure

Each trial began with a fixation cross presented alone for 1,500 milliseconds, followed by the array of faces that was shown for a maximum of 30 seconds or until the response (see Figure 2). False responses were signaled by a short 100-millisecond tone at 1,000 hertz immediately following the key press. The experiment consisted of four blocks. Before each block, a short message informed participants about the to-be-searched-for emotion category. The 120 experimental trials within a block were preceded by two warm-up trials that were randomly selected from the 60 conditions (3 set sizes × 2 target presence × 10 target individuals) within a block. 

Figure 2

 

Schematic overview of the stimulus sequence. Either two, four, or nine faces were presented on a black background. Each stimulus display contained neutral faces and no emotional target face (target absent trials) or one or several neutral faces plus one emotional target face (target present trials).

Experiment 1: Results

No trials with anticipatory responses (RTs < 200 milliseconds) were found. False responses (4.0%) and RTs > 5,000 milliseconds (0.4%) were excluded from the RT analysis. For statistical analysis, we first computed the slopes of the RT × set size function separately for each individual participant and for each condition of interest. Search efficiency was then assessed by computing a 2 (target presence: present vs. absent) × 2 (target emotion: angry vs. happy) × 2 (target face mouth opening: open vs. closed) × 2 (target gender: male vs. female) within-subjects ANOVA over the individual slopes (Figure 3). 

Figure 3

 

Mean RTs from Experiment 1. Lines are regression slopes. Equations display linear functions. Error bars are standard errors of the mean.

The ANOVA revealed four significant main effects. First, the main effect of target presence was significant, F(1, 11) = 39.97, p < 0.001, η_p² = 0.78, reflecting that search slopes were much steeper in target absent trials (170 ms/item) than in target present trials (70 ms/item). This is a signature of an inefficient, serial search in which individual distractors are scanned before the target is found. 

Second, there was a main effect of target emotion, F(1, 11) = 9.90, p = 0.009, η_p² = 0.47, reflecting more efficient search for happy (108 ms/item) than for angry faces (133 ms/item). Thus, contrary to the original threat-capture hypothesis, but consistent with studies using either the NimStim or the KDEF stimulus set, there was a happy face advantage rather than an angry face advantage. 

Third, and central to the present endeavor, there was a main effect of mouth opening, F(1, 11) = 45.23, p < 0.001, η_p² = 0.80. Search for targets with an open mouth was considerably more efficient (90 ms/item) than for closed-mouth targets (151 ms/item). It might be noted that this effect was as strong as the effect of target presence, which is usually the strongest effect in visual search experiments. 

Fourth, there was a small but significant bias for more efficient search with female emotional target faces (116 ms/item) than with male target faces (124 ms/item), F(1, 11) = 11.62, p = 0.006, η_p² = 0.51. 

The reported effects appeared to be almost completely additive: Only the interaction between mouth opening and target presence approached significance, F(1, 11) = 4.19, p = 0.065, η_p² = 0.28, with larger differences between target absent and target present trials (110 ms/item) for the closed-mouth faces than for the open-mouth faces (86 ms/item; for all others F < 3.02, p > 0.111, η_p² < 0.22). 

A corresponding analysis of the error slopes (Figure 4) revealed only two main effects. The main effect for target presence, F(1, 11) = 10.77, p = 0.007, η_p² = 0.50, revealed a shallower slope for errors in target absent than in target present trials, which is a typical finding (because participants tend to overlook a target, but rarely report a target where no target is actually presented). The main effect for mouth opening, F(1, 11) = 5.56, p = 0.038, η_p² = 0.34, reflected a shallower error slope in search for open-mouth targets. Finally, the mouth opening × target presence interaction, F(1, 11) = 5.75, p = 0.035, η_p² = 0.34, shows that the absent/present difference was stronger with closed than with open-mouth faces. Importantly, the results from the error analysis do not complicate interpretation of the RT data because there was no indication for a speed-accuracy trade-off for the variables of mouth opening and facial emotion. 

Figure 4

 

Errors and error slopes for Experiment 1.

We suspected that visible teeth can render any emotional face salient and relatively easy to find in a crowd, and that a search advantage may result when one target face is characterized by visible teeth, while the other is not. To assess whether the visibility of teeth can artificially produce a search advantage, we created two confounded designs, in which the visibility of teeth was confounded with either the angry or the friendly facial expression. The slopes in these confounded designs were then analyzed with a 2 (target presence: present vs. absent) × 2 (target emotion: angry vs. happy) × 2 (target gender: male vs. female) within-subjects ANOVA (see also Figure 5). 

Figure 5

 

Mean target present slopes (in ms/item) depending on confound condition. The left pair of slopes corresponds to the condition in which angry faces showed teeth while happy faces did not; the right pair of slopes corresponds to the condition where happy faces show teeth while angry faces do not.

In the first confounded design we compared search efficiency between happy face targets with visible teeth to angry face targets with a closed mouth. Importantly, the ANOVA revealed a strong main effect of emotion, with more efficient search for friendly (open mouth) faces than angry (closed mouth) faces, F(1, 11) = 50.26, p < 0.001, η_p² = 0.82 (main effect of target presence: F(1, 11) = 46.56, p < 0.001, η_p² = 0.81; all other effects and interactions F < 4.29, p > 0.05). 

In the second confounded design we compared detection of angry face targets with visible teeth and of happy face targets with a closed mouth. Again, the ANOVA revealed a significant main effect of emotion, which was now due to more efficient search for angry (open mouth) faces than friendly (closed mouth) faces, F(1, 11) = 9.03, p = 0.012, η_p² = 0.45 (main effect of target presence: F(1, 11) = 31.19, p < 0.001, η_p² = 0.74; all other main effects and interactions F < 2.17, p > 0.17). 

To relate our findings to previous studies that analyzed only target present trial performance, we repeated the analysis while excluding the target absent trials performance. The results showed reliable emotion effects when the analysis was based on target present trials only F > 9.01, p < 0.012. The results of the confounded designs are also depicted graphically in Figure 5. 

Is search determined by the same distractor rejection process in target present and target absent trials?

The previous analysis suggests that effects in the target present trials are mirrored in the target absent trials, in which the same neutral face distractors were presented in all conditions. To assess this more formally, we conducted an ANOVA on only the target absent trials. The 2 (target emotion: angry vs. happy) × 2 (target face mouth opening: open vs. closed) × 2 (target gender: male vs. female) within-subjects ANOVA revealed a main effect of mouth opening, F(1, 11) = 57.58, p < 0.001, η_p² = 0.84, and a marginally significant main effect of target emotion, F(1, 11) = 4.15, p = 0.067, η_p² = 0.27 (others, F < 2.97, p > 0.11, η_p² < 0.21). These effects are a first and unambiguous indication that scanning and rejecting the same distractors proceeds differently depending on the searched-for target. 

To assess whether search performance was more due to the guidance of attention to the target location or to the efficiency of distractor rejection, we examined the correlation between search efficiency in target present and target absent trials. The rationale was as follows. In target present trials, search efficiency is in principle a function of both guidance and distractor rejection, with the relative contribution guidance and distractor rejection being unknown. In target absent trials, in which no target is present, search efficiency cannot possibly be a function of guidance and is thus exclusively a function of distractor rejection. 

Hence, the contribution of guidance versus distractor rejection can be inferred on the basis of the correlation between search efficiency on target present and target absent trials. A correlation close to r = 1.0 would indicate that the same process drives search efficiency in target present and target absent trials, which would then be distractor rejection. A correlation close to r = 0.0 would indicate that different causes drive search efficiency in target present and target absent trials, suggesting that search performance is largely determined by target guidance on target present trials and distractor rejection on target absent trials. 

We first examined the correlation between target present and target absent trials across the four types of stimuli of interest (i.e., happy and angry faces, with open or closed mouths, respectively). The correlation between the 48 (12 participants × 4 expressions) pairs (target absent and target present) of slopes was r = 0.84, p < 0.001. Thus, there seems to be a substantial overlap in the search efficiency through crowds with and without a target, indicating that searching and rejecting the distractors is an important determinant of performance on both the target absent and the target present trials. In fact, 71% of the variance in target present trials can be predicted on the basis of the variance in target absent trials, suggesting that search performance in both target present and target absent trials is determined mostly by distractor rejection. 

It might be objected that this correlation could be inflated by individual differences; possibly the high correlations reflects mainly that for fast participants search rate is high in all conditions, while for slow participants search rate is low in all conditions. To exclude this possibility, correlations were computed separately for each participant. The average correlation, which was computed by first transforming the 12 correlation coefficients to z-values, and transforming the mean of the z-values back to r, was r = 0.89. Thus, the high correlation between target present and target absent search efficiency was not simply due to inter-individual differences in overall search efficiency. 

To explore whether this association can be found within the categories of happy and angry, open-mouth and closed-mouth faces as well, correlations between target present and absent trials were computed separately for each category. For each of these computations, each participant provided one pair of slopes. Thus, this analysis is based on inter-individual differences and rests on the assumption that guidance and distractor rejection probe different “abilities,” both of which may vary independently between participants. Correlations were high for angry closed-mouth faces, r = 0.81, p < 0.001, and happy closed-mouth faces, r = 0.87, p < 0.001. For open-mouth faces, correlations were substantially lower: r = 0.40, p = 0.192, for angry open-mouth faces, and r = 0.46, p = 0.129, for happy open-mouth faces. This indicates that the same “ability” is probed in target present and absent trials for closed-mouth targets, while different “abilities” are probed in target present and absent trials for open-mouth faces, possibly because search in target present trials can profit from guidance by the open-mouth target. We put “abilities” in quotation marks to indicate that in addition to trait differences, strategic factors might play a role. 

Evidence for attentional guidance by the target face?

If search is exhaustive and proceeds in a perfectly serial manner, the slope in a target absent condition should be twice as steep as in a target present condition (Treisman & Gormican, 1988; Treisman & Souther, 1985; Wolfe, 1998; see Becker, Anderson, Mortensen, Neufeld, & Neel, 2011, for a discussion with relevance for face searches). In a target absent trial, all search items have to be selected to ascertain that the target is indeed absent. In contrast, in target present trials, the target is found on average after inspecting half of the items in the search display. We can test whether or not search was serial and exhaustive by taking serial search as the null hypothesis: Deviations from this null hypothesis would indicate that attention could be guided to some extent to the target, which enabled observers to find the target somewhat earlier than with purely serial search (Wolfe, 1998). 

This reasoning implies that with purely serial search, the slope for target absent (STA) trials should be equal the double slope for target present trials (STP): STA = 2 × STP. In contrast STA > 2 × STP would be predicted by serial search assisted by attentional guidance. We conducted four t-tests (one-tailed) comparing STA to 2 × STP (data were collapsed over the gender variable). 

Results indicated deviations from purely serial search for happy open-mouth faces, t(11) = 2.22, p = 0.024, in which search in target present trials was 48 ms/item faster than expected for purely serial self-terminating search (slope ratio: 2.8); for angry open faces, t(11) = 1.90, p = 0.042, with an advantage of 34 ms/item (slope ratio: 2.9); for happy closed-mouth faces, t(11) = 2.55, p = 0.014, with an advantage of 32 ms/item (slope ratio: 2.3); but not for angry closed-mouth faces, were performance did not deviate from the serial search assumption, t < 1 (STA-2×STP = −4 ms) (slope ratio: 2.0). 

Experiment 1: Discussion

Results indicate that mouth opening and concomitant teeth visibility play an important role in the detection of photorealistic emotional faces in crowds of neutral faces, and that the presence of visible teeth may even explain inconsistencies in results between prior studies. In fact, the display-of-teeth effect on search efficiency happened to be as strong as the effect of target presence on search efficiency, and it was considerably larger than the effect of facial emotion in the present study. Moreover, additional analyses demonstrated that if teeth visibility is confounded with facial emotion, the facial emotion category with visible teeth is found more efficiently: When happy faces show teeth while angry faces do not, happy faces are found more efficiently than angry faces; correspondingly, when angry faces show teeth while happy faces do not, angry faces are searched for more efficiently. To summarize, the present results show that teeth visibility is not only a potent influence in visual search for faces, but may also account for the inconsistencies in the experimental literature where some studies reported a search advantage for angry faces, and others reported an advantage for friendly faces. 

Our analyses of the processes underlying visual search efficiency with emotional faces also yielded a number of noteworthy results. Analyses relating performance on target absent (neutral faces only) trials and target present (one target plus neutral faces) trials revealed very high correlations. For example, the scanning of the same neutral crowd is considerably more efficient when a happy open-mouth face is searched for than when an angry closed-mouth face is searched for. Clearly, this effect cannot be attributed to attentional guidance by the emotional target face, because actually no emotional target face is present in the target absent display. Instead, search efficiency in target absent (neutral faces only) trials should be completely determined by distractor rejection, and in particular the serial comparison of the emotionally neutral crowd faces with the target template, which is a generalized representation of the emotional target(s). In target absent trials, these comparisons probably continue until sufficient certainty is obtained that no emotional target is present, either because no target has been found after scanning all distractors, or because a certain amount of time has elapsed since the beginning of the trial (the time-out criterion). 

The high correlation between target present and absent performances has two obvious implications. First, because no emotional target could possibly guide attention in target absent trials, the high correlation strongly suggests that the duration of distractor rejection processes determined performance not only in target absent trials, but in target present trials as well. This means that large parts of the effect of emotional category is not due to the attraction of attention by the emotional target, but to the time needed to process the distractor faces in the neutral face crowd. 

Second, because the distractors were the same across all conditions, the durations of the distractor rejection processes are apparently flexible and vary with the searched-for target type: The same neutral distractors are scrutinized longer when the emotional target was a closed-mouth angry face than when it was an open-mouth happy face. This implies that aspects of the distractor rejection process are adaptively set, probably according to the difficulty of the task. If discrimination between the emotional target and the neutral distractor is easy, distractor rejection proceeds quickly. In contrast, if the discrimination is hard, distractor rejection proceeds more slowly. Discrimination difficulty is probably determined by the similarity between the distractor and the target (template), as is implicated by prior research and theorizing in the context of visual search (Duncan & Humphreys, 1999; Farmer & Taylor, 1980). 

Interestingly, the correlation between target present and absent trials over participants was lower for the open-mouth faces than for the closed-mouth faces, and was significant only for the latter but not for the former. This result indicates that with open-mouth faces, distractor rejection is not the only determining factor, but that search can additionally profit from guidance provided by the target. This was also reflected in the significant deviations from serial self-terminating search found for both angry and happy open-mouthed targets and for the happy closed-mouth target. These results indicate that emotional faces are capable of guiding attention, in particular when the emotion is expressed in an open-mouth display (a point that will be discussed in more detail in the General Discussion). 

Emotional category also had an effect on search performance, with happy faces being searched for more efficiently. This is consistent with most of the more recent research (e.g., Becker, Anderson, Mortensen, Neufeld, & Neel, 2011; Juth, Lundqvist, Karlsson, & Öhman, 2005; Calvo & Nummenmaa, 2008) and further questions the validity of the original threat capture hypothesis. Finally, results revealed a small effect of target gender, with search being more efficient with female than with male faces. This effect was not anticipated, and gender was only introduced as a methods variable, because human faces are necessarily either male or female. This result may reflect that emotions are, on average, more strongly expressed on female faces than on male faces (see also Table 1). 

Experiment 2

Experiment 1 tested the effect of mouth opening and visible teeth in intact faces, which presumably have considerable ecological validity for research on facial expressions of emotion. A frequent drawback of such stimulus material is that it introduces uncertainty about the effective differences between target categories. In the present research, this uncertainty pertains to the question whether mouth opening (and visible teeth) is actually the only difference between the categories of open-mouth versus closed-mouth expressions. To ensure that the results of Experiment 1 were indeed due to differences in the teeth displays and not other parts of the face, Experiment 2 tested whether the same results can be obtained presenting only the mouth portions of the faces, while omitting other facial areas as much as possible (see also Calvo & Nummenmaa, 2008). 

Experiment 2: Method

Participants

Participants were three men and five women with a mean age of 27 years (range 18–44 years). 

Design and apparatus

These were the same as in Experiment 1. 

Stimuli

From the stimulus material used in Experiment 1, rectangular parts were cut out that showed predominantly the mouth region (Figure 6). The upper edge was approximately located at the level of the nose tip, and the lower edge was approximately located below the level of the chin. Because the resulting cut-out was more than twice as wide than high, the pictures were also cropped on the left and right sides to obtain a format not too dissimilar to Experiment 1. The cropped pictures always showed the complete mouth, located approximately in the center of the picture. After cropping, pictures were reduced to 3.24° × 2.86°, such that the resulting pictures were similar to Experiment 1 in size and pixilation. As in Experiment 1, the same 256-color map was used for all pictures. Columns between pictures were 0.48° wide. The stimuli were displayed in groups of two, four, or nine, as before. 

Figure 6

 

Examples of stimuli presented in the experiment.

Procedure

The procedure was the same as before. 

Experiment 2: Results

No trials with anticipatory responses (RT < 200 milliseconds) were found. False responses (6.0%) and RTs > 5,000 milliseconds (0.0%) were excluded from the RT analysis. Figure 7 shows the mean RTs along with the regression slopes. 

Figure 7

 

Mean RTs from Experiment 2. Lines are regression slopes. Equations display linear functions.

The ANOVA revealed four significant main effects. The main effect of target presence was significant, F(1, 7) = 62.90, p < 0.001, η_p² = 0.90, reflecting that search slopes were about twice as high in target absent trials (84 ms/item) than in target present trials (36 ms/item). There was a main effect for target emotion, F(1, 7) = 35.76, p = 0.001, η_p² = 0.84, reflecting more efficient search for happy (42 ms/item) than for angry stimuli (78 ms/item). Third, there was a main effect of mouth opening, F(1, 7) = 102.24, p < 0.001, η_p² = 0.94. Search for targets with an open mouth was more efficient (14 ms/item) than for closed-mouth targets (106 ms/item). The main effect for gender was not significant, F < 1. 

There were three significant two-way interactions which entailed all combinations of emotion, mouth opening and target presence, F > 22.68, p < 0.002, and a significant three-way interaction between all variables, F(1, 7) = 12.13, p = 0.010, η_p² = 0.63. The emotion × mouth opening interaction revealed that there was almost no emotion effect in the open-mouth condition (angry: 16 ms/item, happy: 11 ms/item), but a considerable emotion effect in the closed-mouth condition (angry: 138 ms/item, happy: 73 ms/item). The emotion × target presence interaction revealed that the emotion effect was larger in target absent trials (angry: 110 ms/item, happy: 57 ms/item) than in target present trials (angry: 45 ms/item, happy: 27 ms/item). The mouth opening × target presence interaction reflects that there was only a small target presence effect with open-mouth faces (present 10 ms/item, absent 17 ms/item), but a large effect with closed-mouth faces (present: 61 ms/item, absent 150 ms/item). Finally, the three-way interaction indicates that the two-way interactions are all mostly due to the very inefficient search in the angry closed-mouth condition (see Figure 7). 

A corresponding analysis of the error slopes (Figure 8) revealed only a main effect for target presence, F(1, 7) = 9.66, p = 0.017, and mouth opening, F(1, 7) = 18.68, p = 0.003, as well as the two-way interaction between these variables, F(1, 7) = 18.16, p = 0.004, all mirroring the RT effects (all other F < 2.50, p > 0.16). The results from the error analysis do not complicate interpretation of the RT data, as there was no indication for a speed-accuracy trade-off. 

Figure 8

 

Errors and error slopes for Experiment 2. Is search determined by the same distractor rejection process in target present and absent trials?

As in Experiment 1, we conducted an ANOVA on the target absent trials, which revealed a main effect for mouth opening, F(1, 7) = 83.81, p < 0.001, and for facial emotion, F(1, 7) = 36.14, p = 0.001, as well as the mouth opening × facial emotion interaction, F(1, 7) = 20.75, p = 0.003. The interaction was due to the fact that the mouth opening effect was much smaller with open mouths than with closed mouths. 

The correlation between search efficiency in target present and target absent trials was again high, whether r is computed over the 32 (8 participants × 4 expressions) pairs of slopes (r = 0.92), or as the average of the within-participants correlations (r = 0.99). 

Experiment 2: Discussion

Experiment 2 supports the conclusions drawn from Experiment 1 regarding the role of mouth opening and visible teeth in visual search. Mouth opening had a very strong effect on search efficiency: with an open mouth, search was almost efficient (zero slope), whereas search was inefficient with closed mouths, in particular for the angry faces. 

While these general results patterns are similar to Experiment 1, search was much more efficient in Experiment 2 for the open-mouth stimuli. A possible explanation is that the isolation of the most salient features in the mouth region (in Experiment 2) facilitated processing, while the embedded presentation of these features in otherwise relatively uninformative portrait photos (in Experiment 1) hampered detection. 

A second deviation from Experiment 1 was the special status for the angry closed-mouth faces, which rendered by far the least efficient search performance and most frequently missed targets, and registered in statistical interactions between emotion category, mouth opening and target presence. Apparently, closed-mouth angry targets were especially difficult to detect on the sole basis of differences in mouth shape. In line with this hypothesis, the zygomaticus major activity (the closed-mouth smile), which is characteristic of happiness, may be a sufficient indicator of target presence with happy stimuli. In contrast, the orbicularis oris activity entailed in the compressed lip display in the closed-mouth anger stimulus may be perceptually more similar to the relaxed mouth in the neutral stimulus. This relatively minor role of the mouth in the anger display is in line with the widely held belief that the iconographic region for anger is not the mouth, but the brows (i.e., corrugator supercilii activity) region (e.g., Fox & Damjanovic, 2006). 

General discussion

The two major aims of the present experiments have been to examine the possibility that visible teeth displays play an important role in the visual search for photorealistic emotional faces in crowds of neutral faces, and to test whether the presence of visible teeth may even explain inconsistencies in results between prior studies. To examine these questions, stimulus materials were used where facial emotions of happiness and anger were shown in a closed-mouth display and in an open-mouth display by the same individuals. The results are clear: With complete emotional faces (Experiment 1) the variation of mouth opening rendered the strongest effects in the analyses, and the effect was even magnified when only the mouth region of the faces were presented (Experiment 2). Importantly, the analyses of the confounded designs provides a proof of concept that mouth opening can be the variable that decides which of the presented faces is searched for most efficiently: When open-mouth happy faces were compared with closed-mouth angry faces, we obtained a happy face advantage, whereas we found an angry face advantage when the closed-mouth happy faces were compared with open-mouth angry faces. 

We have characterized the main experimental variable in a rather phenomenological way as the presence or absence of an open mouth with visible teeth. It is difficult to pinpoint exactly the perceptual differences between closed and open-mouth faces in terms of lower order features, which is a common problem when natural stimuli are presented in visual search (e.g., Zelinsky, 2008). Two obvious candidates are the color difference caused by teeth visibility, and the shape difference caused by the mouth opening. Because it is known that color is the most potent feature to guide attention and that observers rely on color information much more than on shape information (Williams & Reingold, 2001), we conclude that color was most probably the most effective feature dimension in the present study. 

The present experiments also revealed a happy face advantage, although this effect was much smaller than the effect of visible teeth. The finding of a happy face advantage is in line with recent studies using either the KDEF or the NimStim stimulus set (e.g., Calvo & Nummenmaa, 2008; Juth, Lundqvist, Karlsson, & Öhman, 2005; Williams, Moss, Bradshaw, & Mattingley, 2005). It is unclear, however, to what extent previous research was contaminated by confounding the facial emotion category with teeth visibility. Thus, the present experiments are the first to demonstrate the happy face advantage when facial category is manipulated orthogonally to mouth opening and teeth visibility. 

One possible explanation of the happy face advantage seems to be that the smile reveals even more teeth than the snarl. Visual inspection of the presented stimuli, however, does not yield clear evidence for this hypothesis; what seems to be clear is that the smile changes other features of the face more than the snarl. In particular, with respect to the mouth-only stimuli in Experiment 2, the curvature of the lips in happy faces seems to deviate more from mouth-line in neutral distractors than the differences in curvature between angry and neutral faces (see Figure 6, for an example). With respect to the whole face, the open-mouthed smile seems to affect the cheeks and the eyes more than the angry snarl. In the smiling face, the cheeks are “raised,” which actually entails the compression and concordant rounding of tissue. This in turn leads to a different pattern of light reflection, occasionally producing rather light spots on the cheeks. Moreover, with the presently used stimulus set, the displacement of the brows seems much more pronounced in the happy than the angry face, leading to the full disclosure of the eye lids, again producing bright spots in some of the images. 

What are the mechanisms underlying visual search for an emotional target?

Previous research has sensibly interpreted visual search for emotional targets in a straightforward manner: (a) Emotional targets are presented among crowds of neutral distractors, giving them the opportunity to attract or guide attention; (b) evidence for attentional guidance is revealed by a detection advantage in the form of a search asymmetry, in which one emotional target is found more efficiently than an alternative target; (c) evidence for attentional guidance is assumed to be strongest when the distractors are all the same in all conditions, such that differences in target present trials cannot be explained by differences in distractor rejection. The present results suggest that this reasonable story might be wrong. 

The present analyses probed into the mechanisms of visual search for an emotional face in a crowd of emotionally neutral faces by applying a different approach, which focused not only on the performance in trials in which the target is presented, but also on performance in target absent trials. Importantly, most of the variance of performance in target present trials is explained by distractor rejection, and not by attentional guidance. This is evidenced first by the fact that search was generally inefficient with the complete faces, with slopes > 40 ms/item in target present trials. None of the emotional target face categories was found at a single glance, all required a serial scanning of the neutral crowds. This is prime evidence that distractors were inspected in target present trials. Secondly, differences between the four emotional target categories were also reflected in the target absent slopes, in which all faces displayed a neutral expression. In these trials guidance could not have played a role because no emotional target was presented. Thirdly, the important role of distractor rejection is also reflected in the high correlations between target present and target absent trials, implying that much of the variance in target present trials is explained by distractor rejection processes, leaving little variance to be accounted for by attentional guidance. 

The fact that the emotional target category had a strong effect even when the display contained only emotionally neutral distractors implies that the most important variable that determined search efficiency in the present study was not the physically presented target, but rather the mental representation of the target category—the target template: Only the representation of the target category is constant over target present and target absent trials within a block, but varies between blocks of trials; that is, only the target template co-varies with the search performances. In the present study, the most important variable determining the duration of the process of distractor rejection may have been the comparison of individual distractors or small groups of distractors with the template for the emotional target category. The duration of this process is probably determined by the similarity between the template for the emotional category and the inspected distractor (Duncan & Humphreys, 1989).If discrimination between the emotional target and the neutral distractor is easy, each distractor (or group of distractors) is only briefly inspected and quickly rejected when it does not match the target template immediately. That is, distractor rejection ends after an only superficial analysis. In contrast, if target distractor discrimination is difficult, distractors are inspected with more scrutiny, leading to longer inspection times. 

While the effects of distractor rejection processes were very strong, we also found evidence for guidance by the emotional target: The slope ratios indicated variations from serial self-terminating search for both faces with visible teeth and the closed-mouth happy face target. Put another way: times to find the target were somewhat shorter than expected on the basis of purely serial self-terminating search. The pattern of results from the slope ratios largely matches the results from the inter-individual correlations between target absent and target present performance, which were lower for the open moth faces than for the closed-mouth faces. Together, these two pieces of evidence suggest that the open-mouth faces guided attention, thereby shortcutting the serial scan of the neutral distractor faces. 

A caveat to the interpretation of the present results is that attentional guidance from a first glance at the display seems rather unlikely, because all slopes were quite steep (>40 ms/item). Apparently, attentional guidance took place not before, but during the scanning of small groups of crowd stimuli. A possible scenario would be, for example, that after a couple of shifts, the attentional focus landed in the vicinity of the emotional target, and weak salience signals then guided attention to the respective face. 

Emotional versus perceptual factors

Emotion researchers and theorists have been interested in visual search for emotional faces and evidence for search asymmetries mainly because of the possibility that results revealed attentional guidance by affective features. According to Öhman and others (e.g., Eastwood, Smilek & Merikle, 2003; Öhman, Lundqvist, & Esteves, 2001), affective content is extracted preattentively and fed back to the perceptual system where it guides attention towards the affective stimulus. In its original form, it was assumed that the affective content also had to include threat, which is, however, untenable in view of the prevalence of experiments (including the present ones) revealing a happy face advantage. 

The emotion-guides-attention explanation faces difficulties in explaining the current results. First, it cannot explain the very strong effect of the emotional target category in target absent trials, in which no affective stimulus was present to guide attention. Second, it cannot readily explain the very strong effect of teeth visibility, and in particular, that this effect was larger than the effect of emotion category. To accommodate for the present results of teeth visibility, the original emotional guidance account has to be changed to the effect that the attention-guiding content is not categorical in nature, but is a more general dimension, such as emotion intensity. However, the result that the effect of teeth was even stronger in Experiment 2 than in Experiment 1 speaks against this explanation. Experiment 1 presented color photos of complete emotional faces, which on any account should be better emotional stimuli than the cut-outs presented in Experiment 2, in which the emotional stimulus was strongly impoverished due to the elimination of large parts of the facial context. Third, as the open-mouth stimuli and the closed-mouth stimuli differ perceptually, the affective feature is confounded with systematic perceptual differences between the faces. Thus, it becomes difficult to distinguish such a mixed perceptual-emotional explanation from a perceptual-guidance account which states that attention is guided by perceptual features only. Consequently, it would appear that the affective-guidance account is less parsimonious (sensu Occam's razor) than a competing perceptual account: Whereas it is possible to dispose of the affective-guidance account, the perceptual-guidance account is necessary to explain parts of the current data. 

Conclusion

What is the emotional-face-in-a-crowd effect? In its most influential versions, it is the fast finding of an angry face in a crowd of nonangry persons, and it reflects that information about the presence and location of a threat is extracted early (i.e., preattentively) in visual processing and fed back to the visual system to the effect that attention is directed to the stimulus, or position, where the threat information originates (e.g., Hansen & Hansen, 1988; Öhman, Lundqvist & Esteves, 2001). Present evidence suggests, however, that much of the effect with photorealistic emotional faces is due to visible teeth; that when the tooth effect is controlled for, not the angry face, but rather the nonangry emotional face is found more efficiently; that large amounts of the effect are not due to the emotional target, but rather to the identity of the mental representation of the target (the target template); and that large amounts of the effect is not due to attentional guidance by the target, but rather to processes related with the testing and rejecting of the nonemotional distractors. 

Acknowledgments

We acknowledge the support of the Deutsche Forschungsgemeinschaft and the Open Access Publication Funds of Bielefeld University for the publication fee, and of the Australian Research Council (DP110100460 and DP110100588). 

Commercial relationships: none. 

Corresponding author: Gernot Horstmann. 

Email: [e-mail]. 

Address: Center for Interdisciplinary Research and Department of Psychology, Bielefeld University, Bielefeld, Germany.