Emotion categorization does not depend on explicit face categorization

Abstract

Abstract

Abstract:

Abstract  Face perception and emotion recognition have been extensively studied in the past decade; however, the relation between them is still poorly understood. A traditional view is that successful emotional categorization requires categorization of the stimulus as a ‘face', at least at the basic level. Here we tested whether emotional information could still be recognized accurately without explicit categorization of a stimulus as a face. For this purpose we created a stimulus set in which facial stimuli expressing a range of happy-to-fear emotions were morphed into another object category (shoe). Interestingly, participants categorized emotions with great accuracy in stimuli that contained so little face information that they were explicitly categorized as shoes. Hence, our results show that accurate emotion categorization can take place in stimuli that contain surprisingly little face information. This finding raises interesting questions about the extent to which processes leading to emotion recognition and categorical face perception might be separable.

Introduction

Object recognition has been classically viewed to occur in a series of sequential processing stages: detection, categorization, and identification. At the first stage (object detection), the overall shape of the object is segregated from its background (Driver & Baylis, 1996). Then, the detected object is categorized in supra- and subordinate levels (Nakayama, He, & Shimojo, 1995). Eventually, the object is identified at the final level of object recognition. However, this sequential view on object processing has been challenged in several studies. For example, Peterson and Gibson (1994) showed that object category can influence the performance at detection level in stimulus conditions that prevented perceptual categorization, thus arguing against the unidirectional view that detection precedes further categorical processing. Moreover, Grill-Spector and Kanwisher (2005) proposed that detection and basic-level categorization take place almost at the same stage. This is a challenge to the sequential processing idea as well, because here it is assumed that an object, once detected, may be categorized without additional processing (Bar, 2004; Bar et al., 2001). 

A similar debate is present in the literature on face perception. Traditional models of face recognition assume a sequential series of processing steps, including figure-ground segregation and categorization processes yielding a structural description that once achieved is the basis of two further, separate routes of face processing: (a) person identification and (b) recognition of emotional expression (Bruce & Young, 1986; Morris, de Gelder, Weiskrantz, & Dolan, 2001). One debate in the face processing literature is about the extent to which the processes involved in face and emotion processing interact (Calder & Young, 2005). Another more fundamental debate is whether the two processes of face (body) identification and emotion recognition make use of the same initial structural representation of the face (body). There is rising evidence that emotion categorization uses a coarse face representation that is not sufficient for, or altogether different from, face representations involved in face categorization and identification (de Gelder & Rouw, 2001; de Gelder, Snyder, Greve, Gerard, & Hadjikhani, 2004; Garrido, Barnes, Sahani, & Dolan, 2012; Gschwind, Pourtois, Schwartz, Van De Ville, & Vuilleumier, 2012; Johnson, 2005). If so, it should be possible to categorize correctly the emotional expression of a stimulus that contains a face of which the representation is too coarse to be categorized as a face. To evaluate this hypothesis, we tested whether, in stimuli containing increasingly degraded structural face information, accurate categorization of the facially expressed emotion was always associated with an explicit categorization of the stimulus as a face. To that aim, we devised a stimulus set based on a two-way morph procedure involving two object categories and two emotional facial expressions. Specifically, a series of face images from the same identity was morphed from one to another emotional expression, and each image of the series was then morphed to the same exemplar of another object category (shoes). Using this stimulus set, we tested whether emotion categorization is possible at morph levels in which structural face information is degraded to a level that leads to explicit categorization of the stimuli as a shoe rather than a face. Three possible outcomes were envisaged for this experiment: First, it is possible that a high level of face-related information is required to permit accurate emotion categorization. This means that accurate emotion categorization would be possible only for face-object morphs that are almost always explicitly categorized as faces. Second, it is possible that the physical information required to categorize the intermediate morphs as faces in most of the trials is sufficient to permit emotion categorization. Third, emotion categorization may be possible at face-object morph levels that are explicitly categorized as shoes and that give insufficient information for the stimuli to be categorized as a face. 

General methods

Stimuli

Original face images were color images of two males (KDEF identities: AM10 and AM14), each displaying two different emotions of happy and fearful (two identities × two emotions) selected from a subset of Karolinska Directed Emotional Faces (KDEF) (Lundqvist, Flykt, & Öhman, 1998), previously validated to be correctly (> 90%) categorized as happy and fearful in a pilot study on 40 participants. The shoe stimulus was selected from our own shoe database (de Gelder, Bachoud-Levi, & Degos, 1998). 

The stimuli were preprocessed as follows: first, all the original images were cropped and resized to fit into an oval-like template between face and shoe outline by Adobe Photoshop CS4 (Adobe Systems Incorporated, San Jose, CA; http://adobe.com). Then, all the average pixel values for each image were shifted to 128 in Matlab 2010a (Mathworks, Natick, MA; http://www.mathworks.com). Afterwards, all the pictures were overlaid on top of a gray noisy background with the same average luminance of 128. 

The preprocessed images were then submitted to the morphing software, FantaMorph 4.2.6 (Abrosoft Co., Beijing, China; http://abrosoft.com), to generate the morphs between two emotional faces and the shoe. To that aim, we first morphed the two emotional faces from the same identity to each other in six equal morphing steps, resulting in seven stimuli from very happy to very fearful faces. Then, each of the seven emotional face stimuli was morphed to a single shoe stimulus in 12 morphing steps. 

We used two different subsets of the morphed stimuli in the two experiments: (a) The odd number face-shoe morph levels (1, 3, 5, 7, 9, 11, and 13) in Experiment 1 corresponded to face content percentiles of 100, 83, 66, 50, 33, 17, 0; (b) The second half of the face-shoe morph levels (7, 8, 9, 10, 11, 12, and 13) in Experiment 2 corresponded to face content percentiles of 50, 42, 32, 25, 17, 8, 0. Note that axis labels in Figures 2 and 3 are ordinal values, of which the % face content is defined by above-specified labels. 

Figure 1

 

Samples of morphed stimuli, with colored lines representing faces (KDEF identity: AM10) along the fear (red, morph 1, 100% fear) to happy (cyan, morph 7, 0% fear) emotion dimension, and gray level lines representing stimuli along the face (black, morph 1, 100% face) to shoe (light gray fine dashed line, morph 7, 0% face) object dimension. The dotted blue line represents a hypothetical perceptual boundary between the face and the shoe categories.

Figure 2

 

(A) Face-shoe categorization responses as function of face-shoe morph level averaged over all subjects; (b) Fear-happy categorization responses as a function of different levels of face-shoe morphing. Gray-level/line-style coding of plots for different face-shoe morph levels corresponds to gray-level coding as in Figure 1. The morph levels on x-axis represent the ordinal sequence of the morphs described in General methods section.

Figure 3

 

Emotion categorization response as a function of face-shoe morph levels for trials in which the stimulus was categorized as a shoe.

Each face-shoe morph level subset—except the shoe anchor-point—consisted of seven levels of happy-fear morphed stimuli. As a result, a total of 43 stimuli ranging in two dimensions of emotion (fearful face, happy face), and object category (face-shoe) were tested in each experiment (Figure 1). 

The visual stimuli were presented on a 17-inch Samsung LCD monitor with a preset refresh rate of 60 Hz (Samsung, Samsung Town, Seoul, South Korea), using a Pentium Core 2 Duo, 3.0 GHz computer using E-prime 2.0 (Psychology Software Tools Inc., Pittsburgh, PA; www.pstnet.com). The subjects were sitting at a distance of approximately 65 cm from the monitor in a dimly-lit room. 

Data analysis

In each task, the proportion response for each corresponding morph-level was calculated and all the task-relevant psychometric functions were fitted. Furthermore, statistical analysis on the thresholds was performed by SPSS (SPSS Inc., released 2009; PASW Statistics for Windows, Version 18.0, SPSS Inc., Chicago, IL; http://www.spss.com/). 

The psychometric functions were obtained by using local linear fitting (McCullagh & Nelder, 1989; Zychaluk & Foster, 2009). The advantage of this method is that it is model-free; it works best when the exact shape of the underlying psychometric function is uncertain. In this method, the only assumption about the shape of the psychometric function is its smoothness. By a Taylor expansion, any smooth function can be approximated locally by a linear function:  

 

where the parameters , depend on x and their values are estimated by maximizing local log-likelihood,  

 

 

The weights w(x, x_i) determine the influence of each point x_i on the estimate at a point x, so that the further the points x and x_i are apart, the smaller the value of w(x, x_i). 

To decide the optimal bandwidth for each condition, we first estimated the appropriate loss function, which measures the discrepancy between the true psychometric function and the estimate. Next, we used cross-validated deviance to find the optimal bandwidth. Then, 1,000 x-values were generated and their corresponding fit were estimated by locglmfit function in a model-free toolbox for Matlab (Zychaluk & Foster, 2009). Finally, the threshold and slope of the fit were estimated by 200 bootstrap iterations. 

Experiment 1: Emotion categorization versus face/object categorization

Methods

We recruited 10 students (eight female, M_age: 23, age range: 18–27) from Maastricht University through local advertisements for this study. The study was conducted according to local ethics committee. The subjects gave their informed consent to participate in the experiment for either one course credit or 7.5 Euro. All participants were right-handed, with normal or corrected-to-normal vision and naive to the purpose of the experiment. One subject was removed from the analysis because of very strong (> 90%) preference towards face response for all the stimuli including the shoe anchor-point. 

The participants performed two different tasks (face-shoe categorization and emotion categorization) one after the other in four experimental blocks (counterbalanced among the subjects). In the face-shoe categorization task, the participants reported whether they perceived the stimuli as a face or a shoe (face vs. object). In the emotion categorization task, the participants had to report the emotional content (happy vs. fear) of the presented stimulus even if not perceiving the stimulus as face. Left and right button presses corresponded respectively to face and shoe responses in the face-shoe categorization task, and to fear and happy responses in the emotion categorization task. 

One block consisted of eight repetitions of each trial in which one of the 43 images was shown for 66 ms and in which the participants performed the instructed task (8 × 43 trials for a single block). One trial proceeded as follows: the fixation cross was presented in the center of screen for 500 ms, followed by a stimulus for 66 ms. Then a blank gray screen was shown and the participants were supposed to perform the respective task of the block. Trials were presented in a pseudo-random order, such that in each subsequent group of 43 trials each stimulus was presented once. Within each group, stimulus order was random. 

Prior to each block, the instructions were presented on the screen and the subjects were trained for 2 minutes to learn the button associations and get familiar with the stimuli. When training was over, the instructions were displayed again and the experimental block started with the subjects' button press. 

Results and discussion

Participants performed an object categorization (face or shoe) and an emotion categorization task (happy or fear) for each stimulus in separate experimental blocks. We first averaged the response to each face-shoe morph for different emotion (happy-fear morph) levels in all subjects. Then, we fitted nonparametric psychometric curves to the percentage shoe responses as a function of face-shoe level and calculated the 50% threshold of each participant in the face-shoe categorization task. We used repeated-measures ANOVA to compare the differences in face-shoe categorization thresholds among different emotion levels, with emotion as the within-subject factor and participants as the between-subject factor. We found that emotion level did not influence object categorization thresholds, F (6, 48) = 0.39, p > 0.1; sphericity assumed. This conclusion based on analysis of thresholds is corroborated by the average psychometric curves seen in Figure 2A, showing complete overlap across different emotion levels. 

More importantly, we also found that emotion categorization was strikingly robust against degradation of face information (Figure 2B). The results show that for morph levels containing as little as 17% of face information (corresponding to face-shoe morph 6), emotion categorization remained possible. This is supported by the goodness of fit in the 17% face condition, which averaged over participants was 86% (SD = 8.5%). 

Note that there is an overall bias to categorize stimuli as shoes rather than faces (Figure 2A). This is likely due to the choice of a shoe stimulus with anchor points for the morphing procedure that matched well to a face. 

The results from this experiment suggest that successful emotion categorization may occur for ambiguous stimuli that are perceived as a shoe in most trials. However, the object categorization task and the emotional expression categorization tasks were performed separately in different blocks of trials. We therefore cannot exclude that although the stimuli containing only 17% facial information were categorized as shoes in the majority of trials; the correct emotion categorization performance may reflect a minority of trials in which the participants would classify the stimuli as faces if they had been asked (while performing emotion categorization at chance level for the rest of the trials). Alternatively, as we would like to argue, participants may use the 17% face information in the stimuli to categorize emotion correctly without explicitly categorizing the stimulus as a face and, instead, categorizing it as a shoe. 

Note that the results from the shoe anchor-point in the emotion categorization task showed a bias towards fearful emotion in all subjects (Figure 2B). This is likely related to the emotional valance of this specific category of objects (de Gelder, 2005; Niedenthal, 2007). 

Experiment 2: Dual response categorization task

The results from Experiment 1 indicate that emotion categorization can still take place when the observers do not categorize the visual stimulus as a face in most of the trials. The strength of this conclusion, however, was limited by the fact that the object and emotion categorization tasks were done in different experimental blocks. Here, we aimed to strengthen this conclusion by testing whether emotion categorization would remain possible even on those trials in which the weak face information present in the stimulus did not lead to explicit categorization as a face. To test this hypothesis, we used a dual task design, in which participants performed both the object categorization and the emotion categorization task within each single trial. Moreover, based on the observation in Figure 2 showing complete face categorization for ordinal morph levels 1–4 (i.e., 0% shoe responses), we shifted the range of object morph levels towards levels containing less face information (see General methods). This allowed us to collect a sufficient number of trials resulting in a shoe categorization response, required to test the efficiency of emotion categorization in stimuli not categorized as a face. 

Methods

Ten students (eight female, M_age: 23, age range: 18–27) from Maastricht University were recruited through local advertisements for this study. The study was conducted after approval by the local ethics committee. The subjects gave their informed consent to participate in the experiment for either one course credit or 7.5 Euro. All participants were right-handed, with normal or corrected-to-normal vision and naive to the purpose of the experiment. Six participants had already participated in Experiment 1 two weeks prior to Experiment 2. 

The design for this experiment was similar to Experiment 1 using another subset of morphed stimuli (see General methods); the only difference was that the participants were asked to perform both tasks one after each other immediately after each stimulus. After each stimulus presentation, the subject decided both the Emotional content (task E) and the Object category (Task O) of the stimulus. The trials were divided in two blocks, one for each of the two possible task orders (OE or EO). Each participant did one block for each order, but the sequence of task order was counterbalanced over participants (i.e., half of the participants did EO in the first block and EO in the second; the other half did the converse). Task order in a block was cued by a verbal instruction appearing on the screen prior to the start of the block. 

A single block consisted of eight trials for each of the 43 images (344 trials). Trials were presented in a pseudo-random order, such that in each subsequent group of 43 trials each stimulus was presented once. Within each group, stimulus order was random. One trial proceeded as follows: the fixation cross was presented in the center of the screen for 500 ms, followed by a stimulus for 66 ms. Then a blank gray screen was shown and the participants were supposed to perform both tasks one after the other according to the instructed order. Up and down arrow keys corresponded to the happy and fearful response in the emotion categorization task; left and right arrow keys corresponded to face and shoe response in the object categorization task, respectively. Participants were reminded on a trial-by-trial basis of the response requirements by showing the cue words happy and fear just above and below the fixation spot in the emotion categorization task. Similarly, they were reminded of the response requirements by cue words face and shoe to the left and right of the fixation spot in the object categorization task. Participants self-initiated the experiment with a button press. Prior to each block, participants were familiarized with the stimuli and button associations in the two tasks during a brief training period lasting 2 minutes. 

Results and discussion

To decide whether we could pool the data from all trials from all participants to test our main hypothesis, we first explored whether the order of tasks had any effect on the thresholds in the face-shoe categorization task. It is possible, for example, that on a given trial, having responded “shoe” (not face) in the first task might have affected the decision of the participant in the subsequent emotion task. We pooled the data for each task order over all participants, and compared the percentage happy responses as a function of the emotion morph level for the 8% and 17% face conditions. This resulted in two psychometric curves that were indistinguishable, as confirmed by ANOVA of % happy responses, using as factors emotion level and task order, F (1, 8) = 2.04, p > 0.1. Furthermore, there was a rigid psychometric function underlying the emotion categorization response of the subjects with no effect of task order (thresholds for EO-OE orders were 3.73 ± 0.18 and 3.93 ± 0.16, respectively). 

Hence, we proceeded to pool the data from all subjects across task orders and studied the trials in which the participants categorized the stimuli as shoe, pooling data from the 8% and 17% face conditions. The shoe anchor-point (0% face) was not included. When plotting % happy responses as a function of morph level in the combined 8% and 17% face conditions (Figure 3), we found a smooth function with a threshold of 3.8 ± 0.4. Furthermore, to verify the discriminating effect of underlying emotion, we compared the tails of the function with the 50% threshold. To that goal, we grouped the emotion morphs based on the perceived emotion categories (fear: 1, 2; happy: 6, 7) and applied one sample t-test. The results show that there are significant differences for both groups below (fear: t [127] = −12.1) and above (happy: t [119] = 9.7) the threshold. 

Thus, Figure 3 shows that for the trials in which participants categorized stimuli that contained only 8%–17% face information as shoes, there was nevertheless very robust categorization of emotions contained in the degraded and unperceived faces. These data are compelling because the successful emotion categorization was performed in trials in which participants categorized the stimuli as a shoe. Hence, the data from the dual task design in this experiment provide a confirmation and validation of results and conclusions from Experiment 1. 

General discussion

The present study shows that accurate emotion categorization can take place in the absence of explicit face-object categorization, and in the presence of very little residual face information. This raises interesting questions about the amount of face information necessary for emotion categorization, and about the way in which this residual face information is processed. In Experiment 1, we investigated emotion categorization performance for the stimuli in which the face-related information was not sufficient for categorization of the visual stimuli as face. The results showed that for intermediate face-to-shoe morphed stimuli that showed a strong average bias to be explicitly categorized as shoe, the participants could still reliably categorize the emotional content. In Experiment 2, we explored whether the high emotion categorization performance for the stimuli categorized on average as shoe in Experiment 1 was due to those trials in which the stimulus was categorized as a face. For this purpose we used a dual task with participants performing both emotion categorization and face-object categorization after each stimulus presentation. The results indicate that even in the trials in which the subjects were explicitly categorizing the stimuli as shoe, and in which the stimuli were selected to have very little physical face information, they could still reliably categorize the emotional content of the image. 

This result is important because it fits with several robust findings that have been taken as evidence that different, even specialized, processes underlie face and object recognition (Tanaka & Farah, 1993; Valentine, 1988; Yin, 1969). Some researchers have proposed that faces are special, possibly not sharing any or sharing very little processing routes with objects (Farah, Wilson, Drain, & Tanaka, 1998; Grill-Spector, Knouf, & Kanwisher, 2004). Furthermore, it has been previously demonstrated that subordinate categorization in the expertise domain of the viewer can be as accurate and as fast as the basic-level categorization (Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976). More specifically, Tanaka (2001) showed that both basic-level and subordinate level categorizations take place with similar timing and accuracy. The present study suggests that the subordinate categorization of emotions in faces can outperform the explicit basic level categorization between two object categories. 

Several factors may contribute to performance in categorization tasks. Priming is one such factor, and visual face processing mechanisms have indeed been shown to be easily primed by verbal instructions and previously presented stimuli. For example, Bentin, Sagiv, Mecklinger, Friederici, and von Cramon (2002) showed that face-specific brain activity can be evoked by nonface objects after they had been seen embedded in schematic faces. If we consider the stimuli with high face content as priming stimuli for stimuli with low face content, this may also contribute to the successful emotional categorization of stimuli with very low face content. This in itself, however, does not invalidate our conclusions, as our data do suggest that the ‘primed' face representations permit highly successful emotion categorization in stimuli that are explicitly categorized as shoes, and contain as little as 8% of the original face content. A related consideration is that the morphing span of the face-shoe stimuli may affect the category boundary, which is relevant in Experiment 2, where we worked with a small subset of the morphing range biased strongly towards stimuli with low face content. To counter a possible bias to categorize stimuli as shoes, we presented the whole range of stimuli in the practice phase. This may not have prevented fully a bias in participants to classify the stimuli as shoes, but such bias makes the finding of very accurate emotion categorization in these stimuli all the more compelling. 

An important open question in the interpretation of our data is related to the two-Alternative Forced Choice (2AFC) paradigm used in the object categorization task. Because of this, it cannot be excluded that participants, despite a classification of a stimulus as a shoe, were still aware of some human facial information in the shoe. The explicit categorization of a stimulus as a shoe does not fully exclude awareness of the facial information, and this might then drive the highly successful emotional categorization. However, the very brief stimulus presentation, as well as the very small amount of human face information that still drove the emotional categorization responses, and the mixing of the human face information with the face-like features of the (pure) shoe stimulus (chosen to make morphing with a human face feasible), are all factors that limit the likelihood of this explanation. Further experiments, using some additional measure of residual face perception or a type of confidence measure could address this issue, as well as the question whether residual face information guides perception in a conscious or an unconscious manner. 

Because it cannot be fully excluded that physically present face information in stimuli categorized as shoes could still have been used in a conscious or unconscious manner to construct a basic representation of a face, it also cannot be fully excluded that a subordinate emotion categorization is derived from it, in line with more sequential models of face processing (see Introduction). In particular, our finding that accurate explicit fear-happy emotion categorization, but not face-shoe object categorization, survives a strong reduction in the amount of physical face-related information in the stimuli could be seen as supporting a sequential model in which a common face representation provides input to a highly sensitive mechanisms of emotion categorization and a less sensitive mechanism of object categorization. This appears in agreement with the well-known effects of affective salience on visual attention (Calvo & Nummenmaa, 2008; Eimer & Holmes, 2007; Palermo & Rhodes, 2007). However, despite possible alternative theoretical explanations and open empirical questions, our data may also be seen as support for a possible separation between processing streams for face representations used for emotion categorization and face representations used for conscious object categorization. The present evidence gives interesting clues that provide new avenues for further tests of two-pathway theories of face perception. 

Traditional models of face processing are built on the notion that processing involves two sequentially organized stages, one of category assignment and the other of affective valence attribution. The alternative view (developed more in the context of affective valence processing than object recognition per se) postulates two parallel pathways, one of which is devoted to rapid extraction of relevant information before detailed processing of the visual properties (Anderson, Christoff, Panitz, De Rosa, & Gabrieli, 2003; Bar et al., 2001; de Gelder & Rouw, 2001; Garrido et al., 2012; Rudrauf et al., 2008; Vuilleumier, 2005). In this alternative account, emotion categorization uses a coarse representation of faces that might not be sufficient for categorizing the visual object as a face like in the ambiguous morphs we used in our study, but that might still permit successful emotion categorization. The existence of different entry representations and possibly different types of information for different facial information attributes may be tested in future experiments by assessing the effect of structural facial transformations on face and object categorization. A better understanding of how our visual system extracts emotions from a face is theoretically important, and we therefore believe our findings can give useful input to both computational modeling efforts and neurophysiological investigations of emotion and face perception. Further research is needed to understand the information present in the stimuli that drives the categorization of emotion in stimuli that are not explicitly categorized as faces. 

Conclusion

The present study provides the first evidence that the ability to categorize a stimulus as fearful or happy does not require that the participants explicitly categorize the stimulus as a face, and can take place in stimuli that contain very little physical face information. 

Acknowledgments

This study was partly supported by Human Frontiers Science Program RGP54/2004, NWO Nederlandse Organisatie voor Wetenschappelijk Onderzoek 400.04081, EU FP6-NEST-COBOL043403, and FP7 TANGO to Beatrice de Gelder, and NWO VICI grant to Peter De Weerd. 

Commercial relationships: none. 

Corresponding author: Beatrice L. de Gelder. 

Email: [email protected]. 

Address: Cognitive and Affective Neuroscience Laboratory, Tilburg University, Tilburg, The Netherlands.