Holistic perception of individual faces in the right middle fusiform gyrus as evidenced by the composite face illusion

Abstract

The perception of a facial feature (e.g., the eyes) is influenced by the position and identity of other features (e.g., the mouth) supporting an integrated, or holistic, representation of individual faces in the human brain. Here we used an event-related adaptation paradigm in functional magnetic resonance imaging (fMRI) to clarify the regions representing faces holistically across the whole brain. In each trial, observers performed the same/different task on top halves (aligned or misaligned) of two faces presented sequentially. For each face pair, the identity of top and bottom parts could be both identical, both different, or different only for the bottom half. The latter manipulation resulted in a composite face illusion, i.e., the erroneous perception of identical top parts as being different, only for aligned faces. Release from adaptation in this condition was found in two sub-areas of the right middle fusiform gyrus responding preferentially to faces, including the “fusiform face area” (“FFA”). There were no significant effects in homologous regions of the left hemisphere or in the inferior occipital cortex. Altogether, these observations indicate that face-sensitive populations of neurons in the right middle fusiform gyrus are optimally tuned to represent individual exemplars of faces holistically.

Introduction

Neuroimaging studies have identified a network of areas in the human occipito-temporal cortex that respond preferentially to faces (Haxby, Hoffman, & Gobbini, 2000; Ishai, 2008; Puce, Allison, Gore, & McCarthy, 1995; Rajimehr, Young, & Tootell, 2009; Sergent, Ohta, & MacDonald, 1992), with a right hemispheric dominance. Whether faces are familiar or unfamiliar, two of these areas are known to play a role in the extraction of individual face representations (e.g., Eger, Schweinberger, Dolan, & Henson, 2005; Ewbank & Andrews, 2008; Fox, Moon, Iaria, & Barton, 2009; Gauthier, Tarr et al., 2000; Grill-Spector & Malach, 2001; Schiltz et al., 2006): the “fusiform face area” (“FFA”, Kanwisher, McDermott, & Chun, 1997), which is situated in the lateral middle fusiform gyrus, and the “occipital face area” (“OFA”, Gauthier, Tarr et al., 2000), which is located more posteriorly, in a region of the inferior occipital cortex. 

Behavioral studies, on the other hand, have demonstrated that the processing of a facial feature (e.g., eye, nose, mouth, …) is affected by alterations to the identity or the position of one or several other features of the individual face (e.g., Farah, Wilson, Drain, & Tanaka, 1998; Homa, Haver, & Schwartz, 1976; Mermelstein, Banks, & Prinzmetal, 1979; Sergent, 1984; Suzuki & Cavanagh, 1995; Tanaka & Farah, 1993; Tanaka & Sengco, 1997; Young, Hellawell, & Hay, 1987). The most compelling illustration of this phenomenon comes from an adaptation of the composite face effect (Young et al., 1987) to create a visual illusion in which identical top halves of faces are perceived as being slightly different if they are aligned with different bottom parts, even when the bottom parts are irrelevant and not attended to (Figure 1). This composite face illusion (CFI) is a particularly clear demonstration that the features of a face (here the two halves of a single face) cannot be perceived in isolation. That is, the perception of the attended top part depends on the identity of the bottom part and its position (since the illusion vanishes when the two parts are misaligned spatially; for empirical demonstrations in face matching tasks, e.g., Goffaux & Rossion, 2006; Hole, 1994; Le Grand, Mondloch, Maurer, & Brent, 2004; Michel, Rossion, Han, Chung, & Caldara, 2006; Rossion & Boremanse, 2008). This empirical observation is generally interpreted as evidence for the integration of the facial features into a Gestalt, a global picture, a “configural” (Sergent, 1984; Young et al., 1987), or a “holistic” (Farah et al., 1998; Tanaka & Farah, 1993) representation. 

Figure 1

 

The 6 conditions used in the fMRI experiment. For all conditions, participants had to match the top half of the faces (above the thin white line), which were strictly identical in the “Same” and “Bottom different” conditions (top and middle rows). However, when the two halves were aligned, the association of a different bottom half led to a composite face illusion, marked by an increase of “different” responses and slower RTs (left middle row condition). The bottom row depicts examples in which the top halves of the faces were different, a condition that was not included in the previous study with an orthogonal task (Schiltz & Rossion, 2006).

However, the critical question of how individual features are integrated into a holistic face representation in the human brain remains largely unclear. Using an fMR-adaptation block paradigm (Grill-Spector & Malach, 2001), we showed previously that faces were represented holistically in face-sensitive regions of the occipito-temporal cortex, in particular the right “FFA” (Schiltz & Rossion, 2006). To do that, we tested the “composite face effect” mentioned above in prelocalized face-sensitive areas, using adaptation to facial identity as an indication of the sensitivity of an area to real different faces and to faces erroneously perceived as different. During the critical condition of that study, participants had the subjective impression of viewing different top parts of faces during a block of trials, despite the fact that the top face parts remained unchanged over the block. Importantly, this illusion was only present if the face halves were spatially aligned (vs. misaligned; Young et al., 1987). This perceptual illusion induced by holistic face perception was accompanied by an increased BOLD signal (release from adaptation) in the face-sensitive regions of the occipito-temporal cortex, predominantly in the right “FFA” (Schiltz & Rossion, 2006). This observation is compatible with the view that local features are important in the neural representation of faces (e.g., Harris & Aguirre, 2008), but it indicates that in these areas these features cannot be treated independently of the whole facial identity. 

Here we aimed to strengthen and extend the results of this previous imaging study by using the CFI in an event-related fMR-adaptation paradigm, measuring brain activity across the whole brain. We introduced several important modifications to our paradigm, both to test the reliability of these observations and to refine them but also to be able to disclose the full network of areas (face-sensitive or not) that are involved in individual holistic face perception. First, we used an event-related fMR-adaptation experiment instead of a block design. This modification reduces the potential attentional confounds that could have been present in our previous study (e.g., Wojciulik, Kanwisher, & Driver, 1998), because here participants do not know in advance the kind of stimulus that they will see in each trial (i.e., aligned or misaligned faces; a different or identical face to the previous one). Second, and most importantly, participants of the present study had to perform an individual face discrimination/matching task rather than an irrelevant (color detection) task as in our previous study (Schiltz & Rossion, 2006). These modifications allowed us to assess the behavioral composite effect during scanning in the present study. Moreover, while all the attended top parts of faces were identical in the conditions of this previous study, here we implemented a fully factorial 2 × 3 design with the factors identity and alignment that allowed to compare directly the neural correlates of the illusory (identical top parts perceived as different) and the physical (different top parts) identity changes. Hence, we had 6 conditions in total (compared to 4 in Schiltz & Rossion, 2006), as in a recent study using the composite face effect in an adaptation paradigm in event-related potentials (ERPs, Jacques & Rossion, 2009). Precisely, one of the interests of the present study is that its design was very similar to this recent ERP study in which the composite face effect reflecting holistic face processing was observed starting at about 160 ms following stimulus onset, over the right occipito-temporal cortex (Jacques & Rossion, 2009). While the precise time course of face processing cannot be clarified with fMRI, the relationship between the outcome of this ERP study and the present ER fMRI work should shed light on the spatio-temporal course of holistic face processing. Finally, we used a more powerful magnet (i.e., 3 T vs. 1.5 T) here, improving the effectiveness of the region of interest (ROI) approach by disclosing the complete set of face-sensitive regions in the fusiform and the occipital gyri of both hemispheres in each participant. Importantly, the higher field strength enabled the successful recording of the whole brain during a relatively shorter duration (TR), in order to also test directly for the “neural composite face effect” in the whole brain, i.e., without any a priori definition of face-sensitive regions. 

Methods

Participants and imaging procedures

Eleven healthy adult participants (age 19 to 34, 3 males, 5 left-handed) were scanned at the University of Maastricht with a 3-T head scanner (Siemens Allegra, Siemens AG, Erlangen, Germany), with repeated single-shot echo-planar imaging using the BOLD contrast effect as an indirect marker of local neuronal activity (Ogawa, Lee, Kay, & Tank, 1990). In both experiments, we acquired 21 slices with the imaging parameters: slice thickness = 3.5 mm (no gap), slice order descending and interleaved, echo time (TE) = 50 ms, flip angle (FA) = 90°, matrix size = 64 × 64, field of view (FOV) = 224 × 224 mm. For the face localizer, we acquired data over two runs of 6 min and 42 s, and we used a repetition time (TR) of 1500 ms. In the composite experiment, four runs of 11 min, 27 s, and 500 ms were acquired, and we used a TR of 1250 ms. A three-dimensional (3D) T1-weighted data set encompassing the whole brain was acquired to provide detailed anatomy (1 mm³) using a “Modified Driven Equilibrium Fourier Transform” (MDEFT) sequence (scan parameters: TR = 7.92 ms, TE = 2.4 ms, FA = 15°, matrix size = 256 × 256, FOV = 256 × 256 mm, 176 slices, slice thickness = 1 mm, no gap, total scan time = 13 min and 43 s). 

Stimuli and task description

Face localizer

The block-design localizer fMRI experiment aimed at defining the areas responding preferentially to faces (“FFA” and “OFA”) and consisted of 8 blocks per run (36 s per block) of alternating pictures of faces and objects, with 12-s fixation cross epochs between the blocks. Participants performed a one-back identity task (two positives per block). Thirty-six stimuli were presented for 800 ms followed by a 200-ms blank screen during each block. All images (pictures of faces and objects) were presented in grayscale and sustained a size of roughly 4° of visual angle. They were matched for mean luminosity and varied location in X (10%) and in Y (13%) on each trial. Faces were cropped so that no external features (hair and so on) were present. Participants performed two runs of the face localizer, with the order of blocks counterbalanced between runs (run 1 starting with faces, run 2 with objects). 

Composite experiment

During the main composite experiment, observers viewed 90 pairs of colored images with composite faces in each run. All face images were separated into two parts by a white gap of 4-pixel width on average to ensure that participants could define the exact borders of the top and bottom parts in the aligned stimuli. They had to focus on the top part of the stimuli in order to discriminate/match the two top parts in each pair (same–different delayed task). The first stimulus in each pair was presented for 3500 ms, and after a blank screen of 100 ms, the second stimulus was then briefly presented for 150 ms ( Figure 2). In these conditions, it has been shown that observers maintain fixation on the top part of the face, even when the aligned bottom part differs between faces and lead to a strong composite face illusion (de Heering, Rossion, Turati, & Simion, 2008). The 90 trial pairs (6 conditions × 15 trials by condition) were presented randomly and with a jittered baseline interval lasting 3750 ms on average. The composite faces of this experiment were constructed from a set of 15 different top halves of faces (6 females) and 15 different bottom halves of faces (6 females) that were combined to create 60 different composite faces. Over the 4 runs, participants thus viewed 360 composite face pairs (60 trials in each condition) and each composite face was repeated with an equal frequency (n = 12) and appeared equally often in each condition. Faces were matched for mean luminosity and they were cropped so that no external features (hair and so on) were present (Figure 1). The composite face pairs were formed either by aligning the top and bottom parts precisely to each other (while leaving the small gap) or by slightly misaligning the two parts. The lateral offset was placed on the tip of the nose under the pupil (approximately 40 pixels = 0.9°) and it was directed equally often to the left and right sides. The stimuli with offset bottom parts are commonly called “misaligned” composite faces. Aligned faces subtended approximately 4.2° in width × 5.6° in height, while misaligned faces subtended 5.1° in width × 5.6° in height. The gap between top and bottom parts was 3 pixels in height (<0.1°). 

Figure 2

 

Timeline of the events (one trial) presented during fMRI recording. Following variable fixation durations (2–4 TRs of 1250 ms), an adapting face was presented for 3500 ms and participants had to focus on the top half of the face (sample to match). After a brief ISI of 100 ms, a target face was presented for a short duration (150 ms) and participants had to decide whether the top half was identical to the adapting face or not (here the answer is “identical”, but the fact that the bottom half is different leads to an increase of RTs and error rates).

The present paradigm crossed three levels of “identity” (different/illusionary different/identical) and two levels of “alignment” (aligned/misaligned top and bottom parts) in a factorial design to obtain six conditions: aligned different (ali_dif), aligned bottom different (ali_bot_dif), aligned identical (ali_id), misaligned different (misali_dif), misaligned bottom different (misali_bot_dif), misaligned identical (misali_id; Figure 1). Thus the “different” pairs contained both different face tops and different face bottoms, the “bottom different” pairs differed only with respect to the bottom parts and in the “identical” pairs both the top and bottom parts were repeated within a pair. In all conditions, the task was to concentrate on the top face parts and decide whether they were identical or different within face pairs. Stimuli and pairs were displayed in a random order with a PC running E-prime 1.1 (PST) through a projector surface located over the head of the participant and viewed with an angled mirror. 

Data analysis

All FMRI data were analyzed using BrainVoyager QX (Version 1.8, Brain Innovation, Maastricht, The Netherlands) applying a regression analysis. Prior to statistical analysis, preprocessing was done as in Schiltz and Rossion (2006). Preprocessing consisted of a linear trend removal for excluding scanner-related signal, a temporal high-pass filtering applied to remove temporal frequencies lower than 3 cycles per run, and a correction for small interscan head movements by a rigid body algorithm rotating and translating each functional volume in 3D space (small interscan head movements <3 mm/deg in all directions). In addition, the data were also corrected for the difference between the scan times of the 21 slices. Data were smoothed in the spatial domain (4-mm FWHM). In order to be able to compare the locations of activated brain region across subjects, all anatomical as well as the functional volumes were spatially normalized (Talairach & Tournoux, 1988). Subsequently, the functional data were analyzed using multiple regression models (General Linear Model, GLM) consisting of predictors, which corresponded to the particular experimental conditions of each experiment. The predictor time courses used were computed on the basis of a linear model of the relation between neural activity and hemodynamic response (Boynton, Engel, Glover, & Heeger, 1996). 

ROI definition and analysis

The areas responding preferentially to faces were defined independently of the experimental runs, for each individual participant, by means of the conjunction of the contrast (faces vs. objects) between the two face localizer runs. This procedure was used in previous studies and ensured that larger activations to faces than objects identified were those consistent across the two runs (Dricot, Sorger, Schiltz, Goebel, & Rossion, 2008; Sorger, Goebel, Schiltz, & Rossion, 2007). For most regions and participants, all contiguous voxels in the middle fusiform gyrus and the inferior/middle occipital gyrus significant at t > 10.51 (p(Bonf) < 0.0001, one-tailed, corrected for multiple comparisons) were considered for further analysis and defined as the “FFA” and “OFA” regions, respectively. When foci of activations were too large so that fusiform and inferior occipital activation foci overlapped, the statistical threshold was raised to a more conservative value [t > 13.51, p(Bonf) < 0.00001] to separate the two functional regions spatially. When one of the regions of interest could not be found, the threshold was lowered to t > 5.71 (p(Bonf) < 0.001; see Table S1 in Supplementary material for definition of ROIs in all participants). In sum, these different thresholds were used to be able to disclose and isolate clearly all 4 regions of interest in every participant of the experiment (see Dricot et al., 2008; Sorger et al., 2007). Importantly, these areas were defined strictly independently of the experimental manipulation of interest (i.e., the composite illusion). 

The individual regions of interest (ROIs) were then tested for an interaction between the factors “identity” and “alignment” using the beta weights of the GLM obtained in these regions for the ER composite face experiment (for which all trials, 60 in total, for each condition were considered in the analysis). We applied a repeated-measures ANOVA (random effect analysis) and calculated post-hoc comparisons of interest to assess the adaptation effects (different–identical faces) and the composite effect (bottom different–identical faces) for both aligned and misaligned faces. 

Whole-brain analysis

We also performed a (random effect) whole-brain analysis to highlight the regions that were sensitive to the composite face effect, independently of their preferential response to face stimuli using the contrast [aligned (bottom-diff–identical)–misaligned (bottom-diff–identical)]. We identified all the regions that were significant at t > 3.6, p < 0.005, uncorrected for multiple comparisons (minimum of 50 contiguous voxels). 

Supplementary analysis (correct vs. erroneous responses in ali_bot_dif condition)

In the analyses for the bottom-different condition, we included data for both correct and incorrect responses and not only for incorrect responses, because the composite illusion does not only result in incorrect responses, it also generates increased response times even when subjects provide a correct response (e.g., Young et al., 1987; see Jacques & Rossion, 2009). It is therefore almost impossible to determine whether the illusion took place or not on any given trial based on behavioral data, and all trials may be relevant in the experiment. However, we also compared the BOLD signal related to correct and erroneous responses in the critical ali_bot_dif condition, by performing a complementary analysis in which the correct and erroneous ali_bot_dif responses were considered as separate conditions in a repeated-measures ANOVA (random effect analysis). To test whether the BOLD response in the “aligned bottom different” condition differed according to response accuracy, we calculated the post-hoc comparison ali_bot_dif_CORRECT vs. ali_bot_dif_ERROR in the above-mentioned individual ROIs and in the regions reaching significance in the whole-brain analysis. 

Results

Behavioral results

Accuracy rates were high and similar in all conditions, ranging from 84% to 88%, with the exception of the condition of interest (“ali_bot_dif”, different bottom on aligned faces, expected response is “same”), for which accuracy rates dropped to 54% ( Figure 3A). This was also the condition that was performed more than 100 ms slower than all other conditions, for which average correct RTs were highly similar ( Figure 3B). 

Figure 3

 

Behavioral results during fMRI recording. Performance was very good overall, except for the condition for which participants were sensitive to the composite face illusion and pressed the “different” response key in almost half of the trials, with a substantial increase of RTs. This reflects the behavioral composite face illusion.

Statistical results validated these observations. The repeated-measures ANOVA (2 × 3, alignment × identity) on accuracy rates showed main effects of alignment ( F(1,10) = 39.82, p < 0.0001) and identity ( F(2,20) = 14.24, p < 0.0001) and, most importantly, a significant interaction between the two factors ( F(2,20) = 33.91, p < 0.0001). Post-hoc comparisons showed that the conditions did not differ for misaligned faces (“misali_bot_dif”–“misali_id”: p = 0.4; “misali_bot_dif”–“misali_dif”: p = 0.36; “misali_dif”–“misali_id”: p = 0.09). However, on aligned faces, the performance was significantly lower for the bottom condition (“ali_bot_dif”) than the different (“ali-dif”; p < 0.001) and identical conditions (“ali_id”; p < 0.001), which did not differ from each other ( p = 0.74). The analysis performed on correct response times revealed only a significant interaction between alignment and identity ( F(2,20) = 7.04, p = 0.0039) with both main effects failing to reach significance ( alignment: F(1,10) = 2.45, p = 0.15; identity: F(2,20) = 2.17, p = 0.14). For correct RTs, post-hoc comparisons showed that the conditions did not differ for misaligned faces (“misali_bot_dif”–“misali_id”: p = 0.17; “misali_bot_dif”–“misali_dif”: p = 0.96; “misali_dif”-“misali_id”: p = 0.69). However, on aligned faces, the performance was lower for the bottom condition (“ali_bot_dif”) than the different (“ali-dif”; marginally significant, p = 0.06) and identical (“ali_id”) conditions ( p = 0.022), which did not differ from each other ( p = 0.58). 

Neuroimaging results (1): Regions of interest

For each participant, we first identified the bilateral areas responding preferentially to faces in the IOG and MFG as regions of interest (ROIs, “FFA” and “OFA”; Figure 4; see Table S1 in Supplementary material for details about these areas in individual participants). 

Figure 4

 

Sagittal, coronal, and transversal views of the bilateral “FFA” and “OFA”: regions responding more to faces than to non-face stimuli (objects) in the group analysis ( n = 11) that served as regions of interest for the main experiment ( n = 11) investigating the neural correlates of the composite face illusion ( p < 0.05 corrected for multiple comparisons).

In the right “FFA”, the 2 × 3 ANOVA revealed main effects of alignment (larger response to aligned faces; F(1,10) = 6.63, p < 0.028) and identity ( F(2,20) = 13.46, p < 0.001), with a significant interaction between the two factors ( F(2,20) = 3.7, p < 0.043). Post-hoc comparisons showed that the release from adaptation (different–identical faces) was significant for both aligned and misaligned faces ( p = 0.002 and p = 0.02, respectively), and its magnitude did not differ depending on alignment (2 × 2 interaction: p = 0.52). However, and most importantly, the interaction was found because the comparison “bottom different–identical faces” was significant only for aligned faces (CFI: p = 0.0016) and not at all for misaligned faces (control: p = 0.744). A 2 × 2 ANOVA with the 4 critical conditions of interest (same top halves) led to a highly significant interaction between alignment and the presence of different bottom identities (2 × 2 interaction, p = 0.0086). When the two halves of the faces were aligned, the release from adaptation was almost as large for the bottom different condition (“ali_bot_dif”) than when the two halves were different (“ali_dif”; Figure 5; p = 0.58). However, there was no such release when the two halves of the face were misaligned, leading to a significantly larger effect of adaptation for “misali_dif” than “misali_bot_dif” ( p = 0.005). 

Figure 5

 

BOLD response (expressed as beta weights of the GLM) to the composite face stimuli in the regions of interest “FFA” and “OFA”. To illustrate the magnitude of the neural composite effect, beta weights are plotted in the four regions responding preferentially to faces in the occipito-temporal cortex. The condition of interest “ali_bot_dif” is illustrated in dark blue color (i.e., the face parts are aligned and differ only for their bottom parts). It is expected to be significantly larger than “ali_id” (shown in dark green) in the areas that are sensitive to the illusory perception of a different top part of the face. Beta weights refer to the scaling of regressors in a GLM ( n = 11), in which the BOLD response is modeled by the linear convolution of boxcars of neural activity by an assumed BOLD impulse response function. Error bars refer to mean standard errors.

These observations clearly show that even when participants actively process the top part of the identical faces of the pair, presenting a different bottom half causes a significant release from adaptation in the right “FFA”, providing that the two halves are aligned with each other and form a whole face ( Figures 5 and 6). 

Figure 6

 

Time courses of the 4 conditions of interest of the study (for which a “same” response is expected). Note the large release from face identity adaptation in the FFA of both hemispheres, with the only significant effect in the right “FFA”. Despite showing a larger response overall when considering all conditions, the effects were much weaker and non-significant in the “OFAs”. Error bars refer to mean standard errors.

The same analyses were performed in the other three ROIs. In the left “FFA”, the ANOVA revealed main effects of alignment (larger response to aligned faces; F(1,10) = 16.17, p < 0.0024) and identity ( F(2,20) = 7.954, p < 0.0035), but without a significant interaction between the two factors ( F(2,20) = 0.975, p = 0.39). Post-hoc comparisons showed that the release from adaptation (different–identical faces) was significant for both aligned and misaligned faces ( p = 0.0034; p = 0.034, respectively), and its magnitude did not differ depending on alignment (2 × 2 interaction, p = 0.28). Even though the comparison “bottom different–identical faces” was significant for aligned faces (“ali_bot_dif”–“ali_id”: CFI, p = 0.025) and not for misaligned faces (“misali_bot_dif”–“misali_id”: control, p = 0.41), the critical interaction between alignment and the presence of different bottom identities failed to reach significance (2 × 2 interaction, p = 0.21) in the left “FFA”. When the two halves of the faces were aligned, the release from adaptation was almost as large for the bottom different condition than when the two halves were different ( Figure 5; “ali_dif”–“ali_bot_dif”, p = 0.55), while the release was present but not as large when the two halves of the face were misaligned (“misali_dif”–“misali_bot_dif”, p = 0.06). In short, the critical interaction of interest failed to reach significance in the left hemisphere, even though the left “FFA” appeared to show a similar profile of response as the right “FFA”. 

To test directly for the effect of lateralization at the level of the FFA, we ran an ANOVA on the critical conditions of interests, with hemisphere as factor, and failed to find a triple interaction between hemisphere, alignment, and identity of the bottom half of the face ( F(1,10) = 0.44, p = 0.52; interaction between alignment and identity of the bottom half: F(1,10) = 5.45, p < 0.04). 

In the right “OFA”, there was no main effect of alignment ( F(1,10) = 0.14, p < 0.71) but a significant effect of identity ( F(2,20) = 3.78, p < 0.04), without a significant interaction between the two factors ( F(2,20) = 0.25, p = 0.78). Post-hoc comparisons showed that the release from adaptation (different–identical faces) was significant for aligned ( p = 0.0036) but not for misaligned faces ( p = 0.19), even though the interaction failed to reach significance ( p = 0.68). The comparison “bottom different–identical faces” was not significant, neither for aligned nor misaligned faces (“ali_bot_dif”–“ali_id”: p = 0.21; “misali_bot_dif”–“misali_id”: p = 0.6), and there was no significant interaction between alignment and the presence of different bottom identities (2 × 2 interaction, p = 0.48) in this area. Whether the two halves of the faces were aligned or misaligned, the release from adaptation was not significantly larger when the two halves were different than when only the bottom half was different ([“ali_dif”–“ali_bot_dif”]: p = 0.33; [“misali_dif”–“misali_bot_dif”]: p = 0.21 for aligned and misaligned faces, respectively). 

Finally, in the left “OFA”, there was no main effect of alignment ( F(1,10) = 0.24, p < 0.64) and no main effect of identity ( F(2,20) = 1.84, p < 0.18) but a significant interaction between the two factors ( F(2,20) = 3.63, p = 0.045). Post-hoc comparisons showed that this pattern was due to the identical condition for aligned faces showing a reduced response compared to all other conditions ( Figure 5). The release from adaptation (different–identical faces) was significant for aligned ( p = 0.015) but not for misaligned faces ( p = 0.52), even though the interaction failed to reach significance ( p = 0.13). The comparison “bottom different–identical faces” was close to significance for aligned faces ([“ali_bot_dif”–“ali_id”]: p = 0.06) but not for misaligned faces ([“misali_bot_dif”–“misali_id”]: p = 0.70). Consequently, there was a significant interaction between alignment and the presence of different bottom identities ( p = 0.023) in this area. There were no significant differences between the condition with the two different halves and the condition with a different bottom half ([“ali_dif”–“ali_bot_dif”]: p = 0.61; [misali_dif”–misali_bot_dif”] p = 0.16). 

As for the FFA, there was no significant lateralization of the composite face effect at the level of the OFA (triple interaction between hemisphere, alignment, and identity of the bottom half of the face: F(1,10) = 1.67, p = 0.23; interaction between alignment and identity of the bottom half: F(1,10) = 3.91, p = 0.08). 

The supplementary analysis contrasting correct and erroneous responses in the ali_bot_dif condition revealed similar response profiles bilaterally in the fusiform (rMFG: p = 0.18, lMFG: p = 0.08) and prefrontal ROIs (rPFG: p = 0.4, lPFG: p = 0.95), as well as in the right occipital area (rOFA: p = 0.57). Only in the left OFA the BOLD signal was significantly higher for correct than for incorrect responses (lOFA: p = 0.01). 

In summary, in all the four regions of interest, presenting a different bottom in the second face stimulus caused a release from adaptation, even when the attended top parts were identical. Interestingly, the critical interaction (a larger release from adaptation to aligned than misaligned faces), revealing the CFI at the neural level, was found to be highly significant only in the “rFFA”. This interaction was also significant in the “lOFA”, but the pattern of observations was less clear, this interaction being largely due to an absence of adaptation for all conditions where misaligned faces were presented ( Figure 5). 

Neuroimaging results (2): Whole-brain analysis

The results of the whole-brain analysis of the interaction of interest [aligned (bottom-diff–identical)–misaligned (bottom-diff–identical)] revealed two adjacent areas in the right middle fusiform gyrus and three areas of activation in the dorsolateral prefrontal cortex: bilaterally in the inferior frontal gyrus (iPFG) at the border with the precentral gyrus (PrCG, BA44/6) and in the left inferior frontal gyrus (BA 44; Table 1 and Figure 7). The two areas of the right fusiform gyrus were located slightly posterior to the average coordinate of the right FFA but presented a significantly larger response to faces than objects in the face localizer (posterior: p < 0.05; anterior: p < 0.005). Interestingly, this was also the case for the 3 prefrontal regions identified (right iPFG/PrCG: p < 0.0001; left iPFG/PrCG: p < 0.0001; left iPFG: p < 0.01). 

Table 1

 

Coordinates of the five regions yielding significant activations ( p < 0.005, t > 3.6) in the whole-brain analysis testing for a composite face illusion.

Region	Talairach coordinates			Z-score	Size
	x	y	z
Right posterior middle fusiform gyrus	41	−52	−8	6.2	254
Right anterior middle fusiform gyrus	39	−40	−11	5.0	55
Right dorsolateral prefrontal cortex (BA44)	39	−2	30	5.1	215
Left dorsolateral prefrontal cortex (BA44)	−43	−1	25	5.8	226
Left dorsolateral prefrontal cortex (BA9, 44)	−39	13	15	5.1	158

Figure 7

 

Regions yielding significant activations ( p < 0.005, t > 3.6) in the whole-brain analysis testing for a composite face illusion. The two posterior regions were situated in the middle fusiform gyrus (MFG), slightly dorsal to the “rFFA”. In the dorsolateral prefrontal cortex, two regions were located at the border of the inferior frontal and the precentral gyri (iPFG/PrCG) bilaterally. In the left hemisphere, a second dorsolateral prefrontal activation was observed still more anteriorly, in the inferior frontal gyrus (iPFG).

Importantly, there was a clear dissociation between the profiles of response observed in the middle fusiform regions vs. the prefrontal cortex across all the conditions of the composite face experiment. On the one hand, the right fusiform regions both showed identity adaptation effects (different–identical faces, posterior MFG: p < 0.05; anterior: p = 0.06) in addition to the release from adaptation reported above for the composite illusion (bottom different–identical faces). On the other hand, the prefrontal regions showed a maximal response to the condition leading to the increase of error rates and RTs, i.e., when the bottom of the aligned faces differed and participants had to respond “same” ( Figures 7 and 8). This effect does not appear to be related to a release from adaptation, as the three prefrontal regions did not show a simple release from face identity adaptation effect (all ps < 0.82). 

Figure 8

 

Beta weights in response to the composite face stimuli in the five regions yielding significant activations in the whole-brain analysis ( n = 11) testing for a composite face illusion (left side = three areas in the right hemisphere; right side = two prefrontal areas in the left hemisphere). All five regions revealed a significant neural composite face effect but differed in terms of their response to the “different” condition. Only the middle fusiform areas revealed a significant face identity adaptation effect. In the prefrontal cortex, the critical “ali_bot_dif” condition (in dark blue) yielded a higher BOLD response than all other conditions. This functional profile indicates a close link with the increased response difficulty in this illusion condition. Error bars refer to mean standard errors.

Another way of looking at these differences between fusiform and prefrontal areas is by comparing directly the condition for aligned faces when the two faces to match are completely different (“dif”) to the condition with the bottom half being different only. For the two regions identified in the fusiform gyrus, these comparisons did not reach significance ( p > 0.3), while the condition “bottom-different”, which was the most difficult in the experiment, led to the largest signal in all the prefrontal regions (all ps < 0.01, Figure 8). 

Finally, as for the ROI analysis, correct and incorrect trials in the ali_bot_dif condition revealed similar response profiles in all the five above-mentioned areas showing a significant CFI in the whole-brain analysis (posterior MFG: p < 0.82; anterior MFG: p = 0.40; right iPFG/PrCG: p < 0.17; left iPFG/PrCG: p < 0.52; left iPFG: p < 0.35). 

Discussion

As in behavioral studies of the composite face effect with unfamiliar faces, participants had to focus on top halves of two faces presented consecutively and decide whether they were identical or different. In the critical “bottom-different” condition, participants performed significantly worse than in the conditions were both the bottom and top parts were either different or identical. Importantly, this composite effect was found only when the top and bottom halves were spatially aligned. This behavioral effect, i.e., the illusory perception of top halves of faces as being different, was accompanied by a reduced susceptibility to fMR adaptation in the face-sensitive regions of the occipito-temporal cortex, mainly in the right “FFA”. These results strengthen our previous observations of a “neural composite face effect” in fMRI with different participants and stimuli (Schiltz & Rossion, 2006). Most importantly, we demonstrated the same effect by means of an event-related paradigm here, in which the order of the trials from different conditions was fully randomized and unpredictable, thus reducing substantially the probability of disclosing differences between conditions due to attentional confounds (D'Esposito, Zarahn, & Aguirre, 1999). Moreover, while an orthogonal (color detection) task was used previously (Schiltz & Rossion, 2006), here we measured the CFI behaviorally while performing fMRI at the same time. Both our present and past findings thus suggest that populations of neurons in the FFAs, in particular in the right hemisphere, integrate information from the two face halves into a representation of the whole face. 

Because we also added a condition in which both the top and bottom halves of the faces differed in a trial, we were able to show that, for aligned faces, changing only the bottom part in a face pair (i.e., “bottom-different” condition) reduced the susceptibility to fMR adaptation in the right “FFA” to the same extent as changing both the bottom and top parts (i.e., “different” condition). In other words, the BOLD effects related to the illusory and the real identity changes were of roughly the same magnitude in this area. This observation is in line with evidence that a nearly full release from adaptation to face identity is observed in the “FFA” even for unfamiliar faces that are quite similar (e.g., with a 30% difference on a morph between two face identities; Gilaie-Dotan & Malach, 2007). 

In the OFAs, contrary to the FFAs, there was release from adaptation only when both the bottom and top halves were different but not when only the bottom half of the face differed. These observations clearly strengthen the dominant role of the “FFA”, relative to the “OFA”, in particular in the right hemisphere, in representing individual faces holistically. 

The whole-brain analysis also highlighted two neighboring regions in the right MFG presenting release from adaptation when the bottom face half changed. Although the coordinates of these areas do not directly correspond with those of the “rFFA”, they were located just a few millimeters more dorsally (right post MFG: z = −8; right ant MFG: z = −11 vs. “rFFA”: z = −15 ± 4), overlap in part with the “rFFA”, and also respond significantly stronger to faces than objects (at a lower statistical threshold). This observation can be related to two fMRI studies of face processing, which disclosed sensitivity to faces differing by relative distances between features in other areas of the ventral stream than the most face-sensitive regions (Maurer et al., 2007; Ng, Ciaramitaro, Anstis, Boynton, & Fine, 2006). Most interestingly for the present study, Maurer et al. (2007) found a region that was located slightly more dorsal and posterior than the “rFFA” and showed an increased BOLD response when faces differed in the spacing rather than the shape of their features. The most likely explanation of these observations is that there are in fact several patches in the fusiform gyrus responding preferentially to faces, with the area of the middle fusiform gyrus being the region that is most consistently found in a simple face localizer task and accordingly labeled the “FFA”. However, this area may not always be the one showing the most significant effect of an experimental manipulation, even for face stimuli. This underlines the interest to complement face localizer approaches by whole-brain fMRI analyses (see Friston, Rotshtein, Geng, Sterzer, & Henson, 2006; Saxe, Brett, & Kanwisher, 2006) in order to derive more general conclusions than those limited to the role of the 2 or 3 most face-sensitive areas in the human brain. 

Importantly, the whole-brain analysis did not reveal any BOLD effect related to the composite face illusion in the right inferior occipital cortex. Both the ROI and the whole-brain analysis thus further highlight the right MFG, including the “rFFA”, as being the dominant region of the occipito-temporal cortex that is concerned with the integration of individual face parts into a holistic face representation (Harris & Aguirre, 2008; Schiltz & Rossion, 2006). However, it remains possible that holistic processing of individual faces in this area may depend on integration of information with other areas sensitive to faces in the neural face network, such as the “OFA” (see Rossion, 2008b). 

The dominance of a rather anterior visual area such as the middle fusiform gyrus in supporting holistic (i.e., global) representations of individual faces is consistent with the fact that neurons in this higher level visual area have presumably a larger receptive field as compared to neurons in lower level visual areas (e.g., the inferior occipital cortex, or “OFA”). Almost by definition (see Rossion, 2008a), an effect reflecting the holistic processing of an individual face such as the composite face effect requires that several facial features across the whole face are perceived simultaneously (or at least within a short time frame, see Anaki, Boyd, & Moscovitch, 2007; Singer & Sheinberg, 2006). This function is thus much more likely to depend on neurons located quite high in the visual hierarchy, which are sensitive to the entire face (Desimone, Albright, Gross, & Bruce, 1984; Tsunoda, Yamane, Nishizaki, & Tanifuji, 2001) or object (e.g., Grill-Spector et al., 1998; Lerner, Hendler, Ben-Bashat, Harel, & Malach, 2001) stimulus. 

Even though we found evidence for holistic face representations in the right middle fusiform gyrus, our results should not be taken as evidence that the right “FFA” would not process local face parts, as shown in other fMRI studies (Harris & Aguirre, 2008; Liu, Harris, & Kanwisher, 2009; Yovel & Kanwisher, 2004). Obviously, both the processing of local details and of the stimulus as a whole are fundamental aspects of face perception. These results also go beyond the demonstration that this anterior region is sensitive to a generic global face configuration (Liu et al., 2009), to the general structure of a face (Caldara & Seghier, 2009), or to a whole general object/face shape (Hasson, Hendler, Ben Bashat, & Malach, 2001; Lerner et al., 2001). What is demonstrated here and in our previous work is that when observers process one part of an individual face, populations of neurons in the right middle fusiform gyrus, including the “rFFA”, are influenced by the identity of other part(s) (unless the face parts would be spatially misaligned, scrambled, or if the face would be presented upside down; Schiltz & Rossion, 2006). 

A series of recent fMRI studies explored questions that are closely related to the present investigation of holistic face processing and that concern the processing of second-order spatial relations in faces, i.e., distances between facial features (Goffaux, Rossion, Sorger, Schiltz, & Goebel, 2009; Rhodes, Michie, Hughes, & Byatt, 2009; Rotshtein, Geng, Driver, & Dolan, 2007). Although the approaches adopted in these studies differed substantially both in terms of tasks performed (i.e., face categorization, face discrimination, emotion detection, and color detection) and analyses (i.e., differential psychometrics, multivariate analysis, and region of interest analysis), they agree in attributing an important role to the right middle fusiform gyrus (in or close to the “rFFA”) when it comes to processing and integrating information that is spatially distributed over the whole individual face stimulus. 

The right lateralization of the effect disclosed here is in agreement with the recent observation with the same paradigm in ERPs of a composite face effect limited to a small sample of right occipito-temporal electrodes (Jacques & Rossion, 2009). In that study, the influence of the different bottom faces on the perception of the top face started at about 165 ms (N170 component, e.g., Bentin, McCarthy, Perez, Puce, & Allison, 1996) following stimulus onset and lasted for about 100 ms. While source localization studies have generally reported a network of occipito-temporal areas as potential candidates for the generation of the N170, including the right middle fusiform gyrus (e.g., Deffke et al., 2007; Henson et al., 2007), this rather distributed pattern of areas concern the localization of the whole N170 rather than the more modest modulation of the amplitude of this component in the composite identity adaptation paradigm (Jacques & Rossion, 2009). It may thus well be that the spatio-temporal locus of the composite face effect is quite restricted in time and space. The present observations suggest that a primary candidate for this early (160 ms) response to a whole individual face is the middle right fusiform gyrus, in particular the right “FFA”. 

More generally, the right hemispheric dominance for holistic face processing as observed in neuroimaging (Harris & Aguirre, 2008; Lobmaier, Klaver, Loenneker, Martin, & Mast, 2008; Rossion et al., 2000; Schiltz & Rossion, 2006) fits with at least two sources of evidence in the face processing literature. First, divided visual field studies supported a dominant role of the right hemisphere in holistic processing of individual faces (Hillger & Koenig, 1991). Second, acquired prosopagnosia, the inability to recognize faces following bilateral or unilateral right brain damage (Barton, Press, Keenan, & O'Connor, 2002; Bouvier & Engel, 2006; Sorger et al., 2007), has been associated with a deficit in holistic face processing in many patients (e.g., Boutsen & Humphreys, 2002; Joubert et al., 2003; Ramon, Busigny, & Rossion, 2009; Saumier, Arguin, & Lassonde, 2001; Sergent & Signoret, 1992). 

Interestingly, the whole-brain analysis also disclosed three prefrontal regions in which the profile of the BOLD response revealed a neural composite face effect. However, the response profile of these prefrontal regions differed from the pattern observed in the middle fusiform regions. That is, whereas there were identity adaptation effects to both the illusory and physical identity changes of the attended top part of the face in the middle fusiform regions, there was no release to the physical identity change in the anterior prefrontal regions ( Figure 8). These regions showed a maximal and exclusive BOLD response in the “illusion” condition, which leads to the most erroneous and the slowest responses. Thus, while one cannot exclude that these dorsolateral prefrontal cortex (DLPFC) areas—which were face-sensitive in the localizer ¹—play a role in face processing (e.g., Allison, Puce, Spencer, & McCarthy, 1999; Barbeau et al., 2008; Carlson, Grol, & Verstraten, 2006; Halgren et al., 1994; Wilson, Scalaidhe, & Goldman-Rakic, 1993), the behavioral data rather suggest that their activation mainly reflects the increased difficulty of the facial decision (that is accompanied by an increased error rate and a decreased response speed), and thus the decisional consequence of the composite face paradigm in a matching task (Richler, Gauthier, Wenger, & Palmeri, 2008). 

Finally, one question that is worth discussing briefly is the extent to which our observations are specific to faces or if they could be generalized to other object categories as well. On the one hand, behavioral studies indicate that the composite face effect or other effects considered as markers of holistic processing are either specific or much stronger for faces than other non-face object categories (e.g., Robbins & McKone, 2007; Tanaka & Farah, 1993). Moreover, it is important to note that the effects that were found here were restricted to areas responding much more to faces than non-face object categories. These findings suggest that the present observations and our conclusions may indeed be valid only for face stimuli, which would be particularly dependent on such a holistic representation. This is not to say that non-face objects are not perceived holistically. It is obvious that non-face objects are also perceived as integrated wholes in the visual cortex (e.g., Baker, Behrmann, & Olson, 2002; Davidoff & Donnelly, 1990; Kimchi, 1994). However, it is important again to stress that we are referring here to the holistic processing of individual faces, not of the face as a generic category (e.g., the detection of a face in a visual scene, or the perception of a face in a two-tone image with little featural information, Mooney, 1957). For most non-face objects of the visual world, such a generic or basic level categorization is sufficient, whereas faces have to be processed at a more subordinate or individual level (Tanaka, 2001). One critical function of the regions identified here may thus be to subtend holistic perception for individual items of a very special category of visual stimuli, i.e., faces. 

In conclusion, combining fMR adaptation with the composite face illusion has enabled us to strengthen, refine, and extend previous observations indicating that neurons responding preferentially to faces in the right middle fusiform gyrus are optimally tuned to represent individual faces holistically. 

Supplementary Materials

Table S1 - Table S1 

Table S1. Parameters of the face-sensitive regions of interest as identified in the localizer runs for each individual participant in the study. 

Acknowledgments

This study was supported by a research grant (ARC 07/12-007, Communauté Française de Belgique—Actions de Recherche Concertées). Bruno Rossion is supported by the Belgian National Research Fund (FNRS). We would like to thank 3 anonymous reviewers for their helpful comments on a previous version of this manuscript. 

Commercial relationships: none. 

Corresponding authors: Christine Schiltz; Bruno Rossion. 

Emails: [e-mail]; [e-mail]. 

Addresses: EMACS Unit, FLSHASE, University of Luxembourg, Campus Walferdange, B.P.2, L-7201 Walferdange, Luxembourg; Unité Cognition et Développement, Université Catholique de Louvain, 10 Place du Cardinal Mercier, 1348 Louvain-la-Neuve, Belgium. 

Footnote

Footnotes

¹  It should be noted that a larger response to faces in these regions as found in the face localizer paradigm could also be accounted for by the fact that the one-back task on face stimuli is inherently harder than the non-face stimuli (objects of different shapes and colors). This is a caveat that concern a majority of fMRI studies of face processing relying on such localizers, even though the larger activation for faces than objects in the visual face areas (“FFA”, “OFA”, “pSTS”) is independent of this difference in difficulty, being observed through a variety of tasks and control stimuli for faces (e.g., Kanwisher et al., 1997).