Supra-additive contribution of shape and surface information to individual face discrimination as revealed by fast periodic visual stimulation

Abstract

Face perception depends on two main sources of information—shape and surface cues. Behavioral studies suggest that both of them contribute roughly equally to discrimination of individual faces, with only a small advantage provided by their combination. However, it is difficult to quantify the respective contribution of each source of information to the visual representation of individual faces with explicit behavioral measures. To address this issue, facial morphs were created that varied in shape only, surface only, or both. Electrocephalogram (EEG) were recorded from 10 participants during visual stimulation at a fast periodic rate, in which the same face was presented four times consecutively and the fifth face (the oddball) varied along one of the morphed dimensions. Individual face discrimination was indexed by the periodic EEG response at the oddball rate (e.g., 5.88 Hz/5 = 1.18 Hz). While shape information was discriminated mainly at right occipitotemporal electrode sites, surface information was coded more bilaterally and provided a larger response overall. Most importantly, shape and surface changes alone were associated with much weaker responses than when both sources of information were combined in the stimulus, revealing a supra-additive effect. These observations suggest that the two kinds of information combine nonlinearly to provide a full individual face representation, face identity being more than the sum of the contribution of shape and surface cues.

Introduction

Human faces are characterized by two main properties: shape and surface information. Face shape is defined essentially by the bone structure and the soft tissue of the face, while surface information captures the color (light reflectance from the facial surface) and texture information (such as stubble and wrinkles). Over the past few years, behavioral studies have shown that both shape and surface information contribute to person discrimination for personally familiar (Russell & Sinha, 2007), famous (Burton, Jenkins, Hancock, & White, 2005; Lee & Perrett, 1997), and unfamiliar (Jiang, Blanz, & Rossion, 2011; Michel, Rossion, Bülthoff, Hayward, & Vuong, 2013; O'Toole, Vetter, & Blanz, 1999; Russell, Biederman, Nederhouser, & Sinha, 2007; Russell, Sinha, Biederman, & Nederhouser, 2006) faces. Yet neither surface nor shape information alone can completely account for the improved discrimination of faces varying in both their shape and surface combined (Russell et al., 2006; see also Caharel, Jiang, Blanz, & Rossion, 2009; Jiang et al., 2011; O'Toole et al., 1999; Russell et al., 2007). 

Surprisingly, few studies have shown differences between the respective contributions of these two characteristics to face identity discrimination. A recent adaptation study suggested that shape plays a more dominant role for identity discrimination than does surface information (Lai, Oruç, & Barton, 2013). Jiang, Blanz, and Rossion (2011) used the composite and inversion face effects to show that the integration of the face parts into a unified representation, a so-called holistic representation of the face (Rossion, 2013; Tanaka & Farah, 1993; Young, Hellawell, & Hay, 1987), depends relatively more on shape than on surface information. Conversely, studies using contrast negation have highlighted the importance of surface information for face identity discrimination (Russell et al., 2006; Vuong, Peissig, Harrison, & Tarr, 2005). Improved recognition of faces based on surface information is also supported by a principal components analysis of face images (Calder, Burton, Miller, Young, & Akamatsu, 2001; Hancock, Burton, & Bruce, 1996) and by successful recognition of shape-free averages of familiar faces (Burton et al., 2005). 

However, these studies measured behavioral outputs that reflect a mixture of perceptual and decisional processes in explicit face identity discrimination tasks. Hence, it is difficult to assess the relative contributions of shape and surface information to the visual representation of face identity independent of decisional factors. Moreover, in behavioral individual face discrimination paradigms, the variables measured—accuracy rates and response times (RTs)—do not allow for quantification of the respective contributions of shape and surface information to face identity. This is particularly the case because accuracy rates are usually relatively high for recognition based on each of these sources (i.e., between 70% and 80%) and performance is spread over two variables (accuracy and RTs). For these reasons, these studies cannot determine whether shape and surface information provide independent contributions to face identity discrimination (i.e., face identity = shape + surface) or whether one of these cues provides enough information to discriminate face identity at a high level already. In the latter case, there is redundancy of information: If one were able to quantify the contribution of each of these sources of information, a change of both sources of information combined would lead to a smaller effect than the sum of the two presented separately (Identity change < Surface+Shape change, i.e., a subadditive effect, for instance if there is saturation of the response when combining the two sources of information). Alternatively, the two sources of information may also contribute supra-additively if their combination is necessary to reach a threshold for efficient face identity discrimination (Identity change > Surface+Shape change). 

Here we addressed this issue of the relative and combined contributions of shape and surface information to the visual representation of face identity by using an approach that allows the capture of an implicit visual discrimination response between individual faces, for shape and surface information separately as well as for their combination. This approach consists in presenting an image at a constant periodic rate (frequency F) throughout a long sequence. This periodic visual stimulation elicits periodic responses—steady state visual evoked potentials (SSVEPs; Regan, 1966, 1989)—detectable in the human electroencephalogram (EEG). The main strength of this fast periodic visual stimulation (FPVS) approach is in providing a robust visual discrimination response that can be identified objectively (i.e., exactly at an experimentally defined frequency) and directly quantified. Moreover, this response can be obtained without contamination from decisional/motor processes, since the observers do not have to process the faces explicitly (Rossion & Boremanse, 2011; see Rossion, 2014, for a review). 

In a recent study, Liu-Shuang, Norcia, and Rossion (2014) used this approach in an oddball paradigm (for low-level visual stimulation, see Braddick, Wattam-Bell, & Atkinson, 1986; Heinrich, Mell, & Bach, 2009). Facial images were presented at a constant rate of 5.88 Hz (base frequency) for about 60 s. The same face was repeated for the entire stimulation sequence, but at a regular interval—every five faces—a different face, the oddball, was embedded in the sequence, thus resulting in an AAAABAAAACAAAAD… sequence. Since faces changed identity at a rate of 5.88/5 = 1.18 Hz (oddball frequency), the 1.18 Hz and its harmonics (i.e., exact integers of the stimulation frequency) could be used as a measure of the system's response to the changing identity (i.e., individual face discrimination). 

In the present study, the fast periodic oddball paradigm was used to compare the electrophysiological response to a change of identity carried out by surface or shape only, or by both shape and surface information combined. 

Materials and methods

Participants

Ten participants (one male, mean age = 22.17, SD = 1.87, range = 20–25) provided signed and informed consent and were paid for their participation in the experiment. They were all right-handed and reported normal or corrected-to-normal vision. None of the participants reported any history of psychiatric or neurological disorder, or any problems with face recognition. All of them demonstrated normal-range face matching performance (> 40/54 for all; average = 43.90) in the Benton Facial Recognition Test (Benton, Hamsher, Varney, & Spreen, 1983). None reported to have noticed the periodic change of facial identity (one out of five faces) during the EEG experiment. 

Stimuli

Using an established method of face manipulation (Tiddeman, Burt, & Perrett, 2001), we morphed the photographs of each of eight original individual faces (four male) with 10 other same-sex face photographs. For each facial image, 189 delineation points placed on salient features of the face (e.g., face contour, mouth, eyes, eyebrows, nose) were used. All images were standardized for size on interpupillary distance prior to morphing. External features such as ears and hair were not visible. Faces were morphed (Figure 1) along the shape dimension only (keeping surface constant), the surface dimension only (keeping shape constant), or both (identity), resulting in 248 stimuli used in the experiment (240 morphed stimuli—10 corresponding morphs for each of the eight individual faces for each of the three conditions—plus the eight individual faces). The extracted morphed images constituted 70% of the other photographs. This level of transformation is sufficient for faces to be discriminated as new identities, since morphs above 50% are consistently perceived as different individuals (see, e.g., Beale & Keil, 1995). The mean (± SD) height × width of the facial image was 277 ± 11 × 210 ± 9 pixels, corresponding roughly to 5.49° × 7.26°. All faces were placed against a gray background (204/255, 204/255, 204/255), resulting in images of 250 × 320 pixels. Mean luminance of the faces was equalized online during presentation. 

Figure 1

 

Stimuli and experimental design. (A) Image variations along both surface and shape combined (identity), surface only, and shape only. At every stimulation cycle, image size varied between 90% and 110% in a random order. (B) An illustration of the fast periodic oddball stimulation during EEG recording: A base face (A) is presented four times followed by an oddball face (B) during each trial. Thus, there are two embedded frequencies: Facial images are presented at a rate of 5.88 Hz (base frequency), and a face with a different identity is shown at a rate of 1.18 Hz (oddball frequency).

Behavioral pilot experiment

To examine the effects of the image manipulation, 13 female participants not tested in the EEG experiment (mean age = 22.62, SD = 2.36, range = 19–24) participated in a delayed match-to-sample, two-alternative forced-choice behavioral task. Forty morph pairs (20 male) were created by randomly assigning two morphs from the same condition (varying along surface only, shape only, or both dimensions) to a pair. Participants performed 240 trials (2 [both faces in a morph pair were presented as target faces] × 40 pairs × 3 conditions [varying along surface only, shape only, or both]) separated in six blocks. In each block, there were trials from the different conditions. Each trial started with a fixation cross for 250 ms followed by the target face presented for 170 ms, corresponding to the presentation duration of a face stimulus displayed at 5.88 Hz (∼170 ms) in the EEG study, and then a noise mask shown for 200 ms. After that, a blank screen appeared on the screen for 1000 ms, followed by a display showing the target face and a distractor face. The distractor face differed from the target face in surface only, shape only, or both dimensions. The two images were shown side by side until the participants responded. For half of the trials, the target face was on the right side. Trial order was randomized. Participants had to indicate which of the two faces was previously presented by clicking with the mouse over it. Data analysis showed that the manipulation of the facial dimensions affected accuracy rates, F(2, 24) = 27.64, p < 0.0001, η_p² = 0.70, and correct response times, F(2, 24) = 12.96, p < 0.0001, η_p² = 0.52. For both variables, individual discrimination was improved when both surface and shape combined varied (accuracy: M = 0.87, SEM = 0.02; RT: M = 1337 ms, SEM = 63) as compared to when surface only (accuracy: M = 0.72, SEM = 0.02 , p < 0.0001; RT: M = 1573 ms, SEM = 77, p = 0.007) or shape only (accuracy: M = 0.73, SEM = 0.01, p < 0.0001; RT: M = 1512 ms, SEM = 83, p = 0.003) varied. There was no difference in the performance of the task based only on cues of shape or surface (accuracy: p = 0.99; RT: p = 0.44). Thus, even though the stimulus duration was relatively brief here, these behavioral results are almost identical to the findings obtained in previous studies (e.g., Jiang et al., 2011; Russell et al., 2006), with an increase of about 15% for accuracy and a decrease of 15% correct RTs when both sources of information are combined, relative to the performance when only one source of information is available. 

Procedure

During the EEG recording, participants were seated in a dimly lit room with a 1-m viewing distance to the screen. The stimuli were presented on a CRT 17-in. (43-cm) monitor controlled by a computer. In every trial, one of the original individual faces was presented as the “base face” (A) and repeated at a fast rate (5.88 Hz, stimulus onset asynchrony of 170 ms) throughout the 72-s long trials. This rate was used because it provides a large signal-to-noise ratio (SNR) for face stimulation over the right occipitotemporal cortex (Alonso-Prieto, Van Belle, Liu-Shuang, Norcia, & Rossion, 2013). At fixed intervals of every four faces, the oddball face, randomly selected from the 10 corresponding morphed faces in one of the conditions, was presented (B, C, D, …), resulting in a trial sequence AAAABAAAACAAAA… (Figure 1B). Thus, individual faces varying along the shape, surface, or both dimensions combined appeared at a frequency of 5.88 Hz/5 = 1.18 Hz (i.e., every 850 ms). As a result, EEG amplitude at precisely this frequency (1.18 Hz—the oddball frequency) and its harmonics (i.e., 2.36 Hz, 3.53 Hz, …) were used as an index of the visual system's discrimination of individual faces along these dimensions. A custom software package running in Matlab was used to display the images at a rate of 5.88 Hz (base stimulation frequency) through sinusoidal contrast modulation (see, e.g., Rossion, Alonso-Prieto, Boremanse, Kuefner, & Van Belle, 2012; Rossion & Boremanse, 2011). To minimize low-level adaptation effects, the size of the images was randomly varied between 90% (4.94° × 6.53°) and 110% (6.04° × 7.98°) at every cycle (Figure 1), as in previous studies (e.g., Liu-Shuang et al., 2014; see Dzhelyova & Rossion, 2014, for a systematic evaluation of the effect of size variation on these EEG periodic responses). Each trial started with a fixation cross presented on the screen for a variable duration of 2 to 5 s, followed by 2 s of gradual fading in of the face, an 8-s baseline during which only the original face was presented, and then a 60-s stimulation sequence and 2 s of gradual fading out of the face. The whole experiment consisted of 24 trials—four trials with female and four trials with male images for each of the three conditions: change of both surface and shape information, change of surface information only, and change of shape information only (Figure 1A). The order of conditions was randomized across participants, who were instructed to pay attention to the faces and respond when they noticed a color change of the fixation cross. The fixation cross was presented in the center of the face stimuli, just below the eyes (Hsiao & Cottrell, 2008; Peterson & Eckstein, 2012), and briefly (200 ms) changed its color at random times from red to blue eight times within every trial. This orthogonal task was used to ensure that the participants were attentive. Behavioral data from one participant who did not press the response key for most of the trials were excluded. However, including this participant's EEG data did not change the results. Other than that specific participant, participants were accurate (M = 0.96, SD = 0.04, range = 0.90–1) and quick (M = 479 ms, SD = 72.19) at performing this orthogonal task, without differences across conditions in accuracy, F(2, 16) = 0.04, p = 0.96, or correct response times, F(2, 16) = 0.001, p = 0.99. 

EEG acquisition

EEG activity was recorded using a BioSemi ActiveTwo amplifier system with 128 silver/silver chloride electrodes. Two additional electrodes (a common mode sense active electrode and a driven right leg passive electrode) were used as reference and ground electrodes, respectively. Vertical eye movements were recorded with two electrodes positioned above and below the right eye. Horizontal eye movements were recorded with electrodes placed at the corner of each eye. EEG and electrooculogram recordings were sampled at 512 Hz. 

EEG analysis

All EEG processing steps were carried out using Letswave 5 (http://nocions.webnode.com/letswave; Mouraux & Iannetti, 2008) and Matlab 2012 (MathWorks, Natick, MA). EEG data were high-pass filtered at 0.1 Hz (Butterworth filter, fourth order). The continuously recorded EEG file was first cropped in 24 76-s segments (2 s before and 2 s after each stimulation sequence). Since the approach provides a high SNR and a response of interest confined to a small frequency bin, there was no need to reject EEG trials. Nevertheless, we performed two further operations to improve SNR. Blinks were selectively removed by means of an independent component analysis (Jung et al., 2000) using the runica algorithm (Bell & Sejnowski, 1995; Makeig, Bell, Jung, & Sejnowski, 1996) as implemented in EEGLAB. This algorithm outputs a square mixing matrix in which the number of components corresponds to the number of channels (in this case, 128 components). For each participant, only one component representing vertical eye movements was removed (for 7/10 participants this component was the first one accounting for most of the variance). Next, noisy or artifact-ridden channels were reestimated using linear interpolation of the three nearest spatially neighboring electrodes (no more than 5% of the electrodes, i.e., 3.4 electrodes on average across participants were interpolated). All data segments were rereferenced to a common average reference. 

Frequency-domain analysis

Preprocessed data segments were cropped down to an integer number of 1.18-Hz cycles beginning immediately after the baseline (i.e., 10 s after the initial stimulus onset) until approximately 67.83 s (57.83 s, 68 cycles, 29,610 points at 512 Hz in total). The eight trials or epochs of each condition were averaged in the time domain, separately for each condition (both surface and shape combined, surface only, or shape only) and each participant. This procedure maintains the complex phase of the response while canceling out EEG activity that is not phase locked with the stimulus. The obtained average waveforms were transformed in the frequency domain using a discrete Fourier transform (DFT; Frigo & Johnson, 1998) implemented in Matlab 2012, yielding a frequency spectrum of amplitude (μV) ranging from 0 to 512 Hz with a frequency resolution of 0.0172 Hz. The frequency spectrum was normalized by dividing by the number of data points, and amplitude values were extracted for all channels. Due to the nature of the presentation and the response characteristics (i.e., the exact frequency of the stimuli), no windowing of the DFT was used (Bach & Meigen, 1999). To ensure that differences between the three conditions were not due to differences in latency/phase jitter between conditions, an additional analysis was performed in which the DFT was applied to each epoch prior to averaging of the eight spectra by condition. This analysis provided the exact same statistical differences and conclusions (see supplemental analysis and Figure S1). 

To correct for noise level, the spectrum was baseline-corrected by subtracting the average voltage amplitude of the 20 surrounding bins (10 on each side, excluding the immediately adjacent and the two extreme bins) from the amplitude at each frequency bin (see, e.g., Nozaradan, Peretz, Missal, & Mouraux, 2011). This procedure slightly differs from our previous studies (e.g., Rossion et al., 2012), in which baseline correction was performed by estimating the SNR, i.e., dividing the signal by the amplitude at the neighboring frequency bins. While the SNR approach is useful in revealing low-amplitude responses (at high frequency rates, for instance), a sum of ratios may not be recommended for quantification of responses by combining a response spread over multiple harmonics. A baseline subtraction can be justified, since in the absence of a response at the periodic rate, the amplitude at a given frequency bin should be similar to the amplitude at the surrounding frequency bins. Hence, in the absence of a response, the noise-subtracted amplitude should tend towards zero. This procedure also has the advantage that the amplitude is expressed in microvolts (see Hu, Xiao, Zhang, Mouraux, & Iannetti, 2014). Nevertheless, for information and for comparison with our previous studies of individual face discrimination with FPVS, SNR values are also reported. SNR values were calculated as the ratio of the amplitude at each frequency and the average of the 20 surrounding bins (10 on each side, excluding the immediately adjacent bin and the most extreme one; Liu-Shuang et al., 2014; Rossion et al., 2012). Grand averages of the baseline-corrected amplitudes and SNRs for each condition were calculated. 

Based on the grand-averaged amplitude spectrum for each condition, z-scores for each electrode were estimated in order to assess the significance of the response. The z-scores were calculated in a similar way as the baseline correction, using the mean and standard deviation of the 20 frequency bins surrounding the frequency of interest. Given that the oddball response was distributed on several harmonics, a threshold of Z = 3.90, corresponding to a p value of 0.0001, was used to define the presence of a significant response at the frequencies of interest (i.e., base and oddball frequencies and their harmonics). To estimate the response for the oddball frequency, the fifth harmonic corresponding to the base stimulation frequency was excluded, and significant harmonics up to the second harmonic of the base frequency (11.76 Hz) were considered. There were no significant responses beyond the sixth harmonic (7.05 Hz). Topographical maps showed that for all conditions, the largest oddball response was over the right and left lateral occipital sites (Figure 2), in line with previous observations (Liu-Shuang et al., 2014). Thus, for further analysis of the oddball response, a region of interest (ROI) was defined by considering the five neighboring channels on the right (PO8, PO10, PO12, P8, P10) and the left (PO7, PO9, PO11, P7, P9) hemisphere. 

Figure 2

 

(A) Topographical maps displaying the regions with maximal activation for the base rate (top row) and oddball frequencies (bottom row). Individual scales (up to their own maximum value) are used. The face discrimination response is estimated as the summed baseline-corrected amplitudes for the oddball frequency (1.18 Hz) and its harmonics (2.36, 3.53, 4.70, and 7.05 Hz). The response to the base rate is estimated as the baseline-corrected amplitudes at 5.88 Hz. (B) Baseline-corrected amplitude (μV) spectrum for the different conditions (changes in surface only, shape only, or both) over right occipitotemporal ROI. The channel layout map on the right displays the electrodes in the right occipitotemporal region of interest.

For individual data analysis, fast Fourier transformed data for the channels included in the ROIs were pooled and the baseline-corrected amplitude values were extracted. The sum of baseline-corrected amplitudes at the oddball frequency and its harmonics (1.18, 2.36, 3.53, 4.70, and 7.05 Hz) was taken as the system's discrimination response of individual faces based on change of surface information only, shape only, or both (i.e., identity). 

Time-domain analysis

A complementary time-domain analysis was performed to visualize the shape of the periodic changes time-locked to the oddball stimuli. Prior to defining stimulus-locked epochs, a low-pass fast Fourier transform (FFT) filter of 30 Hz (width 5 Hz) was applied to the preprocessed and rereferenced time-domain segments of 58 s (see “EEG analysis” earlier). To remove the dominating contribution of the 5.88-Hz oscillation, an FFT notch filter with a 0.5-Hz bandwidth, selectively removing the base frequency (5.88 Hz) and its four harmonics, was applied. Stimulus-locked epochs started 1000 ms prior to the oddball stimulus onset and lasted for a total duration of 2000 ms. For each trial, 70 oddballs images were presented (70 × 850 ms = 59500 ms), resulting in 68 overlapping epochs for each trial (the first and last epochs were excluded, as a 1-s interval before or after the oddball was not available). Overall, 544 epochs per condition per participant were available, which were averaged separately for each condition and participant and then grand-averaged. 

Results

Base frequency

Grand-averaged baseline-corrected spectra averaged across all electrodes and conditions showed a clear response at the 5.88-Hz stimulation frequency (baseline-corrected amplitude ± SEM = 0.60 ± 0.03 μV; SNR ± SEM = 10.30 ± 0.34), indicating a successful synchronization to the visual stimulation. Overall, the response at 5.88 Hz had a medial occipital topography, peaking on either electrode Oz or POOz, depending on condition. The baseline-corrected amplitude values for the channel showing the maximal response for each condition were 1.96 μV (SNR = 26.50) for the change of both dimensions combined, 1.95 μV (SNR = 25.36) for the change of surface only, and 1.96 μV (SNR = 24.32) for the change of shape only. The same scalp topography was also previously reported at the base-rate response (Liu-Shuang et al., 2014). An ROI of five channels was defined around channel Oz. Amplitude values for the base rate over the medial occipital ROI can be found in Table 1. Scalp distributions of the base rate are shown in the top row of Figure 2A. An ANOVA with Condition (change of surface only, shape only, or both combined) as a within-subject factor over the medial occipital region revealed no significant differences in the baseline-corrected amplitudes among the three conditions, F(2, 18) = 0.04, p = 0.96, η_p² = 0.004. Thus, the response at the base frequency rate does not differ between the three conditions, which is expected in the absence of differences in terms of low-level visual input or attentional resources allocated to the different conditions. 

Table 1

 

Mean (± SEM) baseline-corrected amplitude values (μV) and SNR scores for the base and oddball frequency over the ROIs. Amplitude values for the oddball frequency are calculated as the sum of baseline-corrected amplitudes up to 7.05 Hz. SNR scores are provided as a reference. Notes: MO = medial occipital; OT = occipitotemporal.

Frequency (channels)	Base frequency		Oddball frequency
	MO sites		Left OT sites		Right OT sites
Condition/signal	μV	SNR	μV	SNR	μV	SNR
Change of both	1.45 (0.29)	19.94 (2.75)	0.78 (0.15)	2.60 (0.27)	1.23 (0.23)	3.14 (0.37)
Surface only	1.45 (0.31)	20.06 (3.35)	0.43 (0.08)	1.89 (0.18)	0.53 (0.09)	1.92 (0.15)
Shape only	1.46 (0.31)	19.90 (3.23)	0.19 (0.06)	1.30 (0.13)	0.46 (0.13)	1.82 (0.25)

Oddball frequency

The baseline-corrected amplitude spectrum can be seen in Figure 2B, and the individual values of the summed oddball frequency (1.18 Hz) and its four harmonics for all participants can be found in Figure 3. The response for the condition when both surface and shape combined changed is much larger than for the conditions when the surface only or shape only changed. The response to changes in surface only was dominant on the right but bilateral, while the response to changes of shape only and changes in both characteristics combined was more strongly right lateralized (Figure 2). Examining the individual data (Figure 3) confirmed these observations and suggested that despite interindividual differences, these observations were present at an individual level too. 

Figure 3

 

Individual and mean baseline-corrected amplitude values—sum of harmonics for the oddball frequency (1.18 Hz) and its first four harmonics (2.36, 3.53, 4.70, and 7.05 Hz)—per condition for the left and right occipitotemporal regions.

A repeated-measures ANOVA with the factors Condition (change of surface only, shape only, or both combined) and Hemisphere (left, right) on the baseline-corrected amplitudes confirmed these observations. There was a main effect of Condition, F(2, 18) = 30.86, p < 0.0001, η_p² = 0.77, since the response when both dimensions combined changed (M = 1.0, SEM = 0.18) was larger than the response for changes of surface only (M = 0.45, SEM = 0.08, p = 0.003) and of shape only (M = 0.28, SEM = 0.09, p < 0.0001). Additionally, the surface-only condition resulted in a significantly larger response than the shape-only condition (p = 0.047). A main effect of Hemisphere, F(1, 9) = 8.53, p = 0.017, η_p² = 0.49, indicated that the response over the right hemisphere (M = 0.72, SEM = 0.14) was larger than that over the left (M = 0.43, SEM = 0.09). The interaction between Condition and Hemisphere also reached significance, F(2, 18) = 3.45, p = 0.054, η_p² = 0.28. Post hoc tests indicated a significantly larger response over the right hemisphere for the condition when both surface and shape combined changed, t(9) = 2.63, p = 0.027, and when shape only changed, t(9) = 3.43, p = 0.008, but a bilateral activation when the surface only changed (p = 0.20, no significant difference between right and left hemispheres). The exact same analysis was performed on EEG data which were Fourier transformed before averaging, in order to ensure that these differences were not due to phase/latency jitter differences between conditions (e.g., more variability in the phase of the response across trials for shape than surface conditions). Although, as expected, the SNR was lower with this procedure, the results of this analysis were virtually identical to the analysis just described (see supplemental analysis and Figure S1), indicating that the differences observed were not due to phase/latency jitter differences between conditions. 

Time domain

An advantage in amplitude for the condition of changes in both facial characteristics combined can be observed also in the grand-averaged time-domain data (Figure 4). Following the presentation of the oddball face, a negative increase in the amplitude of the periodic response is observed for channel PO8 in the right occipitotemporal ROI, reaching its maximum (−1.67 μV) at about 264 ms. Complementing frequency data, there was also a negative amplitude increase for the surface-only condition (maximum peak amplitude = −0.91 μV), and the smallest negative increase in amplitude was evident for the shape-only condition (maximum peak amplitude = −0.6 μV). Furthermore, the peak for the surface-only condition (274 ms) appeared to be delayed with respect to both the shape-only condition (255 ms) and the combined condition (264 ms). 

Figure 4

 

Grand-averaged EEG waveform for channel PO8 during an oddball sequence. The time segment starts 1 s before and lasts 1 s after the appearance of an oddball stimulus (indicated with black dashed line, oddball cycle = 850 ms), without selectively filtering out the 5.88-Hz base rate (A) and with the base rate selectively filtered out (B) for the three conditions: change of both surface and shape combined (first row, in black), change of surface only (second row, in red), and change of shape only (third row, in blue). Gray dashed line indicates the time point at which the negative decrease in the amplitude reaches its peak for the surface-only condition.

Additive effects of shape and surface information

The change of both facial characteristics combined led to a significantly larger response than the change of surface only or shape only. To further explore this effect and understand the respective contributions of surface and shape information to face identity discrimination, we arithmetically summed the response obtained in the frequency domain to each dimension and compared this sum to the response obtained when both of them combined varied. Amplitude spectra of the sum of the two conditions over the right occipitotemporal region can be seen in Figure 5. It is clear that the response for the oddball frequency and its harmonics (2.36, 3.53, 4.70, and 7.05 Hz) is larger for the condition when both sources of information are combined than for the sum of the surface-only and shape-only conditions. Specifically, the arithmetical sum of the two conditions is smaller than the response obtained when both sources of information were combined over the right (arithmetical sum: p = 0.007, M = 0.94, SEM = 0.20; combined: M = 1.22, SEM = 0.23) and left (arithmetical sum: p = 0.021, M = 0.52, SEM = 0.14; combined: M = 0.77, SEM = 0.15) hemispheres (see also supplementary material). 

Figure 5

 

(A) Baseline-corrected amplitude spectrum (0.5 to 12 Hz) for change of both dimensions combined and the sum of surface- and shape-only conditions for the right occipitotemporal region of interest. (B) Topographical map demonstrating supra-additive effects: On the left, the sum of the shape shape-only and surface-only conditions; on the right, the difference between the two dimensions combined and the sum of shape and surface only (sum of harmonics for the first four oddball harmonics).

Importantly, this effect cannot be attributed to an increase of attention of general processing in the condition when both characteristics combined varied, as compared to the other two conditions, since behavioral responses in the orthogonal detection task did not differ across conditions. Most importantly, the response at the base frequency (5.88 Hz), which reflects general processes due to the alternation between a face and the gray background that are counted twice in the sum, is about two times larger when adding the response to each of the two facial dimensions as compared to the condition when both varied—i.e., no significant difference was found in comparing the arithmetical sum of the base-rate response when images varying along shape only and surface only were presented and the doubled response obtained when images varying along the two dimensions (shape and surface, i.e., identity) were presented, t(9) = 0.24, p = 0.82. 

Discussion

The aim of this study was to assess the respective and combined contributions of shape and surface information to face identity discrimination independent of decisional processes. To do so, we used a recently developed approach in which a robust and objective electrophysiological discrimination response is measured implicitly by means of FPVS in an oddball paradigm (Liu-Shuang et al., 2014; Rossion, 2014). We make three observations. First, we provide supporting evidence to behavioral studies that both shape and surface information contribute to face identity discrimination, although our EEG results suggest a relatively more important role of surface information. Second, our results suggest that shape and surface information diagnostic for identity dissociate partly in space and time. Specifically, shape perception is characterized by a right lateralized activation, while surface information is processed more bilaterally and slightly more slowly. Third and most important, shape and surface changes alone are associated with much weaker responses than when both sources of information are combined, suggesting a supra-additive contribution of the two sources of information to the representation of individual faces. 

Both shape and surface information contribute to individual face discrimination

The observation that both surface and shape dimensions are important for individual face discrimination is in line with previous behavioral findings (Caharel et al., 2009; Calder et al., 2001; Jiang et al., 2011; O'Toole et al., 1999; Russell et al., 2007; Russell et al., 2006). In the behavioral data collected here, there was no advantage in the contribution to individual discrimination for either shape or surface information, in line with the vast majority of studies using individual discrimination tasks with different face stimuli (e.g., Jiang et al., 2011; O'Toole et al., 1999; Russell et al., 2007; Russell et al., 2006; Troje & Bülthoff, 1996). Yet surface information provided a larger discrimination response in the FPVS EEG study. This discrepancy between behavioral and electrophysiological results suggests that surface information may be more relevant than shape information for individual face discrimination, but this effect could be masked by decisional factors during explicit behavioral tasks. 

The increased electrophysiological response to surface relative to shape information is interesting because previous studies have reported such an advantage only for familiar (Burton et al., 2005; Russell & Sinha, 2007) or learned (Itz, Schweinberger, Schulz, & Kaufmann, 2014) faces, while in the present study the faces presented were unfamiliar for the participants. The difference between familiar and unfamiliar faces is generally attributed to unfamiliar faces being processed by means of image-based cues (Bruce & Young, 1998). Here, our paradigm largely prevents the discrimination of faces based on image cues: Stimulus presentation is very short and rapidly masked by the subsequent face that is presented in the sequence, leaving no time for a detailed analysis of image-based cues. Moreover, the substantial change of size in each face presented is sufficient to remove an early temporal component which is observed when there is no change of size between faces (Dzhelyova & Rossion, 2014). As a matter of fact, the discrimination response was observed essentially over high-level visual regions of the occipitotemporal cortex rather than over low-level visual areas. Yet the use of familiar faces in the present paradigm may further increase the dominance of surface versus shape information. 

Nevertheless, the advantage of surface information found in the present experiment could also be due to several factors, and should be qualified. First, this effect could be specific to our image set or any of its properties (e.g., different level of difficulty to discriminate identity based on shape-only or surface-only cues), though this interpretation is less likely, as our behavioral results replicated those of previous studies. Second, as 3-D shape can be extracted from a very coarse representation defined by the global contour of the face and relative size of the head, it is possible that due to the image size variation introduced here between stimuli, the diagnosticity of shape information was slightly reduced in our study, which investigated the perception of size-invariant changes of shape. Indeed, when image size varies from cycle to cycle, the size of the head is also changing and thus cannot be consistently used as an informative cue. Third, the use of color rather than grayscale stimuli certainly adds to the diagnosticity of the individual faces in the surface reflectance condition (Russell et al., 2007). Fourth, when face identity changed at every fifth stimulation cycle in our oddball paradigm, there was no difference in head orientation (i.e., viewpoint), a factor that should increase the salience of shape information as opposed to surface information (e.g., Bruce & Young, 1998; Michel et al., 2013; Troje & Bülthoff, 1996). Future studies should address how these factors influence face discrimination of unfamiliar people based on surface and shape cues. 

Shape and surface information dissociate in space and time

Overall, the individual face discrimination response for face stimuli having both shape and surface information combined was prominent over the right occipitotemporal cortex, in line with previous observations made with another set of face stimuli (Liu-Shuang et al., 2014). This scalp topography is also in agreement with previous FPVS studies measuring individual face discrimination (Alonso-Prieto et al., 2013; Rossion et al., 2012; Rossion & Boremanse, 2011), and with the typical response to faces as observed in standard event-related potential (ERP) experiments following transient face stimulation (i.e., the N170; Bentin, Allison, Puce, Perez, & McCarthy, 1996; see Rossion & Jacques, 2011, for a review). Here, when faces were discriminated based on shape information alone, the relative dominance of the right hemisphere over the left hemisphere increased even further. This observation is consistent with neuroimaging observations indicating a larger release from fMRI adaptation in functionally defined face-sensitive areas in the right hemisphere for face shape changes (Jiang, Dricot, Blanz, Goebel, & Rossion, 2009) and with an increased amplitude of N170 component over the right hemisphere (Schulz, Kaufmann, Walther, & Schweinberger, 2012). On the other hand, the response to the facial surface information was more bilateral here, which complements previous electrophysiological and neuroimaging results reporting involvement not only of the right but also of the left hemisphere (Caharel et al., 2009; Jiang et al., 2009) in processing of surface information. 

Our time-domain analysis suggests a delay for detecting changes of surface versus shape or both sources of information combined. This observation should be interpreted cautiously, because it may be difficult to provide an unambiguous estimate of time delay between the stimulus onset and the response due to the fast stimulation rate, and the latency of the deflections was measured on grand-averaged data only. Here, the appearance of a different face every 850 ms (1.18 Hz) was followed by a large negative deflection starting at about 200 ms and peaking at about 260 ms. Considering that with a sinusoidal contrast stimulation, a face is fully revealed (i.e., 100% contrast) only at the half stimulation cycle (i.e., 85 ms for a full cycle duration of 170 ms at 5.88 Hz) and thus will be hardly visible for the first 25–30 ms, the timing of this response suggests that the oddball face is discriminated from the base face at about 170 ms. This timing is compatible with findings obtained in standard ERP studies of individual face repetition, showing repetition effects on the peak of the N170 component (see, e.g., Caharel et al., 2009; Jacques, d'Arripe, & Rossion, 2007). Moreover, the time delay observed for processing surface information is entirely in line with previous observations of delayed diagnosticity of surface-based information to individual face discrimination as observed with transient ERP stimulation (Caharel et al., 2009; Itz et al., 2014). Taken together, the results suggest that the neural mechanisms contributing to shape and surface processing for face discrimination are temporally and spatially overlapping but are also partially dissociated. 

The supra-additive contribution of shape and surface information

Changing both surface and shape information combined produced a stronger response than any of these facial characteristics alone. At first glance, this finding complements behavioral observations that individual performance on face identity discrimination is improved when both aspects vary (e.g., Jiang et al., 2011; O'Toole et al., 1999; Russell et al., 2007; Russell et al., 2006). However, critically—and contrary to behavioral studies showing relatively little improvement of face recognition when shape and surface information are combined—we were able here with FPVS to quantify the EEG discrimination response. Thanks to this advantage, we observed that the sum of the response to each of these dimensions is weaker than the response obtained when both dimensions were combined. Although attention allocation to face-related characteristics can affect the response elicited by periodic stimulation in EEG (see, e.g., Hajcak, MacNamara, Foti, Ferri, & Keil, 2013; Keil, Moratti, Sabatinelli, Bradley, & Lang, 2005), this supra-additive effect of changes in both facial features cannot be explained by differences in attention, as performance in the orthogonal task and base-rate response measured over the medial occipital sites did not differ across conditions. Rather, these novel observations suggest that although face identity perception depends on both dimensions—facial shape and surface information—their contribution in identifying people involves mechanisms that cannot be explained by the summed brain response to the same surface and shape stimuli when presented individually. 

One possibility is that shape and surface cues are processed by completely independent populations of neurons (i.e., brain regions), and when they are presented simultaneously, their independent contribution is facilitated and reaches a threshold for individualization of faces in other brain regions. However, in this case, the timing and the spatial distribution of the neural response would have been very different for shape and surface characteristics, a claim not supported by our observations of the predominant response over the right occipital temporal site and the small temporal differences in the discrimination response based on cues derived from variations in shape only, surface only, or both combined. Admittedly, due to the limited spatial resolution of scalp-recorded EEG, we cannot provide a definite answer about the localization of the sources of face discrimination in the different conditions. Nevertheless, measuring release from adaptation to individual face repetition in fMRI has not revealed large differences in localization between shape- and surface-based diagnostic cues (Jiang et al., 2009). Interestingly, in that study, face-selective regions slightly decreased their release from adaptation when stimuli varying along both dimensions were presented, as compared to shape changes only (Jiang et al., 2009). However, these observations were made during an explicit face discrimination task, which was easier when both cues were combined than when shape-only cues were present. Since fMRI accumulates brain activity over many seconds, these effects could also be due to attentional or decisional factors, and are difficult to attribute to implicit changes in the visual representation of individual faces. 

Another intriguing possibility is that an overlapping population of neurons, responding to each of the dimensions, increases its firing rates when presented with stimuli varying along both dimensions. In other words, cells tuned to respond to identity changes fire with reduced magnitude and/or delayed latency (subthreshold level) when only one of the dimensions is changing. Such coding properties of the neurons would be comparable with inferotemporal cells in primates which fire with systematically graded response to facial stimuli and decrease (Edwards, Xiao, Keysers, Foldiak, & Perrett, 2003; Leopold, Bondar, & Giese, 2006; M. Young & Yamane, 1992) or delay their activity for suboptimal exemplars (Perrett, Oram, & Ashbridge, 1998). Furthermore, the neuronal subthreshold activity to stimuli varying only in surface or shape information will be more difficult to detect and will thus be less able to guide perceptual decision responses. This can partly account for the delayed and less accurate discrimination based solely on one of the two facial characteristics. 

Conclusion

In conclusion, our observations suggest that although facial shape and surface information have complementary effects on individual face discrimination, the neural response to changes of both of these diagnostic sources of information combined is much stronger and cannot be explained by any of the individual sources of information or by their linear addition. These supra-additive effects of facial shape and surface information for individual face discrimination suggest that the two kinds of diagnostic cues need to be combined to provide a full representation of face identity. 

Acknowledgments

This work was supported by a grant from the European Research Council (facessvep 284025) to BR and a postdoctoral UCL/Marie-Curie fellowship to MD. The authors would like to thank Benvenuto Jacob for the stimulation software Sinstim. 

Commercial relationships: none. 

Corresponding author: Milena Dzhelyova. 

Email: [e-mail]. 

Address: University of Louvain, Psychological Sciences Research Institute (IPSY), Louvain-la-Neuve, Belgium.