Deconstructing the early visual electrocortical responses to face and house stimuli

Abstract

Abstract

Abstract:

Abstract  The initial timing of face-specific effects in event-related potentials (ERPs) is a point of contention in face-processing research. The occasional reports of a larger P100 to face stimuli compared to other image categories is often attributed to differences in low-level stimulus characteristics. Separating the P100 from the classic N170 effect has not been done except by adjusting stimuli to control for low-level stimulus characteristics, which yields robust face effects only after 130 ms. In the present study we use a stimulus set with minimal controls for low-level characteristics. This produces significantly larger (p < 0.01) P100 and N170 amplitudes for images of faces compared to houses in a group effect. However, with independent component analysis (ICA), we demonstrate that (a) the P100 scalp effect stems from a neural network that is indeed independent of that producing the N170 effect, despite the N170 component being active at the time of the P100; (b) compared to the N170 effect, the P100 effect is less reliable even when it is present because of intersubject variability; and (c) some individuals show a component with a larger response to houses over faces at the time of the P100 that is undetectable at the scalp because the activation of larger spatiotemporally overlapping activity cancels its field projection. Thus, with ICA, we are able to account for the general finding in the literature of a consistent N170 face effect and a less reliable P100 face effect at the level of anatomically independent electrocortical processes.

Introduction

Due to their high temporal resolution, scalp-recorded event-related potentials (ERPs) are well suited for studying the rapid cascade of cortical activation that is associated with time-locked psychological processing. However, such ERPs can present interpretation difficulties because of the superposition of the electrocortical field potentials captured at the scalp. This problem is particularly pronounced when it is expected that relatively independent and functionally distinct cell networks project spatiotemporally overlapping field potentials. A clear example of this limitation is in the controversy concerning the initial ERP effects related to face processing, which we address in this paper. 

The face effect critical timing controversy

Face-specific processing in the brain has been widely studied through the N170 component of the ERP, which responds differently to faces compared to a wide variety of other stimulus categories. The N170 spans from about 130 ms to 200 ms post stimulus onset (Bötzel, Schulze, & Stodieck, 1995; Eimer, 2000; Itier & Taylor, 2002; Rousselet, Husk, Bennett, & Sekuler, 2008), the time required, it is argued, to extract stimulus information necessary for categorizing a stimulus as a face versus other categories of objects. Although the robust face effect of the N170 dominates the face-processing literature, a less reliable P100 face-specific effect challenges the interpretation that the N170 is the first marker of face-specific high level perceptual processing in the visual system (Debruille, Guillem, & Renault, 1998; George, Jemel, Fiori, & Renault, 1997; Halit, de Haan, & Johnson, 2000; Itier & Taylor, 2004; Mouchetant-Rostaing, Giard, Bentin, Aguera, & Pernier, 2000). In this paper, we address the question of the anatomical independence of cortical processes that account for scalp effects during the period of the P100 and N170 ERP components. 

Factors contributing to the controversy

Rousselet et al. (2008) outlined several factors that may account for the disagreement in terms of identifying the initial face-specific ERP effect. First, the discrepancy reported in the literature may be due to low-level differences between stimuli representing faces versus other categories (e.g., differences in color, amplitude spectrum, etc.). A second factor relates to the lack of consistent robustness in statistical designs. Rousselet et al. (2008) dealt with these issues by using a well-controlled stimulus set and sophisticated statistical treatment of the data. A third factor is the spatial and temporal overlapping of multiple electrocortical generators projecting to posterior scalp regions within 200 ms of the stimulus onset. The current investigation explores the same face-effect timing issue using independent component analysis (ICA) to unmix the field projections that constitute the P100 and N170 scalp ERP complex and examining each of the resulting constituent processes separately. 

P100 and N170 ERP components

The P100 is a positive deflection maximal over bilateral medial occipital scalp regions with a peak latency of approximately 100 ms. The P100 shows stimulus category effects that are generally associated with low-level stimulus characteristics (Regan, 1989). However, top-down attention processes have also been shown to affect P100 amplitude (Van Voorhis & Hillyard, 1977) and easily confound the interpretation of early ERP differences in the processing of face stimuli (Johnson & Olshausen, 2003). 

The N170 is a negative deflection immediately following the P100 peaking at about 170 ms after stimulus onset, and its maximum amplitude is distributed over bilateral occipito-temporal regions. The N170 is best known for its larger response to faces than to any other stimulus category (Bentin, McCarthy, Perez, Puce, & Allison, 1996), and more generally it is associated with early automatic high-level integrative processing associated with expertise (Bentin & Golland, 2002; Rossion, Collins, Goffaux, & Curran, 2007). Although the N170 is mostly studied in terms of its relationship to high-level processing, research has also documented low-level effects on the N170 (Goffaux, Gauthier, & Rossion, 2003; Zheng, Mondloch, Nishimura, Vida, & Segalowitz, 2011), again adding to interpretive difficulties. The inversion effect is a robust ERP finding of a larger and later N170 deflection for inverted versus upright faces (Bentin et al., 1996). This inversion effect is characteristic of face processing and similar effects of inverting nonface stimuli are rarely observed. The inversion effect is not only face-category specific, but it also isolates the effect as a high-level process because this manipulation almost perfectly controls for low-level stimulus characteristics. 

Low-level and high-level effects

Although it is likely that the P100 and N170 scalp ERP components are constituted from shared cortical generators (that have spatiotemporally overlapping field projections), Rossion and Caharel (2011) reported that the P100 and N170 effects have functional independence during face and house processing. They included a phase-scrambled version of each stimulus in a face and house stimulus set (with minimal controls for low-level stimulus characteristics) and found that the P100 differences between face and house stimulus categories persisted for phase-scrambled images; therefore, they concluded that the P100 effects reflected sensitivity to low-level stimulus differences between categories. The N170 category effect was absent in the phase-scrambled contrast. These results strongly support the interpretation that the P100 is associated with low-level stimulus characteristics and that the N170 is associated with high-level integrative perception that is largely face specific. 

Control of low-level stimulus characteristics

One can devise specialized stimulus sets that are controlled with respect to low-level differences between image categories, such as comparing face stimuli to noise images containing matched luminance, phase content, and contrast (Allison, Puce, Spencer, & McCarthy, 1999). Using such a stimulus set of equated faces and houses, Rousselet, Husk, Bennett, and Sekuler (2007, 2008) determined that the face effect remains robust only in the period of 130–200 ms. 

However, there are some reasons why highly controlled stimulus characteristics can limit the interpretation of the N170 face effect. First, face-image modification may reduce the face effect by dulling the brain's response to low-level information that is normally used to trigger face-specific processing. Further, controlling for low-level stimulus characteristics across image categories can introduce systematic confounds related to the brain's greater sensitivity to face manipulations than other stimulus categories. For example, variations in spatial frequency content have been shown to affect face recognition (Bachmann, 1991; Costen, Parker, & Craw, 1994; Nasanen, 1999), and face recognition is more vulnerable to spatial frequency overlap than object recognition (Collin, Liu, Troje, McMullen, & Chaudhuri, 2004). Second, the face effect may not be an entirely high-level phenomenon, but may involve a special relationship between high-level integrative processes and early low-level sensory processing. For these reasons, a desirable alternative to stimulus control is needed and can be achieved by examining the underlying electrocortical processes that constitute the spatiotemporally overlapping ERP components in the context of relatively unaltered and ecologically valid face perception. 

Independence of ERP components (underlying brain dynamics)

Advanced electroencephalogram (EEG) analysis using unmixing algorithms such as ICA reveal that ERP components are composed of the sum of temporally dynamic underlying cortical constituent processes (Makeig et al., 1999). ICA is a method for blindly unmixing a set of mixed signals (Bell & Sejnowski, 1995) and has been used to describe the relative contribution of constituent processes to many ERP scalp components, such as isolating cortical processes related to the N2 (Hu, Mouraux, Hu, & Iannetti, 2010), decomposing the N1 into constituent processes during a selective spatial attention task (Makeig et al., 1999), separating independent processes underlying the novelty P3 and P3b (Debener, Makeig, Delorme, & Engel, 2005), and isolating response-related brain responses during a visual selective attention experiment (Jung et al., 2001). Further, De Vos, Thorne, Yovel, and Debener (2012) reported that single trial classification of face stimuli was improved over scalp EEG voltages by applying an ICA decomposition. Additionally they were able to show that the face-sensitive component was responsive to general visual processing rather than reflecting a network that was exclusively involved in face processing. Thus, although the specific electrocortical phenomena that benefit from ICA are diverse, the nature of the information gained by each instance is similar in that (a) independent electrocortical processes are isolated that are otherwise mixed when assessed at a scalp recording site and (b) an improved signal-to-noise ratio at the level of the single trial increases the power of statistical tests. 

Goals of the current study

The first goal was to describe the P100 and N170 at the level of their constituent electrocortical processes unmixed from the scalp data using ICA during the presentation of face versus house stimuli. This would allow for a description of the anatomical independence of ERP effects during the period of the P100 and N170 complex. Once the cortical constituents are isolated, we could examine timing and robustness effects as reported by Rousselet et al. (2008) in the context of a stimulus set that is minimally controlled with respect to low-level characteristics. 

The second goal was to describe the likelihood of finding group and individual-subject effects within the first 200 ms post-stimulus onset at the scalp and within independent components (ICs), and furthermore, to account for observed differences in results based on group versus individual-subject analyses. 

Methods

Participants

Ten healthy adult volunteers ranging in age from 22 to 37 took part in this study. All individuals had self-reported normal or corrected to normal vision; five were female, nine were right handed. 

Stimuli

Twenty-six images were used: 16 faces, eight houses, and two half-circle checkerboards. All stimuli were matched for outer shape and average pixel luminance. All face and house stimuli were front-view gray-scale photographs with an oval crop and presented at a size of 8 cm wide by 10 cm high (15.2° × 19° of visual angle). The 16 face stimuli were made up of four identities expressing two emotions (angry and fearful) and were either upright or inverted in orientation. These emotional faces were used for reasons of convenience and interest in this specific stimulus set, having been used before in our lab (Munro et al., 2007) and were adapted from Gur et al. (2002). The eight house stimuli were made up of four identities that were either upright or inverted. Sample stimuli are depicted in Figure 1A. 

Figure 1

 

(A) Examples of stimuli used in the face/house/checkerboard task. Images were presented for 250 ms and included a random interstimulus interval (ISI) ranging from 500 ms to 750 ms. Participants were asked to respond as quickly and accurately as possible by pressing a left or right button in response to the infrequent left or right half-circle checkerboards. (B) Example stimuli used as animation for trials in the fading blocks of the task. In these blocks stimuli faded into one another without a blank screen ISI. During these blocks participants were asked to respond with their dominant hand to the appearance of an infrequent full-circle checkerboard stimulus.

Figure 2

 

Global field amplitude (GFA) ERP overlays with 99% confidence intervals of the bootstrapped difference waves. The four subplots present specific GFA ERP category contrasts with the various combinations of face versus house, upright and inverted, as indicated, with minuend and subtrahend indicated in thick and narrow black lines. The gray area represents the 99% confidence interval of the 1,000 bootstrapped difference waves empirically derived. The abscissa is red where the confidence interval does not include zero and therefore the waveforms are considered to be significantly different from one another.

Design and procedure

Subjects sat alone in a dark room (the only source of light was from the experimenter's window that was to the right of the participant and about eight feet away; the experimenter's computer monitors were constantly on, generating the light that was passing into the recording room) with the display viewing distance held at a constant 30 cm using a chin rest. Stimuli were presented using E-Prime 1.2 on a Dell CRT monitor with 1024 × 768 screen resolution and 60 Hz refresh. All stimuli were presented on a black background in the center of the screen behind a small gray fixation “+” that was displayed constantly for the duration of the task. Participants made their responses using both hands on a four-key pad. 

The task procedure consisted of eight blocks of trials. Trial blocks were separated by a short break (about 30 seconds) that was terminated by the experimenter. Each block alternated between two types of trial procedures and the initial block type was counterbalanced across participants. 

In one block type the stimuli were presented for 250 ms with a variable interstimulus interval that ranged from 500 ms to 750 ms. The image categories presented in this block were upright and inverted faces, upright and inverted houses, as well a left and a right checkerboard. For blocks of this type, the participants were instructed to respond as quickly and as accurately as possible to left and right checkerboards by pressing the left-most and right-most keys, respectively. Participants were allowed the entire trial duration to give their response. 

In the other block type stimuli were presented as an animation of images that contained a range of spatial frequency filters. Each image was filtered at various low-pass intervals creating seven levels of a spatially filtered gradient that, presented in rapid succession, resembled an animation where each stimulus image appeared and then disappeared from a mean luminance blur (See Figure 1B for an example filter series). The average pixel luminance was held constant over the entire duration of these blocks where each stimulus presentation appeared as a gradual increase and then decrease of spatial frequencies contrast. Although the data from these blocks were not included in any of the statistical tests, these periods of the recordings were included in the ICA decomposition. 

There were a total of 300 trials per block consisting of 50 of each of the stimulus types: upright faces, inverted faces, upright houses, inverted houses, left checkerboards, and right checkerboards. Identity was varied randomly for both face and house stimuli on every trial, as was emotion for face stimuli. 

EEG recording

EEG data were recorded from a BioSemi ActiveTwo system with 128 scalp channels. The analog signal was digitized according to the BioSemi zero-reference principle (the voltage at each site is quantified relative to the common mode sense and driven right leg loop) at 1024 Hz and online low-pass filtered at 512 Hz. As a substitute for impedance measures quantifying signal quality, electrode offsets were maintained below 50 μV. 

EEG preprocessing

All preprocessing was performed in Matlab R2010b using functions from the open source toolbox EEGLAB version 11 (Delorme & Makeig, 2004) as well as in-house functions developed for automated artifact removal on the Shared Hierarchical Academic Research Computing Network (SHARCNet). 

Following the manual removal of break intervals from the recordings, each participant's data file was submitted to an automated preprocessing script. The EEG signals were filtered from 2 Hz to 30 Hz and rereferenced to the average electrode for the preprocessing steps. The continuous data were then windowed into 50% overlapping 600 ms windows. 

A summary of the preprocessing data rejection outcomes is presented in Table 1. The median, minimum, and maximum rejection quantities are presented in order of processing from top to bottom. Each of the artifact detection procedures is described in the following paragraphs. 

Table 1

 

EEG data rejection summary.

Measure	Median	Minimum	Maximum
Low neighbor correlation channels	10	6	15
Bridged channels	12	2	28
Remaining channels for ICA	108.5	95	116
Total in task time (minutes)	32.3	32.2	32.5
Scalp data rejection time (minutes)	1.85	1.5	2.6
ICA activation rejection time (minutes)	0.394	0.09	2.05
Remaining time for ICA (minutes)	30.1	28.7	30.6
Number of cortical ICs	15	8	26
Number of upright face trials	193	159	198
Number of inverted face trials	195.5	166	200
Number of upright house trials	193	156	200
Number of inverted house trials	194	163	199

In order to identify poor recording channels to be removed before ICA, the maximum correlation coefficient r of each channel with its three nearest neighbors was calculated, resulting in n channel correlations for each time window. The 99% confidence intervals of r values were then calculated for each time window. In each time window, channels for which r values were outside the 99% confidence interval were flagged. If a channel was flagged in 10% or more of the time windows it was considered contaminated with noise and rejected. The summary of these rejection criteria is presented in Line 1 of Table 1. 

In order to identify bridged electrodes, the mean and standard deviation of the maximum r values for each channel was calculated across time windows. Each channel's mean r value was then divided by its standard deviation to produce a composite score that captured a bridged electrode's high and relatively invariable correlation to its bridged neighbor. Channels for which the composite score was greater than eight standard deviations away from the 25% trimmed mean of the composite scores across channels were rejected. The summary of these rejection criteria is presented in Line 2 of Table 1. The total number of channels remaining following the rejection criteria is listed in Line 3 of Table 1. 

The total amount of time for the in task EEG recording is listed in Line 4 of Table 1. After removing all of the bad channels, the maximum nearest neighbor correlation coefficient was recalculated for each time window. This time the 99% confidence interval was calculated for each channel across time windows. If the r value of a given time window fell lower than the 99% confidence interval for a particular channel, it was flagged. If 10% or more of the channels were flagged in a given time window this window was rejected. The summary of this rejection criterion is listed in Line 5 of Table 1. Following this rejection of time windows, the data were concatenated back to continuous data with discontinuities marked as boundaries. Any continuous period of data that was shorter than two seconds was also removed. The remaining continuous time intervals were then de-trended prior to submitting the data to the extended Infomax ICA with an N-1 channels PCA reduction. 

Following the first ICA decomposition, an ICA time-pruning procedure was used, where the ICA time-course of activations were used to identify further time periods for rejection. The standard deviation of activation was calculated for each independent component (IC) at each time window. The 99% confidence interval of standard deviation scores were then calculated for each IC across time windows. The identical flagging and rejecting criteria as described for the channel data above were applied to these IC-based confidence intervals and the remaining data were resubmitted to another extended Infomax ICA with an N-1 channels PCA reduction. The summary of these rejection criteria is presented in Line 6 of Table 1, and the total task time remaining after the artifact rejection procedure is listed in Line 7. This second ICA tends to produce a more stable decomposition, as it has the benefit of performing its training on data that do not include periods of time that are known to produce unusually high levels of activation in at least 10% of the components (based on the outcome of the first ICA decomposition). 

Following the second ICA, dipoles were fit to each IC. ICs for which the dipole fit explained less than 85% of the weight variance were marked for rejection. Manual examination of the ICA decomposition was used to reject biological artifact components whose dipoles explained 85% or more of the weight variance (e.g., eye blinks, lateral eye movements, electrocardiogram [ECG] and electromyogram [EMG]) and to correct the criteria if a rejected IC could be described by symmetrical bilateral dipoles; for example, sometimes the P100 or N170 is captured by a bilateral IC and sometimes two ICs are needed (as is indicated in Figure 3). The IC weights were then applied back to the initial continuous DC-to-30 Hz filtered data with periods removed that were flagged during the ICA time-pruning procedure. The summary of the number of remaining cortical ICs is listed in Line 8 of Table 1. 

Figure 3

 

Topopgraphical maps of single subjects' independent components (ICs) that contribute to each IC cluster and the ERP envelope plot for all face and house stimuli combined. The upper portion of this figure shows the topographical projections of all the independent components from each individual that contribute to each of the three IC group clusters identified as contributing to the ERP effects in the first 200 ms following stimulus onset. Red rectangle: Topographical maps for the ICs that contribute to the icP1a cluster, and the topographical projection of the icP1a cluster. Green rectangle: Topographical maps associated with the icP1b cluster. Blue rectangle: Topographical maps for the icN1 cluster. The polarity of the topographical maps corresponds to peak projections during the interval of the P100 in the ERP for all the stimulus conditions combined. The lower portion of this figure shows the envelope maximum positive and negative voltage values for each time point at the scalp. Each of the paired colored lines indicate the voltage range of the projections of each of the three IC clusters to the scalp (icP1a: red; icP1b: green; icN1: blue). The gray area indicates the range of the voltage projections of all three IC clusters projected back to the scalp together indicating that these three clusters account for almost all the variance.

Segmentation

Stimulus-locked ERPs were calculated relative to the onset of all remaining nontarget trials. Trials were pooled into four categories representing all nontarget stimulus categories, namely, face upright, face inverted, house upright, and house inverted. The period of −200 ms to 0 ms relative to stimulus onset was used as baseline. The summary of the number of trials remaining for the analysis by stimulus condition is listed in Table 1 on Lines 9 to 12. 

IC identification

A measure of spatial variance was used to select the independent components that captured the ERP effects at the time of the P100 and N170. For both upright and inverted face-minus-house comparisons, the difference waves were calculated for each scalp site. The period of the P100 and N170 effects were then manually selected for each individual. The unique percentage of variance accounted for during the P100 and N170 effects were calculated for each IC. The percentage of variance accounted for was measured as the mean of the spatial variance of the scalp difference during the period of interest minus the residual scalp variance of the unique IC contributions (the variance at the scalp given all other ICs combined), divided by the total scalp variance. ICs were then ranked and entered into a cumulative percentage of a “variance accounted for” calculation. Starting with the largest, contributors were added until we accounted for 85% of the scalp effect. 

Only components with occipital or posterior-temporal peak projections were included. One class of component that had a peak projection over occipital scalp regions was assigned to the set of residual components (rather than being classified as a cluster of its own, such as icP1a, icP1b, and icN1). This class of component had a clear left-right occipital asymmetry (with opposing valence) and was differentially active at the time of the P100 to the retinotopic contrast of the left and right visual field checkerboards. Although this class of component does not consistently contribute to the face-related contrasts that are the focus of this paper, its presence is worth mentioning here because of the high spatial and temporal overlap of its ERP with the effects of interest. 

In addition, components were included in subsequent analysis if they contained clear and consistent ERP effects during the periods of interest despite being underrepresented in the spatial variance measure because of having very focused field projections. 

Bootstrapped group statistic

The statistical treatment employed in this study follows closely the procedures introduced by Wilcox (2005) and used by Rousselet et al. (2008). Categorical differences between ERPs were assessed for the group using a percentile bootstrap technique averaged across individuals. In this study, 20% trimmed means were used in the place of the full means across all artifact-free trials for a given stimulus category. The trimmed mean was chosen for the analyses in this study based on its benefit in representing electrophysiological data which can often be contaminated with extreme values (see Rousselet et al., 2008, and Wilcox, 2005, for a detailed description of the benefits of the 20% trimmed mean). 

The bootstrapping methods described in this section rely on surrogate sets of data. A surrogate set is a random sample of data points obtained with replacement from the full sample of data points available. For example, a surrogate set of trials for a given stimulus condition containing 100 epochs is a set of 100 data trials selected randomly with replacement from the full set of 100 trials. 

In order to estimate the alternative hypothesis distribution for a given data set, 1000 surrogate-set trimmed mean ERPs were obtained from the segments of each category and subject. Averaging surrogate ERPs across subjects produced 1000 surrogate grand average ERPs per stimulus condition. From these, estimates of the alternate hypothesis distributions for each contrast were then obtained by forming difference waveforms based on subtracting one condition from another. The four specific contrasts examined in this procedure were upright face minus upright house, inverted face minus inverted house, inverted face minus upright face, and inverted house minus upright house. The 99% confidence intervals at each time point in the surrogate-set grand ERP contrasts for the alternate hypothesis were calculated and periods where the interval did not include zero were considered significant. 

Differential activation robustness

Considering that the critical timing controversy regarding face-specific effects in ERP research revolves around an unreliable differentiation between faces and other stimulus categories during the time of the P100, it is important to the goals of this study to not only examine the early effects if they exist but also to measure the robustness of the early ERP effects. Further, an important distinction to make is whether the effect is frail in each individual or whether the effect is robust but only in some individuals. Both of these sources of instability in group effects could explain the current state of the literature describing face specific P100 effects. 

The measure of differential activation robustness (DAR) simply adds one layer of surrogate set sampling to the bootstrapping procedure described above. Specifically, a random sample with replacement is obtained before performing a bootstrap test in a Monte Carlo procedure where the resampling and testing is repeated 100 times. At the end of the 100 Monte Carlo experiments the Boolean values for each data point (0 = fail to reject null hypothesis, 1 = reject null hypothesis) are summed to create a measure of the likelihood of finding an effect at each data point. 

Results

Group bootstrap test on the scalp global field average

The global field amplitude (GFA = standard deviation across electrodes; Rousselet et al., 2008) results are illustrated in Figure 2. Each plot contains the respective GFA ERP overlay (two black lines, the bold line being the first category entered into the subtraction to create the difference waves), the 99% confidence intervals of the differences (gray area), and the periods of significant differences between the two conditions (red line for each time sample that is significant). 

The GFA waveform significantly differentiates the face and house stimulus categories (for both upright and inverted orientations) over the period of about 150 ms to 200 ms, closely matching the effects found by Rousselet et al. (2008) and for periods following this as well. The inversion effect is also significant for face stimuli during this period taking the form of a larger and later deflection for inverted faces over upright faces during the period of about 170 ms to 220 ms. The latency shift of this contrast during the P100 descent and N170 ascent results in short periods of significance surrounding 150 ms where the response to inverted faces trails the response to upright faces. A latency shift during the ascent of the P100 also resulted in a period of significant differences starting before 100 ms. Brief periods of significance were also present in the house-inversion contrast at 100 ms and 200 ms. The confidence intervals of the differences for this comparison remain far closer to the null hypothesis than the face-related comparisons. 

In addition to the effects found during the period of the N170 ERP component for face versus house comparisons, GFA effects were also present during the period of the P100 over approximately 100 ms to 150 ms following stimulus onset. Brief face inversion effects were also found in the GFA signal during the ascending and descending slopes of the P100. 

ERP IC characteristics

Three types of ICs accounted for the ERP face-effects in the first 200 ms following stimulus onset. The topographical projections of these three IC clusters and each individual's contributing ICs are depicted in the upper portion of Figure 3. IC cluster contributions to the grand average scalp ERP envelope (for all stimulus conditions combined) are plotted at the bottom of Figure 3. The thick lines in this envelope plot represent the voltage range of all the scalp channels at every time point. Each of the three colored envelopes represents the range of voltage values projected to the scalp uniquely by the IC clusters. The gray area in this plot represents the range of voltages projected to the scalp by all three IC clusters taken together. The icP1a has a broad medial occipital projection (outlined in red), icP1b has a focused medial occipital pole projection (outlined in green), and the icN1 has a bilateral occipito-temporal projection (outlined in blue). 

The IC cluster contributions to each of the four category grand average scalp ERPs are illustrated in Figure 4. The color configuration of these envelope plots corresponds to the envelope plot at the bottom of Figure 3. 

Figure 4

 

ERP envelope plots for each stimulus condition: faces (top quadrants) and houses (bottom quadrants), upright (left quadrants) and inverted stimuli (right quadrants). Black lines present the maximum positive and negative voltage values of the scalp data at each time point. Each colored envelope presents the voltage range of each of the three IC clusters: icP1a in red, icP1b in green, and icN1 in blue. The gray area presents the voltage range of all three IC clusters projected together.

The icN1 contains ICs from each individual that were categorized based on their largest relative contribution to negative voltage over right-dominant bilateral occipito-temporal scalp regions during the time of the N170 ERP component. Although these components were classified based on their contribution to the total scalp ERP voltage range during the N170 time period, this IC cluster is also active (with a positive polarity at posterior sites) during the period of the P100 and P200. 

The icP1a contains ICs from each individual that were categorized based on their large contribution of positive polarity activation over a broad medial occipital area during the P100 and P200 time periods. The icP1a is illustrated with red lines in all envelope plots. 

One other IC class was identified in nine subjects and had a distinctive occipital pole projection and peak activation closely following that of the icP1a, labeled icP1b. Although the ICs of this cluster were highly spatiotemporally overlapped with the icP1a, they were distinguished based on two characteristics: (a) Their peak voltage at the time of the P100 projected a negative polarity over medial occipital scalp sites and (b) they did not show the large P200 contribution that is characteristic of ICs included in the icP1a cluster; rather, these ICs tended to show a subsequent ERP component following 300 ms. 

The grand average scalp voltage ranges described by these three IC clusters taken together (gray areas in the plots of Figure 4) illustrate the sufficiency of these constituent processes for describing parsimoniously the total scalp ERPs during the first 200 ms following stimulus onset. 

Group IC cluster projections to difference waves envelopes

The vast majority of the two face-versus-house N170 effects is largely accounted for by icN1 (Figure 5). For the face inversion effect, the icN1 cluster not only accounts for the N170 effect (posterior negativity between about 160 ms and 250 ms) but also accounts for much of the inversion effect during the descending slope of the P100 (posterior positivity between about 135 ms and 160 ms). 

Figure 5

 

ERP difference envelope plots for each stimulus contrast: upright face minus upright house (top left), inverted face minus inverted house (top right), inverted face minus upright face (bottom left), and inverted house minus upright house (bottom right). Black lines present the maximum positive and negative voltage values of the scalp data at each time point. Each colored envelope presents the voltage range of each of the three IC clusters: icP1a in red, icP1b in green, and icN1 in blue. The light gray area presents the voltage range of all three IC clusters projected together.

The face-versus-house P100 effects are mostly accounted for by the icP1b cluster (green in Figure 5). The direction of this difference is a larger negative deflection over medial occipital sites for houses over faces. In the envelope plots this pattern persists in the face-inversion comparison but not for the house-inversion comparison. 

Group bootstrap test on the GFA of icP1a, icP1b, icN1, and the residual

Results of the IC bootstrap tests are shown in Figures 6 and 7, where comparison type defines the columns (upright face minus upright house and inverted face minus inverted house in Figure 6, and inverted face minus upright face and inverted house minus upright house in Figure 7) and IC clusters define the rows (from top to bottom are IC clusters icP1a, icP1b, and icN1, followed by the GFA of the residual scalp data). Each plot consists of the same line configuration as the initial GFA results in Figure 2. 

Figure 6

 

IC and residual GFA ERP overlays with 99% confidence intervals of the bootstrapped difference waves for the upright face minus upright house (left) and inverted face minus inverted house (right). The residual represents the scalp signal remaining after removal of the IC projections. The light gray area represents the 99% confidence interval of the 1,000 bootstrapped difference waves. The abscissa is red where the confidence interval does not include zero and the waveforms are considered to be significantly different from each other. Notice that the three ICs account for virtually all the stimulus effects.

Figure 7

 

IC and residual GFA ERP overlays with 99% confidence intervals of the bootstrapped difference waves for the inverted face minus upright face (left) and inverted house minus upright house (right). The residual represents the scalp signal remaining after removal of the IC projections. The light gray area represents the 99% confidence interval of the 1,000 bootstrapped difference waves. The abscissa is red where the confidence interval does not include zero and the waveforms are considered to be significantly different from each other. Notice that the three ICs account for virtually all the stimulus effects.

During the period of the P100, all three IC clusters show differentiation for the face-minus-house contrasts as well as the face inversion contrast. However, the three IC cluster effects take different forms. The icP1a cluster differentiates face-minus-house stimulus images in the form of a latency shift in which face stimuli have a later P100. This characteristic difference slightly persists in the face inversion comparison but to a much lesser degree and does not reach significance in the house inversion comparison (although there are brief times of slight peak differences). The icP1b cluster has a clear peak amplitude difference during the period of the P100; however, this amplitude difference is counterintuitive with respect to the GFA results at the scalp: The icP1b cluster has a clearly larger response to houses than to faces and the polarity of this voltage is negative at medial occipital scalp sites. Thus, this component involves a greater reactivity to the house stimuli but through summation at the scalp contributes to the larger occipital P100 to faces. This effect persists for the face inversion effect but not for the house inversion effect. The icN1 cluster has a P100 effect that takes the form of a slightly larger peak activation to houses than to faces with a positive voltage at the scalp over bilateral occipito-temporal regions. Again, this effect at the time of the P100 persists but to a much lesser extent and is absent for the house inversion comparison. 

At the time of the N170, icN1 clearly dominates the face effect. This is also the largest effect that persists in the face inversion comparison although absent in the house inversion contrast. 

The sufficiency of the three IC clusters described above to account for the GFA scalp effects during the period of the P100 and N170 ERP complex is illustrated in the bootstrap results of the residuals (GFA of scalp data after removing icP1a, icP1b, and icN1). A stimulus-related evoked response does persist in the GFA of the residual scalp data, but only very slight differences exist between any two contrasts. The slight differences following 100 ms for the two face-minus-house contrasts may seem systematic, but their inconsistency at the level of single subject analysis is addressed in the DAR results. 

Differential activation robustness—Bootstrapping results

The differential activation robustness (DAR) results are presented in Figure 8. These are organized such that each of the four columns represents a contrast: from left to right, upright face minus upright house, inverted face minus inverted house, inverted face minus upright face, and inverted house minus upright house. Each row block contains the results of a single waveform, from top to bottom is the scalp GFA, the icP1b, the icP1a, the icN1, and the residual. Each row block contains twelve subrows of plots that depict the results of different levels of analysis. The plots in the top-most subrow of each signal block outlined in a gray box depict the DAR results calculated as a group statistic. The second top-most subrow of each signal block outlined in a black box depicts the DAR results calculated as the average of the single-subject statistics. The remaining ten subrows illustrate the DAR results calculated for each subject individually. 

Figure 8

 

Scalp, IC, and residual GFA differential activation robustness (DAR) results for each of the four contrasts over the time period of −200 ms to +600 ms relative to stimulus onset. Dark blue represents a 100% robust effect where Category 1 has a smaller GFA value than Category 2 in the contrast, dark red represents a 100% robust effect where Category 1 has a larger GFA value than Category 2 in the contrast, and green represents 0% robustness. Each row block contains twelve subrows. The top-most subrow outlined in a gray rectangle depicts the DAR results calculated as a group statistic. The second subrow outlined in a black rectangle depicts the DAR results calculated as the average of single subject statistics. The remaining ten subrows depict the DAR results calculated for each subject separately.

An important characteristic of the DAR measure relates to the differences found between the group measures and the averaged single subject measures. Periods of high robustness can differ between these two measures in two main circumstances. The first is when a group effect is the result of relatively few subjects showing a robust effect. The second is when several (or all) subjects show unreliable effects individually, but taken together as a group the effect is robust. For the current discussion the group effects are displayed to illustrate the windows of robustness given a blanket group statistic. The average of individual robustness measures are examined here as the accurate probability of finding an effect at each time point for each contrast. 

Differential activation robustness for the scalp GFA

The GFA DAR results at the level of the average single subject illustrate two clear periods of likely effects during the first 200 ms for all three face-related contrasts. The most robust period of effect for the GFA waveform is between 150 ms and 200 ms for the two face-minus-house contrasts. The face inversion contrast produced a less robust effect starting at 170 ms and persisting for about 40 ms. The single subject DAR results reveal that all participants produced robust face-minus-house effects starting at around 150 ms. The less robust average single subject effect for the inversion effect is expressed as increased variability in both degree of robustness and latency at the level of the single subjects. 

The GFA DAR results described above reflect a clear replication of the effects presented by Rousselet et al. (2008) for the GFA N170 face effects. The P100 DAR effects at the level of the average subject for the face-related contrasts are not as robust as the DAR results found over the period of the N170. This reduction in the DAR effect over the period of the P100 is consistent with the conflicting results in the literature of P100 face-related effects. The reduced robustness of the P100 face effect is the result of increased individual differences rather than consistent single subject unreliability: For the upright face-minus-house comparison, eight subjects showed 100% robustness in the N170 period (the other two reached 93% and 51%) compared to the P100 period where only two subjects showed 100% robustness and only three others reached 50% or more in the direction of the group effect. 

Differential activation robustness for the GFA of icP1a, icP1b, icN1, and the residual

The icP1a DAR results matched the inconsistent scalp DAR results over the period of the P100. Further, effects in this IC cluster persist through the duration of the N170 in the face-minus-house contrasts. The effects of this IC cluster at the time of the N170 also persist in the face-inversion contrast. The single subject explanation of the reduced DAR from the group statistic to the average single-subject statistic is an inconsistency of effects from subject to subject rather than unreliable effects in each individual. 

Although the icP1b average subject DAR results for face-minus-house contrasts are among the most robust, they do not explicitly appear in the scalp GFA DAR results because they take place at the same time and over the same scalp areas the larger voltages projected by icP1a and icN1. 

The DAR statistic for icN1 closely replicates the initial face effect reported by Rousselet et al. (2008) starting at about 150 ms for the upright face-minus-house contrast. This suggests that even in the presence of potential low-level confounds, the pure face-specific effect can be retrieved using ICA. The inverted face-minus-house contrast shows almost identical DAR results both at the level of the group and single subject. The face inversion effect is shifted slightly later expressing a larger negative deflection for inverted faces starting at approximately 160 ms. The inversion DAR effect also shows a period of greater positivity to inverted faces over upright faces prior to 150 ms. As described in the bootstrapping results, this increased positivity for inverted faces takes the form of a latency shift in the downward slope of the N170 where inverted faces are later than upright faces. 

The sufficiency of the three IC clusters described above for accounting for the scalp GFA results is captured in the DAR of the residual GFA. This analysis shows that even the slight group effect following 100 ms in the face-minus-house contrasts (which is abolished in the average single subject statistic) is the result of effects found in only two of the ten subjects. Further, the specific subjects that contribute the effects differ across the two face-minus-house contrasts. 

Discussion

The experimental paradigm and analysis strategy employed in this experiment addressed two main goals. The first goal was to describe the P100 and N170 ERPs and their categorical effects during a face/house task at the level of their constituent electrocortical processes unmixed from the scalp data using ICA. The second goal was to describe the likelihood of finding categorical effects within the first 200 ms following the stimulus onset at the scalp and within ICs, at the level of both group and individual subject analysis. 

Electrophysiological constituents of the P100 and N170 ERPs

Although it is accepted in the ERP literature that individual ERP components reflect the summation of multiple event-related cortical processes whose electrical projections overlap both temporally and spatially, the spatial distribution and relative voltage contribution of each constituent process is difficult to determine from an examination of the signals as recorded at the scalp. By performing an ICA decomposition on the artifact-rejected continuous EEG traces, we have identified clearly three independent electrophysiological processes that accounted for face-related ERP effects at the scalp during the period of the P100 and N170 complex. Notably, all three processes were relatively active over the period of both the P100 and N170 time periods. 

Is the N170 cortical source accounting for face-house differences found at the time of the P100 but often masked by other lower level processes of the visual cortex during the interval of 80 ms to 150 ms? The icN1 results suggest that the answer is “no.” Although the icN1, which describes the familiar N170 face effect, is active in a manner that projects a positive voltage over occipital scalp regions during the time of the P100, this activation differentiates stimulus conditions in a direction opposite to that typically found at the scalp. The greatest icN1 face effect occurs in the period of the N170, starting at about 150 ms. The average single subject DAR timing of the face effects expressed by icN1 reported here closely replicates the scalp findings of Rousselet et al. (2008). 

The familiar P100 scalp effect found in this analysis (larger P100 for faces than houses) is accounted for by two ICs, namely icP1a and icP1b. The activation patterns of icP1a differentiate stimulus categories in the form of a slope difference during the descent of the peak following 100 ms post-stimulus onset. This icP1a slope difference contributes to the peak difference summed at the scalp shortly following 100 ms post-stimulus onset. 

The second constituent process that contributes to the P100 face effect is the icP1b. This component's effect takes the form of a larger negative voltage at central occipital scalp sites for houses over faces. This characteristic demonstrates an opposite effect than what would be interpreted at the scalp, specifically, that a smaller scalp voltage is the result of increased cancellation between two opposing processes. This small but robust voltage difference (in some subjects) taken together with the slope difference of the icP1a describes the underlying dynamics and robustness of the controversial P100 face effect. This finding illustrates a case of the necessity of unmixing the scalp signals in order to accurately interpret scalp voltage differences. 

The activation patterns of the icN1 mimic the scalp EEG collected during the presentation of face and house stimuli that have sophisticated controls for low-level physical characteristics (e.g., Rousselet et al., 2008), i.e., the icN1 face-minus-house effects replicate scalp studies that show no P100 effects. Thus, although there is some P100 face effect in this data set, the strong icN1 component does not account for it. 

The ability to isolate the N170 face effect source from the P100 effects in continuous EEG signals has implications for the study of face-related processing. Under many circumstances it is desirable to study the timing of specific processing (such as faces) in the context of nonmanipulated stimuli. The ecological validity of face stimuli is important in order for researchers to determine the full effect of face-specific processing. Further, because the face effect is considered more generally to be one of the earliest visual effects modulated by top-down sensory/perceptual processes, robust measures that relate the timing of earliest categorical ERP deviations are paramount. Because ecologically valid stimuli and timing of effects are both critical in the study of top-down sensory perceptual processing, it is important for the interpretability of the results that the constituents underlying the ERP effects at different latencies be unmixed. 

Constituents of effect robustness

N170 effects in the face-processing literature have developed a reputation for being robust, while P100 effects in the same literature have developed a reputation for being unreliable. Within this study the average single subject DAR measure reveals what underlies these reputations. The robust N170 effects measured in the average single-subject DAR at both the scalp and icN1 are expressed as periods of robust effects in all of the individuals and the timing of the effects across individuals is largely consistent. The less reliable P100 average single subject DAR results at the scalp are described as less consistent in time across individuals as well as present in fewer subjects. The P100 scalp DAR results are not increased when measured on icP1a, with relative inconsistency in timing and not present in all individuals. The icP1b average subject DAR is surprising in that it is an effect that is not apparent at the scalp. This effect is robust in six subjects showing a larger negative deflection over medial occipital regions for houses over faces during the period of the P100. 

Conclusion

The inherent interpretation difficulties associated with the mixing of electrocortical field potentials at the scalp in EEG recordings is particularly pronounced in the task of measuring the initial face-related ERP differences because multiple cortical networks project spatiotemporal field projections during the P100 and N170 that are highly overlapping. Using ICA we showed that the constituent cortical processes derived from the scalp EEG confirm the functional independence of the P100 and N170 reported by Rossion and Caharel (2011) in that the scalp effects at the time of the P100 are accounted for by different components than the N170. Further we showed that the constituent processes also confirm the timing of the face effects reported by Rousselet et al. (2008) in that the timing of the effects for the icN1 closely match their results despite the presence of potential low-level stimulus confounds. This study also confirms that the P100 face effect is less robust than the N170 effect due to less consistency across individuals. However, our results also indicate that the P100 at the scalp needs to be decomposed with an ICA and bootstrapping strategy in order to fully understand its effects: in our case, that the P100 at the scalp being larger for faces than for houses also includes a component in which there is a larger activation for houses than faces. Similarly, our results indicate that although the icN1 is active at the time of the P100, its contribution is in the opposite direction, i.e., smaller for faces than houses. By unmixing the spatiotemporally overlapping field potentials that contribute to the P100 and N170 ERP effects, we are able to account for the data and examine individual networks without the necessity of controlling for low-level stimulus differences, a constraint that itself can produce systematic biases across conditions and potentially alter these brain processes that are associated with ecological validity. Finally, these methods provide the single-trial source separation that is necessary for examining the functional relationships among cortical networks in the task of visual information processing. 

Acknowledgments

This work was made possible by funding from the Canadian Foundation for Innovation (CFI) and the Natural Sciences and Engineering Council of Canada (NSERC) to SJS, the facilities of the Shared Hierarchical Academic Research Computing Network (SHARCNET: www.sharcnet.ca) and Compute/Calcul Canada, and Ruben Gur for the original face stimuli. 

Commercial relationships: none. 

Corresponding author: Sidney J. Segalowitz. 

Email: [e-mail]. 

Address: Psychology Department, Brock University, St. Catharines, ON, Canada.