The spatiotemporal profile of cortical processing leading up to visual perception

Abstract

Much controversy exists around the locus of conscious visual perception in human cortex. Some authors have proposed that its neural correlates correspond with recurrent processing within visual cortex, whereas others have argued they are located in a frontoparietal network. The present experiment aims to bring together these competing viewpoints. We recorded EEG from human subjects that were engaged in detecting masked visual targets. From this, we obtained a spatiotemporal profile of neural activity selectively related to the processing of the targets, which we correlated with the subjects' ability to detect those targets. This made it possible to distinguish between those stages of visual processing that correlate with human perception and those that do not. The results show that target induced extra-striate feedforward activity peaking at 121 ms does not correlate with perception, whereas more posterior recurrent activity peaking at 160 ms does. Several subsequent stages show an alternating pattern of frontoparietal and occipital activity, all of which correlate highly with perception. This shows that perception emerges early on, but only after an initial feedforward volley, and suggests that multiple reentrant loops are involved in propagating this signal to frontoparietal areas.

Introduction

Over the past 15 years, our view of the way information is processed in primate visual cortex has dramatically changed. It has become increasingly clear that information is not only processed hierarchically from bottom to top, but that interactions between cortical areas play a crucial role in the way the brain extracts information from its retinal input (Bar et al., 2006; Hupe et al., 1998; Lamme, Van Dijk, & Spekreijse, 1993). An ever growing body of literature shows extra-classical receptive field effects—also known as contextual modulation—occurring ∼100 ms after stimulus onset in V1 and up (Lamme, 1995; Lamme, Rodriguez-Rodriguez, & Spekreijse, 1999; Lamme & Roelfsema, 2000; Lamme et al., 1993). These type of effects point to the influence of higher on lower visual areas and have been widely recognized to result from stimulus-related recurrent processing within visual cortex (Dehaene, Changeux, Naccache, Sackur, & Sergent, 2006; Lamme, 2006; Roelfsema, Lamme, Spekreijse, & Bosch, 2002). 

Early on, such findings were restricted to recordings in macaque V1, but recently EEG, fMRI, and TMS studies have confirmed the importance of recurrent cortico-cortical interactions in human perception (Fahrenfort, Scholte, & Lamme, 2007; Haynes, Driver, & Rees, 2005; Murray, Boyaci, & Kersten, 2006; Pascual-Leone & Walsh, 2001). 

However, it is difficult to uncover the spatiotemporal profile of such interactions. The temporal resolution of fMRI is too limited. In neurophysiology, it is difficult to simultaneously locate cells in high and low visual areas covering the same retinotopic area. Thus, when speaking about “recurrent processing” or “feedback processing,” researchers often posit multiple iterations or loops, without being clear on the number or timing of these loops. For that matter, not everything is known about the function of recurrent processing either, although there are strong indications that it plays a crucial role in the function and phenomenology of figure-ground segregation and visual awareness (Fahrenfort et al., 2007; Hupe et al., 1998; Lamme, 1995; Lamme & Roelfsema, 2000; Lamme et al., 1993). 

In this experiment, we sought to identify the spatiotemporal profile of cortical processing in human visual cortex using EEG and find out which activations correlate with perception and which ones do not. Although it is impossible to resolve the origin of EEG generators in the brain in an unconstrained manner (Nunez & Srinivasan, 2006), a claim with respect to relative anteriority or posteriority of such generators can be made much more easily (Fahrenfort et al., 2007; Foxe & Simpson, 2002). We calculated spherical surface Spline LaPlacian distribution maps of the EEG (Perrin, Pernier, Bertrand, & Echallier, 1989) and subsequently pooled electrodes in an anterior–posterior fashion, making claims only about the relative position of generators with respect to the front–back dimension. In combination with EEG millisecond timing information, this method makes it possible to draw conclusions about the temporal order in which consecutive brain areas become active and about the modes of processing that subserve these activations (Fahrenfort et al., 2007; Foxe & Simpson, 2002). 

Subjects were asked to identify masked texture defined Figure and No Figure stimuli while ERPs (event-related potentials) were recorded (see Figures 1 and 2). We calculated an ERP Figure minus No Figure difference wave in order to isolate figure induced activity and correlated this with behavioral scores reflecting the ability of subjects to discriminate between Figure and No Figure trials. This was done for the entire spatiotemporal profile of cortical processing, allowing inferences about those aspects of processing that do and those that do not correlate with visual perception. 

Figure 1

 

Schematic time course of a trial. Trial started with 300 ms fixation, followed by a 100-ms target, followed by a 16.6-, 33.3-, or 50-ms mask. Subjects were given 1000 ms to indicate whether a figure was presented or not. The inter-trial interval was jittered between 100 and 600 ms.

Figure 2

 

Examples of stimuli used in the experiment. (A) Figure stimulus, (B) No Figure stimulus, (C) Mask, and (D) Orientations used to configure target and mask stimuli. In any given trial, the orientation(s) used in the target stimulus would not be used again for the mask stimulus. Each condition consisted, on average, of the exact same sets of oriented textures.

The behavior–EEG correlation shows an early bilateral feedforward signal which does not correlate with subjects' ability to distinguish Figure from No Figure targets even though there is a difference in the signal generated by Figure and No Figure stimuli. Slightly later in time, an extremely consistent posterior occipital generator shows a strong correlation with perception. In all likelihood, this generator is due to stimulus-related recurrent processing within visual cortex, as its timing and location are highly consistent with the effects of contextual modulation in early visual areas of the macaque (Lamme, Zipser, & Spekreijse, 2002, a very similar effect using human EEG has also been shown in Fahrenfort et al. (2007)). Due to its correlation with perception and its precession to parietal and frontal activations, this generator seems to act as the primary seed of perception. Within 200 ms, bilateral parietal and centrofrontal regions become active. These are followed at approximately 250–300 ms by concurrent frontal and occipital generators both of which correlate highly with perception, uncovering a distributed network of frontoparietal and occipital areas, plausibly involved in the transition of visual perception to a reportable stage (Dehaene, Sergent, & Changeux, 2003; Lamme, 2003, 2006; Lumer & Rees, 1999; Sergent, Baillet, & Dehaene, 2005). 

Methods

Participants

Nineteen psychology students took part in the experiment in partial fulfillment of first year course requirements. All subjects (mean age 23.4, ±6.8) had normal or corrected-to-normal vision. Eighty-four percent were right handed. Sixty-three percent were female. Each subject provided written informed consent. All procedures were approved by the ethical committee of the University of Amsterdam. 

Stimulation

Trial time course and example stimuli can be found in Figures 1 and 2. At the start of each trial, a fixation dot on a grey background turned from dark into bright red, followed after 300 ms by texture target with a duration of 100 ms. In half of the trials, this target contained an orientation defined square (Figure trials), in the other half the target was a homogenous texture (No Figure trials). Each target was followed by a 16.6-, 33.3-, or 50-ms pattern mask containing an orientation defined square annulus ( Figure 2). 

Backward masking, in which a target stimulus is followed shortly by a second stimulus, degrades the visibility of the target stimulus (Breitmeyer, 1984). There are many types of masking, with different explanations for their effectiveness (e.g., Fahrenfort et al., 2007; Lamme et al., 2002; Macknik & Livingstone, 1998). For reviews, see Breitmeyer and Ogmen (2000) and Enns and Di Lollo (2000). The mask used in the present experiment was determined in a pilot phase to fit in well with the general phenomenology of the task and to maximally elicit variable responses between subjects. The aim here was to exploit inter-individual differences in masking effectiveness, so visibility scores could be correlated with differences in the EEG signal over time. The different mask durations had no function other than making the graded response described below meaningful, as a single mask duration would not have resulted in differential responses. 

Subjects were given 1000 ms to respond, after which the fixation dot turned dark red until the start of the next trial. This inter-trial interval varied between 100 and 600 ms. All conditions were randomized and evenly distributed. A total of 6 blocks of 192 trials were recorded per subject. 

Textures consisted of black (0.9 cd/m ²) on white (104 cd/m ²) line elements, spanning 0.06° and 0.37° of visual angle. Line elements could have four possible orientations: 22.5°, 67.5°, 112.5°, and 157.5°. A No Figure target contained a single orientation; a Figure target consisted of two orthogonal orientations. The square in the figure trial subtended 2.47° of visual angle in the center of the screen. Masks consisted of an orientation defined square annulus of the same size as, and in the same central location as the Figure target, and consisted of orientations not used in the preceding target. Border thickness of the mask annulus was 0.39° of visual angle. All stimuli were isoluminant at 66.8 cd/m ². Stimuli were created using Matlab (The MathWorks, Inc., Natick, MA, USA) and were presented using Presentation (Neurobehavioral Systems, Inc., Albany, CA, USA). 

All texture orientations used in a trial were completely counterbalanced over conditions in such a way that Figure and No Figure trials were equal with respect to local stimulation. This was done in order to be able to carry out the EEG subtraction procedure detailed in the Results section (also see Figure 5). For a similar procedure in the context of figure-ground segregation and EEG/MEG, see Caputo and Casco (1999), Fahrenfort et al. (2007), Lamme, Van Dijk, and Spekreijse (1992), and Scholte, Witteveen, Spekreijse, and Lamme (2006), or for other examples of subtraction procedures in EEG, see Hillyard, Hink, Schwent, and Picton (1973) and Thorpe, Fize, and Marlot (1996). 

Task and behavioral measure

Approximately 1 week prior to the EEG session, all subjects were given a 30-min training session to become acquainted with the task. After training they took part in an EEG session in which they carried out the same task. Subjects were instructed to distinguish between Figure and No Figure trials. With their right hand, they pressed a single button if they perceived a No Figure target (Not-Seen response), or one of three buttons (3-point scale) if they perceived a Figure target, depending on perceptual strength. The 3-point scale ensured subjects based their responses on their phenomenology, and not on guessing. Later, the 3-point scale was collapsed into a single response category (Seen response) and not used any further. For each subject, a perfect observer score was calculated based on Seen and Not Seen responses, reflecting a subjects' ability to distinguish between Figure and No Figure trials. The perfect observer score is a linear and subject bias' free measure derived from d′ (Wickens, 2002). 

EEG measurements and pre-processing

EEG was recorded from the scalp using a BioSemi ActiveTwo 48 channel active EEG system (BioSemi, Amsterdam, the Netherlands) at 256 Hz. Forty-eight scalp electrodes were measured (referenced to the ears), as well as two electrodes for horizontal and two for vertical eye movements (each referenced to its opposite counterpart). The data were filtered (high pass >0.5 Hz, low pass <20 Hz, 50 Hz notch), and automatic artefact rejection was applied by removing segments containing voltage steps of more than 50 μV, segments falling outside the −200 μV to 200 μV range, as well as segments containing larger than 300 μV differences within the segment. Ocular correction was applied on the basis of the horizontal and vertical electro-oculograms (Gratton, Coles, & Donchin, 1983). After ocular correction, artefact rejection was applied again by removing all segments outside the −50 μV to 50 μV range. Baseline correction was applied using the signal in the −300 ms to 0 ms interval. All EEG processing was done using Brain Vision Analyzer (Brain Products GmbH, Munich, Germany). 

We converted all ERP signals using a surface Laplacian procedure by interpolating ERP signals to approximate scalp current densities (SCDs) (Perrin et al., 1989) using spherical splines. SCDs are spatial second order derivatives of the potentials measured on the scalp. SCDs are reference free and act as a band-pass spatial filter isolating signals due to sources localized in superficial cortex. The spatial resolution of SCDs is typically in the order of 2 to 3 cm in surface tangential directions (Nunez & Srinivasan, 2006). 

EEG spatiotemporal maps

In order to construct spatiotemporal maps of the EEG, electrodes were pooled in a posterior–anterior fashion, starting with occipital electrodes and moving towards the front around both sides of the scalp (separately for the left and the right hemisphere). Electrode poolings were chosen in such a way that they were separated from each other by approximately 3 cm (see Figure 3). Because SCDs have a spatial resolution of around 2 to 3 cm, this ensured that the scalp current densities were accurate with respect to the front–back dimension. Separate poolings were made for the right and the left hemisphere. Two-dimensional spatiotemporal maps were constructed on the basis of these poolings (see Figure 4 for an example). Three electrodes (Fz, Cz, and Pz) were left out of the poolings because they were located centrally and could therefore not be assigned to either left or right hemispheric poolings. Two electrodes were left out because they violated the 3-cm requirement (P1 and P2). Visualization and statistical analyses of spatiotemporal maps and other time courses were done with Matlab (The MathWorks, Inc., Natick, MA, USA) using custom code. 

Figure 3

 

Layout of the electrode poolings used in the experiment. The top half of the figure shows the electrodes displayed on the scalp. The bottom half shows the same electrodes on a flattened head view and the color scheme associated with the poolings.

Figure 4

 

2D spatiotemporal maps of (A) Figure followed by Mask and (B) No Figure followed by Mask processing. The y-axis represents time. Electrode poolings are at each tick mark of the x-axis, locations in between poolings have been spline interpolated. Color indicates the strength of the scalp current density at each spatiotemporal location.

Statistical testing

Differences between experimental conditions were ascertained using paired t-tests. Corrections for multiple comparisons were made by limiting the FDR (false discovery rate), a method by which the expected proportion of falsely rejected hypotheses is controlled (Benjamini & Hochberg, 1995). 

Formally, the FDR is given by E

$\left[\frac{V}{V+S}\right]$

in which V is the number of false positives and S is the number of true positives. By applying the FDR correction, this value is kept below the threshold q, often set at 0.05 (although higher values can be acceptable depending on the research question). The formula used for finding the p-values at which this is true for a series of ordered p-values from small to large, is P( i) ≤

$\frac{i}{L}$

q, in which L( i) is the temporal or spatiotemporal position corresponding to p-value P( i); thus, L is the number of temporal or spatiotemporal positions (i.e., number of tests being performed). Independence or positive correlation of tests is assumed. For an explanation of how the FDR is used in the field of neuroimaging, see Genovese, Lazar, and Nichols (2002); for an application of the FDR in EEG, see Fahrenfort et al. (2007). 

Results

Figure 4 shows the spatiotemporal profile of cortical processing of Figure and No Figure trials on 2D spatiotemporal maps ( Figures 4A and 4B, respectively). Color represents the scalp current density at each spatiotemporal location. Electrode poolings are represented at each tick mark on the x-axis, occipital in the middle, left-frontal on the left, and right-frontal on the right. Values in between poolings were interpolated using Spline interpolation. Time is represented on the y-axis. 

The Figure and No Figure spatiotemporal maps in Figure 4 are remarkably similar to each other. Both maps show a strong occipital generator in the 100- to 150-ms time frame, flanked by equally strong bilateral parietal generators, followed by two occipital generators later in time. Not surprisingly, this shows that the strongest cortical response due to these texture stimuli are early on and within (or close to) visual cortex. Because of the strength of these responses, it is difficult to infer small differences in cortical processing between Figure and No Figure trials from these raw spatiotemporal maps. 

Therefore, all subsequent analyses were done on SCD difference waves. These were obtained by subtracting averaged No Figure trials from averaged Figure trials. An example of this procedure is shown in Figure 5. This figure shows the time course of the SCD for the occipital pooling for both Figure and No Figure trials, as well as for the difference between the two. It also shows where the difference is significant according to a false discovery rate of 0.05 (FDR, see the Statistical testing section). 

Figure 5

 

Example of the Figure minus No Figure subtraction procedure on posterior pooling I. The SCD time course due to a Figure and a No Figure trial is shown for the occipital pooling. The difference between Figure and No Figure is also shown, in blue. A paired t-test between Figure and No Figure trials was performed at each time point in the time series. Significant differences between Figure and No Figure according to an FDR of 0.05 are shown in black.

Subtracting No Figure from Figure trials has two other major advantages:

1.  
   The Figure minus No Figure subtraction isolates activity related to the processing of the figure. As Figure and No Figure targets are, on average, made up of the exact same sets of oriented textures (see Figure 2), any influence of local stimulation on cortical processing, such as caused by the line elements of the textures themselves, is subtracted out. The only signal left is related to the processing of differences in figure-ground organization between Figure and No Figure trials (for other examples on the topic of texture segregation, see Caputo & Casco, 1999; Fahrenfort et al., 2007; Lamme et al., 1992; Scholte et al., 2006).
2.  
   By the same token, as the rest of the stimulus sequence is exactly equal between Figure and No Figure trials, any direct contribution of other stimuli in the sequence, such as fixation dots and masks, is subtracted out as well. Masks of different durations were evenly distributed over Figure and No Figure trials and were thus subtracted out.

The spatiotemporal profile of the Figure minus No Figure subtraction is shown in Figure 6A. In this 2D map, color represents the difference in scalp current density between Figure and No Figure trials. The axes are the same as in Figure 4. To evaluate differences in cortical processing between Figure and No Figure trials, a random effects analysis was performed by employing a paired two-tailed t-test between Figure and No Figure averages at each space–time point in the spatiotemporal map in Figure 6A, treating the average of each subject at that space–time point as an observation. The correction for multiple comparisons with respect to the number of tests was done by limiting the false discovery rate (FDR, see the Statistical testing section), a method by which the p-value at which significance is evaluated is corrected for the number of tests being performed (Benjamini & Hochberg, 1995). The spatiotemporal locations for which the difference between Figure and No Figure was significant is encapsulated by solid black lines, corrected for multiple comparisons at an FDR of 0.05 (q = 0.05). The solid dots in Figure 6A indicate local minima and maxima of the SCD difference. On the bottom of Figure 6A topographic plots of the critical time windows are shown to provide an unambiguous description of the spatial distribution of the effects, thus confirming the overall picture. 

Figure 6

 

(A) Figure minus No Figure activity. The top shows the 2D spatiotemporal map for the difference between Figure and No Figure processing (effectively the difference between Figures 4A and 4B). The y-axis represents time. Electrode poolings are at each tick mark of the x-axis such that the middle is the most posterior (occipital) location and the left and right of the axis are the most anterior (frontal) left and right positions. Color indicates the strength of the difference between Figure and No Figure trials in scalp current density at each spatiotemporal location. A paired t-test between Figure and No Figure trials was performed at each spatiotemporal location in the map. The solid black lines show the areas within which these differences were significant, corrected for multiple comparisons at an FDR of 0.05. Solid dots indicate local maxima and minima. On the left of the figure, five relevant stages (see Results section) are indicated by encircled numbers with pointing arrows. For each of these stages, a topographic map is shown at the bottom. (B) Figure minus No Figure correlation with detection. The lower figure shows the correlation between subject's ability to discriminate between Figure and No Figure trials and the Figure minus No Figure difference for each spatiotemporal location in the map. Color represents the strength of Spearman's rank correlation. The solid black lines are the same as in the top figure; the solid red lines represent areas within which the correlations are significant at the .05 level. In white, the correlations are given for the areas that are enclosed by both black and red lines. Level of significance is indicated by asterisks: *p < .05, **p < .01 (one-tailed). Spurious activations of significant clusters smaller than 25 spatiotemporal locations were not reported in either figure.

Figure 6A clearly shows that processing does not occur hierarchically from bottom to top (i.e., from center to edges in Figure 6A) in a feedforward fashion, but that massive activation of early visual areas occurs up to at least 400 ms after stimulus presentation, long after more frontal areas have been recruited. These activations are likely to reflect both sustained local processing (Foxe & Simpson, 2002), recurrent interactions within visual areas (Fahrenfort et al., 2007; Foxe & Simpson, 2002; Lamme, 1995), as well as long range interactions between frontal and visual areas (Lumer & Rees, 1999; Rodriguez et al., 1999). From Figure 6A, we can infer a number of stages. Each stage is indicated by an encircled number on the left of the figure, for each of which the scalp topographic flat map is shown on the bottom:

1.  
   A bilateral parietal generator peaking at 121 ms (collapsed average: 121 ms, right: 117 ms/left: 152 ms). A meta-analysis of studies employing macaque intracranial recordings (Lamme & Roelfsema, 2000) has shown average response times in early visual areas of 72 ms (V1), 84 ms (V2), and 77 ms (V3). Dorsally these continue to 129 ms (V7a), 92 ms (V7ip), and ventrally to 106 ms (V4). Given the fact that the earliest Figure–No Figure differences here peak parietally at 121 ms, it is not unlikely that they reflect sustained activity resulting from feedforward processing, although it cannot be ruled out that some feedback is already incorporated at this interval (Foxe & Simpson, 2002). Note that we report peak ERP latencies and average response latencies, not onset latencies, which are considerably shorter.
2.  
   A more posterior occipital generator peaking at 160 ms which, due to the fact that it is later in time and more posterior than the bilateral 121 ms generator is probably due to a combination of feedforward and feedback activity within early visual cortex (Fahrenfort et al., 2007; Foxe & Simpson, 2002). Incidentally, this interpretation is in excellent agreement with a large number of studies showing contextual modulation due to recurrent processing in this time frame in V1 and up using highly comparable stimuli (Lamme, 1995; Lamme et al., 1993; Supèr, Spekreijse, & Lamme, 2001). Several other authors have identified EEG correlates of conscious vision around this time window, some earlier peaking around 100 ms (Fahrenfort et al., 2007; Pins & ffytche, 2003) and some later starting at 130 ms and peaking later on in time (Koivisto, Revonsuo, & Lehtonen, 2006; Koivisto, Revonsuo, & Salminen, 2005).
3.  
   Occipitoparietal and centrofrontal regions peak at around 200 ms. A negativity starting around 200 ms is typically reported as part of the N2 family of components, of which the most notable in this context is the N2pc (N2 posterior–contralateral). This component is largest at posterior scalp sites and is observed over the hemisphere contralateral to the location of an attended object (given that the target stimulus is not located centrally). It has been suggested to reflect perceptual-level attentional selection, for instance to zoom in on a target within an array of distractors (Luck, Girelli, McDermott, & Ford, 1997). This component has been shown to occur virtually unimpaired even when a stimulus is unreportable due to object substitution masking (Woodman & Luck, 2003).
4.  
   A more posterior occipital generator peaks at 246 ms, with a concurring frontal generator. Given their timing and approximate concurrence these may be engaged in long range coordinated recurrent activity enabling conscious access (Dehaene et al., 2006; Lamme, 2006; Lumer & Rees, 1999; Rodriguez et al., 1999).
5.  
   Strong recurring occipitoparietal generators appear at 350–400 ms, flanked by centrofrontal generators, which may well reflect a third iteration of recurrent processing within and/or between these areas. A posterior–parietal component in this time window is classically reported as the P3 or P300, referring to a third positivity (or a positivity around or after 300 ms) in the ERP waveform. The P3 has been associated with a number of psychological variables, the most prominent of which are working memory and attention (Donchin & Coles, 1988; Kok, 2001). More recently, it has been suggested to be the response to the outcome of internal decision making processes (Nieuwenhuis, Aston-Jones, & Cohen, 2005). Accordingly, activity in this and the previous time window has generally been observed to be attenuated by attentional manipulations such as the attentional blink (e.g., Koivisto et al., 2006; Kranczioch, Debener, & Engel, 2003; Sergent et al., 2005).?>

In Figure 6B, we show which of these stages correlate with perception and which ones do not. For each subject the perfect observer score was calculated, reflecting his or her ability to detect masked figures. Perfect observer scores and classic percent correct are shown in Figure 7 for each subject. We calculated the correlation between these perfect observer scores and the Figure minus No Figure difference for the entire spatiotemporal profile in Figure 6A. Thus, for each space–time point in the spatiotemporal map from Figure 6A, Spearman's rank correlation was computed between subjects' average Figure minus No Figure difference SCD and subjects' perfect observer score at discriminating Figure from No Figure trials. The result is shown in Figure 6B, the strength of the correlations in color. The black lines enclose the spatiotemporal locations where the Figure minus No Figure difference is significant, as redrawn from Figure 6A. 

Figure 7

 

For each subject, the perfect observer score for the Figure detection task as well as the corresponding classic percentage correct score is shown. Subjects are uniquely identifiable by both shape and color of the markers.

The dark red lines enclose the spatiotemporal locations within which the correlations between detection accuracy and the SCD difference wave are significant at the .05 level. In white, the correlations are given for each of the areas where both the difference and the correlations are significant (i.e., those areas within which the black lines and the dark red lines overlap). Only correlations in spatiotemporal locations where Figure and No Figure significantly differ were reported, so as to exclude correlations that occurred outside of the periods of neural activity related to the processing of figure from ground. Note for example that we also found significant correlations (i.e., dark red circles in Figure 6B) at about stimulus onset (0 ms) in the right parietal and frontal regions. These might reflect attentional set being higher at trials in which detection is successful and will not be directly related to the processing of the figure stimulus per se (also see Supèr, van der Togt, Spekreijse, & Lamme, 2003). Their location is consistent with right hemispheric dominance for attention for the entire visual field (e.g., Heilman & Van Den Abell, 1980; Mesulam, 1999). Alternatively, they may reflect spurious correlations, as calculating such large numbers of correlations may produce significant results even when fitting noise. 

The map shows that the first bilateral parietal generator due to feedforward processing (stage 1 above, also see Figures 8A and 8C) does not correlate with subjects' ability to detect a figure, whereas the later occipital generator due to recurrent processing (stage 2 above, also see Figure 8B) does. As this occipital activation is the first one to show a strong correlation with perception, and almost all ensuing correlations are highly significant, it seems to act as a seed for further correlations. 

Figure 8

 

Graphs showing data points and their fit for 5 selected relevant generators. For significant correlations, these represent clusters of spatiotemporal locations within the generator where the correlation had a p-value <.05. For non-significant correlations, these represent the 50% contiguous spatiotemporal locations having correlations with the lowest p-values within that generator. (A) The generator peaking at 152 ms at pooling III-L attributed to feedforward processing. (B) The generator peaking at 160 ms at occipital pooling I attributed to feedback processing. (C) The generator peaking at 117 ms at pooling III-R attributed to feedforward processing. (D) The generator peaking around 300 ms on the anterior left. (E) The generator peaking around 300 ms on the anterior right.

Further evidence of the importance of this generator in perception comes from the fact that it is by far the most consistent difference between Figure and No Figure. As the FDR is reduced to 0.0001, the only significant generator surviving this overly strict threshold is this occipital one (see Auxiliary Figure 1), reflecting the fact that it represents the most consistent difference between Figure and No Figure processing. 

Later generators show an alternating pattern of anterior and posterior activity, most of which correlate with perception, although anterior correlations appear more in the left (also see Figure 8D) than in the right hemisphere (also see Figure 8E). This left-hemispheric dominance may be caused by the fact that subjects had to report Figure presence with their right hand, although correlations with activations due to motor preparation and response are unlikely because a response was always required and always given with the right hand (and should thus be subtracted out of the Figure minus No Figure difference). Also, language is known to be predominantly left hemispheric (Vigneau et al., 2006). Left hemispheric dominance of correlations could therefore partly be due to the fact that subjects had to give an appraisal of stimulus strength that may have been verbalized mentally. 

We would like to stress that the descriptions of generators and correlations found are not exhaustive in terms of the neural activity that underlies them. EEG activity is caused by coordinated postsynaptic activity of huge cell assemblies producing dynamic patterns of electric potential on the scalp. Aside from the inverse problem, the skull is also beset by problems of volume conduction, leaving us with a very coarse reflection of neural activity (Nunez & Srinivasan, 2006). Thus, multiple coherent neural events may show up as a single generator or may not show up at all. A single coherent neural event may even show up as multiple distinct generators as the polarity of the difference between experimental conditions shifts over time. Therefore, the only aim in this experiment is to embed the generators and correlations that were identified in a coherent picture of cortical processing given the knowledge we have from other sources such as monkey physiology, and not to give a comprehensive description of all cortical processing. 

Discussion

With the employed subtraction paradigm, we isolated signals that discriminate between Figure and No Figure stimuli. Thus, we obtained a spatiotemporal profile of the EEG activity that is induced by a texture figure. We could discern 5 stages in this signal, starting with a signal resulting from initial feedforward activation, followed by signals reflecting recurrent loops of feedback and feedforward activity within visual cortex, and later between more distant cortical areas. Correlating this activity with behavior, i.e., the ability to detect figures when masked, shows that the early feedforward activation of the parietal cortex does not correlate with perception. This is in line with many studies showing that feedforward processing goes uninterrupted in visual cortex even when a subject is fully unconscious of the stimulus (Fahrenfort et al., 2007; Lamme, Zipser, & Spekreijse, 1998; Lamme et al., 2002). 

More posterior in the brain, and later in time, an occipital generator plausibly due to recurrent processing correlates highly with perception. This suggests that the correlates of perception and visual awareness start to emerge at around 100 ms due to recurrent processing, and are propagated further along the system through multiple recurrent loops (also see Dehaene et al., 2006; Lamme, 2006). This view is consistent with many studies showing the importance of recurrent processing in figure-ground segregation and visual awareness (Fahrenfort et al., 2007; Haynes et al., 2005; Hupe et al., 1998; Lamme & Roelfsema, 2000; Lamme et al., 1993; Scholte et al., 2006; Silvanto, Cowey, Lavie, & Walsh, 2005). 

As have others, the present study thus stresses the importance of the 100+ ms activation of visual cortex in human perception (Fahrenfort et al., 2007; Koivisto et al., 2006, 2005; Pins & ffytche, 2003; Wilenius-Emet, Revonsuo, & Ojanen, 2004). Although the exact timing differs somewhat between these studies—which may relate to differences in stimulus complexity—they are all consistent with the idea that visual awareness emerges as a result of recurrent activity in visual cortex. 

Confirmation of this idea can also be found in studies on the topic of high-level perceptual decision making. A number of studies have employed linear regression techniques to create single trial predictions about subjects' performance in perceptual decision tasks, such as discriminating between a face and car (Philiastides, Ratcliff, & Sajda, 2006; Philiastides & Sajda, 2006; Smith, Gosselin, & Schyns, 2004). These studies suggest that EEG components reflecting early visual perception can be found occipitally at ∼170 ms (plausibly incorporating feedback mechanisms), whereas a late ∼300 ms component reflects a postsensory/decision stage. 

This is not to say that stimulus categorization cannot be triggered by non-perceptual (feedforward) events. For instance, face tuning starts in inferotemporal cortex by ∼100 ms (Liu, Harris, & Kanwisher, 2002; Oram & Perrett, 1992) and Thorpe et al. (1996) have shown that frontal cortex starts detecting the presence of an animal in a natural scene by ∼150 ms. But VanRullen and Koch (2003) have shown that subjects can perform such stimulus categorizations even when highly confident that they did not see the (masked) test stimulus. Crucially, the earliest time at which subjects start categorizing unperceived scenes with above chance performance matches exactly the time at which they do so for consciously perceived scenes. This leads these researchers to conclude that both conscious and unconscious categorization are initially triggered by the same (unconscious) feedforward process. In such a scheme, “perceptual” or high-confidence decisions logically depend on subsequent recurrent activity (for a similar argument, see Jolij, Scholte, Van Gaal, & Lamme, in press). 

A large number of physiological studies show correlates of figure-ground segregation in V1 in the 100+ ms time window due to recurrent processing, which fits well with the 100+ ms timing found in the present experiment (e.g., Lamme et al., 1993) and which has been argued to be a correlate of visual awareness. Although the exact nature of cortical recurrent processing may be more dynamic than can be uncovered with EEG, the present study suggests that in humans too, the phenomenology of figure-ground segregation may originate from early recurrent processing in visual cortex. 

Conclusions

The first correlates of perception emerge right after an initial (automatic) feedforward sweep, which does not correlate with the subjects' ability to detect a figure from a background. All recurrent and feedforward activity after this seed correlates with the ability of subjects to perceive a figure. There are multiple recurrent loops involved in visual perception spanning the entire human cortex in the 100- to 450-ms time frame. 

Supplementary Materials

Supplementary Figure 1 - Supplementary Figure 1 

Supplementary Figure 1. Figure minus No Figure activity. Shows the spatiotemporal profile of the Figure minus No Figure difference, as in Figure 6A, but now with an FDR of 0.0001. The occipital generator starting around 100 ms is the only generator reaching significance at this threshold, and thus reflects by far the most consistent difference between Figure and No Figure processing. 

Supplementary Figure 2 - Supplementary Figure 2 

Supplementary Figure 2. Figure minus No Figure activity midline electrodes. Separate spatiotemporal map of midline electrodes Pz, Cz and Fz which were not covered in the spatiotemporal map of Figure 6A. As can be seen from this map there is no major activity on these midline electrodes that is not covered in the spatiotemporal map of Figure 6A. 

Acknowledgments

We would like to sincerely thank Anne Koeleman for her efforts in collecting the data for this experiment. 

Commercial relationships: none. 

Corresponding author: Johannes J. Fahrenfort. 

Email: [e-mail]. 

Address: Roetersstraat 15, Room 6.21, 1018 WB Amsterdam, The Netherlands.