Mapping the route to visual awareness

Abstract

The “neural correlate” of perceptual awareness is much sought-after. Here, we present an novel approach to the identification of possible neural correlates, in which we exploit the temporal connection that inevitably links the selection process that determines what we become aware of, and the development of awareness itself. Because the speed of selection determines when downstream processes can first become involved in generating awareness, the latency of neural processes provides a way to isolate the neural correlates of awareness. We recorded event-related potentials (ERPs) while observers carried out a visual behavioral task designed to estimate attentional selection latency. We show that within-task trial-by-trial behavioral variability in attentional selection latency correlates to trial-by-trial variability in ERP latency. This was true in a posterior contralateral region, and in central and frontal areas, thereby implicating these as waypoints along which visual information flows on the way to visual awareness.

Introduction

The neural mechanisms involved in visual processing in the human brain are relatively well understood. A considerable number of specialized visual areas have been identified, and various experimental approaches have contributed to characterizing the roles and specific functional properties of each of these areas. Research into (putatively) homologous visual areas in non-human primates has provided further insight into visual processing mechanisms at the neural level, making the visual system probably the best understood of the mammalian sensory systems. However, although the neural mechanisms underlying perceptual processing are therefore well understood, the neural processes subserving perceptual awareness remain largely unknown. 

The brain achieves visual awareness extremely rapidly. In daily life, we rarely notice any delay incurred by neural processing: When we view our own limbs during the execution of an action, for example, the visual feedback appears instantaneous. Likewise, in laboratory settings, the visual information necessary to distinguish even complex visual categories such as animate or inanimate objects (Thorpe, Fize, & Marlot, 1996) and natural or man-made environments (Joubert, Rousselet, Fize, & Fabre-Thorpe, 2007) is available extremely rapidly. Humans can make manual behavioral responses to such tasks within about 400 ms (VanRullen & Thorpe, 2001a; perception). In fact, the majority of this time is spent preparing the motor output associated with the response: Neural measures demonstrate that the visual category is established substantially earlier (within ∼150 ms; Thorpe et al., 1996). Recently, even this time window was shortened when Kirchner and Thorpe (2006) demonstrated that observers can reliably make eye movements to an object from a specific semantic category within as little as 120 ms. 

This ultrarapid visual categorization limits the number of processing stages that information might undergo on the way to awareness and suggests that visual processing involves substantial automatic feedforward mechanisms. However, not all of the information cascading through this processing pipeline necessarily reaches visual awareness. Indeed, phenomena such as inattentional and change blindness (Simons, 2000; Simons & Levin, 1997, 1998) indicate that although we might introspectively feel that we are aware of our entire visual environment, in fact only a very limited subset of incoming sensory information actually reaches awareness. One influential model proposes that the initial rapid feedforward sweep is followed (for some representations) by feedback from higher to lower association areas and that this recurrent processing is the neural hallmark of a sensory representation achieving awareness (Lamme, 2003). However, what determines which neural representations are processed recurrently and which are not? Some kind of selection process must clearly determine which information we become aware of and which is lost by the wayside. 

The nature of this selection process forms the crux of substantial debate. Selection takes place at several levels in the visual hierarchy. Early visual areas are sensitive to high spatial and temporal frequency information, for example, whereas such information is not consciously visible (He & MacLeod, 2001). For the most part, the observer has no conscious control over such selection mechanisms. However, an important part of the selection process is under conscious control. Evidence from change blindness and inattentional blindness suggests that attention plays a crucial role in this selection stage: When attention is diverted to a different stimulus or task, observers fail to notice even dramatic changes to visual stimuli (e.g., Rensink, O'Regan, & Clark, 1997). In this interpretation, attention plays a crucial role in determining which visual information is destined to reach awareness and which is to be lost by the wayside. 

Further evidence that attention is involved in gating consciousness is the century-old notion of prior entry: that “the object of attention comes to consciousness more quickly than the objects which we are not attending to” (Spence & Parice, 2009; Titchener, 1908). Indeed, experiments show that attended stimuli are perceived to precede unattended stimuli by roughly 30–60 ms (see Spence & Parice, 2009, for a review). Behavioral responses are similarly faster for attended than unattended locations, objects, or features (e.g., Posner, 1980). 

Comparable effects of attention on the latency of neural processes however remain contentious. ERP components evoked by attended and unattended stimuli, although often exhibiting different amplitudes, generally have similar latencies (McDonald, Teder-Salejarvi, Di Russo, & Hillyard, 2005). Where systematic difference have been demonstrated, these are typically an order of magnitude smaller than perceptual effects (i.e., <10 ms; Vibell, Klinge, Zampini, Spence, & Nobre, 2007). ERP components whose latency does covary with overt behavioral measures (such as reaction time) have been reported, but these components (e.g., P300; Kutas, McCarthy, & Donchin, 1977, and the sustained posterior contralateral negativity, SPCN, Brisson & Jolicoeur, 2008) are generally relatively late (300–500 ms post-stimulus), putting them well outside the ∼200-ms time window in which perceptual awareness can be achieved (e.g., Thorpe et al., 1996). As such, a neural correlate of the mechanism that guards access to conscious awareness remains elusive. 

Recently, Gosselin and Schyns (2001) provided a number of important insights into the path to visual awareness of faces using a reverse correlation approach they termed Bubbles. By recording ERPs during the viewing of random subsets of visual information under different task sets and correlating these to behavior, they were able to describe the time course of neural processing and describe at which time points the brain is responsive to which visual features (Schyns, Petro, & Smith, 2007; Smith, Fries, Gosselin, Goebel, & Schyns, 2009). Using a similar trial-by-trial analysis strategy, Philiastides and Sajda (2006) explored the time course of decision making on the basis of visual information. Both groups identified EEG activity in two time windows: an early time window around 170 ms during which sensory processing takes place and a later time window around 300 ms, which showed task-specific modulations (Smith et al., 2009). Whereas the latency of the first component is constant, the latency of the second is variable and depends on the amount of available evidence (Philiastides & Sajda, 2006). Altogether, these findings support a distinction between neural activity related to automatic, feedforward processing and neural activity related to information selected for conscious awareness (see also Johnson & Olshausen, 2003; VanRullen & Thorpe, 2001b). 

From a similar temporal perspective, one important empirical implication of a gateway mechanism to awareness has to date remained largely unexploited: If selection determines what reaches visual awareness, then when selection occurs will also determine when the selected stimulus reaches awareness. Logically, the speed of selection determines when downstream processes involved in generating awareness can first become active. As a result, the latency of these neural processes should depend on the latency of selection. 

Given an appropriate method to estimate when attentional selection takes place, the latency of neural processes can therefore provide a way to identify the neural correlates of awareness. One approach is to use reaction time (RT) as an index (e.g., Amano et al., 2006; Kutas et al., 1977; McCarthy & Donchin, 1981; Renault, Ragot, Lesevre, & Remond, 1982). However, RT can be affected by a number of factors unrelated to selection per se, including for example processes involved in decision making and motor control. Second, RT requires an overt behavioral response. The corresponding neural activity will overlap and potentially disguise the true neural correlates of awareness, particularly in neuroimaging approaches with limited spatial resolution, such as electroencephalography (EEG). 

We recently developed a behavioral task to estimate the latency of attentional selection without requiring observers to make speeded overt responses (Carlson, Hogendoorn, & Verstraten, 2006). In this paradigm, the observer views an array of running analog clocks. At a given moment, one of the clocks is flashed. At the end of the trial, the observer reports what the time was on that clock, when it was flashed. The difference between the reported and actual time on the flashed clock is then taken as an estimate of attentional selection latency. Because the observer simply reports what he or she perceived (similar to the attentional gating approach; Reeves & Sperling, 1986), it directly probes what the time was when the target clock was selected by attention. Importantly, because no response is required until after the trial, it evokes minimal unrelated neural activity during the crucial part of the trial. 

Here, we recorded event-related potentials (ERPs) while observers carried out this behavioral task. We show that trial-by-trial behavioral variability in attentional selection latency positively correlates to trial-by-trial variability in ERP latency in three scalp regions in particular. This suggests that these three locations form crucial waypoints along which information flows on the way to visual awareness. 

Methods

Observers

Twelve observers (aged 19–25) participated in the experiment for course credit or monetary compensation. All had normal or corrected-to-normal vision, were right-handed, and gave informed consent prior to participation. None of the observers had been previously exposed to the clock paradigm, and all were naive to the purpose of the experiment. 

Four observers were excluded from the final analysis. One was excluded due to an EEG acquisition error. A second observer was excluded because insufficient trials remained after automatic artifact rejection. Finally, two more were excluded because they showed evidence of residual eye movements even after artifact rejection (see EEG acquisition section below). 

Stimuli

Observers were seated in a dark room at a distance of approximately 50 cm from a LaCie ElectronBlue 22″ color monitor (1024 × 768 resolution, 75-Hz refresh rate) controlled by a PC running Matlab 7.01 (The MathWorks, Natick, MA) with PsychToolbox 2.54 extensions (Brainard, 1997; Pelli, 1997). The stimulus consisted of 10 clock faces on a gray background, arranged in a circle at 7° eccentricity from a central fixation point (Figure 1). Each clock subtended 2.5° of visual angle and featured a single hand rotating clockwise at 1 cycle/s. The initial positions of the clock hands on each trial were randomly determined. 

Figure 1

 

Stimuli and procedure in the baseline and shift conditions.

Procedure

At the start of each trial, observers fixated a fixation point at the center of a blank background. All 10 clocks then appeared simultaneously. At a random time between 1 and 2 s after clock onset, one of the clocks was cued by turning the rim of the clock face from black to bright red for 40 ms. The ten clocks were presented for a total of 2.5 s, after which they were replaced by a single, centrally presented clock. The observer could adjust the position of the clock hand using two buttons. The task was to report the time he or she perceived in the cued clock when it was cued. We have previously shown that the method generates precise estimates of attentional shift time that correspond well with the literature (Carlson et al., 2006) and are robust to variations across subjects and experimental paradigms (e.g., Hogendoorn, Carlson, VanRullen, & Verstraten, 2010; Hogendoorn, Carlson, & Verstraten, 2007). Observer responses were not speeded, and trials were self-paced. Observers completed seven blocks of 100 trials in each of two conditions: a baseline condition and a shift condition. Blocks from the two conditions were completed alternately, and the first block of each condition was discarded as a practice block. 

The only difference between the two conditions was the observer's prior knowledge of the location of the target clock. In the baseline condition, the clock that was to be reported on each trial was indicated before the trial by a small arrow extending from the fixation point. The arrow was presented for 1 s and removed from the screen 1 s before the clocks appeared. In this way, observers could covertly direct their attention to the target clock before the trial started, and thus, no attentional shift needed to be made during the trial. In the shift condition, none of the clocks was indicated before the trial. Trials in the two conditions were otherwise indistinguishable. We deliberately implemented a relatively long delay between the pre-cue and the onset of the stimulus to ensure any transient neural activity resulting from the pre-cue would have passed before the critical phase of the trial. 

The behavioral dependent variable of interest in both conditions was the difference between the veridical position of the hand on the target clock when it was cued and the hand position reported by the observer after the trial: that is, the observer's response error. Reported positions up to 180° ahead of the clock hand were taken as positive errors, and reported positions up to 180° behind the clock hand were taken as negative errors. Although in principle the circular nature of the response could confound large positive and large negative errors, in practice observers' response distributions were sufficiently narrow to negate this problem. 

Presenting a single cue confounds lateralized attention-related responses with lateralized sensory responses. Presenting two cues, and varying which of the two was the target on different trials, would have allowed us to analyze lateralized ERP components such as the N2pc (Woodman & Luck, 1999). Nonetheless, we presented one rather than two cues in order to maximize the exogenous salience of the cue and to avoid the possibility that the top-down task set might introduce unwanted temporal variability. Furthermore, we aimed to identify the neural processes subserving awareness and anticipated that those might have central or bilateral distributions. 

EEG acquisition

During the experiment, we recorded a 64-channel electroencephalogram (EEG), in addition to a horizontal and a vertical electroocculogram (EOG), using a Biosemi EEG amplifier and ActiView software (Biosemi, Amsterdam, The Netherlands) sampling at 1024 Hz. 

We analyzed the data using MATLAB 7.01 with EEGLab extensions (Delorme & Makeig, 2004). Data were referenced to the average of the left and right mastoid electrodes, and then resampled to 250 Hz, low-pass filtered at 40 Hz, and divided into epochs time-locked to cue onset (500 ms before to 1000 ms after). To compensate for low-frequency drift, epoch baselines were equated for the 200 ms preceding cue presentation. 

Epochs from all observers were inspected for ocular artifacts on the basis of EOG traces using a two-stage process. First, all epochs were inspected visually and automatically for EEG artifacts, including blinks, saccadic eye movements, drift, and discontinuities. In a second stage, to assess systematic eye movement activity toward the cued clock, mean horizontal EOG traces (time-locked to cue presentation in the shift condition and time-locked to both cue and pre-cue presentations in the baseline condition) were calculated separately for trials in which the cued clock was presented in the left or right visual field (Woodman & Luck, 2003). Two observers were excluded from analysis because residual horizontal EOG activity time-locked to presentation of the cue exceeded 3 μV. Overall, an average of 93% of epochs from the remaining 8 observers were accepted. 

EEG analysis

Individual artifact-free EEG epochs were divided into ten groups on the basis of the location of the cued clock on each trial. Within each observer and each group, epochs were rank-sorted according to the observer's response error. Rank-sorted epochs were divided into sixteen bins and averaged within each bin (a procedure known as Vincentization; Ratcliff, 1979) to form 16 ranked average epochs per clock location per observer. We sorted EEG epochs separately for each clock location in order to ensure that clock locations were evenly assigned to each of the ranked bins. Attentional resolution has been reported to vary between the lower and upper visual fields (He, Cavanagh, & Intriligator, 1996). A number of ERP components are similarly sensitive to the positive of a target (e.g., Luck, Girelli, McDermott, & Ford, 1997; Perron et al., 2009). Sorting EEG epochs irrespective of clock position might therefore have artificially introduced trial-wise variation in the ERP. 

Epochs in which the cued clock was presented in the right visual field were reflected across the sagittal midline in order to isolate any lateralized effects. Ranked epochs were then collapsed across all ten clock locations, leaving 16 ranked epochs per observer. Mean observer response errors were calculated as estimates of attentional selection latency for each of the 16 bins. Trials in the first and last bins were excluded from further analysis because the range of response errors represented in these bins was substantially higher than in the central 14 bins. The final result of this procedure is a set of 14 independent, ranked epochs for each of eight observers. 

Finally, for each electrode position and each observer, ERP latency was defined as the point in time at which 50% of the relative peak amplitude had been reached. We deliberately used this universal, non-specific definition in order to be able to compare electrode sites with the same analysis procedure. Latencies were averaged across observers for each electrode, separately for each of the 14 ranked epochs. 

Results

Behavior

Mean response error was 42 ms in the baseline condition and 109 ms in the shift condition. A two-tailed paired-samples t-test showed this difference to be highly significant (t(7) = 10.53; p < 0.001). Mean within-observer standard deviation was 98 ms in the baseline condition, which did not differ significantly from the shift condition (mean 117 ms; t(7) = 1.29; p = 0.24). Mean distributions of behavioral responses in baseline and shift conditions are shown in Figure 2. 

Figure 2

 

Behavioral response distributions in the baseline and shift conditions.

Analysis of behavioral results for separate clock locations showed systematic variability in response error as a function of clock location in the shift condition. After collapsing horizontally symmetrical clock locations, a repeated measures analysis of variance on response errors from clocks from each of the 6 possible vertical positions showed a significant effect of vertical clock position (F(5) = 3.487, p = 0.012). Response error was lowest for the bottom position (94 ms) and increased steadily as the target clock was presented higher on the screen (103 ms, 108 ms, 116 ms, and 117 ms), although response error on the top clock was again lower (103 ms). No such effect of clock position was evident in the baseline condition in which no attentional shift needed to be made (F(5) = 1.437, p = 0.237). 

Response errors on individual trials are an indirect measure of attentional selection latency on that trial. A large proportion of the variance in response error can be attributed to the limited precision with which observers perceive, remember, and report the target clock. Negative response errors (i.e., reports that are anti-clockwise to the true position of the clock hand) therefore do not necessarily imply a (highly implausible) negative attentional selection latency. Nonetheless, trials with relatively high (i.e., clockwise) response errors are more likely to have greater attentional selection latency than trials with relatively low (i.e., anti-clockwise) response errors and vice versa. 

In the shift condition, observers were required to shift attention to the target clock before attentional selection could take place. As such, any variance in attentional shift time will contribute to variance in attentional selection latency and, thereby, to the total variance in response error. In line with this, the mean within-observer standard deviation of response errors was slightly larger in the shift condition. Although the difference did not reach significance here, we have shown previously that variability in response error is reliably lower when the target clock is indicated before the trial (Hogendoorn et al., 2010). Any systematic variation in the latency of neural responses related to attentional selection latency can therefore be expected to be more pronounced in the shift condition than in the baseline condition. 

Neurophysiology

For each electrode position, we calculated the correlation between the latency of the dominant ERP waveform at that electrode and the mean behavioral response error. The proportion of explained variance at each electrode position is plotted in Figure 3. In the baseline condition, there were no significant correlations between ERP latency and behavioral responses. In the shift condition, however, we observed strong positive correlations at three scalp regions in particular: (1) a large posterior contralateral region encompassing P3/4, P5/6, PO3/4, and PO7/8 (r = 0.60, r = 0.61, r = 0.62, and r = 0.64, respectively; all p < 0.024); (2) a focal central region centered on Cz (r = 0.72, p = 0.004); and (3) a symmetric, broad frontal region extending anterior including Fz and both contra- and ipsilateral F1/2 (r = 0.58, r = 0.56, and r = 0.58, respectively; all p < 0.039; Figure 3). 

Figure 3

 

Scalp maps showing the proportion of variance in ERP onset latency explained by behavioral estimates of attentional selection latency. Left and right hemispheres are ipsi- and contralateral to the target clock, respectively. In the shift condition only, three regions in particular show strong correlations between ERP latency and behavior: a posterior contralateral region, Cz, and a frontocentral region. Activity in these regions is highly correlated in time to the latency of attentional selection, suggesting that neural information flows through these regions on the way to awareness.

Figure 4 shows mean event-related potentials evoked by the presentation of the cue in both the baseline and shift conditions for each of these three scalp regions. Separate average waveforms are shown for the seven bins with the lowest response error and the seven bins with the highest response error. Although no latency differences are evident in the baseline condition at any of the electrode sites, in the shift condition onsets of the dominant waveforms are delayed in trials with high response error. 

Figure 4

 

Evoked event-related potentials in three scalp regions: a posterior contralateral region encompassing the contralateral electrode of the PO3/4, PO7/8, P3/4, and P5/6 electrode pairs (top), Cz (middle), and a frontal region encompassing Fz, F1, and F2 (bottom). Left panels depict the baseline condition and right panels depict the shift condition. ERPs evoked during trials with below-median response error are depicted in red; ERPs evoked during trials with above-median response error are depicted in blue. Latency differences are evident in the shift condition but not in the baseline condition.

In the shift condition, highly significant positive linear trends related behavioral estimates of attentional selection latency to the onset latency of the dominant waveform in posterior contralateral, central, and frontal scalp regions. Figure 5 shows ERP onset latency as a function of behavioral response error for each of these three regions. Slope coefficients (with 95% confidence intervals) of the linear best fit are 0.09 (CI = 0.02–0.15) over the posterior contralateral scalp region, 0.09 (CI = 0.03–0.14) at Cz, and 0.07 (CI = 0.01–0.12) over frontal electrodes. No significant correlations were evident in the baseline condition for any of these areas. 

Figure 5

 

ERP onset latency in three scalp regions as a function of response error in the baseline condition (top) and in the shift condition (bottom). Error bars depict standard errors of the means across observers. Dashed lines denote linear best fit functions. Whereas in the baseline condition correlations between behavior and ERP latency are non-significant, in the shift condition clear, positive linear trends relate behavior and ERP onset latency. **p < 0.01.

Discussion

We investigated the neural route to visual awareness by linking behavioral and neural measures of attentional selection latency. When observers were required to shift their attention, we were able to identify three scalp regions at which the latency of the dominant evoked ERP waveform correlated with attentional selection latency: a posterior contralateral region, Cz, and a frontal region. In these three areas, early attentional selection was associated with low neural latencies and vice versa. No such relationship between behavioral and neural latencies was evident in a baseline condition in which observers did not shift their attention. Attention has been argued to fulfill a gatekeeper role for visual awareness. Accordingly, activity in these three areas likely reflects the selection and transmission of visual information on the way to visual awareness. 

Previous work has implicated early lateralized ERP activity at PO7/8 (such as the N2pc; Woodman & Luck, 1999) in attentional selection. The posterior contralateral locus therefore might reflect neural processes involved in attentional selection itself. The fact that the activity evoked by trials with early and late attentional selection starts to diverge relatively early (about 200 ms post-cue; Figure 4, top right) lends additional credence to this idea. On average, in the shift condition, observers reported a hand orientation about 110 ms later than veridical. If attentional selection takes place about 200 ms post-cue but the stimulus that is selected is only 110 ms later than the cue, then the level of representation at which attentional selection takes place is about 90 ms old. This is in good agreement with the time necessary for visual information to first reach extrastriate areas (40–100 ms; Di Russo, Martinez, Sereno, Pitzalis, & Hillyard, 2002). As such, it seems plausible that contralateral occipitoparietal cortex is involved in attentional operations (specifically selection) and, thereby, forms the first waypoint on the route to visual awareness. Following this line of reasoning, the central and frontal loci are likely to reflect activity downstream to the selection stage evident at the posterior contralateral area. One possibility is that activity at these two areas reflects the encoding of the selected stimulus into working memory and the subsequent availability of that information for higher order cognitive processes. 

An alternative possibility is that the systematic activity at each region reflects recurrent processing and feedback interactions rather than specific cognitive constructs such as attention. This is in accordance with the view of awareness advocated by Lamme (2000, 2003; also Lamme & Roelfsema, 2000). The striking similarity between the trends relating behavioral and neural latencies at posterior contralateral and central regions (Figure 5, lower panels) further supports the position that activity in both locations reflects an interaction of the two areas. 

By demonstrating correlations between the latency of evoked ERPs and behavioral estimates of attentional selection latency, we were able to map out putative neural correlates of visual awareness. Our results show that a contralateral occipitoparietal region, a central region, and a symmetrical frontal area show time-locked activation to behavioral measures of attentional selection. This temporal link between neurophysiology and behavior strongly suggests that these regions are involved in realizing awareness, reflecting either the attentional selection mechanism itself or another neural process further downstream on the way to awareness. 

Acknowledgments

This work was supported by the Netherlands Organisation for Scientific Research (NWO-MAGW Pioneer Grant). We are indebted to Marjolein Kammers, Steven Luck, Pierre Jolicoeur, Phillippe Schyns, and an anonymous reviewer for invaluable feedback on previous versions of the manuscript. 

Commercial relationships: none. 

Corresponding author: Hinze Hogendoorn. 

Email: [e-mail]. 

Address: Helmholtz Institute, Utrecht University, Experimental Psychology Division, Heidelberglaan 2, 3584 CS Utrecht, The Netherlands.