Neural correlates of object-based attention

Abstract

Much research has been directed toward disentangling the “units” of attention: Is attention directed to locations in space, visual objects, or to individual features of an object? Moreover, there is considerable interest in whether attention increases the gain of neural mechanisms (signal enhancement) or acts by other means, such as reducing noise or narrowing channel tuning. To address these questions, we used a direct measure of signal strength: the amplitude of visual evoked potentials and a task in which selection could be based on a depth order cue but not on location. Attended and nonattended stimuli were presented at different temporal frequencies, and, thus, responses to the two stimuli could be analyzed separately even though they were presented simultaneously. Attention increased the amplitude of the second harmonic component of the response, but not the fourth harmonic. In addition, responses measured at the second harmonic, but not at the fourth harmonic, were larger for stimuli seen as behind. The results are consistent with the fourth harmonic being generated at a stage of processing that is not accessible to attention and where depth order has not been extracted. The second harmonic, on the other hand, is modifiable by attention and shows evidence for differential encoding of depth order.

Introduction

Several theories of attention postulate that attention is organized around space and works like a “spotlight” that moves across the visual field (Eriksen & Eriksen, 1974; Posner, 1980; LaBerge & Brown, 1986; Eriksen & St James, 1986; Treisman, 1988; for reviews, see Cave & Bichot, 1999; Kanwisher & Wojciulik, 2000). Several lines of evidence support this view. Performance measured through accuracy and reaction time is better for a stimulus presented at an attended region of space (Posner, 1980; Posner, Snyder, & Davidson, 1980; Bashinski & Bacharach, 1980; Muller & Findlay, 1987; Downing, 1988), and event-related potential (ERP) studies have shown enhanced processing for stimuli presented to attended locations in the form of increased evoked response amplitudes (Posner et al., 1980; Hillyard & Munte, 1984; Mangun & Hillyard, 1987). 

Other authors have argued that visual attention operates on object features, such as motion, shape, color, and orientation, suggesting that features rather than spatial locations are the primary cues that engage the attentional process (Hillyard & Munte, 1984; Corbetta, Miezin, Dobmeyer, Shulman, & Petersen, 1990; Roelfsema et al., 1998; Baldassi & Burr, 2000). Duncan (1984) suggested that attention is directed to segmented objects. Evidence from both human and animal studies has supported this theory (Roelfsema, Lamme & Spekreijse, 1998; Valdes-Sosa, Bobes, Rodriguez, & Pinilla, 1998; for reviews, see Posner and Driver, 1992; Scholl, 2001). It has also been argued that location- and object-based attention mechanisms are not mutually exclusive, but they are alternatively evoked by different task demands (Vecera & Farah, 1994), and more recent theories have incorporated both location- and object-based types of selection (Treisman, 1993; Desimone & Duncan, 1995). 

Within the object/feature selection model, it is difficult to distinguish experimentally between the effects related to features of objects as opposed to the object per se. Moreover, the use of subjective recall to measure processing of unattended stimuli can confound attentional effects with memory effects (cf., Driver, 2001). More direct, neurally based measures, such as fMRI and the ERP, present alternative methodologies for assessing processing within attended and nonattended streams. O’Craven, Downing, and Kanwisher (1999) showed that attention to objects (houses and faces) produced higher neural activity in brain areas specialized for processing that kind of information (parahippocampal “place area” and fusiform “face area”). This experiment excluded the spatial attentional cue by superimposing the attended and nonattended stimuli. A second strength of their paradigm lay in the fact that they could measure, at the same time, the response to a feature of the objects (motion) that was not relevant for the primary task, as well as responses related to the objects per se. 

ERPs have been used extensively to study neural activity generated by attended and nonattended stimuli. Early studies used transient ERPs and manipulated spatial attention (Hillyard & Munte, 1984; Mangun & Hillyard, 1987). In those studies, a flash of light or grating was presented to one hemifield and the subjects were cued to the side where the stimuli would subsequently appear. Spatially directed attention to a cued stimulus produced an enhancement of particular components of the signal (P1/N1), suggesting an active gain control mechanism (for a review, see Mangun, 1995). 

Attentional effects have also been demonstrated using the steady-state response evoked by flickering, spatially cued stimuli (Morgan, Hansen, & Hillyard, 1996; Muller et al., 1998; Di Russo & Spinelli, 1999; Muller & Hillyard, 2000; Di Russo, Spinelli, & Morrone, 2001). In Morgan et al. (1996) and Muller et al. (1998), task-relevant and task-irrelevant stimuli were temporally modulated at different temporal frequencies, and responses were separately measured at the corresponding response frequencies. Enhancement of the steady-state signal was found for stimuli presented at the cued spatial location. In DiRusso and Spinelli (1999) and DiRusso et al. (2001), flickering chromatic or achromatic gratings were presented in one hemifield. In one condition, attention was directed to that hemifield through a contrast discrimination task while visual evoked potentials (VEPs) were measured. In another condition, attention was directed to the opposite hemifield by having the observers perform a letter discrimination task. Attention directed to the hemifield with the test stimuli resulted in larger signal amplitudes for both chromatic and achromatic gratings and marginally faster response latencies for the achromatic gratings. 

More recently, Valdes-Sosa, et al. (1998) found evidence for object-based ERP effects using superimposed stimuli that could not be selected on the basis of location. In that study, two transparent, rotating dot fields were presented, one comprised of red dots and the other of green dots. After a period of oppositely directed rotations, one of the fields translated linearly and the motion onset VEP to that motion was recorded. The observers were cued as to which color stimulus was task relevant (direction discrimination), and VEPs were measured to displacements that occurred for either task-relevant or task-irrelevant stimuli. Valdes-Sosa, et al. (1988) found a strong suppression of P1/N1 components related to the unattended motions. Attentional modulation was weak or absent in control conditions that did not elicit the percept of separate transparent surfaces. Torriente, Valdes-Sosa, Ramirez, and Bobes (1999) presented random element patterns that split into moving and static subsets that were spatially interspersed. The observers were given a task that required attention either to the static or moving elements. The amplitude of the N170 component to motion was reduced when attention was directed to the static elements. 

Here we combined the temporal tagging method used by Morgan et al. (1996) and Muller et al. (1998) with a motion display that used occlusion cues to segregate overlapping stimuli into different apparent depth planes. The advantage of the tagging method is that one can directly monitor processing in the attended and unattended streams simultaneously and with high precision for response dynamics. Rather than allowing spatial selection, as in by Morgan et al. (1996) and Muller et al. (1998), we provided features that supported the perceptual segregation of the attended and nonattended targets into separated depth planes. This combination of features will constitute our operational definition of an “object” in this study. 

Methods

Observers

Thirteen adults (6 males and 7 females, aged 21 to 50 years) participated. Two participants were excluded from the analysis because their signal-to-noise ratio was too low. The exclusion was blind to the effects of attention or any of the other dependent measures. Each observer had 6/6 or better acuity in each eye, normal stereopsis on the Frisby free-space stereo-test, and was fully refracted for the viewing distance. The research followed the tenets of the World Medical Association Declaration of Helsinki, and informed consent was obtained from the subjects after explanation of the nature and possible consequences of the study. The local institutional human experimentation committee approved the research. 

Stimulus

The display was generated using conventional raster graphics (800 × 600 pixels at 72 Hz refresh) on a monochrome computer monitor. The stimulus consisted of a series of 16 crosses spread across an 18.5 × 24-deg display with a background luminance of 50 cd/m². Each bar was 3 deg high and 0.3 deg wide. The center-to-center spacing of the crosses was 6.03 deg. The bars were oscillating sinusoidally at two different but very close temporal frequencies (TF) (Movie 1). This allowed us to analyze the data separately for the two different components even though the responses were being generated simultaneously in the attended and nonattended streams. The vertical bars moved together right and left at TF1 = 2.4 Hz, and the horizontal bars moved up and down at TF2 = 3.0 Hz. The relative amplitude of the motion was 40% of the bar length (1.2/deg peak-to-peak motion centered the 3 deg bars). 

 

Movie 1

 

This movie is an example of one of the eight conditions that we used in our experiment showing how the two components were moving.

We counterbalanced luminance (5, 95 cd/m²) and depth order (in front, behind) of the vertical and horizontal bars in order to obtain four different stimulus configurations (Figure 1). Because the luminance of the bars differed, we could assign one of the colors to occlude the other, creating an unambiguous monocular cue for depth order. 

Figure 1

 

The figure shows the four different configurations of the stimuli used in our experiment. The vertical and horizontal bars were matched for luminance and depth order. The vertical bars (TF1) oscillated rightward and leftward at 2.4 Hz while the horizontal bars oscillated upward and downward at 3.0 Hz (TF2). We collected data for each of these four stimuli in two different conditions, paying attention to the vertical bars paying attention to the horizontal bars.

Experimental Procedure

For each configuration described above, we collected data in two different attentional conditions (attend vertical bars or attend horizontal bars) for a total of eight conditions. Each trial presentation lasted 8.3 s, and we recorded 10 trials for each condition. Trials were run in blocks of 5, and each block was repeated twice with the order of presentation of each block being randomized. 

Before starting each trial, the observer was instructed to attend to either the vertical or the horizontal bars while maintaining fixation in the center of the display, without making any eye movements. The observers were also instructed to withhold eye blinks during the trial. The observers initiated trials with a button and could interrupt the trials or abort them as needed to retain blink-free fixation. Each observer was previously trained with about 4 to 6 trails before starting the session to be sure they understood the task and could perform it correctly. 

VEP Recording and Analysis

The brain electrical activity was recorded with Grass gold-cup surface electrodes. Electrode impedance was between 3 and 10 kilo-ohms. We recorded from a chain of 5 electrodes placed over the posterior occipital lobe, each referenced to linked ears. The central electrode was placed at the midline 3 cm above the Inion. Two electrodes were placed 3 and 6 cm laterally on each side of the midline electrode from left to right; these derivations will be referred to as O3, O1, Oz, O2, and O4. The inter-electrode distance was 3 cm (see Figure 2). 

Figure 2

 

The image shows schematically the electrode positions on a head seen from above. The posterior part of the head is on the bottom of the figure. Oz was placed 3 cm above the Inion. O3, O1, O2, and O4 are the occipital sites (channels), and the distance between each site was 3 cm starting from Oz. A1 and A2 are the two reference electrodes placed on the ear lobes.

The electroencephalogram (EEG) was amplified by 50,000 times with Grass Model 12 amplifiers and was digitized to 16 bits accuracy at a sampling rate of 432 Hz. Analog filter settings were 0.3 to 100 Hz, measured at −6 dB points. We averaged the data for each subject for all the trials related to the eight separate conditions. Responses were isolated from the EEG by analyzing the distinct temporal harmonic components generated by the vertical (1F1, 2F1, 4F1…) and the horizontal bars (1F2, 2F2, 4F2…). Spectrum analysis at these harmonics was performed with an adaptive matched filter technique using a recursive least squares adaptive filter (Tang & Norcia, 1995). Complete spectra (e.g., Figure 3) were computed with a mixed-radix discrete Fourier Transform (dft routine; MATLAB, Mathworks, Inc.). 

Figure 3

 

Example of spectral analysis of data from a single observer. The data are from the Oz derivation recorded under two different conditions: attending to vertical (the upper, orange spectrum) and attending to horizontal bars (the bottom, blue spectrum). The vertical bar oscillated at TF1 = 2.4 Hz and the horizontal bars oscillated at TF2 = 3.0 Hz. Attention increased the amplitude of the second harmonic of the attended stimulus relative to the unattended stimulus.

Results

Because each of the two stimulus components was tagged by a different temporal frequency, we could measure the responses generated by both the attended and the nonattended stimulus components in the same condition at the same time. Figure 3 shows the spectrum of the response for a single observer. 

We concentrated our data analysis on the second and the fourth harmonics because these even harmonics dominate the response to symmetric oscillatory motion. As can be seen in Figure 4 (top panel), evoked response amplitude at the second harmonic component was maximal at Oz for both TF1 and TF2 stimuli (error bars are ±1 SEM). However, the effect of attention was present on all channels. We did not find an effect of attention at the fourth harmonic. 

Figure 4

 

Plots of the amplitude of the VEP signal for the second harmonics (top panel) and fourth harmonic (bottom panel) as a function of different channels. Data averaged across subjects. On the left side are plotted the signal related to the vertical bars (2F1 and 4F1) when they were attended (blue symbols) and when they were nonattended (red symbols). On the right is plotted the signal related to the horizontal bars (2F2 and 4F2) when they were attended (green symbols), and when they were nonattended (orange symbols). The experimental noise levels are shown as dotted lines. The noise estimates were calculated as the mean amplitudes at a pair of adjacent nonharmonic frequencies. These frequencies were located ± 0.6 Hz from the response frequency of interest (see Norcia, Sato, Shinn, & Mertus, 1985; Norcia, Hamer, Jampolsky, & Orel-Bixel, 1995).

The effects of attentional instruction, depth order, luminance, and orientation of the bars were analyzed using a multi-variate approach to repeated measures analysis (multi-variate analysis of variance [MANOVA]). 

For each main effect and interaction, we computed Hotelling’s T² statistic:  

 

which, in our design, is F_(1,n-1)-distributed, where n = number of subjects; and,  

 

is the mean response vector, where y_i is the 32-element vector of scalp potentials recorded for the full set of permutations of our five design factors, for each subject, j; and,  

 

is the pooled covariance matrix; and c is a contrast vector corresponding to the particular effect being tested. 

Details and derivations can be found in Rencher (1995, p. 158). 

Data reported below are from the Oz derivation, which was chosen because it had the largest and most reliable response across observers. 

We analyzed the data separately for the two harmonics of interest (second, fourth). Earlier pilot studies indicated that temporal frequency and bar orientation variables do not interact with attention. The error bars represented in the graphs are always ±1 SEM. 

First, we analyzed the data related to the four different stimulus configurations, collapsing across the attention variable. Figure 5 plots mean amplitude data for the second and fourth harmonics as a function of frequency (TF1, TF2) and relative luminance. There was no main effect of luminance, nor were there any significant interactions involving the luminance variable (see Figure 5). However, the same analysis for the variable depth order showed an increment of the second harmonic signal amplitude related to the tagged component that is behind (see Figure 6). This effect was not present for the fourth harmonic. Overall, there was a significant main effect of depth order (F_(1,10) = 22.4; p = .0008) with amplitudes larger for responses generated by occluded stimuli. We found a significant interaction involving depth and harmonic (F_(1,10) = 19.36; p = .001) reflecting depth order effects present only at the second harmonic. Additionally, there was a significant depth-by-frequency interaction (F_(1,10) = 14.1; p = .004). At the second harmonic, the F2 stimulus generated a relatively large response when it was behind (see Figure 6). 

Figure 5

 

Second and fourth harmonic data from Oz averaged across subjects. The two graphs show the amplitude of the signal at the second harmonic and at the fourth pooled across the variable attention when the bars were 5-cd/m² luminance (green bars) or 95-cd/m² luminance (gray bars). There are no effects of luminance.

Figure 6

 

Data from Oz averaged across subjects. The two graphs show the amplitude of the signal at the second harmonic and at the fourth pooled across the variable attention when the bars were behind (green bars) or in front (gray bars). The response to each temporal frequency oscillation at the second harmonic is larger when the bars are behind.

Knowing the influence of depth order and luminance on the signal baseline, we looked at the effect of the variable attention. The average of the signal amplitude across subjects for each of the 5 occipital channels was significantly larger for the attended component of the stimuli compared to the nonattended one in all of the conditions. The effect of attention as a function of frequency is shown in Figure 7 for the Oz derivation for both second and fourth harmonic components. Overall, there is a main effect of attention (F_(1,10) = 35.07; p = .0001) with the attended condition having larger amplitudes. There is an attention-by-harmonic interaction (F_(1,10) = 22.8; p = .0007) consistent with the presence of an attentional modulation of about 30% at the second harmonic that is absent at the fourth harmonic. There were no other effects involving the attention variable. Importantly, we found that both depth order and luminance have no effect in relation to the signal enhancement due to attention at the second harmonic (F_(1,10) = 0.002; p = .97 and F_(1,10) = 0.113; p = .74). Also at the fourth harmonic, none of the variables interact with attention. 

Figure 7

 

Data from Oz averaged across subjects. On the left panel, we represent the signal amplitude at the second harmonics for the vertical (2F1) and the horizontal (2F2) bars. For both 2F1 and 2F2, the attended condition (orange bars) shows significantly larger amplitude than the nonattended one (blue bars). This graph shows the enhancement of the signal with attention in our experiment. On the right side, we plotted the amplitude of the signal for both the vertical (4F1) and the horizontal (4F2) bars at the fourth harmonic. In this case, there was no difference between the attended (orange bars) and the nonattended (blue bars) conditions. Spatial Attention Control Experiment

There is a possibility, related to the spatial attention hypothesis, that the observers were paying attention just to the tips of the bars (the segments that were not overlapping at any time), say in the immediate area around fixation. It is possible, but unlikely, that attentional modulation of responses generated by this small fraction of the display would be able to swing the total VEP voltage by the amount we measured. 

In order to eliminate this possibility, we ran a control experiment in five subjects where we used the same temporal frequency of the bars and motion amplitude as in the original experiment. The difference was that in this experiment the bars themselves were shortened so that they completely overlapped over the stimulus cycle (1.2 deg motion of 1.2 deg bars). The observers were instructed to attend to either the vertical bars (F1) or the horizontal bars (F2) in separate conditions. We used only one of the configurations illustrated in Figure 1. Responses were measured at the second and fourth harmonics of the two component frequencies, and we pooled that data across temporal frequency (F1/F2) for attended and nonattended conditions. We found that observers were still able to modulate the magnitude of their responses: responses averaged 60% larger at the second harmonic for attended versus nonattended stimuli (p < .001; paired comparison t test). There was no significant effect of attention at the fourth harmonic (p = .5029), replicating the results of the first experiment. For spatial attention to act on these stimuli, the spotlight would have to be movable both with high precision (to avoid the region of the nonattended bar) and with great speed (because the faster of the bars was moving at 3 Hz). 

Discussion

Sustained attention to a component of a complex moving pattern produced an enhancement of evoked activity at the second harmonic but not the fourth harmonic. In our study, the two components were spatially overlapped, and it is difficult to imagine that a spatial location mechanism such as an attentional “spotlight” could separate them. On the other hand, the components had a well-segregated perceptual depth ordering, and attentional selection along the depth axis might underlie the effect. In this regard, it is important to note that depth order itself produced reliable differences in response amplitude that were independent of attentional instruction for the second harmonic. 

The use of the tagging VEP technique allowed us to monitor the response from the two stimulus components at the same time. Although we did not use an auxiliary behavioral task to monitor attentional performance, the fact that we measured an enhancement of the signal related to the specific temporal frequency corresponding to the attended bars means that the subjects were successful in doing the task. An auxiliary task could possibly have increased the effect, so our data may represent a lower bound on the magnitude of modulation that is possible under optimal conditions. 

Our interpretation of this pattern of results is that the fourth harmonic is generated early in the chain of visual processing at a site that is both prior to attentional access and the extraction of depth order. We can make this inference based on a general principle of neural systems: the higher harmonics are preferentially generated during the earliest part of the time-evolution of the evoked response. Those harmonics dominate the leading edge of the evoked response, which is invariably steeper than the trailing edge (cf., the time-frequency analysis of Norcia et al., 1985). We thus suggest that the second harmonic component is dominated by later processing stages where depth order has been extracted and attention can act. It is possible that attention can only act on segmented “objects,” and our attentional effect thus lies either at or after the depth-order extraction stage. Previous psychophysical studies have also suggested that attention is deployed at the level of segmented objects (Duncan, 1984; for reviews, see Driver, 2001 and Scholl, 2001). 

One could also propose that attention is needed to perform the depth segmentation. Regardless of the direction of causality, it is apparent that we have tapped two very distinct levels of processing: a preliminary stage representing itself in the fourth harmonic and a later stage that contributes to the second harmonic. 

Relationship to Previous ERP Studies

As noted in the “Introduction,” most previous ERP studies have been conducted within the spatial attention framework, with the observers being cued as to the location of the task-relevant target. Attention to location results in enhancement of early components in the P1 (approximately 80–120 msec) to N1 (approximately 150–200 msec) latency ranges, without a corresponding change in scalp topography (for reviews, see Mangun, 1995, and Anllo-Vento & Hillyard,1996). Other ERP studies have presented relevant and irrelevant stimuli sequentially to the same location with selection being based on a feature such as color (Hillyard & Munte, 1984; Anllo-Vento & Hillyard, 1996), motion (Anllo-Vento & Hillyard, 1996), or spatial frequency (Harter and Previc, 1978; Martinez, Di Russo, Anllo-Vento, & Hillyard, 2001). These studies have found an increased negativity in the 150–300 msec time-range (selection negativity [SN]) for attended stimuli, with the scalp topography of the SN being substantially different than that of N1/P1. 

In our study, using steady-state potentials, we showed that attention increased the signal strength at a neural level similar to previous studies, but that spatial attention is not required for amplitude enhancement. Within the limits of our sampling, the attentional effect does not involve substantial changes in response topography (see Figure 4). 

Of more direct relevance to this work are the experiments of Valdes-Sosa, Cobo, and Pinilla (1998), Torriente et al. (1999), and Pinilla, Cobo, Torres, and Valdes-Sosa (2001). In each of these studies, motion cues were used to elicit the percept of two overlapping objects lying in different depth planes. As noted by the authors of these studies, location-based spotlight mechanisms could not be used to select stimuli. Torriente et al. (1999) suggested that their results and, by implication, those of Valdes-Sosa, Cobo, et al. (1998) and Pinilla et al. (2001) could be interpreted within the framework of a “motion-filter” (e.g., McLeod, Driver, & Crisp, 1988; McLeod, Driver, Dienes, & Crisp, 1991; Driver, McLeod, & Dienes, 1992). In this view, stimuli of different motion characteristics activate separate motion-selective channels that can be selected via attentional mechanisms. 

As a general speculation and to stimulate further research (other than positing a filter-based model for selection of moving stimuli), it is possible to reformulate the spotlight metaphor to include the third dimension, in which case selection on the basis of perceptual depth order could comprise a variant of spatial attention. Valdes-Sosa, Cobo, et al. (1998) and Pinilla et al. (2001) have proposed an internal surface-based representation (e.g., He & Nakayama, 1995) as an alternative basis for selection, which is tantamount to an explicitly 3-D version of the spotlight model. Objects, by their very nature, exist in three dimensions and there are many monocular cues, such as the occlusion cue used in this study, that could signal a separation in depth that would be useful for attentional selection. 

Acknowledgments

This work was supported by grant EY06579 (A.M.N. and M.W.P.) from the National Eye Institute of the National Institutes of Health and by a Rachel C. Atkinson Fellowship (F.P.). A preliminary version of this work was presented at the 2001 Annual Meeting of the Optical Society of America, October 13-14, 2001, Irvine, CA. Thanks to Stefano Baldassi for helpful comments and advice. Commercial Relationships: None.