EEG precursors of detected and missed targets during free-viewing search

Abstract

Abstract

Abstract:

Abstract  When scanning a scene, the target of our search may be in plain sight and yet remain unperceived. Conversely, at other times the target may be perceived in the periphery prior to fixation. There is ample behavioral and neurophysiological evidence to suggest that in some constrained visual-search tasks, targets are detected prior to fixational eye movements. However, limited human data are available during unconstrained search to determine the time course of detection, the brain areas involved, and the neural correlates of failures to detect a foveated target. Here, we recorded and analyzed electroencephalographic (EEG) activity during free-viewing visual search, varying the task difficulty to compare neural signatures for detected and unreported (“missed”) targets. When carefully controlled to remove eye-movement-related potentials, saccade-locked EEG shows that: (a) “Easy” targets may be detected as early as 150 ms prior to foveation, as indicated by a premotor potential associated with a button response; (b) object-discriminating occipital activity emerges during the saccade to target; and (c) success and failures to detect a target are accompanied by a modulation in alpha-band power over fronto-central areas as well as altered saccade dynamics. Taken together, these data suggest that target detection during free viewing can begin prior to and continue during a saccade, with failure or success in reporting a target possibly resulting from inhibition or activation of fronto-central processing areas associated with saccade control.

Introduction

In our daily lives, we are constantly moving our eyes, acquiring visual information from one fixation to the next through fast eye movements termed saccades. Their purpose is to bring objects of interest into foveal vision, which has the highest spatial acuity. While there is a certain level of arbitrariness in these eye movements, they are by no means random. What precisely guides them during visual search has been an active area of research for more than 30 years (for a review see Eckstein, 2011). 

The literature on single-fixation search—where observers are not allowed to make eye movements—indicates that search decisions can be made using the visual periphery alone (see, e.g., Baldassi & Verghese, 2002; Palmer, Verghese, & Pavel, 2000). There is also clear evidence that eye movements are guided by information about the target gleaned from the periphery (Findlay, Brown, & Gilchrist, 2001), and there is evidence that such presaccade detection is influenced by attention to specific features of the target (Verghese, 2001). During free viewing there is also clear evidence that subjects can identify a target in the periphery and before saccading to it, and that fixation serves primarily to confirm detection (Kotowicz, Rutishauser, & Koch, 2010). Moreover, neural correlates of presaccadic selection of peripheral targets have been found in the frontal (Thompson & Bichot, 2005) and supplementary eye fields (Stuphorn & Schall, 2006) using invasive recordings in nonhuman primates. 

Comparable neurophysiological data in human subjects stem from intracranial recordings (Blanke et al., 1999; Kirchner, Barbeau, Thorpe, Régis, & Liégeois-Chauvel, 2009; Lachaux, Hoffmann, Minotti, Berthoz, & Kahane, 2006), transcranial magnetic stimulation (Campana, Cowey, Casco, Oudsen, & Walsh, 2007; Grosbras & Paus, 2002; Muggleton, Juan, Cowey, & Walsh, 2003; Ro, Farnè, & Chang, 2003), and functional magnetic resonance imaging (Bressler, Tang, Sylvester, Shulman, & Corbetta, 2008; Cornelissen et al., 2002). However, these studies involve controlled search tasks often requiring fixation and covert shifts of attention. Similarly, as a consequence of large saccade-related potentials, encephalographic recordings from humans have remained limited to highly constrained scenarios, for example requiring fixation or a very constrained repertoire of saccades (see, e.g., Dandekar, Ding, Privitera, Carney, & Klein, 2012; Drewes & VanRullen, 2011; Gutteling, van Ettinger-Veenstra, Kenemans, & Neggers, 2010; Hilimire, Mounts, Parks, & Corballis, 2011; Hinkley, Nagarajan, Dalal, Guggisberg, & Disbrow, 2011; Kelly, Foxe, Newman, & Edelman, 2010; Ptak, Camen, Morand, & Schnider, 2011). Notable exceptions are Dandekar, Privitera, Carney, and Klein (2012) and Kamienkowski, Ison, Quiroga, and Sigman (2012), who analyzed EEG during free-viewing search focusing on evoked responses following the saccade. However, to our knowledge, no neurophysiological data are available in humans to indicate the earliest moments of target detection during normal free-viewing search. In addition, the neural substrates of “inattentional blindness” (Mack & Rock, 1998)—a common phenomenon in which an unexpected object is foveated but not reported—are largely unknown (Chun & Marois, 2002). 

In the present study, we aimed to identify the earliest neural signatures of target detection during free viewing and to probe the correlates of detection failures. To this end, we simultaneously recorded EEG and eye position while subjects were engaged in an unconstrained search task of varying difficulty. We found target-related activity concurrent with saccades as well as evoked activity preceding the saccade to target by as much 150 ms over central areas, consistent with anticipatory motor preparation. Before the behavioral response to the target, 10-Hz alpha activity was reduced in fronto-central areas. Conversely, detection failures were associated with elevated alpha activity at least 1.5 s before and after the saccade. We also find that success and failure to report a target are associated with altered saccade dynamics which in turn covary with alpha power in these same areas. Together these data suggest that detection performance during free viewing may be modulated by activity in premotor areas (including frontal or supplementary eye fields), and this is reflected in both modulated alpha activity and altered saccade control. 

Methods

Subjects

Thirty-five subjects with normal or corrected-to-normal vision participated in the experiments. The average age of the subjects was 26.0 ± 6.0 years, with seven women and 28 men volunteering. All experiments were approved by the Institutional Review Board of the City College of City University of New York, and all subjects gave informed written consent before the experiment. 

Search paradigm

Subjects were instructed to freely scan an image in order to locate, as quickly as possible, a prescribed target. This may be viewed as a form of the well-known Where's Waldo? task. Each search image contained a single target among many distractors (Figure 1). Prior to each trial, the subjects were shown the target for as long as they desired to memorize its identity. After a keyboard press, an isolated fixation point appeared in the center of the screen and was used to correct for possible drift in the eye-tracker calibration. Upon stable fixation, the search stimulus appeared and subjects freely scanned the presented image. Once the target was detected, subjects were instructed to press a keyboard button. Two tasks of varying difficulty were investigated (Tasks 1 and 2). As a control, the easier task was also repeated without the manual response (Task 1B). Instead, subjects communicated target detection by maintaining fixation on the target for at least 1 s. In all instances, the trial could be aborted with a button press regardless of gaze point, thus allowing the subject to terminate the search if the target was forgotten or not found. Data from these aborted trials was not used in the analysis. In the easier task (Task 1), subjects responded with their left hand. In Task 2, responses were balanced between left and right hand by asking different subjects to use either the left or the right hand. The experiment consisted of blocks of 120 trials, with a break between blocks. Subjects were free to do one or more blocks at their discretion. The number of subjects performing Tasks 1, 1B, and 2 were N = 13, N = 17, and N = 24, respectively. 

Figure 1

 

Search tasks and typical saccade behaviors. Top panel: A trial begins with the target identity shown to the subject. After a button push, a fixation dot is presented. The search screen appears once fixation on the dot is detected. Subjects search freely until they locate the target. In Task 1 and 2, detection is reported with a keyboard press while directly looking at the target. In Task 1B the keyboard press is omitted. Bottom panels: Example of scene together with a typical saccade path during search. Categorization of saccades into “targets,” “distractors,” or “misses” is performed by comparing the fixation point with the identity of the corresponding object (see Methods for more detail).

Search stimuli

Each scene contained a total of 49 (57) items in Task 1/1B (Task 2), with both target and distractors chosen randomly from a set of four (24) elements (see Figure 1). The objects of Task 1/1B consisted of a colored (red or green) rectangular bar oriented horizontally or vertically and spanning an area of 200 pixels. In Task 2, an object was defined by the color (red, green, yellow, or blue) and order of two vertical bars (total area of 336 pixels). For each scene, distractors were drawn from a subset of four such objects, with the lone target being a left-right-reversed form of one of these. Locations for all objects were randomly selected under the constraint that only a single item could be placed within a rectangular area of 84 × 84 pixels (Task 1) or 50 × 50 pixels (Task 2). The search image covered the entire screen (resolution: 1024 × 768 pixels; screen size: 33.9 cm × 27 cm), with subjects situated approximately 60 cm from the screen. Thus, 1° of visual angle corresponded to 32 pixels. 

Eye-movement recordings

Monocular eye movements were recorded with an EyeLink 2000 system (SR Research, Kanata, Ontario, Canada) at a sampling frequency of 500 Hz, with calibration of position occurring at the outset of each block. Calibration drift was corrected with a mandatory fixation point in the center of the screen before each trial. Stimulus presentation and eye tracking were programmed using Experiment Builder (SR Research). Saccade and fixation events were detected by the built-in Eyelink software. Specifically, saccades were defined as changes in gaze position of more than 0.1° with accelerations of at least 8000°/s², and saccade onset was given by the time at which gaze velocity exceeded 30°/s. All events were stored in an event file together with saccade properties (duration, vertical and horizontal velocity, and start and end locations), fixation duration, and event markers indicating the start of each trial and key-press response times. 

Classifying saccades

Examples of saccade paths and fixations are shown in the bottom panels of Figure 1. When followed by the required keyboard response, the first saccade into the area within 1.56° of the target object was labeled the “target saccade” (shown in red), while subsequent saccades/fixations were ignored (purple). In the absence of a response within the next five saccades, fixations on targets were classified as “misses” (shown in blue), while “return saccades” were those saccades to target which were immediately preceded by saccades away from the target area. Finally, all fixations outside of the target area were denoted as “distractor saccades” (black dots). To avoid ambiguity in detection time, target saccades involving returns were excluded from analysis (5.0%, 5.6%, and 24.6% of trials for Task 1, 1B, and 2 respectively, [Figure 2F]). Furthermore, to account for forgotten targets or severe gaze-position errors, trials in which a key-press response was received while fixating outside of the target area were also not analyzed (12.7% and 16.0% of trials for Task 1 and Task 2). 

Figure 2

 

Distribution of subject-averaged behavioral measures. Data represent mean values for each subject (13, 24, and 17 subjects for Task 1, 2, and 1B, respectively). Box plots display the medians across subjects (red lines), 25% and 75% quartiles (box), range (whiskers), and outlier subjects (red +). (A) Number of saccades per trial. The increased number of saccades during Task 2 reflects an increased level of difficulty. (B) Average saccade distance. (C) Average fixation duration, excluding target saccades. (D) Fraction of detected targets per trial. (E) Number of misses per successful trial, that is, saccade to target not reported within 5 subsequent saccades. (F) Number of returns per detection, that is, saccades to target following a recent target fixation. There is no statistical difference between Task 1 and 1B in any of these measures (Wilcoxon rank-sum test, p > 0.23). Task 2 differs significantly from Task 1/1B in all measures (p < 0.01) except for the fractions of detected targets per trial (p > 0.15).

EEG recording

EEG was recorded at a sampling rate of 1024 Hz from 64 scalp electrodes positioned according to the international 10/10 standard (Figure 3E) using an ActiveTwo system (BioSemi, Amsterdam, Netherlands). It was synchronized to the eye-position signals using a trigger pulse issued via the parallel port at the start of each trial. Synchronization was validated through inspection of single-saccade EEG records across the frontal electrodes, revealing sharp spike potentials shortly after saccade onset (Figure 3). No noticeable jitter was apparent across trials (the spike potential is approximately 8 ms long). 

Figure 3

 

Saccade-related potentials. (A) Velocity of the eye movements in pixels per millisecond, averaged across saccades. Velocity is decreased during fixations before and after the saccade. (B) EEG averaged across all saccades for all electrodes and time relative to saccade onset (color represents voltage). The saccade-locked evoked potentials show the spike potential (sharp frontal negativity shortly after saccade onset, 2–10 ms) followed by the lambda complex (frontal positivity during saccade and start of fixation, 15–85 ms), and occipital activity during fixation (100–150 ms). (C) Saccade-locked evoked potential difference between upward and downward saccades, with spatial distribution shown at 47 ms. (D) Saccade-locked evoked potential difference between leftward and rightward saccades. These data are an example generated from Task 1, combining data sets from all subjects. (E) Arrangement of electrodes 1 through 64: The first 32 electrodes are on the left hemisphere, and the last 32 on the right. Within these two sets of 32, electrodes are roughly sorted from the front to the back of the head. Three inferior electrodes in the back of the head are not shown (24, 28, 61).

EEG data analysis

All data analysis was performed using MATLAB (MathWorks, Natick, MA). Analysis for each task combined saccades from all subjects and all trials (i.e., no single-subject analysis). 

Epoching and filtering

6-s epochs were extracted from the continuous EEG record (±3 s relative to saccade onset) and high-pass filtered to remove slow baseline drifts. We employed a causal finite impulse response (FIR) filter with compact temporal extent (zero-delay positive impulse followed by 200-ms negative taps of equal magnitude). This operation amounts to baseline subtraction of every sample with the 200 ms preceding it. To minimize memory requirements, the signal was then down-sampled to 64 Hz or 128 Hz depending on the duration to be analyzed. 

Saccade matching

To minimize the contribution of eye movements to the saccade-related evoked responses in the EEG, each target saccade was matched to a set of distractor saccades having similar dynamical properties: duration, amplitude, velocity, orientation, duration of the previous fixation, and endpoint coordinates. For each target saccade, a set of at least three corresponding distractor saccades with the smallest Mahalanobis distance across these dimensions was selected. This matching procedure was also performed for “misses.” We confirmed that all dynamical properties were well matched using a two-sample Kolmogorov–Smirnov test (p > 0.05 in all instances). Through subtracting the EEG epochs of the distractors from those of the targets (or misses), the saccadic potentials are effectively removed (see Figure 4). For Task 2 we also found for each miss saccade a matching target saccade (we had fewer target saccades available for matching, and thus some targets were used several times). 

Figure 4

 

Saccade-locked evoked potential for Task 1, difference between targets and matched distractors, and effects of subspace subtraction. (A) Evoked potentials for target saccades, that is, EEG locked to saccade onset and averaged across all saccades to targets from all trials and all subjects. (B) Evoked potentials for distractor saccades (averaged over distractor saccades that were matched in dynamical properties to target saccades). (C) Evoked-potentials difference of targets minus matched distractors. (D) t-statistic for every electrode and each time bin (7.8 ms), with color shown only for bins that are statistically significant (FDR p < 0.001). (E through H) Same data after regressing eye-movement-related activity using subspace projections of the three components shown in Figure 3B through D. All activity that could be explained from a linear superposition of such eye-movement components has been removed, including activity that was generated by preceding and subsequent saccades.

Ocular-component removal

While the subtraction procedure described previously removes potentials time-locked to the present saccade, it does not suppress activity resulting from preceding and subsequent saccades. Fortunately, this non-time-locked activity may be removed with a subspace subtraction, which exploits the known spatial distribution of eye movements (L. C. Parra, Spence, Gerson, & Sajda, 2005). The dominant eye-movement activity can be captured by three spatial components (Figure 3B through D): vertical and horizontal eye movements and presaccadic spike potential (see Results for a detailed description of these components). To construct the spatial distribution of the spike potential, the mean event-related potential across all saccades was computed (Figure 3B), with the most negative sample in the first 10 ms after saccade onset representing the component. Meanwhile, vertical and horizontal eye-movement components were captured as the mean difference between left-right and up-down saccades. These three components were regressed out of the EEG prior to the forthcoming analysis (see Appendix A). Figure 4 demonstrates the effect of this subspace subtraction on the evoked responses. 

Channel rejection

We zeroed the activity in channels having mean power (averaged across time and trials) greater than 1500 μV², with this threshold value chosen to remove severely corrupted channels. Through zeroing, these channels did not contribute to the analysis (averaging) of event-related potentials (ERPs) or oscillatory activity. 

Sample rejection

Individual samples that were 16 dB above the average power were zeroed (power was computed separately for each channel and averaged across all samples and saccades; 16 dB corresponds to six standard deviations in magnitude). This procedure was performed after all other preprocessing and filtering and was repeated two times, in each step recomputing the average powers. Overall, sample rejection affected no more than 2% of the data. 

Evoked-response analysis

We employed linear discriminant analysis to identify components of the scalp EEG which discriminate between target (or miss) and distractor saccades on a single-saccade basis. The analysis was carried out in a time-specific fashion, in that a separate classifier was computed for each time sample t (see Appendix B). To improve the signal-to-noise ratio and obtain more robust classification-parameter fits, saccades were pooled across subjects—that is, each saccade contributed to the analysis as a separate data point regardless of trial or subject number. Given an input vector of EEG x(t) at time t, the classifier produces a scalar output y(t) whose sign indicates the prediction—positive for targets, negative for distractors—and whose magnitude relates the confidence of the prediction. Classification performance is then computed by comparing the estimates y(t) with the ground truth: we constructed the conventional receiver-operator-characteristic curve on the training set, with the area under the curve (AUC) reflecting the probability of correct classification (Fawcett, 2006). To enable statistical significance testing, we also computed training-set values of the AUC under the null hypothesis by randomizing the saccade labels and computing the classification performance. One hundred randomizations were performed, with the AUC values from all temporal samples aggregated to form the distribution of AUC values under the null hypothesis (we aggregated this bootstrap sample across time, as the distributions did not vary greatly with time). Temporal samples t whose corresponding AUC value is greater than the 99th percentile value of this distribution (red-shaded area in Figure 5B) are deemed to be significant at the p < 0.01 level. In other words, the neural activity at that time carries significant discriminating (i.e., target vs. distractor) activity. Finally, we correct for multiple comparisons while controlling the false discovery rate (FDR) at 0.01 following Benjamini and Hochberg (1995). The resulting significant AUC values are marked in Figure 5B with red dots. 

Figure 5

 

Discriminant evoked potentials for targets versus distractors. Data locked to saccade onset for Task 1 (top left), Task 2 (top right), and Task 1B (bottom center). (A) Histograms of the button-response times (blue) and the end of saccades (black). (B) Area-under-the-curve performance for discrimination of EEG between targets and matched distractors. Values in the red-shaded area are statistically significant (p < 0.01, shuffle statistic, uncorrected for multiple comparisons). Points marked with red dots are significant after correction for multiple comparisons controlling false discovery rate (p < 0.01). (C) Correlation of EEG at each electrode with discriminant activity computed for each time point (i.e., forward model). (D) Spatial distribution of forward model over the scalp for selected time points.

Oscillatory-power analysis

We also investigated whether differences in oscillatory power could discriminate between saccade categories. To that end, we employed a technique which computes the EEG component with the largest difference in power between two conditions (see Appendix C). The signal was band-pass filtered to the alpha band (7.5–12.5 Hz) with a Morlet filter and the power difference captured by the maximum (or minimum) eigenvalue of the difference of covariance matrices, denoted here as λ and measured in microvolts squared (see Appendix C). This value λ quantifies the maximal difference in power observed between the two conditions and was used here as a metric of performance. Once again, the analysis was performed in a time-specific manner and a distribution of maximum (and minimum) eigenvalues under the null hypothesis was constructed by randomizing saccade labels and repeating the analysis on the randomly labeled data. Thus, to be deemed significant at the p < 0.01 level, the maximum eigenvalue must be greater than the 99th percentile value in the maximum-eigenvalue null distribution (red-shaded area in Figure 6A). Similarly, the minimum eigenvalue is denoted as significant if it falls below 99% of the probability mass in the null distribution of minimum eigenvalues (blue-shaded area in Figure 6A). It should be noted that negative eigenvalues indicate a decrease in power. 

Figure 6

 

Alpha-band power is reduced prior to target detection and is elevated before and after missed targets. Difference in power of 10-Hz oscillations during Task 2 for miss saccades versus matched target saccades (top), target versus matched distractors (bottom left), and miss saccades versus matched distractor saccades. (A) Largest positive/negative eigenvalue of covariance difference, indicating increase/decrease in alpha oscillations for miss saccades as compared to target saccades (red/blue respectively). Values in the red/blue-shaded areas are statistically significant at p < 0.01 (computed using shuffle statistic). (B) Absolute value of maximum eigenvectors across electrodes and time, indicating how much each electrode contributes to the component with largest power for miss saccades versus target saccades. (C) Spatial distribution across the head of these same oscillatory components at selected times. Saccades were compiled across all subjects resulting in 4,114 target saccades and 1,127 miss saccades. Miss-to-target matching used 1,127 matched pairs (some targets were used multiply). Matching to distractors used 3 times as many distractors as targets or misses (12,342 and 3,381, respectively).

Results

Targets are “missed” in the difficult search task

Subjects searched freely for a target among distractors in two tasks of varying difficulty (Figure 1). Task 1 was designed to be easy such that targets might be detected prior to fixation (Kotowicz et al., 2010), and we aimed to identify the earliest markers of detection in the EEG. Conversely, Task 2 was designed specifically to elicit many detection failures, that is, a target is foveated but remains unreported until it is foveated again (a “miss”). For all tasks, subjects were able to locate the specified object in more than 80% of trials (Figure 2). The relative difficulty of Task 2 is evident by the increased number of saccades per trial (Figure 2A). Moreover, it is apparent that the identity of each object was more carefully examined in Task 2, as reflected by an increased fixation duration (Figure 2C). 

There were more missed targets in Task 2, as intended, and also an increased number of “return saccades” (Figure 2E and F), where the subject returns to the target immediately after first foveation. Task 1 had slightly fewer distractors with larger separation (approximately 8%), and thus we expected an increase in the average saccade distance. The increase over Task 2 was 16% and thus larger than expected (Figure 2B), suggesting that subjects captured not only a wider field of view but on average also a larger number of distractors for each fixation. Interestingly, saccades to targets did not differ in distance between any of the tasks (mean: 4.6°, 4.4°, and 4.5° for Task 1, 1B, and 2; p > 0.10, Wilcoxon rank-sum test). To control for the effects of the motor response on the EEG measures, Task 1 was repeated while omitting the key-press response. There were no significant behavioral differences between Tasks 1 and 1B, except of the button response, indicating that the nature of the response does not alter the search process. 

Saccade-related potentials reflect early occipital activity

Figure 3 shows EEG activity locked to saccade onset and averaged over a total of 8,641 saccades collected during Task 1 across all subjects. The horizontal and vertical eye-movement velocity profiles are marked by a sharp increase at onset, with a diminished average velocity during preceding and subsequent fixations (Figure 3A). A characteristic negativity at the frontal electrodes is evident at the onset of the saccade (Figure 3B). This “spike potential” comprises muscle activity and corneo-retinal potentials (Gaarder, Krauskopf, Graf, Kropfl, & Armington, 1964; Riemslag, Heijde, Van Dongen, & Ottenhoff, 1988; Thickbroom & Mastaglia, 1985) and is followed by a positivity (approximately 15–85 ms) with similar spatial distribution (Figure 3B). This positivity is known as the lambda complex. Consistent with previous literature (Jagla, Jergelová, & Riecanský, 2007) we find that its intensity increased with saccade amplitude, and its duration was proportional to saccade duration but extending 40–50 ms beyond the end of the saccade. Furthermore, a weakly positive potential whose topology is indicative of early visual processing in occipital areas appears at 110–160 ms (Figure 3B), similar to what was found by Dandekar, Privitera, et al. (2012). 

The saccade-related potentials of Figure 3B through D are averaged across all types of saccades (targets, distractors, misses) and do not significantly differ between the different tasks. For the purpose of differentiating activity between target, distractor, and miss saccades, such stereotypical profiles can be considered artifacts. Thus, in the subsequent analysis these components were removed from the data using a subspace-subtraction technique (see Methods). Our aim was to identify activity which differs from these well-characterized components and thus carries information relating to the nature of the object (e.g., target vs. distractor). 

Target-related potentials precede foveation of easy but not difficult targets, likely related to motor preparation

We first sought to identify neural activity discriminating target and distractor saccades. To that end, we compute the ensemble-averaged EEG stemming from target and (matched) distractor saccades for Task 1 (Figure 4A and B). Through subtracting the activity shown in Figure 4B from that of Figure 4A, the discriminating activity becomes apparent (Figure 4C). A conventional test for significant differences for each electrode and each time sample indicated strong differences following the saccade and a possible difference preceding the saccade onset (two-sample t-test, df = 3894; FDR p < 0.001; Figure 4D). After subspace subtraction, the timing of these differences is largely preserved (Figure 4E through H). However, the spatial profile of these differences is changed, since regression removed all activity that could be explained by a linear combination of eye-movement-related components. 

To analyze evoked-response differences in more detail, and in particular to detect the earliest point at which significant target-related activity is observed, we used linear discriminant analysis. Through combining multiple electrodes prior to statistical evaluation of a difference, this technique has the potential to increase signal-to-noise ratio and has thus been used to extract more subtle differences in evoked responses that are otherwise not apparent in individual electrodes (see, e.g., Gerson, Parra, & Sajda, 2005; Ratcliff, Philiastides, & Sajda, 2009; see Methods for details). Figure 5B shows the discrimination performance (measured in AUC) as a function of time for a classifier which attempts to predict the nature of the saccade (i.e., “target” or “distractor”) based on the measured neural activity. It is interesting to note that for Task 1, statistically significant activity emerges already at t = −150 ms, indicating that the neural activity prior to saccade onset carries information as to the object of (future) foveation. 

This early component has a central distribution with a slight right lateralization (in this Task 1, subjects responded with the left hand). Importantly, this early activity is not present in Task 2, in which the responses occurred with a greater latency (Figure 5A). Moreover, when probing Task 1 for the speed of response, we find that the early activity is present only prior to fast responses (median split in response time at 413 ms). To directly test whether this discriminating activity is an early preparatory signal associated with the button press (e.g., the “readiness potential” of Deecke, 1990) we repeated Task 1 without the manual response. Indeed, as shown in Figure 5B, any discriminating activity prior to t = 0 failed to appear. These results suggest that this early activity is indeed related to the upcoming motor response. 

Careful inspection of Figures 4A and B of Task 1 indicates that the preparatory potentials preceding the saccade are negative, consistent with the findings of Jagla et al. (2007), Klostermann et al. (1994), and Moster and Goldberg (1990). In the present data this negativity is less pronounced for targets, resulting in a relative positivity indicative of the upcoming target (see Figure 4C, centered at −110 ms). 

Discriminating activity is present also during and after saccade to target

For the easier task, discriminating activity in the form of a fronto-central negativity appears during the saccade (see snapshots at 63 ms for Task 1/1B in Figure 5D). This spatial distribution of target-related activity persists until some time after (see snapshot at 172 ms for Task 1, but also Task 2). Remarkably, discriminating components are also over occipital electrodes (see 63 ms and 109 ms for Task 1B), which is noteworthy because target-related occipital activity is not conventionally found during a saccade. 

Postsaccade, a target-related occipital negativity emerges at 150 ms and 200 ms (see Figure 5C and D at 172 ms for Task 1 and 242 ms for Task 2). To better understand the differing timing of this early postsaccade activity, we repeated the same discriminant analysis separately for short and long target saccades (median split on saccade length at 3.58°) for Tasks 1 and 2. We found that the onset of the discriminant potentials subsequent to the saccade is delayed by approximately 50 ms for the longer saccades, suggesting that even within task, nearby targets may be “seen” sooner than distant targets. Careful inspection of the saccade-locked ERPs in Figure 4 reveals that this occipital negativity appears as a positivity for all saccades (as in Figure 3B at 141 ms) but is less pronounced in target saccades and thus appears with a negative sign after subtracting distractor activity. 

In summary, for the simpler task, evoked potentials indicative of the impending targets are present during the saccade regardless of motor response, specifically over occipital electrodes. In all tasks, the timing of postsaccadic evoked potentials is comparable but not identical to those observed in conventional target-detection paradigms, that is, rapid-serial-visual presentation (RSVP; Gerson et al., 2005; Johnson & Olshausen, 2003; VanRullen & Thorpe, 2001), constrained saccades (Dandekar, Privitera, et al., 2012), and more recent studies with free viewing (Kamienkowski et al., 2012). The difference in precise timing between different studies may relate to task difficulty. 

Success or failure to detect a target is associated with sustained modulation in fronto-central alpha power

The analysis of target-detection failures focused on Task 2, as this task generated the largest number of miss saccades (approximately one in every other trial, Figure 2E). We did not find a difference in the saccade-evoked responses of miss saccades versus distractor saccades, as we did for target saccades versus distractor saccades. 

Given our hypothesis that detection failures result from lapses in attention, we next analyzed oscillatory activity in the alpha band (7.5–12.5 Hz). 

Power was measured in the EEG component showing the largest difference in power between two conditions (see Methods). As before, we matched dynamical properties of saccades to assure that potential differences are not due to differing saccade dynamics. We first compared missed and found targets (i.e., miss saccades vs. target saccades). A significant increase in oscillatory alpha-band power (p < 0.01, shuffle statistic) was found before and after the saccade when the target was missed (top panels in Figure 6A). Interestingly, this preceding activity also has a fronto-central distribution, with an additional right-anterior component (top panels in Figure 6C). Note that we ensured a balance between left- and right-hand responses during data collection for this task, so any lateralization is unlikely to result from motor mu rhythms (alpha activity over motor cortex associated with motor activity). 

To determine whether the elevated alpha activity during detection failures is due to an alpha increase for miss saccades and/or a decrease for target saccades, we repeated the analysis separately for target and miss saccades (Figure 6, Targets vs. Distractors and Misses vs. Distractors). When compared to matched distractor saccades, alpha power was reduced before target saccades (also for Task 1 and 1B, not shown). Incidentally, after the target saccade there was a reduction of alpha activity over left/right central motor areas, as expected due to the keyboard press. For the miss saccades, alpha power was increased relative to matched distractors during an extended period of time before and after the saccade (at least 1.5 s). The same differences were found also when comparing target and miss saccades to a common pool of distractor saccades that serve as an equal baseline of alpha activity (Figure 8). More careful analysis of miss saccades revealed that increased alpha activity is associated with failures to report the target specifically after prolonged inspection of the target; namely, when all miss saccades that remain within the target region for more than one fixation are excluded from the analysis, the difference in alpha power to matched distractors disappears. 

Success or failure to detect a target is associated with altered saccade dynamics

This last finding led us to speculate that saccade dynamics may differ between target and miss saccades and that this in turn correlates with alpha power. Indeed, the dynamical properties of the saccades—and their flanking fixations—differ significantly between target, miss, and distractor saccades (Figure 7A). When repeating the analysis of alpha power to assess the effect of the different dynamical properties, we find that alpha activity is increased for short and fast saccades, for saccades that remain close to the center of the screen, and for saccades that are flanked by short fixations (Figure 7B). Evidently, saccade control prior to finding a target differs from when the target remains unreported (Figure 7A), and saccade control modulates alpha oscillations on a prolonged time scale of several seconds (± 1.5 s analysis window in Figure 7B). 

Figure 7

 

Saccade properties vary between saccade type and correlate with alpha-band power. (A) Median value of dynamical saccade properties and flanking fixations. Comparisons used Wilcoxon rank-sum test, with p-values indicated on top for each comparison (not corrected, number of saccades as in Figure 6). Miss saccades differ from distractor saccades in all properties except for the preceding fixation duration and eccentricity of saccade endpoint. Target saccades differ from distractor saccades in all properties except duration. (B) Dependence of alpha power on different saccade and fixation properties. (B.1) Power difference between second and third quartiles for each property is measured here as in Figure 6, that is, as the largest eigenvalue of the covariance difference. Shown here is the average value over the entire analysis window −1.5 s to 1.5 s, with error bars indicating the standard deviation over that period. All properties have a significant effect on alpha power (p < 0.01 significance levels indicated in light-shaded area; thresholds [green] are very similar for different properties). (B.2) Spatial distribution of alpha-power components at selected times corresponding to the maximum difference for the same six dynamical properties. This analysis was performed on distractor saccades only, using a larger number of saccades (89,000) to facilitate the split into quartiles.

Figure 8

 

Supplementary figure. Same as Figure 6, but now comparing alpha-band power differences for target saccades (left) and miss saccades (right) to a common pool of distractor saccades. We used 16,456 as a common pool of unmatched distractor saccades.

Discussion

There is a long history of work using scalp EEG to investigate neural correlates of target detection during highly constrained viewing conditions. The literature on neural activity during free viewing is much more recent (Dandekar, Privitera, et al., 2012; Dimigen, Sommer, Hohlfeld, Jacobs, & Kliegl, 2011; Graupner, Pannasch, & Velichkovsky, 2011; Kamienkowski et al., 2012; Luo, Parra, & Sajda, 2009; Ossandón, Helo, Montefusco-Siegmund, & Maldonado, 2010; Rämä & Baccino, 2010), possibly because such an investigation requires (a) precisely synchronizing gaze-position signals with EEG, (b) controlling for ocular artifacts, and (c) accounting for the effects of filter design when making inferences about timing relative to rapid events such as saccades. Existing work on the EEG during free viewing has been confined to fixation-locked evoked potentials, and with the exception of Kamienkowski et al. (2012) and Luo et al. (2009), it has not dealt with neural processing related to free-viewing search or target detection. 

In this article, we provide a comprehensive treatment of the neural correlates of free-viewing visual search, including an analysis of detection failures. By aggregating over a large number of saccades combined across subjects, we obtained sufficient statistical power to discriminate neural activity based on the object of foveation. In addition, we employed machine-learning tools to compute components maximizing the difference (across conditions) in evoked and oscillatory activity, allowing us to detect discriminating activity distributed across electrodes. 

Pre- and postsaccadic discriminating evoked potentials

The analysis stemming from Task 1 indicates that target objects could be detected as early as 150 ms prior to the first foveation of the target (Figure 5B.1), as evidenced by a fronto-central potential which is more negative prior to distractor foveation. This finding extends previous work by Luo et al. (2009), which identified a target-related fronto-central positivity at −80 ms during a constrained search paradigm. While presaccadic negativity over fronto-central areas has been well characterized and related to activity of the frontal eye fields (FEF; Klostermann et al., 1994; Moster & Goldberg, 1990), its connection to target detection is less clear. The fact that this early activity is absent in Tasks 1B and 2, where a manual response was omitted or delayed, respectively, seems to implicate the readiness potential (RP; Deecke, 1990). 

Moreover, its right-lateralized spatial distribution (top left of Figure 5D at −141 ms) is also consistent with the RP. The presaccadic discriminating activity is also consistent with other known preparatory potentials (Brunia & Boxtel, 2001): For instance, the contingent negative variation, which is associated with the expectancy of an impending task-relevant stimulus (Walter, Cooper, Aldridge, Mccallum, & Winter, 1964), has a distribution and timing similar to those of the RP. Regardless of the relationship of this activity to that found in more constrained experimental paradigms, it is apparent that subjects in Task 1 detect the target significantly ahead of target fixation. 

This finding is consistent with earlier behavioral data on Task 1 indicating that target location can be determined prior to fixation for a subset of trials (Kotowicz et al., 2010), namely, subjects had above-chance performance in reporting the location of a target even when the target was removed from the scene just before the first fixation on target (at −40 ms). Consistent with this, we find that fewer saccades are needed for this “easy” task as compared to Task 2 (Figure 2A) and the saccades extend over longer distances (Figure 2B), suggesting that subjects can evaluate a larger field of view and thus may detect the target with peripheral vision. Interestingly, in the absence of an overt manual response in Task 1B this presaccadic discriminant activity is not observed (see bottom Figure 5B). 

Why did we not find presaccade evoked responses in the absence of a button-push response? If the target is detected in the periphery ahead of time, surely some visual processing must proceed the saccade. However, it is possible that the timing of this visual processing is not time locked with the saccade onset and thus does not rise above significance in our saccade-locked ERP analysis. This interpretation is supported by the observation that a target-related positive potential appears over occipital cortex already during the saccade (bottom of Figure 5D). This activity bears resemblance in polarity and distribution to the classical P1 component (Pratt, Willoughby, & Swick, 2011) involved in tasks requiring increased spatial attention (Mangun, Hopfinger, Kussmaul, Fletcher, & Heinze, 1997). Given the propagation delay from the retina to visual cortex (approximately 60 ms), it is possible that this occipital activity is the result of visual input preceding the saccade. Alternatively, it could represent a top-down biasing of visual cortex in anticipation of the upcoming target. 

The target-related activity during saccades in Task 1/1B (63 ms) appears with the same spatial distribution some time later in Task 2 (172 ms). We speculate that in the easier task, this activity was set in motion in the previous fixation and continued during the saccade to target, while in the more difficult task, it was only initiated after target foveation. All three tasks show significant activity that is predictive of the impending target immediately preceding the saccade (e.g., −23 ms in Task 2). Its spatial distribution, however, is variable, and thus we hesitate to draw any conclusions from this finding. 

The postsaccadic discriminant activity is consistent across the three tasks (Figure 5): a focal frontal negativity in the midline during and shortly after the saccade (60–170 ms), occipital negativity (150–250 ms), and a broad central-parietal positivity starting at 300–400 ms. This postsaccade activity is similar to that observed during RSVP (Gerson et al., 2005; Johnson & Olshausen, 2003; Thorpe, Fize, & Marlot, 1996; VanRullen & Thorpe, 2001), where stimulus-related discriminating activity appears as early as 150 ms after stimulus presentation and response-related activity is seen at 300–400 ms (P300). A similar timing of discriminant EEG activity was found in a recent study on free-viewing search (Kamienkowski et al., 2012). The task in that study differed from ours in that targets were crowded by nearby distractors, making them hard to detect without direct foveation. A similar P300 component was also found in a study comparing target detection during fixation as compared to constrained saccades (Dandekar, Ding, et al., 2012). Here and in these earlier studies, the timing of these components differs somewhat depending on task difficulty and distance of target, indicating that the first discriminating moment may vary with task difficulty. 

It is interesting to contrast these components with classical evoked potentials such as the N1 or P300. (One should keep in mind that the timing here is relative to saccade onset, whereas classical visually evoked components are measured relative to stimulus onset.) While the N1 is found over many recording sites with variable timing (Mangun & Hillyard, 1991), the frontal activity observed here at 60–170 ms has a seemingly disparate spatial distribution. Given its topology and coincidence with saccades, it is more likely related to activity in the FEF or supplementary eye fields (Schall, 2002; Stuphorn, Brown, & Schall, 2010; Stuphorn & Schall, 2006; Thompson & Bichot, 2005; Thompson, Hanes, Bichot, & Schall, 1996). Indeed, early visually evoked responses (around 45–60 ms poststimulus) have been found in the FEF (Kirchner et al., 2009), and it is thus possible that the early discrimination observed during Task 1 corresponds to such fast responses. The occipital N1 (Wascher, Hoffmann, Sänger, & Grosjean, 2009) is presumably unrelated to the activity seen here at 170–250 ms, which stems from a reduction is positivity during target saccades. Finally, one may be tempted to label the positivity at 300 ms as a free-viewing analogue of the classic P300. However, neither the timing nor the (more central) scalp topology supports such a claim. Thus, further studies will be required to infer the origin of the discriminative components observed in this study. 

Target-detection failure and alpha power

Studies on inattentional blindness are concerned with failures to report unexpected but salient objects, which are often unrelated to the main task (Mack & Rock, 1998). Here we analyzed success and failures to detect a task-relevant stimulus, namely, the search target itself. There are many reasons why humans may miss targets during visual search; these include but are not limited to visual clutter (Henderson, Chanceaux, & Smith, 2009), rare targets (Wolfe, Horowitz, & Kenner, 2005), and the stochastic nature of visual processing (Eckstein, 1998). In the present study, successful detection was associated with decreased alpha activity prior to the first saccade to target (−1.5 s < t; Figure 6). In contrast, failures to detect a target were associated with an increase in alpha power before and after saccade to the missed target (−1.5 s < t < 1.5 s). We also found that success and failure to detect a target are associated with altered saccade dynamics (Figure 7) and that this in turn correlates with a decrease or increase in alpha power. However, the elevated alpha activity cannot be attributed to differing saccade dynamics, as it persists when dynamical properties have been matched (bottom right of Figure 6). 

Increased alpha activity is a well-known correlate of reduced attention (Kelly, Lalor, Reilly, & Foxe, 2006; Klimesch, Sauseng, & Hanslmayr, 2007; Thut, Nietzel, Brandt, & Pascual-Leone, 2006) and is thought to reflect reduced excitability of a cortical area or its active inhibition (Pfurtscheller, 2001; Romei, Rihs, Brodbeck, & Thut, 2008). The increase found here has a fronto-central topology with a right-anterior component. It should also be pointed out that the increased alpha activity is significantly more anterior than the right-parietal alpha activity found in earlier (non-search-related) studies (O'Connell et al., 2009). 

Decreased alpha activity over sensory cortex is often found during active sensory processing. In the context of a target search, one may attribute the presaccadic decrease in alpha power to engagement of cortical areas which may be extracting salient features from the periphery. The decrease in alpha power preceding found targets (as compared to distractors; Figure 6B) may also indicate an overall increased alertness before a target is successfully identified. From intracranial recordings in humans, decreased alpha-band activity has been observed over the FEF and supplementary eye fields shortly before and after a saccade (Lachaux et al., 2006). It is noteworthy that the alpha-power changes detected here precede and outlast the saccade by as much as 1.5 s (and maybe further, if we extended the analysis window), thus reflecting sustained changes in cortical synchronization. The right lateralization found in the present study is consistent with previous studies using transcranial magnetic stimulation, which show a right-dominance of FEF in attentional effects (Grosbras & Paus, 2002) and elevated stimulus-evoked potentials over the right hemisphere following error saccades (Ptak et al., 2011). Incidentally, the component discriminating misses (and targets) from distractors bears a topology extending over the right-prefrontal electrodes (a location which shows the earliest evoked activity during target recognition in RSVP paradigms; VanRullen & Thorpe, 2001). 

The distribution of increased alpha activity observed here over fronto-central electrodes is consistent with the location of frontal and supplementary eye fields, which are involved in saccade control. Thus it is possible that the increase in alpha activity observed for miss saccades may result from a difference in saccade control, which is reflected in altered saccade dynamics (Figure 7A). Conversely, increased alpha activity may signal an inhibition of these areas and, as a consequence, alter dynamical properties of the saccades. Sensitivity of alpha activity was observed for all dynamical features we looked at (Figure 7B), and alpha modulation is extended over a period of time (±1.5 s) which is much longer than the duration of these dynamical features. This suggests that alpha modulation is not tightly regulated but rather coincides with prolonged states. 

In summary, we have reported here correlations between three sets of variables naturally modulated during this free-viewing search task: (a) success or failure to detect a target, (b) fronto-central alpha activity, and (c) saccade dynamics. We found a correlations between all three variables and a specific link between detection success and alpha power when controlling for saccade dynamics. However, only a prospective intervention, perhaps with transcranial magnetic stimulation (TMS), could address the precise causal relationship between fronto-central alpha activity, detection performance, and saccade dynamics. 

Caveats on spatial origin of activity

While there is increasing evidence for lateralization of specialized functions in the human FEF, it should be noted that the left and right FEF act largely equally on their contralateral visual fields (Blanke et al., 1999; Gutteling et al., 2010): Right- and leftward saccades elicit similar activity in the left and right hemispheres, respectively. The spatial filtering used here (subtraction of left/right movement components; Figure 3) would not reveal such activity. Analysis without this spatial filtering reveals significant activity in the frontal electrodes (Figure 4A through D), including activity possibly originating in the eye fields which may correlate with eye-movement direction. Further analysis of the present data, performed separately for left and right saccades, may be necessary to extract such hemifield-specific activity (see, e.g., Bellebaum & Daum, 2006; Parks & Corballis, 2010). 

Conclusion

This study has explored the neural correlates of free-viewing visual search. We found presaccade motor-related potentials indicating that in “easy” tasks, targets may be detected as early as 150 ms prior to saccade to target. In addition, occipital evoked activity related to the upcoming target is present even during and immediately after the saccade, suggesting that visual processing or top-down bias thereof may be concurrent to eye movements. Moreover, target detection, alpha power, and saccade dynamics are modulated and all correlate with each other during free-viewing search. When controlling for saccade dynamics, we find that alpha activity over fronto-central sites varies with success or failure to report a target. This suggests that activation or inhibition of fronto-central cortical areas may play a crucial role in the success of target detection. 

Acknowledgments

The authors are grateful to Simon Kelly for the careful review and thorough feedback on an early version of this manuscript. This work was supported by DARPA under contract N10PC20050 and by the Fundação para a Ciência e a Tecnologia (Fellowships SFRH/BD/33207/2007 to JCD). 

Commercial relationships: none. 

Corresponding author: Lucas C. Parra. 

Email: [email protected]. 

Address: Department of Biomedical Engineering, City College of New York, City University of New York, New York, NY, USA. 

Appendix A: Subspace subtraction

The purpose of subspace subtraction here is to remove (regress out) any activity that could be explained (linearly) from the spatial profile of voltages on the scalp observed during eye movements. Since voltages add up linearly in the EEG, eye movements characterized by those spatial profiles can, after regression, no longer contribute to the signal. Denote the mean voltage across the 64 electrodes for the spike potential as the vector v_s and denote the vertical and horizontal potentials at 47 ms as v_v and v_h. These three components, V = [v_sv_vv_h], can be subtracted from all EEG samples given in vectors x(t) with the following subspace projection: x̃(t) = (I − VV⁻¹)x(t), where V⁻¹ represents the pseudoinverse of V (for a derivation of this, see L. C. Parra et al., 2005). This subtraction is analogous to the separation of components that occurs for instance in independent component analysis, which is routinely used to remove movement artifacts from the EEG (Jung et al., 1998). Contrary to that technique, the components representing movement here are not identified “blindly” using an independence assumption but rather are reliably quantified based of observable information (i.e., “trained” with examples of left/right or up/down eye movements). 

Appendix B: Linear discriminant analysis

A conventional method of analyzing event-related potentials is to subtract, for each channel, the mean activity of one type of event from the average activity of a standard event (or baseline activity). This is done here by subtracting the activity of distractors from targets (Figure 4). While taking the difference for each electrode is conceptually straightforward, this process ignores the dependence of activity in different channels (i.e., the correlation between electrodes resulting from a common neuronal source), thus complicating statistical analysis and, more importantly, failing to leverage the information available across multiple channels. By instead using a weighted average across electrodes, one can increase the (correlated) signal strength and reduce the (uncorrelated) noise. An effective approach for finding an optimal weighting for this average is linear discriminant analysis (Duda, Hart, & Stork, 2001), with different algorithms optimizing slightly different discriminant criteria (L. Parra et al., 2002). Linear discriminant analysis aims to find an orientation in space—that is, a component of the data—that best separates two data sets as follows: Denote the weights for this averaging with w. The goal of discriminant analysis is to find w such that the scalar (one-dimensional) signal, y(t) = w^Tx〰(t) differs maximally across two different conditions, in this case, targets versus distractors. This technique was first introduced to EEG analysis by L. Parra et al. (2002) and has since been used repeatedly for single-trial analysis of EEG. Comparable methods are used for single-trial analysis of fMRI (see, e.g., Lohmann, Volz, & Ullsperger, 2007). We used penalized logistic regression to find the optimal w for each sample in the data (easy-to-use MATLAB code is available in Dmochowski, Sajda, & Parra, 2010). 

Appendix C: Power-difference analysis

In comparing oscillatory activity, linear discrimination is not an appropriate method for finding an optimum weighted average of the electrodes (discrimination of the two conditions is based on the sign of y(t), which continually changes for oscillating activity). Instead, one must consider the size of the absolute value of the oscillation, or power, to identify modulation of oscillatory activity. Thus it is more sensible to find a projection of the data that differs in power. If a specific frequency band is of interest, then power should be measured after the signal has been adequately band-pass filtered. Here, we band-pass filter using a Morlet quadrature pair, h(t) = exp(i2πtf_c – t²Δf²/2) represent center frequency and bandwidth and i² = −1. Denote the filtered signals as ◯(t) = h(t)*x(t) and compute with this the covariance R₁ and R₂ for conditions 1 and 2 (for example, misses and distractors). We were interested in determining the projection of this filtered data exhibiting the maximum power difference. This can be obtained as the eigenvector of the covariance difference, (R₁ – R₂)w = λw, with the largest eigenvalue λ. Note that the values in w will in general be complex-valued, indicating a summation over electrodes with a phase delay relative to each other. A more detailed discussion of this method is forthcoming. For here it suffices to say that the signal extracted, ȳ(t) = w^T◯, represents a component of the EEG, with oscillatory activity in the desired frequency band which differs maximally between the two conditions. The method is comparable to the popular Common Spatial Pattern algorithm (Ramoser, Müller-Gerking, & Pfurtscheller, 2000) while improving robustness with regards to outliers and noise.