Visual memory during pauses between successive saccades

Abstract

Selective attention is closely linked to eye movements. Prior to a saccade, attention shifts to the saccadic goal at the expense of surrounding locations. Such a constricted attentional field, while useful to ensure accurate saccades, constrains the spatial range of high-quality perceptual analysis. The present study showed that attention could be allocated to locations other than the saccadic goal without disrupting the ongoing pattern of saccades. Saccades were made sequentially along a color-cued path. Attention was assessed by a visual memory task presented during a random pause between successive saccades. Saccadic planning had several effects on memory: (1) fewer letters were remembered during intersaccadic pauses than during maintained fixation; (2) letters appearing on the saccadic path, including locations previously examined, could be remembered; off-path performance was near chance; (3) memory was better at the saccadic target than at all other locations, including the currently fixated location. These results show that the distribution of attention during intersaccadic pauses results from a combination of top–down enhancement at the saccadic target coupled with a more automatic allocation of attention to selected display locations. This suggests that the visual system has mechanisms to control the distribution of attention without interfering with ongoing saccadic programming.

Introduction

Most natural visual tasks require sequential inspection of the visual array by saccadic eye movements. Saccades are important for two reasons. First, since visual acuity is best in the central fovea, and falls sharply with increasing eccentricity, saccades are needed to bring the line of sight to regions of interest to ensure that selected visual details can be resolved. Second, even when visual resolution is adequate, the limits on the ability to identify, recognize, or remember multiple objects means that we need to sequentially direct attention to those objects or regions that are of immediate relevance to task performance. Saccades would appear to provide the appropriate vehicle for carrying attention from place to place. This latter role for saccades has encouraged the belief that saccadic eye movements are closely, and perhaps inextricably, tied to selective perceptual attention. It is this assumption that we examine in this paper. 

Given the complex sets of operations that must be performed during visual tasks, it is not surprising that a variety of approaches to studying the links between saccades and attention have developed. Some approaches have focused on modeling where people choose to look, examining the decision rules that determine saccadic landing sites on the basis of their immediate value to the task at hand (Araujo, Kowler, & Pavel, 2001; Bichot & Schall, 1999; Eckstein, Drescher, & Shimozaki, 2006; Einhäuser, Rutishauser, & Koch, 2008; Findlay, 1997; Koch & Ullman, 1985; Legge, Klitz, & Tjan, 1997; Motter & Simoni, 2007; Najemnik & Geisler, 2005; Peters, Iyer, Itti, & Koch, 2005; Pomplun, 2006; Rao, Zelinsky, Hayhoe, & Ballard, 2002; Torralba, Oliva, Castelhano, & Henderson, 2006; Viviani & Swensson, 1982). A prevalent assumption in many of these studies is that saccadic landing positions are chosen by means of a global analysis of the visual array, centered on the fovea, that begins anew with each successive fixation. The various models that have been proposed by the authors cited above have been successful at predicting aggregate characteristics of saccades, such as the spatial distribution of preferred landing sites after many seconds of inspection. Perceptual attention—defined as the allocation of internal processing resources to a given object or region—is assumed to remain centered on the line of sight, and is given no special role independently of saccades. 

Another set of studies, of more direct relevance to the present paper, did make distinctions between perceptual attention and saccades. These studies focused not on choices of where to direct saccades, but rather on the way in which attention enables saccades to reach chosen targets accurately in the presence of competing stimuli nearby. These studies probed the state of attention during the latency interval preceding individual saccades and found that perceptual recognition or perceptual identification are better at the saccadic goal than elsewhere (Baldauf & Deubel, 2008; Caspi, Beutter, & Eckstein, 2004; Cohen, Schnitzer, Gersch, Singh, & Kowler, 2007; Deubel & Schneider, 1996; Gersch, Kowler, & Dosher, 2004; Godijn & Theeuwes, 2003; Hoffman & Subramaniam, 1995; Kowler, Anderson, Dosher, & Blaser, 1995; McPeek, Maljkovic, & Nakayama, 1999; Wilder, Kowler, Schnitzer, Gersch, & Dosher, in press). Efforts to direct attention to locations other than the saccadic goal resulted in some performance loss: either reduced levels of perceptual accuracy or an increase in saccadic planning time (Kowler et al., 1995). Neurophysiological studies done under conditions comparable to the psychophysical work have supported strong links between attention and saccades, with several studies finding activity related to both perceptual attention and saccades in neural areas such as lateral intraparietal cortex (e.g., Ipata, Gee, Goldberg, & Bisley, 2006), frontal eye field (e.g., Moore & Fallah, 2001), and superior colliculus (e.g., Kustov & Robinson, 1996). At least some of the pre-saccadic perceptual enhancements observed in the behavioral work can be attributed to signals relayed from frontal eye field (an area closely tied to the generation of saccades) to visual area V4 (Moore & Armstrong, 2003). Taken together, the research on attention and saccades has shown that the decision to make a saccade is coupled with a shift of perceptual attention to the target. These shifts of attention, which appear to be generated along with the saccadic commands and produce marked attenuation of visual signals from non-target regions, are valuable for ensuring accurate aiming of saccades to selected goals, reducing the likelihood that the line of sight will be drawn to objects nearby.¹ 

Although the prior work makes a compelling case for linking saccadic planning to the spatial distribution of perceptual attention, there are clear drawbacks to such an arrangement. In particular, it seems implausible that perceptual processing would be well served by an attentional field that is narrowly focused around the target of each successive saccade. Such a narrow focus restricts the ability to evaluate the content of visual scenes, to make decisions about the foveal stimulus, and to identify useful places to look. The conflict between the need for global perceptual analysis of a scene prior to saccades, and the spatially local character of pre-saccadic shifts of attention, encouraged us to examine the extent to which saccades and attention can be decoupled. Specifically, are there mechanisms that allow perceptual attention to be distributed independently of ongoing saccadic planning, without impairing the execution of the pattern of saccades? 

Gersch, Kowler, Schnitzer, and Dosher (in press) recently found evidence for such independence in a study of saccadic sequences. In their experiment, saccades were made along a path that was either marked by a color cue, or memorized, while perceptual attention was assessed by an orientation identification task during randomly selected intersaccadic pauses. (By “intersaccadic pause” we mean the intervals of fixation, typically lasting 0.2–0.3 seconds, between successive saccades.) In the case of the memorized saccadic paths, perceptual performance was better at the immediate goal of the saccade than at non-goal locations (see also Gersch et al., 2004). But when saccades were made along color-cued paths, attention extended beyond the immediate saccadic goal to locations along the cued path, including locations previously examined. Saccadic performance was equivalent for both cued and memorized paths. Finding different patterns of attention with equivalent pattern of saccades shows that whatever mechanism was responsible for the spread of attention along the color-cued saccadic path was neither necessary to carry out the saccadic sequence nor did it interfere. Baldauf and Deubel (2008) and Godijn and Theeuwes (2003) obtained related results, finding that attention was allocated to a pair of cued saccadic targets (even non-adjacent targets; Baldauf, Wolf, & Deubel, 2006) prior to the initiation of the sequence. 

Gersch et al.'s (in press) results can be related to those of Bichot, Rossi, and Desimone (2005), who studied activity in V4 during the pauses between saccades of a monkey engaged in a visual search task (Mazer & Gallant, 2003). Bichot et al. found that neural activity was enhanced, not only at the target of the immediate saccade (Armstrong & Moore, 2007; Moore & Armstrong, 2003), but also in response to stimuli that shared critical features (color or shape) with the search target. Bichot et al. did not relate neural activity to the pathway of sequences of saccades; thus, it was not possible to determine from their experiment whether the enhanced activity reflected long-range saccadic planning or the spread of perceptual attention due to, for example, shared visual features (Melcher, Papathomas, & Vidnyánsky, 2005; Motter, 1994; Sàenz, Buraĉas, & Boynton, 2003; Treue & Martínez Trujillo, 1999). 

The ability to distribute attention to significant locations in space without at the same time disrupting the planning or execution of saccades (Gersch et al., in press) provides an exception to the pervasive link between perceptual attention and immediate saccadic planning. Such a distribution can benefit perception without disrupting ongoing saccadic planning or saccadic control. 

Goals of the present study

In the present study, like Gersch et al. (in press), attention was measured during sequences of saccades made along color-cued paths. In their study, attention was assessed by means of a perceptual identification task (identifying the orientation of a briefly presented grating stimulus). In the present study, perceptual attention is assessed by means of a visual memory task. A visual memory task was used for two reasons: 

First, performance on perceptual identification tasks can be well above chance even for weakly attended locations. Thus, even a modest alteration in the distribution of pre-saccadic attention could be of substantial benefit to perception. A visual memory task, on the other hand, can provide a more stringent test of the ability to attend to non-goal locations. The limited capacity of immediate visual memory forces attention to play a crucial “all-or-none” role in determining which portions of a visual array are remembered. Thus, with a memory task only locations receiving the highest levels of attention are likely to impact memory performance (Alvarez & Cavanagh, 2004; Reeves & Sperling, 1986; Sperling, 1960). 

The second reason to study visual memory during sequences of saccades is to better understand how the planning and execution of saccades affects the processing of visual scenes. While the relative perceptual clarity of different portions of the array may be important to performing many visual tasks, the ability to remember portions of the array from one fixation to the next is also crucial (Ballard, Hayhoe, & Pelz, 1995; Epelboim & Suppes, 2001; Melcher, 2001; Melcher & Kowler, 2001). Several studies have examined the link between visual memory and saccades. Irwin and colleagues, for example, presented arrays of letters briefly during the latency interval of single saccades and found that letters located near the saccadic goal were remembered better than letters elsewhere (Irwin, 1992; Irwin & Andrews, 1996; Irwin & Gordon, 1998). Henderson and Hollingworth (1999, 2003) used the detectability of changes made to objects in scenes as the index of memory. They found that changes made during a saccade toward an object were more likely to be noticed than changes made during a saccade away from an object (see also Currie, McConkie, Carlson-Radvansky, & Irwin, 2000). While these studies established an advantage in memory for targets of individual saccades, they did not study saccadic sequences, or map the relative strength of visual memory in a variety of spatial locations relative to the saccadic path, or relate memory strength to visual properties of the array. 

The present study set out to map the strength of visual memory in a variety of spatial locations on and off the pathway of saccades and to relate the map to the spatial and temporal properties of the saccadic sequences. The main goal was to characterize the spatial distribution of attention and to find out to what extent the spatial distribution of attention, as assessed by visual memory, could be dissociated from the immediate saccadic goal. 

Approach and rationale

Following the approach of Gersch et al. (in press), subjects were asked to make sequences of saccades along paths marked by a perceptual cue (a color difference between saccadic targets and non-targets). To assess the strength of attention at locations on and off the cued saccadic path, we measured the ability to recall a single probed letter from an array presented during a randomly chosen intersaccadic pause. The comparison of performance across the different probed locations will determine whether there is any advantage to locations on the cued path, including locations previously examined. 

A second condition was included in which the location of the probed letter was disclosed before the trial. This condition was tested to find out whether the distribution of attention during saccadic sequences could be voluntarily altered to include the pre-cued location, and if so, whether such alterations required some sacrifice in saccadic performance. 

We used a specified saccadic path rather than a free-viewing task such as search (as in, for example, Bichot et al., 2005) in order to be certain of the planned trajectory of the saccadic path. This approach would let us to infer the role of saccadic planning, as opposed to the role of visual attributes of the display, in controlling attention by allowing us to compare performance for known sequences of designated saccadic targets to that obtained for previously examined locations. 

Methods

Eye movement recording

Movements of the right eye with head stabilized were recorded by a Generation IV SRI Double Purkinje Image Eyetracker (sensitivity <1 arcmin) (Crane & Steele, 1978). Tracker output was filtered (100 Hz) and sampled every 2 ms (for details, see Gersch et al., 2004). 

Observers

Three paid volunteers were tested (JT, GT, and ML), each with normal, uncorrected vision. Each was unaware of the purpose of the experiment. 

Stimulus and task

Stimuli were displayed on a Dell P793 CRT monitor (13 deg × 12 deg; viewing distance 115 cm; resolution 1.46 pixels/minarc; refresh rate 75 Hz). Background luminance was 54 cd/m ² and maximum luminance was 108 cd/m ² at the refresh rate used. 

The display (see Figure 1) was a 5 × 5 array of 1° diameter outline circles separated by 1.5° (center-to-center). Five of the circles were green ( x = 0.280 y = 0.602, luminance = 81.6 cd/m ²) and the rest red ( x = 0.628 y = 0.338, luminance = 22 cd/m ²). The 5 × 5 array was bordered by 4 rectangular areas that each held three crosses which served as starting and ending locations for the saccadic sequences. Subjects made saccades to look from one green circle to the next, beginning at the central green cross on one of the 4 sides (chosen randomly) and ending at the central red cross on the opposite side. 

Figure 1

 

(A) Examples of experimental displays. Each contains 25 circles (diameter = 1°, center-to-center separation = 1.5°). The 5 green circles designate the different saccadic paths, starting at the green cross and ending at the red cross (the dashed lines in the figure marking the saccadic paths are for illustration only and were not shown during the actual experiment). Eight different saccadic paths were tested (the 4 shown plus their mirror images). Display orientation varied so that start position (green cross) was either top, bottom, right, or left. (B) Sequence of events during a typical trial. Time runs from top to bottom. The superimposed black line is a representative eye trace showing the path of saccades made from the starting cross (top panel) to the ending cross at the bottom. The letter array and superimposed noise fields appeared briefly (117 ms; second panel) during a randomly chosen intersaccadic pause while the eye was fixating near the middle of the path. After 2 seconds, the location of the probed letter to be reported was indicated by displaying the probed location in either yellow (for on-path locations) or purple (for off-path locations). The indicator remained until the subject reported the letter that appeared at that location by means of a button press. Following the report, the correct answer was shown.

Visual memory was assessed by the ability to remember and identify a letter from an array of 25 letters (1/circle) that was flashed briefly during a randomly selected intersaccadic pause. Each letter was chosen randomly and independently from a set of 10 (A, B, C, D, E, J, K, L, M, N), shown in block style using a custom made font. Horizontal and vertical extent was approximately 45 min arc. The letters were black against a medium gray background (54 cd/m ²). Four frames of the letter array were interleaved with 5 frames of visual noise (13 ms/frame). The noise was a matrix of 20 × 20 dots (dot size = 3 × 3 pixels) whose luminance levels were Gaussian distributed ( SD = 33% maximum display contrast). Interleaved noise was included to maintain consistency with prior work (Dosher & Lu, 2000; Gersch et al., 2004; Gersch et al., in press). 

The location of the probe letter to be identified was chosen randomly from the central set of 9 to avoid testing at the edges of the display. The letters appeared during a randomly chosen time when the eye was likely to be fixating one of the 3 on-path locations within the central 9 of the display. In the main experimental sessions, the location of the to-be-identified letter was not cued in advance of the trial. Separate sessions were run in which the location was cued before and during each trial by setting the color of the probed location to either yellow (for on-path locations) or purple (for off-path locations). 

Procedure

The sequence of events during trials is shown in Figure 1B (time is running from top to bottom). The subjects fixated a green cross and started the trial when ready by pressing a button. After 100 ms a beep sounded for 50 ms, which was the signal to begin making the sequence of saccades. Eight different saccadic paths were tested (the four in Figure 1A along with their mirror images). Display orientation also varied so that start position (green cross) was either top, bottom, right, or left. Subjects were instructed to make a sequence of saccades to each circle along the path, maintaining a steady, brisk pace, not altering the rate of saccades in anticipation of or in response to the letter array (the same instructions as used in Gersch et al., 2004; Gersch et al., in press). To randomize the time of appearance of the letter array during the trial and also to increase the likelihood of the letter array appearing during a pause between saccades (rather than during the saccades themselves), an on-line algorithm monitored the eye-movement data for the occurrence of saccades. The saccades were detected by means of a velocity criterion that was empirically verified for the 3 subjects. The algorithm flagged the first saccade that occurred after the expiration of a random delay, which was set to 300 to 1500 ms after the signal to begin making saccades. Thirty to 160 ms after this flagged saccade, the 9 critical frames (letter + noise) were presented. Subjects continued to scan the display until they reached the red cross on the other side of the screen. Trials lasted 2 s, long enough for the sequence to be completed on the vast majority of trials. 

After the trial was over, the location of the letter to be reported was indicated on a post-trial display by changing the color of the circle in the probed location to either yellow (for on-path locations) or purple (off-path locations). The letter report was given by pressing a button on a 10-key button-box. Feedback was given by presenting the letter within the display of circles after the response was given. 

Sessions were also run in which (1) perceptual performance was tested while steady fixation was maintained at one of the 3 central on-path locations chosen randomly, and (2) saccades were made using the identical stimuli without a letter report taken at the end of the trial. 

Trials were run in blocks of 60–100. Data collection and calibration were done in laboratory visits of about 2 hours each on any given day. There were about 100–150 laboratory visits/subject, distributed over a period of 10 months. 

Analyses of eye movement data

The beginning and ending positions of saccades were detected off-line by means of a computer algorithm employing an acceleration criterion. The “critical saccade” was defined as the first saccade that occurred after the appearance of the letter frames. Eye position at the onset of the critical saccade was used to designate which circle was fixated at the time of the presentation of the critical frames. 

To establish that the saccadic sequences were followed correctly, each saccade was categorized as either following the prescribed path (“good”) or according to the type of error. The majority of errors fell into two categories: saccades that landed off the path or saccades that skipped over a location on the path. Saccades directed back to an on-path location originating from a location off the path and saccades that were directed from one off-path location to another are grouped as “other” in the presentation of the data. Note that only trials in which the critical saccade was on the path were included in the analysis of the perceptual data. 

Other saccadic characteristics that were analyzed were (1) offset error (distance between fixation position and the center of the fixated circle) of the “good” saccades; (2) the average number of targets hit per trial; and (3) the average time interval preceding saccades. 

Trials were omitted from the perceptual results if off-line analyses showed that the letter array appeared at any time during a saccade (<15%). Occasional trials (∼3%) were eliminated because saccades were initiated before the start signal. Data were based on a total of 9227 trials for JT (4847 dual-task, 3960 steady fixation and 420 saccades-only), 5447 trials for GT (2189 dual-task, 2900 steady fixation and 358 saccades-only), and 7483 for ML (2582 dual-task, 4620 steady fixation and 281 saccades-only). Trials eliminated from the analyses of the perceptual results were included in the overall analysis of saccadic performance. 

Statistical analysis: Generalized estimating equations

Analyses of memory performance evaluated the influence of saccadic scanning. Since the dependent variable in this case was binary (correct or incorrect letter identification), logistic regression was used (Hosmer & Lemeshow, 2000). Logistic regression determines the percentage of variance in the dependent variable (the letter report) that is explained by the independent variables, namely, condition (saccadic scanning vs. fixation), path status (on vs. off), and location (ahead vs. behind). 

Logistic regression applies maximum likelihood estimation after transforming the dependent variable into a logit variable (where logit refers to the natural log of the odds of a correct report). The coefficients of the fitted model for the separate independent variables (trial condition, path status, and location) represent the log-odds ratio, which is the natural log of the odds ratio. (The odds ratio is the ratio of the number of correct letter reports to the number of incorrect reports.) Significant main effects of the independent variables are shown by significant coefficients in the fitted model (and their corresponding odds ratios). In addition, the interaction coefficients of the fitted model represent the significance of the interaction between these two independent variables on letter identification. 

To include the data from our three subjects in the analysis, the method of generalized estimating equations (GEE) was used to fit the logistic regression model (Liang & Zeger, 1986). The GEE method takes into account possible within-subject correlations, thus allowing one model to be fit to the data set that consists of multiple observations from 3 subjects. 

Results

The distribution of attention during saccadic scanning and during maintained fixation

We will first describe performance when there was no pre-cue indicating the location of the letter to be probed so that subjects had no reason to preferentially attend to one or another location. Thus, the observed “default” distribution of attentional strength over space reflects the contribution of saccadic planning (or other aspects of the task or display) unaffected by any external incentive to differentially attend to one location or another. Performance will be described both when maintained fixation was maintained throughout the trial and during the performance of the saccadic sequences. 

Memory during maintained fixation

Visual memory performance was measured when the eye remained fixated for the entire trial on one of the three central on-path locations. The set of 3 fixated positions were the same as the 3 tested when the letter array appeared during intersaccadic pauses (see below). In all other respects, the stimuli and procedures used during fixation trials were the same as those during saccadic scanning (see Methods). Tests of performance during fixation and during saccadic scanning were performed on the same days. 

The percentage of post-cued letters recalled correctly during maintained fixation was 42% for JT, 34% for GT, and 38% for ML. Multiplying these percentages by the number of locations tested ( n = 9) works out to 3–4 letters remembered (3.8 for JT, 3.1 for GT, and 3.4 for ML), consistent with the expected capacity of short-term visual memory (Alvarez & Cavanagh, 2004; Sperling, 1960). 

The probability of recalling a letter during maintained fixation depended on two things: retinal eccentricity and path status ( Figure 2A). Memory for letters appearing in one of the 3 on-path (i.e., fixated) locations was better than for letters at off-path (i.e., never fixated) locations at equivalent eccentricities. Memory, either on or off the path, declined with eccentricity. This effect of path—which varied in magnitude across the 3 subjects—shows that either the color differences themselves or the learned significance of the color differences (on-path vs. off-path) influenced which letters were more likely to be encoded into memory during maintained fixation. The performance during fixation (regardless of the source of the on-/off-path differences) establishes a baseline against which the effects of saccadic planning can be evaluated. 

Figure 2

 

Visual memory performance during (A) maintained fixation and (B) pauses between saccades when the location of the probed letter was not disclosed by a cue presented before the trial. Proportion correct reports of letter identification as a function of retinal eccentricity for letters appearing on (green solid lines) or off (red dashed lines) the saccadic path. Data were pooled across the 4 starting locations, 8 different saccadic paths ( Figure 1), and locations with the same eccentricity, with the exception that on- and off-path data were never pooled together. In Figure 2B “Ahead” refers to data obtained when the eye was paused at the first on-path location in the central 9, and “Behind” refers to data obtained when the pause was at the last on-path location in the central 9 (see Figure 4). The dashed black line in each graph represents chance performance (10% correct). Each plotted proportion is based on 50–75 observations during maintained fixation and 75–125 observations during intersaccadic pauses. Error bars show ±1 standard error.

Characteristics of sequences of saccades

The saccadic sequences were performed accurately. Figure 3A shows that the vast majority of saccades followed the prescribed path, with saccades rarely skipping a location or landing off the path. 

Figure 3

 

Summary of saccadic accuracy. Proportion of saccades in 3 categories: “Good” saccades that remained on the path, saccades that strayed off the path, and saccades that skipped over a location on the path. Results are shown separately for (A) No pre-cue and (B) Pre-cue trials. Each proportion is based on 10,000–16,000 saccades/subject.

The on-path saccades landed an average of 18′–24′ from the center of the 1 deg diameter target circles. Average intersaccadic pause durations were 200–260 ms, allowing >4.6 of the 6 targets (5 on-path circles + the ending cross) to be looked at during the trials (see Table 1). These characteristics are typical of saccadic sequences (Gersch et al., 2004; Vishwanath & Kowler, 2003; Zingale & Kowler, 1987), regardless of the presence or absence of a color cue marking the path (Gersch et al., in press). 

Table 1

 

Characteristics of saccades.

No pre-cue a
With concurrent letter identification					Good saccades h
Subject	Proportion of total saccades c				Error at saccade offset d (minarc)	Average number of targets hit per trial e	ISP f (ms)
	Good g	Skips h	Off the path i	Other j	Mean ( SD) N		Mean ( SD) N
JT	0.98	0.001	0.01	0.009	18 (9) 15765	4.7	261 (85) 15765
GT	0.93	0.002	0.02	0.04	20 (11) 12431	5.8	204 (55) 12431
ML	0.82	0.02	0.08	0.08	24 (12) 11021	4.9	239 (82) 11021
Without concurrent letter identification					Good saccades h
Subject	Proportion of total saccades c				Error at saccade offset d (minarc)	Average number of targets hit per trial e	ISP f (ms)
	Good g	Skips h	Off the path i	Other j	Mean ( SD) N		Mean ( SD) N
JT	0.81	0.00	0.11	0.08	23 (11) 2370	4.6	247 (90) 2370
GT	0.91	0.001	0.04	0.05	21 (11) 1159	5.8	202 (51) 1159
ML	0.69	0.006	0.16	0.14	26 (12) 1459	4.7	236 (83) 1459
Pre-cue b
With concurrent letter identification					Good saccades h
Subject	Proportion of total saccades c				Error at saccade offset d (minarc)	Average number of targets hit per trial e	ISP f (ms)
	Good g	Skips h	Off the path i	Other j	Mean ( SD) N		Mean ( SD) N
JT	0.95	0.001	0.03	0.02	19 (10) 13077	4.5	265 (94) 13077
GT	0.74	0.004	0.11	0.15	23 (12) 9245	5.7	197 (57) 9245
ML	0.76	0.02	0.10	0.12	25 (13) 9542	4.8	245 (89) 9542
Without concurrent letter identification					Good saccades h
Subject	Proportion of total saccades c				Error at saccade offset d (minarc)	Average number of targets hit per trial e	ISP f (ms)
	Good g	Skips h	Off the path i	Other j	Mean ( SD) N		Mean ( SD) N
JT	0.77	0.00	0.12	0.11	23 (11) 1005	4.5	246 (95) 1005
GT	0.83	0.001	0.08	0.09	21 (11) 2246	5.7	204 (54) 2246
ML	0.68	0.017	0.15	0.15	27 (12) 1078	4.7	244 (91) 1078

 

Note: ^a“No pre-cue” refers to trials in which the location of the letter reported at the end of the trial was not cued prior to trial start.

 

^b“Pre-cue” refers to trials in which the location of the letter reported at the end of the trial was cued prior to trial start.

 

^c“Total saccades” refers to all saccades except secondary, corrective saccades that followed a primary saccade to a target.

 

^d“Error at saccadic offset” refers to vector distance between eye position at the time of saccadic offset and the center of the nearest circle.

 

^e“Average number of targets hit per trial” refers to number of saccadic targets on the path that were successively fixated during a trial.

 

^f“ISP” refers to the Intersaccadic Pause duration, the interval preceding each good saccade.

 

^g“Good” refers to saccades that followed the prescribed saccadic path.

 

^h“Skips” refers to saccades that skipped the immediately next location on the path and brought the line of sight to a subsequent on-path location.

 

ⁱ“Off the path” refers to saccades that brought the line of sight to a location off the prescribed path.

 

^j“Other” refers to the remaining types of erroneous saccades (off-path to on-path locations, off-path to off-path locations, backward saccades).

The saccadic performance described above was comparable to that observed in control sessions in which the same sequential patterns of saccades were made without the concurrent memory test (see Table 1). Saccades were actually more accurate with the concurrent memory task, with a higher proportion of on-path saccades, and smaller saccadic offset errors. The differences in offset errors when scanning and without the concurrent task were significant (JT: t(2865) = 21.09, p < 0.0001; GT: t(1383) = 2.96, p < 0.01; ML: t(1865) = 5.98, p < 0.0001) ( Table 1, No pre-cue portion). One subject (JT) had longer mean intersaccadic pauses (difference of 14 ms) with the concurrent memory ( t(3039) = 7.11, p < 0.0001) ( Table 1, No pre-cue portion). 

Memory performance during saccadic sequences

Memory performance was poorer overall during saccadic sequences than during maintained fixation

The percentage of letters recalled correctly dropped from >34% during maintained fixation to 22% (JT), 24% (GT), and 26% (ML) during saccadic scanning. These percentages are equivalent to only 2.0 (JT), 2.1 (GT), and 2.3 (ML) letters remembered from the array, fewer than during fixation. Statistical tests using logistic regression and generalized estimating equations (GEE; see Methods, and Gersch et al., in press) showed that subjects remembered significantly fewer letters during scanning than during fixation (odds ratio (OR) = 2.09, p < 0.0001). 

Performance was better for locations on the saccadic path than off the path

The major trends, which held for all 3 subjects, can be seen in Figure 4, which shows ML's performance when the array appeared during intersaccadic pauses at each of the 3 central on-path locations. The numbers inside each circle show the proportion of letters recalled correctly. Memory performance was substantially better for locations on the saccadic path than for the surrounding locations off the saccadic path. Memory for off-path locations was at or near chance. Memory was also better at the saccadic target (the locations denoted by the green arrow, top and middle panels in Figure 4) than at any other on-path location. 

Figure 4

 

Visual memory performance for subject ML when the array appeared while the eye was paused at the first (top), second (middle), and third (bottom) of the central 3 on-path locations. Green circles are on the saccadic path, and red circles are off the path. The location marked by the dashed green circle is the current position of the eye. Locations marked by green arrows (not shown to subjects) are saccadic targets. Numbers inside each circle and the gray levels represent the proportion of correct reports. For each of the 3 locations along the path, data were pooled across the 4 starting locations and 8 different saccadic paths (see Figure 1). Proportions were based on 50–125 observations.

Figure 2B summarizes performance for all subjects as a function of retinal eccentricity. The green solid-line functions show on-path performance. The red dashed-line functions show off-path performance, with data combined across off-path locations with the same retinal eccentricity. The functions on the right side of each graph were obtained while the eye paused at the first of the 3 central on-path locations, and the functions on the left side were obtained for pauses at the last of the 3 on-path locations. Positive values on the abscissa (labeled “Ahead”) refer to saccadic targets; negative values (“Behind”) refer to the on-path locations that were previously examined. Figure 2B shows that the differences between on- and off-path performance were, if anything, larger during intersaccadic pauses than during maintained fixation (compare to Figure 2A), with off-path performance not significantly better than chance (10% correct) (JT, 12.3% correct; t = 2.284, df = 1046, p > 0.01; GT 12.8% correct, t = 1.945, df = 538, p > 0.01; ML 11.5% correct; t = 1.103, df = 584, p > 0.01). On-path performance was also poorer during saccadic scanning relative to maintained fixation (odds ratio (OR) = 1.95, p < 0.0001) but remained substantially better than off-path performance, thus preserving the strong on-path advantage. 

Effects of eccentricity along the saccadic path were overridden by effects of saccadic planning

During saccadic scanning, performance was best at the saccadic target ( Figure 2B). Only when the eye had reached the final on-path location, and thus none of the on-path locations were saccadic targets, did memory return to the typical pattern in which performance was determined by retinal eccentricity. 

The effect of top–down saccadic planning, which produced a spike in performance at the immediate saccadic target, did not consistently extend to the saccadic target locations further ahead on the path. Performance at the on-path location two targets ahead of current fixation was better than performance at the on-path location two targets behind for subjects ML and JT, but not for GT (see Figure 2B; odds ratio not significant over all 3 subjects, OR = 2.41, p > 0.01). 

In summary, these results show that memory performance was determined by two factors operating concurrently: a strong enhancement at the immediate saccadic goal and a global enhancement of the saccadic path, including locations previously examined. 

Influence of time within the trial

The results obtained during saccadic scanning, while the eye was pausing at each of the central on-path locations ( Figure 2B), were taken from three different temporal portions of the trial. To what extent did time within the trial, by itself, affect performance? Figure 5 shows performance for the three different temporal epochs of trials for both the maintained fixation and saccadic conditions. For the saccadic condition, each epoch corresponds to trials when the letter array appeared while the eye had paused at the first, second, or third on-path location. For the fixation condition, trials were divided into 3 groups according to when the letter array appeared during a trial so that the average time of appearance of the letter array for each group was equivalent to that during saccadic scanning. The results in Figure 5 show that time within a trial, by itself, did not influence performance during either fixation or saccadic scanning. Thus, as the eye advanced along the path, the spatial distribution of attention changed, but the overall level of memory performance remained the same. 

Figure 5

 

Letter identification performance (mean proportion of correct identifications) as a function of when during the trial the letter array appeared during (A) saccadic scanning and (B) maintained fixation. For saccadic scanning results, means were averaged over the central 9 locations for each of the 3 possible on-path fixation positions, corresponding to temporal intervals early, middle, and late within the trial. For maintained fixation results, means were averaged over the same 3 possible on-path fixation positions and then divided into three temporal intervals comparable to those obtained for saccadic scanning. These intervals were early (300–550 ms after trial start), middle (550–800 ms), or late (>800 ms). Error bars show ±1 standard error.

Could the effects of saccadic scanning on memory be overridden by pre-cues disclosing the location of the probed letter?

For the results presented thus far, each of the central 9 display locations had an equal chance of being probed. There were no pre-cues disclosing the probed location before the trial, and thus no experimentally induced bias to differentially attend to one or more locations. In separate experimental sessions, this procedure was changed. The location of the letter to be probed at the end of the trial was disclosed by a pre-cue presented before the trial started. A pre-cue condition was included in order to find out whether attention could be shifted away from the saccadic goal without interfering with the saccadic sequence. Such interference has been observed in the past for tasks requiring single saccades (Kowler et al., 1995). The outcome would have bearing on the degree to which saccades and attention can be dissociated by means of “top–down” strategies during saccadic sequences. 

During fixation, the single pre-cued letter was remembered more than 80% of the time for all cases except GT's largest eccentricity ( Figure 6A). Memory during saccadic scanning ( Figure 6B) was considerably poorer (odds ratio (OR) = 3.06, p < 0.0001). The pre-cues, however, were influential in improving performance. They significantly reduced the on-path advantage relative to that obtained without pre-cues [interaction coefficient = 1.1016, p < 0.0001] (compare Figure 6B, pre-cue, with Figure 2B, no pre-cue). Two of the three subjects (JT and ML) continued to show an on-path advantage even with the pre-cue. For GT the on-path advantage disappeared. In addition, JT and ML (but not GT) continued to show better performance at the saccadic goal than at current fixation, just as they had done without the pre-cue. These results show that the pre-cues improved memory for letters off the saccadic path but did not completely override the on-path advantage or the effects of saccadic planning. But these improvements, and their significance for attention, cannot be understood without also examining the effect of the pre-cues on the saccades. 

Figure 6

 

Visual memory performance during (A) maintained fixation and (B) pauses between saccades when the location of the probed letter was disclosed by a pre-cue presented before the trial. Proportion of correct reports of letter identification as a function of retinal eccentricity for letters appearing on (green solid lines) or off (red dashed lines) the saccadic path. Data were pooled across the 4 starting locations, 8 different saccadic paths ( Figure 1), and locations with the same eccentricity, with the exception that on- and off-path data were never pooled together. In Figure 6B “Ahead” refers to data obtained when the eye was paused at the first on-path location in the central 9, and “Behind” refers to data obtained when the pause was at the last on-path location in the central. Proportions at each eccentricity were based on 50–75 observations during steady fixation and 50–200 observations during intersaccadic pauses. Error bars show ±1 standard error.

Pre-cues changed the saccadic pattern

The reduced on-path advantage resulting from the pre-cues was achieved at a cost to saccades. Pre-cues led to an increase in the proportion of erroneous “off-path” saccades ( Figure 3B). These effects were largest for GT, the subject who showed the largest improvement in memory due to the pre-cues (JT: χ ² = 87.26, p < 0.0001; GT: χ ² = 825.51, p < 0.0001; ML: χ ² = 59.70, p < 0.0001) (see also, Table 1, Pre-cue portion). 

The cost to saccades of using the pre-cues can be evaluated by examining attentional operating characteristics (AOCs) (Sperling & Dosher, 1986) showing the tradeoffs between memory and saccadic performance. Memory performance was represented by the overall proportion of correct reports. Saccadic performance was represented by the proportion of “good” (on-path) saccades in Figure 7A and by the duration of the intersaccadic pause containing the letter array in Figure 7B. In both cases, the improved memory performance observed with pre-cues was associated with poorer saccadic performance: either fewer “good” on-path saccades, particularly for GT (Figure 7A), or longer intersaccadic pause durations (Figure 7B). This tradeoff shows that the pre-cues encouraged a more conservative saccadic strategy, in which “top–down” adjustments in attention were achieved by sacrificing the timing and accuracy of saccades. 

Figure 7

 

Attentional operating characteristics (AOCs) showing the cost in saccadic performance of using pre-cues that signaled the location of the probed letter. The ordinate of each AOC shows the proportion of letters correctly identified. The abscissa shows (A) the proportion of saccades following the prescribed path (“Good Saccades”), and (B) mean saccadic latency (duration of the intersaccadic pause preceding the saccade). Following convention for AOCs, the abscissa for mean saccadic latency is inverted so that performance improves (latency decreases) from left to right. Data are shown for sessions with (squares) and without pre-cues (triangles) indicating the location of the probed letters. Data were collapsed across all 3 central fixation positions of the eye. Standard errors each datum point were smaller than the plotting symbols in both the x and y directions.

Discussion

Shifts of attention and saccades are closely tied together, with attention moving to the selected target before the saccade. There are, however, clear disadvantages for perception, as well as visual memory, in linking attention exclusively to saccadic plans. We asked whether a dissociation between attention and saccadic planning, in which attention need not be focused exclusively on the saccadic goal, could be achieved during the performance of saccadic sequences without disrupting the pattern of saccades. We found that dissociations between saccades and attention could be achieved while using saccades to look at targets along color-cued paths. Using a visual memory task to assess attention, we found that memory performance was better for locations on the cued saccadic path than for locations off the path. The advantage for on-path locations, other than the immediate saccadic target, was not connected directly to saccadic planning because the on-path advantage included locations that were previously examined. 

Regardless of the source of the on-path enhancement (see below), the results show that there are mechanisms to support the distribution of attention during saccadic scanning independently of the immediate saccadic plans. Although visual cues were important in allowing the broader distribution of attention (also, Gersch et al., in press), other factors, such as perceptual segmentation cues, or even overlearning, could, in principle, play comparable roles, depending on the task. 

Gersch et al. (in press) found that attention could be distributed along a cued saccadic path in experiments where a perceptual identification task was used to assess attention. The present paper used a visual memory task, which can provide a more sensitive indication of attentional strength. The differences we found between performance on and off the cued saccadic path using a visual memory task were larger than those found with the identification task, with memory for off-path locations falling to chance levels. The memory task also revealed an unexpected strong prioritization within the preferred on-path locations, with letters at the saccadic target, rather than letters at fixation, showing best performance. 

What determines the distribution of attention during saccadic scanning?

These results point to two main processes that operated concurrently to govern the default distribution of attention during saccadic scanning: a “top–down” shift of attention to each saccadic target in sequence and a spread of attention along the saccadic path to locations that shared critical features with the saccadic target. 

The distribution of attention along the saccadic path, which also was found during maintained fixation, is similar to the spread of attention across the visual field observed previously for stimuli sharing critical features (color, for example) with an attended target. Such “feature-based attention” has been studied using brief stimulus presentations in the absence of saccades, with some suggestions that the major role for feature-based attention is to facilitate the selection of saccadic targets, or the guidance of saccades (Bichot et al., 2005; Lu & Itti, 2005; McAdams & Maunsell, 2000; Melcher et al., 2005; Motter, 1994; Sàenz et al., 2003; Treue & Martínez Trujillo, 1999; Wolfe, 1994). We found the attentional advantage along the saccadic path applied to previously viewed locations, and not just to saccadic targets, showing that any spread of feature-based attention is not directly involved in immediate saccadic guidance. It plays at best a supporting (but nevertheless important) role, for example, narrowing the set of possible saccadic targets in a crowded field (Motter & Belky, 1998), or enhancing the perceptual contrast between a selected target and its surround (Cohen et al., 2007). Achieving a sequence of accurate saccades depends on more than feature-based attention. Top–down processes are needed to isolate individual targets and dictate the order of locations to be scanned. 

The patterns of attention we observed are most relevant to those visual tasks that encourage or depend on orderly, planned saccadic sequences. Examples of such tasks, which have been studied in the past, include visual search through arrays of characters (Hooge & Erkelens, 1996, 1999); visuomotor tasks requiring predictable actions, such as tapping a series of rods (Epelboim et al., 1995), or manipulating sets of virtual blocks (Hayhoe, Bensinger, & Ballard, 1998); tasks requiring navigating through the environment (Pelz & Canosa, 2001; Turano, Geruschat, & Baker, 2003); reading (McConkie et al., 1988; Schnitzer & Kowler, 2006); or problem solving (Epelboim & Suppes, 2001; Suppes, Cohen, Laddaga, & Floyd, 1983). In these tasks, the sequential selection of saccadic targets may produce the patterns of attentional enhancement much like those we observed here during sequences of saccades to selected targets. By contrast, less constrained tasks, such as visual search through unstructured arrays (e.g., Motter & Simoni, 2007; Najemnik & Geisler, 2005), have been modeled successfully without allowing for pre-saccadic shifts of attention. Whether such tasks also are affected by pre-saccadic shifts of attention that bias the attentional field is an issue that remains to be resolved. 

Limits on visual memory during fixation pauses and the role of task tradeoffs

Our results are consistent with the view that saccadic planning influences what is remembered during any fixation pause (Currie et al., 2000; Henderson & Hollingworth, 2003; Irwin, 1992; Irwin & Andrews, 1996; Irwin & Gordon, 1998). Memory was clearly much better for the saccadic target than any other location (even current fixation). It is important to note, however, that the limitations imposed by saccadic planning on memory could be offset by cues disclosing the probed locations in advance of trials. Attending to the cued locations, however, resulted in a sacrifice in saccadic performance: longer intersaccadic pauses and poorer saccadic accuracy. This tradeoff verifies the links between saccadic planning and the top–down control of attention. From a practical standpoint, the observed tradeoffs have useful implications for the control of saccades in natural tasks. The option to delay the rate of scanning, or put up with occasional saccadic errors, may be small prices to pay in natural tasks for acquiring greater control of the spatial range of attention and effective visual memory. 

We also found that memory performance overall was poorer during saccadic scanning than during fixation. Reasons for these losses could include factors related to the planning of the saccades themselves or related to the retinal changes produced by saccades. Further work will be needed to address the question of the source of overall performance loss. 

Implications for the neural control of attention and saccades

It is tempting to suppose that the pattern of memory performance we observed across the different locations of the display (see Figure 4), with enhancement on the saccadic path, particularly at the immediate target, represents a composite “saliency map”—a representation of the visual array that assigns weights to locations according to both stimulus properties and “top-down” significance to the task. It is useful to examine the implications of our results in the context of these proposed “maps.” 

Neural areas such as FEF, LIP, SC, or V4 are all plausible sites for such composite maps, with the assumption typically made that a winner-take-all computation downstream from the map determines the location of the saccadic endpoint (Bichot & Schall, 1999; Colby & Goldberg, 1999; Gottlieb, 2007; Shen & Paré, 2007; Thompson, Bichot, & Sato, 2005; Treue, 2003). Thus, according to prevailing views, the saliency map plays a central role in determining saccadic endpoints. If such a computation of saccadic endpoints based on information in the map were to be successful and avoid generating frequent saccades to vivid but unimportant targets, then the “winner” would have to be pre-ordained by a strong top–down signal. The likely sources of such signals in sequential scanning tasks such as ours are neural areas of executive control, such as prefrontal cortex (PFC). Neurons in PFC have been found which represent sequences of saccadic and other motor plans and control the order of executing the responses (Averbeck, Sohn, & Lee, 2006; Fujii & Graybiel, 2003; Hasegawa, Blitz, & Goldberg, 2004; Mushiake, Saito, Sakamoto, Itoyama, & Tanji, 2006; Ninokura, Mushiake, & Tanji, 2004; Shima, Isoda, Mushiake, & Tanji, 2007). But once an accurate top–down signal representing the saccadic target has been produced, it is reasonable to ask what advantage might there be in sending this information through a global saliency map, only to have to recover it again, at the risk of saccadic error? 

The costs to saccades of sending the saccadic plan through a global map—the risk [greater likelihood] of saccadic mislocalizations, or the need for additional computational stages—may be outweighed by the benefits for perception and memory. Saccadic plans provide an effective way to represent and convey top–down information about the locations that contain important or task relevant information. A strong signal representing the selected saccadic target, which is then able to spread its influence over space by, for example, exploiting the connections among units with common feature preferences, could be a highly efficient way to bias perceptual attention to a host of regions with potential value to the ongoing task. Such biases, particularly if they can be manifested without disrupting the execution of saccades, as we have shown they can, would have the effect of selectively improving perception and memory for relevant objects or locations. The resulting perceptual analyses or decisions would then be available to higher-level mechanisms that continually develop new behavioral plans, including new top–down choices about where to look. The benefits of sending saccadic plans to a global map would be realized even if the map should prove not to be an essential stage of saccadic preparation. The main point is that saccadic plans and the accompanying changes in attention can be a tool to enhance portions of the visual field, which then contribute to the development of subsequent and longer range saccadic and behavioral plans. 

Acknowledgments

We thank Jacob Feldman, Manish Singh, Doug DeCarlo, and Thomas Papathomas for their helpful comments and suggestions. 

Commercial relationships: none. 

Corresponding author: Timothy M. Gersch. 

Email: [email protected], [email protected]. 

Address: Mahoney Center for Brain and Behavior, Center for Neurobiology and Behavior, Columbia University College of Physicians and Surgeons, 1051 Riverside Drive, Kolb Research Annex, New York, NY 10032, USA. 

Footnote

Footnotes

¹  Attention can be allocated to selected spatial locations during maintained fixation, showing “independence” of eye movements and attention. But under these circumstances, saccadic plans and attentional decisions are not in conflict because during maintained fixation there are presumably no active saccadic plans. The issues we address here pertain to attention during intervals preceding planned and executed saccades. We are also not assuming that shifts of attention should be treated as the neural equivalent of planned, but unexecuted, saccades (Rizzolatti, Riggio, & Sheliga, 1994), a claim that in its strongest form seems doubtful (Horowitz, Fine, Fencsik, Yurgenson, & Wolfe, 2007; Juan, Shorter-Jacobi, & Schall, 2004).