Visual working memory modulates low-level saccade target selection: Evidence from rapidly generated saccades in the global effect paradigm

Abstract

Abstract

Abstract:

Abstract  In three experiments, we examined the influence of visual working memory (VWM) on the metrics of saccade landing position in a global effect paradigm. Participants executed a saccade to the more eccentric object in an object pair appearing on the horizontal midline, to the left or right of central fixation. While completing the saccade task, participants maintained a color in VWM for an unrelated memory task. Either the color of the saccade target matched the memory color (target match), the color of the distractor matched the memory color (distractor match), or the colors of neither object matched the memory color (no match). In the no-match condition, saccades tended to land at the midpoint between the two objects: the global, or averaging, effect. However, when one of the two objects matched VWM, the distribution of landing position shifted toward the matching object, both for target match and for distractor match. VWM modulation of landing position was observed even for the fastest quartile of saccades, with a mean latency as low as 112 ms. Effects of VWM on such rapidly generated saccades, with latencies in the express-saccade range, indicate that VWM interacts with the initial sweep of visual sensory processing, modulating perceptual input to oculomotor systems and thereby biasing oculomotor selection. As a result, differences in memory match produce effects on landing position similar to the effects generated by differences in physical salience.

Introduction

Eye movements play a central role in the efficient completion of almost all goal-directed human activity. When performing everyday tasks, such as making a cup of tea, the eyes are directed sequentially to individual objects (the kettle, faucet, teabags, and so on) immediately before each object is needed (Land & Hayhoe, 2001); task performance depends on a series of visual search operations, each triggered when a new object becomes relevant. This observed coupling of eye movements to task structure demonstrates that oculomotor selection is subject to strong top-down control (Malcolm & Henderson, 2010; Yarbus, 1967) and cannot be driven solely by low-level stimulus salience (Itti & Koch, 2000). Several forms of top-down guidance have been identified (for a review, see Hollingworth, 2012a), including knowledge of the typical locations of objects in scenes (Henderson, Weeks, & Hollingworth, 1999; Neider & Zelinsky, 2006; Torralba, Oliva, Castelhano, & Henderson, 2006), memory for the particular environment in which the search occurs (Brockmole, Castelhano, & Henderson, 2006; Castelhano & Henderson, 2007; Chun & Jiang, 1998; Hollingworth, 2009, 2012b; Võ & Wolfe, 2013), and memory for the visual properties of the currently relevant object, allowing the formation of a target template (Bravo & Farid, 2009; Malcolm & Henderson, 2009; Vickery, King, & Jiang, 2005; Wolfe, Horowitz, Kenner, Hyle, & Vasan, 2004; Yang & Zelinsky, 2009). 

This last form of control—guidance by knowledge of the visual properties of the relevant object—is a key process that integrates research on attention, eye movements, working memory, and visual search. Theories of attentional selection require a means to specify and keep active the perceptual features of the relevant object (Bundesen, 1990; Desimone & Duncan, 1995; Duncan & Humphreys, 1989; Wolfe, 1994), and most theories propose that visual working memory (VWM) is the substrate of this target template (Bundesen, Habekost, & Kyllingsbaek, 2005; Desimone & Duncan, 1995). When an object is required by the task and needs to be found, features of that object are loaded into VWM, which then guides attention and gaze toward regions of the visual field containing those features. Substantial evidence indicates that VWM can interact with selective operations to bias selection in favor of memory-matching objects and that this influence is at least partially automatic, occurring when attending to a memory-matching object impairs performance (Han & Kim, 2009; Hollingworth & Luck, 2009; Hollingworth, Matsukura, & Luck, 2013; Hollingworth, Richard, & Luck, 2008; Mannan, Kennard, Potter, Pan, & Soto, 2010; Olivers, 2009, 2011; Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005; Soto, Humphreys, & Heinke, 2006). 

Although a large body of research has demonstrated attentional guidance by VWM, the precise locus of interaction has remained unclear. Does VWM interact with the initial sensory processing of visual stimuli, or does it influence only later operations in vision, when competition is maximal (Kastner & Ungerleider, 2000; Luck, Girelli, McDermott, & Ford, 1997)? The evidence so far is mixed. VWM modulates the response of neurons selective for complex, natural objects in areas IT (Chelazzi, Duncan, Miller, & Desimone, 1998; Chelazzi, Miller, Duncan, & Desimone, 1993) and V4 (Chelazzi, Miller, Duncan, & Desimone, 2001), but these effects arise only approximately 150 to 200 ms after stimulus onset, with little or no influence on the initial sensory response, suggesting a relatively late locus for VWM effects. Studies examining feature-based attention have found much earlier effects as a function of match to the current feature template (Bichot, Rossi, & Desimone, 2005; Zhang & Luck, 2009; Zhou & Desimone, 2011), but these studies had no means to confirm that the effects were driven by VWM per se. In particular, feature-based attention studies tend to repeat the same target feature across many consecutive trials, and it is possible that such repetition leads to the representation of the template in long-term memory rather than in VWM (Carlisle, Arita, Pardo, & Woodman, 2011). 

Recent evidence suggesting an early locus of interaction came from a study examining the influence of VWM on rapidly generated saccades to single, abrupt onset targets (Hollingworth et al., 2013). In this study, participants were required to remember a sample color presented at the beginning of each trial in preparation for a memory test at the end of the trial. While they maintained this color in working memory, participants executed a saccade to a sudden-onset disk on the horizontal midline that did or did not match the remembered color. The disk was the only object visible in the display. Orienting saccades were generated more rapidly to target disks that matched the memory color than to targets that did not. In one key experiment, for example, mean saccade latency was 125 ms for memory-matching targets and 136 ms for nonmatching targets. Thus, the working memory representation must have influenced very early processes (i.e., those occurring within 125 ms of stimulus onset). In addition, memory match influenced the metrics of the saccade, with saccades to memory-matching targets landing significantly closer to the center of the object. These results, observed for saccades with latencies well under 150 ms, indicate that VWM modulates the low-level mechanisms underlying rapid orienting responses. 

To examine the effect of VWM on saccade target selection in a competitive context, Hollingworth et al. (2013) included a remote distractor condition, in which a distractor disk was included in the target display, above or below central fixation. Because the target and distractor had an angular separation of 90°, saccades were typically directed discretely to one of the two objects. When the distractor matched VWM and the target did not, 47% of saccades were directed toward the distractor. When the target matched VWM and the distractor did not, only 6% of saccades were directed to the distractor. This type of interaction between VWM and saccade targeting can account for other findings in which VWM-matching items capture attention and recruit gaze in the presence of multiple stimuli (Han & Kim, 2009; Hollingworth & Hwang, 2013; Hollingworth et al., 2008; Hollingworth & Luck, 2009; Mannan et al., 2010; Olivers, 2009; Olivers et al., 2006; Soto et al., 2005; Soto et al., 2006; Soto & Humphreys, 2009). 

The remote distractor condition of Hollingworth et al. (2013) demonstrated that VWM can influence target selection when gaze is oriented discretely to one of two objects. Such an effect could arise via direct modulation of competitive processes within the oculomotor system, but it could also depend on mediating mechanisms, such as the capture of covert attention, which can be dissociated from saccade planning under some conditions (Gregoriou, Gotts, & Desimone, 2012; Schall, 2004). In addition, mean saccade latencies in the remote distractor condition were relatively long (185–225 ms), compared with the single-onset condition, and thus it is not possible to conclude from these data that VWM influenced the competition between potential saccade targets by modulating initial sensory processing. In the present study, we developed a paradigm that more directly probed selective operations within the oculomotor system itself and that was likely to produce saccade latencies similar to those observed in the single-onset condition of Hollingworth et al. (2013). Specifically, we probed the influence of VWM on the metrics of saccade landing position in a global-effect paradigm. 

The global, or averaging, effect describes the tendency for saccades to land between two nearby objects when a saccade is to be directed to one of them (Coren & Hoenig, 1972; Findlay, 1982; Ottes, Van Gisbergen, & Eggermont, 1984), and it is observed most prominently for very rapidly generated saccades (Chou, Sommer, & Schiller, 1999; Ottes, Van Gisbergen, & Eggermont, 1985). The phenomenon is thought to be driven by the interaction between activity gradients within an oculomotor map coding saccade target locations (Godijn & Theeuwes, 2002; Marino, Trappenberg, Dorris, & Munoz, 2012; Meeter, Van der Stigchel, & Theeuwes, 2010; Wilimzig, Schneider, & Schöner, 2006), such as that found in the intermediate layers of the superior colliculus (e.g., Munoz & Wurtz, 1995). When two targets are close in space, local excitatory interactions (or simple summation; Meeter et al., 2010) cause peaks corresponding to the individual objects to merge into a single peak centered between them, and the saccade is then directed to this intermediate location. Longer-range inhibitory connections prevent such spatial integration when objects are relatively far apart, leading to discrete selection. 

Of central relevance to the present study, landing position for averaging saccades varies according to the relative physical properties of the objects. All else being equal, saccades tend to land closer to the more salient object: Saccades land closer to a larger object (Findlay, 1982), to an object of greater dissimilarity from the background (Deubel, Findlay, Jacobs, & Brogan, 1988), and to an object of greater brightness (Deubel, Wolf, & Hauske, 1984). In each of these studies, two closely spaced objects were presented at different eccentricities on the horizontal midline, to the left or right of central fixation, and participants attempted to execute a saccade to the outer, target object. When one of the two objects was more physically salient than the other, mean saccade landing position was shifted systematically toward the more salient object, and this bias was observed both for salient targets and for salient distractors. Mechanistically, salient objects will generate a more rapidly developing and/or more robust activity gradient in oculomotor maps, biasing the averaging process so that the peak of the merged distribution lies closer to that object. Thus, if a match with the current content of VWM increases the initial visual salience of an object, the averaging process likewise should be biased toward the matching object. 

Recently, Silvis and Van der Stigchel (in press) reported evidence relevant to this topic from a VWM manipulation in a variant of the global effect paradigm. Participants maintained a color in VWM as they executed a saccade to a pair of equidistant objects presented in close proximity. They were instructed to orient generally to the object pair, without specification of one object as the target. Saccades tended to land nearer the object that matched the remembered color. This result demonstrates that VWM match modulates saccadic selection in the presence of multiple objects (see also Hollingworth et al., 2013). However, the mean saccade latency observed by Silvis and Van der Stigchel (in press) was approximately 350 ms, substantially longer than saccade latencies observed in global effect studies probing differences in salience. For example, the mean saccade latency in Findlay (1982) was approximately 140 ms. Silvis and Van der Stigchel (in press) examined their data as a function of saccade latency quartile, but even saccades in the fastest quartile were generated relatively slowly, with an upper bound of 289 ms, and the authors did not observe a statistically reliable effect of memory match within this quartile. Thus, these data do not address the very early time course of the effect. In the present study, we adopted the method used in earlier global effect experiments (Deubel et al., 1984; Deubel et al., 1988; Findlay, 1982) in order to probe the influence of VWM on very rapidly generated saccades. 

In our experiments, participants maintained a color in VWM as they executed a saccade to the more eccentric of two closely spaced objects on the horizontal midline (Figure 1). The match between the remembered color and the colors of the target and distractor objects was manipulated. We tested two predictions derived from the hypothesis that VWM interacts early with sensory processing to modulate the initial visual salience of an object (Hollingworth et al., 2013). First, the landing position of the saccade should be biased toward whichever object matches the content of VWM: VWM match should generate effects on landing position similar to those that have been observed previously for differences in physical salience. In particular, landing position should be biased toward a memory matching distractor despite participants' intention to orient to the target. Second, because we propose that VWM interacts with the initial wave of sensory processing and input to oculomotor systems, these effects should be observed even for saccades with latencies near the limit of human saccade generation times. 

Figure 1

 

(A–D) Sequence of events on a trial of Experiment 1. Participants fixated a central cross. A color memory square was presented for 300 ms (A), followed by an interstimulus interval of 700 ms (B). A pair of objects was then presented to the left or right of fixation, and participants attempted to execute a saccade to a more eccentric object, which was also the larger of the two (C). After the target was fixated, there was a 200-ms delay, followed by the presentation of two memory test stimuli (D), the original color and a foil drawn from the same color category. Participants responded to select the color that matched the remembered color. The eye icons show horizontal fixation position. (E) Sample saccade task stimuli for the three memory-match conditions.

In addition to testing these predictions, we examined the interaction between two forms of bias—the automatic effects of VWM match and strategic orienting to the target object—by observing the temporal evolution of the VWM-match effect with increasing saccade latency. It is well established that as saccade latency increases in the global effect paradigm, the eyes land progressively closer to the target (e.g., Ottes et al., 1985), reflecting the increasing influence of strategic mechanisms to bias selection in favor of the intended target object. In the present paradigm, when the distractor matched memory, memory match was expected to bias landing position for short latency saccades, before strategic influences became functional. However, for longer latency saccades, these two forms of bias should be in conflict, allowing us to examine the relative influences of strategic control and VWM bias. 

Experiment 1

Participants maintained a color in memory as they executed a saccade to one of two objects that appeared on the horizontal midline (Figure 1). First, a color patch was presented, to be held in memory for a within-category discrimination test at the end of the trial. Next, colored target and distractor disks appeared simultaneously. The target disk was always the more eccentric object, and it was larger than the distractor. Participants were instructed to generate a saccade as quickly as possible to the outer of the two objects. The match between the sample color and the colors of the target and distractor was manipulated: The target matched the memory color category, and the distractor did not (target match); the distractor matched the memory color category, and the target did not (distractor match); or neither matched the memory color category (no match). After fixating the target, participants saw two test colors and reported which of the two matched the original color patch. 

Method

Participants

In all three experiments, participants were between 18 and 30 years of age, reported 20/20 uncorrected vision, and received course credit or pay for their participation. Twelve participants completed Experiment 1. 

Stimuli

Memory- and saccade-task stimuli appeared against a gray background with a central, white fixation cross subtending 0.3°. 

The memory sample display (Figure 1A) consisted of a 1.6° × 1.6° colored square at the center of the screen. The color category was selected randomly from red, green, and blue. Within the category, the color value was selected randomly from four similar colors. The 1931 CIE color coordinate system values (x, y, and luminance) were measured using a Tektronix model J17 colorimeter. The four reds were x = 0.53, y = 0.27, 19.0 cd/m²; x = 0.61, y = 0.30, 17.8 cd/m²; x = 0.65, y = 0.33, 17.7 cd/m²; and x = 0.63, y = 0.34, 19.1 cd/m². The four blues were x = 0.17, y = 0.15, 11.8 cd/m²; x = 0.16, y = 0.14, 10.0 cd/m²; x = 0.17, y = 0.12, 9.3 cd/m²; and x = 0.17, y = 0.10, 8.8 cd/m². The four greens were x = 0.32, y = 0.59, 33.1 cd/m²; x = 0.30, y = 0.60, 32.2 cd/m²; x = 0.29, y = 0.53, 33.1 cd/m²; and x = 0.26, y = 0.44, 34.5 cd/m². 

In the memory test display (Figure 1D), two 1.6° × 1.6° color squares were presented to the left and right of the central fixation cross at an eccentricity of 2.5° (all reported eccentricities are to the center of the object). One color was the same as the sample color (correct alternative), and the other was drawn randomly from the remaining three colors in the target category (foil). The left-right positions of the two alternatives were determined randomly. This within-category discrimination task minimized the role of verbal encoding. Encoding the sample stimulus with a simple verbal label (e.g., “blue”) would not have been sufficient to choose between the two color alternatives. 

The saccade task display (Figure 1C) contained two objects: a saccade target disk (0.97° diameter) and a smaller distractor disk (0.65° diameter). All stimuli appeared on the horizontal midline. The target was displayed either to the left or right of central fixation. Direction was selected randomly, and eccentricity was selected randomly within a range (4.6° to 7.0°). The distractor always appeared 2.3° closer to the central fixation point than did the target. Thus, the interobject distance between target and distractor remained constant across trials. The relation between the remembered color and the colors of the target and distractor disks was manipulated: target match, distractor match, or no match. 

When the target or distractor matched the memory category, the match was either exact or inexact. On exact-match trials, the color was the same as the sample color. On inexact-match trials, the color was selected randomly from the remaining three colors in the same category. An inexact-match color always became the foil color in the memory test display. Because a category-matching color in the saccade display was equally likely to be the correct color or the foil color in the upcoming memory test, participants could derive no benefit from strategically attending to the colors of the stimuli in the saccade display (see Olivers et al., 2006). 

Apparatus

Stimuli were displayed on a 17-in. CRT monitor with a refresh rate of 120 Hz. The position of the right eye was monitored by an SR Research EyeLink 1000 eye tracker sampling at 1000 Hz. A chin and forehead rest maintained a viewing distance of 70 cm and minimized head movement. Manual responses were collected with a serial button box. Screen events, eye events, and manual responses were coordinated by E-prime software (Schneider, Eschmann, & Zuccolotto, 2002). 

Design and procedure

Upon arriving for the experiment session, participants provided informed consent and were instructed in the task. The eye tracker was calibrated. The eye tracker was recalibrated during the experiment if the position estimate deviated from the calibration points by more than approximately 0.75°. 

Each trial began with central fixation. The experimenter initiated the trial. After a delay of 400 ms, the memory sample square was presented for 300 ms (Figure 1A), followed by a blank (fixation cross only) delay of 700 ms (Figure 1B). Then, the saccade target and distractor were presented (Figure 1C). Participants were instructed to ignore the distractor and to execute a saccade directly to the target. The target was always larger and more eccentric than the distractor, minimizing any difficulty in distinguishing between the two objects. 

When a fixation was detected in the target region, the target display remained visible for an additional 200 ms and was then replaced with the memory test display (Figure 1D). Participants pressed one of two buttons to indicate whether the left or right square was exactly the same color as the sample square presented at the beginning on the trial. The button response terminated the trial. The next trial was initiated when the participant had returned his or her gaze to the central fixation cross. 

Participants completed a practice session of 16 trials, followed by an experiment session of 400 trials: 100 trials of target match, 100 trials of distractor match, and 200 trials of no match. The match trials were divided evenly between exact and inexact match. Trials from the different conditions were randomly intermixed. 

Note that in this design, the color of the saccade target matched the category of the memory color on only 25% of trials, and when it matched the memory category, it was equally likely to be an exact match or an inexact match. Therefore, any effect of memory match on saccades could not have been caused by an expectation that the target would match the remembered color. 

Data analysis

Eye-tracking data were analyzed offline. A combined velocity (>30°/s) and acceleration (> 8000°/s²) threshold was used to define saccades. Trials were eliminated from the analysis if the participant was not fixating within 1° of the center cross when the target stimulus appeared (14.8% of trials), if saccade latency was greater than 400 ms or less than 60 ms (3.6% of remaining trials), or if the first saccade did not land within 3.5° of the target center (1.9% of remaining trials). A total of 19.4% of trials were eliminated. In the three experiments, elimination of these trials did not alter the pattern of results, and the proportion of eliminated trials did not differ among memory-match conditions. 

Results

Eye movement results

The primary measure was the landing position of the first saccade following the onset of the saccade task stimuli. We also examined the latency of the saccade and the relation between latency and landing position. For the landing position analysis, we report the horizontal landing position relative to the center of the saccade target object. Landing positions short of the saccade target center were assigned negative values, and positions beyond the saccade target center were assigned positive values. In all three experiments, analyses over the unsigned, absolute landing error produced the same pattern of results. Inclusion of the vertical component of the saccade did not alter the pattern of results. Limiting the eye movement analyses to trials on which the participant responded correctly on the color memory test also did not alter the pattern of results, so all trials were included. Finally, the data did not differ as a function of exact/inexact match, and this factor was collapsed. The absence of an effect of exact/inexact memory match is not surprising, given that target stimuli were presented 4.6° to 7.0° in the periphery. If saccade-task stimuli are presented near the fovea, a difference between exact/inexact match can be observed. For example, in experiment 2 of Hollingworth et al. (2013), saccades were significantly more likely to be directed to a near-foveal distractor (1.3° eccentricity) when it was an exact match versus an inexact match. 

Landing position:

The distributions of saccade landing position relative to the target are displayed in Figure 2A. Saccades in the no-match condition tended to land between the distractor and the target. This condition provides a baseline measure of the standard global effect. When the target matched the memory color category, the distribution of landing position shifted toward the target location relative to the no-match baseline. Similarly, when the distractor matched the memory color category, the distribution shifted toward the distractor location relative to the no-match baseline. The mean landing position from each subject on each of these three trial types was entered into a one-way, within-subjects analysis of variance (ANOVA), which confirmed that the differences in mean landing position across trial types were reliable, F(2, 22) = 35.6, p < 0.001, η_p² = 0.76. A follow-up test showed that the mean landing position was closer to the target in the target-match condition (−1.05°) than in the no-match condition (−1.22°), F(1, 11) = 22.3, p < 0.001, η_p² = 0.67. In addition, the mean landing position was farther from the target in the distractor-match condition (−1.40°) than in the no-match condition, F(1, 11) = 20.6, p < 0.001, η_p² = 0.65. 

Figure 2

 

(A) Distributions of horizontal landing position error (relative to the target center) as a function of memory-match condition in Experiment 1. For this and all subsequent graphs, leftward saccades were normalized for depiction as a rightward saccade. The blue ring and light-blue field illustrate the horizontal location of the target object. The red ring and light-red field illustrate the horizontal location of the distractor. (B) Mean landing position error in Experiment 1 as a function of memory-match condition, for all trials and for trials divided by saccade latency quartile. The end of each bar marks the mean landing position relative to the target and distractor regions depicted in the figure. Error bars are condition-specific, within-subject 95% confidence intervals (Morey, 2008).

Latency:

Overall, saccades were generated rapidly, with a mean saccade latency of 140 ms across the entire experiment. This is consistent with saccade latencies observed in other global effect experiments implementing a similar design (e.g., Findlay, 1982). Memory match produced a statistically marginal effect on latency, F(2, 22) = 3.73, p = 0.053, η_p² = 0.23. Numerically, mean latency was lower in the target-match condition (138 ms) than in the no-match (143 ms) and distractor-match (140 ms) conditions. Thus, more accurate saccades in the target-match condition cannot be attributed to participants taking longer to initiate saccades in that condition. 

Landing position as a function of latency:

To examine the time course of memory-match effects, for each participant the landing position data were divided into quartiles by the latency of the initial saccade. Defining the quartile boundaries in a participant-by-participant manner ensured that each participant had an equal distribution of trials across quartiles. The first quartile had a mean latency of 112 ms, the second 129 ms, the third 143 ms, and the fourth 178 ms. The mean horizontal landing position is displayed in Figure 2B as a function of quartile. Saccades were reliably more accurate in the target-match condition than in the distractor-match condition in all four quartiles. In particular, there was a reliable effect of VWM match on landing position in the fastest quartile, F(1, 11) = 16.3, p = 0.002, η_p² = 0.60. Saccade latencies in this quartile varied between 63 and 142 ms, with a mean latency of 112 ms and with 73% of saccades generated in less than 120 ms. Thus, the finding of significant effects of VWM match in this quartile indicates that VWM modulates relative salience extremely rapidly. 

The effect of memory match increased with increasing saccade latency, generating a reliable interaction between memory match (target match/distractor match) and quartile in a two-way ANOVA, F(3, 33) = 3.50, p = 0.026, η_p² = 0.24. Saccade latency quartile produced a marginal effect on landing position in a follow-up one-way ANOVA that was limited to the target-match condition, F(3, 33) = 2.61, p = 0.068, η_p² = 0.19, with the eyes landing closer to the target as latency increased. Thus, with more time, the competition between target and distractor was more likely to be biased in favor of the target (Ottes et al., 1985). In contrast, there was no effect of latency quartile in a one-way ANOVA limited to the distractor-match condition, F < 1, indicating that the memory-matching distractor interfered with the typical evolution of competition in favor of the target. 

Color memory results

The mean accuracy on the color memory test was 79.1%. There was no effect of whether the display contained a matching object (79.3%) or no matching object (78.1%), F < 1. To examine the influence of exact/inexact match on memory performance, the data from the target-match and distractor-match conditions were examined as a function of exact/inexact match in a two-way ANOVA. There was no reliable main effect of target/distractor match, F(1, 11) = 3.33, p = 0.095, η_p² = 0.23, with 77.8% correct in the target-match condition and 80.8% correct in the distractor-match condition. However, there was a reliable main effect of exact/inexact match, F(1, 11) = 7.45, p = 0.020, η_p² = 0.40. Accuracy was higher when the match was exact (81.9%) than when it was inexact (76.7%). These factors did not interact, F < 1. 

The effect of exact/inexact match on color memory accuracy could have arisen in two ways. First, participants might have confused the color values of the memory square and saccade stimuli when making their response; on some trials, they may have incorrectly reported the saccade stimulus color rather than the memory color. When the match was inexact, this would have led to an incorrect response. A second possibility is that, on inexact-match trials, attention to and perceptual processing of the saccade stimuli may have interacted with the memory representation of the color patch, shifting that representation toward the saccade stimulus color. Such a shift would have increased the probability of selection of the foil color in the two alternative tests. 

Discussion

In Experiment 1, the landing position of averaging saccades was biased toward the object in the pair that matched a color value maintained in VWM. This effect was observed even for the fastest quartile of saccades, with a mean latency of only 112 ms. Saccades of this latency lie near the limit of human capabilities. Thus, the data provide strong support for the hypothesis that VWM content interacts with initial sensory processing, modulating sensory input to oculomotor systems. In addition, VWM match counteracted strategic orienting to the target. When the target object matched memory, saccades tended to land progressively closer to the target as saccade latency increased, reflecting the influence of strategic selection. However, when the distractor matched VWM, there was no relation between latency and landing position: The landing position of even relatively long-latency saccades remained biased toward the memory-matching distractor. 

Experiment 2

In Experiment 1, the task provided no incentive for participants to strategically attend to the color of the items in the saccade display. However, it is possible that participants believed that the color would be useful. To eliminate completely the possibility of a strategic bias, color was made an incidental property of the remembered object in Experiment 2. Participants remembered the orientation of a star-shaped object that was rendered in a task-irrelevant color. Consequently, the task provided no demand to remember color, eliminating any reason for participants to strategically attend to color-matching objects. However, intentionally encoding one property of an object typically leads to incidental encoding of other properties (Hollingworth et al., 2013; Hyun, Woodman, Vogel, Hollingworth, & Luck, 2009), and we therefore predicted that the color of this object would be present in VWM and influence saccades to the target. 

Method

Participants

Twelve new participants completed the experiment. 

Stimuli

The stimuli were the same as in Experiment 1, except for the stimuli in the memory task. The memory stimulus was a four-pointed, star-shaped object subtending 2.0° (see Figure 3). It was drawn in a color selected randomly from the set of colors used in Experiment 1. The orientation of the memory stimulus was chosen randomly on each trial. In the memory test display, the original memory stimulus was paired with a foil that differed by 10° of orientation (direction chosen randomly). 

Figure 3

 

Sample memory display (left panel) and test display (right panel) stimuli for the orientation memory task in Experiment 2.

Design and procedure

The design and procedure were the same as in Experiment 1, with two exceptions. First, the memory test was orientation discrimination rather than color discrimination. Second, when one of the saccade task stimuli matched the color of the memory item, the match was always exact. 

Data analysis

Trials were eliminated according to the same criteria as in Experiment 1: if the participant was not fixating within 1° of the center cross when the target stimulus appeared (8.4% of trials), if the first saccade did not land within 3.5° of the target center (2.7% of remaining trials), or if saccade latency was greater than 400 ms or less than 60 ms (0.6% of remaining trials). 

Results and discussion

The results of Experiment 2 replicated the principal findings in Experiment 1. The absolute magnitudes of the memory-match effects were reduced compared with those in Experiment 1 (presumably because the incidental encoding of color in this experiment was weaker than the intentional encoding in Experiment 1), but the effects remained significant. In addition, as saccade latency increased, saccades in the target-match condition landed progressively closer to the target, but saccades in the distractor-match condition remained biased toward the distractor location. 

Eye movement results

Landing position:

The distributions of landing position relative to the target are displayed in Figure 4A. There was a reliable effect of memory-match condition, F(2, 22) = 24.5, p < 0.001, η_p² = 0.69. Mean landing position was closer to the target on target-match trials (−0.84°) than on no-match trials (−0.92°), F(1, 11) = 13.9, p = 0.003, η_p² = 0.56. The mean landing position was farther from the target on distractor-match trials (−1.12°) than on no-match trials, F(1, 11) = 19.2, p = 0.001, η_p² = 0.63. Although reliable, the effects of memory match were reduced in magnitude compared with those in Experiment 1. This finding is consistent with previous studies indicating weaker effects of memory match for task-irrelevant features of the remembered object compared with task-relevant features (Hollingworth et al., 2013; Olivers et al., 2006). The effect of a task-irrelevant feature suggests that the present method may be more sensitive to memory-match effects than traditional visual search tasks; memory-match effects were eliminated entirely from the visual search task of Olivers et al. (2006) when the matching feature was an incidental property of the remembered object. 

Figure 4

 

(A) Distributions of horizontal landing position error (relative to the target center) as a function of memory-match condition in Experiment 2. (B) Mean landing position error in Experiment 2 as a function of memory-match condition, for all trials and for trials divided by saccade latency quartile. The end of each bar marks the mean landing position relative to the target and distractor regions depicted in the figure. Error bars are condition-specific, within-subject 95% confidence intervals (Morey, 2008).

Latency:

Saccades were generated in an average of 164 ms across the entire experiment. Memory match produced a reliable effect on latency, F(2, 22) = 8.40, p = 0.002, η_p² = 0.43. As in Experiment 1, mean latency was lower in the target-match condition (161 ms) than in the no-match (164 ms) and distractor-match (168 ms) conditions. 

Landing position as a function of latency:

The mean horizontal landing position is displayed in Figure 4B for each latency quartile. The first quartile had a mean latency of 127 ms, the second 151 ms, the third 171 ms, and the fourth 212 ms. Saccades were reliably more accurate in the target-match condition than in the distractor-match condition in all four quartiles. In particular, there was a reliable effect of VWM match on landing position in the fastest quartile, F(1, 11) = 5.23, p = 0.04, η_p² = 0.32. 

As in Experiment 1, the effect of memory match increased with increasing saccade latency, generating a reliable interaction between memory match (target match/distractor match) and quartile, F(3, 33) = 6.59, p = 0.001, η_p² = 0.37. Saccade latency quartile produced a reliable effect on landing position in the target-match condition, F(3, 33) = 4.18, p = 0.013, η_p² = 0.28, with the eyes landing closer to the target as latency increased. There was no effect of latency quartile in the distractor-match condition, F < 1. Again, a memory-matching distractor interfered with the typical evolution of competition in favor of the target. 

Overall, saccades in Experiment 2 landed closer to the target object than in Experiment 1 and were generated more slowly than in Experiment 1. The longer delay in executing the saccade presumably allowed more time to resolve the competition between target and distractor in favor of the target, consistent with prior reports (Ottes et al., 1985). 

Orientation memory results

Mean accuracy on the orientation discrimination task was 69.8%. There was no effect of the match between the color of the remembered shape and the saccade stimuli, F(2, 22) = 2.54, p = 0.102, η_p² = 0.19, with 69.8% correct in the target-match condition, 68.1% correct in the no-match condition, and 71.6% correct in the distractor-match condition. 

Experiment 3

As in other studies examining the influence of VWM on perceptual selection (Hollingworth et al., 2013; Olivers et al., 2006; Soto et al., 2005), we sought to ensure that the effects of memory match could not be attributed to perceptual priming from simply having viewed the memory stimulus before the saccade task. Experiment 3 was the same as Experiment 1, except there was no memory test at the end of the trial, and participants were not asked to remember the color patch that appeared at the beginning of the trial. Instead, they were told that the color patch simply informed them that the trial was about to begin. Thus, participants in Experiment 3 saw the same color stimulus before the saccade task as presented in Experiment 1, but there was no demand to maintain that color in VWM during the trial. 

Method

Participants

Twelve new participants completed the experiment. 

Design and procedure

The design and procedure were the same as in Experiment 1, except there was no memory test at the end of the trial: The trial ended with fixation of the target object. 

Data analysis

Trials were eliminated according to the same criteria as in Experiment 1: if the participant was not fixating within 1° of the center cross when the target stimulus appeared (10.2% of trials), if the first saccade did not land within 3.5° of the target center (1.4% of remaining trials), or if saccade latency was greater than 400 ms or less than 60 ms (0.8% of remaining trials). 

Results and discussion

With no demand to remember the color patch, all effects of color match were eliminated, demonstrating that the results in previous experiments could not have been caused by perceptual priming. 

Landing position

Landing position distributions are displayed in Figure 5A. The influence of memory match on landing position was eliminated entirely, with no effect of memory-match condition, F < 1. Mean landing position was −0.76° in the target-match condition, −0.73° in the no-match condition, and −0.76° in the distractor-match condition. 

Figure 5

 

(A) Distributions of horizontal landing position error (relative to the target center) as a function of memory-match condition in Experiment 3. (B) Mean landing position error in Experiment 3 as a function of memory-match condition, for all trials and for trials divided by saccade latency quartile. The end of each bar marks the mean landing position relative to the target and distractor regions depicted in the figure. Error bars are condition-specific, within-subject 95% confidence intervals (Morey, 2008).

Latency

Memory match also had no hint of an effect on latency, F < 1. Mean latency was 172 ms in the target-match condition, 172 ms in the no-match condition, and 173 ms in the distractor-match condition. 

Landing position as a function of latency

The mean horizontal landing error is displayed in Figure 5B for each saccade latency quartile. In no quartile did accuracy in the target-match and no-match conditions differ. 

General discussion

The fine-grained metrics of averaging saccades are influenced by physical differences in salience between the two objects, with the eyes tending to land closer to the more salient object (Deubel et al., 1984; Deubel et al., 1988; Findlay, 1982). In the present study, we examined whether the match between an object's color and a color maintained in VWM would produce an effect similar to that observed for physical differences. We predicted that if the content of VWM interacts with the initial sensory processing of visual stimuli to increase the relative salience of items matching memory (Hollingworth et al., 2013), landing position should be biased toward the memory-matching object, and this effect should be observed even for the fastest saccades in the distribution. 

These predictions were confirmed. Overall, the eyes landed closer to the target when the target matched the remembered color than when the distractor matched the remembered color (see also Silvis & Van der Stigchel, in press). In Experiment 1, the difference in mean landing position between the target-match and distractor-match conditions was 0.35°, 15.2% of the distance between the two objects (2.3°). In Deubel et al. (1984), who manipulated relative luminance with an interobject distance of 4.0°, an equivalent absolute difference in landing position would have required a luminance ratio of approximately 1.8:1 (estimated from Figure 2 of Deubel et al., 1984). An equivalent difference in terms of the percentage distance between the two objects would have required a luminance ratio of approximately 4.2:1. These comparisons are limited by methodological differences between the two studies, but it is clear that memory match has a substantial effect on landing position and that obtaining an equivalent effect with a manipulation of physical difference would require stimuli that are easily distinguishable. 

This raises the question of whether VWM altered the appearance of the stimuli. Although we did not have a means to assess conscious perception of the stimuli, several recent studies suggest that VWM content interacts with perceptual processing to bias conscious perception (Kang, Hong, Blake, & Woodman, 2011; Scocchia, Cicchini, & Triesch, 2013; Scocchia, Valsecchi, Gegenfurtner, & Triesch, 2013) or change the timing of conscious perception (Gayet, Paffen, & Van der Stigchel, in press; Pan & Luo, 2012). Using a continuous flash suppression technique, Gayet et al. (in press) found that stimuli matching VWM content were perceived earlier than nonmatching stimuli. Pan and Luo (2012) observed that the perceived duration of memory matching stimuli was longer than that for nonmatching stimuli, consistent with a more robust sensory response to the former (Eagleman & Pariyadath, 2009). Although these results are broadly consistent with the results reported here, they cannot speak directly to the time course of VWM modulation, as effects on conscious perception could arise either through initial modulation of the sensory response or through later feedback from higher-level representations of the remembered stimuli (Desimone & Duncan, 1995). 

In the present study, VWM modulation of landing position was observed even in the fastest quartile of saccades, with a mean saccade latency as low as 112 ms, providing strong support for the hypothesis that the maintenance of features in VWM interacts with the initial sensory processing of visual stimuli. To provide converging evidence regarding the time course of VWM effects, we reanalyzed the data of Hollingworth et al. (2013, experiment 1), in which participants executed a saccade to a single-onset target that did or did not match the category of a color held in VWM. Saccades landed closer to the center of the target when it matched VWM. As in the present study, we divided the data into saccade latency quartiles. Consistent with the present results, a reliable effect of VWM match on saccade landing position was observed even in the fastest quartile, F(1, 11) = 9.28, p = 0.01, η_p² = 0.46. Mean saccade latency in this quartile was 113 ms. Results from a similar experiment reported by Schneegans, Spencer, Hollingworth, and Schöner (2011) also yielded a significant effect of memory match in the fastest quartile of saccades, F(1, 11) = 10.1, p = 0.009, η_p² = 0.48, with a mean latency of 107 ms in the quartile. Thus, VWM influences the metrics of eye movements with latencies that fall within the express-saccade range (e.g., Fischer & Ramsperger, 1984). 

These results are consistent with evidence from studies of feature-based attention that have found very early effects on the sensory response to visual stimuli (Bichot et al., 2005; Zhang & Luck, 2009; Zhou & Desimone, 2011). In these experiments, there was no direct manipulation of VWM content, but the tasks were likely to have introduced a demand to maintain template features in VWM. For example, Bichot et al. cued monkeys to search for a target of a particular color within a search array composed of colored shapes. Because the color cue was removed before the search commenced, and because the target color changed regularly (approximately every five trials) during the experiment (Carlisle et al., 2011), it is plausible that VWM was used to maintain the current target template (Woodman & Arita, 2011). When the monkey fixated a distractor, and a different object fell within the receptive field of a V4 neuron selective for the color of that object, the sensory response was enhanced when that color was the target color. The sensory response to target and nontarget colors diverged almost immediately from the onset of the fixation, consistent with an interaction between template features (presumably maintained in VWM) and the first sweep of sensory information at the onset of the fixation. 

The present paradigm differs from studies of feature-based attention in that participants had no incentive to attend to items that matched the content of VWM. We went to considerable lengths to eliminate strategic orienting to memory-matching objects by making the remembered color antipredictive of the saccade target color, by making the saccade-task colors unpredictive of the correct response on the memory test, and, in Experiment 2, by making color a task-irrelevant feature of the memory stimulus. Thus, the present effects reflect a relatively automatic influence of VWM representations on saccadic orienting. Nonetheless, it is likely that feature-based selection and the VWM-based effects observed here depend on the same underlying mechanism. If VWM guides selection when it is irrelevant to or in conflict with participant goals, then the same mechanism would produce efficient orienting to behaviorally relevant stimuli when VWM is used, instead, to represent features of the desired object. Although they are likely to be overlapping mechanisms, VWM and feature-based attention are not necessarily equivalent, as maintenance in VWM is not always sufficient for attentional guidance (Hollingworth & Hwang, 2013; Houtkamp & Roelfsema, 2006; Woodman & Luck, 2007) and attentional guidance can be achieved without VWM representations (Carlisle et al., 2011). The difference between items in VWM that do and do not interact with perceptual selection may derive from differences in the extent to which they involve sustained, delay period activation in visual sensory regions and thus the extent to which they interact with the perceptual processing of visual stimuli. 

The present results also inform the debate over bottom-up versus top-down influences on perceptual selection and attention capture, which has hinged on resolving whether early forms of orienting are driven solely by stimulus properties (Theeuwes, 1991) or are contingent on attentional set (Folk, Remington, & Johnston, 1992). Previous evidence has suggested that rapidly generated saccades are influenced only by the physical properties of the stimulus (Ludwig & Gilchrist, 2002; van Zoest, Donk, & Theeuwes, 2004), consistent with the former view. The present results, along with those of Hollingworth et al. (2013), clearly show that saccades with latencies near the limit of human capabilities can be modulated by the maintenance of perceptual features in VWM. In fact, the tendency for saccades to land near the memory-matching distractor in the present study, at saccade latencies near 100 ms, can be considered the most direct demonstration to date that even the most rapid forms of orienting are modulated by strategic factors. 

Finally, the present findings necessitate modification of existing models of saccade target selection (Meeter et al., 2010; Trappenberg, Dorris, Munoz, & Klein, 2001; Wilimzig et al., 2006). These models provide general accounts of the competitive dynamics of saccade target selection, and each can emulate the averaging effect observed when rapid saccades are generated to closely spaced objects. In addition, each of these models combines bottom-up, sensory guidance with top-down spatial guidance, such as that derived from knowledge of the target location. However, none implements a feature-based mechanism of guidance that could account for the automatic effects of VWM maintenance on selection or for strategic control based on knowledge of the target's surface features (i.e., a target template). To implement this type of functionality, Schneegans et al. (2011) integrated the neurodynamic model of saccade target selection developed by Schöner and colleagues (Kopecz & Schöner, 1995; Wilimzig et al., 2006) with the neurodynamic model of VWM developed by Johnson, Spencer, Luck, and Schöner (2009). In the combined model, a saccade-planning field (coding locations for saccade target selection) and a VWM field (coding values in color space) are coupled to a shared, low-level sensory field coding color values across space. The maintenance of a color in VWM feeds back to the low-level sensory field, preactivating that value across the visual field. This feedback implements feature-based attention. When two stimuli appear in a global-effect display, the peak in the sensory field at the location of the memory-matching object is generated more quickly and is more robust. This difference in sensory response is equivalent to the difference that would be generated by stimuli with different physical salience. When local excitatory interactions in the saccade-planning field cause the two peaks to merge, the difference in sensory input for the two objects causes the central tendency of the merged peak to be biased toward the location of the memory-matching object. Thus, the model captures the basic landing position results in the present study in a fairly simple architecture based on the assumption that VWM modulates sensory input to systems responsible for oculomotor selection. 

Acknowledgments

The study was supported by NIH Grants R01 EY017356 (to Andrew Hollingworth) and R01 MH076226 (to Steven Luck). 

Commercial relationships: none. 

Corresponding author: Andrew Hollingworth. 

Email: [e-mail]. 

Address: Department of Psychology, The University of Iowa, Iowa City, IA, USA.