Bold moves: Inevitable saccadic selection in visual short-term memory

Abstract

Selection for visual short-term memory (vstm) provides a basis for many cognitive functions. Saccadic eye movements sway this selection in favor of stimuli previously seen at locations congruent with their target. In three experiments, we provide converging evidence that this saccadic selection is implemented as a fundamental, inevitable selection process, rather than a top-down strategy. In particular, benefits for congruent over incongruent items were largely constant across set sizes ranging from two to eight items (Experiment 1), showing that saccadic selection imposes priorities on vstm irrespective of memory load and is effective even when only few representations need to be maintained. Moreover, a decrement in performance for incongruent items occurred reliably, whether the congruent location contained a task-relevant item or an irrelevant noise patch (Experiment 2). Finally, saccadic selection was immune to a strong manipulation of the observer's attentional priorities (Experiment 3). Given the prevalence of saccades in natural vision, our results demonstrate a fundamental and ecologically relevant selection mechanism for vstm: Saccades systematically eliminate information seen at non-target locations, while information at the saccade target remains available to recall. This simple heuristic is effective in the absence of informative cues and may incapacitate voluntary selection mechanisms that are incongruent with ongoing movement plans.

Introduction

Visual information processing is fundamentally capacity-limited (Marois & Ivanoff, 2005). One major limitation is the amount of information that can be maintained in visual short-term memory (vstm; Bays & Husain, 2008; Cowan, 2001; Luck & Vogel, 1997; Nakayama, 1990; Pashler, 1988). To provide the groundwork for higher cognition, therefore, the visual system must rapidly select information for vstm and discard other signals from the rich pool of available sensory data. 

Indeed, capacity is large at the first stages of visual memory formation, but it declines within a few hundred milliseconds of the disappearance of the visual source (Averbach & Sperling, 1961; Sperling, 1960). The cueing of covert attention grants visual information selective access to vstm, shifting the balance between multiple items (Awh, Vogel, & Oh, 2006; Becker, Pashler, & Anstis, 2000; Gegenfurtner & Sperling, 1993; Kalogeropoulou, Jagadeesh, Ohl, & Rolfs, 2017; Schmidt, Vogel, Woodman, & Luck, 2002). In natural vision, spatial attention is tightly coupled to saccadic eye movements. Indeed, enforcing fixation during a retention interval deteriorates vstm performance (Williams, Pouget, Boucher, & Woodman, 2013). Recently, we and others demonstrated that saccades have a strong influence on the content of vstm (Hanning, Jonikaitis, Deubel, & Szinte, 2016; Ohl & Rolfs, 2017; 2018): They protect memory of stimuli previously seen at their target location (congruent location) and lead to forgetting of visual information elsewhere (incongruent locations). This saccadic selection is substantive if the preparation of the eye movement starts within a second after stimulus disappearance—the time scale at which saccades occur in natural vision. Based on these results, we proposed that saccades constitute a primary form of selection in active visual processing (Ohl & Rolfs, 2017). Here, to challenge this fundamental nature of saccadic selection in vstm, we addressed three alternative accounts. 

First, it could be argued that saccades merely affect weak memory representations. Here, we mean a memory representation that is either imprecise or more easily forgotten—potentially caused by fewer resources allocated to its maintenance. Experimental manipulation of the number of stimuli to be maintained (i.e., memory set size) results in such memory representations—increasing memory load decreases memory precision and increases the proportion of guessing (Bays, Catalao, & Husain, 2009; Zhang & Luck, 2008). If saccadic selection affected weak memory representations only, then saccades would exert no impact on the content of vstm at small set sizes, which would challenge the tenet of a fundamental selection mechanism. 

Second, saccadic selection in vstm could be an implementation of a top-down strategy. For example, the saccadic influence on vstm could be similar to—or an instance of—retro-cueing, in which the deployment of covert attention to one item in vstm improves memory performance for that item relative to those at other locations. Indeed, the tight coupling between saccades and covert attention (Deubel & Schneider, 1996; Kowler, Anderson, Dosher, & Blaser, 1995; Ohl, Kuper, & Rolfs, 2017; Rolfs & Carrasco, 2012) could bias the competition of items in vstm. This strategy would be particularly helpful to prioritize memory representations and free up resources when a large amount of information needs to be remembered that otherwise would exceed memory capacity. Thus, from the perspective of a top-down strategy, the impact of saccadic selection on vstm performance should vary as a function of memory set size, as has been found by the vast majority of studies on retro-cueing (reviewed by Souza & Oberauer, 2016; see also Discussion). Again, such a result would suggest that saccadic selection is not fundamental. 

Finally, saccadic selection in vstm could result from an implicit or explicit bias by which the observer judges the saccade target to be more relevant than other competing sources of information. In this alternative scenario to a fundamental selection mechanism, saccades toward locations that have never contained relevant information should not cause a decrement in vstm performance at locations incongruent with the saccade target. 

Here, we assessed these alternative accounts systematically and found that saccadic selection for vstm meets none of their predictions. Instead, saccades influence vstm largely independently of memory load, even if it was unlikely or ruled out that information had to be remembered at the target of the eye movement. 

Experiment 1

In a first experiment (Figure 1), participants had to remember the orientation of gratings flashed briefly in arrays that consisted of a variable number of relevant stimuli (set size). After the disappearance of the array, a movement cue prompted participants to generate a saccade that—after an additional delay—was followed by a response cue probing one location in the array. Participants reported from memory whether the stimulus at the probed location was tilted clockwise or counterclockwise relative to vertical. Importantly, the movement cue was uninformative as to the probed location. This orthogonal manipulation of saccade and memory task allowed us to study the dependence of saccadic selection on memory load. 

Figure 1.

 

Experimental procedure of Experiment 1. We presented memory arrays consisting of either 2, 4, 6, or 8 tilted Gabor patches (other locations were filled with noise patches) for 100 ms. The observer's task was to maintain each patch's orientation in memory. A response cue appeared 1200 ms after memory array offset and highlighted the test location. Observers were asked to report the orientation (clockwise vs. counterclockwise) of the Gabor patch that had been presented at that highlighted location. In between memory array offset and presentation of the response cue, we displayed a movement cue that prompted observers to move their gaze to the cued location (within 400 ms). The location of the response cue could be either congruent or incongruent with the saccade target. Importantly, the saccade target was uninformative about which Gabor's orientation would have to be reported. The saccade was equally likely to be directed to any of the stimulus locations that contained a Gabor patch. The figure shows an example of an incongruent trial for the set size 4.

In Experiment 1, we manipulated memory load by presenting set sizes ranging from two to eight orientations. Typically, remembering a small number of stimuli is more accurate and more precise than remember many stimuli—a characteristic finding for a limited-resource system such as vstm (Phillips, 1974). Previously, we have suggested that saccades form a fundamental selection mechanism in memory based on experiments with a constant memory set size of four (Ohl & Rolfs, 2017; 2018). Here, we set out to determine how inescapable the consequences of this mechanism are by testing whether saccades can effectively select among memory representations even when only two items need to be maintained. 

These manipulations of memory set size also served as a test to determine whether saccades affect visual memory via top-down mechanisms akin to retro-cueing. Performance benefits from retro-cueing reliably increase with memory set size (see Souza & Oberauer, 2016 for review; Astle, Summerfield, Griffin, & Nobre, 2012; Gilchrist, Duarte, & Verhaeghen, 2015; Kuo, Stokes, & Nobre, 2012; Nobre, Griffin, & Rao, 2008; Sligte, Scholte, & Lamme, 2008; Souza, Rerko, Lin, & Oberauer, 2014; van Moorselaar, Olivers, Theeuwes, Lamme, & Sligte, 2015b; Vandenbroucke, Sligte, de Vries, Cohen, & Lamme, 2015), suggesting that participants adapt memory resources flexibly to the current task demands (e.g., by removing noncued representations from an overloaded visual memory; but see also Gressmann & Janczyk, 2016 for discussing that retro-cueing effects might be independent of set size). Saccadic selection, in contrast, may be less compromising and affect performance irrespective of the number of relevant objects to be held in memory. Rather than implementing a top-down strategy that participants contrived in the absence of other clues, saccadic selection would thus implement a simple heuristic that information at our movement goals is most relevant. 

Method

Participants

Eight naïve observers (ages 21-37 years; 4 female; 7 right-handed; 7 right-eye dominant) participated in four sessions of Experiment 1 (one training and three test sessions), separated by at least one night between consecutive sessions. We excluded one participant from the analysis as this participant's performance was indistinguishable from chance (95% confidence interval, overlapped with 0.5). 

Observers were paid 7€ per session as compensation and received an additional 7€ after completion of all sessions. We obtained written informed consent from all observers at the beginning of the first session. All observers had normal or corrected-to-normal vision. The study followed the guidelines of the Declaration of Helsinki of 2008 and was approved by the ethics committee of the Psychology Department of the Humboldt-Universität zu Berlin. 

Materials and procedure

The experiment was conducted in a dimly lit room. To reduce head movements, observers positioned their heads on a chin and forehead rest. Using an Eyelink 1000 Desktop Mount eye tracker (SR Research, Ottawa, ON, Canada), we recorded monocular eye positions of the dominant eye at a sampling frequency of 1 kHz. We displayed visual stimuli on a gamma-corrected Sony Trinitron CRT with a refresh rate of 100 Hz and a spatial resolution of 1280 × 800 pixels. The distance between the observer's eyes and the screen center was 57 cm. The experiment was implemented in Matlab (Mathworks, Natick, MA, USA) using the Psychophysics toolbox 3 (Brainard, 1997; Kleiner et al., 2007; Pelli, 1997) and the Eyelink toolbox (Cornelissen, Peters, & Palmer, 2002), running on a Mac mini-computer (Apple, Cupertino, CA, USA). 

Participants were asked to remember the orientations of a number of gratings (Gabor stimuli) and, after a delay, to report one of those orientations as indicated by a response cue (Figure 1). A trial started with a fixation stimulus (0.17 dva diameter white disk surrounded by a black contour with a diameter of 0.68 dva and width of 0.085 dva) presented at the screen center on a gray background (luminance of 77 cd/m²). Eight black circular placeholders (1.95 dva diameter) appeared simultaneously with the fixation symbol indicating the location of the upcoming stimuli for the memory task. Placeholders were arranged equidistantly on an imaginary circle at an eccentricity of 6 dva from the screen center. After 500 ms, a memory set appeared for 100 ms, consisting of either 2, 4, 6, or 8 oriented Gabor patches (±45° from vertical, 50% contrast, randomly assigned spatial frequency of either 1.5 or 2.25 cycles per degree, random phase, and a 0.65° standard deviation [SD] Gaussian envelope). Unoriented noise patches (pixel noise, band-pass filtered from one-half to twice the spatial frequency of the Gabors, 50% contrast, 0.65° SD Gaussian envelope) appeared simultaneously at all placeholders that did not contain a Gabor. After a delay period of 400 ms following the disappearance of the memory array, we presented a movement cue (0.26 dva long black line segment pointing out from the outline of the fixation spot) that identified one of the eight placeholders. Participants had a maximum of 400 ms to move the eyes to the identified target. Note that in this experiment (in contrast to Experiment 2), saccades always targeted a location at which a Gabor patch had been presented. Following another delay period of 800 ms, we presented a response cue (thickening of one placeholder's outline from 0.05 dva to 0.085 dva) prompting observers to report (by pressing one of two buttons) the remembered orientation of the Gabor patch that had previously occupied this location (clockwise or counterclockwise relative to vertical). 

Importantly, the movement cue was uninformative about the location of the subsequent response cue (i.e., the location of the saccade target did not inform about the location of the memory test). However, because saccade targets always coincided with a stimulus location in Experiment 1 the probability that the location of the saccade target and the response cue coincided (congruent trials) or did not coincide (incongruent trials) depended on the memory set size (50% congruent and incongruent trials for set size 2; 25% congruent trials and 75% incongruent trials for set size 4; 16.67% congruent and 83.3% incongruent trials for set size 6; 12.5% congruent trials and 87.5% incongruent trials for set size 8). 

Each experimental session consisted of 20 short blocks composed of 24 trials each. In each session, we tested the different memory set sizes in separate and randomly interleaved blocks. To obtain the same total number of congruent trials for each set-size condition, we ran two blocks with a set size of two, four blocks with a set size of four, six blocks with a set size of six, and eight blocks with a set size of eight. Each observer completed a total of 1,440 trials in the 3 test sessions. We chose the number of participants and the number of trials per experimental condition based on a previous study (Ohl & Rolfs, 2017) that yielded highly reliable effects in a similar combination of a saccade and a memory task. 

We ran standard nine-point calibration and validation routines that aligned eye and screen coordinates at the beginning of the experiment, after breaks, and whenever necessary. Before trial onset, we displayed a small fixation point and a trial only started when the observers’ eye positions were inside a circular region (1.5 dva diameter) centered on the fixation point for a minimum of 200 ms. Trials were aborted when we detected blinks or when the eyes crossed an invisible boundary with a radius of 1.5 dva from the central fixation point (in the interval between trial onset and presentation of the response cue). Additionally, trials in which saccadic reactions times exceeded a limit of 400 ms, were aborted. All aborted trials were repeated at the end of the block in randomized order. 

Data analysis

For inferential statistics, we used an arcsine transformation of the dependent variable performance (i.e., proportion correct) and conducted repeated-measures analyses of variance (rmANOVA). Error bars are ±1 within-subject standard error of the mean (SEM; Baguley, 2011; Morey, 2008). We complemented the rmANOVA with computation of Bayes factors (BF) using the R package BayesFactor (Morey & Rouder, 2015) with default g-priors (scale = 0.5). BFs quantify evidence for a hypothesis given the data (fixed effect of model terms plus random effect of participants) relative to the evidence for a baseline model including only the random factor of participants. As an example, a BF of 10 indicates that the data are 10 times more likely for a given model as compared with a model that includes only the random factor of participants. Typically, BFs smaller than 3 are regarded as weak evidence, BFs between 3 and 10 as substantial evidence, and BFs between 10 and 100 as strong evidence (Kass & Raftery, 1995). Here, we will report log BFs, such that BFs smaller than 1.1 are regarded as weak evidence, BFs between 1.1 and 2.3 are regarded as substantial evidence, and BFs between 2.3 and 4.6 are regarded as strong evidence. Log-transformed BFs larger than 4.6 indicate decisive evidence. 

For the detection of saccades, we first transformed raw eye positions into two-dimensional velocity space offline. We classified successive eye positions as saccades if they exceeded the median velocity by 5 SDs for at least 8 ms (Engbert & Mergenthaler, 2006). Saccadic events that were separated by less than 20 ms were merged into a single saccade. The first saccade that landed within a radius of 3.6° from the center of the saccade target (i.e., 60% of the target's eccentricity) was classified as the response saccade. We excluded trials that included saccades with an amplitude larger than 1 dva before execution of the response saccade. Moreover, we rejected trials from further analyses that included blinks or missing samples in the recordings of eye position. A total of 8,896 trials (88%) entered the final data analysis of Experiment 1. The raw data are available through the Open Science Framework at https://osf.io/jmf95/. 

Results

Performance decreased strongly with increasing memory set size (Figure 2a; F(3, 18) = 63.1, p < 0.001)—a finding characteristic of the resource-limited vstm rather than iconic memory (Phillips, 1974). Moreover, congruency of the reported and the saccade location influenced visual memory performance across all set sizes (F(1, 6) = 15.3, p = 0.008), with better performance for the stimulus location that coincided with the subsequent saccade target. Importantly, the saccadic influence on visual memory did not depend on memory set size as indicated by the absence of an interaction between congruency and memory set size (F(3, 18) = 0.6, p > 0.250). That is, the memory advantage for stimuli congruent with the saccade target was independent of memory set size. The calculation of log-transformed BFs supported these results. A statistical model including the two main effects yielded most evidence (BF = 34.76), followed by a model including the two main effects and their interaction (BF = 33.33). That is, a model including only the two main effects is 4.2 times more likely given the data than a model including both main effects and their interaction (derived from: e^34.76 / e^33.33 = 4.2). 

Figure 2.

 

Results of Experiment 1. (a) Mean performance ±1 within-subject SEM in congruent (blue) and incongruent (orange) trials is displayed as a function of set size (2, 4, 6, or 8 items in the memory array). (b) Spatial specificity of the saccadic influence on visual memory performance. Mean performance ± 1 SEM is shown as a function of angular distance between the memory probe location and the saccade target in the array, plotted separately for each memory set size. SEM, standard error of the mean.

Memory performance also varied with angular distance between saccade target and location of the memory probe (Figure 2b). Performance was best for stimuli presented at the location congruent with the saccade target and decreased with increasing angular distance. However, memory performance for stimuli located opposite to the saccade target was elevated relative to intermediate locations (see also Germeys, De Graef, Van Eccelpoel, & Verfaillie, 2010; Ohl & Rolfs, 2017). An rmANOVA revealed that both angular distance (F(4, 24) = 5.1, p = 0.004) and set size (F(3, 18) = 43.5, p < 0.001) significantly influenced memory performance. The interaction between angular distance and memory set size did not reach significance (F(12, 72) = 1.0, p > 0.250). Accordingly, the evidence for a model including only the two main effects (BF = 80.59) was 16.8 times greater than evidence for a model including main effect and their interaction (BF = 77.77). 

The advantage for congruent trials cannot be accounted for by a speed-accuracy tradeoff. Manual response times (RTs) were faster in congruent (905 ± 25 ms) than incongruent trials (1108 ± 25 ms). An rmANOVA revealed a significant influence of congruency (F(1, 6) = 22.2, p = 0.003) and memory set size (F(3, 18) = 9.2, p < 0.001), but no interaction between the two main effects (F(3, 18) = 0.3, p > 0.250). 

Finally, saccade parameters varied little between experimental conditions. Saccade latencies increased slightly (but not significantly) with memory set size (F(3, 18) = 2.5, p = 0.091) by 10 ms from a set size of 2 stimuli (179 ± 2.8 ms) to a set size of 8 stimuli (189 ± 2.5 ms). Saccade amplitude varied significantly with set size (F(3, 18) = 13.3, p < 0.001). Amplitudes decreased with increasing set sizes from 5.72 ± 0.04 dva (set size of 2) to 5.41 ± 0.03 dva (set size of 8). 

Discussion

Experiment 1 confirms the recent finding that saccades influence the selection of items for storage in visual memory (Ohl & Rolfs, 2017; 2018). A substantive set size effect for both congruent and incongruent items reveals that this selection occurs in vstm and independently of the number of items that compete for memory resources. Indeed, none of our analyses of performance (including accuracy and reaction time) showed an interaction of set size and congruency. Even if observers needed to remember no more than two stimuli, saccades effectively prioritized congruent memory representations over incongruent ones, just like they did if eight stimuli had to be remembered. 

Note, however, that the validity of the movement cue was not constant for all memory set sizes in Experiment 1. For a set size of 2 items, 50% of the trials were congruent whereas for a set size of 8 items, the saccade was more likely to go to any location in the display (12.5% congruent trials). It is possible, therefore, that the variation of validity inherent to the design contributed to our findings and, perhaps, obscured an interaction between congruency and memory set size. We address this possible concern and extend the scope of our findings in Experiment 2. 

Experiment 2

In Experiment 2, all locations in the stimulus array were equally likely to be targeted by the subsequent saccade, and each location was probed in only 1/8 of the trials. Thus, in contrast to Experiment 1, the validity of the movement cue remained constant across all memory set sizes. In our paradigm, all locations that did not contain Gabor stimuli instead contained noise patches. This manipulation enables an additional interesting analysis: We can now test whether saccadic selection in vstm depends on the task relevance of the stimulus presented at the location congruent with the saccade target. Specifically, we can sort incongruent trials into trials in which the saccade aimed at a location that was previously occupied by an oriented Gabor versus a location that was previously occupied by a noise patch. This allowed us to investigate whether saccadic selection follows a rather simple heuristic that disregards information at nontarget locations indiscriminately (i.e., irrespective of the memory content at the saccade target location), or a strategy that sacrifices information at incongruent locations when the saccade target contains relevant information. 

Previous studies have established that the encoding of the saccade goal results in costs for representations already maintained in vstm (Schut, Van der Stoep, Postma, & Van der Stigchel, 2017; Tas, Luck, & Hollingworth, 2016; Van der Stigchel & Hollingworth, 2018). Saccades to an empty space generated during the retention interval did not interfere with memory maintenance, whereas saccades to a new, memory task-irrelevant object decreased memory performance (Tas et al., 2016). Based on these findings, it is reasonable to predict similar memory performance at incongruent locations irrespective of task-relevant versus task-irrelevant information at the saccade goal as long as a visual object is displayed at the goal of the saccade. It is unclear whether a continuously present placeholder at the saccade goal is also automatically encoded into vstm and therefore interferes with vstm as much as a new secondary object. It is of particular interest in Experiment 2 to test whether task relevance of a memory representation at the saccade goal can modulate saccadic selection in vstm. 

Methods

Experiment 2 was identical to Experiment 1 except for the following changes: First, to allow for saccades toward noise patches in all set-size conditions, we tested memory set sizes of two, four, or seven (instead of eight) stimuli. Second, saccades were equally likely to be directed to any location of the eight placeholders. Thus, the location indicated by the response cue coincided with the saccade target (congruent condition) in only 1/8 of the trials across all memory set sizes. Moreover, saccades could also target locations that had previously been occupied by a noise patch and we distinguished incongruent trials of two types: those in which the saccade targeted a location previously occupied by a Gabor and those in which the saccade targeted a location previously occupied by a noise patch with no particular orientation. This enabled us to study the role of previously presented content at the saccade target in the selection of information for vstm. 

Participants

Nine naïve observers (ages 19-31 years; 6 female; 9 right-handed, 6 right-eye dominant) participated in 5 sessions of Experiment 2 (1 training and 4 test sessions). Consecutive sessions were separated by at least one night. None of the nine subjects had participated in Experiment 1. We excluded one participant from the analysis as a consequence of very low performance (indistinguishable from chance level). 

Materials and procedure

In Experiment 2, we displayed visual stimuli on a gamma-corrected VIEWPixx /3D (VPixx Technologies Inc., Saint Bruno, QC, Canada) in scanning backlight mode (luminance in a range of 0 to 100 cd/m²; pixel response time ∼1 ms) at a spatial resolution of 1920 × 1080 pixels and a refresh rate of 120 Hz. The experiment was running on a DELL Precision T3600 (Debian GNU Linux 8) using MATLAB (Mathworks, Natick, MA, USA), the Psychophysics toolbox 3 (Brainard, 1997; Kleiner et al., 2007; Pelli, 1997), and the Eyelink toolbox (Cornelissen et al., 2002). 

Data analysis

All analysis procedures were identical to those used in Experiment 1. A total of 15,598 trials (97%) entered the final data analysis of Experiment 2. The raw data are available through the Open Science Framework at https://osf.io/jmf95/. 

Results

The overall pattern of results in Experiment 2 is strikingly similar to that in Experiment 1. First, we observed a general decrease in memory performance with increasing memory set size, irrespective of congruency condition (Figure 3a). Second, performance in the congruent and incongruent conditions forms largely parallel lines across set sizes. That is, as in Experiment 1, targets whose location coincided with the subsequent saccade target location were more often remembered correctly than stimuli at incongruent locations, and this advantage for congruent locations was not affected by memory set size. Finally, we observed the congruency effect both after saccades to a noise patch and saccades to oriented Gabors—performance in the two incongruent conditions largely overlaps. A 3 (congruency) x 3 (memory set size) rmANOVA corroborated these findings. We observed a significant main effect of memory set size (F(2, 14) = 100.2, p < 0.001) and congruency (F(2, 14) = 8.1, p = 0.005). Importantly, the influence of congruency did not vary as a function of memory set size, as indicated by the absence of an interaction (F(4, 28) = 0.7, p > 0.250). Indeed, a model including only the two main effects (BF = 51.53) was 6.2 times as likely to account for the data than a model including both main effects and their interaction (BF = 49.70). In an additional rmANOVA, we compared the influence of memory set size and congruency only for the two incongruent conditions (Gabor vs. noise patch). Again, we observed a significant main effect of memory set size (F(2, 14) = 50.48, p < 0.001). However, the two incongruent conditions were not statistically different (F(1, 7) = 1.3, p > 0.250) and they did not interact with memory set size (F(2, 14) = 1.5, p = 0.249). Accordingly, a model including only the main effect of set size (BF = 30.09) was 2.9 times as likely as a model including both main effects (BF = 29.03). Thus, saccades influence the selection for vstm irrespective of memory set size and irrespective of whether the memory content at the saccade target is a competing stimulus or not. 

Figure 3.

 

Results of Experiment 2. (a) Mean performance ± 1 SEM in congruent (blue), incongruent Gabor patch (orange), and incongruent Noise patch trials (gray) is displayed as a function of set size (2, 4, or 7 items in the memory array). (b) Spatial specificity of the saccadic influence on visual memory performance. Mean performance ± 1 SEM is shown as a function of angular distance between the memory probe location and the saccade target in the array, plotted separately for each memory set size. SEM, standard error of the mean.

As in Experiment 1, the saccadic influence was spatially specific: Memory performance dropped sharply just one stimulus away from the saccade target (Figure 3b). Both angular distance (F(4, 28) = 8.4, p < 0.001) and memory set size (F(2, 14) = 80.9, p < 0.001) influenced visual memory performance; their interaction did not reach significance (F(8, 56) = 0.7, p > 0.250). 

A speed-accuracy tradeoff cannot account for the observed findings. Manual RTs were faster for congruent (798 ± 40 ms) than incongruent trials (noise patch: 909 ± 12 ms; Gabor: 1078 ± 47 ms). An rmANOVA revealed significant main effects for congruency (F(2, 14) = 8.7, p = 0.003) and memory set size (increasing reaction times with increasing set size; F(2, 14) = 6.5, p = 0.010), but no significant interaction (F(4, 28) = 1.4, p > 0.250). 

Finally, saccade latencies varied significantly across congruency conditions (F(2, 14) = 11.8, p < 0.001): Saccade latencies to the location of a noise patch (209 ± 1.8 ms) were somewhat higher than to the location of an oriented Gabor (congruent Gabor 198 ± 1.2 ms, incongruent Gabor 199 ± 0.8 ms). Furthermore, saccade latencies varied barely (and not significantly) with memory set size (F(2, 14) = 2.9, p = 0.087; set size 2: 205 ± 0.8 ms, set size 7: 203 ± 0.9 ms). There was no sign of an interaction between congruency and set size (F(4, 28) = 1.7, p = 0.185). We observed significant main effects of congruency (F(2, 14) = 18.4, p < 0.001) and memory set size (F(2, 14) = 7.2, p = 0.007) on saccade amplitude. Moreover, amplitudes depended on the interaction between congruency and set size (F(4, 28) = 2.9, p < 0.038). Specifically, saccades to locations that had contained noise patches were shorter on average (5.26 ± 0.06 dva) than saccades to locations that had contained oriented Gabors (congruent Gabors 5.73 ± 0.03 dva; incongruent Gabors 5.58 ± 0.04 dva) and the effect of set size (set size 2: 5.45 ± 0.03 dva, set size 4: 5.44 ± 0.02 dva, set size 7: 5.53 ± 0.04 dva) was more pronounced when saccades were made to locations of oriented Gabors. 

Discussion

Experiment 2 corroborated the results of Experiment 1: Saccades prioritized congruent representations in vstm irrespective of the number of items competing for memory resources, including even the smallest set sizes. Indeed, it appears irrelevant if the stimulus previously presented at the saccade target contained a competing stimulus (with a defined orientation) or merely a placeholder (unoriented noise). Together, these findings substantiate the proposal that saccadic selection is a fundamental selection mechanism for vstm that indiscriminately eliminates information about stimuli at non-target locations. It is worth noting that both experiments presented thus far yielded strikingly similar results despite the fact that set size and validity of the movement cue covaried in Experiment 1, but not in Experiment 2. This observation suggests that saccadic selection in visual memory is immune to manipulations of movement cue validity. In line with this idea, we recently observed a benefit for stimuli congruent with the saccade target even when the saccade target location was least likely to be probed in the memory test (Ohl & Rolfs, 2017). In Experiment 3, we set out to explicitly test the influence of movement cue validity on visual memory performance in this paradigm. 

Experiment 3

In Experiment 3, we manipulated the validity of the movement cue, that is, how much information about the memory test location is provided by highlighting the saccade target location. This experimental manipulation allows us to test our claim that saccadic selection in vstm is obligatory (Ohl & Rolfs, 2017) and overrides the top-down strategy of using the available information regarding the memory test location as provided by the movement cue. 

Based on the striking similarity of findings in the previous two experiments—despite their difference in movement cue validity—we reason that saccadic selection in vstm should occur irrespective of movement cue validity. Our recent finding that memory performance was best for representations congruent with the saccade goal even when this location was the least likely location to be probed for memory (Ohl & Rolfs, 2017) further corroborates this idea. In Experiment 3, we will contrast the influence of valid, neutral and invalid movement cues on memory performance in a single experiment. Dual task protocols that assess saccadic influences on perceptual discrimination demonstrated an obligatory shift of attention to the saccade target. That is, even when participants were informed in advance about the location of the discrimination target, their discrimination performance was best at the saccade target location (Deubel & Schneider, 1996). Moreover, the object at the saccade target is automatically encoded into vstm (Schut et al., 2017; Shao et al., 2010) and exerts a detrimental influence on other (spatially incongruent) memory representations (Schut et al., 2017; Tas et al., 2016; Van der Stigchel & Hollingworth, 2018). The aim of Experiment 3 is to find out whether endogenous attention can further modulate the seemingly automatic saccadic selection process in visual memory. 

Methods

We kept the set size of four oriented Gabors constant across the experiment. As in Experiment 1, the movement cue pointed always to a location that contained an oriented stimulus in the memory array. The saccade target was either most likely to be probed for the memory test (i.e., movement cue correctly highlighted the response cue location in 50% of the trials), as likely as any other location (i.e., movement cue at response cue location in 25% of the trials), or least likely to be probed for the memory test (i.e., movement cue pointing to the response cue location in only 12.5% of the trials). Thus, in a condition in which the saccade target was most likely to be probed, the movement cue was 4 times more likely to highlight the future response cue location than in the condition in which the saccade target was the least to be probed. Participants were explicitly informed about these likelihoods at the beginning of the experiment, and a message before the beginning of a new block highlighted whether the saccade target location was more likely, equally likely or less likely to be probed than another location in that block. The message disappeared and the block started once the participant pressed a key on the keyboard. 

The temporal and spatial parameters of Experiment 3 were identical to Experiments 1 and 2. Each experimental session consisted of 12 blocks composed of 40 trials each, with the different conditions of movement cue validity being tested in separate and randomly interleaved blocks. Each observer completed a total of 1,920 trials across the 4 test sessions. 

Participants

Ten naïve observers (ages 19-39 years; 6 female; 8 right-handed, 5 right-eye dominant) participated in 5 sessions of Experiment 3 (1 training and 4 test sessions). We excluded two participants from the analysis as a consequence of very low performance (indistinguishable from chance level). 

Materials and procedure

All materials and procedures were identical to Experiment 2. 

Data analysis

All analysis procedures were identical to those used in Experiments 1 and 2. A total of 14,902 trials (97%) entered the final data analysis of Experiment 3. The raw data are available through the Open Science Framework at https://osf.io/jmf95/. 

Results

The results in Experiment 3 provide further evidence for obligatory saccadic selection in vstm. First, we observed better memory performance for stimuli presented at a location congruent with the saccade target location as compared to incongruent locations (Figure 4a). Second, the difference in performance between the congruent and incongruent conditions seems barely affected by movement cue validity. That is, saccadic selection in vstm occurs irrespective of the information provided about the future memory test location. A 2 (congruency) x 3 (movement cue validity) rmANOVA corroborated these findings. We observed a significant main effect of congruency (F(1, 7) = 17.4, p = 0.004), whereas movement cue validity (F(2, 14) = 0.2, p > 0.250) and the interaction between congruency and movement cue validity were not significant (F(2, 14) = 0.2, p > 0.250). A model including just the main effect of congruency (BF = 11.38) was 5.9 times more likely to account for the data than a model including both main effects (BF = 9.60) and 22.4 times as likely as a model including both main effects and their interaction (BF = 8.27). Thus, saccadic selection in vstm occurs irrespective of the information provided by the movement cue. 

Figure 4.

 

Results of Experiment 3. (a) Mean performance ± 1 within-subject SEM in congruent (blue) and incongruent (orange) trials is displayed as a function of the likelihood that the saccade target was probed as opposed to a specific location incongruent with the saccade target location (3:1, 1:1, or 1:2). (b) Spatial specificity of the saccadic influence on visual memory performance. Mean performance ± 1 SEM is shown as a function of angular distance between the memory probe location and the saccade target in the array, plotted separately for each validity condition. SEM, standard error of the mean.

As before, we observed a spatially specific influence of saccades on visual memory. In each movement cue condition, memory performance was best at the saccade target location and remained at a lower level for incongruent locations (Figure 4b). Correspondingly, the angular distance between saccade target location and memory test location significantly influenced memory performance (F(4, 28) = 7.1, p < 0.001), whereas neither movement cue validity (F(2, 14) = 0.6, p > 0.250) nor the interaction affected memory performance (F(8, 56) = 0.4, p > 0.250). 

We did not observe a speed-accuracy tradeoff. Manual RTs in congruent trials (566 ± 26 ms) were faster than for incongruent trials (852 ± 26 ms). The rmANOVA showed a significant main effect for congruency (F(1, 7) = 29.2, p = 0.001), no influence of movement cue validity (F(2, 14) = 0.1, p > 0.250), and a trend for their interaction (F(2, 14) = 3.3, p = 0.068). 

Movement cue validity (F(2, 14) = 0.3, p > 0.250) had no influence on saccade latencies. Similarly, we did not observe a significant main effect of movement cue validity (F(2, 14) = 0.2, p > 0.250) on saccade amplitude (range of saccade amplitudes in the different conditions from 5.67 dva to 5.81 dva). 

Discussion

Experiment 3 further supported our surprising observation form the first two experiments: The degree of information regarding the memory test location—as provided by the movement cue validity—had no impact on visual memory performance in active observers, despite the fact that observers were explicitly informed about these probabilities. Memory performance was best for stimuli presented at a location congruent with the saccade target location irrespective of whether memory was unlikely or very likely to be probed at the saccade goal. Together, these findings further provide converging evidence for our recent claim that saccadic selection in vstm is obligatory (Ohl & Rolfs, 2017). Experiment 3 created a situation in which the saccade target location is either more or less likely to be probed. We would like to point out, however, that this does not preclude the possibility that observers can deploy attention—in response to a valid retro-cue highlighting the most likely test location—to an additional object at a location other than the saccade target location. 

General discussion

We use saccadic eye movements to continuously sample information from our visual environment. Here, we report new evidence that these eye movements constitute a fundamental selection mechanism in visual memory. Whereas memory performance decreased consistently and substantially with memory load—a hallmark of vstm as opposed to iconic memory (Phillips, 1974)—the effect of congruency between saccade target and reported location remained constant across all set sizes (from two to eight items; Experiments 1 and 2). That is, irrespective of memory load, participants remembered items that were previously shown at the saccade target (congruent condition) more often than those at other locations (incongruent conditions). Thus, even when only as little as two orientations had to be remembered, saccades effectively imposed priorities on the memory representation—providing strong evidence for the inevitability of this selection mechanism. This saccadic-selection effect was independent of the presence of relevant information at the saccade target location (Experiments 2). Moreover, saccadic selection was immune to a strong manipulation of cue validity, in which the saccade target was either three times as likely (valid condition) or half as likely (invalid condition) to be probed for memory performance (Experiments 3). Saccadic selection for vstm thus appears to follow a simple principle: to protect information at locations congruent with the saccade target while eliminating information elsewhere. 

What do we learn from the finding that saccadic selection was independent of the number of stimuli in a scene that compete for resources in visual memory? Even in sparse visual environments—containing as few as two items—saccades biased vstm in favor of visual information at the target at the cost of information at other locations. Comparing set size effects on memory performance, Schut et al. (2017) observed lower precision of visual memory reports in a saccade condition than during fixation. In their study, much like in ours, set size did not affect this difference in memory precision. The authors suggested that the automatic encoding of the saccade target into vstm drives the decline of visual memory at saccade-incongruent locations. The present findings are in line with this idea. Whereas we did not probe memory during prolonged fixation, we did find that the cost of encoding the saccade target into vstm was spatially unspecific, affecting stimuli at all locations incongruent with the saccade target (see also Hanning et al., 2016; Ohl & Rolfs, 2017; 2018). This finding generalizes also to hand movements (Hanning & Deubel, 2018; Heuer, Crawford, & Schubö, 2017). 

An important question is whether memory representations that are congruent with the location of the saccade goal are enhanced or if, instead, representations at incongruent locations are eliminated. Whereas saccadic eye movements seem to enhance memory performance during free viewing (Williams et al., 2013), saccades in response to a cue entail overall costs for memory performance when compared with fixation (Hanning et al., 2016; Ohl & Rolfs, 2017; Schut et al., 2017). These differences are likely due to the dual-task nature of the protocol, which increases task load in the saccade conditions. Future research, aimed at a better understanding of the processing costs associated with the dual task requirements (involving motor and memory components) will help us assess the differences in memory processes recruited during fixation compared with those recruited in active observers. 

Given the current state of the literature, it is most likely that movements in general interfere with memory maintenance (Lawrence, Myerson, & Abrams, 2004; Lawrence, Myerson, Oonk, & Abrams, 2001; Pearson & Sahraie, 2003; Postle, Idzikowski, Sala, Logie, & Baddeley, 2006). Although costs for memory content can arise from visual transients during prolonged fixation (van Moorselaar, Gunseli, Theeuwes, & Olivers, 2015a), postsaccadic visual input is not an important source of interference (Tas et al., 2016). The major source for saccade-related memory costs results from selection and encoding processes before saccade onset (Hanning et al., 2016; Tas et al., 2016). At the same time, memory representations congruent with the location of the saccade goal appear to be protected from interference. Preserving a stable representation of stimuli presented at the saccade goal is necessary to enable perceptual continuity across saccades (Rolfs, 2015; Van der Stigchel & Hollingworth, 2018) and allow for fast correction of saccadic landing errors (Hollingworth, Richard, & Luck, 2008) even on the scale of microsaccades (Ohl & Kliegl, 2016; Ohl, Brandt, & Kliegl, 2011). 

Our study indicates that saccades form a rigid selection mechanism in visual memory. This mechanism is markedly different from the flexible nature of retro-cueing of covert attention (Griffin & Nobre, 2003). In contrast to the movement cue in our first two experiments, retro-cues inform observers about the probability that an item will be relevant for the later memory test. The majority of studies showed that advantages from retro-cueing increase with set size (see Souza & Oberauer, 2016 for review; Astle et al., 2012; Gilchrist et al., 2015; Kuo et al., 2012; Nobre et al., 2008; Sligte et al., 2008; Souza et al., 2014; van Moorselaar, Olivers, Theeuwes, Lamme, & Sligte, 2015b; Vandenbroucke et al., 2015; no interaction was found by Gressmann & Janczyk, 2016; Makovski, Sussman, & Jiang, 2008; Matsukura, Luck, & Vecera, 2007; Trapp & Lepsien, 2012). The dependence of retro-cueing on set size suggests that noncued items are removed from working memory to free up limited resources when memory load is high. Saccadic selection cannot be explained by this strategic deployment of covert attention in memory. Changes in attentional priority imposed by saccades generally share features with exogenous shifts of attention (Carrasco, 2011; Ohl & Rolfs, 2017; Rolfs & Carrasco, 2012)—they arise transiently in response to saccades, they are short-lived (Ohl & Rolfs, 2017; 2018), and their impact is largely independent of set size. 

Interestingly, saccadic selection in vstm seems to be in conflict with voluntary, top-down selection of memory content. Even when providing a valid movement cue—essentially a valid retro-cue informing which memory items are most likely to be tested—participants did not make use of this information. Memory performance was similar irrespective of whether the saccade target location was most likely, or least likely to be probed for recall, and saccadic selection was present irrespective of the cue validity. Thus, we provide further support for the conclusion that saccadic selection in vstm is obligatory (see also Ohl & Rolfs, 2017). However, we cannot exclude the interpretation that participants did not even consider the validity information (or that they simply did not care). We consider this explanation unlikely, however, because our manipulation of cue validity was rather strong (validity changed by a factor of six between blocks of trials) and observers were continuously informed about the expected contingencies before each block of trials. Moreover, the striking similarity of the findings in the first two experiments, which occurred despite changes in movement cue validity, point to the same conclusion. Namely, the validity of the movement cue in our paradigm does not guide selection in vstm. This renders it unlikely that the observed influence of saccades is the result of an implicit or explicit bias (e.g., intentionally prioritizing a location based on information provided by the movement cue). More likely these effects emerge from a fundamental selection mechanism in vstm. 

Thus, our study suggests an obligatory coupling of saccades and attention also in the selection of memory representations, but we cannot finally resolve the role of endogenous attention in our paradigm. It remains an open question, for instance, how saccadic selection would affect observers’ ability to retroactively deploy attention to one distinct location incongruent with the saccade target (using a valid retro cue), if that single location (rather than all incongruent locations) is most likely to be tested for memory performance. For instance, both reward and pointing movements influence visual memory maintenance when tested in separate experiments, but when tested in the same experiment reward affected memory performance only at the action-relevant location, whereas actions influenced memory performance irrespective of the reward manipulation (Heuer & Schubö, 2018). In their study, the reward information was presented 200 ms before onset of the movement cue, thus providing also more time for voluntary selection than in the present study. To conclude, we observed memory benefits at locations congruent with the saccade target location irrespective of whether different movement cue validities were presented in separate blocks (see Experiment 3) or an anti-cue was used throughout the experiment (Ohl & Rolfs, 2017). These results suggest that saccadic selection hampers or even incapacitates voluntary selection of content in vstm in the presence of saccadic eye movements. 

Saccadic selection also occurred irrespective of whether the saccade-target-congruent location contained a relevant item or an irrelevant noise patch, a finding that appears incompatible with a deliberate strategy to consider vstm content congruent with the saccade target as more relevant than information from other sources. Quite the contrary: Saccades exert a detrimental influence on information from non-target locations in an uncompromising fashion. Previous studies suggest, however, that saccadic influences on visual memory are conditional on the presence of a visual object at the saccade target location. Memory costs occur only for saccades to a visual object, but not when saccades target an empty location (Tas et al., 2016). Thus, the automatic encoding of the saccade goal into vstm and its resulting interference with current memory representations (Schut et al., 2017; Shao et al., 2010; Tas et al., 2016; Van der Stigchel & Hollingworth, 2018) are likely to underlie saccadic selection in vstm. 

Based on our findings, we suggest that saccades protect stimuli at congruent locations from interference, which could result either from new (intra- or postsaccadic) visual input (Makovski & Jiang, 2007; Matsukura et al., 2007; Sligte et al., 2008; van Moorselaar, Gunseli, Theeuwes, & Olivers, 2015a) or the automatic encoding of stimuli at the saccade goal into vstm (Schut et al., 2017; Tas et al., 2016; Van der Stigchel & Hollingworth, 2018). Stabilizing selected visual memory traces to decrease the detrimental effect of saccade related interferences on visual memory then helps maintain memory representations at movement-congruent locations across saccades. 

Conclusion

In the present study, we observed that saccadic selection occurred in vstm, largely independently of memory load, irrespective of the task relevance of the stimulus seen at the saccade target, and irrespective of how likely it is that memory is probed at the saccade goal. Saccades therefore selectively protect memory at locations congruent with their target, and eliminate information at nontarget locations without compromise. Using this powerful and ecologically relevant selection mechanism, the visual system is equipped to make rapid choices in the absence of informative cues, deciding what to remember and what to forget. 

Acknowledgments

This research was supported by a DFG research grant to S.O. and M.R. (OH 274/2-1 and RO 3579/6-1) as well as the DFG's Heisenberg program (RO 3579/8-1). 

Commercial relationships: none. 

Corresponding author: Sven Ohl. 

Email: [e-mail]. 

Address: Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany.