The cost of making an eye movement: A direct link between visual working memory and saccade execution

Abstract

To facilitate visual continuity across eye movements, the visual system must presaccadically acquire information about the future foveal image. Previous studies have indicated that visual working memory (VWM) affects saccade execution. However, the reverse relation, the effect of saccade execution on VWM load is less clear. To investigate the causal link between saccade execution and VWM, we combined a VWM task and a saccade task. Participants were instructed to remember one, two, or three shapes and performed either a No Saccade-, a Single Saccade- or a Dual (corrective) Saccade-task. The results indicate that items stored in VWM are reported less accurately if a single saccade—or a dual saccade—task is performed next to retaining items in VWM. Importantly, the loss of response accuracy for items retained in VWM by performing a saccade was similar to committing an extra item to VWM. In a second experiment, we observed no cost of executing a saccade for auditory working memory performance, indicating that executing a saccade exclusively taxes the VWM system. Our results suggest that the visual system presaccadically stores the upcoming retinal image, which has a similar VWM load as committing one extra item to memory and interferes with stored VWM content. After the saccade, the visual system can retrieve this item from VWM to evaluate saccade accuracy. Our results support the idea that VWM is a system which is directly linked to saccade execution and promotes visual continuity across saccades.

Introduction

To process visual information, the visual system redirects the high-acuity fovea to objects of interest in the visual field by rapidly moving the eyes, i.e., saccades (Purves, Augustine, & Fitzpatrick, 2001). Although saccades dramatically alter the visual input, we subjectively experience a stable visual world, that is, a world in which items of interest are available for processing both before and after a saccade. In what manner the visual system stitches together a continuous experience from visual input interrupted by saccades is still debated (Burr, Morrone, & Ross, 1994; Deubel, Schneider, & Bridgeman, 1996; Matin, 1974). 

One potential candidate to bridge the gaps in visual processing during saccades is an attentional selection mechanism that is tightly coupled to saccade preparation, i.e., presaccadic shifts of attention (Deubel & Schneider, 1996; Irwin & Gordon, 1998; Schneider & Deubel, 1995). The coupling between attentional selection and saccades is shown by an increase in the discriminability of visual input near a saccade target (relative to stimuli further away from the saccade target), even when the stimuli are no longer available after the execution of the saccade (Deubel, 2008; Irwin & Andrews, 1996). Information at (and surrounding) the attended location is acquired by attentional processes and stored in a memory buffer from which information can subsequently be retrieved after the saccade (Deubel & Schneider, 1996; Irwin & Gordon, 1998). A presaccadic shift of attention may allow the visual system to determine whether the stored visual input matches the postsaccadic visual input. By determining congruence between pre- and postsaccadic visual input, motor errors and/or changes in the external world can be detected and compensated for, e.g., by corrective eye movements (Johnson, Hollingworth, & Luck, 2008; Miall & Wolpert, 1996; Richard, Luck, & Hollingworth, 2008). Previous research indicates that the presaccadic shift of attention, together with the corollary discharge (an efference copy of the motor command executed by the visual system), underlies visual continuity across saccades (Bridgeman, 1995; Cavanaugh, Berman, Joiner, & Wurtz, 2016; Melcher, 2011; Tian, Ying, & Zee, 2013). 

The storage of presaccadic visual input, or transsaccadic memory, is often thought to be dependent on visual working memory (VWM; Deubel & Schneider, 1996; Irwin, 1991; Irwin & Andrews, 1996; Luck & Hollingworth, 2008). VWM is a buffer, with a limited amount of storage capacity, which we can consciously access and use to update visual information (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008; Ma, Husain, & Bays, 2014). As storage capacity in VWM is limited: When more information is stored in VWM, individual items can be represented with less detail, resulting in a decrease in the precision of responses (Bays et al., 2009). Storing presaccadic visual input is important, as it allows for comparison with postsaccadic visual input. For example, after the saccade has been completed, the postsaccadic visual input can be integrated with the visual input stored in VWM to estimate saccade accuracy and increase the precision of information that is processed by the visual system (Ganmor, Landy, & Simoncelli, 2015; Oostwoud Wijdenes, Marshall, & Bays, 2015; Wolf & Schütz, 2015). Other possible mechanisms include transsaccadic learning, as several authors have shown that transsaccadic changes of stimuli (e.g., shape and spatial frequency) can alter how observers respond to presaccadic retinal stimuli (Bosco, Lappe, & Fattori, 2015; Herwig & Schneider, 2014; Herwig, Weiß, & Schneider, 2015; Valsecchi & Gegenfurtner, 2016). Transsaccadic learning is thought to occur because the visual system learns to associate presaccadic peripheral visual input and postsaccadic foveal visual input over the course of many saccades (Weiß, Schneider, & Herwig, 2014). Both transsaccadic learning and transsaccadic integration could underlie comparisons of presaccadic and postsaccadic retinal input, potentially bridging the gap in visual processing during saccades. 

The ability of the visual system to evaluate stored presaccadic visual information after a saccade has been made, has been shown in several studies (Hollingworth, Richard, & Luck, 2008; Irwin, 1992; Oostwoud Wijdenes et al., 2015; Richard et al., 2008). In particular, studies on gaze correction have shown that visual information acquired during presaccadic attentional shifts can be used to refixate a target object if it is displaced during a saccade (Hollingworth et al., 2008). In the study by Hollingworth and colleagues (2008), participants were presented with a circular array of stimuli. The saccade target was indicated by a brief increase and decrease in size of one of the stimuli. When the participant executed a saccade to this cued target, the array rotated, such that the participant landed in between the previously cued object and an uncued (distractor) object. To correctly perform the corrective saccade task, the observers had to execute a corrective saccade based on the available visual information alone, as variance in the motor system was uninformative in regards to the postsaccadic location of the target. The study showed that participants consistently executed a secondary (corrective) saccade to the displaced target. The authors thus showed that information about the saccade target is acquired before execution of the saccade, allowing for corrective saccades. Another study on corrective saccades showed that VWM content may bias and increase the latency of visual corrective saccades (Hollingworth & Luck, 2009). In the study by Hollingworth and Luck (2009), observers were tasked to remember the exact color of a stimulus. The crucial manipulation was a distractor object in a visual corrective saccade task that changed into a color that was being maintained by the participant. Thus, corrective saccades were slower and biased towards the distractor object when compared to a distractor with irrelevant features. These results indicate that VWM is tightly coupled with motor execution in a visual corrective saccade task. 

Overall, prior studies indicate that corrective saccades are guided by current VWM content. The opposite relation, hence how VWM content is affected by the execution of a saccade, is less clearly defined. To specify, the proposed interaction between saccade execution and VWM is that visual information at and surrounding the saccade target is obligatorily and preferentially encoded into VWM before a saccade is executed. We hypothesize that, before the eye movement is executed, a prediction of the visual features of the “to-be-fixated” object is made. This prediction will most likely be as accurate as possible, which means it will have a VWM load comparable to actively committing one extra item to VWM (or possibly an even higher load). We also hypothesize that the visual system will prioritize information acquired by presaccadic attentional shifts, as it is essential for the evaluation of saccade accuracy, which in turn is important for a stable perception of the environment. If this is indeed the case, then the encoding of the saccade target in VWM should decrease the precision of other information that is stored in VWM, as features of the saccade target mandatorily compete for VWM resources. 

In the current study, we investigated whether and how VWM content is affected by visual information that is acquired before the execution of a saccade. Furthermore, the implications of retrieving information that is required for a subsequent gaze correction for VWM were investigated. We expected that information that was present in VWM at the time of saccade execution would be represented less accurately (higher standard deviation in the distribution of responses) because of the encoding of the saccade target into VWM (Experiment 1). In addition, to correctly perform a corrective saccade task, the stored visual features of the saccade target must be retrieved from VWM. We expected this process of retrieval to be delayed when competition for VWM resources was higher (similar to Hollingworth, 2008). To investigate whether these effects were exclusive to VWM load and to exclude the effects of task-load in general, we conducted a second experiment. Participants had to perform a dual-task in which they had to retain auditory or visual information and execute a saccade. Prior research has shown, that auditory working memory load affects corrective saccade latency less than VWM load (Hollingworth, 2008), but that the effect of executing a saccade on items stored in auditory working memory are unknown. We expected items stored in auditory working memory to be less affected by executing a saccade, compared to items in VWM, due to auditory information occupying a resource separate from VWM. 

Experiment 1

Method

Participants

Fifteen participants (six male, nine female) aged 18 to 28 (M = 21.9) were tested, for 90 min each. Participant amounts were chosen to match prior research (Hollingworth et al., 2008; Schut, Fabius, Van der Stoep, & Van der Stigchel, 2016). All observers participated for a monetary compensation of €6/hr. Participants reported normal or corrected-to-normal vision and were naïve as to the purpose of the study. Written informed consent was obtained from all participants. One participant chose to end the experiment prematurely and was excluded from further analyses. During the testing of another participant, a power outage occurred, delaying the experiment by 10 min. The study was reviewed and approved by the Faculty Research Ethics Committee (FETC) of the University of Utrecht. The data and analyses are registered and available online (Schut, 2017). 

Stimuli and apparatus

Participants performed three tasks in blocks of several trials: a saccade task (124 trials per participant), a VWM task (124 trials per participant), or both tasks in the dual-task blocks (186 trials per participant). The tasks were presented in blocks of trials and did not differ in stimulus presentation, only in preblock instructions. Moreover, the order of the tasks was randomized per participant. 

In the VWM task, the participant was instructed to remember one, two, or three shapes presented sequentially at the center of the screen. We chose to present the stimuli in sequence so that we could run an auditory analogue to the VWM task (Experiment 2). The VWM shapes were morphed by manipulating the width and height, while keeping the surface area at a constant value (surface area of the shapes = 2.25°). For example, a square with a width-height-ratio of 1.0:1.0 would be 1.5° wide and 1.5° high. The VWM shape categories used in the VWM task were a square, diamond (a square rotated by 45°), and triangle. The width-height-ratio of the VWM shapes was a continuous value, randomly drawn from one of two uniform distributions (with width-height-ratios ranging from 1.0:1.2 to 1.0:2.0). The participants were instructed to remember the exact form of the VWM shapes (a maximum of three stimuli). The VWM shape categories and width-height-ratios were randomly chosen per trial, with the constraint that all VWM shapes in a trial were unique in both shape category and width-height-ratio. After a retention interval, a test stimulus was presented. The test stimulus was one of the three VWM shapes categories (e.g., square) with an equal width and height (width-height-ratio of 1.0:1.0); the width-height-ratio of this stimulus had to be altered to match the remembered object by pressing the arrow keys. The smallest step size of a single press of a key was a change of 0.0025 in width-height-ratio. The step size increased if one of the arrow keys was pressed by number of seconds since release × 30. The step size reset to a change of 0.0025 if a key was released. This was done to ensure accuracy in changing the values by tapping the keys, and to avoid frustration by reducing the amount of time needed to indicate a large change by holding down a key. Once the participant thought the response stimulus matched the remembered object, they pressed the space bar to confirm their answer. The participant could only end the trial if one of the arrow keys had been pressed. 

The saccade task was performed in between the presentation of the VWM shapes and the VWM test stimulus. In the saccade task, six colored circles with a diameter of 1.5° were presented on the screen. Colors were drawn from a subset of four colors: red (12.1 cd/m²), green (18.9 cd/m²), blue (9.9 cd/m²), and magenta (14.2 cd/m²). The restriction in assigning the colors was that no neighboring circles could have the same color. The six colored circles were placed equidistantly on an imaginary circle with a radius of 4.5° from the fixation stimulus (a gray dot of 0.6° in diameter, 4.1 cd/m²). The array of stimuli was rotated randomly per trial, with the additional restriction that no stimuli could appear on the cardinal axes. One of the circles was cued as the saccade target by expanding its size to 2.1° in diameter for 100 ms. 

The experiment took place in a darkened room. Stimuli were presented on an Asus ROG swift PG278Q (27 inch, 60.1 by 34.0 cm) monitor with a spatial resolution of 2560 × 1440 pixels and the refresh rate set at 120 Hz. Participants were seated 70 cm from the monitor with their heads resting on a desk-mounted chin- and headrest. Eye movement data was collected using an EyeLink 1000 (SR Research Ltd., Oakville, ON, Canada), positioned at 65 cm from the participant. The left eye was recorded at 1000 Hz. Saccades were detected offline with the default values of the EyeLink algorithm for saccade detection. 

The experiment was programmed in Python 2.7 using the Pygaze library to connect to the eye tracker and to define areas of interest (Dalmaijer, Mathôt, & Van der Stigchel, 2013). Eye tracker data files were analyzed with Python 2.7, and statistical analyses were performed using R 3.1.3 (Ihaka & Gentleman, 1996). 

Procedure

Participants completed the experiment in three blocks of trials: a saccade only block, a VWM only block, and a dual-task block in which participants performed both the saccade task and VWM task. A typical trial within the dual-task block occurred as follows (see Figure 1A): A fixation stimulus was presented, after which one, two, or three VWM shapes were shown sequentially in the center of the screen (see Figure 1B). The VWM shapes were either morphed to have a low width-height-ratio or a high width-height-ratio. Participants were instructed to remember the exact form of all VWM shapes that were presented. Each VWM shape was shown for 900 ms, followed by a 100 ms blank interval. After all VWM shapes were presented, a fixation point was shown. In trials in which less than three stimuli were shown, the duration of the fixation period was increased to match the retention time of a trial with three VWM shapes. After an interval of 400 ms, six colored circles were shown. This indicated to the participant that the saccade task had started. One of the circles was cued by expanding in size for 100 ms and subsequently returning to its original size in the next refresh cycle of the screen. The participant was instructed to execute an eye movement to this stimulus as soon as a circle was cued. If the participant did not initiate a saccade within 800 ms, the array rotated by itself. During the eye movement, the entire array of stimuli rotated such that the participant's gaze landed in between the cued stimulus (target) and another stimulus (distractor), a 30° rotation. Eye-movements were determined by the recorded gaze position leaving a circular region of 1° visual angle around the last sampled point. In one third of the saccade trials, the array did not rotate. The participant was tasked with executing a saccade to the postsaccadic position of the cued stimulus (Figure 1C). After a second saccade was detected, 200 ms passed and all circles were removed from the screen. Finally, a stimulus representing a shape category was presented in the center of the screen. Participants were tasked with manipulating the width-height-ratio of the response stimulus (by using the arrow keys) to match the shape they had seen at the start of the trial. To reduce end-point biases, participants could manipulate the width-height-ratio of the response stimulus 30% further than the range of the distribution the VWM shapes were drawn from (examples of extreme VWM shape values shown in Figure 1D). The participant confirmed their answer by pressing the space bar. In saccade only trials, the stimulus presentation matched the dual-task trials, but participants were required to maintain fixation for 1200 ms instead of initiating saccades. 

Figure 1

 

The procedure of Experiment 1. Square panels indicate the stimuli that were presented (A, C); rounded panels illustrate the experimental design (B, D). Note that the square panels within the entire figure are presented in inverted contrast. (A) An overview of a trial. In the given example, the memory load = 1. Blue solid lines indicate the VWM portion of the trial; orange dashed lines indicate the saccade task portion of the trial. (B) An overview of the VWM task. Subjects were tasked with remembering one to three stimuli during the “Memory Array” phase and were asked to reconstruct the width-height-ratio of one of these stimuli in the “Reconstruct” phase. (C) The procedure of the saccade task. The two saccades of importance are shown: “First saccade” and “Corrective saccade.” In the VWM only task, the participants kept their gaze on the fixation point for 1200 ms. (D) Example stimuli for the extreme values of the three shape categories.

Data analysis

Our parameters of interest were the performance on the VWM task in the VWM only block as compared to the dual-task block, and the saccade latency and accuracy during the saccade only block as compared to the dual-task block. We calculated the difference between the shown width-height-ratio of the correct VWM shape and the width-height-ratio that was reported by the participant. The parameters of interest were mean error and precision (the inverse of the standard deviation). As shown by Bays et al. (2009), when working memory resources are occupied, response precision decreases. We investigated whether mean error was significantly different between the conditions using a Bayesian repeated-measures ANOVA. A difference in mean error would indicate bias, rather than less accurate reporting. If no difference was present, we would fit a linear mixed model predicting the precision of responses to the VWM task with the fixed factors of Task (VWM only or dual-task) and Working Memory Load as a contrast-coded factor (one, two, or three items). We checked the normality of the residuals in the linear mixed models using a Shapiro-Wilk test, and reported the mean and standard deviation of the residuals. We linearized the responses on the VWM task (reported width-height-ratios), to make the distance in the width-height-ratio identical between ratios above zero and below zero, where zero indicates a width-height-ratio of 1:1. 

We compared corrective saccade latency and accuracy within the saccade only block and the dual-task block. We were particularly interested in the relation between VWM and corrective saccades as we reasoned that corrective saccades can only be correctly executed if the saccade target is retrievable from VWM. First, to determine at which moment the corrective saccade occurred, we analyzed the offset of the first saccade. The first saccade was defined as the saccade with an offset of 60 to 800 ms relative to saccade target onset (99.2% of trials). The next saccade with an onset of 60 to 800 ms relative to the offset of the first saccade, was taken as the corrective saccade (95.7% of remaining trials). We chose a lower bound of 60 ms since the conduction delay for afferent signals to transform retinal input into an oculomotor response is around 30 to 60 ms at minimum (Dorris, Klein, Everling, & Munoz, 2002; Dorris, Paré, & Munoz, 1997; Fischer & Boch, 1983; Fischer & Weber, 1993; Paré & Munoz, 1996). This suggests that, for visual information to be processed, saccades must have a latency beyond 30 to 60 ms. The upper bound reflected the time it took for the array to rotate by itself if a participant did not initiate a saccade after the saccade cue appeared. A corrective saccade was categorized as being correctly executed if the saccade landed in the direction of the corrective saccade target (i.e., the angle between the target and the saccade, should not be greater than 90°) and within 2° of the corrective saccade target. Saccades that did not meet these two criteria were categorized as incorrect, but not excluded (485 out of 2243 trials). 

We also performed control analyses using linear mixed models with the latencies of the first saccades as the dependent variable. Previous research has shown that first saccade latency and corrective saccade latencies may be closely linked in a corrective saccade paradigm (Schut et al., 2016). Therefore, we wanted to establish whether the potential effect of altered corrective saccade latencies with higher VWM load were not simply due to altered first saccade latencies. 

We opted for Bayesian statistics, which can differentiate between significant null results versus nonconclusive evidence (for an overview of Bayesian Hypothesis Testing, see Wagenmakers, Lodewyckx, Kuriyal, & Grasman, 2010). Bayes Factors (BF) can be interpreted as the ratio of evidence for one hypothesis over another. For example, a test which shows BF = 100 for the alternative model, indicates that there is 10 times more evidence than a test in which the evidence for the alternative model shows BF = 10. To interpret the strength of evidence of a BF, Kass & Raftery (2012) have provided guidelines. A BF of between 1 and 3 is described as providing evidence that is “not worth more than a bare mention.” A BF of 3 to 20 provides “positive” evidence, 20 to 100 “strong” evidence, and above 100 “very strong” evidence (Kass & Raftery, 2012). For readability, we report all BFs in favor of one model over the other, where BF₁₀ is the evidence for the alternative hypothesis over the null hypothesis and BF₀₁ is the evidence for the null hypothesis over the alternative hypothesis (since BF₁₀ is equal to BF₀₁⁻¹). 

Bayesian linear mixed models were constructed with the BayesFactor package (Morey & Rouder, 2015). The BF was calculated for the models that included the fixed effect and was divided by the BF of the models that did not include the fixed effect to test the predictive value of adding that factor to the model (Bayesian Model Averaging; Hoeting, Maigan, Raftery, & Volinski, 1999). Interaction effects were only tested for models that included all main effects present in the interaction effect, similar to the approach used in the Bayesian ANOVAs in JASP (Wagenmakers, 2015). Each BF was multiplied by the proportion of models that included the factor (prior probability). Additional fixed effects were only included if the BF was 3, indicating positive evidence, or higher in favor of the more complex model. As a measure of uncertainty we provided the (Bayesian) 95% confidence intervals, also known as the 95% credible intervals (95% CI; Chen & Shao, 1999). The 95% CIs represent the 2.5th and 97.5th quantile of the posterior density function. This interval can be interpreted as being 95% confident that the true value of a parameter lies within the calculated 95% CI, after observing the data. These results were corroborated with frequentist linear mixed models, which were constructed using the lme4 R package (Kuznetsova, Brockhoff, & Christensen, 2013). Between frequentist linear mixed models, we compared Bayesian Information Criterions (BICs), as a measure of model performance. For model comparison purposes, an interaction effect was only included if both main effects were present in the model. A fixed effect was only included if the BIC was 10 points lower in the complex model as compared to a simpler model without the factor. 

For frequentist linear mixed model comparisons, the X² and p values are reported per model comparison, with a significance criterion of α = 0.05. For the best performing model, we report the parameter estimates β and the corresponding t values. Lastly, linear mixed models were visualized by plotting the linear fit of the best performing model. The fit was plotted without the random effects (i.e., removing the influence of separate intercepts per participant). 

To test similarity of remembering an extra item or performing a saccade, a Bayesian hypothesis-test was performed using the Savage-Dickey method (Verdinelli & Wasserman, 1995; Wagenmakers et al., 2010), in which the difference between the reduction of precision by addition of a saccade task and the reduction of precision per additional VWM item in the posterior distributions were compared. Thus, a difference between conditions (δ) would indicate the relative cost of performing the saccade task, with δ = 0 showing that performing a corrective saccade has the same cost, as remembering an additional memory item. We compared the distribution to the prior distribution (a Cauchy distribution with a width of 0.707) centered on 0, as a null hypothesis that δ = 0. 

The Bayesian models were checked for autocorrelation and failure to converge, by examining the trace of the Markov-chain Monte Carlo (MCMC) sampling process. A failure to converge would indicate that the MCMC sampling process was not able to estimate a parameter correctly. The measure of convergence between MCMC chains (Rhat) in every model was not higher than 1.1, indicating that's there was no serious convergence issues in the MCMC process, and thus there was reliable parameter estimation (Brooks & Gelman, 1998; Gelman, Rubin, Gelman, & Rubin, 1992). For all analyses that required resampling we ran 10,000 iterations. 

As an exploratory analysis, we investigated the effect of stimulus history on working memory precision. The effect of stimulus history was particularly present in the study by Bays and Husain (2008), where items closer to the saccade onset were reported with more precision. We were interested whether the effect of performing a dual task differed between items within the sequence. To analyze this, we only included trials in which three working memory items were presented in the analyses. We used model comparison techniques as described previously, reporting BIC values and results for Bayesian model averaging tests. Based on these tests, we examined the best performing model in better detail, to investigate which items in the sequence were remembered better than others. 

For exploratory purposes, we investigated the differences of random guesses between VWM conditions with a mixture model analysis. As described by Bays and Husain (2008), working memory performance can be deconstructed into a Gaussian component, and a uniform component. The uniform component is a better model for trials in which participants guessed, and where any answer is equally likely. The mixture models contain three parameters: (a) the mean of the Gaussian distribution as a measure of the response bias, (b) the standard deviation of the Gaussian distribution as a measure of the response precision, and (c) the height of the uniform component as a measure of the guess rate. With these analyses, we could identify whether participants guessed more as working memory load increased. When we modelled these (more complex) mixture models, we found that there were no differences in guess rates between conditions (single task vs. dual-task, and one, two, or three VWM items, BF₀₁ > 70). Furthermore, using the product-pace method (Lodewyckx et al., 2011), we found that the more parsimonious Gaussian only model (as described in the prior paragraphs of the methods section) outperformed the mixture model. Since the mixture model did not outperform the Gaussian only model, and there were no differences between guess rates in any of the conditions, we chose to describe the data with a Gaussian component only. Likely, the uniform component required an experimental design with a larger number of random guesses. For example, in the Bays and Husain (2008) study, the number of working memory items went up to six items, which would require more guesses. In the current paper, the set size only goes up to three, which would probably constrain the amount of completely random guesses to trials in which the participants were not paying attention. The model code and analysis script are available online (Schut, 2017). 

Visual working memory precision

First, we analyzed whether the VWM task was performed less precisely if the participant was also performing the saccade task. As a control analysis, we first determined whether a significant response bias was present between conditions. We did not expect any differences in the mean response, but rather for participants to be less precise as working memory resources were occupied by an eye movement task. A Bayesian analysis of variance confirmed these hypotheses, showing that differences in mean-error was not predicted by task (BF₀₁ = 13.9), working memory amount (BF₀₁ = 134.0), task and working memory amount (BF₀₁ = 1993.0) or a full factorial model (BF₀₁ = 1.21 × 10⁵). Therefore, a derivative of the standard deviation could be used for further inferences. Our mean data suggested that participants, on average, performed the VWM task less precisely in the dual-task condition (M = 6.29, SD = 2.73) as compared to the single task condition (M = 8.06, SD = 3.54). Furthermore, the mean data suggests that participants were more precise in answering when one item was retained (M = 9.10, SD = 3.87) compared to when two (M = 6.97, SD = 2.64) and three items were retained (M = 5.45, SD = 1.99). 

To quantify this, a linear mixed model was performed, with the precision of responses on the VWM task as the dependent variable and Task (VWM only block or dual-task block), and Working Memory Load (one, two, or three items presented) as fixed effects. Lastly, a random intercept per participant was added to the model. To test the predictive value of the addition of the fixed effects and interaction effect, BF of the models that included our effects were compared to models that did not. The models including Task (VWM only or dual-task) as a fixed effect were 421 times more likely to explain the data than models that did not include the factor Task (BF₁₀ = 11.72). Similarly, the models that included a fixed effect for Working Memory Load were about 7,000 times more likely to explain the data than models that did not, BF₁₀ = 7038.70. Conversely, the models that included an interaction effect between Task and Working Memory Load were less likely given the data, BF₀₁ = 26.85. The parameter estimates are shown in Table 1. 

Table 1

 

Parameter estimates for Bayesian Model Averaging analysis. Note: Bayes Factors have been rescaled to the participant model for readability.

 

These results were consistent with the frequentist linear mixed models, showing that the best performing linear mixed model (including Task and Working Memory Load without interaction effect, BIC = 428) had more explanatory value than models that only included Task as a fixed effect, BIC = 444, X²(2) = 24.818, p < 0.01, and models that only included Working Memory Load as fixed effect, BIC = 433, X²(1) = 9.59, p < 0.01. The more complex model that included the interaction effect between Task and Working Memory Load, BIC = 436, X²(2) = 0.95, p = 0.62, did not outperform the simpler models without an interaction effect. A Shapiro-Wilk test confirmed that the residuals of the model with Task and Working Memory Load as fixed effects were normally distributed (M = 1.7 × 10⁻¹⁵, SD = 0.73, W = 0.91, p = 0.14), indicating it is an appropriate model for the data. The predicted β-coefficients and corresponding t values of the model with two fixed effects are shown in Table 2. 

Table 2

 

Parameter estimates for frequentist linear mixed model predicting performance on the working memory task.

 

Together, the analyses supported our hypothesis that performing a saccade task while retaining unrelated items in VWM, reduces response precision for the subsequent VWM task. Furthermore, the additions of a saccade task to a working memory task as well as memory load in a VWM task do not interact. This result indicates that the amount of VWM working memory items did not affect the magnitude of the reduction of VWM precision. 

The fit of the linear model and the measured mean performance are shown in Figure 2A. The model shown includes fixed effects for Task and Working Memory Load and a random intercept per participant. The average response precision on the tasks as predicted by the model is lower in the dual-task (M = 6.37, 95% CI = 5.23–7.48) compared to the VWM only block (M = 7.96, 95% CI = 6.86–9.11). Participants also got worse as the number of items increased (one item: M = 8.92, 95% CI = 7.69–10.19, two items: M = 6.98, 95% CI = 5.77 to 8.21, three items: M = 5.60, 95% CI = 4.32–6.85). Figure 2A visually indicates a slight overlap of the 95% CI between the two tasks and the working memory load, due to uncertainty in the estimate of the mean score that the other scores are centered around. Therefore, we also plotted the joint posterior distributions of Task and Working Memory Load in Figure 2B. Negative values for Task show that the dual-task block was performed less accurately than the VWM only block. Similarly, negative values for VWM precision show the decrease of precision per added item. Two separate point clouds are plotted, one for the difference in precision between one and two items and the other for the difference in precision between two and three items. A Bayesian hypothesis test supports that the difference between dual task and VWM task is different from 0, BF₁₀ = 23.93, 95% CI = −2.72 to −0.53. Furthermore, the difference between one and two items retained as well as the difference between two and three items retained is different from 0, BF₁₀ = 23.43, 95% CI = −3.26 to −0.63 and BF₁₀ = 3.11, 95% CI = −2.69 to −0.08. Lastly, we found marginal evidence for no difference between one and two items retained versus two and three items retained, BF₀₁ = 1.49. Importantly, we hypothesized that the cost of making a saccade may be close to the cost of remembering an additional item, as saccade targets have been shown to be mandatorily encoded into VWM prior to a saccade. A Bayesian hypothesis test indicates that performing the saccade task elicits a similar load on working memory as remembering an additional item (BF₀₁ = 3.74, when comparing the dual-task versus the VWM task to one versus two items, BF₀₁ = 3.85 when comparing single task versus dual-task to two versus three items). These results agree with results from a frequentist paired t test, t(12) = −0.40, p = 0.35. Thus, participants were significantly less precise in reporting the items when they were instructed to perform a concurrent saccade task. The loss of precision of performing a dual-task is comparable to remembering an extra item. 

Figure 2

 

(A) The fit of the mixed effects model, which predicts VWM task precision based on Working Memory Load and Task. The measured mean precision per Task and Working Memory Load is indicated by the squares and triangles. The 95% CI is shown as a gradient, with higher densities being less transparent. (B) The joint posterior of the parameters Task and Working Memory Load. Negative values on the y axis indicate lower precision on the dual-task with respect to the VWM task. Negative values on the x axis indicate lower precision per addition of a working memory item. The blue points indicate the difference between one and two items retained; the orange points indicate the difference between two and three items retained. The joint 95% CIs are shown as an orange and a blue circle.

To test whether the execution of a corrective saccade was the driving factor behind decreased memory performance in the dual-task, we compared trials in which the array rotated during the first saccade (corrective saccade trials) to trials in which the array did not rotate during the first saccade (single saccade trials). When adding the additional fixed effect of rotation (single saccade trials vs. corrective saccade trials) to the mixed effects model with task and VWM load as fixed effects, we found no predictive value of corrective saccades for the precision of VWM responses, BF₀₁ = 4.08. We infer that VWM information seems to be in competition with resources acquired around the time of the execution of the first saccade, as executing a corrective saccade did not significantly influence precision of responses in the VWM task. 

First saccade latency and accuracy

As a control analysis, we examined differences in first saccade parameters between conditions before examining differences in corrective saccade metrics. We observed that the median first saccade latency in the dual-task was 206 ms (SD = 79.4 ms, Mean amplitude = 4.94°) if one item was being retained, 203 ms (SD = 79.1 ms, Mean amplitude = 4.96°) if two items were retained, and 199 ms (SD = 81.5 ms, Mean amplitude = 4.82°) if three items were retained. In the saccade only block, the first saccade latency was 213 ms (SD = 92.5 ms, Mean amplitude = 4.96°) if one item was presented, 205 ms (SD = 104.3, Mean amplitude = 4.86°) and 203 ms (SD = 84.5 ms, Mean amplitude = 4.99°) if two or three items were presented, respectively. Using a one-sided Bayesian t test, we tested the effect of Task (saccade only vs. dual-task) on first saccade latency. This test provided evidence that there was no effect of Task on first saccade latency, BF₀₁ = 121.88 and that there was no effect of Working Memory Load on first saccade latency, for one versus two items: BF₀₁ = 3.54 and for two versus three items BF₀₁ = 8.23. Similarly, the data provides more evidence for no differences for Task in first saccade amplitude, BF₀₁ = 9.47. Similar results were found for the effect of items presented on first saccade amplitude, for one versus two items: BF₀₁ = 8.31 and for two versus three items: BF₀₁ = 4.19. 

Corrective saccade latency and accuracy

After investigating the effect of corrective saccades on VWM precision, we examined the effect of VWM load on corrective saccade metrics by comparing the saccade only block and the dual-task. We investigated the effect of Task (saccade only vs. dual-task) and Working Memory Load (one, two, or three items presented) on two corrective saccade metrics: a saccade accuracy (proportion of saccades landing within 2° of the target) and saccade latency (time in ms between previous saccade offset and corrective saccade onset). We expected corrective saccades to be performed less accurately or initiated more slowly in the dual-task. 

Examining the data showed that the median corrective saccade latency in the saccade only block was 243 ms (SD = 127 ms) and for the dual-task was 232 ms (SD = 124 ms). When one item was presented, median saccade latency was 236 ms (SD = 121 ms). The median saccade latency was 238 ms (SD = 118 ms) when two items were presented, and 242 ms (SD = 138 ms) when three items were presented. The mean proportion of accurate corrective saccades was 0.83 (SD = 0.38) in the saccade task and 0.76 (SD = 0.43) in the dual-task. For Working Memory Load, we find a proportion of 0.81 (SD = 0.40) correctly executed saccades when one item is presented, and proportions of 0.77 (SD = 0.42) and 0.77 (SD = 0.42) correctly executed saccades for two and three items presented, respectively. 

Two linear mixed models were constructed to analyze corrective saccade latency and accuracy. One model contained corrective saccade latency as a dependent variable and the other with corrective saccade accuracy as a dependent variable. The fixed effects entered into the model were Task and Working Memory Load, with an interaction effect between the fixed effects. A random intercept was added for each participant. Model comparison for corrective saccade latency showed that the model including only Task as a fixed effect outperformed the models that did not, BF₁₀ = 12.3. Working Memory Load was not predictive within our models, BF₀₁ = 1.06, and neither was the interaction effect between Working Memory Load and Task. BF₀₁ = 1532. A similar pattern of results was found for corrective saccade accuracy. Whereas Task was predictive (BF₁₀ = 8.13) for corrective saccade accuracy, Working Memory Load and the interaction between Working Memory Load and Task were not (BF₀₁ = 1.05, BF₀₁ = 696). This shows that in the dual-task corrective saccade accuracy was lower and latency was higher than in the single-task condition. The BF₁₀s for the models are shown in Table 3. 

Table 3

 

Parameter estimates for Bayesian Model Averaging analysis. Note: Bayes Factors have been rescaled to the participant model for readability.

 

The model comparison using frequentist linear mixed models, yielded congruent results. The model including Task as a fixed factor and a random intercept (BIC = 27952) outperformed models with fixed effects for Working Memory Load and Task (BIC = 27954 to 27954–27975, p > 0.25). We constructed a generalized linear mixed model with one fixed effect (Task) and random effects for saccade accuracy. This model showed that the inclusion of the factor Task had significant predictive value, z = −3.11, p < 0.01, similarly outperforming more complex models including parameters for Working Memory Load, BIC = 2209 for the model including Task, BIC = 2221–2223, p < 0.01, for the other models. The estimates for the frequentist linear mixed models for saccade accuracy and saccade latency are shown in Table 4. Both models showed normally distributed residuals for the random intercept per participant, M = 3.71 × 10⁻¹² ms, SD = 36.93 ms, W = 0.96, p = 0.79 for saccade latency, M = −0.01 ms, SD = 0.74 ms, W = 0.92, p = 0.22 for saccade accuracy. 

Table 4

 

Parameter estimates for frequentist linear mixed models on corrective saccade latency and accuracy.

 

Based on these model comparisons, we further examined the models that only included Task as a fixed factor and a random intercept per participant to predict corrective saccade latency and corrective saccade accuracy. We normalized the means, which were centered around a latency of 268 ms (95% CI 239–297 ms) and a 0.78 proportion correct (95% CI = 0.68–0.88 ms). 

Within the Bayesian linear mixed models, saccade latency in the dual-task was 17.4 ms (95% CI = 28.3–6.67 ms) which was higher than in the saccade only task. Similarly, the model for saccade accuracy showed that the proportion of accurate corrective saccades was, on average, 0.05 ms lower in the dual-task when compared to the saccade only task (95% CI = 0.02–0.09). These results are shown in Figure 3. 

Figure 3

 

Fit of the Bayesian linear mixed model including Task as fixed effect to the measured corrective saccade latency (A) and proportion of accurate corrective saccades (B). The dark line is the calculated model; the error bars indicate the 95% CI. The transparent squares show the observed point estimates. Note that the figure shows relative values, rather than absolute values.

Exploratory analysis: Stimulus history

Next, we examined the effect of stimulus history on the cost of a saccade. For these analyses, we only investigated trials in which three items were presented. First, we performed a Bayesian model comparison, which included precision on the VWM task as a dependent variable, as well as Task (VWM only vs. dual-task) and Item Index (first, second, third item shown) as fixed variables. The results show positive evidence for models including the factor Task, BF₁₀ = 6.7. For the fixed effect Item Index, we found inconclusive evidence, BF₀₁ = 1.01. The results show positive evidence against the inclusion of an interaction effect between Task and Item Index, BF₀₁ = 17.0. We further analyzed the model with Task and Item Index as fixed effect and found that the second item was reported worse (M = −1.25, 95% CI = −2.26 to −0.27) compared to the first item (BF₁₀ = 4.01, M = 0.50, 95% CI = −0.44 to 1.44) and third item (BF₁₀ = 7.19, M = 0.74, 95% CI = −0.21 to 1.72). Together, these results speak in favor of a difference in working memory precision between the positions of items in a sequence. Importantly, the models suggest that this effect does not interact with the cost of a saccade task on visual working memory precision. 

Discussion

Our results in Experiment 1 showed that participants were less precise on a VWM task when performing an additional saccade task. Furthermore, corrective saccades within the dual-task were executed slower and less accurately on average compared to saccades in the saccade only condition. Although the corrective saccades were slower and less accurate, the first (or prior) saccade seemed unaffected by the addition of a VWM task. Presumably, the corrective saccade requires information to be successfully retrieved from VWM, whereas initial saccade execution does not depend on VWM encoding. The cue to execute the first saccade is externally available, leading to increased saccade latency for corrective saccades, but not first saccades. The results indicate that the cost of performing a saccade is similar to storing an additional item in VWM. Performing a corrective saccade does not seem to influence VWM precision, indicating that prior to the first saccade, feature information is already mandatorily encoded in VWM. To assess whether these differences are strictly tied to storing visual information in VWM, rather than these results being explained by the added demands of performing two tasks, we conducted a second experiment. Experiment 2 was similar to Experiment 1, but now also included trials in which participants performed an Auditory Working Memory (AWM) task, instead of only a VWM task. 

Experiment 2

Method

Participants

In Experiment 2, 13 observers (three male, 10 female) aged between 18 and 28 (M = 22.9), participated. The study was approved by the faculty ethics committee, and followed the same guidelines as Experiment 1. Experiment 2 lasted for two hours; participants were monetarily compensated with €6/hr. 

Stimuli and apparatus

The experiment consisted of three tasks, mixed per block: working memory (WM) trials, saccade only trials, and dual-task trials (WM and saccade task). The modality of WM trials was either visual (VWM) or auditory (AWM). For the VWM trials, the same rectangular morph stimuli were used as in Experiment 1. Sine waves in a log-linear range between 350 Hz and 2000 Hz were used in the AWM task. The auditory stimuli were presented on a single speaker placed behind the monitor. To be able to match the characteristics of the two stimulus modalities, we opted for a two-alternative, forced-choice paradigm. To match the difficulty of the tasks, a QUEST staircase procedure was used (Watson & Pelli, 1983). Three stimuli were presented, followed by a retention interval. After the retention interval, one of the remembered stimuli and a novel stimulus (i.e., the remembered stimulus plus or minus a staircased value) were presented in a random order. Participants responded by indicating which of the two sounds was the remembered stimulus (i.e., the first or second stimulus presented). The threshold was set at a performance of 75% correct and 100 trials were run. If a difference lower than 1.0 in width-height-ratio was not detectable by the participant in the visual task, or a difference lower than 0.227 in log space was not detected in the auditory task (a difference of approximately 100 Hz at the low end of the frequency range and 485 Hz at the high end of the frequency range), the participant was excluded from the experiment (two participants). Stimuli for the saccade task were identical to that for Experiment 1. 

Procedure

The experiment was divided into two phases: the threshold phase and the experimental phase. First, participants completed a staircase procedure in the threshold phase for the VWM task and the AWM task. During the threshold phase, participants performed 42 saccade trials to familiarize them with the task. After the staircase procedure, the experimental phase began. The aim of the thresholding phase was to find a magnitude of modification that resulted in a performance that was not at chance (or at ceiling) between AWM and VWM trials (around 75%). To illustrate, in the VWM thresholding phase three rectangles were shown to the participants (Figure 4). Participants were instructed to remember the exact shape of the rectangles shown. The rectangles were presented in sequential order and were presented for 600 ms on the screen followed by a 400-ms blank. After all stimuli were presented, a blank screen was shown for 1000 ms (the retention interval). Afterwards, participants were presented a response cue in the form of a number. The number indicated to which stimulus participants were supposed to respond. For example, after being presented three items, the number 3 instructed the participants to report the shape of the third rectangle that was presented previously. Next, the two rectangles appeared, one of which was the actual nth stimulus, the other being a rectangle with a modified width-height-ratio. Trials in the AWM thresholding phase were similar, but rather than finding a 75% correct threshold for the width-height-ratio, the 75% correct threshold for differences in tone frequency was determined. The third task during the thresholding phase, the saccade task, was identical to the saccade task described in Experiment 1. 

Figure 4

 

Procedure of the thresholding task in Experiment 2. The smaller black squares indicate that the fixation cross was presented for 400 ms. The dashed green line follows the procedure for the auditory task; the solid blue line, for the visual task. The dual-task followed a similar procedure, but the retention interval was replaced by a gaze correction task (see Figure 1 for a graphical representation).

After the thresholding phase was completed, the experimental phase started. Experiment 2 had a 2 (Modality: AWM vs. VWM) × 3 (Task: dual-task, WM only, or saccade only) × 2 (Working Memory Load: two or three items) design. The procedure was like the thresholding phase; only the retention interval was replaced by the saccade task. The trials were presented in a block where we predetermined whether participants had to perform the VWM or AWM task. Within this block, participants completed WM only, saccade only, and dual-task trials in randomized order. In contrast to Experiment 1, the different conditions did not differ in preblock instruction, but rather in the stimuli presented on the screen, due to the mixed design nature of Experiment 2. That is, if part of a trial (WM only, or saccade only) did not have to be performed, it was simply replaced by a fixation point. 

Data analysis

Similar analyses were used as for Experiment 1. However, since our dependent variable was dichotomous (i.e., correct/incorrect), we used generalized linear mixed models with a logit link function. This means that the model is linearly fit in log space (Dixon, 2008). This is good practice for binomial data, as the model never reaches mean values below 0 or above 1 (Dixon, 2008). To fit the generalized linear mixed model, we used the lme4 package (as previously described); for the Bayesian generalized, linear mixed model we used the MCMCglmm R package (Hadfield, 2010). Importantly, we investigated whether the difference between the dual-task and WM only was present in the VWM condition, while not being present in the AWM condition. To this end, we modeled the response as being predicted by the Task (WM only or dual-task), Working Memory Load (two or three items), and an interaction effect between Task and Modality (VWM vs. AWM). Random intercepts were added per participant for each model; residuals were checked for normality with Shapiro-Wilk tests. 

We excluded saccade trials based on the same criteria used in Experiment 1. We excluded trials based on first saccade latencies and corrective saccade latencies being smaller than 50 ms or larger than 800 ms (11 trials out of 1,440 excluded). The exclusion rate in Experiment 2 was likely lower than in Experiment 1 because participants had more practice with the saccade task prior to starting the experimental trials. 

Results and discussion

Working memory performance

To analyze the data in a similar manner to Experiment 1, we constructed a generalized linear mixed model for working memory performance, but rather than only including Task (WM only vs. dual-task) and Working Memory Load (as a factor, two or three stimuli presented) we added an interaction effect for Modality (visual or auditory). The model with Task and Working Memory Load, together with the added interaction between Task and Modality (BIC = 2166) significantly outperformed a null model, in which only a random intercept per participant was modeled, BIC = 2179, X²(4) = 43.5, p < 0.01, and a full-factorial model, BIC = 2187, X²(3) = 1.4, p = 0.70. The estimates of the best performing model are shown in Table 5; the residuals for this model were normally distributed, M = −0.004, SD = 0.21, W = 0.94, p = 0.51. Results for a Bayesian model comparison between the proposed model and a null model yield similar results, BF₁₀ = 33000, as well as comparing the proposed model to a full-factorial model, BF₁₀ = 982. This indicates that the linear mixed model including Task, Working Memory Load, and an interaction effect between Modality and Task is an adequate model for the data. 

Table 5

 

Parameter estimates for frequentist generalized linear mixed model predicting performance on the working memory task.

 

To recapitulate, we expected to replicate the effects we found in Experiment 1 for the VWM task. That is, better performance in the VWM only condition as compared to the VWM dual-task condition. Additionally, no difference was expected between the AWM only and AWM dual-task condition. First, we describe the average proportion of correct responses per variable that was entered in the model as a fixed effect. When two items had to be remembered the proportion of correct responses was 0.82 (SD = 0.39). The proportion of correct responses was 0.78 (SD = 0.41) when three items were presented. In the AWM trials, a proportion of 0.85 (SD = 0.36) of responses was correct. In the VWM trials this proportion was 0.75 (SD = 0.43). In the AWM only trials the proportion of correct responses was 0.86 (SD = 0.35), and in AWM dual-task trials we observed a proportion of 0.84 (SD = 0.37). For VWM only trials, the proportion of correct responses was 0.79 (SD = 0.41) and in the VWM dual-task trials the proportion of correct responses was 0.73 (SD = 0.44) correct. 

The model revealed a significant effect of Task, z = 2.61, p < 0.01, and Working Memory Load, z = −2.46, p = 0.01, 95% CI −0.56 to −0.06. Importantly, the results show that participants performed the VWM task worse in the dual-task VWM condition than in the single task VWM condition, z = −2.22, p = 0.03. There was no effect of Task for the AWM condition, z = −1.09, p = 0.27. Figure 5 shows both the model (Figure 5A) and the joint posterior of the effects of the AWM dual-task trials and the VWM dual-task trials (Figure 5B). The interaction between Task and Modality shows that the cost of making a saccade for the VWM dual-task is on average 0.07 greater (in proportion space) than for AWM dual-task (95% CI −0.02 to 0.16). 

Figure 5

 

The effect of performing a dual-task on both auditory and visual working memory performance. (A) Fit of the linear mixed model, 95% CI's are shown as smears, with density of the interval as the alpha value. (B) The joint posterior between single task and dual-task trials for auditory stimuli (y axis) and visual stimuli (x axis). The 95% CI of this plot is indicated by an orange ellipse. *p < 0.05

To quantify the difference in performance between the AWM dual-task and the VWM dual-task, we performed a Bayesian hypothesis test. The data was 7 × more likely to be explained under the model where there was a difference between the AWM dual-task and VWM dual-task (δ ≠ 0, BF₁₀ = 7.08). Therefore, our hypothesis that the effect of a saccade task would have less effect on an AWM task than on a VWM task was supported. A Bayesian hypothesis test was performed for the difference between the performance in the AWM dual-task and the AWM single task. The results indicated that the model which assumed no difference between the AWM dual-task versus AWM single task was more likely to explain the data than a model which assumed a difference, BF₀₁ = 32. Hence, performing a saccade task while retaining unrelated items in AWM does not affect AWM performance, whereas it does affect VWM performance. We concluded that, the decrease in WM performance between a dual-task and a single task in Experiment 1 was not exclusively due to increasing task demands. 

First and corrective saccade latency and accuracy

When investigating the effect of Working Memory Load on corrective saccade latency in Experiment 2, we used the best fitting model from Experiment 1. In the AWM task, first saccades in the dual-task condition had a median latency of 208 ms (SD = 88 ms) and 218 ms (SD = 77 ms) in the saccade only condition. In the VWM trials, the median corrective saccade latency in the dual-task condition was 218 ms (SD = 91 ms) and 199 ms (SD = 81 ms) in the saccade only condition. The model used to describe the corrective saccade latency included a fixed effect of Task, and an interaction effect between Task and Modality, and a random intercept per participant. The model comparison revealed that the model with fixed effects (BIC = 12062) performed worse than the model with only a random intercept per participant, BIC = 12042, X²(3) = 1.52, p = 0.68. A Bayesian linear mixed model analysis provided positive evidence in favor of the null model (BF₀₁ = 9.4), indicating that the parameters in the model were not predictive. Thus, we can conclude that the parameters in the model do not significantly predict corrective saccade latency. We also ran the analyses with corrective saccade end-point accuracy as the dependent variable. Like the saccade latency results, the null model outperformed the model with fixed effects, BF₀₁ = 153. We concluded that in Experiment 2, saccade latencies and saccade accuracy were not significantly affected by the modality of items retained or the number of items retained. 

General discussion

In the current study, we have investigated the load that executing a saccade puts on the visual working memory (VWM) system. The results of Experiment 1 indicated that the cost of performing a saccade was similar to storing one extra item in VWM. Participants were less accurate in reporting a VWM stimulus after executing a saccade during the retention of the stimuli. The accuracy reduction of items held in VWM was present whether participants executed a single saccade or two saccades (i.e., a saccade to the target followed by a corrective saccade). This accuracy reduction implies that before executing a saccade, information is mandatorily encoded in VWM. Moreover, corrective saccades were executed slower and less accurately in Experiment 1, when participants were retaining VWM items. This increase in corrective saccade latency could have reflected a slowed retrieving of the corrective saccade target from VWM when other items are occupying VWM. The results indicated that predicting future visual input had similar implications for VWM as actively committing an item to VWM, which in turn showed a direct link between VWM and saccade execution. 

The results from Experiment 1 demonstrated that VWM precision decreased with an increasing VWM load. A targeted eye-movement lead to a VWM performance reduction that was similar to storing an extra item. Although the degree of reduction of VWM precision due to an additional saccade seemed high, the result was not unprecedented. For example, previous research by Melcher and Piazza (2011) showed that reports of enumeration became significantly worse when performing a saccade task, similar to retaining an extra item in VWM. Although we described the loss of precision of a saccade as a cost compared to remembering an extra item, VWM does not seem to be a resource with discrete “item slots” that can be stored. Instead it has been suggested that VWM is a continuous resource that can be distributed over multiple objects (Bays et al., 2009; Ma, Husain, & Bays, 2014; Melcher & Piazza, 2011). We argue that under either model of VWM resources (i.e., discrete or continuous), our results would be possible. 

We also investigated to what extent the results in Experiment 1 were due to increased task-load in general. The results of Experiment 2 indicated that the costs of an eye movement for WM precision are specific to VWM. It does not affect Auditory Working Memory (AWM). These results were consistent with the traditional models of working memory (WM), in which VWM (visuospatial sketchpad) and AWM (phonological loop) are systems with separate capacities connected by a central executive (Baddeley, 1992; Baddeley & Hitch, 1974). Due to the separation of these capacities, it is generally easier to perform a multimodal dual-task, rather than a unimodal dual-task (Logie, Zucco, & Baddeley, 1990). Therefore, it seems that performing an eye movement and storing the presaccadic information of the saccade target taxes the VWM system, rather than the AWM system. 

The performance in the thresholding phase of Experiment 2 did not perfectly map onto the performance in the experimental phase of Experiment 2. The participants were overall better at the AWM than the VWM task. This was, however, not problematic for the conclusions drawn about the effects that executing a saccade has on working memory. In Experiment 1, an effect of performing a saccade on the accuracy of items held in VWM was observed, which did not vary with VWM load. Furthermore, we expected performance to be reduced by adding a secondary saccade task to the AWM task. Additionally, we chose an analysis method which was less at risk of interference due to ceiling effects, i.e., a log-linear model. Even if the AWM condition resulted in less overall WM load, an effect of executing saccades on AWM should have been observable, as it was in Experiment 1, and in the VWM condition of Experiment 2. Therefore, the difference in thresholding performance was not problematic for the conclusions drawn. 

While the effect of eye movements on VWM precision were clear and in line with a previous study (Hollingworth et al., 2008), the effect of VWM load on corrective saccade latency was less consistent. First of all, we did not find an effect of VWM load on corrective saccade latency or on corrective saccade accuracy, which has been found previously for both endogenously and exogenously driven saccades (Hollingworth & Luck, 2009; Stuyven, Van der Goten, Vandierendonck, Claeys, & Crevits, 2000). The only effect of task-load on eye movements that we observed was an effect of whether participants were performing a VWM task next to the corrective saccade task on corrective saccade latency and accuracy. These results may be explained by the visual system prioritizing presaccadically acquired information, as this information is utilized to maintain visual stability across saccades (Hollingworth et al., 2008; Zhao, Gersch, Schnitzer, Dosher, & Kowler, 2012). Moreover, there was no difference between single- and dual-task corrective saccade latency in Experiment 2, in contrast with the results of Experiment 1. This suggests that the effect of VWM load on corrective saccade latency was not very robust when compared to the inverse relation (the effect of saccades on VWM load). Whereas previous studies did observe a consistent effect of VWM load on corrective saccade latency (i.e., higher VWM load increased corrective saccade latency, Hollingworth & Luck, 2009; Hollingworth et al., 2008), we only found this effect in Experiment 1. 

Alternatively, the difference with previous studies may have arisen because in previous research, the features that were held in the VWM task were similar to, instead of different from, the items in the saccade task (i.e., remember a color and saccade to a colored object, Hollingworth & Luck, 2009). Even if the features did not directly interfere with each other due to similarity, both features at least occupied the same feature space. Research indicated that VWM capacity may be modulated by the similarity of items in feature space. For example, remembering a large amount of colors is more difficult than remembering objects with more diverse features, because of potential binding errors (Bays et al., 2009; Ester, Sprague, & Serences, 2015; Fougnie, Asplund, & Marois, 2010; Treisman & Gelade, 1980). By design, we opted to keep the features that had to be retained in the visual task (width-height-ratio of a gray object) as dissimilar as possible from the features in the saccade task (one of four colors of a circular object). The slight discrepancy between previous research and our current conclusions may be due to using unrelated features in the memory task and the saccade task. We suggested that the effect of VWM load (of items that are unrelated to the saccade task) on corrective saccade latency, is quite small at baseline, but may become more pronounced when the features of items in a saccade task and a VWM task overlap. 

Furthermore, spatial properties of memorized items have been shown to be important for the cost of a saccade. In a study by Williams, Pouget, Boucher, and Woodman (2013), observers were tasked with remembering the features of a stimulus. When unconstrained in their viewing behavior, participants executed more saccades to the location of the items they had to remember than to other, less relevant, locations. The authors hypothesized that visuospatial selection mechanisms aid in the maintenance of object representations. Additionally, the authors show that when attention is drawn away from the location of a remembered object by a distractor, participants report memorized items less accurately. Studies that dissociate the location of the saccade target and the location of remembered items showed that saccades away from the location of a memorized item resulted in worse memory performance (Ohl & Rolfs, 2016; Hanning, Jonikaitis, Deubel, & Szinte, 2015). Tas, Luck, & Hollingworth (2016) found that saccades to a secondary object during the retention interval of a visual working memory task, interfered significantly with the proportion of correct answers. In contrast, this effect was not present when the participants were instructed to make a free saccade (in a certain direction). The lack of a reduction in memory performance when making a saccade to a blank portion of the screen showed the necessity of an object needing to be present for VWM to be affected by a saccade, rather than the saccade itself affecting VWM. In the current study the saccade targets and working memory items were presented at different locations and contained nonoverlapping features. These divergent spatial properties likely resulted in a larger cost due to presaccadic attentional shifts away from the remembered location. 

As spatial properties were of great importance to performance on the visual working memory task, we stress that in the AWM task the perceived location of the stimulus was hard to define. For example, auditory information was typically localized towards a visual target (Bertelson & Radeau, 1981; L. Chen & Vroomen, 2013; Van der Stoep, Postma, & Nijboer, 2017). This could have caused participants to perceive the sound coming from the center, where the visual stimuli were presented as well. Alternatively, participants could have disregarded the spatial properties of the auditory stimulus entirely, as processing of the spatial location of the auditory stimuli was not emphasized in the current study and the location of the speaker was hidden from view. Two processes may have played a role in the current set of experiments: AWM was less affected by saccades due to the loading of VWM (rather than AWM) by saccades, and/or AWM was less affected by saccades due to a lack of spatial conflict between auditory working memory items and saccades (i.e., the storage of nonspatial features in the AWM task; McDonald & Ward, 1999). 

Storing a representation of the object in VWM prior to executing a saccade might serve several functions. First, based on the current literature on transsaccadic integration, we assume that the visual system is constantly storing presaccadic representations of visual input in VWM (Ganmor et al., 2015; Herwig, 2015; Oostwoud Wijdenes et al., 2015; Wolf & Schütz, 2015). The stored representations of presaccadic visual input are integrated with the postsaccadic foveal image. A possible function of integration is to compensate for the reduction of visual processing during a saccade and bridge the delay between pre- and postsaccadic visual input (Herwig et al., 2015; Kok, Brouwer, van Gerven, & de Lange, 2013; Miall & Wolpert, 1996). Secondly, stored presaccadic visual input, when integrated with the postsaccadic foveal image, increases the precision of the representation of the objects (similar to cue-integration processes; Ernst & Banks, 2002; Fetsch, DeAngelis, & Angelaki, 2013; Ganmor et al., 2015). Presumably, when a discrepancy (of a large magnitude) is detected between stored presaccadic visual input and a postsaccadic foveal image, the stored presaccadic and postsaccadic visual input were not integrated (Poth, Herwig, & Schneider, 2015; Poth & Schneider, 2016; Schneider, 2013). Furthermore, in case a mismatch is perceived between the stored visual input and foveated postsaccadic image, the information stored in VWM could guide corrective saccades (similar to the concept of a forward model; Kok et al., 2013; Miall & Wolpert, 1996; Tian et al., 2013; Webb, 2004). 

Forward models contain the assumption that motor systems predict (before a movement is initiated) the state of the system after the movement. However, it is currently unclear whether the presaccadic visual input is stored or that a prediction of postsaccadic visual input is made and stored prior to the saccade. Several studies on transsaccadic learning have, by inducing changes to saccade targets during the saccade, shown that observers could be taught to associate different presaccadic images with postsaccadic visual input (Bosco et al., 2015; Herwig & Schneider, 2014; Herwig et al., 2015; Valsecchi & Gegenfurtner, 2016). These studies provide evidence for the account that what was stored in VWM is, at least partly, a prediction of upcoming retinal input. In either case, whether visual information was presaccadically stored, or both stored and influenced by prediction, our study provided evidence that these presaccadic processes tax the VWM system. We proposed that VWM was, amongst others, a buffer system, directly linked to saccade execution. VWM can be used to bridge the lack of visual processing during a saccade and may guide corrective saccade by retrieving the state of the world prior to the initial saccade, promoting visual stability. 

In conclusion, this study shows that saccades are tightly coupled to the allocation of items to VWM. The visual system must consistently commit predictions of upcoming visual input to a temporary buffer, which can retroactively be compared with the postsaccadic visual input. Our results demonstrate that the cost of predicting of upcoming visual input was quite high, or at least similar to items committed volitionally to VWM. The cost of executing an eye movement on VWM in this experiment most likely reflects an extreme situation in which the cost of an eye movement is high. The visual environment within the task was highly mutable (i.e., it changed 66% of the time during an eye movement), and the stimuli were shown at different locations throughout a trial, requiring saccades away from memorized items. The current findings indicate that the visual system can flexibly allocate the resources of VWM for either saccade tasks or actively committing items to working memory, and that making an eye-movement compulsorily and exclusively uses these visual resources. 

Acknowledgments

We thank Veronique Van der Kleij for proofreading the manuscript and correcting typographical errors. This research was funded by a VIDI Grant 452-13-008 from the Netherlands Organization for Scientific Research to Stefan Van der Stigchel. 

Commercial relationships: none. 

Corresponding author: Martijn J. Schut. 

Email: . 

Address: Experimental Psychology Helmholtz Institute Heidelberglaan, Utrecht, The Netherlands.