Transsaccadic memory of multiple spatially variant and invariant object features

Abstract

Transsaccadic memory is a process by which remembered object information is updated across a saccade. To date, studies on transsaccadic memory have used simple stimuli—that is, a single dot or feature of an object. It remains unknown how transsaccadic memory occurs for more realistic, complex objects with multiple features. An object's location is a central feature for transsaccadic updating, as it is spatially variant, but other features such as size are spatially invariant. How these spatially variant and invariant features of an object are remembered and updated across saccades is not well understood. Here we tested how well 14 participants remembered either three different features together (location, orientation, and size) or a single feature at a time of a bar either while fixating either with or without an intervening saccade. We found that the intervening saccade influenced memory of all three features, with consistent biases of the remembered location, orientation, and size that were dependent on the direction of the saccade. These biases were similar whether participants remembered either a single feature or multiple features and were not observed with increased memory load (single vs. multiple features during fixation trials), confirming that these effects were specific to the saccade updating mechanisms. We conclude that multiple features of an object are updated together across eye movements, supporting the notion that spatially invariant features of an object are bound to their location in memory.

Introduction

When we look around, we experience the visual scene as continuous. However, we receive a limited amount of information from a single fixation, necessitating multiple saccades to gather information from the scene. Given the disjointed nature of saccades, we should expect to experience snapshots of the visual scene—in contrast to our actual experience (Hollingworth & Henderson, 1998; Mackay, 1973). Visual constancy is thought to be achieved through the formation of an internal representation that is preserved across saccades through mechanisms involving visual memory (Higgins & Rayner, 2015; Prime, Tsotsos, Keith, & Crawford, 2007). To build an internal representation, information about viewed objects at each fixation in the visual scene must be represented in memory and then updated across each saccade, which is referred to as transsaccadic memory (Germeys, De Graef, & Verfaillie, 2002; Henderson & Hollingworth, 1999; Henderson & Siefert, 1999; Irwin, 1991, 1992; Irwin & Andrews, 1996; Medendorp, 2011; Melcher & Colby, 2008; Palmer & Ames, 1992; Pollatsek, Rayner, & Collins, 1984; Prime et al., 2007; Prime, Niemeier, & Crawford, 2006). 

Studies have demonstrated that transsaccadic memory is similar to visual short-term memory (VSTM) and may involve the same mechanisms (Alvarez & Cavanagh, 2004; Bays & Husain, 2008; Cowan, 2011; Cowan & Rouder, 2009; Donkin, Nosofsky, Gold, & Shiffrin, 2013; Luck & Vogel, 1997; Pashler, 1988; Zhang & Luck, 2008). For example, Prime et al. (2007) have demonstrated that the limit of the number of objects remembered across saccades was similar to that of VSTM during fixation. Presumably, remembering multiple features of the same object rather than multiple objects should also show similar limits for transsaccadic memory as have been shown for VSTM (Bays, Wu, & Husain, 2011; Fougnie, Asplund, & Marois, 2010; Oberauer & Eichenberger, 2013; Wheeler & Treisman, 2002). Indeed, it has been shown that people can distinguish remembered objects with multiple features across saccades (Carlson, Covell, & Warapius, 2001; Irwin, 1998; Irwin & Andrews, 1996); however, to our knowledge no study has yet characterized how saccades influence this memory (e.g., precision, bias). It has been suggested that VSTM for different features of an object is stored in independent memory stores (Allen, Baddeley, & Hitch, 2006; Bays et al., 2011; Pasternak & Greenlee, 2005; Robertson, 2003; Treisman, 1998; Treisman & Schmidt, 1982; Wheeler & Treisman, 2002; Wolfe & Cave, 1999) but nevertheless may be bound by location (Cave & Wolfe, 1990; McCollough, 1965; Reynolds & Desimone, 1999; Schneegans & Bays, 2017; Theeuwes, Kramer, & Irwin, 2011; Treisman, 1998; Treisman & Gelade, 1980; Wolfe & Cave, 1999). 

Once a saccade is made, this remembered information must be updated to account for the saccade. It is important to note that updating is a spatial process, and thus the saccade requires updating the location of objects. Some studies have directly measured the updating of object location by measuring the remembered locations of objects across saccades. Hayhoe, Lachter, and Feldman (1991) showed that we are able to integrate spatial arrangements across eye movements to determine shapes from dots. Prime et al. (2006) showed that we are just as good at determining the intersection point of two lines across saccades as we are during fixation. Golomb & Kanwisher (2012) demonstrated that the remembered location of an object was better when the retinotopic location was recalled rather than the spatiotopic location, but in both cases the remembered location of the object was relatively accurate and precise. Generally, these studies suggest that we are able to accurately update locations of objects (dots or even oriented lines) across saccades. Other studies have provided indirect evidence of updating in transsaccadic memory by showing perceptual integration effects on spatially invariant features—for example, color—at the same predefined spatial location across saccades (Irwin, 1992; Melcher, 2005; Melcher & Morrone, 2003; Oostwoud Wijdenes, Marshall, & Bays, 2015). Consistent with these findings, it has been hypothesized that transsaccadic integration involves the remapping of not just object locations but also object features (Cavanagh, Hunt, Afraz, & Rolfs, 2010; Melcher & Colby, 2008; Prime et al., 2006; Prime, Vesia, & Crawford, 2011). 

While these studies on transsaccadic integration have provided insight into the mechanisms of transsaccadic memory and updating, they have tended to investigate only how single features of objects are remembered across saccades—for example, just location or just orientation (but see Prime et al., 2006). What remains unclear is whether and how complex objects with multiple feature conjunctions and their locations are remembered and updated across saccades. Since objects in the world tend to have multiple spatially invariant features (e.g., shape) besides location, if the object's remembered location is updated during saccades, are its other spatially invariant remembered features also updated or influenced by the saccade? If saccades affect memory of spatially invariant object features, this would imply that the location is tightly bound to these features in memory (Johnson, Verfaellie, & Dunlosky, 2008; Matthey, Bays, & Dayan, 2015; Melcher & Colby, 2008; Prime et al., 2011; Reynolds & Desimone, 1999; Schneegans & Bays, 2016, 2017; Wheeler & Treisman, 2002). Alternatively, a saccade might influence only the remembered location. This would imply that the memories of the object's spatially invariant features are loosely linked to or even independent of its location (Robertson, 2003). 

To gain insight into this question, we used a psychometric-testing paradigm to quantify how multiple features of an object (location, orientation, and size) are remembered across saccades. We tested how well participants remembered the conjunction of features or a single feature of an object with or without an intervening saccade. These data have been previously published in abstract form (Khan et al., 2013). 

Methods

Participants

Fifteen participants took part in this experiment (eight men, seven women), 11 of whom were unaware of the purpose of the experiment. Their ages ranged from 21 to 36 years (M = 23.6, SD = 3.8). All participants had normal or corrected-to-normal vision and provided written consent to take part in the experiment, which was preapproved by the Health Sciences Research Ethics Board at Queen's University, Kingston, Canada. 

Apparatus

Participants performed the experiment in a completely darkened room. A 20-in. Mitsubishi DiamondPro 2070 CRT stood at a viewing distance of 30 cm (16 × 12 in., 2,048 × 1,536 pixels, refresh rate: 86 Hz). Participants' heads were stabilized using a head and chin rest as part of an EyeLink tower-mount setup (SR Research, Mississauga, Canada). A setting of low contrast (25%) and brightness (0%) values was used to prevent the participants from using visual cues from the monitor frame to perform the task; the monitor edge was not visible to the participants. Eye movements from the right eye were recorded using an EyeLink 1000 video-based recording system (SR Research) at 1000 Hz. Stimuli were displayed using Experiment Builder (SR Research). Responses were keyed using an SR Research Gamepad with two buttons (left and right). Prior to each block of trials, the eye tracker was calibrated by having the participant fixate a series of nine positions on the display (the center and eight positions surrounding the periphery of the display). 

Procedure

Each trial in the experiment consisted of two bars presented in sequential order. The participant's task was to memorize the features of the first bar—location, orientation, and size—and indicate how the second bar differed from the first. Figure 1 depicts the sequence of stimuli during a trial. Each trial began with a red dot (0.5° diameter) appearing at one of two locations (9° right or 9° left from center screen, aligned horizontally) on a black background. After 750 ms, a red bar appeared for 750 ms (the first bar). The first bar's features (location, orientation, and size) randomly varied from trial to trial, with a range of 1° left to 1° right relative to screen center in increments of 0.4° for location, 5° counterclockwise to 5° clockwise in increments of 2° for orientation, and 1.8° to 2.3° in increments of 0.1° for size. Next, the first bar was extinguished and the first dot remained on for a further 300 ms. The second dot then appeared for 900 ms at either the same location as the first dot (fixation trial) or a second location, requiring an eye movement (saccade trial). There were an equal number of fixation and saccade trials, and they were presented in randomized order. A second bar (also red) then appeared for 750 ms. The second bar's features (location, orientation, and size) also randomly varied from trial to trial, with the same parameter set as the first bar. A blank then appeared for 100 ms and was followed by a question screen. Figure 1 depicts a trial in the single-feature location condition, with the question “Was the second bar to the left or right of the first bar?,” which required the participant to indicate how the second bar differed in location from the first, thus providing an estimation of the remembered features of the first bar. The question was presented in red text on a black background centered on the screen. The participant answered the question by pressing one of two buttons on a game controller (left and right, corresponding to the first and second options in the question). The question remained on the screen until the participant pressed the button. There was an intertrial interval of 1,000 ms before the next trial began, during which a blank screen was displayed. 

Figure 1

 

Task stimuli and sequence for the single-feature location experiment. Each trial began with the presentation of a red fixation dot at one of two locations (9° right or left from center) on a black background for 750 ms. Next, a bar appeared and remained illuminated for 750 ms. After the bar was extinguished, the fixation dot remained illuminated for an additional 300 ms. A second fixation dot was then presented at either the same location as the first fixation (fixation trial, not shown) or at the second location, requiring an eye movement (saccade trial, shown). After 900 ms, the second bar was presented for 750 ms. The first and second bars varied randomly in location (−1° to 1° from center screen, in 0.4° intervals horizontally), orientation (5° counterclockwise to 5° clockwise from vertical, in 2° intervals), and size (1.8° to 2.3° in length, in 0.1° intervals). The fixation and second bar were then replaced by a blank screen for 100 ms followed by a question screen: “Was the second bar to the left or right of the first bar?” (Arial, font size 30, not shown to size). The question remained visible until the participant entered a response on the gamepad.

Besides the single-feature location condition, participants performed a single-feature orientation condition, with the question “Was the second bar rotated clockwise or counterclockwise compared to the first bar?,” and a single-feature size condition with the question “Was the second bar shorter or longer than the first bar?” In addition, participants performed a multiple-feature condition, in which they were asked to remember all three features together. Participants were asked the three questions in sequence. Each new question appeared immediately after the participant had pressed the button to answer the previous one. The presentation order of the questions was counterbalanced from block to block (six blocks total) in order to equalize possible memory effects. 

There were 96 trials in each block. Each participant performed 24 blocks (six location-feature only, six orientation-feature only, six size-feature only, six multiple-feature) of 96 trials each (2,304 trials total). Participants performed all blocks in random order over the course of a few weeks, approximately 1–3 blocks/day. 

Data analysis

We collected a total of 34,560 trials. Saccade timing and position were automatically calculated offline using a saccade-detection algorithm with a velocity criterion of 50°/s, and verified visually. Trials where the tracker lost eye position, where participants made a saccade or a blink, or where there was an incorrect fixation (>3° radius from fixation dot) during the time when the bars were displayed were removed from the data set, leaving 28,277 trials (82% of total trials). For each feature, we calculated the difference between the first and second bars, resulting in a range of −2° to 2° difference in location in 0.4° intervals, a range of −10° to 10° difference in orientation in 2° intervals, and a range of −0.5° to 0.5° difference in size in 0.1° intervals. Due to this difference calculation, the number of trials for the extreme ends of ranges were very small, and so we collapsed across the outer two intervals—for example, for location the −2° and −1.6° intervals were collapsed to be −1.8°. 

We fitted each participant's responses in each condition with psychometric functions using psignifit 3.0 toolbox with the Bayesian Inference fitting procedure (Frund, Haenel, & Wichmann, 2011) to estimate the parameters of the psychometric fit using MATLAB (MathWorks, Natick, MA). Psychometric functions were built with a Gaussian sigmoid due to its symmetry, which matched the expected properties of the task. No priors were imposed for either the mean or slope of the psychometric function. The priors of upper and lower thresholds were set to 0, as we did not expect lapse rates during the task, which is confirmed by the participants' control data (e.g., Figure 2A). We then extracted the point of subjective equality (PSE) and just-noticeable difference (JND) from each fit. The PSE was determined as the degree of stimulus change needed for the observer to have 0.5 choice probability. The JND is a measure of the degree of stimulus change required in order for an observer to report a congruent change. The JND is computed as half the difference between the stimulus changes needed to elicit 0.25 and 0.75 choice probabilities. We removed Participant 2's data from further analysis because that participant performed the task incorrectly for unknown reasons; the psychometric functions did not follow the same pattern as for all other participants. We performed repeated-measures ANOVAs and Bonferroni–Holmes-corrected paired t tests on the PSEs and JNDs to compare across saccade versus fixation trials, left versus right eye positions, and single- versus multiple-feature conditions. 

Figure 2

 

Location-feature psychometric functions. Psychometric functions are shown for the single-feature location condition for a single participant (A) and across all participants (B), as well as for the multiple-feature condition (C–D). Within each graph, the mean proportions (dots) and fitted psychometric functions (lines) are shown for each of the four trial types; left fixation (black solid line and circles), right fixation (gray solid line and circles), left saccade (black dashed line and open circles), and right saccade (gray dashed line and open circles).

Results

Location-feature memory is influenced by saccades

We investigated the effects of an intervening saccade on feature memory. In order to quantitatively assess performance, participant data were fitted with a psychometric curve. The psychometric functions of both the fixation and saccade conditions for the left and right eye positions are shown for Participant 1 (Figure 2A) as well as across all participants (Figure 2B) for the single-feature location condition. The PSE value indicates when the location of the second bar is perceived to be at the same remembered location as the first bar and thus reflects where the participant remembers the first bar to be. For example, a leftward shift in the PSE indicates that the first bar was remembered to be slightly to the left of where it actually was for this participant. The JND provides information on the degree to which an observer can distinguish between differences in stimuli; smaller numbers indicate a better ability to distinguish between changes in stimuli, or less uncertainty. Thus, a larger JND means less precision in the memory of the first bar. We performed two-way repeated-measures ANOVAs with final eye position (left or right) and condition (saccade or fixation) as factors separately for PSE and JND. In terms of saccade trials, we chose final eye position as opposed to initial eye position arbitrarily. For PSE, we found a significant main effect of eye position, F(1, 13) = 5.2, p < 0.05, and a significant interaction effect, F(1, 13) = 12.3, p < 0.01, but no significant condition effect (p > 0.05). Bonferroni–Holm post hoc paired t tests showed no significant effect of eye position in the fixation condition (p = 0.58), but there was a significant difference for the saccade condition, t(13) = 3.1, p = 0.009. Leftward saccades (to the left eye position) increased the tendency of participants to remember the first bar as being more right that it was (mean PSE = 0.66°), whereas rightward saccades resulted in the opposite tendency (PSE = −0.9°; Figure 3B). 

Figure 3

 

Orientation-feature psychometric functions. Psychometric functions are shown for the single orientation (A) and multiple-feature (B) conditions across all participants. The x-axis shows the relative orientation of the first and second bars. Data are plotted in the same manner in as Figure 2.

For JND, there were a significant main effect of condition, F(1, 13) = 51, p < 0.001, but no significant main effect of eye position (p > 0.05) nor any interaction effect (p > 0.05). This analysis shows that the JND was significantly larger in the saccade condition (mean JND = 1.014) compared to the fixation condition (0.62), suggesting a degradation of memory of location with saccades. Location memory was not worse with one eye position (left vs. right) compared to the other, which is not surprising as there is no reason to presume that object locations in one visual field are remembered less well than in the other for healthy participants. In summary, saccades induced biases in the remembered locations of the objects and degraded memory of their locations. 

We also tested whether saccades had the same effect on location when the participant was under increased memory load (Participant 1: Figure 2C; all participants: Figure 2D)—that is, in the multiple-feature condition in which participants had to remember all three features within a given trial. Similar to the location-only condition, there was a significant main effect of PSE for eye position, F(1, 13) = 13.9, p < 0.01, and a significant interaction effect, F(1, 13) = 26, p < 0.001, but no significant effect of condition (p > 0.05). Post hoc testing revealed a significant difference between eye position in the saccade condition once again, t(13) = 4.4, p = 0.001, mean left saccade PSE = 1.87°, mean right saccade PSE = −1.74°, but not in the fixation condition (p = 0.085). Similarly, there was a significant condition effect on JND, F(1, 13) = 49.5, p < 0.001, mean fixation JND = 0.889, mean saccade JND =1.695, but no eye-position effect (p > 0.05) nor any interaction effect (p > 0.05). Thus, increasing memory load did not change the effect saccades had on remembering the location of the bar. Saccades still resulted in a memory degradation as well as a bias with respect to eye position, which was exaggerated with increased memory load. 

Orientation-feature memory is influenced by saccades

Next, we investigated the effect of saccades on spatially invariant features. We first examined whether orientation memory was affected by saccades in the single-feature orientation condition (Figure 3A, all participants). 

The intervening saccade between stimulus presentations did not significantly change the PSE (p > 0.05) of the psychometric curve compared to the fixation trials. However, there was a significant effect of eye position, F(1, 13) = 7.5, p < 0.05, across both fixation and saccade trials, where participants remembered the bar to be oriented more clockwise at the (final) left eye position (mean PSE = 0.47°) and more counterclockwise for the (final) right eye position (mean PSE = −0.16°). There was no interaction effect (p > 0.05). 

There was a significant increase in JND during saccade trials (mean JND = 2.2) compared to fixation trials (mean JND = 1.5), F(1, 13) = 44, p < 0.001, but no eye-position effect (p > 0.05) or interaction effect (p > 0.05). 

When orientation had to be recalled in conjunction with other features (Figure 3B), we again found no main effect on PSE of saccades compared to fixation (p > 0.05), nor a significant effect of eye position (p > 0.05); however, there was a significant interaction effect, F(1, 13) = 5.2, p < 0.05. Post hoc testing revealed significant differences in the PSE between left (mean PSE = 0.42°) and right eye positions (mean PSE = −0.31°) for the saccade trials, t(13) = 2.97, p < 0.05, but not fixation trials (p > 0.05). Thus, as in the feature-only condition, there was a tendency to remember the bar to be oriented more clockwise for the left eye position compared to the right eye position, but in the increased-memory-load condition this effect held only for saccade trials. 

For JND, like in the single-feature condition, we only found a significant main effect of condition, with larger JNDs for the saccade condition (mean JND = 2.85) compared to the fixation condition (mean JND = 2.35), F(1, 13) = 11.1, p < 0.01. 

To summarize, in both the single- and multiple-feature paradigms, saccades biased the remembered orientation of the first bar according to the direction of the saccade. This was also the case for the single-feature fixation condition. In addition, saccades increased the uncertainty about the remembered orientation compared to fixation. 

Size-feature memory is influenced by saccades

We investigated the impact of saccades on memory of another spatially invariant object feature, size (Figure 4A). When participants were asked to remember just the size of the first bar, there was a significant effect of condition on PSE, F(1, 13) = 37.2, p < 0.01, where the first bar was remembered to be smaller during saccade trials (−0.057°) than during fixation trials (0.05°). There was no significant effect of eye position (p > 0.05), but there was a significant interaction effect, F(1, 13) = 4.8, p < 0.05. Post hoc testing revealed no differences between left and right fixation trials (p > 0.05), whereas there was a significant but small difference between leftward (−0.12°) and rightward (0.01°) saccade trials, t(13) = 2.3, p < 0.05. 

Figure 4

 

Size-feature psychometric functions. Psychometric functions are shown for the single size (A) and multiple-feature (B) conditions across all participants. The x-axis shows the relative size of the first and second bars. Data are plotted in the same manner as in Figure 2.

For JND in the single-feature size condition, there was a significant effect of condition, F(1, 13) = 5, p < 0.05 (fixation = 0.194, saccade = 0.238), as well as a significant eye-position effect, F(1, 13) = 7.8, p < 0.05 (leftward = 0.23°, rightward = 0.19°), but no significant interaction effect (p > 0.05). 

When size had to be recalled in conjunction with other features, we again found an effect of condition on PSE, F(1, 13) = 6.7, p < 0.05. Similar to the single-feature condition, the size of the bar was remembered to be smaller during saccade trials (−0.032°) compared to during fixation trials (0.077°). Although the pattern in saccade trials in the multiple-features size condition was similar to that in the single-feature size condition (Figure 4), we did not find a significant eye-position effect (p > 0.05) or interaction effect (p > 0.05). 

For JND in the multiple-features condition, there was no significant effect of condition (p > 0.05) and no interaction effect (p > 0.05), but there was a significant eye-position effect, F(1, 13) = 4.8, p < 0.05 (leftward = 0.36°, rightward = 0.26°). 

For both the single- and multiple-feature conditions, the size of the first bar was remembered to be smaller during saccade trials compared to fixation trials. In addition, in the single-feature condition only, the bar was remembered to be smaller during leftward final fixations compared to rightward ones. We also found an increase in the uncertainty of the first bar's size, but only during the single-feature condition. Finally, there was an increase in uncertainty for leftward compared to rightward eye positions in both single- and multiple-feature conditions. To summarize the results so far, we found that saccades changed the PSEs for both location and size. In addition, these effects seem to be independent of increasing memory load (one vs. three features). Saccades did not induce a significant change in PSE in orientation, however, an interaction effect of saccade direction was observed with greater memory load. These results demonstrate that the bar's remembered location as well as its spatially invariant features (orientation and size) were influenced across saccades. 

We also found that saccades increased the memory uncertainty of each of the features of the first bar in general in both the single- and the multiple-feature condition. It is possible that remembering information while performing an intervening saccade may induce changes in memory in a similar manner as increasing memory load. In other words, the intervening saccade might simply increase memory load because it is an additional task. Thus, it is this increase in memory load that results in the changes in the PSEs observed rather than their being due to the saccade remapping mechanisms. 

To test this, we compared the PSEs and JNDs of the single-feature to the multiple-feature conditions for fixation trials. If the saccade only increases memory load, then we should see similar patterns of changes when the memory load is increased without saccades as when a saccade is performed. 

Feature memory is degraded but not biased by increasing memory load

We used two-way repeated-measures ANOVAs with number of features recalled (1 vs. 3; e.g., Figure 2B vs. 2D) and eye position as factors only for fixation trials. 

We found that increasing memory load did not significantly change the PSE for any of the three features (all ps > 0.05), nor did eye position have an effect (p > 0.05), nor were there any interaction effects (p > 0.05). 

For JND, we found main effects of number of features recalled for location, F(1, 13) = 14.8, p < 0.01 (single = 0.62°, multiple = 0.89°), and orientation, F(1, 13) = 54.8, p < 0.001 (single = 1.5°, multiple = 2.4°), but no effect of eye position (p > 0.05) and no interaction effects (p > 0.05) for either feature. For size, we found no effect of number (p > 0.05) and no interaction effect (p > 0.05), but there was a small but significant eye-position effect, F(1, 13) = 6.7, p < 0.05 (leftward = 0.27°, rightward = 0.21°), as was also found earlier. 

The null effects on PSE with increased memory load suggest that the changes seen in the PSEs during saccade trials are due to updating mechanisms rather than memory load. However, we found increases in the JND both during saccade trials and under increased memory load during fixation trials (at least for location and orientation). These findings suggest that in addition to updating effects, saccades also increase memory load. 

Order of recall questions

Finally, we also tested whether the order of the questions had any effect on memory when all three features were to be remembered at the same time. We used one-way repeated-measures ANOVAs separately for each feature with order as a factor (1 vs. 2 vs. 3) collapsed across condition (saccades vs. fixation). We found no effect of order on PSE (Figure 5A–5C) for any of the features (p > 0.05). For JND (Figure 5D–5F), we found a significant effect of order for location, F(2, 26) = 4.8, p < 0.05. Post hoc analysis revealed that location memory was significantly worse when the location feature question was asked last as opposed to first (p < 0.05), but there were no other significant differences. There was no effect of order on either orientation (p > 0.05) or size (p > 0.05). In summary, the order of questions had no effect on the PSE and had a small effect of increasing uncertainty of the recall of the location of the bar. 

Figure 5

 

Question order. The point of subjective equality (A–C) and just-noticeable difference (D–F) are shown for the multiple-feature condition, separated by location (A, D), orientation (B, E), and size (C, F), separated by the order of the questions—for example, in (A), 1 signifies that the location question was first, 2 signifies that the location question was second (with either size or orientation being first), etc. The bars depict the mean point of subjective equality and just-noticeable difference, and the error bars are the standard error of the mean across participants. ^*p < 0.05.

Discussion

Saccades biased the remembered location, orientation, and size of the bar dependent on the direction of the saccade. Overall, these saccade-induced biases remained similar even when we increased memory load. We propose that saccadic updating mechanisms influence both spatially variant and invariant remembered features of an object, suggesting a link between object location and features in memory. In addition, saccades increased uncertainty in the perceptual report, possibly due to contributions from saccade-induced memory load or extraretinal noise. 

Saccades induce biases in the remembered location of an object

Saccades were observed to induce biases in all three remembered features, both spatially variant and invariant. The largest effect of the intervening saccade was an overestimation of the remembered bar's location away from the direction of the saccade. Similar overestimations of target location have been observed during tasks requiring actions, such as pointing (Henriques, Klier, Smith, Lowy, & Crawford, 1998). Therefore, it is possible that the observed overshooting effects could be a result of inaccurate updating from an overcompensation of the saccade amplitude. However, this hypothesis is unlikely, as many studies have observed similar biases with perceptual reports and attributed them to visual-space compression toward the fixation point at the time of encoding (Golomb & Kanwisher, 2012; Sheth & Shimojo, 2001; van der Heijden, van der Geest, de Leeuw, Krikke, & Müsseler, 1999). In agreement with Golomb and Kanwisher (2012), our results support the idea that remembered locations are updated accurately across saccades, with discrepancies being attributed to the encoding process rather than the update. 

Nonspatial features of the object are also influenced by the intervening saccade

We found that the intervening saccade also influenced both the orientation and the size of the object. The effect on remembered orientation was observed under the higher-memory-load condition, with the bar being remembered more clockwise for leftward saccades and more counterclockwise for rightward saccades. In addition, the size of the bar was consistently remembered to be smaller when a saccade intervened before recall. 

To confirm that the abovementioned effects were related to updating rather than attributable to increased memory load, we compared single-feature to multiple-feature trials during fixation. We were unable to replicate any of the biases observed during the saccade trials when memory load was increased for fixation trials. Based on this, we conclude that the effects observed were specific to the intervening saccade, implicating updating mechanisms. 

These changes in the perceived orientation and size of the bar due to the saccade are remarkable because they imply that updating mechanisms distort the remembered features of the object even when those features are spatially invariant. In a study on transsaccadic memory, Prime et al. (2007) observed increased errors in remembering the luminance or orientation of targets as the size of the saccade increased. Our results thus point toward the idea that nonspatial features of an object are influenced by the intervening saccade and thus must be bound or linked together with their location in some manner (Kahneman, Treisman, & Gibbs, 1992; Nissen, 1985; Schneegans & Bays, 2017; Serences & Yantis, 2006; Treisman, 1996), consistent with the hypothesis that transsaccadic integration also involves the remapping of object features (Cavanagh et al., 2010; Melcher & Colby, 2008; Prime et al., 2006; Prime et al., 2011). It has been suggested that VSTM of different object features is stored in independent memory stores (Bays et al., 2011; Pasternak & Greenlee, 2005; Wheeler & Treisman, 2002), based on observations of misbinding, where features of an object are remembered to be part of another object (Allen et al., 2006; Robertson, 2003; Treisman, 1998; Treisman & Schmidt, 1982; Wheeler & Treisman, 2002; Wolfe & Cave, 1999). Our results do not contradict these findings, as we did not test multiple objects; however, they do imply a tight linkage in the memory of the different features of the same object with their location, since all features were biased by the eye movement. This tight linkage to location makes sense, as binding is successfully performed the majority of the time and misbinding occurs only under forced experimental conditions, and even then not all the time (Treisman, 1977), or in the case of brain lesions (Cohen & Rafal, 1991; Humphreys, Cinel, Wolfe, Olson, & Klempen, 2000; Robertson, 2003). Our results are in line with studies proposing either that information from the same location is implicitly bound together—resulting in adaptation aftereffects specific to congruently presented features, as in the McCollough effect (McCollough, 1965; Wolfe & Cave, 1999)—or that spatial attention is used to bind features of an object to its location (Cave & Wolfe, 1990; Reynolds & Desimone, 1999; Theeuwes et al., 2011; Treisman, 1998; Treisman & Gelade, 1980). Even the notion of independent memory stores of individual features does not preclude them from being linked through location. After all, the brain somehow needs to piece together feature information computed in different brain areas to know which visual objects have what specific features (see later). In agreement with this view, Schneegans and Bays (2017) have recently suggested that nonspatial features are bound to the object's location yet are independent of one another. This is consistent with our findings that the intervening saccade influences nonspatial features due to being linked to the updated location of the object. 

Specific biases on spatially invariant features

We observed specific but small biases in the remembered orientation and size after saccades. For orientation, participants remembered the top of the bar to be oriented toward the final fixation position after the saccade. The size of the bar was remembered to be smaller during saccade trials than during fixation trials, and for leftward versus rightward saccades. We confirmed overall that there were no differences in the other features that could explain observed differences in each of the features—for example, the average location of bars for leftward saccades was different from rightward saccades. These biases cannot be explained by noise induced by a saccade, as this would cause not directional biases but rather an increase in uncertainty (discussed later). An alternative explanation could be that the observed biases are related to the remembered location of the object, in that it was remembered to be farther away from fixation after a saccade. For example, it has been recently demonstrated that an object is perceived to be smaller the farther away it is in the periphery (Baldwin, Burleigh, Pepperell, & Ruta, 2016). Therefore, observed bias in size reports can be explained by a peripheral mislocalization of the first object farther toward the periphery. However, we do not believe this is the case, because the overestimation of location was relatively equal for location (approximately 2° in either direction), whereas the biases for orientation and size were different for leftward versus rightward saccades (e.g., −0.12° leftward vs. 0.01° rightward for size). Moreover, even if this were the case, it would still imply that object features were linked and updated across saccades, as the updated object location influenced the remembered size. In sum, regardless of the specific bias, the emergence of a bias specific to saccade trials supports the notion that the features of the object are linked and updated together across saccades. 

Saccades increase memory load

Apart from the biases induced due to the saccade, we also found that saccades increased the JNDs or uncertainty in a similar manner as increasing memory load during fixation trials. It has been suggested that planning and executing saccades may increase memory load (Bays & Husain, 2008; Shao et al., 2010). For example, Tas, Luck, and Hollingworth (2016) have demonstrated that saccades resulted in reduced memory capacity for task-relevant objects, presumably due to automatic encoding of the task-irrelevant saccade target into memory. However, the increased uncertainty we observed may not necessarily or exclusively be due to increased memory load. If memory representations are stored retinotopically and updated during a saccade, a noisy integration of extraretinal information may also contribute to uncertainty (Baker, 2003; Golomb & Kanwisher, 2012; Henriques et al., 1998; Prime et al., 2007). Whether both these factors lead to the observed increase in uncertainty may be dependent on whether the mode of storage utilized by VSTM in this context is continuous (shared) or discrete. VSTM has been proposed to comprise either a limited number (generally four) of discrete memory slots (Cowan, 2011; Luck & Vogel, 1997; Pashler, 1988), continuous but shared memory resources rather than discrete slots (Alvarez & Cavanagh, 2004; Bays & Husain, 2008; Wilken & Ma, 2004), or a hybrid of the two (Cowan & Rouder, 2009; Donkin et al., 2013; Zhang & Luck, 2008). With shared representations, both updating and memory load (Alvarez & Cavanagh, 2004; Fougnie et al., 2010; Oberauer & Eichenberger, 2013; Wheeler & Treisman, 2002) may explain increased uncertainty, whereas with the slot hypothesis increases in uncertainty should be exclusively due to updating noise, since memory is not saturated (Bays & Husain, 2008; Prime et al., 2007). 

Our findings from the fixation trials, however, support a continuous shared model of memory resources; we found that JNDs increased when three features were remembered compared to one feature. Previous studies have shown that increasing the number of remembered features of the same object shows similar limits as increasing the number of objects (Alvarez & Cavanagh, 2004; Bays et al., 2011; Fougnie et al., 2010; Oberauer & Eichenberger, 2013; Wheeler & Treisman, 2002). For example, two studies have shown that increasing memory load decreases performance for complex objects (Alvarez & Cavanagh, 2004; Olsson & Poom, 2005). Along these lines, Bays et al. (2011) have suggested that like objects, the memory of multiple features within an object also degrades with increasing memory load for visual working memory. In contrast, supporters of the discrete-slot hypothesis propose a feature cost-free integrated object account of memory, where remembering multiple features does not add memory cost (Luck & Vogel, 1997). Considering this, the observed effect of saccades on JND is likely due to a combination of both noisy updating and shared memory allocation. 

Eye-centered representation of remembered object location

The overestimation of the target's location after a saccade has been demonstrated to reflect an eye-centered representation of location (Golomb & Kanwisher, 2012; Henriques et al., 1998; Prime et al., 2007). An eye-centered reference frame for transsaccadic memory has also been demonstrated by others (Hayhoe et al., 1991; Prime et al., 2007; Verfaillie, 1997). We asked our participants to perform the task in complete darkness, thus ensuring that no allocentric references were available across saccades (such as the monitor frame). We biased the task such that updating in our task would likely take place in eye-centered coordinates. Complete darkness necessitated that eye position be used to estimate the location of the bar. However, we note that during the fixation condition, the persistence of the fixation dot may provide an allocentric cue, thus inflating location performance. Could the observed cost of saccades be attributed to the loss of allocentric information rather than saccadic updating? Golomb and Kanwisher (2012) observed that spatial-report performance decreased with additional saccades. If the cost incurred by saccades were solely due to loss of an allocentric cue, we should not expect to see additional costs with more than one saccade. Our findings are consistent with the proposition that the observed cost in memory performance is due to saccade updating; it cannot solely be attributed to loss of an allocentric cue. Thus, our manipulation revealed biases in the remembered nonspatial features of the bar and supports the idea that object locations are updated for perception. 

Underlying neurological mechanisms

The influence of the intervening saccade on both object location and spatially invariant features implies interactions between areas known to be in involved in updating and those involved in memory and perception—or, alternatively but not exclusively, brain areas that are involved in all these processes. Updating processes have been demonstrated using single-cell recording, imaging, lesion and stimulation studies in multiple brain areas including the parietal cortex and extrastriate and striate cortices (Colby, Duhamel, & Goldberg, 1995; Duhamel, Colby, & Goldberg, 1992; Khan, Pisella, Rossetti, Vighetto, & Crawford, 2005; Medendorp, Goltz, Vilis, & Crawford, 2003; Merriam, Genovese, & Colby, 2003, 2006). In terms of transsaccadic memory, multiple areas including the early visual cortex, parietal cortex, and frontal cortex have been shown to be involved in remembering the location of objects across saccades (Malik, Dessing, & Crawford, 2015; Prime et al., 2008, 2010; Tanaka, Dessing, Malik, Prime, & Crawford, 2014). Recently, the parietal cortex has also been implicated in the updating and memory of shape information across saccades (Subramanian & Colby, 2014). Other areas in the brain—such as extrastriate areas, particularly area V4—have been shown to be sensitive to visual features as well as location (De Beeck & Vogels, 2000), and to receive information from saccade-related areas (Burrows, Zirnsak, Akhlaghpour, Wang, & Moore, 2014; Han, Xian, & Moore, 2009; Moore & Armstrong, 2003). Interestingly, a recent study showed the involvement of multiple areas including the right inferior parietal cortex and extrastriate areas (including V4) during a task where participants were required to remember, update, and report object orientation (Dunkley, Baltaretu, & Crawford, 2016). Finally, these same areas also encompass the network of brain regions involved in working memory (Eriksson, Vogel, Lansner, Bergström, & Nyberg, 2015; Gazzaley, Rissman, & D'Esposito, 2004). In summary, the large overlap and interactions between brain areas involved in location and feature updating and memory processes supports the idea of a linkage between updating and memory processes. 

Conclusions

We tested how well participants remembered spatially variant and invariant features of an object across saccades to determine how features of an object are updated across saccades. We observed that saccades induced systematic biases and increased uncertainty when participants remembered the object's location but also when they remembered its orientation and size. We conclude that all features of an object are linked and updated together across saccades. 

Acknowledgments

AZK is funded by the Canada Research Chair program. AZK and GB are funded by NSERC Canada. 

Commercial relationships: none. 

Corresponding author: Aarlenne Z. Khan. 

Email: [email protected]. 

Address: School of Optometry, University of Montreal, Montreal, Canada.