The relationship between delay period eye movements and visuospatial memory

Abstract

Abstract

Abstract:

Abstract  We investigated whether overt shifts of attention were associated with visuospatial memory performance. Participants were required to study the locations of a set of visual objects and subsequently detect changes to the spatial location of one of the objects following a brief delay period. Relational information regarding the locations among all of the objects could be used to support performance on the task (Experiment 1) or relational information was removed during test and location manipulation judgments had to be made for a singly presented target item (Experiment 2). We computed the similarity of the fixation patterns in space during the study phase to the fixations made during the delay period. Greater fixation pattern similarity across participants was associated with higher accuracy when relational information was available at test (Experiment 1); however, this association was not observed when the target item was presented in isolation during the test display (Experiment 2). Similarly, increased fixation pattern similarity on a given trial (within participants) was associated with successful task performance when the relations among studied items could be used for comparison (Experiment 1), but not when memory for absolute spatial location was assessed (Experiment 2). This pattern of behavior and performance on the two tasks suggested that eye movements facilitated memory for the relationships among objects. Shifts of attention through eye movements may provide a mechanism for the maintenance of relational visuospatial memory.

Introduction

Baddeley (1986) suggested that, analogous to articulatory rehearsal of verbal material, the oculomotor system may play a role in keeping visuospatial information “in mind” during a delay period between study and test. This suggestion that eye movements support memory echoed similar sentiments expressed by Noton and Stark (1971a, b), who theorized that eye movements are a critical aspect of perception and the subsequent activation of the corresponding internal image during recognition. Known in the literature as “scanpath theory,” this proposal was supported by reports of striking similarity between the eye movement patterns observed during memory retrieval and those which occurred during the initial study episode (Fisher & Monty, 1978; Noton & Stark, 1971a, b; Parker, 1978; but see Didday & Arbib, 1975, and Walker-Smith, Gale, & Findlay, 1977, for a different interpretation of these results). Since then, behavioral and neuroscience evidence has indicated that the allocation of spatial attention may support spatial working memory (see Awh & Jonides, 2001, for a review; Awh, Jonides, & Reuter-Lorenz, 1998; Lawrence, Myerson, & Abrams, 2004; Pearson & Sahraie, 2003; Smyth & Scholey, 1994). Given that the brain networks responsible for the planning and execution of eye movements are partially overlapping with those involved in spatial attention and spatial working memory (Awh, Armstrong, & Moore, 2006; Belopolsky & Theeuwes, 2009; Hoffman & Subramaniam, 1995; McPeek & Keller, 2002), overt shifts of attention during the maintenance of visuospatial information may benefit memory performance. Thus, investigating the relationship between eye movements and patterns of memory performance on tasks that tap into different types of mnemonic processing can help further elucidate the nature of visuospatial memory representations and the neural regions that support them. 

The relationship between eye movements and memory has been explored with tasks that require the maintenance and/or retrieval of spatial information. For example, during mental imagery and memory retrieval of spatial information, eye fixations were directed toward locations that were initially studied (Brandt & Stark, 1997; Johansson, Holsanova, Dewhurst, & Holmqvist, 2012; Johansson & Johansson, 2013; Laeng & Teodorescu, 2002). Evidence supporting the link between eye movements and spatial working memory performance, specifically, has come from several studies that have reported that the disruption of eye movements during the retention of visuospatial information impairs memory performance (Lawrence et al., 2004; Pearson & Sahraie, 2003; Postle, 2006). Instructing participants to either maintain fixation or move their eyes in a task-irrelevant manner likely prevented both overt and/or covert memory rehearsal. However, the task manipulations in these studies could have affected memory performance for reasons secondary to the eye movements themselves. Disrupting eye movements fundamentally changes the nature of the task by imposing additional requirements (e.g., maintain fixation) and may even increase memory demands (e.g., remembering to look at the display in a certain manner). Furthermore, while previous studies directly manipulated eye movements through task instructions, these within subject manipulations could have unintentionally affected the way participants move their eyes during the “free eye movement” conditions (Godijn & Theeuwes, 2012; Pearson & Sahraie, 2003). 

Tremblay, Saint-Aubin, and Jalbert (2006) assessed the contribution of delay period eye movements to visuospatial memory performance during the recall of studied locations in serial order, which requires both the formation of spatial memory representations as well as memory for the correct temporal sequence of the study item presentations. Their findings indicated that eye movements during the delay period to previously studied locations benefitted memory performance for serially presented visuospatial information (but, see Godijn & Theeuwes, 2012), but questions remain about whether memory for the spatial and/or the temporal information was affected by these overt shifts of attention. To address this issue, we examined the role of overt shifts of attention (which occurred naturally, without the use of explicit task instructions regarding eye movements) for visuospatial memory in two experiments, and without the additional requirement of memory for temporal order. 

One aspect of short-term visuospatial memory rehearsal that has been overlooked by previous investigations is the relative contribution of eye movements to the absolute item positions and/or the spatial relations among studied items. Jiang, Olson, and Chun (2000) demonstrated that change detection for a tested location was significantly reduced when the surrounding items were either removed or reconfigured between study and test. This evidence suggests that item representations are likely not stored entirely independently from one another, but instead, items within a visual scene are encoded with respect to one another, and the spatial relations among items are represented within short-term memory. That said, visuospatial memory performance still remains above chance when context information is removed; thus, it seems that both absolute and relative spatial representations are held in short-term memory. Thus, a major question left unanswered by previous research is whether overt and/or covert shifts of attention served as a rehearsal mechanism for the absolute spatial locations or whether delay period shifts of attention serve to bolster memory for the spatial relationships among studied locations. 

The present work investigated whether participants spontaneously rehearsed previously studied locations during the delay period of a visuospatial delayed-match-to-sample task in the absence of explicit instructions regarding eye movements. Further, we examined the extent to which eye movements benefitted memory by strengthening representations for the relative spatial positions among items, or whether eye movements enhanced the absolute spatial representations. In two experiments participants viewed a set of novel, abstract objects while load (number of objects studied) and duration of the delay period (time between study and test displays) were varied in order to achieve a wide range of performance on the task. A blank screen was presented during the delay period. In Experiment 1, visuospatial memory was probed after the delay period by re-presenting all of the studied objects in either an intact or manipulated test display (see Figure 1, upper panel). In manipulated displays, the position of one of the objects was moved in space from where it was initially studied. By contrast, the test displays in Experiment 2 only contained a single object, which was re-presented in either the identical or an altered spatial location. Thus, in both experiments, memory for the spatial location of an object was assessed; however, participants could use the stored relational information to guide their decision (Experiment 1), or participants were forced to rely more heavily on memory for the absolute spatial information, as the relational information among items was no longer available (Experiment 2). 

Figure 1

 

The display sequence of the two visuospatial memory experiments. Upper panel: Experiment 1: During the study phase, three, four, or five objects were presented for 2 s, followed by a visual mask (500 ms). Following a variable delay period, the objects were displayed in either identical locations (intact trials) or one object was shifted in location as in the example above (manipulated trials). Lower panel: Experiment 2: Study and delay phase were identical to Experiment 1. However, during the test phase of Experiment 2, only the target object was displayed. In Experiment 1, participants could rely on either absolute or relative spatial information to detect changes in spatial position, whereas in Experiment 2, participants had to rely predominantly on absolute spatial information. In both examples, the red object was shifted to the left by 2.5° during the test phase compared to its location during the study phase.

To examine whether overt shifts of attention were utilized during the delay period to support memory performance, the spatial distribution of eye movements during the delay period was compared to that of the study phase using a newly developed similarity metric. Greater similarity values reflected greater congruence between the spatial distribution of fixations during the study and delay periods. The relationship between the similarity metric and subsequent recognition both across and within participants was then assessed for each experiment. The across-participant analysis examined whether the participants who made more delay period eye movements back to the studied locations overall were also the participants who performed better on the task; the within-participants analysis examined whether the trials in which participants made more delay period eye movements back to the studied locations were more often the trials on which participants responded accurately. Positive correlations with similarity and task performance in both experiments, that is, regardless of whether relative position information with respect to the other objects is made available, would indicate that memory for the absolute spatial locations were maintained and rehearsed via eye movements. By contrast, positive correlations, with similarity and task performance only when information regarding the relative positions among the objects is present in the test display (Experiment 1) but not when the target object is presented in isolation (Experiment 2), would indicate that the rehearsal of spatial locations during the delay period facilitates the formation of relational, rather than absolute, spatial memory representations. More specifically, such findings would suggest that the purpose of these eye movements is to integrate the spatial relations among the studied objects rather than to encode absolute spatial locations of each object (Ryan & Villate, 2009). This work will then reveal the extent to which eye movements are associated with successful retention of visuospatial information, and in particular, the specific nature of the visuospatial information that is associated with delay period eye movements. 

Experiment 1

Methods

Participants

Participants were 16 young adults (nine female) aged 18–32, (M = 22.25; SD = 4.23) with normal or corrected-to-normal vision. Participants were recruited from the Rotman Research Institute participant pool and the University of Toronto. All participants provided informed consent. 

Apparatus and classification of fixations

Stimuli were presented on a 19-in. Dell M991 monitor (resolution 1024 × 768; Dell, Round Rock, TX). Monocular eye movements were recorded with a head-mounted Eyelink II eye tracker (sample rate = 500 Hz; SR Research Ltd., Mississauga, Ontario, Canada). Eye movement calibration was performed at the beginning of the experiment, and drift correction (>5°), if needed, was performed immediately prior to the onset of each trial. Saccades were determined using the built-in EyeLink saccade-detector heuristic; acceleration and velocity thresholds were set to detect saccades greater than 0.5° of visual angle. Blinks are defined as periods in which the saccade-detector signal was missing for three or more samples in a sequence. Fixations are defined as the samples remaining after the categorization of saccades and blinks. 

Stimuli, procedures, and design

Participants were presented with a delayed-match-to-sample task in which they decided whether each of the objects within the study and test displays matched in spatial location (Figure 1, upper panel). Eye movements were recorded throughout the experiment. Participants were not given explicit instructions on how to move their eyes, thus they were engaged in free viewing at all times. At the start of each trial, a set of multicolored, abstract objects (three, four, or five objects) appeared on a gray background for 2 s, followed by a visual mask for 500 ms. The objects were created in Corel Draw v. 12 (Corel, Ottawa, Ontario, Canada) and scaled so that they were all equivalent in size (visual angle = 2°) and were brightly colored to minimize perceptual interference. The objects were uniquely designed to minimize resemblance to real-world objects, discourage use of associated verbal labels to aid in remembering the locations (e.g., “the cat is to the left of the boy”) and were among the set used in prior work (Olsen, Rondina, Riggs, Meltzer, & Ryan, 2013; Ryan, Leung, Turk-Browne, & Hasher, 2007; Ryan & Villate, 2009) Visual masks were made in Adobe Photoshop (Adobe Systems Inc., San Jose, CA) by distorting the study image using a “wave” function; each trial had a unique visual mask. The resulting masks were gray screens that contained highly distorted or pixilated versions of the original objects and pixels were “scattered” so that they extended beyond the original location of the object. The masks were used to minimize visual persistence on the retina so that local luminance changes could not be used to detect a change in an object's spatial position. After the visual mask, a blank gray screen was displayed for 770 to 20,000 ms (median delay period = 5608 ms). Following the delay period, a prompt was displayed for 500 ms to cue the onset of the test display. The test display presented the previously studied objects in either the identical spatial locations (intact display) or one of the objects was shifted 2.5° away (either a horizontal or lateral shift) from the studied location (manipulated display), which resulted in a disruption to the relative and absolute position of the object with respect to the other presented objects. Displays were constructed before the experiment and consisted of 288 “sets” that were counterbalanced across participants. Within the study and test displays, objects could be presented in 1 of 24 possible locations. Twelve locations were located around an imaginary circle (similar to a clock) with a radius of 9.1° and the corresponding “lure” locations were shifted 2.5° laterally or horizontally (either inward or outward) away from the corresponding original location. One hundred forty-four trials were presented in the experiment, half of which were intact and half of which were manipulated trials. Participants used a handheld response box to make their test response (i.e., “intact”/“manipulated” judgments), and were instructed to respond as quickly and accurately as possible. Participants were not told in advance of the number of study objects or the length of the retention interval (trial order was randomized). The number of objects displayed in each trial (three, four, or five) was balanced equally across the intact and manipulated task conditions; similarly, the length of the delay period (24 different delay periods) was balanced equally across the intact and manipulated conditions, as well as the three load conditions. Intact and manipulated displays, as well as the study and test displays, were counterbalanced across participants to control for stimulus-specific effects on viewing. 

Behavioral analysis

Repeated measures ANOVA was used to examine the effect of test display, load, and delay length on behavioral performance (see Supplementary Materials). These factors were also included in a logistic regression, which examined the effect of eye movements on behavioral performance. These two analyses produced a similar set of main and interaction effects; thus, only the results of the logistic regression analysis are reported below. 

Eye movement analysis

We examined the extent to which the spatial distribution of delay period fixations mimicked the spatial distribution of fixations that occurred during the study phase. If participants used overt shifts of attention to “revisit” studied locations in order to boost visuospatial memory performance, the spatial distribution of delay period fixations would be similar to those during the study phase. A similarity metric between the two sets of fixation patterns (the set of x-y coordinates of each fixation and the corresponding fixation duration) was computed using a simple algorithm developed in Python. Fixation patterns were converted into a single vector, which were collapsed across fixation order, with each entry in the vector corresponding to a different spatial location (i.e., the vector included all possible screen locations). The value of the entry for each position was proportional to the fixation duration that occurred at that screen location. In this formulation, longer fixation durations at a certain screen location corresponded to larger (higher amplitude) vector values in the corresponding entry of the vector. To ensure that the similarity metric was not biased by a disparity in the number of study phase and delay phase fixations, only the first seven fixations from each time period were submitted for analysis (on average, during Experiments 1 and 2, participants made seven fixations during the study phase, and on average they made 14 and 11 fixations during the delay phase of Experiments 1 and 2, respectively). These fixation “maps” were then spatially smoothed with a Gaussian filter (full width at half maximum = 100 pixels), to incorporate neighborhood weighting. That is, the amplitude of the vector entries in neighboring positions to the fixation point scaled with a Gaussian drop off. This ensured that nearby but not exactly co-incident fixations were still granted partial weighting by the similarity metric. Similarity between two fixation maps was calculated by computing the Euclidean dot-product between the vectors representing each map. Lastly, the similarity metric was scaled from 0 to 100 such that low scores indicated low similarity between study and delay period fixation patterns and higher scores indicated higher similarity (see Figure 2). 

Figure 2

 

Examples of two different trials that yielded either a relatively high (similarity score = 72.3; upper panel) or a relatively low (similarity score = 0.81; lower panel) similarity score between the study phase and delay period fixation patterns. Fixations are indicated by the turquoise circles overlaid onto the task display background. Yellow squares indicate the location of the studied objects (but did not appear on the screen during the experiment).

Statistics

Statistical correlations and repeated measures ANOVA were computed using SPSS (version 20.0) and logistic regression analysis was performed in R (version 2.15) using a linear mixed effects model. Significant results were reported for logistic regression coefficients associated p values < 0.05 and odds ratios are reported as an estimate of effect size (Pampel, 2000). 

Results

Permutation analysis

A permutation analysis was performed in which the data were initially resampled so that fixation patterns from the study phase of a given trial were compared to delay period fixation patterns from a randomly chosen trial. The similarity analysis was then performed on each trial of the shuffled data in order to create a null distribution for comparison with the correctly labeled data. Because the object locations varied from trial to trial, fixation patterns compared across trials should result in relatively low similarity scores. The obtained similarity values were then permuted 1,000 times to provide a null distribution of data to which the correctly labeled data could be compared. As expected, this analysis of the permuted data yielded resulted in low similarity scores (95% CI [8.36, 9.23]). 

Similarity analysis results: Examining the spatial overlap between study-phase and delay-phase fixations

A range of similarity scores was obtained by comparing the study and delay period fixations for each trial and for each subject. The mean computed similarity value was 18.75 (95% CI [17.99, 19.51]); critically, this mean value was clearly outside of the 95% CI (8.36, 9.23) of the null distribution as determined by the permutation analysis. One participant's mean similarity score was identified as an outlier using Tukey's (1977) method (based on the interquartile range) and this participant was excluded from all further analyses. 

Relationship between similarity scores and accuracy

Participants' mean similarity scores were correlated with their overall task accuracy (percent correct). A significant positive relationship between mean similarity score and memory performance was found (r = 0.71, p_two-tailed = 0.003; Figure 3, left). This significant between-subject correlation provided intriguing initial evidence that overt shifts of attention during the delay period between study and test may facilitate memory performance. 

Figure 3

 

Relationship between overall study-delay fixation similarity scores and task accuracy. Similarity score (an index of spatial overlap between study and delay period fixation patterns) is positively related to overall memory performance when relational information was present in the test display (Experiment 1; n = 15) but is only weakly related to memory performance when the test probe was presented in isolation (Experiment 2; n = 16). Each point corresponds to a single participant.

Participants who had higher overall similarity scores also achieved higher overall accuracy on the task. This suggests that revisiting studied locations during the delay period of a particular trial should affect memory performance on that same trial. To test this hypothesis, the variability of similarity scores across trials within each participant was examined with respect to test accuracy. A logistic regression analysis was performed that included task accuracy (correct vs. incorrect) as the binary outcome variable and test display (intact vs. manipulated), delay period length, memory load (three, four, or five study objects), and similarity score as the predictor variables. 

The results from the logistic regression revealed a significant main effect of similarity score on accuracy (odds ratio: 1.34; p = 0.038); participants were more likely to accurately remember the spatial relations during trials in which the similarity score was high. Performance was more accurate for intact compared to manipulated displays (odds ratio: 0.45; p = 0.029). A main effect of load (odds ratio: 0.80; p = 0.013) was also observed; however, the test display and load variables interacted in their effect on accuracy (odds ratio: 1.68; p < 0.001). Accuracy decreased with load for intact trials and increased with load for manipulated trials (see Supplementary Figure 1A). The logistic regression results also indicated a main effect of delay period on accuracy (odds ratio: 0.73; p < 0.001); performance decreased as time between the study and test phases of the task increased (see Supplementary Figure 1B). In addition, an interaction between the delay period and test display on accuracy was observed (odds ratio: 1.27; p = 0.026), due to the fact that the negative correlation between delay period duration and task accuracy was stronger for intact trials (r = −0.44) than for manipulated trials (r = −0.30). 

In summary, participants who had higher overall similarity scores tended to perform more accurately on the task than participants with lower overall similarity scores. Similarly, within each participant, higher similarity scores were associated with more accurate memory performance across trials. This indicates that revisiting previously studied locations during the time between study and test benefits either memory for the relations among studied objects and/or for the absolute locations themselves. Thus, these overt shifts of attention might serve to strengthen or facilitate the formation of absolute spatial memory representations or the spatial relations among studied objects. 

Similarity analysis across the delay period

After establishing that similar study and delay period fixation patterns resulted in higher memory performance on the task, we next investigated whether the time in which these maintenance period fixations occurred was a factor. To address this question, a “time bin” analysis was performed, in which fixations from ten 2-s intervals during the delay phase were independently compared with the study phase fixations, and a similarity score was computed separately within each of the successive delay period time bins (Figure 4). Results from this analysis showed that similarity was highest early in the delay period (repeated measures ANOVA revealed a main effect of time bin: F = 4.44, p < 0.001). 

Figure 4

 

Similarity scores plotted separately for the ten 2-s time bins, demonstrating how fixation similarity patterns change as a function of time elapsed since study. Error bars depict SE.

We next investigated whether including time bin as a factor in the logistic regression improved the ability for similarity score to predict accuracy. Two linear mixed logistic regression models were used: a reduced model, which contained similarity score as a within-subject factor (collapsing across all other factors), and a full model, which contained both similarity and time bin as factors (as well as the interaction term). The two models were compared with a likelihood ratio test. The full model was a better predictor of accuracy than the reduced model (χ² = 4.03, p = 0.047). In addition, in the full model, there was a linear time by similarity score interaction: z = −2.02, p < 0.043. The negative z-score indicated that the effect of similarity score on accuracy declines over time. To investigate this relationship further, logistic regression was performed for each time bin separately, to assess the strength of the association between similarity score and accurate task performance throughout the delay period. Results indicted that during the first four time bins (0–8 s after the study display) there was a significant or marginally significant effect of similarity score on accuracy (0–2 s: p = 0.076; 2–4 s: p = 0.045; 4–6 s: p = 0.033; 6–8 s: p = 0.059). As for the remaining time bins (which contained fixations occurring more than 8 s after the study display), similarity score was not a significant predictor of task accuracy (ps > 0.195). In summary, the time bin analysis demonstrated that the spatial patterns of early delay period fixations were more similar to those of the study phase compared to those of later in the delay period and that these early delay period fixations were significant predictors of task accuracy. 

Relationship between study period fixations and accuracy

An additional analysis was conducted on the study period fixations themselves to investigate the specificity of the relationship between delay period eye movement rehearsal patterns and task performance. Across subjects, no significant correlation was found between accuracy and the mean number of study period fixations (r = 0.17, p = 0.545). Similarly, a within-subjects logistic regression analysis found no relationship between the number of fixations made during study and task accuracy (p = 0.851). These analyses suggest that task performance was related specifically to eye movement fixation pattern similarity between the study and delay periods and not to the pattern of viewing enacted during encoding alone. 

Experiment 2

In Experiment 2, the testing procedure was altered so that test displays presented only a single target object, which precluded the use of the relative spatial relations among all of the objects in order to detect a change in spatial position (Bays & Husain, 2008; Jiang et al., 2000). Therefore, successful performance predominantly required participants to remember the absolute spatial position of each object. While participants may have incidentally encoded and/or rehearsed the relative positions of the studied objects, this information would not have been necessary or helpful for task performance. Again, the similarity metric was used to examine the extent to which eye movements during the delay period mimicked those during the study phase, and were predictive of subsequent memory performance. 

Methods

Participants

Participants were 16 young adults (12 female) aged 19–30, (M = 21.38; SD = 2.70) with normal or corrected-to-normal vision. Participants were recruited from the Rotman Research Institute participant pool and the University of Toronto and provided informed consent. 

Apparatus

Apparatus and eye movement recording procedures were identical to Experiment 1. 

Procedure

The task used in Experiment 2 was identical to that used in Experiment 1, with the following exception. During the test phase, only one object was presented, but as in Experiment 1, this target object was presented in either the same spatial location in which it was viewed during the study phase or in a shifted spatial location (Figure 1, lower panel). Note that the target object was moved exactly the same distance (on average 2.5° in visual angle) as in Experiment 1. 

Results

Similarity scores and the null distribution were calculated in the same manner as in Experiment 1. The mean similarity value obtained in Experiment 2 was 23.99; 95% CI (23.22, 24.76), and was well outside the 95% CI of the null distribution computed on the permuted trials (11.04. 12.08). In contrast to Experiment 1, the relationship between study-delay fixation pattern similarity and task performance was not statistically significant (Figure 3, right; r = 0.43, p_two-tailed = 0.096). Revisiting studied interest regions during the delay period did not have the same beneficial effect on memory for absolute location information as it did when both absolute and relational information was available as tested in Experiment 1. 

Logistic regression analysis was again performed in which task accuracy was the binary outcome variable and test display (intact vs. manipulated), delay period length, memory load (three, four, or five objects) and similarity score were the predictor variables. The similarity between study period and delay period fixations was not associated with increased memory performance (p = 0.799). The main effect of condition (p = 0.327) and the main effect of load (p = 0.139) were not significant. As in Experiment 1, accuracy decreased with load for intact trials and increased with load for manipulated trials, resulting in a significant test display by load variable interaction (odds ratio: 1.21; p = 0.03; Supplementary Figure 1C). As in Experiment 1, delay length was significantly related to accuracy (odds ratio: 0.68; p < 0.001): performance decreased as time between the study and test phases of the task increased (Supplementary Figure 1D). Finally, a test display by delay variable interaction was observed (odds ratio: 1.76; p < 0.001). This interaction was driven by the significant negative relationship between delay length and accuracy on intact trials (r = −0.63, p = 0.001), which was not present during manipulated trials (r = 0.31, p = 0.140). 

Although high similarity between fixation patterns at the study and delay period was observed, this within-subjects analysis demonstrated that, in contrast to the findings from Experiment 1, revisiting the studied locations during the delay period was not beneficial for task performance. All together, these results suggest that eye movements made to previously occupied locations may help form and/or strengthen memory for spatial relations among the objects in memory, but that shifts of eye movements do not strengthen the memory for the absolute position of each object location. 

Discussion

The present results provide intriguing new evidence for the role of overt shifts of attention during visuospatial memory maintenance. While the functional role of eye movements during memory maintenance has previously been explored, to the best of our knowledge, this is the first study that specifically probed the nature of the memory representations (absolute vs. relational) that are affected by spontaneously produced eye movements. Relational spatial memory was superior for participants who reinstated fixation patterns during the delay period that mimicked the fixation patterns enacted during the study phase. In the same manner, when fixation patterns from the study and delay phases were compared within participants, greater fixation pattern similarity was associated with higher memory performance. We propose that shifts of eye movements—i.e., overt shifts of attention—were used to actively build and maintain visuospatial information from the study phase so that it could be accurately compared to the test probe. Critically, while overt shifts of attention were spontaneously executed during both experiments, eye movements during the delay period were not associated with subsequent memory for the absolute spatial locations, but they were related to more accurate memory for spatial relations. 

An alternate interpretation of the current findings is that the eye movement patterns reported here simply reflect better encoding of the object locations during the study phase of the trial. In other words, it is possible that participants who more effectively encoded the object locations initially were more likely to revisit those same locations later, and therefore, more effective encoding instead of rehearsal per se benefitted subsequent memory. While we cannot definitively rule out this explanation, we believe that this “epiphenomenal” account of the data is not likely to be accurate for two reasons. First, fixation patterns at study were not predictive of performance at test. Previous work from our group has demonstrated that the number of fixations made during the encoding of face stimuli was significantly related to subsequent memory (Chan, Kamino, Binns, & Ryan, 2011). However, in the current study, neither the number of fixations made to each of the studied objects, nor the amount of time spent fixating on the studied objects was related to memory performance. While the current experiment was not designed to test for this dissociation, this suggests that the number of fixations made during encoding may support memory for items, as in our prior work, but not for spatial relations, as tested here. Secondly, similarity scores were higher overall in Experiment 2 than in Experiment 1, but accuracy was worse and similarity patterns were not correlated with task accuracy. In summary, eye movements during study were not related to subsequent memory performance overall, but instead, delay period eye movements were significantly related to better memory for spatial relations. 

Influenced by Hebb's work in 1968, Noton and Stark (1971a, b) provided early evidence that a person's eye movements might follow a unique “scanpath” for a particular image during study and subsequent retrieval of that image (see Foulsham & Kingstone, 2013, for a recent evaluation of scanpath theory). In a later study, Brandt and Stark (1997) suggested, “…that eye movements during imagery reflect the mental process of activating and arranging the part images of a complex scene into their proper locations” (p. 33). Scanpath theory would make similar predictions as those tested here: The extent to which study and delay period fixations spatially overlap would facilitate memory retrieval and predict subsequent memory performance. The current investigation, however, did not examine the temporal order of fixations that scanpath theory also suggests is an important component for reactivation of the stored memory trace. Furthermore, it is still unclear whether the eye movement records themselves become part and parcel of the stored memory representation, as proposed by Stark and colleagues. Thus, future investigations will be needed to fully test whether the current data support the various predictions of scanpath theory. 

The present results may also inform the recently re-energized debate of “looking at nothing” (Ferreira, Apel, & Henderson, 2008; Richardson, Altmann, Spivey, & Hoover, 2009)—a proposal based on findings that participants often make eye movements to previously studied, but now empty, locations when memories for information (e.g., visual objects or semantic facts simultaneously presented in the auditory modality) associated with those locations are probed (Hoover & Richardson, 2008; Johansson & Johansson, 2013; Richardson & Spivey, 2000; Spivey & Geng, 2001). Johansson and Johansson (2013) investigated the role of eye movements in facilitating the recall of spatial relations—and critically, they found that gaze position during memory retrieval facilitated the ability to recall spatial relationships among studied objects (but not memory for the objects themselves). Interestingly, in the studies by Richardson and Spivey (2000), participants made eye fixations back to the studied locations even when location memory itself was not probed, nor was accurate performance contingent upon successful retention of location information, or object-location associations; however, such behavior was not associated with memory performance. Likewise, participants in Experiment 2 of the present study exhibited rehearsal of previously studied locations in their eye movement patterns even when it was not beneficial for task performance. Taken together, the current findings, along with those that investigated the role of eye movements during memory recall, suggest that eye movements play a special role in accessing and/or maintaining spatial relationships among study objects, but these eye movements do not confer benefits for the properties of objects or facts that were associated with specific spatial locations. 

The repeated sampling of studied regions in Experiment 1 likely involves the extended engagement of neural regions that are involved both in the processing of spatial relations and in the modulation of visuospatial attention. It has been traditionally thought that the parietal lobes support processing of both relational and absolute spatial information, and that the left and right hemispheres differentially contribute to these two types of spatial relations (Jager & Postma, 2003). However, a recent neuroimaging investigation found that relational representations rely more on parietal regions while absolute representations involve the hippocampus during spatial navigation (Baumann, Chan, & Mattingley, 2012). These parietal lobe regions also may overlap with areas known to be involved in visuospatial attention and furthermore may be functionally coupled with neural regions responsible for oculomotor planning such as the frontal eye fields and the superior colliculus (Belopolsky & Theeuwes, 2009). Thus, delay period shifts of attention may serve to reactivate the neural circuitry involved in eye movements (frontal eye fields), visuospatial attention (frontal-parietal network), spatial processing, and memory (parietal and hippocampal circuits). This reactivation could result in stronger relational spatial representations, and consequently, more accurate performance when this maintained representation is eventually compared to the test probe. 

While the present findings suggest that spontaneous overt shifts of attention may serve as a visuospatial rehearsal mechanism, this interpretation does not rule out the role of covert shifts of attention in rehearsal. For example, Godijn and Theeuwes (2012) recently provided evidence that overt shifts of attention were no better than covert shifts in memory for serially presented spatial locations. Thus, taken together with these findings, it is likely that the rehearsal system does not rely solely on the execution of oculomotor actions, but can be additionally subserved through covert shifts of attention, a process that is supported by a similar neural architecture. Indeed, electrophysiological recordings in nonhuman primates have revealed that the frontal eye fields are involved in the allocation and maintenance of spatial information even in the absence of overt eye movements (Armstrong, Chang, & Moore, 2009). Thus, further investigation using cognitive neuroscience methodology will lead to a better understanding of the relative roles of overt and covert shifts of attention in memory for absolute and relative spatial locations. 

Conclusions

In summary, overt shifts of attention were associated with superior relational spatial memory performance. These findings have important implications for better understanding memory impairments observed in conditions such as Alzheimer's disease, in which eye movements are also affected (Crutcher et al., 2009; Daffner, Scinto, Weintraub, Guinessey, & Mesulam, 1992). Given the association between eye movements and successful memory performance as observed here, future work could focus on the utilization of eye movement patterns for early detection of memory decline and for evaluating the efficacy of cognitive rehabilitation programs. 

Supplementary Materials

docx S1 

Acknowledgments

We would like to thank Rida Anmol, Maria Ayala, Jana Kube, and Lingqian Li for assistance with data collection. This research was supported in part by a Tier II Canada Research Chair Award and a Canadian Institutes of Health Research Operating Grant awarded to Jennifer D. Ryan. 

Commercial relationships: none. 

Corresponding author: Rosanna K. Olsen. 

Email: [e-mail]. 

Address: Rotman Research Institute, Baycrest, Toronto, ON, Canada.