Long-term memory and hippocampal function support predictive gaze control during goal-directed search

Abstract

Eye movements during visual search change with prior experience for search stimuli. Previous studies measured these gaze effects shortly after initial viewing, typically during free viewing; it remains open whether the effects are preserved across long delays and for goal-directed search, and which memory system guides gaze. In Experiment 1, we analyzed eye movements of healthy adults viewing novel and repeated scenes while searching for a scene-embedded target. The task was performed across different time points to examine the repetition effects in long-term memory, and memory types were grouped based on explicit recall of targets. In Experiment 2, an amnesic person with bilateral extended hippocampal damage and the age-matched control group performed the same task with shorter intervals to determine whether or not the repetition effects depend on hippocampal function. When healthy adults explicitly remembered repeated target-scene pairs, search time and fixation duration decreased, and gaze was directed closer to the target region, than when they forgot targets. These effects were seen even after a one-month delay from their initial viewing, suggesting the effects are associated with long-term, explicit memory. Saccadic amplitude was not strongly modulated by scene repetition or explicit recall of targets. The amnesic person did not show explicit recall or implicit repetition effects, whereas his control group showed similar patterns to those seen in Experiment 1. The results reveal several aspects of gaze control that are influenced by long-term memory. The dependence of gaze effects on medial temporal lobe integrity support a role for this region in predictive gaze control.

Introduction

When people or other primates view repeated scenes or visual arrays compared to novel ones, their eye movements change (i.e., repetition effect, Althoff & Cohen, 1999). They make fewer fixations and sample fewer regions (Althoff & Cohen, 1999; Brockmole & Henderson, 2006; Castelhano, Mack, & Henderson, 2009; Dragan, Leonard, Lozano, McAndrews, Ng, Ryan, & Hoffman, 2017; Ryan, Althoff, Whitlow, & Cohen, 2000; Smith, Hopkins, & Squire, 2006), make longer fixations (Ryan, Hannula, & Cohen, 2007; Smith & Squire, 2017; Solyst & Buffalo, 2014) and shorter saccades (Solyst & Buffalo, 2014), exhibit more predictable eye movement patterns (Althoff & Cohen, 1999), detect targets more quickly (Brockmole & Henderson, 2006; Chau, Murphy, Rosenbaum, Ryan, & Hoffman, 2011; Dragan et al., 2017; Peterson, Kramer, Wang, Irwin, & McCarley, 2001; Tseng & Li, 2004; Wynn, Bone, Dragan, Hoffman, Buchsbaum, & Ryan et al., 2016), or demonstrate more accurate memory for target locations (Brockmole & Henderson, 2006). Eye movements also change if the repeated stimulus is manipulated. People tend to make more fixations in, and spend more time scanning, the manipulated area than the nonmanipulated area (Ryan et al, 2000). Collectively, changes in gaze suggest that the brain may use prior knowledge to make predictions about informative spatial regions worthy of directing and holding gaze (Henderson, 2017). 

Although memory-dependent changes in fixation patterns are well documented, these studies measured the changes in eye movements after only a short retention period (from a few seconds to minutes; e.g., Hannula & Greene, 2012; Hannula & Ranganath, 2008; Hannula & Ranganath, 2009; Hannula, Ryan, Tranel, & Cohen, 2007; Ryals, Wang, Polnaszek, & Voss, 2015; Smith et al., 2006; Smith & Squire, 2008, 2017; Wynn et al., 2016); thus it remains unclear whether various changes in eye movements are driven by short-term adaptation with repetition or if they are also evident in longer-term memory (e.g., one month). Also uncertain is the kind of memory system(s) involved in repetition-related eye movement patterns. The increased prevalence of hippocampal synchrony during recall, seen in both sharp-wave ripples (Leonard & Hoffman, 2017) and gamma-band phase locking (Montefusco-Siegmund, Leonard, & Hoffman, 2017) suggests the hippocampus may provide information to guide eye movements during recall. Whether memory-dependent changes in eye movements that depend on medial temporal lobe (MTL) integrity also require conscious awareness (Smith et al., 2006; Smith & Squire, 2008) or occur in the absence of conscious recollection of stimuli (Olsen Sebanayagam, Lee, Moscovitch, Grady, Rosenbaum, & Ryan , 2016; Ryan et al., 2000; Ryan & Cohen, 2004b; Ryan & Cohen, 2004a), the changes in eye movements are thought to be one manifestation of how the hippocampus and adjacent MTL structures may support memory for relational information represented in visual scenes (Cohen & Eichenbaum, 1993; Eichenbaum & Cohen, 2001; Konkel & Cohen, 2009; Konkel, Warren, Duff, Tranel, & Cohen, 2008; Olsen et al., 2015). 

The present study focused on two aspects of memory-dependent changes in eye movements. First, we investigated whether the repetition effects measured through eye movements are seen in long-term, explicit memory. Second, we examined which features of the repetition effects are MTL dependent. We reanalyzed a subset of the data collected by Chau et al. (2011) in which the repetition effects during flicker change detection tasks were measured across multiple sessions. In their study, only search time and verbal report were measured to demonstrate the repetition effect. Here we analyzed additional eye movement features, including fixation duration, saccadic amplitude, and gaze proximity to the target location. We did so over longer repetition delays (hours to weeks) than what has typically been examined. In the first experiment, the eye movements of healthy participants were recorded while they performed flicker change detection tasks; these eye movements were then compared with eye movements on repeated scenes that were measured a day and a month after the initial experimental session. The changes in eye movements on repeated scenes would be taken to reflect long-term memory effects because of the long retention periods. We also tested whether or not different eye movement measures on repeated scenes reflected either explicit or implicit memory by comparing eye movements when participants explicitly remembered the manipulated objects in change detection tasks to when the objects were forgotten. 

Materials and methods

Experiment 1: Healthy participants

Participants

Ten York University students (ages 19–26 years, mean (SD) age 22.3 (1.49) years) with normal or corrected-to-normal vision volunteered to participate in the experiment. Written informed consent was obtained from all participants, and the study was approved by York University's Human Participants Review Committee. 

Apparatus and stimuli

The position of each participant's head was stabilized by a chin rest that was placed 51 cm away from a monitor (38 cm × 30 cm, resolution: 1280 × 1024). Participants’ eye movements were recorded with the iVew X infrared eye tracking system (SensoMotoric Instruments, SMI, Berlin, Germany). Visual stimuli presentation was controlled by Presentation (Neurobehavioral Systems, Berkeley, CA, USA). 

One hundred twenty natural scene images were used in the change detection tasks, including wildlife, city, rural, and indoor scenes. The images covered the full screen, subtending approximately 43 × 34 degrees in visual angle (dva). Target (changing) objects were manipulated by changing either their locations, colors, or presence (comprising 24.17, 4.17, and 71.67% of all trials, respectively) through Adobe Photoshop. Only one target was manipulated per image, and its location (quadrant on screen), size, and category (animate/inanimate) were controlled across trials to prevent biases in participants’ search strategies. Figure 1A shows sample stimuli. 

Figure 1.

 

Experimental design. (A) Sample stimulus. Image A is the original image, whereas Image A′ is a manipulated version of Image A. The target object outlined in yellow is present in Image A, but it disappears in Image A′. (B) Procedure of a flicker change detection trial on the training and testing sessions.

Procedure

Each participant completed one training session and two follow-up testing sessions. These sessions were conducted on different days to test memory of target objects. A session consisted of three blocks that contained 20 images each; thus each participant viewed 60 images within a session. 

During the initial training session, participants viewed only novel images. An example of a changing image was presented before the experiment started. On each trial of the experiment, the original image alternated with its modified image for 500 ms each, and a gray screen was inserted between these two images, presented for 50 ms. Participants visually searched the alternating images to find the target (i.e., manipulated) object. When they fixated on the target object for 1.2 seconds, or a maximum search time of 60 seconds had passed, the original and modified images alternated without the intervening gray screen; thus the target object became visible (Figure 1B). 

Subsequently, two testing sessions were held, one with a lag of one day and the other with a lag of 30 days after the training session (lag one and lag 30, respectively). Half of the images (30 images) shown during the training session were repeated in the first testing session, and the other half were repeated in the second testing session. In the testing sessions, participants viewed half novel and half repeated images in a random order within each experimental block. All participants viewed the same image sets within a training session. 

The procedure was the same with that of the training session, except for the additional verbal report for memory test. After each trial, a screen automatically showed the recognition questions asking participants whether they had seen the image before (“Have you seen this picture before?”) and whether they remembered the target object (“If so, did you remember which object was changing?”). Participants were required to answer the questions within eight seconds, and their verbal responses were recorded. If participants responded yes to both questions, the trial was classified as target-remembered. If they responded yes to the first question (familiar scene) but no to the second question (target-forgotten), the trial was classified as scene-familiar. Lastly, if they responded no to both questions, the trial was classified as scene-forgotten. 

Experiment 2: Amnesic patient DA and age-control cohort

Participants

DA was a 58-year-old, right-handed man with 17 years of education. He has extensive bilateral MTL damage affecting his hippocampus and perirhinal, entorhinal, and parahippocampal cortices in relation to a diagnosis of viral encephalitis in 1993. The damage is more severe within the right hemisphere, affecting anterior and posterior regions of temporal cortex and regions outside of the temporal lobe, including ventral frontal, occipital, and anterior cingulate cortices. His encephalitis resulted in extensive anterograde and temporally graded retrograde amnesia, affecting his episodic memory to a greater extent than his semantic memory (for further details, see Rosenbaum, Moscovitch, Foster, Schnyer, Gao, Kovacevic, & Levine, 2008). DA's performance was compared to that of five right-handed, male control participants (48–56 years old) with no known history of neurological or psychiatric disorders. All participants had normal or corrected-to-normal vision and were paid $10/hour for study participation. Written informed consent was obtained from all participants, and the study was approved by the York University and Baycrest Health Sciences ethics committees. 

Apparatus and stimuli

The apparatus and stimuli were identical to those used in Experiment 1, except that only 40 images were used. 

Procedure

As in Experiment 1, each participant completed three sessions. The first testing session was conducted five minutes after the training session (lag 0), and the second testing session was held 24 hours later (lag 1). Once participants fixated on the target object for one second, or a maximum search time of 40 seconds had passed, the object flickered as described in Experiment 1 to become visible. Considering the poor memory performance of DA in other tasks, the trial duration was made shorter than that of Experiment 1, so that he could be tested more quickly after training. During the testing sessions, the memory test questions were given after each trial (“Had you seen this picture before?” and “If so, did you remember where the object was? Did you remember what the object was?”). If participants answered yes to remembering either the location or identity of the target object, or both, the trial was classified as target-remembered. If they remembered the scenes but not the target objects, the trial was classified as scene-familiar. If they remembered neither the scenes nor the target objects, the trial was classified as scene-forgotten. There was no time limit for verbal report so that participants had enough time to respond. The experimenter presented the next trial by pressing a button when participants were ready. 

Data analysis

Data were analyzed by using custom MATLAB codes. We did not include trials in which (1) the eye tracking was not accurate due to calibration errors or non-linearities in the signal, and (2) fewer than five fixations occurred. 4.17% and 1.56% of the data in Experiments 1 and 2 were discarded through these processes, respectively. Occasionally, trials were not terminated even though participants looked at the target objects for fixed durations (1.2 seconds in Experiment 1, 1 second in Experiment 2). These trials exaggerated search time and affected other variables as well. To resolve this issue, we discarded the rest of the eye movement data after participants fixated on the target objects for the required fixation time. To examine the repetition effects, we paired novel (in the training session) and repeated (in the testing sessions) trials by images and compared the eye movements across these trials. The trends of the eye movement data across the two testing sessions were not significantly different throughout all analyses; thus we collapsed the data across the testing sessions. Then, the subsets of repeated trials were compared to each other. In particular, we compared target-remembered trials with a combination of scene-familiar and scene-forgotten trials (both target-forgotten trials) due to lack of scene-forgotten trials. This comparison contrasts explicit object-in-scene memory with implicit memory (or even no memory) of the target objects. For the comparisons, we conducted two-tailed paired samples t-test in Experiment 1 and used these results to set the direction of one-tailed tests in Experiment 2, assuming that the direction of effects would be similar across the experiments. The effect size was measured by Cohen's d (Cohen, 1988; Cohen, 1992) and partial eta squared (η²). For DA, we conducted nonparametric permutation tests to compare his performance in novel and repeated trials. The data were permuted 10,000 times, and the test statistic was the mean differences between novel and repeated trials. 

Results

Experiment 1: Healthy participants

Search accuracy and proportion of different trial types

We defined target-found trials based on the distance between the last fixation of a trial and the center of a target object (≤ 2.5 dva). For all participants, the proportions of target-found trials were 70.53% (12.41%), 76.16% (8.01%), and 81.19% (3.08%) (mean (SD)) during the training and the two testing sessions, respectively. Target objects were found more often when images were repeated in the testing sessions than when they were shown for the first time in the training session (lag 1: 19.7 (3.3) to 23.4 (4.99), M_diff = 3.7, SE = 1.30, t(9) = 2.846, p = 0.019, d = 0.874 (Cohen's d); lag 30: 21 (3.3) to 25 (2.21), M_diff = 4, SE = 1.26, t(9) = 3.184, p = 0.011, d = 1.424). 

When we only considered participants’ subjective report of memory, independent of search performance, the number of target-forgotten trials (scene-familiar and scene-forgotten trials) increased over the two testing sessions (12.6 (5.7) to 17.8 (7.05), M_diff = 5.2, SE = 2.06, t(9) = 2.525, p = 0.032, d = 0.811), whereas the number of target-remembered trials tended to decrease in the second testing sessions (15.3 (7.01) to 11 (7.5), M_diff = −4.3, SE = 2.25, t(9) = −1.910, p = 0.088, d = −0.592). The number of falsely reported target-remembered and scene-familiar trials were rare and even absent for most participants (false target-remembered, lag 1: 0.5 (0.71), lag 30: 1.5 (2.07); false scene-familiar, lag 1: 1.5 (2.64), lag 30: 2.5 (2.12)). 

Search time

Mean search time was longer in novel trials than in repeated trials (M_diff = 7.37 seconds, SE = 1.58 sec, t(9) = 4.653, p = 0.001, d = 1.613), indicating that search became more efficient when stimuli were repeated. In addition, search time was shorter when target objects were explicitly remembered than when they were not successfully recalled (M_diff = −11.12 sec, SE = 1.48 seconds, t(9) = −7.525, p < 0.001, d = −2.592) (Figure 2). This suggests that shorter search time in repeated trials is tightly associated with explicit object-in-scene memory represented in target-remembered trials as demonstrated in Chau et al. (2011). 

Figure 2.

 

Search time distributions across different trial types. X on each box-and-whisker plot indicates the mean search time of the trial type. *p < 0.05, **p < 0.01, ***p < 0.005.

Fixation duration

Because we asked the participants to fixate on the target objects for 1.2 seconds to terminate visual search, the durations of the last fixations were not included in the analysis. Figure 3 shows the repetition effects on fixation duration. Mean fixation duration was significantly shorter on repeated trials than on novel trials (M_diff = −44.70 ms, SE = 13.74 ms, t(9) = −3.254, p = 0.01, d = −0.653). A possible explanation for this repetition effect is that participants extracted visual information rapidly at each fixation to determine whether or not they were viewing the target location in the repeated scenes. Furthermore, fixation duration in target-remembered trials was significantly shorter than that in target-forgotten trials (M_diff = −25.50 ms, SE = 5.89 ms, t(9) = −4.331, p = 0.002, d = −0.544). Therefore fixation duration appears to be associated with explicit memory for repeated target objects. 

Figure 3.

 

Fixation duration distributions across different trial types. X on each box-and-whisker plot indicates the mean fixation duration of the trial type. *p < 0.05, **p < 0.01, ***p < 0.005.

Distance to the area of interest

We measured distances from each ordinal fixation to the area of interest (AOI, i.e. the center of the target objects). This measurement demonstrates gaze proximity to the target as the trial progressed, and whether this varied as a function of trial type. We used a temporal sliding window technique: Each window contained 5 consecutive fixations, and windows were shifted by one ordinal fixation (e.g., fixation numbers 1–5, 2–6, etc.). For each window, distances from each of the five fixations to the AOI were computed and then averaged. The same process was conducted throughout the entire sliding window. Only target-found trials were included in the analyses to ensure performance on all trials ultimately converged on the target AOI, allowing the distance time courses to be the dependent measure of interest. 

The mean distance to the AOI was shorter in repeated than in novel trials (M_diff = −2.06 dva, SE = 0.29 dva, t(9) = −7.098, p < 0.001, d = −2.184). When repeated trials were broken down by explicit target memory, distance to the AOI was shorter in target-remembered trials than in target-forgotten trials (M_diff = −3.41 dva, SE = 0.36 dva, t(9) = −9.566, p < 0.001, d = −3.035). The results suggested that explicit memory of target objects guided participants’ eye movements more directly to the AOI (Figure 4A). 

Figure 4.

 

(A) Distributions of distance to the AOI. X on each box-and-whisker plot indicates the mean distance to the AOI of the trial type. *p < 0.05, **p < 0.01, ***p < 0.005 (B) Time courses of the distance to the AOI over the first 16 ordinal fixation bins and (C) over the last 16 ordinal fixation bins, aligned to the end of search. Error bars indicate SEM.

We ran a repeated-measures analysis of variance to compare the distance to the AOI in novel and repeated trials in the early phase of visual search. We evaluated the first 16 ordinal fixation bins, containing 20 fixations, because participants made at least 20 fixations in most of trials. Figure 4B depicts the distance to the AOI within this time window. We found a main effect of stimulus repetition (F(1, 9) = 55.151, p < 0.001 η² = 0.860), indicating that the distance to the AOI was shorter in repeated trials than in novel trials. We also found a main effect of ordinal fixation bin (F(15, 135) = 5.541, p < 0.001, η² = 0.381), whereas the interaction between stimulus repetition and ordinal fixation bin was not significant (F(15, 135) = 0.376, p = 0.983, η² = 0.04). Hence, the shorter distance to the AOI in repeated trials was evident even in the early phase of the search. When the subsets of repeated trials were compared, the distance to the AOI varied depending on trial type (F(1, 9) = 27.928, p = 0.001, η² = 0.756), whereby the distance to the AOI was shorter on target-remembered trials than the distance on target-forgotten trials. There was no main effect of the ordinal bin (F(15, 135) = 0.764, p = 0.714, η² = 0.078) nor of the interaction between trial type and ordinal bin (F(15, 135) = 0.575, p = 0.890, η² = 0.06). Therefore, when target objects were explicitly remembered, fixations landed closer to the target than on trials when targets were forgotten. This measurement suggests that explicit memory of the repeatedly seen target objects is tightly related to shorter distances to the AOI, measurable within the first few fixations of search initiation. 

We also conducted the same analysis on the last 16 ordinal fixation bins by aligning the ordinal fixation bins of each trial to the end of search (Figure 4C). Within this time window, we found a main effect of stimulus repetition indicated that distance to the AOI was shorter in repeated trials than in novel trials (F(1, 9) = 19.933, p = 0.002, η² = 0.689). There was a main effect of ordinal fixation bin (F(15, 135) = 221.071, p < 0.001, η² = 0.961) and an interaction between stimulus repetition and ordinal fixation bin (F(15, 135) = 4.036, p < 0.001, η² = 0.31). Note that we included only target-found trials in the analysis; thus the difference in the distance to the AOI between the two types of trials predictably decreased by the end of search. When time courses in target-remembered and target-forgotten trials were compared, the distance to the AOI in target-remembered trial was shorter than that in target-forgotten trials (F(1, 9) = 78.823, p < 0.0001, η² = 0.898). There was a main effect of ordinal fixation bin (F(15, 135) = 56.47, p < 0.001, η² = 0.863) and of the interaction between stimulus repetition and ordinal fixation bin(F(15, 135) = 13.203, p < 0.001, η² = 0.595). These results are consistent with an early bias to scan near the target due to explicit memory after scene recognition, and that once gaze falls near to the target, detection follows shortly thereafter regardless of memory type. 

This raises the possibility that if people start search around the target object by chance, they are likely to detect the target more quickly compared with when they begin search far from the target. To examine this possibility, mean distance from the initial search position (i.e., the first image-guided fixation) to the AOI was compared across different memory types. The mean distance from the initial search position to the AOI was not different between novel and repeated trials (M_diff = 0.32 dva, SE = 0.48 dva, t(9) = 0.667, p = 0.522, d = 0.247). Difference between the target-remembered and target-forgotten trials was not significant, either (M_diff = −0.24 dva, SE = 0.93 dva, t(9) = −0.259, p = 0.802, d = −0.120). The results suggest that mean distance to the AOI was not confounded by initial search positions and that this measurement reflects a strong top-down, memory-guided search advantage for the explicitly remembered target objects in repeated scenes. 

Saccadic amplitude

Saccadic amplitude was measured to examine the spread of eye movements, for instance, whether participants more locally or globally searched for the target objects depending on memory type. Larger saccadic amplitudes indicate global search, whereas smaller saccadic amplitudes indicate local search (Tatler & Vincent, 2008; Unema, Pannasch, Joos, & Velichkovsky, 2005). In the present study, however, saccadic amplitude was not affected by stimulus repetition (Figure 5, M_diff = −0.03 dva, SE = 0.26 dva, t(9) = −0.128, p = 0.901, d = −0.045). When we further broke down repeated trials into target-remembered and target-forgotten trials, there was no significant difference in saccadic amplitude between the two trial types, either (M_diff = 0.03 dva, SE = 0.22 dva, t(9) = 0.14, p = 0.892, d = 0.051). Hence, neither stimulus repetition nor explicit memory strongly affected saccadic amplitude. 

Figure 5.

 

Distributions of saccadic amplitude across different trial types. X on each box-and-whisker plot indicates the mean saccadic amplitude of the trial type.

Experiment 2: Amnesic patient DA and age-matched controls

Search accuracy and proportion of different trial types

For all participants, including DA, the proportion of target-found trials were 50.81% (12.41%), 56.98% (12.76%), and 55.11% (14.47%) (mean (SD)) during the initial training and two testing sessions, respectively. DA could perform the change detection task normally despite his deficits in memory. Target objects were found more often in repeated trials than in novel trials in the first testing session (lag 0: 10.83 (0.75) to 14 (1.26), M_diff = 3.17, SE = 0.48, t(5) = 6.635, p < 0.001 (one-tailed, see data analysis), d = 3.042). In contrast, the number of target-found trials did not vary depending on scene repetition in the second testing session (lag 1: 11.33 (2.07) to 13.33 (3.2), M_diff = 2, SE = 1.81, t(5) = 1.107, p = 0.160, d = 0.742). 

For DA's control group, there was no difference in target-remembered trials across the two testing sessions (11.4 (3.13) to 8.8 (1.79), M_diff = 2.6, SE = 1.36, t(4) = 1.906, p = 0.065, d = 1.020). The number of target-forgotten trials did not significantly differ across two testing sessions (7.6 (2.51) to 10.2 (2.17), M_diff = −2.6, SE = 1.86, t(4) = −1.398, p = 0.118, d = −1.109). Falsely reported target-remembered and scene-familiar trials were rare (number of false target-remembered trials, lag 0: 0.6 (0.89), lag 1: 0.8 (0.45); false scene-familiar, lag 0: 1 (0.71), lag 1: 2.4 (0.55)). DA did not remember or feel familiar with any repeated stimuli, therefore all repeated trials were classified as scene forgotten. 

Search time

Figure 6 depicts how repetition effects influenced search time. The mean search time of DA's control group was shorter in repeated trials than in novel trials (M_diff = −7.21 sec, SE = 2.06 sec, t(4) = −3.496, p = 0.013, d = −2.00). In particular, search time in target-remembered trials was much shorter than that in target-forgotten trials (M_diff = −17.46 seconds, SE = 2.40 seconds, t(4) = −7.276, p = 0.001, d = −4.402), suggesting that explicit target memory was reflected in the search time measure. Unlike the control group, a nonparametric permutation test showed that DA's search time in novel and repeated trials did not differ (M_diff = 1.06 sec, SE = 2.45 seconds, p = 0.366, d = 0.077), failing to show search benefit in repeated trials. Because DA did not report any explicit memory of target objects, we did not compare subsets of repeated trials. 

Figure 6.

 

Yellow diamonds represent DA's mean search time and box-and-whisker plots represent search time distributions of DA's control group. X on each box-and-whisker plot indicates the mean search time of the trial type. *p < 0.05, **p < 0.01, ***p < 0.005.

Fixation duration

DA's control group did not make shorter fixations in repeated trials than in novel trials (M_diff = −30.82 ms, SE = 16.17 ms, t(4) = −1.906, p = 0.065, d = −0.472), but fixation duration was shorter in target-remembered trials compared with target-forgotten trials, consistent with the fixation duration seen in the young adults in Experiment 1 (M_diff = −37.81 ms, SE = 15.72 ms, t(4) = −2.406, p = 0.037, d = −0.567) (Figure 7). DA's mean fixation duration did not vary between novel and repeated trials (M_diff = 9.06 ms, SE = 13.57 ms, p = 0.258, d = 0.147), implying that the memory effects on fixation duration relied on hippocampal integrity. 

Figure 7.

 

Yellow diamonds represent DA's mean fixation duration and box-and-whisker plots represent fixation duration distributions of DA's control group. X on each box-and-whisker plot indicates the mean fixation duration of the trial type. *p < 0.05, **p < 0.01, ***p < 0.005.

Distance to the area of interest

For DA's control group, mean distance to the AOI was shorter in repeated than in novel trials (M_diff = −1.41 dva, SE = 0.54 dva, t(4) = −2.618, p = 0.030, d = −1.301) (Figure 8A). Within repeated trials, mean distance to the AOI was much shorter when participants remembered target objects than when they forgot the target objects (M_diff = −2.63 dva, SE = 0.73 dva, t(4) = −3.612, p = 0.012, d = −2.484). Hence, the search advantage in repeated trials was likely to be derived from explicit recall of target objects. Whereas the control group showed the repetition effect, stimulus repetition did not affect DA's mean distance to the AOI—in repeated trials, he approached the target objects as direct and quickly as in novel trials (M_diff = −0.54 dva, SE = 0.86 dva, p = 0.343, d = −0.130). 

Figure 8.

 

(A) Yellow diamonds represent DA's mean distance to the AOI and box-and-whisker plots represent distributions of distance to the AOI of DA's control group. X on each box-and-whisker plot indicates the mean value of the data. *p < 0.05, **p < 0.01, ***p < 0.005 (B) Time courses of the distance to the AOI over the first 16 ordinal fixation bins and (C) over the last 16 fixation bins (aligned to the search end) of DA's control group. (D) and (E) show DA's time courses of the distance to the AOI over the first and last 16 ordinal fixation bins, respectively. Error bars indicate SEM.

Figure 9.

 

Yellow diamonds represent DA's mean saccadic amplitude and box-and-whisker plots represent saccadic amplitude distributions of DA's control group. X on each box-and-whisker plot indicates the mean saccadic amplitude of the trial type.

Figure 8B shows the distance to the AOI over the first 16 ordinal bins (20 fixations). A repeated analysis of variance showed no main effect of stimulus repetition based on our alpha level of p < 0.05 (F(1, 4) = 7.425, p = 0.053, η² = 0.650). There was, however, a main effect of ordinal fixation bin (F(15, 60) = 3.773, p < 0.001, η² = 0.485) and an interaction between stimulus repetition and ordinal fixation bin (F(15, 60) = 2.341, p = 0.01, η² = 0.369). These results indicate that distances to the AOI in novel and repeated trials varied across time. Within repeated trials, there was a main effect of memory type (F(1, 4) = 12.714, p = 0.023, η² = 0.761), whereby distances to the AOI were shorter in target-remembered trials than in target-forgotten trials. Despite this overall effect, the main effect of ordinal fixation bin (F(15, 60) = 0.514, p = 0.923, η² = 0.114) and the interaction between stimulus repetition and ordinal fixation bin (F(15, 60) = 0.999, p = 0.468, η² = 0.200) were not significant. This analysis suggests that for healthy participants, distance to the AOI reflects explicit memory of repeated target objects in the early phase of search, but not aligned to the end of search (Figure 8C), where we found no significant differences in the distance to the AOI between novel and repeated trials (F(1, 4) = 0.012, p = 0.919, η² = 0.003) nor between target-remembered and target-forgotten trials (F(1, 4) = 0.997, p = 0.375, η² = 0.2). Qualitatively, DA's data showed no evidence that gaze was closer to the AOI with trial repetition (i.e., he showed no implicit memory effect of gaze being directed toward the target) (Figure 8D). Within the last 16 ordinal fixation bins, DA's data fluctuated until the end of visual search (Figure 8E). 

As in Experiment 1, the distance from the initial search position to the AOI was measured to test whether the initial search position modulated the distance to the AOI. DA's control group showed the different patterns over the two testing sessions. In the first testing session (i.e., 5 minutes after the testing session) the mean initial search position was closer to the AOI in repeated trials than in novel trials (M_diff = −5.96 dva, SE = 1.71 dva, t(4) = −3.489, p = 0.013, d = −2.805) but the difference between novel and repeated trials disappeared in the second testing session (M_diff = −1.42 dva, SE = 2.06 dva, t(4) = −0.691, p = 0.264, d = −0.586). Memory of target objects and their locations could be more vivid and precise after a short retention period; hence, this memory representation could make participants to start search around target locations. The number of target-remembered and target-forgotten trials was very few when the testing sessions were broken down, so we could not compare the initial search positions depending on memory types. DA's initial search position relative to the AOI was not different in novel and repeated trials in either testing session (lag 0: M_diff = 3.92 dva, SE = 2.72 dva, p = 0.156, d = 0.487; lag 1: M_diff = −0.53 dva, SE = 1.36 dva, p = 0.420, d = −0.094). 

Saccadic amplitude

For DA's control group, mean saccadic amplitude in repeated trials did not differ from that in novel trials (M_diff = 0.83 dva, SE = 0.48 dva, t(4) = 1.722, p = 0.08, d = 0.975). When repeated trials were broken down, the differences in saccadic amplitude were also not significant between target-remembered and target-forgotten trials (M_diff = −0.34 dva, SE = 0.23 dva, t(4) = −1.485, p = 0.106, d = −0.994). Unlike the other eye movement measurements that we reported, saccadic amplitude did not specifically reflect explicit memory of target objects. DA's saccadic amplitude also did not vary with stimulus repetition (M_diff = 0.53 dva, SE = 0.4 dva, p = 0.077, d = 0.321). 

Discussion

This study examined whether eye movements reflect long-term, explicit memory of repeated stimuli during goal-directed visual search and whether these repetition effects are associated with MTL integrity. In the first experiment, healthy participants generally showed search advantages when they explicitly remembered the target objects in repeated scenes. The repetition effects were also evident even after a one-month–long retention period, indicating that the effects are present for long-term memoranda. Their search time and fixation durations were shorter when scenes were repeated, and among these repeated trials, when target objects were explicitly remembered than when they were forgotten, suggesting efficient visual search. In addition, participants’ eye movements moved more directly to the target when they remembered the repeated targets. Saccadic amplitude, however, was not strongly affected by scene repetition or explicit memory of target objects. In the second experiment, we compared eye movements of an MTL-damaged amnesic person (DA) with those of a neurotypical control group. DA performed visual search tasks normally, but he could not recall any of the repeated stimuli. In addition, his eye movements in novel and repeated trials were indistinguishable, failing to show repetition effects, whereas the control group showed the changes in scanning behavior similar to those observed in young participants in Experiment 1. The current results provide evidence that the hippocampus contributes to maintaining explicit memory of repeated stimuli and to the subsequent changes in eye movements. In contrast to fixation locations and durations, saccadic amplitude was not strongly modulated by stimulus repetition, and there was no evidence that explicit object-in-scene memory alters the overall magnitude of saccades during search. 

Our fixation duration results are in apparent contrast to other studies measuring fixations during repeated and novel stimulus presentation (e.g., Ryan et al., 2007; Smith & Squire, 2017; Solyst & Buffalo, 2014). These seemingly incongruent results might have resulted from different task instructions, which could modulate eye movements (Castelhano et al., 2009; Olsen et al., 2016; Smith & Squire, 2017). Most likely, the differences are due to our use of a goal-directed search task, not a (passive) free-viewing task. A potential role of explicit memory of target objects during goal-directed search could be that it enables participants to more rapidly analyze visual information at each fixation until they find the targets, reducing fixation durations. We are also measuring search at longer intervals from initial presentation, so long-term memory may alter fixation durations independently of the short-term/adaptation effects that may be factors in the previously-reported results. 

We found that people moved their eyes more directly to the target objects (i.e., shorter distance to the area of interest) when these objects were explicitly remembered, and this effect emerged from the early phase of visual search. The present result is consistent to a previous finding that eye movements directed more quickly to the target positions in repeated scenes than in novel ones (Brockmole & Henderson, 2006; Wynn et al., 2016). Furthermore, this viewing behavior was the same no matter how close or far the location of the first gaze fixation was from the target location, except when the retention period was relatively short (5-minute retention in Experiment 2) so that more vivid memory of target locations could guide the first fixation. This suggests that more direct eye movements to remembered targets may reflect strong top-down memory-guided modulation of eye movements, rather than modulation by bottom-up factors, such as initial search positions that randomly fall around the target locations. A recent study measured the scanpath similarity between novel and repeated visual search (Wynn et al., 2016). Critically, scanpaths deviated rapidly on repeated trials, and the deviation in scanpaths from those of novel search was correlated with greater search efficiency (faster detection), which may indicate that people take visual “shortcuts” to reach the target. This scanning strategy could inhibit non-essential fixations that do not contribute to the current search goal, facilitating search efficiency and may provide a potential explanation for the present observation that gazes is rapidly directed toward the targets in repeated scenes. 

It is notable that eye movements changed after a single exposure to each scene stimulus during the training session. These repetition effects persisted despite the long interval between the training and the testing sessions (up to one month). These findings suggest that the repetition effects are apparent in long-term memory and that they are not derived from simple adaptation. In contrast, previous studies typically administered recognition tests immediately after the study phase (Dragan et al., 2017; Hannula & Greene, 2012; Hannula & Ranganath, 2008, 2009; Hannula et al., 2007; Ryals et al., 2015; Smith et al., 2006; Smith & Squire, 2008, 2017; Wynn et al., 2016). Information accumulated over the lifetime that is regarded as long-term semantic memory also influences our visual experiences (Moores, Laiti, & Chelazzi, 2003; Võ & Wolfe, 2012; Võ & Wolfe, 2013). Eye movements made while viewing pre-experimentally familiar stimuli, such as famous faces are different from those for unfamiliar stimuli (Althoff & Cohen, 1999), indicating that these effects can be induced by longer-term memory of the objects. 

The present visual search paradigm might have led to greater improvement in explicit memory for the target objects than other visual memory tasks. Specifically, along with the target object and its scene context, the way a target object changed could also be encoded. Furthermore, target objects were revealed for four seconds after visual search, which might have provided additional opportunities to memorize target objects. Hence, it is possible that explicit memory of target objects could have led to longer-lasting changes in eye movements (i.e., 30 days after the initial exposure). Future studies could use complementary visual search paradigms that omit some of the features of the present task and measure eye movements after a long retention period to examine this possibility. Whereas we consistently observed repetition effects across different eye movement measures and experimental sessions, the repetition effect on saccadic amplitude was not evident. We note that there were nominal but nonsignificant changes in amplitude that may indicate a smaller effect, or that a specific subset of saccades during search may be affected. In these cases, our omnibus measures and statistical power may have lacked the necessary sensitivity to detect such effects; further studies with more refined eye-movement measures and larger samples could test this directly. 

DA's eye movements were not affected by stimulus repetition, implying that hippocampal integrity may be required to exhibit repetition effects. DA's brain damage is not only confined to the hippocampus but extends into the MTL and is also present in the other regions, including the right anterior temporal cortex. The bilateral extent of hippocampal damage (where unilateral lesions are typically without this severe amnesia), along with previous studies showing a role for the hippocampus and in object-in-scene memory, make this a parsimonious account of the present results from DA. In addition to the deficits associated with hippocampal damage, physiological markers of information processing in the hippocampus are also modified with repetition in this task. Gamma oscillation phase locking is increased during repeated trials (Montefusco-Siegmund et al., 2017) and the sharp-wave ripple, thought to coordinate both intra hippocampal and extrahippocampal memory representations, is stronger for target-remembered trials (Hussin, Leonard, & Hoffman, 2018) and occurs preferentially when gaze is near the target on repeated trials (Leonard & Hoffman, 2017). 

These findings are consistent with the view that the hippocampus supports configural/relational memory (Cohen & Eichenbaum, 1993; Eichenbaum & Cohen, 2001; Hannula & Ranganath, 2008; Hannula & Ranganath, 2009; Hannula et al., 2007; Konkel & Cohen, 2009; Konkel et al., 2008; Olsen et al., 2015; Ryals et al., 2015; Ryan et al., 2000). Specifically, the visual search task used in the present study could benefit from item-context relational memory, which, in turn, has been shown to manipulate eye movements. Therefore DA's lack of repetition effects might be due to impairment in both explicit object memory and relational memory, which may involve unconscious processes. Previous findings indicated that the repetition effect does not require consciousness (Hannula & Ranganath, 2009; Olsen et al., 2016; Ryals et al., 2015; Ryan et al., 2000). As mentioned earlier, task requirements could influence eye movements; thus we hypothesize that, at least in the present case, performance in goal-directed visual search is more likely to be correlated with conscious awareness. Because there is a correct answer in the current visual search task and the search is time-limited, search would be strongly guided by explicit memory of remembered targets, which leads to efficient search. On the other hand, participants need to explore additional possible areas until they detect target objects if they are merely familiar with a repeated scene or do not remember the target. In the same vein, two contradictory findings from work by Smith and Squire suggest that the repetition effect could be either conscious or unconscious depending on different task instructions (Smith & Squire, 2008; Smith & Squire, 2017). 

Most of the effects in this study are consistent across different sessions and participant groups, and the effect sizes are relatively strong; however, negative results should be interpreted with caution due to the sample sizes in these experiments. Future research with increased sample sizes could determine whether the marginal or trend-level effects are in fact weaker but meaningful to further elucidate the role of the MTL in repetition effects. 

The current study demonstrated that the repetition effects in eye movements are seen even when recalling long-term memories and that they occur mainly due to conscious recollection of target objects, mediated by the MTL. The MTL, including the hippocampus, may support this process through the creation and/or maintenance of representations that enable spatial predictions for gaze allocation (Burgess, Maguire, & O'Keefe, 2002; Sarel, Finkelstein, Las, & Ulanovsky, 2017). 

Acknowledgments

Supported by the Sloan Foundation to KLH, a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and York Research Chair to RSR, JKT, MF, and KLH, the Canada First Research Excellence Fund and Vision: Science to Applications (VISTA) Research Grant to RSR and MF, and the Canada Research Chairs Program and the Air Force Office for Scientific Research (USA) to JKT. 

Commercial relationships: none. 

Corresponding author. Kari L. Hoffman. 

Email: [e-mail]. 

Address: Department of Psychology, Vanderbilt University, PMB 407817, 2301 Vanderbilt Pl, Nashville, TN 37240-7817, USA.