Impairment of visual memory for objects in natural scenes by simulated central scotomata

Abstract

Because of the close link between foveal vision and the spatial deployment of attention, typically only objects that have been foveated during scene exploration may form detailed and persistent memory representations. In a recent study on patients suffering from age-related macular degeneration, however, we found surprisingly accurate visual long-term memory for objects in scenes. Normal exploration patterns that the patients had learned to rereference saccade targets to an extrafoveal retinal location. This rereferencing may allow use of an extrafoveal location as a focus of attention for efficient object encoding into long-term memory. Here, we tested this hypothesis in normal-sighted observers with gaze-contingent central scotoma simulations. As these observers were inexperienced in scene exploration with central vision loss and had not developed saccadic rereferencing, we expected deficits in long-term memory for objects. We used the same change detection task as in our patient study, probing sensitivity to object changes after a period of free scene exploration. Change detection performance was significantly reduced for two types of scotoma simulation diminishing foveal and parafoveal vision—a visible gray disc and a more subtle image warping—compared with unimpaired controls, confirming our hypothesis. The impact of a smaller scotoma covering specifically foveal vision was less distinct, leading to a marginally significant decrease of long-term memory performance compared with controls. We conclude that attentive encoding of objects is deficient when central vision is lost as long as successful saccadic rereferencing has not yet developed.

Introduction

When we explore a complex visual scene, usually only those objects that have been selectively attended are remembered explicitly. Unattended objects, for example, the appearance of an unexpected object in simple visual displays (Mack & Rock, 1998), often go unnoticed. But even when observers are instructed to look for a change in a scene, changes introduced during eye movements are often missed (Bridgeman, Hendry & Stark, 1975; Grimes, 1996) or take surprisingly long to be detected when an original and a modified scene are presented in alternation, separated by a brief blank (flicker paradigm; Rensink, O'Regan, & Clark, 1997; Simons & Rensink, 2005). When the changed object becomes fixated, however, detection performance improves significantly (Henderson & Hollingworth, 1999; Hollingworth, Schrock, & Henderson, 2001), an effect that has been attributed to selective attentive processing that is required to encode objects into immediate visual memory (Rensink et al., 1997). Notably, the benefit of attentive object processing is not restricted to immediate memory. Even after attention has been withdrawn, object changes within the same semantic category (token changes) or subtle viewpoint changes are remembered in rich detail (Hollingworth, 2006). Robust long-term memory for objects has, for example, been shown across multiple intervening attended objects within the same scene or many different intervening scenes (Brady, Konkle, Alvarez, & Oliva, 2008; Hollingworth, 2004; Hollingworth & Henderson, 2002; Standing, 1973) and even after retention periods of at least 1 day (Hollingworth, 2005; Vogt & Magnussen, 2007). 

One important precondition for successful object encoding into visual long-term memory is, as mentioned above, previous attentive selection that goes along with foveation in normal vision (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995). Objects that have never been fixated, and therefore not attended, lead to chance performance in change detection (Hollingworth et al., 2001; Hollingworth & Henderson, 2002; Simons & Rensink, 2005). The close relation between foveation and memory performance is further reflected by an increased later memory success if target objects are fixated more often or for a longer duration during exploration (Hollingworth & Henderson, 2002; Pertzov, Avidan, & Zohary, 2009; Tatler, Gilchrist, & Land, 2005). Moreover, successful change detection critically depends on the distance between the locus of fixation and the changed object (Hollingworth et al., 2001; O'Regan, Deubel, Clark, & Rensink, 2000), and encoding of objects into visual long-term memory may be successful only within a small radius around the current fixation (Nelson & Loftus, 1980). 

Here, we ask whether an impairment of central vision interferes with attentive object processing, entailing deficient visual long-term memory for objects. When central vision is lost, all attentive processing that is typically carried out with the fovea needs be relocated to the remaining peripheral vision. This may not be without problems. When observers are forced to detect changes by covertly shifting attention in peripheral vision, detection becomes significantly delayed and less reliable than when objects can be foveated (Hollingworth et al., 2001, experiment 2). In a similar fashion, attentive encoding into visual long-term memory may be impaired when attention needs to be decoupled from the fovea during scene exploration. We recently investigated this question in a sample of patients with age-related macular degeneration (AMD), in whom central vision was degraded (Geringswald, Herbik, Hofmüller, Hoffmann, & Pollmann, 2015). Patients were asked to freely explore and memorize everyday objects in natural, computer-generated real-world scenes and were then probed for one specific target object that could either remain the same or be replaced by an object from the same semantic category (token change) in an otherwise identical test scene. A salient onset cue was used to guide attention away from the locus of a potential change to ensure that performance had to rely on memory (Hollingworth, 2003). Surprisingly, we found that visual long-term memory for objects was as good as in normal-sighted, age-matched controls. Taken together with the normal fixation parameters in the patient group, we hypothesized that the patients must have had developed rereferencing of their eye movements to a more eccentric, still functioning, part of the retina (preferred retinal locus [PRL]; von Noorden & Mackensensen, 1962; White & Bedell, 1990; Whittaker, Cummings, & Swieson, 1991) that could then be used as a pseudofovea for attentive encoding of objects into visual long-term memory. 

Alternatively, it may simply be the case that encoding of objects into visual long-term memory does not depend on attentive foveation at all, as long as observers are given sufficient time to relocate attention to the periphery. To investigate whether visual long-term memory for objects requires attentive foveation in normal vision, we replicated the change detection experiment used in our previous study simulating a gaze-contingent central scotoma in normal-sighted observers who had no previous experience with central vision loss. In the full scotoma condition, we used a scotoma with a radius of 4° of visual angle, roughly encompassing the scotoma sizes of three patients in our previous study who had already developed absolute scotomata and were able to perform the experiment with their worse eye (horizontal scotoma radii of 3.5°, 9°, and 2.5° of visual angle). Visual long-term memory performance in two of these three patients was of similar quality as in healthy controls. The full scotoma condition in the current study was thus designed as a clearly visible gray patch superimposed on the natural scene, extending across foveal and parafoveal vision to simulate an absolute scotoma as in these patients. If object encoding is independent of object foveation (in the fovea or, after foveal vision loss, a pseudofovea), untrained observers should be able to perform as well as unimpaired controls at change detection. If, on the other hand, attentive foveation is a prerequisite to object encoding into visual long-term memory in normal vision, untrained observers, who do not have developed saccadic rereferencing to an extrafoveal PRL, were expected to show reduced long-term memory performance. As long as saccadic rereferencing has not been developed, search with a central scotoma affords a high degree of top-down control to avoid objects of interest from becoming fixated and thereby falling into the scotoma. This top-down controlled exploration was expected to be reflected in the gaze parameters. 

The majority of our patients, however, were at earlier stages of the disease and had developed relative scotomata, that is, visual depression without a complete loss of light perception. To increase the generality of our findings, we therefore used a novel, more subtle image-warping implementation that vanished foveal and parafoveal vision in an additional warp scotoma simulation, hypothesized to simulate earlier stages of naturally occurring central vision loss. Because of the degraded visibility of the borders compared with the full scotoma, the warp scotoma simulation additionally enabled us to test whether participants might have learned to rereference saccades to a PRL during the experiment. Recent training studies in normal-sighted observers have indicated that saccadic rereferencing can develop after only a few hours of training (Kwon, Nandy, & Tjan, 2013; Walsh & Liu, 2014). One important precondition, however, was the clear visibility of the scotoma and its borders. If any saccadic rereferencing might develop in our experiment, this was expected to be more difficult in the more subtle warp scotoma condition lacking clear borders. Exploration should then be more cumbersome in comparison with the full scotoma condition, in which the scotoma and its borders were clearly visible. If no saccadic rereferencing developed, gaze patterns were expected to be highly similar in both simulations. 

Recent data on object search in natural scenes suggest that attentive foveation may not be necessary for efficient object encoding as long as parafoveal vision remains available (Nuthmann, 2014). In a third experimental group, we therefore tested more selectively the contribution of foveal vision to attentive object encoding into visual long-term memory by using a small gray-patch scotoma with a radius of 1.5° of visual angle. If parafoveal encoding is sufficient for object encoding into visual memory, change detection performance was expected to be comparable with controls. If foveal vision is a prerequisite to object encoding into long-term memory, however, change detection performance was expected to remain inferior. 

Materials and methods

Participants

Overall, we tested 84 participants. During the initial period of data collection, 20 participants were randomly assigned each to the control (11 women, three left handed, 24 years of average age), full (16 women, one left handed, 24 years of average age), and warp scotoma (12 women, two left handed, 24 years of average age) conditions. The foveal scotoma condition was added post hoc and included 20 new participants initially but was increased by four additional participants because of poor data quality compared with the other experimental conditions (see the Data Exclusion section; 11 women, three left handed, 22 years of average age). All participants were healthy and reported normal or corrected-to-normal visual acuity. Participants were not informed about the purpose of the study until they had completed the experiment. The procedures followed the tenets of the Declaration of Helsinki (World Medical Association, 2013), and the study was approved by the Ethics Committee of the University of Magdeburg. Informed written consent was obtained prior to the experiments. Participants were compensated with course credits or received a compensation of 6 Euros per hour. 

Apparatus

Stimulus presentation and response collection were controlled using the Psychtoolbox (Brainard, 1997; Pelli, 1997) and the Eyelink Toolbox (Cornelissen, Peters, & Palmer, 2002) under Matlab on a PC with a 22-inch Samsung SyncMaster 2233RZ LCD monitor. The monitor was 474 mm (1,680 pixels) wide and 296 mm (1,050 pixels) high, and the vertical refresh rate was 120 Hz. Responses were recorded with a ResponsePixx Handheld five-button response box (VVPixx Technologies Inc.; http://www.vpixx.com). The stimuli were viewed binocularly from a distance of 85 cm, leading to a pixel size of 0.019° of visual angle. The eye position of the left eye was recorded using an Eyelink 1000 Desktop Mount (SR Research Ltd., Mississauga, Ontario, Canada), using corneal reflection and pupil tracking. The temporal resolution of the eye tracker was 1,000 Hz. The gaze data retrieved for the gaze-contingent scotoma simulation were filtered by the heuristic one-sample filter (Cornelissen et al., 2002; Stampe, 1993) implemented in the Eyelink software, removing single-sample noise artifacts. Head movements were minimized by stabilizing the participants' heads using a chin and forehead rest. Participants were tested individually in a dimly lit, sound-attenuated chamber. 

Stimuli

The stimuli used in the experiment were the same as in our previous study (Geringswald et al., 2015). Forty-eight experimental scenes and four additional practice scenes, created with the 3D Traumhaus Designer 9 Premium software (2009), depicted typical indoor environments, such as kitchens or living rooms, and contained 10 to 21 semantically consistent, everyday objects. Test scenes were created by replacing one unique target object with a different, equally sized object from the same semantic category for each scene (token change, see Figure 1 for examples). Target locations were equally distributed across quadrants across all scenes. The scene images extended 22.80° × 17.10° of visual angle and were centered on a gray background. The target objects were on average 2.29° of visual angle wide (SD = 0.98° of visual angle) and 2.49° of visual angle long (SD = 1.25° of visual angle). Masks for each scene were created by randomizing the phase structure of the initial scenes using the Matlab code as provided by Prins (2007). 

Figure 1

 

Visualization of the simulated scotomata. Schematic diagram of an experimental trial and exemplary target objects. (a) Controls explored scenes with unrestricted vision (first column). The full scotoma was a gray patch superimposed on the visual scene (second column). In the warp scotoma simulation (third column), foveated parts of the scene were logarithmically shifted to the center, making them disappear. The full and warp scotoma extended across foveal and parafoveal vision with a radius of 4° of visual angle, smoothly fading out across 1° of visual angle at the borders. The foveal scotoma (fourth column) was a gray patch selectively disrupting foveal vision with a radius of 1.5° of visual angle, smoothly fading out across 0.5° of visual angle at the borders. (b) Each trial consisted of a blank screen (1000 ms), followed by the fixation cross (1000 ms), the initial scene (20 s), the initial scene and the onset cue (250 ms), the initial scene (200 ms), the mask (200 ms), and the test scene with the postcue pointing toward the target object (presented until response). In this example, the target object is the clock, the onset cue is invalid, and the target object is exchanged with a different type of clock in the test scene. (c) Exemplary pairs of target objects from four different scenes with the object presented during free exploration on the left and its replacement in the change condition on the right.

A green disk with a diameter of 2.01° of visual angle served as the onset cue. It was placed on the center of an imaginary rectangle encompassing the target object in the valid cue condition. In the invalid cue condition, the onset cue was randomly placed in one of the three nontarget quadrants. The postcue was a green arrow extending 3.02° × 3.02° of visual angle, pointing unambiguously to the respective target object in the test scene. 

The fixation cross was composed of two black line segments with a line width of 0.63° and a length of 3.14° of visual angle. Additional horizontal and vertical lines extending across the screen and intersecting at the center of the fixation cross were presented to facilitate its visibility. 

The simulated central scotomata (moving mask technique; Rayner & Bertera, 1979) extended 4° of visual angle in radius in the full and warp scotoma conditions and 1.5° of visual angle in radius in the foveal scotoma condition. In the full and foveal scotoma conditions, we used a patch colored the same gray as the background. In the warp scotoma condition, we used image warping and Gaussian blurring shaders that efficiently perform all image manipulation directly on the graphics hardware as implemented in the Psychtoolbox. The pixels within the scotoma were logarithmically shifted toward its center, making the currently fixated stimulus disappear (see Figure 1 for a visualization). For the image warping, we logarithmically transformed the normalized radii of all pixels within a circle of 4° of visual angle around the current fixation location using the formula log(1 + 100 * r)/log(100). The difference between the original and the shifted Cartesian coordinates was then fed to the shader, performing the coordinate shifts and interpolation on the texture. In addition, the texture was blurred with a Gaussian two-dimensional filter with a size of 21 and a standard deviation of 20.5. At the border, the scotomata gradually transitioned from fully visible to fully transparent. For the full and warp scotomata, this was done by setting a circular area with a radius of 3.5° of visual angle to fully visible and applying a circular averaging filter with a diameter of 0.5° of visual angle in the alpha channel of the scotoma. Thus, the fully opaque size of the rendered scotomata extended 3° of visual angle in radius, smoothly fading out at the edges. For the foveal scotoma, the radius set to fully visible extended 1.25° and the circular averaging filter had a diameter of 0.25° of visual angle, respectively, leading to a fully opaque size of 1° of visual angle in radius, smoothly fading out at the edges. 

The position of the artificial scotomata was updated with the gaze coordinates retrieved from the Eyelink 1000 tracking system. The Eyelink 1000 average end-to-end delay was 2.8 ms, and the worst-case latency until the update of the gaze-contingent stimulus manipulation on the monitor was two frames (16.7 ms). No other additional filter algorithms—for example, for fixation or saccade identification—were implemented. Thus, the estimated worst-case delay between actual gaze position and stimulus update was about 20 ms. When no gaze samples were available—for example, because of eye blinks or signal losses—the scotoma remained on the last measured valid gaze position until a new gaze sample became available. 

Procedure

Participants were asked to indicate whether a target object of a previously inspected indoor scene was the same or whether it had changed in an otherwise identical test scene with button presses of the left and right index finger, respectively. They were instructed to memorize as many objects as possible during free exploration and to commit their response only after a green postcue arrow, pointing unambiguously to the target object, had appeared. Participants were further informed precisely about the nature of a potential token change and the corresponding button assignment by means of a printed version of a scene example not used in the experiment beforehand. A reminder of the button assignment was presented on the screen between trials. 

Each session started with a 13-point gaze calibration, followed by a short training on four separate practice scenes, one in each of the cue × test scene conditions, to familiarize participants with the task. The actual experiment included 48 unique indoor scenes, 12 in each of the four cue × test scene conditions. The target appeared three times in each quadrant per condition. Trial sequences were randomized for each participant, and the assignment of the scenes to the conditions was balanced across participants. Every trial started with the presentation of a blank for 1000 ms followed by the fixation cross for 1000 ms, the initial scene presented for 20 s, the onset cue within the scene for 250 ms, the original initial scene again for 200 ms, the mask for 200 ms, and the test scene until the participant responded (Figure 1). Auditory feedback was provided for correct (a 2000-Hz high-pitch tone) and wrong answers (a 500-Hz low-pitch tone). Before each trial, the spatial accuracy of the eye tracker was validated using 13 points. If the average deviation exceeded 1° of visual angle, participants were recalibrated. After each trial, participants were allowed self-determined breaks, and the experimenter proceeded with the subsequent trial when the participant was ready to continue. In the artificial scotoma conditions, the gaze-contingent scotoma simulations were present throughout the whole trial. One session lasted approximately 1 hr. 

Data analysis

Change detection performance analysis and all statistical tests were carried out using R (Version 3.2.3; R Core Team, 2015). Change detection sensitivity was measured as A′ for each participant and each onset-cue condition, typically ranging from 0.5 (chance) to 1 (perfect sensitivity; Snodgrass & Corwin, 1988; see also Stanislaw & Todorov, 1999). The hit rate was calculated as proportion of correct responses when the target changed and the false alarm rate as the portion of errors when it remained the same. Analyses of variance (ANOVAs) were performed using type III sums of squares. Post hoc two-tailed t-test comparisons were adjusted according to Holm (1979), and for planned comparisons between samples, we used Welch's t test. We compared change detection sensitivity between the control, full, warp, and foveal scotoma groups with separate two-way mixed-design ANOVAs with the within-subjects factor cue (invalid, valid) and the between-subjects factor experimental group. 

Gaze data were analyzed with a custom-made Python script applying an adaptive velocity-based algorithm based on the work by Nyström and Holmqvist (2010). We followed the procedures for obtaining saccades and fixations (intersaccadic events that exceeded a duration of 100 ms) as described in our previous work (for example, Geringswald, Herbik, Hoffmann, & Pollmann, 2013). 

Data exclusion

As the reliable presentation of the gaze-contingent simulated scotomata depended on the availability of the gaze coordinates retrieved from the eye tracker, we excluded all trials in which too many gaze samples were missing due to signal loss. We defined a trial as invalid when it met one of the following conditions: more than 20% signal loss during the 20-s exploration of the initial scene, onset-cue, or the following 200-ms presentation of the initial scene or more than 30% signal loss during the test scene. The latter criterion was more liberal because many participants either blinked or looked at the button box before responding, leading to higher signal loss at the end of the trial. In addition, we excluded all trials in which an average spatial accuracy of the eye tracker of at least 1° of visual angle could not be achieved after recalibration. This led to the removal of 15.00% (SD = 11.04%), 12.29% (SD = 14.84%), 12.81% (SD = 13.25%), and 21.27% (SD = 21.06%) of all trials in the control, full, warp, and foveal scotoma groups, respectively. We furthermore excluded all trials in which the response time was shorter than 200 ms or exceeded the mean plus 2.5 standard deviations for each participant. This led to a removal of 2.60% (SD = 2.01%), 2.08% (SD = 2.03%), 2.50% (SD = 1.60%), and 1.65% (SD = 1.84%) of all remaining trials in the control, full, warp, and foveal scotoma groups, respectively. 

Following this procedure, we excluded participants for whom more than half of the trials in one of the cells of the cue × test factor combinations were missing. This led to the exclusion of one participant of the full (overall 58.33% of all data excluded), one in the warp (overall 43.75% of all data excluded), and four in the foveal (overall 93.75%, 43.75%, 37.50%, and 54.17% of all data excluded) scotoma condition. 

After data exclusion, 82.40% (SD = 10.91%) , 87.94% (SD = 10.22%), 86.18% (SD = 11.44%), and 83.96% (SD = 9.82%) of all trials for the control, full, warp, and foveal scotoma conditions, respectively, remained in the analysis, and the number of trials included was not significantly different between experimental groups (one-way mixed-design ANOVA with the between-subjects factor experimental group), F(3, 74) = 1.03, p = 0.38, η²_G = 0.04). 

Results

We sought to investigate whether encoding of objects into visual long-term memory, a process that depends on attentive foveation in normal vision, is impaired when central vision is diminished by a simulated scotoma in healthy observers. For this purpose, we measured change detection performance with three types of scotoma simulations and compared sensitivity for changes with unimpaired controls. In the full scotoma condition, the scotoma was simulated as a clearly visible gray patch, extending across foveal and parafoveal vision, superimposed on natural scene stimuli. In the warp scotoma group, we used a more subtle simulation implemented with image-warping techniques. In the foveal scotoma condition, we specifically tested the contribution of foveal vision to attentive object encoding. Anticipating the main results, visual long-term memory was significantly impaired when central vision, entailing the fovea and parafovea, was impaired. Increased exploration demands due to the loss of central vision were reflected in a reduced number of fixations and larger saccadic amplitudes. Results were highly similar for the full and warp scotoma simulations. Deficits in overall change detection performance were also apparent when foveal vision was disrupted selectively by the smaller foveal scotoma. A decrease in long-term memory performance was marginally significant. 

Change detection sensitivity

Averaged A′ values for all experimental groups are shown in Figure 2. An overall one-way mixed-design ANOVA on A′ with the between-subjects factor experimental group, including controls and all three types of scotoma simulation, was significant, F(3, 74) = 9.06, p < 0.001, η²_G = 0.27. To follow up differences in change detection performance between groups, we first compared controls and the full scotoma group in which potential deficits of object encoding should be most prominent. 

Figure 2

 

Averaged A′ as a function of onset cue validity and experimental group, ranging from 0.5 (chance) to 1 (perfect sensitivity). Error bars depict the standard error of the mean.

Full scotoma

A two-way mixed-design ANOVA on A′ with the within-subjects factor cue (invalid, valid), and the between-subjects factor experimental group (control, full scotoma) revealed a significant main effect of experimental group, F(1, 37) = 20.41, p < 0.001, η²_G = 0.23, that was due to an overall decline in change detection performance when vision was impaired by the scotoma (A′ = 0.82) compared with unimpaired controls (A′ = 0.92). The main effect of onset cue validity was also significant, F(1, 37) = 17.29, p < 0.001, η²_G = 0.17, with higher sensitivity in the valid (A′ = 0.91) than in the invalid (A′ = 0.83) condition. The interaction between experimental group and cue validity did not approach significance, F(1, 37) = 0.64, p = 0.43, η²_G < 0.01. This result is important because it indicates that the onset cues efficiently guided attention in the scotoma group, which is a prerequisite for the assumption that performance in invalid trials is based on object memory. The ANOVA indicated that a simulated scotoma leads to a general drop in performance both for the valid and invalid onset cue condition. To ensure that, specifically, visual memory suffers in the presence of simulated central vision loss, we reanalyzed sensitivity using only data from the invalid onset cue condition that afforded the use of object memory. A Welch two-sample one-tailed t test confirmed that memory performance was significantly impaired in the full scotoma condition (A′ = 0.79) compared with controls (A′ = 0.87), t(32.58) = 2.40, p < 0.05. Figure 2 suggests that, although significantly reduced, memory performance was better than chance in the full scotoma group. This assumption was confirmed by a t test of sensitivity in the invalid onset cue condition against chance, t(18) = 10.30, p < 0.001. 

Invalid onset cues were used to attract attention away from the target position to ensure that change detection had to rely on visual memory. Hollingworth (2004) reported a recency advantage for the two most recently fixated objects, suggesting that these could be held in working memory. Estimates of visual working memory capacity differ across individuals and test paradigms but rarely exceed four items (for a recent review, see Luck & Vogel, 2013). To assess the role of visual working memory to change detection performance in our experiments, we calculated the average median number of intervening fixations on nontarget objects between the last fixation of the target and the offset of the initial scene for all invalid cue trials. The average amount of intervening fixations was significantly greater than three fixations for controls (14.63 fixations), t(19) = 11.47, p < 0.001, and the full scotoma group (13.71 fixations), t(18) = 8.84, p < 0.001. More specifically, up to three intervening fixations were observed in 19.40% and 20.80% of invalid cue trials in the control and full scotoma groups, respectively. Given this relatively high number of trials, it is possible that long-term memory performance, measured with the invalid cue condition, was inflated by visual short-term memory. We therefore recalculated A′ after excluding all trials in which fewer than four intervening fixations were made to ensure that the target was no longer held in visual short-term memory and all trials in which the target was never fixated (3.53 and 8.52% in the control and full scotoma group, resepectively). Change detection sensitivity slightly dropped to 0.86 in controls and did not change numerically in the full scotoma group (A′ = 0.79). A Welch two-sample one-tailed t test confirmed significantly reduced long-term memory performance in the full scotoma condition, t(35.39) = 1.82, p < 0.05. 

Warp scotoma

The warp scotoma was constructed to simulate earlier stages of central vision loss, before the onset of an absolute scotoma. Results of the two-way mixed-design ANOVA on A′ closely mirrored those of the full scotoma condition. A significant main effect of experimental group, F(1, 37) = 19.14, p < 0.001, η²_G = 0.23, revealed significantly impaired change detection performance in the warp scotoma group (A′ = 0.83) compared with controls (A′ = 0.92). Valid onset cues improved performance (A′ = 0.92) compared to with invalid onset cues (A′ = 0.83; main effect cue validity), F(1, 37) = 29.75, p < 0.001, η²_G = 0.26, to a similar extent for controls and the warp scotoma group (nonsignificant interaction), F(1, 37) = 0.14, p = 0.71, η²_G < 0.01. Although memory performance in the invalid cue condition was clearly greater than chance, t(18) = 13.10, p < 0.001, in the warp scotoma group, it was significantly reduced compared with unrestricted viewing (warp scotoma, A′ = 0.79; control, A′ = 0.87; one-tailed), t(36.15) = 2.65, p < 0.01. A comparison of the full and the warp scotoma group with a two-way mixed-design ANOVA on A′ further illustrated the highly similar change detection performance under both simulation types. Overall performance was not significantly different between both scotoma types, experimental group: F(1, 36) = 0.27, p = 0.61, η²_G < 0.01, and the significant main effect of onset cue, F(1, 36) = 12.39, p < 0.01, η²_G = 0.13, confirmed that valid onset cues successfully captured attention to the cued location, leading to improved change detection sensitivity to a comparable extent (nonsignificant interaction), F(1, 36) = 0.20, p = 0.66, η²_G < 0.01. 

Similar to the full scotoma condition, the average amount of intervening fixations in the invalid cue condition was significantly greater than three fixations (18.84 fixations), t(18) = 8.81, p < 0.001. The recalculated A′ after excluding 19.79% of trials in the invalid cue condition in which fewer than four intervening fixations were made and 15.42% of trials in which the target was never fixated slightly dropped to 0.77 and was significantly reduced compared with controls (Welch two-sample one-tailed t test), t(28.82) = 1.91, p < 0.05, confirming visual long-term memory impairment. 

Foveal scotoma

The foveal scotoma simulation aimed at testing specifically the contribution of attentive foveating while leaving parafoveal vision intact. The overall change detection sensitivity was significantly impaired even when the scotoma simulation spared parafoveal vision, as indicated by the significant main effect of experimental group, F(1, 38) = 10.92, p < 0.01, η²_G = 0.13 (foveal scotoma, A′ = 0.87; control, A′ = 0.92). Similar to the full and warp scotoma analyses, performance was significantly improved under valid onset cues (A′ = 0.93) compared to with invalid onset cues (A′ = 0.85; main effect cue validity), F(1, 38) = 34.82, p < 0.001, η²_G = 0.30, and this improvement was comparable between the foveal scotoma group and controls (nonsignificant interaction), F(1, 38) = 0.80, p = 0.38, η²_G < 0.01. Visual memory measured with the invalid cue condition was well above chance, t(19) = 24.94, p < 0.001, in the foveal scotoma group. A Welch two-sample one-tailed t test comparing the foveal scotoma group to controls on invalid cue trials indicated a trend toward decreased memory performance with the foveal scotoma that was marginally significant (foveal scotoma, A′ = 0.83; control, A′ = 0.87), t(33.49) = 1.57, p = 0.06. 

The average amount of intervening fixations in the invalid cue condition was significantly greater than three fixations (11.73 fixations), t(19) = 9.63, p < 0.001. To test specifically the contribution of long-term visual memory, we excluded 24.43% of trials from the cue invalid condition in which fewer than four intervening fixations were made and 1.76% of trials in which the target was never fixated. The recalculated A′ slightly dropped to 0.82. The Welch two-sample one-tailed t test comparing long-term memory performance to controls again indicated a nonsignificant trend toward decreased long-term memory performance with the foveal scotoma, t(34.74) = 1.34, p = 0.09. To test whether visual long-term memory performance was better with the foveal than with the full scotoma, we ran an additional Welch two-sample one-tailed t test, yielding no significant differences (A′ = 0.79), t(29.77) = 0.85, p = 0.20. 

Scene exploration

For the analysis of scene exploration, we aggregated the gaze parameters measured during the presentation of the initial scene, in which objects had to be memorized by the participants, over onset cue validity and performed one-way ANOVAs with the between-subjects factor experimental group on the number of fixation, fixation duration, and saccade amplitude. The impact of the scotoma simulations on the number of fixation, fixation duration, and saccade amplitude can be seen in Figure 3. 

Figure 3

 

Averaged number of fixations (left), fixation duration (middle), and saccade amplitude (right) during initial scene exploration as a function of experimental group. Error bars depict the standard error of the mean.

The main effect of experimental group was significant for fixation number, F(3, 74) = 4.04, p < 0.05, η²_G = 0.14. The warp scotoma led to significantly reduced fixations (warp scotoma: 54.35 fixations), t(36.52) = 2.81, p < 0.05, compared with controls (61.58 fixations). Although the number of fixations was similarly increased for the full scotoma (54.93 fixations), the comparison with controls failed to reach significance, t(35.00) = 2.40, p = 0.09, most likely because of slightly greater variability between participants. Participants in the foveal scotoma group made a comparable amount of fixations as controls (61.35 fixations), t(37.02) = 0.08, p = 1. Differences in fixation numbers did not reach significance comparing scotoma groups with each other (all absolute ts < 2.51, ps > 0.08). Numerically, fixation durations were slightly increased in the full (300 ms), warp (300 ms), and, to a lesser degree, foveal (268 ms) scotoma simulations compared with controls (262 ms); however, none of these differences was statistically reliable, F(3, 74) = 2.17, p = 0.10, η²_G = 0.08. The impact of the scotoma simulations on eye movements was most evident in the saccade amplitudes, F(3, 74) = 39.44, p < 0.001, η²_G = 0.62. Saccade amplitudes were significantly greater during exploration with the full scotoma (5.18° of visual angle), t(26.43) = −9.58, p < 0.001; warp scotoma (4.78° of visual angle), t(27.88) = −8.50, p < 0.001; and foveal scotoma (3.72° of visual angle), t(36.53) = −4.94, p < 0.001, compared with controls (2.89° of visual angle). Saccade amplitudes were comparable between the full and warp scotoma, t(35.71) = 1.37, p = 0.18, and were significantly greater in both simulations compared with the foveal scotoma (all ts > 4.52, ps < 0.001). 

Discussion

We investigated whether a simulated central scotoma that forces observers to carry out all visual processing with peripheral vision impairs visual long-term memory for everyday objects in natural scenes. This research question was inspired by previous studies demonstrating that visual long-term memory for an object depends on previous fixation (Hollingworth, 2006)—and concurrent attending (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995)—of the object. Therefore, it was surprising that in a recent study on patients suffering from natural central vision loss, their explicit visual long-term memory was as accurate as for healthy, age-matched controls (Geringswald et al., 2015). We had reasoned that intact attentive object encoding was due to the successful development of saccadic rereferencing to an extrafoveal retinal location (von Noorden & Mackensensen, 1962; White & Bedell, 1990; Whittaker et al., 1991). Under optimal conditions, the development of saccadic rereferencing in observers confronted with central scotoma simulation takes several hours (Kwon, Nandy, & Tjan, 2013; Walsh & Liu, 2014). To investigate the contribution of saccadic rereferencing to successful extrafoveal object encoding into long-term memory, we thus examined change detection performance using the same paradigm as in our patient study, this time in normal-sighted observers who were for the first time confronted with a gaze-contingent scotoma simulation. 

The main finding was that visual long-term memory for objects was significantly impaired under simulated central vision loss. Change detection sensitivity in the invalid onset cue condition was used to measure memory performance. Cues that validly indicated the target location before a potential change increased change detection performance to a similar extent as in controls, showing that observers oriented their attention to the cued locations. This was a prerequisite to interpret change detection in the invalid cue trials as based on memory, because attention was successfully captured by a nontarget region within the scene. To ascribe change detection performance to visual long-term memory, we further needed to rule out a contribution of visual working memory. Across all experimental groups, 12 or more fixations were made on average between the last fixation of the target and the offset of the initial scene in invalid cue trials, and in approximately 70% of all trials, four or more intervening fixations on nontarget objects were made before the memory test. Because this clearly exceeds the capacity of visual short-term memory (Luck & Vogel, 2013), change detection must have relied on long-term memory in these trials. To completely rule out a working memory contribution to change detection performance, we ran an additional analysis excluding trials with fewer than four intervening fixations. The results of these analyses yielded a clear pattern. The full and warp scotoma simulations led to reduced change detection compared with controls. The smaller foveal scotoma also led to numerically reduced change detection, but this difference was less reliable, expressed as a marginally significant trend. The smaller scotoma interfered much less with scene exploration than the larger scotomata. This is shown by the gaze parameters, which are more comparable between the foveal scotoma and controls than both other scotoma groups. The effect of the foveal scotoma on long-term memory, however, remained somewhat inconclusive. A potential decrease of long-term memory compared with controls was not statistically reliable when foveal vision was knocked out selectively, suggesting that the allocation of attentive object encoding to parafoveal vision was more efficient than when objects had to be encoded in peripheral vision with the larger scotomata. This pattern is in agreement with a recent study in which foveal vision was not necessary to efficiently locate and verify a verbally cued target object in natural scenes, and costs on target verification emerged only when parafoveal vision was degraded in addition (Nuthmann, 2014). On the other hand, memory performance was not significantly better with the foveal compared with the full scotoma, knocking out foveal and parafoveal vision. The loss of foveal vision alone might therefore not be without problems in untrained observers, going along with increased demands on attentive object encoding. Long-term memory performance with the foveal scotoma might thus take an intermediate position between unimpaired viewing and impairment of foveal and parafoveal vision by the larger scotomata. Further studies incorporating more sensitive within-subject designs and a larger range of scotoma sizes, testing attentive object encoding at different eccentricities, would be desirable to investigate whether a specific minimum scotoma size can be identified that leads to clear long-term memory deficits or whether the relationship between scotoma size and quality of long-term memory follows a more gradual pattern. 

The simulation of small scotomata is not without problems. To ensure that the scotoma covered foveal vision, we included only trials in which the average spatial accuracy of the measured gaze did not exceed 1° of visual angle. In the larger scotomata, deviations in this range are less problematic because the scotoma would still cover foveal vision entirely. In the foveal scotoma group, on the other hand, partial foveal processing of the objects may occur as soon as the local spatial deviation exceeds 0.5° of visual angle, potentially inflating visual long-term memory performance. The increased saccade amplitude compared with controls, however, indicated that observers targeted saccade locations outside the scotoma in agreement with previous reports (Nuthmann, 2014, for a review), suggesting that the scotoma reliably covered foveal vision. 

Compared with controls, the large simulated scotomata led to a reduced average number of fixations during exploration and an absolute increase of fixation duration. This finding contrasts with the normal fixation patterns that we observed in patients suffering from natural central vision loss (Geringswald et al., 2015) and supports the hypothesis that intact long-term memory in AMD patients was mediated by a successful adaptation of saccadic rereferencing to an extrafoveal PRL. Although recent research has demonstrated that such a PRL can similarly be developed by healthy observers after a few hours of visual search training (Kwon et al., 2013; Walsh & Liu, 2014), it is highly unlikely that this was the case in our current study. Training of saccadic rereferencing requires that the scotoma and its borders are clearly visible to the observer. If saccadic rereferencing had taken place in our study, gaze control and, potentially, memory performance should have been better in the clearly visible full scotoma condition compared with the more subtle warp scotoma simulation. This was, however, not the case. Memory performance as well as gaze parameters were virtually identical for both large scotoma simulations. In fact, the reduced number of fixations rather suggests that saccades were inhibited in a top-down controlled fashion. As long as saccadic rereferencing has not yet developed, the automatic process of foveating an attended stimulus (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Theeuwes, Kramer, Hahn, & Irwin, 1998) quickly becomes a disadvantage during scene exploration with an artificial scotoma because the attended stimulus will be covered by the scotoma after each eye movement. The measured impairment of attentive object encoding may therefore be a result of active saccade inhibition during object exploration. This reasoning is in line with the finding that active saccade inhibition to a target location impairs attentive stimulus processing (Dhawan, Deubel, & Jonikaitis, 2013) as well as change detection sensitivity in the flicker paradigm (Hollingworth et al., 2001). 

In contrast to the gray-patch scotoma simulations, the real-time logarithmic shift of visual information in the warp scotoma simulation may have produced motion artifacts during fixation, for example, because of spatial jitter in eye movement measurement. We tried to counteract such effects by blurring visual information within the scotoma with a Gaussian kernel to reduce sharp motion transients. Potential motion artifacts may have interfered with object encoding outside the scotoma due to an increased foveal information load compared with the full scotoma condition. However, neither the virtually identical memory performance under both simulation types nor the saccade amplitudes targeting, on average, locations well outside the scotoma indicate that the warping procedure influenced exploration or encoding in a different manner than the full scotoma. 

Finally, although long-term memory performance was significantly impaired by the large central scotoma simulations extending across foveal and parafoveal vision, participants were nevertheless able to detect object changes well above chance. This shows that foveation is not a necessity for attentive object encoding. However, the quality of long-term memory suffers when central vision becomes unavailable. 

Conclusion

Can attentive encoding of objects into visual long-term memory be successful if central vision becomes unavailable? Intact explicit long-term memory in patients suffering from long-standing central vision loss suggests a flexible deployment of attentive object encoding to peripheral vision. When central vision is diminished in unexperienced observers, however, attentive object encoding with peripheral vision remains limited. These results demonstrate the importance of central vision for attentive object encoding on one hand. On the other hand, they suggest that the successful development of saccadic rereferencing to an extrafoveal location may be indispensable for normal visual memory function in the presence of central vision loss. 

Acknowledgments

This work was supported by the Deutsche Forschungsgemeinschaft (PO548/14-1). We thank Nico Adelhöfer and Marc Rose for assistance in data acquisition. 

Commercial relationships: none. 

Corresponding author: Franziska Geringswald. 

Email: [e-mail]. 

Address: Otto-von-Guericke-Universität Magdeburg, Institut für Psychologie II, Magdeburg, Germany.