Spatiotopic updating across saccades in the absence of awareness

Abstract

Despite the continuously changing visual inputs caused by eye movements, our perceptual representation of the visual world remains remarkably stable. Visual stability has been a major area of interest within the field of visual neuroscience. The early visual cortical areas are retinotopic-organized, and presumably there is a retinotopic to spatiotopic transformation process that supports the stable representation of the visual world. In this study, we used a cross-saccadic adaptation paradigm to show that both the orientation adaptation and face gender adaptation could still be observed at the same spatiotopic (but different retinotopic) locations even when the adapting stimuli were rendered invisible. These results suggest that awareness of a visual object is not required for its transformation from the retinotopic to the spatiotopic reference frame.

Introduction

Despite the continuous movements of the eyes and body, our visual world remains stable. In other words, an object could be imaged at very different positions on our retina (when eyes move), but our perceptual representation of that object remains stable in the visual world. Key to this visual stability is the transformation of visual object representation from the retinotopic (coordinates centered on the retina) to spatiotopic (coordinates centered on the outside world) reference frame across saccades (Cicchini, Binda, Burr, & Morrone, 2013; Crapse & Sommer, 2012; Fabius, Fracasso, Nijboer, & Van Der Stigchel, 2019). Previous studies showed that neurons in the extrastriate visual cortex (such as V4) and the lateral intraparietal cortex could temporarily remap their receptive fields to compensate for an impending saccadic eye movement (Duhamel, Colby, & Goldberg, 1992; Tolias, Moore, Smirnakis, Tehovnik, Siapas, & Schiller, 2001; Wurtz, Joiner, & Berman, 2011). Meanwhile, other studies also indicated explicit spatiotopic neural representation in middle temporal area and parietal areas (D'Avossa, Tosetti, Crespi, Biagi, Burr, & Morrone, 2007; Duhamel, Bremmer, BenHamed, & Graf, 1997), although this has remained a topic of debate (Gardner, Merriam, Movshon, & Heeger, 2008; Merriam, Gardner, Movshon, & Heeger, 2013). In any case, either by continuously updating or remapping the retinotopic maps, or by transforming the retinotopic representation to explicit spatiotopic representation, our brain would be able to keep track of the salient objects in the scene and achieve visual stability. 

While the input visual information during saccades is suppressed, our conscious representation of the visual scene across saccades seems to be smooth and continuous, yet we typically do not keep track of the whole visual scene. Selective attention is one of the potential mechanisms to help maintain visual stability (Crespi, Biagi, d'Avossa, Burr, Tosetti, & Morrone, 2011; Melcher, 2008; Melcher, 2011; Szinte, Jonikaitis, Rangelov, & Deubel, 2018). Attentional selection contributes to visual stability by restricting information processing to salient or task-relevant objects. Thus the transsaccadic spatiotopic updating of salient objects would allow the brain to track important features or items in the scene. With multiple objects, the allocation of the selective attention would influence the spatiotopic updating and previous results showed that unattended stimuli could induce decreased but still measurable adaptation aftereffect in the spatiotopic location (Melcher, 2009; Melcher & Colby, 2008). However, although attention plays an important role in gating information to awareness, attention and awareness are not the same. Here we ask if the visual stimulus is invisible, could the spatiotopic updating process still happen? In other words, is spatiotopic updating so critical to our visual function that this process occurs even when we are not aware of the objects in the visual scene? Previous studies have shown that attention can be drawn to unconscious stimuli (Cohen, Cavanagh, Chun, & Nakayama, 2012; Jiang, Costello, Fang, Huang, & He, 2006), and the unconscious stimuli can still be processed to a certain level in the neural pathway (Axelrod, Bar, & Rees, 2015; Fang & He, 2005; Lin & He, 2009; Sterzer, Stein, Ludwig, Rothkirch, & Hesselmann, 2014). Thus the key question addressed in this study is the following: is awareness of a visual object necessary for its reference frame transformation from retinotopic to spatiotopic across saccades? 

Retinotopic versus spatiotopic representations are dissociated by object locations before and after saccadic eye movements. To investigate the question raised above, in addition to using eye movement that dissociates the object's retinotopic and spatiotopic locations, we also need a tool to probe the neural representation in the corresponding locations before and after the saccade. Adaptation paradigms are effective in studying neural representations in different reference frames because they allow a relatively long temporal delay in measuring the adaptation effect, so that if an object has achieved representation at the spatiotopic reference frame, we would expect to see adaptation effect when the test probe is presented at the same spatiotopic location (even if its retinotopic location is different from that of the adapting stimulus). Adaptation paradigms also have the advantage of being able to target specific levels of neural representation in the visual pathway by selectively adapting to properties with different levels of complexity (Boynton & Finney, 2003; Clifford & Rhodes, 2005; Georgeson, 2004; Kohn, 2007; Rushton, 1965). In our study, we took advantage of two forms of visual aftereffects that were previously shown capable of generating spatiotopic aftereffects, namely the tilt aftereffect (TAE) and the face gender aftereffect (FGAE) (Cha & Chong, 2014; He, Mo, & Fang, 2017; He, Fritsche, & Lange de, 2018; Melcher, 2005, 2009; Nakashima & Sugita, 2017; Wolfe & Whitney, 2015; Zimmermann, Morrone, Fink, & Burr, 2013; Zirnsak, Gerhards, Kiani, Lappe, & Hamker, 2011). We first verified that both TAE and FGAE could be observed at the spatiotopic location, which implied that the adapting stimulus had undergone retinotopic-to-spatiotopic transformation. 

Next, to render the adapting stimulus invisible so we could investigate whether the aftereffects could still be observed at the spatiotopic location from the invisible adaptor, we adopted the continuous flash suppression (CFS) approach. CFS is an effective way to render adapting stimuli in one eye invisible by presenting a stream of rapidly changing noise to the other eye. CFS has the advantage of achieving prolonged suppression duration and being less influenced by visual properties of the to-be-suppressed stimulus (Fang & He, 2005; Kim & Blake, 2005; Tsuchiya & Koch, 2005). There is evidence that different types of adaptation aftereffects are differentially influenced by interocular suppression. Not surprisingly, more complex stimulus properties such as face gender and identity information are more vulnerable to suppression, compared with simple stimulus features such as flicker, motion, or orientation (Alais & Melcher, 2007; Kaunitz, Fracasso, & Melcher, 2011; Tsuchiya & Koch, 2005; Yang, Hong, & Blake, 2010). In this study, we investigated the role of awareness in the retinotopic to spatiotopic reference frame transformation, by using CFS to suppress the awareness of the target visual objects. Our results show that for visual targets not consciously perceived, both local orientation information and face gender information could be transformed from retinotopic to spatiotopic reference frame. 

Methods

Participants

Twelve participants (seven females, mean age = 23.2) took part in the main experiment. Half of the participants (n = 6) also took part in the eye movement recording experiment. All participants had normal or corrected-to-normal vision. All participants provided written informed consent and were paid to take part in the study, which was approved by the Institutional Review Panel at the Institute of Biophysics, Chinese Academy of Sciences. 

Stimuli

Stimuli were displayed on two synchronized 23.8-inch LCD displays (Dell U2414H, 1920*1080 at 60 Hz refresh rate) and viewed from a distance of 80 cm through stereo mirrors. All visual stimuli were generated using MATLAB Psychophysics Toolbox (Brainard, 1997). The presentation of a frame (18 × 12 degrees of visual angle (dva)) with dashed lines facilitated stable convergence of images in two eyes and also provided background coordination information for the saccade task. A cross (0.56 × 0.56 dva) presented in the left or right part of the frame served as the fixation point. 

The adaptor for tilt aftereffect was a tilted (±15°) Gaussian-windowed sinusoidal luminance Gabor that subtended 5 dva (Figure 1B). The frequency of the Gabor was 0.8 c/deg. The test stimuli were similar to the adaptor, tilted from −4.5° to 4.5°. For the face adaptation, male and female faces were used as adaptors subtending 5 dva. The morphs were generated using Morph 3.0 (Gryphon Software, San Diego, CA, USA) with 100 intervening morphs. Morph number 50 was regarded as a neutral center point within the morphing space. 

Figure 1.

 

Experiment paradigms for different conditions. (a) The locations of adaptation and test stimuli before and after the saccade. The cross presents the fixation point. The black arrow represents the saccade direction (from left to right). A, adaptation location (also the full adapt test location); S, spatiotopic location; R, retinotopic location; Cs, control spatiotopic location; Cr, control retinotopic location. (b) Adaptor and test stimuli for tilt aftereffect and face gender aftereffect. (c) Time sequences in the experiment. The adapter was presented for 2 s (top-up adaptation) after 0.8 s fixation in the left cross. After a 0.8 s preview of the right cross, participants need to saccade to the right cross after the extinction of the left cross. Then a test stimulus was present for 0.1s in one of four locations randomly.

Procedure

There were two conditions, visible and invisible, for each adaptation stimulus type in separate sessions to avoid task complexity. A total of 2688 trials were obtained for each participant across all conditions. In the visible condition, after the initial adaptation period (25 seconds), the participants first fixated at the left cross for 0.8 second. Then the top-up adaptor was presented to the participant's non-dominant eye for 2 seconds at the upper-middle location of the monitor. Following the 0.8 second (SD = 0.1 second) preview of the next fixation cross on the right side, while still maintaining fixation on the left cross, the participants made a saccade to the right fixation cross (6 dva from the left cross) prompted by the extinction of the current fixation cross on the left. Then a test probe was presented for 100 ms at one of four possible locations (retinotopic, spatiotopic, retinotopic-control, or spatiotopic-control) pseudo-randomly selected with equal probability (Figure 1). Participants needed to report the direction of tilt of the Gabor or the gender of the face. 

The invisible condition was the same as the visible condition, except that dynamic Mondrian patterns (10 Hz, subtending 5 dva) were simultaneously presented to participants’ dominant eye in both initial and top-up adaptation periods. To ensure that the dynamic Mondrian patterns could effectively suppress the adapting stimuli (Stein & Sterzer, 2014; Yang, Brascamp, Kang, & Blake, 2014), we first presented the adaptor at 80% contrast to test whether it could be suppressed in both initial adaptation (25 seconds) and 20 trials of top-up adaptation (2 seconds each trial) for each participant. Participants were asked to press a button if they detected the adaptor in the initial adapting period or in any trial. If the adaptor broke the suppression in more than 5% of trials, we then reduced the contrast of the adaptor by 5% and tested again. This process resulted in the adaptor been seen under CFS suppression in no more than 5% of the trials. The contrast of adaptor was recorded and used in the formal experiment (average contrast for Gabor patch: 79.7% ± 0.8%; average contrast for face: 78.3% ± 2.3%). During the adaptation period, if participants could see the Gabor or tell the gender of the face, they pressed a button (spacebar) to indicate the Mondrian patterns did not fully suppress awareness of the adaptor. These trials were excluded from further analysis. 

In addition, we included a full adaptation condition in which participants maintained the fixation on the left without making a saccade during the whole period with the test stimulus presented in the same location as the adaptor. The logic of the experiment is that if an aftereffect could be observed at the spatiotopic location, then it would imply that the adapting stimulus had achieved spatiotopic representation, in other words, had undergone retinotopic to spatiotopic transformation. 

Eye movement measurements

To verify that the participants were generally able to follow the instructions, half of the participants (n = 6) took part in an eye movement experiment, which was the same as the main experiment, but half in the number of trials (1344 trials). Eye movements of the participants were monitored by the Eyelink 1000 Plus system (SR Research), which sampled gaze positions with a frequency of 1000 Hz. Only the left eye was recorded. The system detected a start and an end of a saccade when eye velocity exceeded or fell below 22°/s and acceleration was above or below 3800°/s². At the beginning of each session during the experiment, a nine-point calibration and validation procedure was conducted. If the calibration did not meet the defined requirements, calibration was repeated until successful. The averaged horizontal eye positions over the time course of the trial for each participant were showed in Supplementary Figure S1. The eye position traces were aligned with the midpoint of the saccade. 

Analysis

MATLAB was used to analyze the data. The psychometric response curve was fitted with a Bayesian-based cumulative Gaussian function (psignifit toolbox in MATLAB) (Schütt, Harmeling, Macke, & Wichmann, 2016) to measure the aftereffects. The magnitude of the TAE was defined as half the difference of tilt to annul the effects of adapting clockwise, compared with counter-clockwise gratings. The FGAE was calculated with a similar method. Example fitting results for one participant were shown in Figure 2. It showed the tilt aftereffect in four different locations when the adaptor was visible. One-half of the distance between two fitted curves was the measured magnitude of the aftereffect. 

Figure 2.

 

Fitted curves of Tilt Aftereffect results for one participant in four test locations without CFS stimuli: (a) spatiotopic, (b) retinotopic, (c) control-spatiotopic, and (d) control-retinotopic location. Similar results were found for the other 11 participants. Red and blue curves represent clockwise and counterclockwise adaptors, respectively. The vertical bars represent the estimated 50% threshold.

Results

Participants were well able to maintain their fixation and execute the required eye movements (Supplementary Figure S1). The mean distances between eye position and fixation center were 0.11° (SD = 0.09°) and 0.36° (SD = 0.28°) before and after saccades. Saccades, which need to be executed within 500 ms after the extinction of the left fixation cross, were on average accurate and prompt, with 143.6 ms (SD = 117.3) mean saccade latency. In only 1.15% of all trials, the saccades were not executed before the test stimulus presentation. Because of the very small proportion of these delayed saccades, our results were not affected by whether we exclude these trials or not in the following statistical analysis. 

For participants who finished separate sessions with and without eye movement recording, no significant differences were found between the two sessions (dependent sample t-tests for all conditions, p > 0.05, Supplementary Figure S2). There were also no significant differences between participants with and without eye movement recording (independent sample t-tests for all conditions, p > 0.05, Supplementary Figure S3). Thus we combined these data in the further statistical analysis. 

The strength of TAE and FGAE for each participant was calculated as half of the difference on the x-axis between the two points of subjective equality (PSEs) based on the psychometric functions following adaptation in two opposite orientation (TAE) or gender (FGAE) (see Figure 2 for an example). Statistics were then performed on the group data. 

We performed two-way ANOVAs to examine the effects of two factors (two levels of adaptor awareness and five different adapt-test relationships) on the magnitude of TAE and FGAE. For the TAE, both the main effects of adaptor awareness and adapt-test relationship are significant (adaptor awareness: F(1,11) = 61.48, p < 0.001, \(\eta _p^2 = 0.848\); adapt-test relationship: F(4, 44) = 61.61, p < 0.001, \(\eta _p^2 = 0.849\)). The interaction between adaptor awareness and the adapt-test relationship was also significant (F(4, 44) = 11.71, p < 0.001, \(\eta _p^2 = 0.516\)), indicating that the impact of adaptor awareness depended on the relationship between adapt-test locations. Post hoc analysis showed that the TAE in spatiotopic location is significantly larger than the control-spatiotopic location in both visible (t = 5.91, p < 0.001) and invisible condition (t = 3.26, p < 0.01), suggesting the existence of a spatially specific adaptation effect at the spatiotopic location, regardless of awareness state of the adapting stimulus. 

For the FGAE, again both the main effects of adaptor awareness and adapt-test relationship are significant (adaptor awareness: F(1, 11) = 14.49, p = 0.003, \(\eta _p^2 = 0.568\); adapt-test relationship: F(4, 44) = 12.15, p < 0.001, \(\eta _p^2 = 0.525\)). However, the interaction effect between adaptor awareness and adapt-test relationship is not significant (F(4, 44) = 1.83, p = 0.141, \(\eta _p^2 = 0.142\)), suggesting that the impact of adaptor awareness was not dependent on the relationship between adapt-test locations. Post hoc analysis showed that the FGAE in spatiotopic location is not significantly larger than that in the control-spatiotopic location in both visible and invisible conditions (p > 0.05). 

For the visible condition (without CFS), the one-sample t-tests with Holm correction (N = 10, 5 locations* 2 state awareness (with(out) CFS) for TAE and FGAE respectively) indicate that both TAE and FGAE could be induced at the spatiotopic location (TAE: M = 0.93°, p < 0.001; FGAE: M = 7.56%, p < 0.001), and not surprisingly, at the retinotopic location (TAE: M = 2.26°, p < 0.001; FGAE: M = 16.67%, p < 0.001). Results show that the TAE and FGAE partially transfer to control-retinotopic location (TAE: M = 0.48°, p < 0.01; FGAE: M = 5.46%, p < 0.05) and control-spatiotopic location (TAE: M = 0.27°, p < 0.05; FGAE: M = 7.85%, p < 0.01). The full adaptation condition (no saccade) reveals the strength of the TAE (M = 2.38°, p < 0.001) and FGAE (M = 10.31%, p < 0.001) in the classic condition (Figure 3, left panels) (also see normalized results in Supplementary Figure S4). 

Figure 3.

 

Adaptation aftereffects (a, TAE; b, FGAE) for the No-CFS and CFS conditions in different locations. Average results from 12 participants show significant TAE and FGAE effects in spatiotopic locations when the adaptors were visible. The effect partially transferred to the two control locations. For invisible adaptor, robust adaptation effects were observed in spatiotopic locations, but not in two control locations. Error bars show ± 1 SE of the mean. Multiple comparisons were Holm corrected. * Adjusted p < 0.05; ** adjusted p < 0.01; *** adjusted p < 0.001.

For the invisible condition (with CFS), interestingly, results show that both stimuli could still generate robust aftereffects at the retinotopic (TAE: M = 0.85°, p < 0.02; FGAE: M = 6.62%, p < 0.02) and spatiotopic locations (TAE: M = 0.25°, p < 0.02; FGAE: M = 3.88%, p < 0.03), whereas no aftereffect was observed at the control-spatiotopic location (TAE: M = 0.02°, p = 0.97; FGAE: M = 1.09%, p = 0.88) nor at the control-retinotopic location (TAE: M = 0.02°, p = 0 .97; FGAE: M = 0 .19%, p = 0.88). For the full adaptation condition without saccade, significant TAE and FGAE were observed (TAE: M = 0.69°, p < 0.01; FGAE: M = 6.83%, p < 0.05) (Figure 3, right panels). Comparing with results in the visible adaptation condition, the spread of aftereffects to control locations did not occur when participants had no awareness of the adaptation stimulus; however, the adaptation effect remained robust at the spatiotopic location. 

Discussion

We used the adaptation paradigm to investigate whether visual objects could be transformed from retinotopic to spatiotopic reference frame while observers were not aware of their presence. We first established that both the orientation and the face gender adaptation were capable of generating tilt and face gender aftereffects, respectively, when tested at different retinotopic but the same spatiotopic location. The critical observation is that when the adapting stimulus was rendered invisible, both aftereffects could still be observed at the spatiotopic location. 

In contrast to awareness being not necessary for the spatiotopic updating, the buildup of spatiotopic neural representation requires spatial attention (Crespi et al., 2011; Melcher, 2008; Melcher, 2009; Melcher, 2011; Melcher & Colby, 2008; Szinte et al., 2018). Crespi et al. (2011) found that when participants were conducting a demanding attention task on the foveal stimuli, BOLD responses evoked by moving stimuli unrelated to the fovea task were mainly tuned in retinotopic coordinates. But the BOLD responses were tuned in spatiotopic coordinates when subjects could easily attend to the motion stimuli. In our study, when the adaptors were visible, the spatial attention to the adaptor location might help the buildup of the adaptation effect in the spatiotopic location. Previous studies showed that the stimuli under CFS could still influence spatial attention (Jiang et al., 2006), which may enable our observation that both TAE and FGAE could occur at the spatiotopic location without visual awareness. 

Attentional facilitation to the saccade destination may also influence the adaptation effects. In our study, the saccade target did not overlap with test locations and eccentricity-matched control locations were included for both spatiotopic and retinotopic conditions. Thus the possible effects of attention facilitation to the saccade target were avoided because of the equal probability of test presence among four different locations (Afraz & Cavanagh, 2009). Besides, because the adaptation and test stimuli were always presented in the periphery, there was no switch between foveal and peripheral locations in testing the aftereffects, presumably generating more stable aftereffect measurements. 

It has been debated whether visual feature information or just the spatial information is transferred in the trans-saccadic remapping. Recent studies demonstrated that feature information like orientation (Ganmor, Landy, & Simoncelli, 2015; Wutz, Drewes, & Melcher, 2016; Zimmermann, Weidner, & Fink, 2017), shape (Demeyer, De Graef, Wagemans, & Verfaillie, 2009), motion (Fabius, Fracasso, & Van Der Stigchel, 2016; Fracasso, Caramazza, & Melcher, 2010; Melcher & Fracasso, 2012; Turi & Burr, 2012), and facial expressions (Wolfe & Whitney, 2015), could be remapped across saccades. Our results provide further support that transsaccadic remapping takes place at the feature level. The process of feature remapping would enable the construction of spatiotopic representations of visual features. 

The time course of spatiotopic updating might also influence the adaptation effects among different locations across saccades (Burr, Tozzi, & Morrone, 2007; Melcher & Morrone, 2003). There is evidence showing that the preview duration is a necessary requirement for the spatiotopic representation to fully build up (Golomb, Marino, Chun, & Mazer, 2011; Golomb, Nguyen-Phuc, Mazer, McCarthy, & Chun, 2010; Golomb, Pulido, Albrecht, Chun, & Mazer, 2010; Mathôt & Theeuwes, 2010; Morrone, Cicchini, & Burr, 2010; Zimmermann, Morrone, & Burr, 2015, 2014; Zimmermann, Morrone, Fink, & Burr, 2013). Thus the relatively long target-preview duration (0.8 second) used in our study likely contributed to a stronger object representation at the spatiotopic location. It is also possible that spatiotopic updating may have different temporal dynamics for different stimulus types and states of awareness. For example, a recent study using rotating motion illusion suggested that spatiotopic updating could occur rapidly (e.g., within 150 ms) (Fabius et al., 2019). 

Recent functional magnetic resonance imaging adaptation studies showed reduced BOLD response in the extrastriate visual cortex when two repeated gratings were presented at the same spatiotopic location before and after a saccade (Dunkley, Baltaretu, & Crawford, 2016; Fairhall, Schwarzbach, Lingnau, Van Koningsbruggen, & Melcher, 2017; Zimmermann, Weidner, Abdollahi, & Fink, 2016). These repetition suppression effects indicate a transfer of representation (and consequently adaptation effect) from retinotopic to spatiotopic reference frame, which is in accord with our finding of spatiotopic adaptation effect with visible grating adaptors. 

Our results show that when the adaptor was visible, a robust tilt aftereffect could be observed in the spatiotopic location (with the largest effect in the retinotopic location and smaller effects in the control locations). For the face gender adaptation, the magnitude of aftereffects was similar among the spatiotopic and other two control locations (smaller than the retinotopic location), which is consistent with a previous study that showed no significant difference between spatiotopic and control locations (Afraz & Cavanagh, 2009). Such results indicate that, in addition to the transformation from retinotopic to spatiotopic reference frame, when the adapting face was visible, there was a spatially non-local adaptation effect. In other words, there was a more spatially invariant representation when an object was consciously perceived, in contrast to a more spatially local object representation in the absence of awareness. The role of awareness in spatially invariant representation was also revealed for object viewpoint in a recent study using Necker cubes as stimuli (Cho & He, 2019). With awareness, the spatially nonspecific effect was also observed for TAE, but quite a bit weaker, presumably because of the intrinsic local nature of orientation processing in the visual cortex. 

More interestingly, when the adaptor was rendered invisible, our results show that there was still a significant representation of the adaptor at its spatiotopic location for both orientation and face gender information, but not in the two eccentricity-matched control locations. In other words, both local orientation and face gender information could be transformed from the retinotopic to spatiotopic reference frame without awareness. The spatiotopic updating of an object from its retinotopic reference frame, a process that is critical for achieving a stable perceptual representation of the visual world, can occur even when the object is not explicitly perceived. 

Acknowledgments

Supported by Strategy Priority Research Program of Chinese Academy of Science (XDB32020200), Key Research Program of Frontier Sciences Chinese Academy of Sciences (CAS; KJZD-SW-L08), Beijing Municipal Science & Technology Commission (Z181100001518002), Beijing Advanced Discipline Fund, National Natural Science Foundation of China (32000787), and Youth Innovation Promotion Association CAS (2021089). 

Commercial relationships: none. 

Corresponding author: Sheng He. 

Email: [e-mail]. 

Address: University of Minnesota, N10, Elliott Hall, 75 River Rd, Minneapolis, MN 55455, USA.