Predictive remapping of visual features beyond saccadic targets

Abstract

Visual stability is thought to be mediated by predictive remapping of the relevant object information from its current, presaccadic location to its future, postsaccadic location on the retina. However, it is heavily debated whether and what feature information is predictively remapped during the presaccadic interval. Here we examined the spatial and featural properties of predictive remapping in a set of three psychophysical studies. We made use of an orientation-adaptation paradigm, in which we induced a tilt aftereffect by prolonged exposure to an oriented adaptor stimulus. Following this adaptation phase, a test stimulus was presented shortly before saccade onset. We found strong evidence for predictive remapping of the features of this test stimulus presented shortly before saccade onset, evidenced by a large tilt aftereffect elicited when the adaptor was positioned at the postsaccadic retinal location of the test stimulus. Conversely, the adaptation state itself, caused by the exposure to the adaptor stimulus, was not predictively remapped. Furthermore, we establish that predictive remapping also occurs for stimuli that are not saccade targets, pointing toward a forward remapping process operating across the whole visual field. Together, our findings suggest that predictive feature remapping of object information plays an important role in mediating visual stability.

Introduction

Each time we move our eyes, the image of objects in the world shifts its position on the retina, yet our perception is remarkably stable. Previous research has revealed that predictive remapping could contribute to this visual stability. Predictive remapping refers to the phenomenon in which neurons become active in response to stimuli outside their receptive fields shortly before a saccade moves their receptive fields onto the stimulated regions (Duhamel, Colby, & Goldberg, 1992). Predictive remapping has been demonstrated in many cortical regions, such as the lateral intraparietal area (Duhamel et al., 1992), the frontal eye field (Goldberg & Bruce, 1990; Umeno & Goldberg, 1997), superior colliculus (Walker, Fitzgibbon, & Goldberg, 1995), and early visual cortex including V2, V3, and V3a (Nakamura & Colby, 2002), and has been shown to depend on the intention to execute eye movements rather than shifting covert attention alone (Colby, 1996). Predictively increasing activity of visually responsive neurons in these areas according to postsaccadic stimulus information could facilitate the processing of visual information across saccades, which is crucial for achieving perceptual stability. 

Although predictive remapping has been widely studied, there is an ongoing debate regarding whether and how feature information of visual objects is remapped during this process (Cavanagh, Hunt, Afraz, & Rolfs, 2010; Ezzati, Golzar, & Afraz, 2008; Harrison, Retell, Remington, & Mattingley, 2013; He, Mo, & Fang, 2017; Lescroart, Kanwisher, & Golomb, 2016; Mayo & Sommer, 2010; Melcher, 2005, 2007, 2010; Pelli & Cavanagh, 2013; Zimmermann, Weidner, & Fink, 2017; Zirnsak & Moore, 2014). On the one hand, several psychophysical studies have suggested that visual feature information, such as orientation and letter information, is transmitted around the time of a saccade (Harrison et al., 2013; He et al., 2017; Melcher, 2007). Furthermore, previous studies suggest that foveal and peripheral feature information are integrated across saccades in a statistically optimal manner, which might rely on predictive feature remapping (Ganmor, Landy, & Simoncelli, 2015; Hübner & Schütz, 2017; Wolf & Schütz, 2015). More specifically, it is suggested that relevant features of a test stimulus, which are extracted before the saccade, are transferred to their postsaccadic retinal location based on the computation of the saccade vector. On the other hand, Rolfs, Jonikaitis, Deubel, & Cavanagh (2011) have proposed that it is merely the attentional pointers, not the feature information, that are predictively remapped across saccades. Through linking the attentional pointers at the current and future retinotopic locations together, the feature information at these two distinct locations is combined at higher processing stages. 

The tilt aftereffect (TAE), in which prolonged exposure to a stimulus (the adaptor) results in a perceptual shift of a test stimulus away from the adaptor, is a sensitive method to address the question of feature remapping (Knapen, Rolfs, Wexler, & Cavanagh, 2010; Melcher, 2007). Namely, orientation feature integration between the pre- and postsaccadic locations can be inferred from observing a TAE. There has been considerable confusion, however, concerning what is supposedly remapped prior to executing a saccade. Specifically, it is unclear whether the adaptor (or the state of adaptation, induced by the adaptor stimulus), the test stimulus, or both, is remapped (see Figure 1C and 1D). Moreover, the spatial properties of remapping are a current topic of debate. In particular, it is not clear whether receptive fields are shifted to their postsaccadic location (forward remapping; Biber & Ilg, 2011; Dorr & Bex, 2013; Duhamel et al., 1992; Melcher, 2007) or toward the saccade target (convergent remapping; Zirnsak & Moore, 2014; Zirnsak, Steinmetz, Noudoost, Xu, & Moore, 2014). Since in most of the previous studies the probe location has coincided with the saccade target location, these studies have been unable to differentiate between convergent and forward remapping effects, and more recent studies that aimed to dissociate these effects have provided conflicting results (Neupane, Guitton, & Pack, 2016b; Zirnsak et al., 2014). Interestingly, a behavioral study by Zirnsak, Gerhards, Kiani, Lappe, and Hamker (2011), in which probe and saccade target location were dissociated, reported evidence for convergent remapping and no evidence for forward remapping. However, this result was based on a small sample (N = 3), which limits the inferences that can be drawn (Button et al., 2013), and the test location for forward remapping was located far in the periphery of the visual field, potentially abolishing a forward remapping effect. Consequently, further investigations about the presence of forward or convergent remapping effects are necessary. 

Figure 1

 

Experimental design and hypotheses. (A) Experimental design. An adaptor was first presented at the initial fixation location for 3 s. After a random delay period, participants were asked to make a horizontal eye movement to the saccade target following the shift of fixation (black dot). Immediately after the presentation of the test stimulus, participants were asked to report whether it was tilted to the left or right relative to vertical. The test stimulus could appear at one of three locations (left, center, or right) and before or after saccade onset. (B) Time course of a trial. The gray area denotes the time period during which a test stimulus could be presented (before, during, or after a saccade). Trials on which a test stimulus was presented during the saccade were removed prior to the analysis. FP = fixation point; ST = saccade target. (C) Left: Adaptation effect. After a prolonged exposure to the adaptor, the neural population that is sensitive to the location of the adaptor becomes adapted (the “hole” in the figure). Middle: Adaptation remapping hypothesis. Upon preparation of a rightward saccade (gray arrow), the adaptation is remapped to its anticipated postsaccadic location. As a consequence, in this example a tilt aftereffect is expected at the left location. Right: Test-stimulus remapping hypothesis. Upon preparation of a rightward saccade (gray arrow), the test stimulus is remapped to its anticipated postsaccadic location. As a consequence, in this example a tilt aftereffect is expected at the right location. (D) The expected pattern of results under conditions of no remapping (left column), adaptation remapping (middle column), and test-stimulus remapping (right column).

In the current study, we investigated whether stimulus orientation is predictively remapped, and whether adaptation itself is remapped, as has been suggested before. Further, we examined whether presaccadic remapping also occurs for nonsaccade targets, in order to distinguish between forward and convergent remapping. To this end, we made use of the orientation-adaptation paradigm to test the TAE at each critical location—that is, the initial fixation location, the saccade target location, and the future retinotopic location of the adaptor. To preview: We found predictive feature remapping of the test stimulus when presented shortly before a saccade, in line with the results of Melcher (2007). Remapping occurred irrespective of status as a saccade goal, suggesting that the visual system employs forward predictive remapping of features. 

Methods

The current study consisted of three experiments. In the first experiment we tested whether predictive feature remapping occurs for stimuli that are saccade targets and whether adaptation itself remaps. In the second experiment we tested whether predictive feature remapping similarly occurs for peripheral stimuli that are not saccade targets. The third experiment acted as a control experiment to further corroborate the results of Experiments 1 and 2. 

Participants

A total of 72 subjects participated in the three experiments, engaging in a total of 82,080 trials. Each experiment had 24 subjects (Experiment 1: 11 women, 13 men, mean age = 23.6 years, range = 19–43; Experiment 2: 16 women, eight men, mean age = 22.8 years, range = 18–30; Experiment 3: 15 women, nine men, mean age = 24.4 years, range = 20–34). The sample size was based on an a priori power calculation, computing the required sample size to achieve a power of 0.80 to detect an effect size of Cohen's d ≥ 0.6, at α = 0.05 for a within-subject comparison. All participants reported normal or corrected-to-normal vision and were unaware of the purposes of the study. Participants were recruited from the institute's subject pool in exchange for either monetary compensation or study credits. The experiments were approved by the Radboud University Institutional Review Board and were carried out in accordance with the guidelines expressed in the Declaration of Helsinki. Written informed consent was obtained from all participants prior to the start of the study. 

Apparatus

All stimuli were generated with custom scripts written in Python (Version 2.7, available at http://www.python.org) and were presented on a 24-in. flat-panel display (BenQ XL2420T, resolution = 1,920 × 1,080, refresh rate = 60 Hz). The visible area of the display measured 48° × 27° at a viewing distance of about 64 cm. The participants' head position was stabilized with a chin rest. Eye movements were monitored by an EyeLink 1000 plus (SR Research, Ottawa, Canada) eye tracker, sampling at 1000 Hz. Only the right eye was recorded. Saccade initiation was detected online, with a velocity threshold of 30°/s and an acceleration threshold of 8000°/s². A nine-point calibration and validation procedure was conducted at the beginning of each block. 

Stimuli and experimental design

Experiment 1

Participants were tested in a quiet and dimly lit laboratory. Each trial began with the presentation of a fixation dot at the center of the screen. This fixation dot also served as the drift-correction target and remained visible until the participant's gaze was within 1° of it and the space bar was pressed. The sequence of events and time course in a single trial are illustrated in Figure 1A and 1B. 

After the initiation of the trial, a black fixation dot (diameter = 0.4°) and an oriented Gabor patch (oriented +20° or −20° relative to vertical) were presented at the center of the screen against a uniform midgray background for 3 s. The Gabor patch consisted of a sinusoidal wave grating (spatial frequency = 2 c/°; phase = 0.25; contrast = 1.0), windowed by a Gaussian envelope (SD = 1.67°). Participants were asked to fixate the dot until it disappeared. After 3 s, the Gabor patch disappeared and participants continued maintaining fixation at the central dot for a delay of 100–200 ms. After the delay, the fixation dot was horizontally displaced to the left or right side of the screen (8°), which served as a cue for the participant to make a saccade to the new fixation location. A test stimulus (Gabor stimulus with one of five orientations: −2°, −1°, 0°, 1°, 2°) was then flashed briefly at one of three locations (left, center, or right) for 50 ms. In the subsequent data analysis, we pooled the data for left- and rightward saccades (no difference, all p > 0.12), expressing all data in the reference frame of the rightward-saccade condition. In this reference frame, the right test-stimulus location corresponds to the saccade target location, the center test-stimulus location corresponds to the initial fixation location, and the left test-stimulus location corresponds to the future, postsaccadic retinotopic location of the adaptor. Crucially, the onset of the test stimulus varied in the range of 50–350 ms after the displacement of the fixation dot, such that it could occur either before or after the onset of the saccade, given that human saccade latency is estimated to lie around 200 ms (Robinson, 1964). The participant's task was to indicate whether the test stimulus was tilted to left or right with respect to vertical, regardless of its location. 

Participants completed three sessions of the task, comprising a total of 1,260 trials. There were 210 trials for each combination of the two adaptor tilt orientations and three test-stimulus locations. If the participant's gaze deviated more than 2° from the central fixation dot during the adaptation period, or landed at a location that was more than 2° away from the saccade target, auditory and visual feedback was given and the trial was aborted. All aborted trials were discarded and retested in a random order, until all trials were completed successfully. 

Experiment 2

In order to test whether predictive feature remapping also occurs for stimuli that are not saccade targets, we repeated Experiment 1 but presented both adaptor and test stimuli 4° above fixation. Consequently, the test stimulus was never a saccade target. In addition, as Experiment 1 yielded no evidence of adaptor remapping prior to a saccade, we did not test for remapping at the future retinotopic location of the adaptor in this experiment. 

The trial sequence in Experiment 2 was identical to that of Experiment 1. Each trial began with the presentation of an oriented Gabor patch 4° above central fixation. Participants were next asked to move their eyes to the periphery following the shift of the fixation dot. The test stimulus was flashed 4° above the initial fixation location or the saccade target location to measure transfer of feature information between these two locations. Experiment 2 consisted of two sessions. For each combination of the two test-stimulus locations and adaptor tilt orientations, 270 trials were collected, resulting in a total of 1,080 trials. 

Experiment 3

In Experiment 3, the task was similar to Experiment 1 except that two oppositely oriented adaptors were presented simultaneously at the two peripheral locations. In a given trial, participants initially fixated at the center of the screen while two oppositely oriented Gabor patches (either +20°/−20° or −20°/+20° from vertical) were presented simultaneously for 3 s at 8° left and right of the center of the screen. Next, participants were prompted to move their eyes to the left or right peripheral location, following the shift of the fixation dot. The test stimulus was flashed at either the initial fixation location or the saccade target location. Experiment 3 consisted of two sessions. For each combination of the two test-stimulus locations and adaptor tilt orientations, 270 trials were collected, resulting in a total of 1,080 trials. 

The logic behind Experiment 3 is as follows. Imagine a trial in which the participant performs a saccade from the center to the right peripheral location. Under the forward-remapping hypothesis, receptive fields are expected to shift in the rightward direction, parallel to the saccade vector. Note that this is equivalent to remapping feature information in the direction opposite to the saccade vector. Therefore, a test stimulus that is centrally presented prior to the saccade would be remapped to the left peripheral location. Under the convergent-remapping hypothesis, receptive fields are remapped toward the saccade target. Therefore, a test stimulus that is centrally presented, far away from the saccade target, would not be remapped to the left peripheral location, and no TAE is expected. 

Data analysis

All data analyses were performed with MATLAB (R2016a; MathWorks, Natick, MA) using the Palamedes Matlab toolbox for fitting psychophysical data (Prins & Kingdom, 2018). The significance threshold was set to 0.05. All data and code are available from the Donders Institute for Brain, Cognition and Behavior Repository at https://hdl.handle.net/11633/di.dccn.DSC_3018034.01_694. 

Outlier criteria

Experiment 1

A total of 37,114 trials were obtained for Experiment 1. Only successfully completed trials were considered in the further analyses. We excluded a trial from the analyses if fixation was broken before fixation displacement (7.75% of all trials) or the participant did not execute the required eye movements or missed the displaced fixation dot by more than 2° (10.87% of all trials). In the remaining 30,202 trials, saccade latency was defined as the temporal distance between the onset of the fixation-dot displacement and the initiation of the saccade that followed. Trials with saccade latencies shorter than 90 ms (0.23%) or longer than 500 ms (1.04%) were excluded. We also excluded trials whose response time was shorter than 200 ms (0.3%) or more than three standard deviations above the subject's mean response time (1.27%). Finally, trials in which the test stimulus was presented during the execution of the saccade were also excluded (15.71%). In total, 24,692 (81.55% of successful trials) trials were included in the analysis. 

Experiment 2

A total of 34,770 trials were obtained for Experiment 2. We excluded trials from further analysis if fixation was broken before fixation displacement (9.62% of all trials) or the participant did not execute the required eye movements or missed the displaced fixation dot by more than 2° (16.19% of all trials). Of the remaining 25,796 trials, trials in which the saccade latency was shorter than 90 ms (0.07%) or longer than 500 ms (1.04%) were excluded. We also excluded trials in which the button response time was shorter than 200 ms (1.34%) or more than three standard deviations above the subject's mean response time (1.08%). Finally, the trials in which test stimulus was presented during the saccade period were also excluded (16.75%). Together, 20,593 (79.83% of successful trials) trials were included in the analysis. 

Experiment 3

A total of 32,792 trials were obtained for Experiment 3. We excluded trials from the analyses if fixation was broken before fixation displacement (11.47% of all trials) or the participant did not execute the required eye movements or missed the displaced fixation dot by more than 2° (9.42% of all trials). Of the remaining 25,942 trials, trials with saccade latencies shorter than 90 ms (0.07%) or longer than 500 ms (1.90%) were excluded. We also excluded trials whose response time was shorter than 200 ms (5.32%) or more than three standard deviations above the subject's mean response time (0.78%). Finally, trials in which test stimulus was presented during the execution of the saccade were also excluded (14.87%). In total, 20,820 (80.26% of successful trials) trials were included in the analysis. 

Quantification of time bins

To plot the TAE magnitude as a function of time, we first separated all trials into two bins at the group level, based on whether the test stimulus was presented before saccade onset or after saccade offset. Trials in which the test stimulus was presented during the saccade were removed. For both bins, the trials were then further subdivided into two time bins by a median split with respect to the test-stimulus onset time. Trials with an onset time that was equal to the median were assigned to the later time bin. This resulted in a total of four time bins. We used four time bins to maximize the trial numbers in each time point and condition to be able to reliably fit the psychometric functions. In Experiment 1, the total number of trials in each time bin was, respectively, 5,715, 5,801, 6,531, and 6,582. Mean test-stimulus onset time with respect to saccade onset (for presaccadic trials) or offset (for postsaccadic trials) was −133 ms (SD = 73 ms), −31 ms (SD = 17 ms), 36 ms (SD = 21 ms), and 115 ms (SD = 28 ms). In Experiment 2, the total number of trials in each time bin was 5,016, 5,075, 5,213, and 5,289. Mean test-stimulus onset time was −117 ms (SD = 63 ms), −29 ms (SD = 17 ms), 32 ms (SD = 18 ms), and 104 ms (SD = 26 ms). In Experiment 3, the total number of trials in each time bin was 6,049, 6,089, 4,320, and 4,362. Mean test-stimulus onset time was −156 ms (SD = 68 ms), −38 ms (SD = 22 ms), 29 ms (SD = 17 ms), and 97 ms (SD = 26 ms). 

In order to follow up on the time course of presaccadic predictive remapping, we further split the trials with test stimuli presented before saccade onset into four narrower time bins to quantify the time course of remapping in more detail. For this analysis, we took the three quartiles instead of the median for defining the boundaries of the time bins. As a result, in Experiment 1 the total number of trials in each time bin was 2,850, 2,865, 2,916, and 2,885. Mean test-stimulus onset time with respect to saccade onset was −189 ms (SD = 66 ms), −77 ms (SD = 11 ms), −45 ms (SD = 9 ms), and −16 ms (SD = 9 ms). In Experiment 2 the total number of trials in each time bin was 2,484, 2,532, 2,530, and 2,545. Mean test-stimulus onset time was −159 ms (SD = 67 ms), −75 ms (SD = 10 ms), −44 ms (SD = 8 ms), and −15 ms (SD = 8 ms). In Experiment 3 the total number of trials in each time bin was 3,029, 3,020, 2,981, and 3,108. Mean test-stimulus onset time was −210 ms (SD = 58 ms), −103 ms (SD = 18 ms), −58 ms (SD = 11 ms), and −20 ms (SD = 11 ms). 

Quantification of TAE

In order to quantify TAE magnitude, we fitted psychometric functions to the pooled group data. Fitting the pooled group data was preferred over fitting single-subject data due to the limited number of trials per condition for each subject. First, for each combination of test-stimulus location and adaptor tilt in each time bin, we expressed the proportion of rightward responses as a function of the test-stimulus orientation with respect to vertical. For convenience, the leftward-saccade trials were first collapsed with rightward-saccade trials in each bin. Subsequently, we fitted cumulative normal distribution functions to these data. The point of subjective equality was defined as the midpoint of the psychometric function, at which the test stimulus was perceived equally often as tilted to the right and to the left. The magnitude of the TAE was then measured as half of the difference between the point of subjective equality of the leftward- and rightward-tilted adaptor conditions, for each time bin and each test-stimulus location separately. In Experiment 3, two adaptors were presented simultaneously. The TAE for test stimuli presented at the initial fixation location (center) was always calculated with respect to the orientation of the adaptor that was opposite of the saccade target, whereas the TAE for test stimuli presented at the saccade target location was calculated based on the adaptor at the saccade target location. 

Statistical analyses

We used permutation tests to statistically compare differences of TAEs between time bins (before saccade) and the interaction effect between locations (incorporating the initial fixation and the future saccade target location only) and the time bins (two time bins before eye movement) at the group level. First, to test for differences in TAEs between time bins, the condition labels of the first and second time bin of each participant were randomly shuffled. The resulting permutation group data were fitted with cumulative normal functions and used to compute the difference in TAEs between the time bins. This procedure was repeated 10,000 times. As p values we report the proportion of permutations that led to an equal or more extreme TAE difference than the one we observed in the experiment. The exchangeability requirement for permutation tests is met, because under the null hypothesis of no difference in TAE between the first and second time bins, the condition labels are exchangeable. 

Second, in order to test for an interaction effect between locations and time bins, we first computed the differences of TAEs between initial fixation and saccade target location at each time bin, and then randomly shuffled the time-bin labels of those differences for each participant. The exchangeability requirement for permutation tests is met, because under the null hypothesis of no interaction effect between locations and time bins, the TAE differences between locations should not be influenced by the time-bin factor, and therefore the time-bin labels are exchangeable. Again, this procedure was repeated 10,000 times. As p values we report the proportion of permutations that led to an equal or more extreme outcome than the one we observed in the experiment. 

Results

We collected psychophysical data in a series of three experiments, each using 24 human participants. In total, we recruited a sample of 72 participants and 82,080 trials. 

Selective remapping of future target stimuli but not adaptation

Our first aim was to test whether the test stimulus or adaptation is remapped. To this end, we compared the temporal profile of the TAE for test stimuli presented at the saccade target location, initial fixation location, and future, postsaccadic retinotopic location of the adaptor. Specifically, when a test stimulus was presented well before the saccade initiation (Figure 2, first column), we found that the perceived orientation of the test stimulus at each location was systematically biased away from the adaptor stimulus that was previously presented at the center of the screen (Figure 2). This repulsive bias, which is well known as the TAE in orientation perception, was quantified as the difference in the point of subjective equality between a left-tilted and a right-tilted adaptor (illustrated as the black bar between the psychometric curves). It was strongest when the test stimulus was presented at the initial fixation (center) location (middle row), where the adaptor had been presented, and markedly reduced but still present at the other two locations. We next investigated if, when, and where the TAE was transferred shortly before subjects initiated a saccade. We found that shortly before an eye movement, the TAE was significantly reduced at the future retinotopic location of the adaptor (Figure 2, “FRA” location, violet lines; comparison between first and second time point: p = 0.0165). Also at the initial fixation location, the TAE was reduced before an eye movement (Figure 2, “IF” location, orange lines; comparison between first and second time point: p = 0.0039). However, the TAE at the future saccade target location was significantly enhanced before the onset of the saccade (Figure 2, “ST” location, green lines; comparison between first and second time point: p < 0.0001). This opposite behavior over time between the locations resulted in a significant (p = 0.0002) interaction between target location (initial vs. future saccade location) and time (first vs. second time bin), showing that TAE increased at the future saccade target location and decreased at the initial location. 

Figure 2

 

Psychometric curves for orientation judgements in Experiment 1. The number above each column represents the mean test-stimulus onset relative to saccade onset (first and second columns) or offset (third and fourth columns) for each time bin. Test-stimulus locations: ST = saccadic target location; IF = initial fixation location; FRA = future retinotopic location of adaptor. For each panel, we plotted the percentage of a “right” response (y-axis) as a function of the orientation of a test stimulus (x-axis) for each time bin and location. Positive x values mean the test stimulus was tilted more clockwise relative to vertical, while negative x values mean more counterclockwise. The black lines indicate that the difference between the point of subjective equality (PSE; the angle at which participants judge a test stimulus was oriented left or right equally) of the leftward- (solid line) and rightward-tilted (dashed line) adaptor conditions—the value of the tilt aftereffect—was defined as half of ΔPSE.

These results are consistent with, and extend, those reported by Melcher (2007). When the test stimulus was presented at the saccadic target location, the features of the test stimulus were predictively remapped to the presaccadic foveal location that was previously adapted. Importantly, however, we found no TAE at the future postsaccadic location of the adaptor, to which the adaptation would be hypothetically remapped. Put simply, it is the orientation feature information of a stimulus presented shortly before the saccade, but not a previously seen adaptor and its consequences, that is predictively remapped before saccade onset. 

Selective remapping of nonsaccade targets

In Experiment 1, we observed predictive feature remapping of the test stimulus toward its postsaccadic location. However, since in this crucial condition the test stimulus was always a saccade target, we cannot differentiate between a mechanism that remaps stimuli toward the saccade target (convergent remapping) and one that more generally remaps stimuli across the visual field to their postsaccadic locations (forward remapping). In order to test whether remapping also occurs for nonsaccade targets, we flashed both the adaptor and the test stimulus 4° vertically above fixation. The idea behind this design is straightforward: If predictive remapping occurs only for saccade targets, we would expect no TAE when the test stimulus is presented 4° above the fixation target. However, if predictive remapping also occurs for stimuli that are not saccade targets, an increase of the TAE for peripherally presented test stimuli should be observable during the presaccadic period. 

Despite the fact that different locations were used for the adaptor and the test stimulus, we found a similar pattern of results as in Experiment 1. Specifically, before an eye movement the TAE was significantly increased at the future target location (Figure 3, “PT” location, green lines; comparison between first and second time point: p = 0.0068). However, the TAE at the adaptor-stimulus location was decreased for the second compared to the first time bin (Figure 3, “AS” location, orange lines; comparison between first and second time point: p = 0.0197). This opposite behavior over time between the locations also resulted in a significant (p = 0.0019) Test-stimulus location (adaptor stimulus vs. future target location) × Time (first vs. second time bin) interaction. This result suggests that predictive remapping likewise occurs for stimuli that are not saccade targets, consistent with a forward-remapping account. 

Figure 3

 

Psychometric curves for orientation judgements in Experiment 2. Same conventions as in Figure 2, but only peripheral target (PT) and adaptor-stimulus (AS) locations were tested, while both the adaptor and test stimulus were presented 4° above fixation.

The results of Experiments 1 and 2 indicate that predictive remapping of orientation occurs, irrespective of whether the stimulus is a saccade target. However, due to the short spatial distance between the test stimulus and saccade target location, and between the adaptor location and foveal fixation (both 4°), one may still argue that the findings of Experiment 2 could be explained by remapping of stimuli close to the fixation target or the fovea. To more directly contrast the convergent and forward remapping hypotheses, we designed Experiment 3, in which two oppositely oriented adaptors were presented simultaneously at peripheral locations while the test stimulus was flashed at the initial fixation location or saccade target location. 

Forward remapping hypothesizes a remapping of receptive fields in the same direction as the saccade. Therefore, a test stimulus presented at the initial fixation location (center) will, just before a rightward saccade, be remapped to the left, opposite of the saccade vector. Convergent remapping, on the other hand, hypothesizes a remapping of receptive fields toward the future saccade location. In this case, no TAE would be predicted for a stimulus presented at the initial fixation location, because no receptive fields are remapped to this location (see Methods for more details). 

The results indicated a positive TAE for stimuli presented at the initial fixation location, just before participants made a saccade (Figure 4, “IF” location, orange lines; second column: p = 0.0337), in line with forward remapping. Conversely, the TAE for stimuli presented at the saccade target location was significantly decreased before saccade onset (Figure 4, “ST” location, green lines; comparison between first and second time point: p < 0.0001). Further, there was a significant (p = 0.0001) Test-stimulus location (initial fixation vs. saccade target) × Time (first vs. second time bin) interaction. This pattern of results suggests that the test stimulus was predictively forward remapped prior to the eye movement. This result also provides further evidence against adaptation remapping, in line with Experiment 1. If the information of the adaptor at the saccade target had been remapped toward the initial fixation location, we should have observed an attractive TAE (expressed in relation to the opposite adaptor) for test stimuli presented at initial fixation. Instead we observed a repulsive TAE, thus further corroborating the absence of adaptation remapping. 

Figure 4

 

Psychometric curves for orientation judgements in Experiment 3. Same conventions as in Figure 2, but only saccade target (ST) and initial fixation (IF) locations were tested, while two oppositely oriented adaptors were presented in the periphery before saccade target onset. The tilt aftereffect at the initial fixation location was calculated with respect to the orientation of the adaptor that was opposite of the saccade target.

Comparison of TAE across experiments

As illustrated by Figure 5, a direct comparison of the three experiments confirms that the orientation information of the test stimulus, instead of the adaptor or its consequences, was predictively remapped to its future retinotopic location shortly before an eye movement (Figure 5A). This effect also occurred when the test stimulus was presented above the saccade target (Figure 5B) or at a peripheral location (Figure 5C)—which is a nonsaccade target location—suggesting that the visual system uses forward predictive remapping of features across the whole visual field. 

Figure 5

 

Comparison of tilt aftereffect across experiments. (A) Experiment 1: The adaptor was shown at the initial fixation location. For analysis, leftward-saccade trials were collapsed into rightward-saccade trials. The vertical dashed line indicates the onset/offset of a saccade. ST = saccadic target location; IF = initial fixation location; FRA = future retinotopic location of adaptor. (B) Experiment 2: The adaptor was shown 4° above fixation. Only initial fixation and saccade target location were probed. Other parameters were identical to Experiment 1. PT = peripheral target location; AS = adaptor-stimulus location. (C) Experiment 3: Two oppositely oriented adaptors were presented at a peripheral location. The test stimulus was presented at the initial fixation location or the saccade target location. ST = saccadic target location; IF = initial fixation location. All vertical error bars represent one standard deviation of the bootstrapped distribution. All horizontal error bars represent one standard deviation of the distribution of test-stimulus onset times within the respective time bin.

In order to follow up on the time period of test-stimulus remapping, we further split all trials with test stimuli presented before saccade onset into four narrower time bins to quantify the time course of remapping in more detail (Figure 6). In these analyses, we found that while the TAE at each location did not change much up to 80 ms before saccade onset, a dramatic change occurred within the 50 ms before saccade onset, suggesting that predictive remapping occurred very close to the saccade onset in our experiment. This result is consistent with that of Duhamel et al. (1992), who showed that the neurons in the lateral intraparietal area start responding to the visual stimuli in their future field 80 ms before saccade onset, suggesting that the feature remapping we observe more likely represents a predictive oculomotor effect instead of the presaccadic attention shift, which presumably occurs much earlier. 

Figure 6

 

Tilt aftereffect in finer time bins before saccade onset in Experiments 1, 2, and 3. (A) In Experiment 1, while the tilt aftereffect at each location did not change much up to 80 ms before saccade onset, a dramatic change occurred within 50 ms before saccade onset, suggesting that predictive remapping occurred very close to the saccade onset. ST = saccadic target location; IF = initial fixation location; FRA = future retinotopic location of adaptor. (B) A similar pattern to that of Experiment 1 was observed in Experiment 2. PT = peripheral target location; AS = adaptor-stimulus location. (C) Similar to Experiments 1 and 2, feature remapping occurred close to the saccade onset in Experiment 3. ST = saccadic target location; IF = initial fixation location. All vertical error bars denote one standard deviation of the bootstrapped distribution. All horizontal error bars denote one standard deviation of the distribution of test-stimulus onset times within the respective time bin.

Discussion

We used an orientation-adaptation paradigm to investigate whether and how feature information is predictively remapped prior to saccades. In Experiment 1 (see Figure 5A), and consistent with the results reported by Melcher (2007), we found strong evidence for predictive remapping of visual information that is presented shortly before saccade onset but no remapping of adaptation, as had been previously hypothesized (Melcher, 2007; Rolfs et al., 2011). Notably, predictive feature remapping occurred very shortly before saccade onset (that is, <80 ms before saccade onset; see Figure 6). In Experiments 2 and 3 (see Figure 5B and 5C), we provided evidence that presaccadic remapping of features also occurs for stimuli that are not a saccade target, consistent with forward remapping, which further underscores the generality of this mechanism (Neupane et al., 2016a, 2016b). 

No predictive remapping of adaptation

The results of Experiments 1 and 3 indicate that while features of stimuli presented shortly before the impending saccade are remapped to their future retinal location, the adaptation effect itself is not remapped during this time period. While there is also a significant TAE at the saccade target location and future retinotopic location of the adaptor at the first time bin in Experiment 1 of our study (Figure 5A), this reflects a spatially nonspecific TAE that spreads across the whole visual field (Knapen et al., 2010). In our experiments, the adaptor stimulus is presented during an initial fixation period, long before participants are instructed to prepare a saccade. Therefore, at the time participants could prepare a specific saccade plan, the adaptor stimulus had already disappeared. Since the saccade preparation occurs after the adaptor-stimulus offset, any processing of the adaptor stimulus is likely finished by the time participants prepare the saccade. As the remapping dynamics also clearly show, only stimulus information that is presented very shortly before the saccade is remapped. This is also in line with the notion that adaptation occurs in a retinotopic reference frame (Knapen et al., 2010; Wenderoth & Wiese, 2008), possibly due to a reduction of excitability in the adapted neurons. It is unlikely that such a reduction of neuronal excitability can be remapped by the planning of a saccade. 

Contrary to our results, a recent article by He et al. (2017) observed predictive feature remapping of adaptation. In that study, however, participants were required to make the same saccade on every trial, and the test stimulus always appeared at the same location (i.e., the future retinotopic location of the adaptor). Since the saccade plan was already known to the participant at the beginning of each trial, it seems likely that participants could build up a spatiotopic representation of the adaptor, even before they saw the cue to execute the saccade. In line with this, Zimmermann, Morrone, Fink, and Burr (2013) found that postsaccadic spatiotopic effects of the TAE are not immediately present after the eye movement but require substantial time to build up. In contrast, in our experiment the saccade direction on a given trial was known to the participant only at the time of the fixation-dot displacement, leaving little time to transform the adaptor information into a spatiotopic representation before and after the saccade. 

Importantly, while the study by Zimmermann et al. (2013) is broadly consistent with our finding that there is no presaccadic remapping of adaptation when the preview duration of the saccade target is short, they presented the test stimulus always after the saccade, thus measuring postsaccadic TAEs. Given that pre- and postsaccadic remapping may not share the same properties, our study complements the previous research on postsaccadic spatiotopic adaptation effects. Furthermore, the evidence for presaccadic remapping of the test stimulus in our study suggests that predictive remapping of feature information can occur even for short saccade target preview durations, and thus predictive feature remapping as such is not dependent on an extended saccade planning time. 

Remapping of features or attentional pointers?

The question whether feature information is involved in the predictive-remapping process has been extensively debated in the past decade. Rolfs et al. (2011) found that visual performance was gradually enhanced at the future retinotopic location even before the onset of eye movements. Since the target was very difficult to detect and required a high degree of attention toward the particular location, the authors proposed that attention, rather than feature information, is predictively remapped prior to a saccade. This hypothesis was further supported by several subsequent studies (Harrison et al., 2013; Hunt & Cavanagh, 2011; Jonikaitis, Szinte, Rolfs, & Cavanagh, 2013; Puntiroli, Kerzel, & Born, 2015). However, in recent years a number of studies have provided evidence that feature information, in addition to attentional pointers, is also involved in transsaccadic remapping (Cha & Chong, 2013; Demeyer, De Graef, Wagemans, & Verfaillie, 2009, 2010; Demeyer, Graef, Verfaillie, & Wagemans, 2011; Eccelpoel, Germeys, Graef, & Verfaillie, 2008; Edwards, VanRullen, & Cavanagh, 2017; Fracasso, Caramazza, & Melcher, 2010; Gordon, Vollmer, & Frankl, 2008; Habtegiorgis, Rifai, Lappe, & Wahl, 2018; Harrison & Bex, 2014; Hayhoe, Lachter, & Feldman, 1991; He et al., 2017; Herwig & Schneider, 2014; Koller & Rafal, 2018; Melcher, 2007; Nakashima & Sugita, 2017; Oostwoud Wijdenes, Marshall, & Bays, 2015; Paeye, Collins, & Cavanagh, 2017; Prime, Niemeier, & Crawford, 2006; Prime, Vesia, & Crawford, 2011; Sligte et al., 2017; Wittenberg, Bremmer, & Wachtler, 2008; Wolfe & Whitney, 2015; Zimmermann et al., 2013; Zimmermann et al., 2017; Zimmermann, Weidner, Abdollahi, & Fink, 2016; Zirnsak et al., 2011). Our study is in line with these studies, and further extends the findings by showing that orientation features of an actively processed stimulus, rather than the adaptation effects due to previous stimulation, are remapped. 

Notably, several fMRI studies have also shown evidence for predictive feature remapping (but see Dunkley, Baltaretu, & Crawford, 2016; Fairhall, Schwarzbach, Lingnau, Van Koningsbruggen, & Melcher, 2017). Zimmermann et al. (2016) found that visual feature information was dynamically remapped from a retinotopic coordinate into a spatiotopic coordinate system in ventral visual areas V3, V4, and VO. Merriam, Genovese, and Colby (2007) found remapping of information associated with the execution of eye movements not only in higher order extrastriate areas (V3A, hV4) but also in V1 and V2, although smaller in magnitude, consistent with an earlier study in nonhuman primates (Nakamura & Colby, 2002). How is this feature information transferred within the visual system? A possible explanation for this might be that feature remapping is the effect of the combination of corollary discharge and bottom-up information. Activity elicited by the test stimulus could be remapped under the guidance of corollary-discharge signals (Rao, Mayo, & Sommer, 2016; Sommer & Wurtz, 2006; Sperry, 1950). The basic idea of corollary discharge is that when the motor system generates a movement command for muscles to produce a movement, a copy or corollary of this command is also sent to other regions of the brain to inform them about the impending movement. Thus when a saccade is prepared by the oculomotor system, a corollary-discharge signal containing information about the onset and target location of the imminent eye movement could be used to redirect the flow of feature information in visual cortex (Fries, 1984; Tolias et al., 2001). In particular, while the neurons whose receptive fields cover the stimulus location will be activated by the bottom-up signal at first, this signal will be combined with the corollary discharge in extrastriate cortex and then, via the superior colliculus, to neurons whose receptive fields will overlap with the stimulus region after the eye movement. 

Convergent and forward predictive remapping

In their seminal study, Duhamel et al. (1992) reported that a set of neurons in the lateral intraparietal area predictively shift their receptive fields from their current location to their future retinotopic location prior to a saccade. This type of predictive remapping was termed forward remapping, since receptive-field locations are shifted parallel to the saccade vector, as has been observed in several studies (Nakamura & Colby, 2002; Umeno & Goldberg, 1997; Walker et al., 1995). However, another type of predictive remapping has been proposed, which is termed convergent remapping, suggesting that the receptive fields shift toward the saccade target location rather than their postsaccadic location (Tolias et al., 2001; Zirnsak et al., 2014). Due to limitations in the experimental paradigms, forward and convergent remapping are sometimes difficult to distinguish. In particular, in many previous studies the test stimulus has often constituted the saccade target, and in this case forward and convergent remapping theories make indistinguishable predictions. 

In our current study, when the test stimulus was presented outside the saccade target location (Experiments 2 and 3), we still observed a robust forward presaccadic remapping effect. This result is in line with a previous electrophysiological study in V4 (Neupane et al., 2016a). In contrast, convergent remapping has been reported in the frontal eye field (Zirnsak et al., 2014). We speculate that the convergent remapping in the frontal eye field, which is a nonvisual area, may not be functionally related to shifting of receptive fields but rather to anticipating and selecting relevant stimuli near the saccade target location, to facilitate processing of saccade targets. Conversely, for the visual system maintaining stable representations of features across saccades is critical for seamless visually guided behaviors, which may be enabled by forward remapping. However, even though we provide evidence for forward remapping, no direct evidence against convergent remapping was observed in our experiments. Therefore, it is possible that forward and convergent remapping could occur concurrently. In addition, recent evidence suggests that these two types of remapping may have different time courses, with forward remapping preceding convergent remapping (Neupane et al., 2016b). Therefore, although we find evidence for forward remapping in our study, it is possible that convergent remapping would dominate at later time points. 

Slopes of psychometric functions

In Experiment 1 (Figure 2), the slopes of the psychometric functions for the initial fixation position are shallower at early time bins (before saccade) than at late time bins (after saccade). This may reflect worse discriminability due to adaptation. While discrimination performance can be improved if an adaptor is oriented orthogonally to the orientation of a test stimulus, it is typically impaired if the orientation of an adaptor differs from the orientation of a test stimulus by 7° to 45° (Regan & Beverley, 1985; Schwartz, Hsu, & Dayan, 2007). In line with this explanation, an opposite pattern of slope changes was found in Experiment 3, in which we used peripheral adaptors. 

Conclusion

We found strong support for predictive remapping of the orientation feature of a test stimulus that was presented shortly before saccade onset. This presaccadic remapping also occurred for stimuli that were not saccade targets, and had the characteristics of a forward remapping process that operates across the whole visual field. Forward predictive feature remapping may constitute an important mechanism for mediating visual stability. 

Acknowledgments

TH was supported by a grant from the China Scholarship Council (CSC; 201608330264). MF was supported by a grant from the European Union Horizon 2020 Program (ERC Starting Grant 678286, “Contextvision”). FPDL was supported by grants from the Netherlands Organization for Scientific Research (NWO; Vidi Grant 452-13-016), the James S. McDonnell Foundation (Understanding Human Cognition, 220020373), and the EC Horizon 2020 Program (ERC Starting Grant 678286, “Contextvision”). 

Commercial relationships: none. 

Corresponding author: Floris P. de Lange. 

Email: . 

Address: Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands