Predictive feature remapping before saccadic eye movements

Abstract

Saccadic eye movements cause rapid and dramatic displacements of the retinal image of the visual world, yet our conscious perception of the world remains stable and continuous. A popular explanation for this remarkable ability of our visual system to compensate for the displacements is the predictive feature remapping theory. The theory proposes that, before saccades, the representation of a visual stimulus can be predictively transferred from neurons that initially encode the stimulus to neurons whose receptive fields will encompass the stimulus location after the saccade. Visual adaptation aftereffect experiments performed by Melcher (2007) provided psychophysical evidence for this theory. However, it was argued that the visual aftereffects were not measured at the “appropriate” remapped location (Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011). Therefore, whether the remapped representation contains feature information (e.g., orientation, motion direction, or contrast) is still a subject of intense debate. Here, to explore the nature of the predictive transfer during trans-saccadic perception, we measured visual aftereffects (tilt aftereffect, motion aftereffect, and threshold elevation aftereffect) at the appropriate remapped location of adapting stimuli before saccades. We observed a significant tilt aftereffect and motion aftereffect, but little threshold elevation aftereffect. Furthermore, the tilt aftereffect and motion aftereffect exhibited spatial specificity. These findings provide strong evidence for the predictive feature remapping theory and suggest that intermediate visual processing stages (i.e., extrastriate visual cortex) might be critical for feature remapping. Finally, we propose that the feature remapping process might also contribute to the spatiotopic representation of visual features.

Introduction

Our perception of the visual world appears to be stable and continuous, even though saccades (rapid eye movements) dramatically shift the image of the world on the retina. Several theoretical viewpoints have been proposed to explain the stability of trans-saccadic perception, including perceptual renewal with each fixation (O'Regan & Noë, 2001; Rensink, 2000), integration in a trans-saccadic buffer (Jonides, Irwin, & Yantis, 1982), storage in visual memory (Irwin, 1991), and spatiotopic representation of the visual world (D'Avossa et al., 2006; Duhamel, Bremmer, Benhamed, & Graf, 1997). While each of these theories can explain certain aspects of the trans-saccadic stability issue, the discovery of predictive remapping (Duhamel, Colby, & Goldberg, 1992; Sommer & Wurtz, 2006), the activity increase of many neurons in retinotopic brain areas when saccades are about to bring stimuli into their receptive fields, suggests a more comprehensive and plausible framework for understanding trans-saccadic perception (Cavanagh, Hunt, Afraz, & Rolfs, 2010; Melcher & Colby, 2008). 

Predictive remapping was observed in the lateral parietal area (Colby, Duhamel, & Goldberg, 1995; Duhamel et al., 1992; Wang et al., 2016), the frontal eye field (Goldberg & Bruce, 1990; Sommer & Wurtz, 2006), the superior colliculus (Walker, Fitzgibbon, & Goldberg, 1995), and extrastriate visual cortex (Nakamura & Colby, 2002). In anticipation of a saccade, neurons in these brain areas were shown to have a normal, current receptive field and a future receptive field (i.e., the spatial location where the receptive field will be after the intended saccade). Remapping could occur even before the saccade onset, reflected as a combination of neuronal activity increase in the future receptive field and concurrent decrease in the current receptive field. 

Although predictive remapping is thought to contribute to visual stability by updating the internal image with each saccade, details of predictive remapping remain to be thoroughly investigated. Some researchers propose that visual stability is based on remapping of attention pointers (Cavanagh et al., 2010; Joiner, Cavanaugh, & Wurtz, 2011; Mirpour & Bisley, 2012, 2016; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011). They propose that a map of attentionally significant locations is updated before saccades. Once a saccade is completed, feature information can be selectively extracted from these locations. In other words, presaccadic activity of remapping neurons conveys only spatial information, but not feature information. On the other hand, some other researchers argue that feature information along with spatial information is remapped before saccades, which is implemented by a predictive activity transfer from neurons that initially encode the stimulus to neurons whose receptive fields will encompass the stimulus location after the saccade. (Harrison, Retell, Remington, & Mattingley, 2013; Melcher, 2007; Melcher & Colby, 2008; Subramanian & Colby, 2014; Szinte, Jonikaitis, Rolfs, Cavanagh, & Deubel, 2016). Therefore, whether presaccadic activity of remapping neurons conveys feature information (e.g., orientation, motion direction, or contrast) is still under intense debate. 

Melcher (2007) used the tilt aftereffect (TAE) to measure any transfer of visual orientation adaptation from the current fixation position to the saccadic target location around the time of saccades. Several visual adaptation aftereffects, including TAE, MAE (motion aftereffect), and TEAE (contrast threshold elevation aftereffect), are especially suitable for studying predictive feature remapping because these aftereffects are highly specific to adapted features, well constrained in spatial extent, and persist longer than a typical fixation (Anstis, Verstraten, & Mather, 1998; He & MacLeod, 2001; Melcher, 2005). In Melcher's study, after a brief adaptation to a tilted grating presented at the initial fixation point, participants were asked to make a saccade and judge the orientation of a test grating briefly presented at either the initial fixation point or the saccadic target location. Melcher found that the intention to make a saccade simultaneously triggered a reduction of the TAE magnitude at the initial fixation point and a TAE increase at the saccadic target location, even before saccades. He suggested that the orientation adaptation (or orientation information) could be transferred from the initial fixation point to the future fixation point (i.e., the saccadic target location) before saccades, therefore providing evidence for predictive feature remapping. Recently, Rolfs et al. (2011) argued that, in Melcher's experiments, TAE was measured at the location (the reversed remapped location) opposite to the actual remapped location of the adapting stimuli (see Figure 1), and they believe that these experiments did not actually study the behavioral correlates of predictive remapping (but see also Zirnsak, Steinmetz, Noudoost, Xu, & Moore, 2014; Zirnsak & Moore, 2014). 

Figure 1

 

Appropriate remapped location of visual adaptor. In Melcher's study (2007), subjects adapted to a tilted grating presented at the initial fixation point. Then they were asked to make a saccade and judge the orientation of a test grating briefly presented at either the initial fixation point or the saccadic target location. According to Rolfs et al. (2011), the visual adaptor (i.e., the tilted grating) at the initial fixation point activates neurons that encode the adaptor's expected retinal location after the saccade. The remapping vector (dashed arrow) actually opposes the saccade vector (solid arrow). Therefore, the appropriate remapped location of the adaptor corresponds to the retinal position that the adaptor will have only following the saccade. FP: fixation point; ST: saccadic target; ARL: appropriate remapped location; AS: adapting stimulus; and RRL: reversed remapped location.

Here, to further test the predictive feature remapping theory, we measured the TAE, MAE, and TEAE at the appropriate or “actual” remapped location of adapting stimuli before saccades as proposed by Rolfs et al. (2011; Figure 2A). If feature information (orientation, motion direction, and contrast) could be predictively remapped, we expect to detect adaptation aftereffect at the appropriate remapped location before saccades. Since these aftereffects are mediated by brain areas at different processing levels (Georgeson, 2004; Kohn, 2007; Webster, 2015), this approach provides a comprehensive characterization of predictive feature remapping during trans-saccadic perception. 

Figure 2

 

Stimuli, procedure, time course of events, and results in Experiment 1. (A) The spatial location and the remapped location of the grating adaptor. Subjects made a saccade from the fixation point to the saccadic target. The saccade direction and the remapping direction are shown by the solid arrow and the dashed arrow, respectively. Test stimuli were presented at the remapped location before saccades. (B) Schematic description of a trial for measuring the TAE. (C) Time course of events in a trial. RL: remapped location; HEP: horizontal eye position; FP: fixation position; ST: saccadic target; AS: adapting stimulus; and TS: test stimulus. (D) Psychometric functions showing orientation judgments after adapting to the left or right tilted adaptor. The abscissa refers to the orientation of test stimuli. L and R indicate that a test stimulus was left or right tilted. The ordinate refers to the percentage of trials in which subjects indicated that a test stimulus was right tilted. (E) TAE magnitudes in the saccade and no-saccade conditions. Data are plotted for each subject (lines and circles) as well as the group means (bars). The asterisk indicates a statistically significant difference between the two conditions (*p < 0.05). Error bars denote 1 SEM calculated across subjects for each condition.

Methods

Subjects

A total of thirty-four naive subjects (18 male and 16 female) participated in the study. The numbers of subjects in Experiments 1–6 were six, five, five, seven, six, and five, respectively. None of them was involved in more than one experiment. Participants were right-handed with reported normal or corrected-to-normal vision and had no known neurological or visual disorders. Their ages ranged from 20 to 30. Participants gave written, informed consent in accordance with the procedures and protocols approved by the human subject review committee of Peking University. 

Apparatus and stimuli

Visual stimuli were presented on a uniform gray background at a viewing distance of 57 cm. The background luminance was 43.4 cd/m² in Experiment 4 and 10.5 cd/m² in other experiments. The display was an IIYAMA color graphic monitor (model: HM204DT; refresh rate: 60 Hz; resolution: 1024 × 768; size: 22 in.). In Experiment 4, stimuli were rendered with a video card with 8-bit input resolution and 14-bit output resolution using Cambridge Research System Bits++. The output luminance of the display monitor was linearized using a look-up table in conjunction with photometric readings from a colorCAL colorimeter (Cambridge Research System). Subjects' head position was stabilized using a chin and head rest. We used an SMI iView X™ Hi-Speed 500 eye tracker to monitor eye position and the SMI BeGaze™ Eye Tracking Analysis Software to process eye movement data. The velocity threshold for saccade detection was set as 30°/s. 

In Experiments 1, 2, and 5, adapting stimuli were two circular patches of sinusoidal gratings with a randomized phase (radius: 1.96°; contrast: 1.0; spatial frequency: 1.5 cycles/°; orientation: −20° or +20°; “−” and “+” indicate that a grating was left or right tilted; Figure 2A). Test stimuli were similar to the adapting stimuli, except that their contrast was 0.2, and they were oriented at one of five angles (−6°, −3°, 0°, 3°, 6°). 

In Experiment 3 and 6, adapting stimuli were two 100% coherent RDKs (random dot kinematograms) consisting of 150 dots (contrast: 1.0; diameter: 0.08°). The dots moved at a velocity of 4°/s, either upward or downward, within a virtual circular area subtending 5.88° in diameter. Test stimuli were similar to the adapting stimuli, except that their contrast was 0.08 and they moved at one of five speeds (2°/s upward, 1°/s upward, 0°/s, 1°/s downward, 2°/s downward). 

In Experiment 4, adapting stimuli were two circular patches of sinusoidal gratings with a randomized phase (radius: 1.96°; contrast: 1.0; spatial frequency: 1.5 cycles/°; orientation: 0° or 90°). Test stimuli were otherwise identical to the adapting stimuli except that their orientation was 0° and their contrast was one of five values (0, 0.005, 0.01, 0.02, and 0.03). In all the experiments, the centers of adapting and test stimuli were 7.07° away from the initial fixation point, in either the upper right or the upper left direction (Figure 2A). Test stimuli were presented at the remapped location of adapting stimuli. 

Design

For all the experiments in the study, we used a method of constant stimuli to measure visual aftereffects at the remapped location of adapting stimuli. In an adaptation block, there was only one adaptor. Before the presentation of test stimuli, subjects experienced 30-s preadaptation and 3-s topping-up adaptation. 

Experiment 1 measured the tilt aftereffect (TAE). There were two experimental conditions: the “saccade” condition and the “no saccade” condition. Each condition had 10 blocks of 40 trials (five blocks for each adaptor). A saccade block began with a preadaptation at the upper right of the initial fixation point (Figure 2A). In a trial, after a topping-up adaptation and a 200–300-ms blank interval, the position of the fixation point was displaced by 10° to the other side of the screen, which served as a cue for participants to make a saccadic eye movement to the new position of the fixation point. Following the displacement, after a 50–250-ms blank interval, a test stimulus was presented for 33 ms at the upper left of the initial fixation position (i.e., the remapped location of the adaptor). At the end of the trial, subjects were asked to make a two-alternative-forced-choice (2-AFC) response to indicate that the test stimulus was left or right tilted (Figure 2B). Since human saccade latency is about 200 ms (Robinson, 1964), it is very likely that the test stimulus appeared before the onset of the saccade. Note that only trials in which a test stimulus appeared before saccade onset were included for data analysis. In a block, each of the five test stimuli was presented eight times. Blocks in the no-saccade condition were similar to the saccade blocks, except that there was no displacement of the fixation point and hence no saccade was triggered. 

Experiment 2 was designed to examine whether there was TAE at the remapped location when the test stimuli were presented before the onset of a saccadic cue. The key manipulation of this experiment was to display the saccadic target for a variable duration before the test stimuli and the saccadic cue (Figure 3A). There were three experimental conditions: the “long preview” condition, the “short-preview” condition, and the no-saccade condition. Each condition had 10 blocks of 40 trials (five blocks for each adaptor). A long preview or short preview block began with a preadaptation at the upper right of the initial fixation point. In a long preview or short preview trial, after a topping-up adaptation, a saccadic target was presented 10° to the other side of the screen, followed by an 800–1200-ms (the long preview condition) or 300–400-ms (the short preview condition) interval. During the interval, a test stimulus was presented for 33 ms at the upper left of the initial fixation point. The gap between the onset of the saccadic target and the onset of the test stimulus was 400–800 ms (the long preview condition) or 100–150 ms (the short preview condition). Then the initial fixation point disappeared, which served as a cue to make a saccadic eye movement to the saccadic target. After the saccade, subjects were asked to make a 2-AFC judgment to indicate that the test stimulus was left or right tilted. In a block, each of the five test stimuli was presented eight times. Blocks in the no saccade condition were similar except that the initial fixation point remained on the screen throughout blocks, and no saccade was made. 

Figure 3

 

Procedure, time course of events, and results of Experiment 2. (A) Schematic description of a trial for measuring the TAE. (B) Time course of events in a trial. HEP: horizontal eye position; FP: fixation position; ST: saccadic target; AS: adapting stimulus; and TS: test stimulus. (C) Psychometric functions showing orientation judgments after adapting to the left or right tilted adaptor. The abscissa refers to the orientation of test stimuli. L and R indicate that a test stimulus was left or right tilted. The ordinate refers to the percentage of trials in which subjects indicated that a test stimulus was right tilted. (D) TAE magnitudes in the long-preview, short-preview, and no-saccade conditions. Data are plotted for each subject (lines and circles) as well as the group means (bars). Asterisks indicate a statistically significant difference between two conditions (*p < 0.05; **p < 0.01). Error bars denote 1 SEM calculated across subjects for each condition.

Experiment 3 measured the motion aftereffect (MAE). There were two experimental conditions: the long preview condition and the no-saccade condition. Each condition had 10 blocks of 40 trials (five blocks for each adaptor). The procedure of Experiment 3 was similar to that of Experiment 2 except that both the adapting and test stimuli were RDKs. Following adaptation, subjects were asked to make a 2-AFC judgment on the motion direction of the test stimuli (upward or downward). 

Experiment 4 measured the threshold elevation aftereffect (TEAE). There were two experimental conditions: the long preview condition and the no saccade condition. Each condition had 10 blocks of 40 trials (five blocks for each adaptor). A long preview block began with a preadaptation at the upper right of the initial fixation point. In a trial, after a topping-up adaptation, a saccadic target was presented 10° to the other side of the screen, followed by a 200–300 ms blank interval. Then, a test stimulus at the upper left of the initial fixation point was presented in one of two intervals. The two successive intervals were each presented for 100 ms, accompanied by an auditory beep and separated by a 300 ms blank interval. Subsequently, after a 200–300 ms blank interval, the initial fixation point disappeared, which served as a cue to make a saccadic eye movement to the saccadic target. Note that the orientation of the test stimulus was fixed, and the adapting stimulus was either parallel or orthogonal to the test stimulus. Subjects needed to identify which of the two intervals the test stimulus was presented in. Blocks in the no-saccade condition were similar except that the initial fixation point remained on the screen throughout a block, and no saccade was made. 

Experiments 5 and 6 aimed to examine the location specificity of predictive remapping by measuring visual aftereffects, not only at the remapped location of the adapting stimuli, but also at a control location (Figure 5A). Similar to Experiments 2 and 3, Experiments 5 and 6 measured the TAE and MAE, respectively. The control location was 7.07° away from the initial fixation point in the lower right direction. 

Figure 4

 

Results of Experiments 3 and 4. (A) Psychometric functions showing motion direction judgments after adapting to the upward- or downward-moving adaptor. The abscissa refers to the direction of test stimuli. U and D indicate that a test stimulus moved upward or downward. The ordinate refers to the percentage of trials in which subjects indicated that a test stimulus moved downward. (B) MAE magnitudes in the long-preview and no-saccade conditions. (C) Psychometric functions showing contrast detection performance after parallel or orthogonal adaptation. (D) TEAE magnitudes in the long-preview and no-saccade conditions. Data are plotted for each subject (lines and circles) as well as the group means (bars). Asterisks indicate a statistically significant difference between two conditions (**p < 0.01). Error bars denote 1 SEM calculated across subjects for each condition.

Figure 5

 

Results of Experiments 5 and 6. (A) The spatial location and the remapped location of the grating adaptor and the control location. RL: remapped location; AS: adapting stimulus; and CL: control location. (B) Psychometric functions showing orientation judgments after adapting to the left or right tilted adaptor. The abscissa refers to the orientation of test stimuli. L and R indicate that a test stimulus was left or right tilted. The ordinate refers to the percentage of trials in which subjects indicated that a test stimulus was right tilted. (C) TAE magnitudes at the remapped location and the control location. (D) Psychometric functions showing motion direction judgments after adapting to the upward- or downward-moving adaptor. The abscissa refers to the direction of test stimuli. U and D indicate that a test stimulus moved upward or downward. The ordinate refers to the percentage of trials in which subjects indicated that a test stimulus moved downward. (E) MAE magnitudes at the remapped location and the control location. Data are plotted for each subject (lines and circles) as well as the group means (bars). Asterisks indicate a statistically significant difference between two conditions (*p < 0.05; **p < 0.01). Error bars denote 1 SEM calculated across subjects for each location.

For all the experiments, the orders of experimental conditions and test stimuli were randomized. Subjects were given one practice block for each experimental condition. They took a rest of at least 3 min between blocks to minimize any carryover adaptation effects from previous blocks. 

Data analysis

To quantify the TAE magnitude, for each of the ±20° tilted adaptors, the percentage of trials in which a test grating was perceived to be right tilted was plotted as a psychometric function of its real orientation. We used a cumulative normal function to fit the psychometric values and interpolated to find the perceived vertical. TAE magnitude was defined as half of the orientation difference between the perceived verticals caused by the two adaptors. To quantify the MAE magnitude, for the upward- or downward-moving adaptor, the percentage of trials in which a test RDK was perceived to move downward was plotted as a psychometric function of its real speed. The psychometric values were fit with a cumulative normal function, and we interpolated to find the speed expected to be perceived stationary. MAE magnitude was defined as half of the speed difference between the two adaptor conditions. For the TEAE measurement, after parallel or orthogonal adaptation, the percentage of trials in which a low-contrast test grating was correctly detected was plotted as a psychometric function of its contrast. We used a Weibull function to fit the psychometric values and interpolated to find the contrast detection threshold at 80% correct (Figure 4B). TEAE magnitude was defined as the ratio of the contrast detection threshold after parallel adaptation to that after orthogonal adaptation. 

Results

Experiment 1: Predictive remapping of orientation before saccade

We measured the TAE at the remapped location of the adapting stimuli in the saccade condition and the no-saccade condition. One-sample t tests showed a significant TAE in the saccade condition, t(5) = 3.92, p = 0.011, but little TAE in the no-saccade condition, t(5) = 0.92, p = 0.398. Moreover, a paired t test showed that the magnitude of the TAE in the saccade condition was significantly larger than that in the no-saccade condition, t(5) = 3.82, p = 0.012 (Figure 2D and 2E). These results demonstrated that the orientation information could be predictively remapped before saccades. Given the dependence of TAE on early and intermediate visual cortical areas (Blakemore & Campbell, 1969; Boynton & Finney, 2003; Fang, Murray, Kersten, & He, 2005; Maffei, Fiorentini, & Bisti, 1973; Movshon & Lennie, 1979), the predictive remapping of orientation during trans-saccadic perception might entail these areas. 

Experiment 2: Time-dependent predictive remapping of orientation

Although Experiment 1 has shown that orientation information could be transferred before saccades, it remains unknown whether this transfer process is time dependent. Here we displayed the saccadic target for a variable duration before the test stimuli and the saccadic cue, therefore manipulating the preview time (Figure 3A). At the remapped location of the adapting stimuli, one-sample t tests showed significant TAEs in the long-preview condition, t(4) = 8.14, p < 0.01, and the short-preview condition, t(4) = 3.97, p = 0.017, but little TAE in the no saccade condition, t(4) = 0.16, p = 0.878. Moreover, paired t tests showed that both the TAE magnitudes in the long preview condition and the short preview condition were significantly larger than that in the no saccade condition: long preview versus no saccade: t(4) = 7.05, p < 0.01; and short preview versus no saccade: t(4) = 3.67, p = 0.021 (Figure 3C and 3D). Furthermore, the TAE magnitude in the long preview condition was significantly larger than that in the short preview condition, t(4) = 6.08, p < 0.01. These results revealed that the predictive remapping effect increased with the latency of the planned saccade, suggesting that the orientation remapping process is time dependent. 

Experiment 3: Predictive remapping of motion direction before saccade

To examine whether motion direction could be predictively represented before saccades, we measured the MAE at the remapped location of the motion adaptors. In Experiments 3–6, we adopted the long preview protocol from Experiment 2 because of its effectiveness in generating the transferred TAE. One-sample t tests showed a significant MAE in the long-preview condition, t(4) = 3.62, p = 0.022, but little MAE in the no-saccade condition, t(4) = 0.28, p = 0.791. Moreover, a paired t test showed that the MAE magnitude in the long-preview condition was significantly larger than that in the no-saccade condition, t(4) = 4.58, p = 0.01 (Figure 4A and 4B). These results demonstrated that the motion direction information could be remapped before saccades. It is well known that MAE originates in early and intermediate visual cortical areas, especially area MT (Kohn & Movshon, 2003; Seiffert, Somers, Dale, & Tootell, 2003; Tootell et al., 1995; Van Wezel & Britten, 2003). Therefore, the neural site responsible for the predictive remapping of motion direction might be located in these areas. 

Experiment 4: Contrast information cannot be predictively remapped

We measured the TEAE at the remapped location of the high-contrast grating adaptors to examine whether contrast can be remapped before saccades. One-sample t tests showed that there was no significant TEAE in either the long-preview condition, t(6) = 0.779, p = 0.465, or the no-saccade condition, t(6) = 0.105, p = 0.919. Moreover, a paired t test showed that there was no significant difference between these two conditions, t(6) = 0.723, p = 0.497 (Figure 4C and 4D). These results demonstrated that the contrast information could not be predictively remapped before saccades. Since TEAE is believed to be profoundly mediated by neurons in the primary visual cortex (V1) and the lateral geniculate nucleus (Fang et al., 2005; Sclar, Lennie, & Depriest, 1989; Solomon, Peirce, Dhruv, & Lennie, 2004), we speculate that V1 and LGN might play a negligible role in predictive feature remapping during trans-saccadic perception. 

Experiments 5 and 6: Location specificity of predictive feature remapping

Although we have found that orientation and motion direction features could be transferred to the remapped location of the adapting stimuli before saccades, it remains unknown whether the predictive remapping is constrained to the remapped location we have tested before. To address this issue, we measured the TAE and the MAE at the remapped location and another location, i.e., the control location (Figure 5A). In Experiment 5, one-sample t tests showed a significant TAE at the remapped location, t(5) = 9.07, p < 0.001, but not at the control location, t(5) = 1.67, p = 0.155. Moreover, a paired t test showed that the magnitude of the TAE at the remapped location was significantly larger than that at the control location, t(5) = 6.66, p < 0.001 (Figure 5B and 5C). In Experiment 6, one-sample t tests showed significant MAEs at both the remapped location, t(4) = 8.20, p < 0.001, and the control location, t(4) = 2.88, p = 0.045. Notably, a paired t test showed that the magnitude of the MAE at the remapped location was significantly larger than that at the control location, t(4) = 3.95, p = 0.017 (Figure 5D and 5E). These results demonstrated that the predictive remapping of orientation and motion direction was location specific. 

Discussion

We measured the visual adaptation aftereffects (TAE, MAE, and TEAE) at the ‘appropriate' remapped location of the adapting stimuli before saccades to investigate whether the predictive representation of basic visual features (orientation, motion direction, and contrast) can form at the remapped location prior to saccades. At the remapped location, we found a significant presaccadic TAE and MAE, but failed to observe a significant TEAE. Furthermore, the TAE and MAE showed spatial specificity before saccades. Together, these findings suggest that predictive representations before saccades can form at the ‘appropriate' remapped location and they contain orientation and motion direction information, but not contrast information, supporting the predictive feature remapping theory. 

Rolfs et al. (2011) studied the functional correlates of predictive remapping of targets of saccadic eye movements. They revealed a robust increase in visual performance at the remapped, future retinal locations of a sequence of movement goals occurring before the onset of saccades. They attributed the performance increase to attentional pointers to the remapped locations updated by a predictive remapping process. Notably, our study here provided clear evidence that, at the remapped location, in addition to the spatial attentional effect found by Rolfs et al. (2011), certain feature information was also transferred from the adaptor location, which cannot be easily explained by the remapping theory of attention pointers (Cavanagh et al., 2010). 

In a neurophysiological study, Nakamura & Colby (2002) investigated the remapping mechanism in visual cortical areas of macaque monkeys. They found that the representation of a briefly presented stimulus was remapped around the time of saccades in extrastriate areas V2, V3, and V3A, while remapping was very rare in striate cortex (V1). Merriam, Genovese, and Colby (2007) used fMRI (functional magnetic resonance imaging) to examine whether human striate and extrastriate cortex have access to remapped information. They found that remapping was strongest in extrastriate areas V3A and hV4 and was less robust in V1 and V2. These studies are consistent with our psychophysical findings here. Orientation and motion direction are mediated not only by striate cortex, but also by extrastriate cortex (Blakemore & Campbell, 1969; Boynton & Finney, 2003; Fang et al., 2005; Kohn & Movshon, 2003; Maffei et al., 1973; Movshon & Lennie, 1979; Seiffert et al., 2003; Tootell et al., 1995; Van Wezel & Britten, 2003). Contrast is mainly processed at striate cortex and even earlier stages of the visual hierarchy (e.g., lateral geniculate nucleus; Fang et al., 2005; Sclar et al., 1989; Solomon et al., 2004; Yu, Zhang, Qiu, & Fang, 2016). It has been proposed that predictive remapping takes place at intermediate processing stages (i.e., extrastriate visual cortex), which is supported by extensive interconnections between extrastriate visual cortex and several key areas for predictive remapping, like LIP and FEF (Baizer, Ungerleider, & Desimone, 1991; Blatt, Andersen, & Stoner, 1990; Schall, Morel, King, & Bullier, 1995; Stanton, Bruce, & Goldberg, 1995). This proposition explains why only the TAE and MAE, but not the TEAE, could be transferred to the remapped location. 

In Experiment 2, we found that the magnitude of the remapped TAE increased with the previewing time, suggesting that the predictive remapping of the orientation information is not instantaneous, and it develops continuously during the interval between the onset of the saccadic target and the onset of the saccadic cue. Recently, Zimmermann, Morrone, Fink, and Burr (2013) used a similar protocol to measure trans-saccadic TAE and found that the spatiotopic representation of orientation across saccades needed hundreds of milliseconds to build up robustly, which is resonant with our finding here. Here, we speculate that the predictive remapping mechanism might contribute to the formation of spatiotopic representation of visual features. When subjects fixate at the initial fixation point, activities of neurons whose receptive fields cover the location of the adaptor decrease gradually during the time course of adaptation. Upon subjects' intention to make a saccade, this activity decrease is transferred to neurons whose receptive fields will encompass the adaptor location after the saccade, as demonstrated by the evident TAE and MAE at the remapped location of the adaptors before saccades. The transfer of the activity decrease (or the transfer of adaptation) therefore leads to postsaccadic spatiotopic aftereffects as observed in previous studies (Melcher, 2005; Melcher & Colby, 2008; Zimmermann et al., 2013; Zimmermann, Weidner, Abdollahi, & Fink, 2016). The location specificity revealed in Experiments 5 and 6 implies that spatiotopic aftereffects could not simply be attributed to the spreading of retinotopic aftereffects from the location of visual adaptors (see Knapen, Rolfs, Wexler, & Cavanagh, 2010). Otherwise, we should have observed the same amount of aftereffects at the control location as those at the remapped location. Notably, in our study, we failed to find any transfer of the TEAE. An earlier study by Melcher (2005) identified a significant spatiotopic TAE but not spatiotopic TEAE during trans-saccadic perception. The consistency between these two studies supports our speculation. 

Why is the remapped TAE dependent on the previewing time? One possible explanation is cortical feedback. Feature remapping may take place initially in frontal and parietal cortex, and the remapping effects we observed here may reflect the influence of feedback projections from the source of high-level signals that drives remapping to extrastriate cortical areas (Baizer et al., 1991; Blatt et al., 1990; Cavada & Goldman-Rakic, 1989; Cavanaugh, Berman, Joiner, & Wurtz, 2016; Kamitani & Tong, 2005; Schall et al., 1995). Another possible explanation is that, although feature remapping is instantaneous, remapping of feature adaptation (i.e., transfer of feature adaptation) between receptive fields is time consuming. Note that, in the current study, we use visual adaptation as a tool to quantify feature remapping, rather than measure the feature remapping process directly. 

Although many previous electrophysiological and fMRI studies have shown that neurons in LIP, SC, FEF, and extrastriate cortex increase their responses to a target outside their classical receptive fields when saccades are about to bring that target into their receptive fields (Colby et al., 1995; Duhamel et al., 1992; Merriam et al., 2007; Nakamura & Colby, 2002; Umeno & Goldberg, 1997; Walker et al., 1995; Wang et al., 2016; Zirnsak, Steinmetz, Noudoost, Xu, & Moore, 2014), so far there is almost no direct neural evidence showing that the response increases contain feature information or are feature selective in visual cortex (only Subramanian & Colby, 2014, examined shape selectivity in macaque LIP). In the future, this issue can be resolved by using more detailed neurophysiological measurements or more sophisticate data analyses (e.g., fMRI decoding analysis). 

Acknowledgments

This work was supported by NSFC 31230029, MOST 2015CB351800, NSFC 31421003, NSFC 61621136008, NSFC 61527804, and NSFC 31671168. Dongjun He was supported in part by the Postdoctoral Fellowship of Peking-Tsinghua Center for Life Sciences and China Postdoctoral Science Foundation (2016M590006). 

Commercial relationships: none. 

Corresponding author: Fang Fang. 

Email: . 

Address: School of Psychological and Cognitive Sciences, Peking University, Beijing, China.