How forward remapping predicts peri-saccadic biphasic mislocalization

Abstract

Neurons in many visual and oculomotor “priority-map” brain areas display forward receptive field (RF) remapping: they respond to stimuli appearing before a saccade at the spatial location that their RF will occupy after the saccade. Concurrently, psychophysical studies have shown that flashes around saccade onset are systematically mislocalized in various patterns. One prominent pattern is a biphasic pattern, where flashes right before a saccade are mislocalized in the saccade direction (forward) and flashes right after a saccade are mislocalized opposite to the saccade direction (backward). Although forward RF remapping and biphasic mislocalization have been suspected to be linked, how this works has never been explained. Here, we show how persistent flash-evoked activity and decoding of the flash position after the saccade combine to produce this biphasic mislocalization pattern. We implement a rate model, consistent with the essential properties of RF remapping, and show that biphasic mislocalization results from insufficient remapping before the saccade, and residual/inappropriate remapping after the saccade. Less remapping before the saccade produces larger forward mislocalization of pre-saccadic flashes, and less remapping after the saccade produces smaller backward mislocalization of post-saccadic flashes. Forward RF remapping thus captures a biphasic peri-saccadic flash-mislocalization pattern consistent with behavioral data.

Introduction

When we view the world, we move our gaze to bring light from different parts of the scene to the fovea. A given static stimulus in the visual scene therefore falls on different parts of the retina before and after a saccade. It has been shown that when a stimulus is briefly flashed around the time of the saccade (from slightly before the saccade to slightly after) and human subjects are asked to report the position of the flash after the saccade (via an eye movement, an arm movement like pointing, or a perceptual report of the flash location), they systematically mislocalize the position of the flash in characteristic patterns that have been extensively described (Awater & Lappe, 2006; Binda, Cicchini, Burr, & Morrone, 2009; Cicchini, Binda, Burr, & Morrone, 2013; Honda, 1995; Kaiser & Lappe, 2004; Lappe, Awater, & Krekelberg, 2000; Pola, 2011b; Pola, 2010; Ross, Morrone, Goldberg, & Burr, 2001; Schlag & Schlag-Rey, 2002; van Wetter & van Opstal, 2008a). Two major trends in these patterns are a) a classical, biphasic mislocalization where pre-saccadic flashes are mislocalized in the direction of the saccade and post-saccadic flashes are mislocalized opposite to the direction of the saccade (Honda, 1989; Honda, 1991; Honda, 1995; Honda, 1999), and b) a convergent mislocalization where peri-saccadic flashes are mislocalized toward the saccade target (Hamker, Zirnsak, Ziesche, & Lappe, 2011; Ross, Morrone, & Burr, 1997). These observed patterns have been noted to depend on the presence of visual references (Lappe, Awater, & Krekelberg, 2000), luminance (Georg, Hamker, & Lappe, 2008), saccade amplitude (van Wetter & van Opstal, 2008a), and so on. A variety of explanations (Hamker et al., 2011; Ziesche & Hamker, 2011) have been proposed for different aspects of these varied patterns of mislocalization (Klingenhoefer & Krekelberg, 2017), including visual transmission latency (Pola, 2004; Pola, 2007; Pola, 2011a), a sluggish eye-position signal (Dassonville, Schlag, & Schlag-Rey, 1992; Schlag & Schlag-Rey, 2002), spatial representational geometry (Richard, Churan, Guitton, & Pack, 2009; VanRullen, 2004), and peri-saccadic receptive field (RF) remapping (Binda et al., 2009; Ross et al., 1997; Ross et al., 2001; Wurtz, 2008; Zirnsak & Moore, 2014; Zirnsak, Steinmetz, Noudoost, Xu, & Moore, 2014). 

Here, we focus on the last of these explanations, that forward peri-saccadic remapping of neuronal RFs may account for mislocalization. Two kinds of remapping have been reported (Marino & Mazer, 2016), although distinguishing them and delineating their relative contributions can be complex (Hartmann, Zirnsak, Marquis, Hamker, & Moore, 2017; Neupane, Guitton, & Pack, 2020; Wang et al., 2024b; Zirnsak, Lappe, & Hamker, 2010) (see Discussion). In the classical (or forward) form of remapping, neurons in many areas of the brain like the lateral intraparietal area (LIP) (Duhamel, Colby, & Goldberg, 1992; Wang et al., 2016), frontal eye field (FEF) (Sommer & Wurtz, 2006; Umeno & Goldberg, 1997), superior colliculus (SC) (Churan, Guitton, & Pack, 2011; Churan, Guitton, & Pack, 2012; Walker, Fitzgibbon, & Goldberg, 1995), V4 (Neupane, Guitton, & Pack, 2016a; Neupane, Guitton, & Pack, 2016b), and V3a (Nakamura & Colby, 2002) respond to stimuli at the spatial location that their RF will occupy after the saccade. In a good proportion of neurons, this is observed for stimuli appearing even before the saccade begins, suggesting a role for corollary discharge (CD) (Helmholtz, 2000; Sperry, 1950; van Holst & Mittelstaedt, 1950) information about the impending saccade in this remapping process. In the second convergent form of remapping (Neupane et al., 2016a; Neupane et al., 2016b; Tolias et al., 2001; Zirnsak & Moore, 2014; Zirnsak et al., 2014), around the time of the saccade, neurons in FEF and V4 have been shown to respond more to stimuli flashed closer to the saccade target, compared to their response during steady fixation. It has been generally believed (Zirnsak & Moore, 2014; Zirnsak et al., 2014) that the classical biphasic and convergent forms of mislocalization correspond directly to the classical (forward) and convergent forms of peri-saccadic remapping. We showed in a preprint in 2023 how classical remapping, combined with persistent flash-evoked activity and decoding of the flash position from the post-saccadic population representation, produces a biphasic mislocalization pattern (Berreby & Krishna, 2023). To our knowledge, this was the first time a mechanism connecting classical remapping to biphasic mislocalization had been described. Here, we implement this mechanism in a model, consistent with the essential properties of RF remapping, and show that biphasic mislocalization results from insufficient remapping right before the saccade, and residual/inappropriate remapping after the saccade. We also show that forward RF remapping thus captures the biphasic peri-saccadic flash mislocalization pattern seen in behavioral data. 

Methods

We consider the simple example of a horizontal rightward saccade of size 8° (Figures 1A, 1B), starting with the participant fixating the center of a screen, at screen coordinates (0°, 0°). The specific choices of amplitude, position and direction in this paragraph are for illustrative purposes only. At some instant before, during or after the saccade, a flash is displayed above the center of the screen, at screen coordinates (0°, 10°). After this first saccade, the participant must report the perceived location of the flash on the screen by making a second saccade to it; we consider other ways of reporting the perceived location in the Results. We postulate, following current theory (Bisley & Mirpour, 2019; Bisley, Mirpour, & Alkan, 2020; Cavanagh, Hunt, Afraz, & Rolfs, 2010; Goldberg, Bisley, Powell, & Gottlieb, 2006; Mirpour & Bisley, 2016), that remapping updates the population response profiles in the brain’s (fundamentally) retinotopic priority maps across a saccade, such that the map represents the locations of task-relevant and salient stimuli both before and after the saccade (Bisley et al., 2020; Mirpour & Bisley, 2012; Mirpour & Bisley, 2016). Flashes that appear well before the saccade are both salient and task-relevant, since the participant has to report their location after the saccade. The pre-saccadic flash response accurately represents the spatial location of the flash such that a saccade made directly to this response peak would foveate the flash. However, since there is an intervening saccade to a different saccade target in this task, as indicated by a large body of psychophysical and physiological data (Basu & Murthy, 2020; Bisley et al., 2020; Ipata, Bisley, & Krishna, 2023; McPeek & Keller, 2002; Mirpour & Bisley, 2016; Mohsenzadeh, Dash, & Crawford, 2016; Munuera, Morel, Duhamel, & Deneve, 2009; Zerr, Thakkar, Uzunbajakau, & Van der Stigchel, 2016), when the participant makes a saccade 8° toward the right, this population response profile is (to a first approximation) accurately remapped by 8° to the left. Thus, after the initial saccade, a second saccade made to the new population response peak at (−8°, 10°) foveates the spatial location where the flash appeared so that there is no mislocalization. 

Figure 1.

 

Model components. (A) Basic task design. A rightward horizontal saccade of amplitude 8° is made from the center of the screen S (0°, 0°) at t = 0 ms, to the endpoint E (8°, 0°). Some time around the saccade, a flash is displayed on screen at (0°, 10°). Its perceived location, on the basis of which a report will be made, need not coincide with its true location. In the presence of mislocalization, the perceived location of the flash may be shifted forward (to the right, in the direction of the saccade) or backward (to the left, opposite to the saccade). In this example, the flash is mislocalized by ±3.5°. (B) Time course of the eye movement. The saccade begins at t = 0 ms (black vertical line, S = Start), progresses at constant speed, and reaches its endpoint T_sac = 35 ms after saccade onset (green vertical line, E = End). (C) Temporal CD signal η(t). η is asymmetric, chosen so as to peak around saccade onset while keeping a significant portion of its area lying after t_crit = 46 ms (blue vertical line). Decoding of the flash location occurs at t_dec = 300 ms (purple vertical line), after remapping is complete and η has decayed back to a negligible value. (D) Temporal profile of the feedforward input signal. The population input is delayed relative to the flash onset by Δ_FF = 40 ms, peaking at t_crit = 46 ms (blue vertical line), \(\tau _{\rm{FF}}^{\rm{on}} = 6\,\mathrm{m}\mathrm{s}\) after it first begins increasing. (E) Spatial profile of the input signal, a Gaussian of standard deviation σ_FF = 2.5°. (F) W₁(x): mexican-hat connectivity kernel underlying persistent activity. W₂(x): antisymmetric connectivity kernel underlying the drift of population responses.

To represent the evolution of the firing rate of priority-map neurons across time, in response to feedforward input from lower visual areas and in the presence of peri-saccadic forward remapping, we propose an explicit rate model, based on dynamic field theory (Schöner, 2016). In the model, we assume that a brief task-relevant flash stimulus (that the subject has to localize in the task) presented well before the saccade elicits persistent activity; it is this persistent activity that is then remapped across the saccade (see Discussion). Here, we provide an overview of our model. A more detailed description, including all relevant equations and a summary table of all key parameters Table A1, is provided in the Appendix (Method details). For simplicity, we focus on describing the one-dimensional case. We consider 1,001 neurons coding for 200° of visual angle. The spatial coordinate system is chosen such that positive x coordinates represent locations to the right of the fovea, and x = 0 corresponds to the center of the fovea. Without loss of generality, we choose the temporal origin t = 0 ms so as to be the saccade onset time. The neuron coding for position x has, at time t, an associated membrane potential u(x, t). This membrane potential directly and uniquely determines the deviation r(x, t) of the neuron’s firing rate from its baseline, through a sigmoid activation function. Each neuron receives feedforward input from lower visual areas, and may be excited or inhibited through CD-modulated lateral connections with its neighbors. 

We assume that feedforward input first reaches the model population Δ_FF = 40 ms after stimulus onset on screen, and that the input to the population peaks \(\tau _{\rm{FF}}^{\rm{on}} = 6\,\mathrm{m}\mathrm{s}\) after that. This combined delay, t_crit = 46 ms, is represented by the blue vertical line in Figures 1C, 1D. 

The temporal CD signal (η, Figure 1C) modulates the speed of remapping in a time-dependent fashion, and is evoked by impending saccades. It is ordinarily 0, beginning to rise some time before a saccade and dropping back to 0 some time after it. We represent it as the product of a rising sigmoid and a falling sigmoid, normalized such that its integral is 1. The relative distribution of the area under the curve of η before/after t_crit is the most significant factor determining the relative values of the maximum positive/negative errors predicted by the model (Figure A2). Therefore, to produce a balanced biphasic mislocalization pattern, more remapping must happen after saccade onset than before it. 

Figure 2.

 

Spatio-temporal profiles of model input (A, C, E, G, I) and responses (B, D, F, H, J), in response to a flash presented at the screen position (0°, 10°). Red vertical line: stimulus presentation time. Black vertical line: saccade onset time. Green vertical line: saccade completion time. Purple vertical line: decoding time (t_dec = 300 ms). Purple horizontal line and annotation: barycenter location at t_dec. Neurons are laid out vertically according to their preferred horizontal retinal location (y axis). The cyan line overlaid on population responses represents the evolution of the barycenter of the population’s activity across time. (A, B) Pre-saccadic flash presented abundantly early (t = −250 ms). Persistent activity is elicited before remapping begins, and is thus fully remapped to the appropriate location. No mislocalization occurs. (C, D) Pre-saccadic flash presented a short time before saccade onset (t = −1 ms). The elicited representation appears too late to be fully remapped, and the flash is mislocalized forward by 3.5°. (E, F) Intra-saccadic flash (presented while the saccade is occurring, at t = 17.5 ms). The input falls somewhere in between the pre-saccadic and post-saccadic stimulus locations. In order to accurately localize the flash, its representation must be remapped, though not to the extent that would be appropriate for a pre-saccadic flash. Concurrently, due to the flash’s appearance while the remapping process is already ongoing, only partial remapping happens. For the onset time shown here, these two processes cancel out, yielding correct localization. (G, H) Post-saccadic flash presented immediately after the saccade is complete (t = 35 ms). As the remapping process is still active, the flash’s representation gets inappropriately remapped. The flash is mislocalized backward by 3.5°. I-J: Post-saccadic flash presented long after the saccade is complete (t = 250 ms). No mislocalization occurs.

Figure 3.

 

Flash mislocalization patterns produced by the model, and experimentally observed. Because of forward remapping in the direction of the saccade, our model predicts a biphasic pattern (black curve) consistent with the empirically-observed relationship between perceptual error and flash onset time (semi-transparent colored curves and solid red curve, resp. individual trajectories (Honda, 1990; Honda, 1991; Honda, 1993; Honda, 1999) and locally-weighted scatterplot smoothing trajectory from pooled data, ±95% bootstrap CI). Flashes that appear before the remapping process begins are fully and accurately remapped. Flashes that appear closer to saccade onset (black vertical line) appear after the remapping process has already begun, and are only partially remapped (in the saccade direction). Flashes appearing shortly after the end of the saccade (green vertical line) elicit responses that shouldn’t be remapped, but are still remapped. Partial remapping in the saccade direction leads to forward mislocalization (in the saccade direction) for pre-saccadic flashes and backward mislocalization (opposite to the saccade direction) for post-saccadic flashes. This is because the eye moves (by the amplitude of the saccade) between the pre-saccadic and post-saccadic flashes, causing the flash to appear at different retinal locations in the two cases. Flashes that appear well after the saccade are neither remapped nor mislocalized.

Rather than modeling the visual flash stimulus and its transformation along the path from the retina to the model population, we directly consider the feedforward input received by the population for a brief flash, lasting \(\tau _{\rm{FF}}^{d} = 2\,\mathrm{m}\mathrm{s}\) on screen. We separately consider the temporal envelope I_time(t) (Figure 1D) and the spatial envelope I_space(x) (Figure 1E) of the feedforward input, which together fully define its spatio-temporal shape, I(x, t). The temporal envelope was chosen according to the findings of Bisley, Krishna, and Goldberg (2004), and sports a rapid rise followed by a slower decay, which are offset from the onset of the flash on the screen by Δ_FF = 40 ms, the assumed propagation delay to the model population. The spatial envelope is Gaussian, centered on the retinal position of the stimulus, with a standard deviation of σ_FF = 2.5°. 

Lateral connections are specified in a translation-invariant fashion: at any given time, the strength of the lateral connection between any two neurons in the population solely depends on the relative offset between the locations that they code for. It is represented by the time-dependent connectivity kernel W, where W(x, t) is the strength of the functional connection between any given neuron and a neuron coding for a location offset by x degrees relative to the location that the first neuron codes for, at time t. We consider such lateral connections from a purely functional standpoint; that is, we make no claim that these connections are direct, rather than being conveyed by distinct excitatory and inhibitory subpopulations, or that synaptic weights themselves are being rapidly modulated. 

W is defined as the sum of a constant Mexican hat, Amari-type (Amari, 1977) interaction kernel (W₁, Figure 1F), and an antisymmetric component (Zhang, 1996) (W₂, Figure 1F) that is scaled by the temporal CD signal η and by the spatial CD signal β. Representations are spatially static when η is near zero, and drift otherwise, at a speed proportional to the product βη(t). Because the distance travelled by a bump of activity is simply the integral of its speed, and η is normalized to have an integral of 1, the remapping amplitude scales linearly with β—assuming full exposure to η. Flashes presented sufficiently early before the saccade are fully exposed to η, and thus drift by β/β₀, with β₀ some constant in deg⁻¹. As such, for any given saccade amplitude A_sac, the corresponding remapping amplitude can be obtained by setting β = A_sac · β₀. This formulation allows us to dissociate the time course of remapping (set by η) from its maximum amplitude (set by β). 

Because we only consider individual flashes, we assume that perceived flash locations can be decoded from population responses using the barycenter, or center of mass, of the responses. In the single-flash scenario under consideration, where no more than one activity bump is evoked, this is equivalent to decoding from the peak of the population’s activity. In order to derive a quantitative mislocalization curve, we consider the activity t_dec = 300 ms after the saccade, and compare the location read out using this decoding scheme to the correct location of the flash in the appropriate coordinate system. 

Our model is closely related to, and inspired by, the one proposed by Wang et al. (2024b). We used similar symmetric (W₁) and asymmetric (W₂) connectivity kernel templates in order to induce drifting, and otherwise made independent modeling choices. While some of these are minor, our model does differ from theirs in several important ways. First, for the model to display stable persistent activity in response to brief flashes, we use a saturating, sigmoid activation function, whereas their model uses a non-saturating, rectified linear unit activation function, and appears to require sustained feedforward input in order to display stable persistent activity. Second, their model does not consider the impact of eye movements on decoding, a key aspect of the present work. Third, we consider a more flexible parameterization of the temporal CD signal, with parameters chosen so as to peak around saccade onset time, while still resulting in empirically consistent mislocalization patterns (Figure A2). Last, we only consider CD-gated directional connections enabling forward remapping, without considering the attention-modulated center-surround connections supporting convergent remapping present in their model, as the present work is primarily concerned with modeling forward, rather than convergent, remapping. 

To assess the consistency of our model’s predictions with patterns of mislocalization experimentally observed in humans performing a similar task, we digitized data from Honda (1990); Honda (1991); Honda (1993); Honda (1999); see Method details. 

Results

The total amplitude of remapping is dependent upon both how fast responses drift, and how long they drift. Responses that are fully exposed to η are remapped to the left by A_sac degrees. Thus, flashes that are presented sufficiently early for the resulting population response to be evoked before remapping begins (i.e., before η begins increasing) are fully remapped. Responses to flashes presented too close to saccade onset, however, only partially experience remapping, because they are only evoked after η has already begun increasing. Responses to flashes presented shortly after the saccade, which should not be remapped at all in order to be accurately localized, are nevertheless partially remapped, as they are evoked before η has fully decayed back to 0. Depending on whether the flash is presented before or after the eye has moved, the resulting incomplete or inappropriate remapping will lead to forward or backward mislocalization, respectively. 

We consider five scenarios, each with the same fixed flash location of (0°, 10°) in screen coordinates: 1) early pre-saccadic flash (Figures 2A, 2B)—this is fully remapped and correctly localized; 2) late pre-saccadic flash (Figures 2C, 2D)—this is insufficiently remapped and mislocalized forward; 3) intra-saccadic flash (Figures 2E, 2F)—this is incompletely remapped, but needs less remapping due to falling somewhere between (0°, 10°) and (−8°, 10°) (in retinal coordinates) on the retina; 4) early post-saccadic flash (Figures 2G, 2H)—this is inappropriately remapped and mislocalized backward; and 5) late post-saccadic flash (Figures 2I, 2J)—this is neither remapped nor mislocalized. In each case, we describe the situation when the behavioral report is made via a second saccade toward the perceived flash location; however, our analysis generalizes fully to other forms of reports (see Discussion). 

Early pre-saccadic flash

When the flash is presented well before the saccade (Figures 2A, 2B), while the eye is fixating at screen location (0°, 0°), it is fully and appropriately remapped. The peak of the persistent activity elicited by the flash is stable and unmoving at retinotopic coordinates (0°, 10°) until the remapping process begins, gradually drifts while remapping takes place, and comes to a rest at (−8°, 10°), its intended destination, remaining there until readout occurs. After the saccade, the eye is fixating the position (8°, 0°) on the screen. Thus, a saccade of (−8°, 10°) (i.e. 8° to the left and 10° upwards), made based on the peak at (−8°, 10°) on the retinotopic priority map, will lead to the eye fixating the point (0°, 10°) on the screen. This is where the flash appeared. Thus, no mislocalization occurs. 

Late pre-saccadic flash

When the flash appears just before the saccade begins (Figures 2C, 2D), the remapping process is already well underway by the time the input evoked by the flash reaches the population. Because the flash is shown before saccade onset, while the eye is fixating at (0°, 0°), the response first appears at (0°, 10°), as before. When it comes to a rest at the end of the remapping period, its peak only reaches the retinotopic location (−4.5°, 10°), 3.5° to the right of (−8°, 10°), the retinal location post-saccadically mapping to the spatial location of the flash. In other words, there is a backward shift (opposite to the saccade direction) of the population response profile peak. After the saccade, as in each of the five cases we consider here, the eye is fixating the position (8°, 0°) on the screen. Based on the population peak at (−4.5°, 10°), the perceived flash location will be reported with a saccade of amplitude (−4.5°, 10°), leaving the eye fixating the point at the screen coordinates (3.5°, 10°). Thus, a pre-saccadic flash shown just before saccade onset is mislocalized forward, in the saccade direction. 

Intra-saccadic flash

When the flash appears while the saccade is underway (Figures 2E, 2F), the precise timing of its appearance may lead to either forward or backward mislocalization. Responses to intra-saccadic flashes will always experience partial remapping (relative to the saccade amplitude); however, they will also be first evoked at a location that falls somewhere between the pre-saccadic and the post-saccadic retinal location of the stimulus. The further along the saccade the flash is presented, the closer the elicited response to the post-saccadic retinal location of the stimulus, and the less time is available to experience remapping. For any given choice of model parameters, there is a unique stimulus presentation time for which these two effects precisely cancel out, yielding no mislocalization. This is the case on Figures 2E, 2F. 

Early post-saccadic flash

When the flash appears soon after the saccade is over (Figures 2G, 2H), while the eye is fixating at (8°, 0°), population responses are still being remapped when the input evoked by the flash reaches the population. When readout occurs, the population response profile peaks at the retinotopic location (−11.5°, 10°), 3.5° to the left of the actual retinal flash location (−8°, 10°); i.e. there is a backward shift (opposite to the saccade direction) of the population response profile peak. Since the eye is fixating at (8°, 0°) on the screen, a saccade-based behavioral report toward the retinotopic population profile peak at (−11.5°, 10°) leaves the eye fixating at (−3.5°, 10°) on the screen, thus showing a backward mislocalization of −3.5° (opposite to the saccade direction). Forward RF remapping thus leads to backward mislocalization of flashes soon after saccade end (while the RFs are still in a remapped state). 

Late post-saccadic flash

When the flash appears long enough after the saccade is over (Figures 2I, 2J), the evoked response does not drift, as the remapping process is already complete. Since the saccade has been executed before the flash appeared, the eye is fixating at (8°, 0°) when the stimulus is presented and the population response profile peaks at the retinotopic location (−8°, 10°). A saccade-based behavioral report toward the retinotopic population profile peak at (−8°, 10°) leaves the eye fixating at (0°, 10°) on the screen, thus showing no mislocalization. 

We have thus demonstrated how classical forward RF remapping can predict biphasic spatial mislocalization of peri-saccadic flashes: pre-saccadic flashes are mislocalized in the saccade direction and post-saccadic flashes are mislocalized opposite to the saccade direction. The principles underlying the results detailed in Figure 2 for five notable stimulus onset times can be generalized to any stimulus onset time. Simulating the evolution of our model neurons for a dense set of stimulus onset times allows us to derive the full quantitative mislocalization curve for peri-saccadic flashes based on the classical forward RF remapping mechanism under consideration, depending on when the flash appears with respect to the saccade (Figure 3). The essential idea here is that forward remapping is only partial right before the saccade, leading to forward mislocalization of pre-saccadic flashes, and that there is residual remapping after the saccade, leading to backward mislocalization of post-sacccadic flashes. Smaller magnitudes of remapping before the saccade lead to larger forward mislocalization of pre-saccadic flashes, and smaller magnitudes of remapping after the saccade lead to smaller backward mislocalization of post-saccadic flashes. The mislocalization curves we obtain from our minimal model correspond well with the available data for mislocalization of brief flashes presented in darkness (Honda, 1990; Honda, 1991; Honda, 1993; Honda, 1999) (Figure 3; see also Discussion and Figure A3). 

Until now, we have considered experiments where participants report the perceived location of peri-saccadic flashes using a second saccadic eye movement (Honda, 1989; Klingenhoefer & Krekelberg, 2017). Participants can also report the perceived flash location with respect to the markings on a ruler that is permanently presented or presented after the saccade (Ross et al., 1997) or by making a pointing or reaching movement toward the perceived flash location (Burr, Morrone, & Ross, 2001; Morrone, Ma-Wyatt, & Ross, 2005), say on a touchscreen. We consider the case with the ruler first, where the report is made by comparing the perceived flash location to a marked ruler that is presented well after the initial saccade and reporting the mark with which it overlaps. This situation can be analyzed identically to the one above, where the report is made with a saccadic eye movement, with the only difference that instead of making an eye movement to the perceived flash location, the participant compares the readout of the population profile peak with the markings on the ruler (which is not affected by remapping or mislocalization since it is presented well after the saccade). A biphasic pattern of spatial mislocalization is therefore predicted here as well. When the ruler is actually presented well before the saccade and remains on until well after the saccade, again the situation is similar since the participant can use the ruler markings after the saccade when reporting the location of their percept. The only difference here is that for flashes presented well before the saccade, when there is enough time before the saccade to compare the perceived location to the ruler, the participant can also compare the population profile peak to the ruler markings before the saccade (and there will be no mislocalization since there is no remapping at this point), and there is no need to compare the (accurately) remapped perceived location to the ruler after the saccade. 

We finally consider the situation when the report is made by making a pointing or reaching movement toward the perceived flash location. We do note that this has been a less common paradigm to study mislocalization, and at least for convergent mislocalization, the effects have been reported to be much weaker. However, to the extent that the arm movement system uses the same representations as the eye-movement and ruler-based report above, classical forward remapping is expected to similarly lead to the same biphasic pattern of mislocalization. The only additional assumption needed here is that when the population profile peak is read out after the saccade, the system needs explicit or implicit access to the gaze center (head and eye orientation) so that an arm movement can be appropriately programmed and executed to the perceived flash location, which is in a gaze-centered frame of reference (Byrne, Cappadocia, & Crawford, 2010; Crawford, Henriques, & Medendorp, 2011; Fiehler, Schütz, & Henriques, 2011; Henriques, Klier, Smith, Lowy, & Crawford, 1998). 

Discussion

We showed in a preprint in 2023 how classical remapping, combined with persistent flash-evoked activity and decoding of the flash position from the post-saccadic population representation, produces a biphasic mislocalization pattern (Berreby & Krishna, 2023). To our knowledge, this was the first time a mechanism connecting classical remapping to biphasic mislocalization had been described. While at least two existing models (Binda et al., 2009; Hamker et al., 2011; Ziesche & Hamker, 2011) do include both RF remapping and mislocalization in the phenomena they capture, they also have a variety of other features obscuring the connection between the RF remapping and the mislocalization. Here, we implement the mechanism that we had previously described in an explicit model, consistent with the essential properties of RF remapping. We show that biphasic mislocalization results from insufficient remapping for flashes right before the saccade, and residual/inappropriate remapping for flashes right after the saccade. Insufficient remapping occurs for responses to flashes right before the saccade, because they appear after the remapping process has begun. Inappropriate remapping occurs for responses to flashes appearing immediately after the saccade, because they appear when the remapping process is being terminated and the RFs are going back to their fixation positions. 

Because the appropriate neural data on a mislocalization task are not available—as we expand on later in this Discussion—we use a simple yet plausible model to generate neural responses that are consistent with available physiological knowledge, in order to illustrate how a decoder can then generate the biphasic mislocalization pattern. We note at the outset that our goal here is neither to provide a comprehensive account of all the response patterns that have been reported with respect to peri-saccadic remapping nor to propose a neurocomputational model that will reveal or describe underlying circuit dynamics; we instead aim to show how a plausible decoding rule can operate on forward-remapped neuronal responses to produce biphasic mislocalization. The shift mechanism that we use for implementing remapping is similar to that in the model of Wang et al. (2024b), which was itself originally described in a seminal paper by Zhang (1996) modeling the head-direction system. This paper by Zhang also has a detailed discussion of possible physiological implementations for generating a drift via asymmetric weights. The pattern of weights that determines the drift properties in turn depends on the direction; given our goals described above, we do not attempt to explain how this pattern of weights is created dynamically for each saccade direction. 

Model assumptions and mechanisms

We use a shifting remapping mechanism within a single network to demonstrate how forward remapping predicts biphasic mislocalization. We note that our model only captures the underlying physiology in a functional sense. For example, when the spatial location of the flash falls in different visual hemifields before and after the saccade, the neuron whose RF overlaps the flash location pre-saccadically and the one whose RF overlaps the flash location post-saccadically may be in different sides of the brain (Berman, Heiser, Saunders, & Colby, 2005; Colby, Berman, Heiser, & Saunders, 2005; Dunn & Colby, 2010; Merriam, Genovese, & Colby, 2003). Although our abstract model implements all key mechanisms (including flash response persistence and remapping) within a single network/brain area, it is possible that these mechanisms are distributed across different brain areas. Thus, it may be that while full remapping for flashes well before the saccade is visible in one area, the response persistence for flashes well before the saccade is actually found in another area, and is transmitted to the remapping mechanism once it starts. Further, mechanisms other than shifting remapping may be responsible for forward remapping, as we detail in the next paragraph. 

Our model makes several other assumptions regarding the details of the classical forward RF remapping. We represented remapping as the gradual drift, or shift (Wang et al., 2016), of population responses, and we assumed that decoding is based on the center of mass of the post-saccadic population response—which, in the scenario under consideration, also corresponds to the peak of the population response. Should we consider jumping (Sommer & Wurtz, 2006)—rather than shifting—RFs, our conclusions would remain unaffected; in such a case, with two transiently co-occurring bumps of activity, the population barycenter would drift similarly to what we observe in our model, as one activity bump gradually increases in amplitude and the other decays. It currently remains unclear why existing studies have come to different conclusions about whether forward remapping is accomplished by shifting or jumping RFs; for our purposes, both these modes of forward remapping will lead to biphasic mislocalization via similar principles. We discuss the equivalence of barycenter trajectories for jumping and drifting RFs with a brief mathematical treatment in the Appendix. Other mechanisms for remapping may also be involved. For example, Goldberg and Bruce (1990) proposed a role for FEF post-saccadic neurons; such neurons are also found in LIP (Zhou, Liu, Lu, Wu, & Zhang, 2016). 

Independent of the mechanism used for remapping, we show here how the backward shift of the population response profile (opposite to the saccade direction) can lead to a biphasic mislocalization pattern for peri-saccadic flashes. The reason for this is clear: mislocalization is actually a result of incomplete or residual remapping, that is, the response profiles are insufficiently remapped for flashes immediately before the saccade (appearing after the remapping process has begun) or are inappropriately remapped for flashes appearing immediately after the saccade (by engaging a residual remapping process, while the remapping process is being terminated and the RFs are going back to their fixation positions). A flash-veridical remapping, in that it produces no mislocalization of flashes at all, would fully remap all pre-saccadic stimuli (including flashes) that need to be processed post-saccadically (Cavanagh et al., 2010; Yao, Ketkar, Treue, & Krishna, 2016; Yao, Treue, & Krishna, 2018) and have remapping decay abruptly after the saccade so that it did not apply to the processing of post-saccadic flashes. It is plausible that such abrupt dynamics are not easy to implement using neural circuits. Alternatively, accurately localizing peri-saccadic flashes may not be an important optimizing principle for the brain. 

Our model uses the property that the magnitude of remapping of the peri-saccadic task-relevant flash (that has to be localized) decreases as the flash onset time goes from well before the saccade to well after the saccade. This is not in conflict with existing reports that classical forward RF remapping is maximal for flashes presented right before the saccade, rather than being maximal for flashes presented well before the saccade (Kusunoki & Goldberg, 2003; Wang et al., 2016). This is because these results were obtained using the responses to task-irrelevant, distracting flashes, and these responses are unlikely to generalize to task-relevant flashes presented during a localization task. Areas like FEF, SC, and LIP function as priority maps (Bisley & Goldberg, 2010; Goldberg et al., 2006; Gottlieb, Kusunoki, & Goldberg, 1998) and neurons in these areas respond much more strongly and in a prolonged fashion to task-relevant stimuli, especially when information about them has to be retained for future use (Barash, Bracewell, Fogassi, Gnadt, & Andersen, 1991a; Barash, Bracewell, Fogassi, Gnadt, & Andersen, 1991b; Bisley, 2011; Bisley & Goldberg, 2006; Bisley & Goldberg, 2003). 

We note specifically here that, for a flash well before the saccade, it is not the transient response that is remapped across the saccade. Instead, in our model, a task-relevant flash appearing in a neuron’s RF well before the saccade elicits a sustained persistent response from that neuron, and around the time of the saccade, when the CD becomes active and remapping begins, this sustained response is remapped to the neuron whose RF will overlap the flash location after the saccade. The key requirement in our model is that the response to task-relevant flashes well before the saccade is fully remapped, while the response to these flashes right before the saccade is not—this requirement emerges because in our model, the decoding is done only post-saccadically and in a simple manner. The requirement is satisfied in our model through persistence of the activity evoked by the flash well before the saccade, that is then remapped. Without this property, to explain why flashes well before the saccade are not mislocalized, one would need to invoke another mechanism; for example, a pre-saccadic decoding mechanism that decodes the spatial location of these flashes then executes a movement toward them after the saccade. 

As there is no direct empirical neural data from monkeys performing a task where they have to localize briefly presented flashes, these aspects of the model await direct experimental confirmation. However, in a double-step task with flashed stimuli, spatially-appropriate forward-remapped responses to future saccade targets briefly flashed (for 25–50 ms) outside the RF have been elicited from the FEF by Goldberg and Bruce (1990); a similar response was shown earlier in SC quasi-visual cells with flashes that disappear well before the saccade by Mays and Sparks (1980a). Sustained delay-period activity emerges in both FEF and LIP neurons when a monkey is performing a delayed memory-guided saccade in response to a brief 25–50 ms flash marking the location the monkey should saccade to (Bruce & Goldberg, 1985; Falkner, Goldberg, & Krishna, 2013; Falkner, Krishna, & Goldberg, 2010; Zhang, Falkner, Krishna, Goldberg, & Miller, 2017). Moreover, FEF neurons can show delay-period activity even without a visual stimulus ever being present in their RF (learned-saccade task). Our minimal model will require additional mechanisms in order to model the response to task-irrelevant flashes, since we have not incorporated a mechanism that changes the model parameters or adjusts the response persistence based on whether the flash is task-relevant. Additionally, the time course of the CD that we use should be considered as the effective time course of the CD when functioning within the remapping circuit. The timing of its onset is consistent with the one known source of CD in the SC (via the mediodorsal nucleus of the thalamus; Rao, Mayo, & Sommer, 2016; Sommer & Wurtz, 2008; Subramanian, Alers, & Sommer, 2019); other potential sources include the ipsilateral and contralateral FEF. 

Another assumption in our model is that the population response profile is read out after the initial saccade, when the participant makes their behavioral report. This assumption is generally critical for the model, although not in every single case we have considered above (e.g., for flashes well before the saccade). We note that, when flashes appear after, say, 150 ms before the saccade, the participant is in a dual-task situation, having to register the flash and make the impending saccade, after which they have to report where they perceived the flash. Postponing the read-out until after the saccade, and just before the behavioral report is made, is therefore plausible behaviorally. In physiological data, predictive responses from the post-saccadic RF are rarely available before the saccade in population responses to flashes (Sommer & Wurtz, 2008; Umeno & Goldberg, 1997; Umeno & Goldberg, 2001; Wang et al., 2016), even for task-relevant flashes (Goldberg & Bruce, 1990)—while neurons do respond to pre-saccadic flashes appearing at the spatial location of their post-saccadic RFs, the responses themselves only begin after the saccade has begun. 

The delayed time course of forward-remapped responses is in fact the key element of a proposal by Burr and Morrone (2011) that LIP RFs are transiently spatiotopic around a saccade. Our model uses post-saccadic decoding after the representation has stabilized to determine the value of the perceptual report (by means of a saccade, or a pointer, or a comparison with a scale, etc.). We used a value of 300 ms because the barycenter has stabilized by this time (Figure 2) and this time is consistent with when reports are generally made in such tasks. The time at which the barycenter has stabilized is set by when the temporal CD η has decayed; if η decays back to zero faster after the saccade, the barycenter will also stabilize faster. Using longer values will not affect our results or conclusions significantly. However, decoding before the barycenter has stabilized will lead to an effect (Figure A4) where the forward mislocalization of pre-saccadic flashes is increased (because the response to pre-saccadic flashes is inadequately remapped) while the backward mislocalization of post-saccadic flashes is reduced (because the response to post-saccadic flashes has not had the time to be inappropriately remapped). This is therefore also a psychophysical prediction that emerges from this version of our model. The variation of the barycenter with time after the saccade can be read out from the right column in Figure 2. 

We do not specify a mechanism that determines when or how these processes start to determine when the report is made; these are probably determined by the subject’s understanding of when the underlying representation and resulting decision have stabilized appropriately to allow adequate and consistent task performance. Our treatment is similar to that in standard models where the target of a saccade is first decoded via a population computation, such as a barycenter or weighted vector average (Lee, Rohrer, & Sparks, 1988). To then model reaction time, a process like LATER (Noorani & Carpenter, 2016) can be used. This uses a simple stochastic evidence-independent rise to threshold to model when the movement is made. Reaction times and speed-accuracy trade-off data have not been presented in the context of mislocalization tasks, and we therefore focus on the where of the localization, and leave out the when from our modeling (Findlay & Walker, 1999). 

Comparison with empirical mislocalization data

We show in Figure 3 that the mislocalization curves we obtain from our model correspond well to the biphasic pattern visible in available data for mislocalization of brief flashes presented in darkness in the work of Honda (Honda, 1990; Honda, 1991; Honda, 1993; Honda, 1999). Even so, we do not claim that forward RF remapping is the sole basis for the biphasic forward mislocalization pattern found in darkness in Honda’s work. Similarly, though the basic results obtained here translate to other saccade directions and amplitudes, specific situations with very small and very large saccades may necessitate specific considerations regarding the extent of forward remapping (van Wetter & van Opstal, 2008a). What we do show here is that to the extent that classical forward RF remapping mechanisms impact the same representations that are used for the spatial localization of peri-saccadic flashes, they predict a biphasic mislocalization pattern for peri-saccadic flashes that is quite similar to that shown empirically by Honda. 

The only other work that we know of that has measured mislocalization of brief flashes in darkness is that of van Wetter and van Opstal (van Wetter & van Opstal, 2008a; van Wetter & van Opstal, 2008b), where, for unclear reasons, a biphasic pattern has not been reported and post-saccadic flashes continue to be mislocalized in the forward direction; we take this up further in Figure A3. This work also pointed out an important issue with the other explanations for forward biphasic mislocalization that have been proposed (based on either referencing an explicit temporally sluggish eye-position signal (Dassonville, Schlag, & Schlag-Rey, 1992; Honda, 1991; Schlag & Schlag-Rey, 2002), slow visual latencies (Pola, 2004; Pola, 2007), or some combination of the two (Ross et al., 1997): their simplest implementations predict that mislocalization magnitude should correlate with saccade amplitude, and this was not found in the data of van Wetter and van Opstal. Van Wetter and van Opstal suggested that the mislocalized percepts could be due to shifts in visual perception (visual shift hypothesis) and that remapping may be involved. The model for remapping that we use here also predicts larger mislocalizations with larger saccade amplitudes (Figure A3), but this can be somewhat mitigated if the CD time course is also allowed to change with saccade amplitude. Further, if the trial-by-trial variations in CD are unlinked from the actual saccade kinematics on each trial, the model can in principle capture the lack of correlation between the trial-by-trial residual localization errors and saccade kinematics that van Wetter and van Opstal reported. Additional work is needed with more empirical data to resolve these issues. 

Contextualizing forward remapping and biphasic mislocalization

It is generally considered (Qian, Goldberg, & Zhang, 2022; Zirnsak & Moore, 2014; Zirnsak et al., 2014) that forward remapping is related to forward/backward biphasic mislocalization, with all the action happening along the saccade direction, and another form of remapping, convergent remapping (Hamker, Zirnsak, Calow, & Lappe, 2008; Neupane et al., 2016a; Neupane et al., 2016b; Neupane, Guitton, & Pack, 2020; Zirnsak & Moore, 2014; Zirnsak et al., 2014), is related to convergent mislocalization toward a saccade target (Lappe, Awater, & Krekelberg, 2000; Ross et al., 1997; Ross et al., 2001). It has been proposed that the phenomena of compression toward the saccade target or other available visual references may reflect the large spatial and temporal uncertainty of peri-saccadic perception (Cicchini et al., 2013). Mislocalization is a very complex phenomenon that almost certainly emerges from a combination of different factors, which are likely context- and individual-dependent (Hamker et al., 2011; Klingenhoefer & Krekelberg, 2017). A variety of other factors, which are not mutually exclusive, including an explicit eye-position signal (that may be sluggish) (Dassonville, Schlag, & Schlag-Rey, 1992; Schlag & Schlag-Rey, 2002), visual latencies (Pola, 2004; Pola, 2007; Pola, 2011a), effects related to the flash-lag effect (Hogendoorn, 2020; Hubbard, 2014), and other mislocalization situations under conditions of spatio-temporal uncertainty (Schlag, Cai, Dorfman, Mohempour, & Schlag-Rey, 2000; Shimojo, 2014), visual references (Burr & Morrone, 2012; Chakrabarty, Nakano, & Kitazawa, 2017; Lappe, Awater, & Krekelberg, 2000; Luo, Garaas, & Pomplun, 2014), effects related to scene motion (Honda, 1995), eccentricity effects and cortical magnification factors (Richard et al., 2009) may all play a role. 

Our single-process model here based on forward RF remapping does not predict any compression toward the saccadic target or other references, and additional mechanisms will be needed to incorporate them. However, in our model, forward RF remapping still plays a crucial role in updating the spatial representation of relevant stimuli across the saccade, so that, for example, when making a saccade toward a target 8° to the right, a flash just above the saccade target before the saccade needs to be remapped to be perceived after the saccade as having appeared above the saccade target (rather than 8° to the right of it). Additional mechanisms, related to attention, or convergent remapping, or the role of visual references in localization, etc., may all play a role in their superimposing the convergent mislocalization pattern on this baseline updating (Burr & Morrone, 2012). In this context, we note that while we have emphasized the retinotopic nature of RFs in LIP, FEF, SC, V4, and so on, there are clear and large eye-position effects on their responses (that often take the form of eye-position gain fields (Andersen, Essick, & Siegel, 1985)) whose timing and importance in peri-saccadic phenomenology is being debated (Morris, Bremmer, & Krekelberg, 2013; Morris, Kubischik, Hoffmann, Krekelberg, & Bremmer, 2012; Xu, Karachi, & Goldberg, 2012; Xu, Wang, Peck, & Goldberg, 2011). 

Reconciling with previous interpretations

As we noted in the Introduction, distinguishing between forward and convergent remapping in physiological studies and delineating their relative contributions can be complex. Early studies reporting forward remapping generally did not have the spatial sampling required to distinguish it from convergent remapping (Hamker et al., 2011; Zirnsak, Lappe, & Hamker, 2010; Zirnsak et al., 2014), whereas later studies that have extensively sampled different spatial locations have come to somewhat divergent conclusions about whether forward or convergent remapping better characterizes the observed data from different areas, while agreeing that these conclusions likely depend upon the context and time window being analyzed (Chen, Zirnsak, & Moore, 2018; Hartmann et al., 2017; Neupane et al., 2016a; Neupane, Guitton, & Pack, 2020; Wang et al., 2024b; Zirnsak & Moore, 2014; Zirnsak et al., 2014). We note here that this ongoing debate on the spatio-temporal properties of peri-saccadic remapping is entirely based on using task-irrelevant flashed probes to estimate peri-saccadic responses, and that there is no data available from contexts where the probe’s location is task-relevant. It is however clear from the well-documented psychophysics of double-step saccades (Becker & Jürgens, 1979; Hallett & Lightstone, 1976) and two-saccade sequences with short intersaccadic intervals (Jurewicz, Liao, & Krishna, 2023; McPeek & Keller, 2002; McPeek, Skavenski, & Nakayama, 2000), as well as the ability to accurately maintain attention at a fixed spatial location across a saccade (Cavanagh et al., 2010; Harrison, Stead, Wallis, Bex, & Mattingley, 2024; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011; Yao et al., 2016), that spatially accurate, forward remapping of the responses to a task-relevant stimulus must exist such that it is able to guide spatially accurate saccades and attention shifts (Bays & Husain, 2007; Cavanagh et al., 2010; Duc, Bays, & Husain, 2008). Consistent with this, there are several studies demonstrating that task-relevant, spatially appropriate, forward remapping can be found for saccades (Goldberg & Bruce, 1990; Mays & Sparks, 1980a), as well as attentional priority (Marino & Mazer, 2018; Mirpour & Bisley, 2012; Mirpour & Bisley, 2016; Yao, Treue, & Krishna, 2018). 

In a recent study, Qian, Goldberg, and Zhang (2022) concluded that forward remapping alone would predict either no mislocalization (for decoders that fully compensate for the remapping) or a mislocalization of pre-saccadic flashes opposite to the direction of the saccade (for unaware decoders that do not make any allowance for the remapping)—the latter mislocalization of pre-saccadic flashes opposite to the saccade direction would be the inverse of what is observed empirically. A similar conclusion, following a brief but related line of argument, was reached previously by Klingenhoefer and Krekelberg (2017). This conclusion would be remarkable, if true, since it would contravene a long-standing assumption in the field. Here, and in our 2023 preprint (Berreby & Krishna, 2023), we show that this conclusion is, however, not true. When decoded appropriately, classical remapping actually does predict biphasic mislocalization of brief flashes similar to that seen in empirical data (Honda, 1989; Honda, 1991; Honda, 1993; Honda, 1995; Honda, 1999). The difference of our conclusion from that of these two previous studies comes from the fact that it is critical to decode (as we do) the population response profile after the saccade when (and on the basis of which) the participant makes their behavioral report. Qian, Goldberg, and Zhang do not specify the exact mechanism that leads from this population response to the flash location reported by the observer. However, it can be inferred from their description that for pre-saccadic flashes, Qian, Goldberg and Zhang use a decoder of the population response (in, say, LIP, FEF, or SC) that compensates for the shift in current eye position between the time when the flash was presented, and when the behavioral report is made. This compensation cannot come from remapping in their scheme, because remapping has already been invoked to produce the backward shift of the population response, which in most models is primarily in retinotopic coordinates (although evidence for hybrid/mixed reference frames in LIP and FEF has been presented (Caruso, Pages, Sommer, & Groh, 2018). Instead, in their scheme, at least for saccade-based behavioral reports of flash location, some unspecified mechanism allows the visuomotor system to take the intervening shift in eye position between the flash onset and the behavioral report into account, and derive a gaze vector that takes the eye to the same location it would have gone to (based on the population response) without the intervening shift in eye position. This is why in their model, a backward shift of the population response (owing to remapping) leads to backward mislocalization, even though the eye has moved forward in the time between the flash onset and the saccadic report. It is unclear what the physiological correlate of this unspecified mechanism is supposed to be. This discussion may however be moot: the same group has recently put forth, in a preprint, a decoding scheme that overlaps with what we show here (Wang, Tsien, Goldberg, Zhang, & Qian, 2024a). There is, therefore, no longer a difference in viewpoint between our two groups, as they now appear to agree with us that classical forward remapping can indeed explain peri-saccadic biphasic mislocalization, when combined with an appropriate post-saccadic decoder that operates on persistent flash-evoked activity shaped by a classical forward remapping process. We also note here that we do agree entirely with Qian, Goldberg, and Zhang that when decoders are aware of RF remapping and compensate for it, no mislocalization is predicted. 

Our model is consistent with the proposal of Colby, Duhamel, and Goldberg (1995) where LIP activity (and by extension, activity in FEF, SC, V4, etc.) represents space primarily using an oculocentric coordinate frame. In this model, forward remapping, presumably based on a CD, updates the neural representation of the visual scene around every saccade (Yao, Treue, & Krishna, 2018), and the peri-saccadic mislocalization of brief flashes is an artifact of measuring the effects of dynamic peri-saccadic representations using a specialized probe. An eye-position signal is only needed for reports made with the arm (pointing/reach, where incidentally, empirical studies usually report relatively little mislocalization (Burr et al., 2001; Morrone et al., 2005)); the arm-movement system just needs to explicitly or implicitly use knowledge of the eye position well after the saccade in order to perform the visuomotor transformation required to point toward the visually experienced flash location (Crawford, Henriques, & Medendorp, 2011). Reports made via a saccade or using a ruler do not need an eye-position signal. 

Conclusions

Via a simple illustrative model, consistent with the physiology of RF remapping and trans-saccadic information transfer (Goldberg & Bruce, 1990; Ipata, Bisley, & Krishna, 2023; Mays & Sparks, 1980b) and equipped with appropriate post-saccadic decoding, we show that forward RF remapping in priority-map brain areas produces biphasic peri-saccadic mislocalization that is consistent with behavioral data. The biphasic mislocalization results from insufficient remapping before the saccade, and residual/inappropriate remapping after the saccade. This validates a long-standing assumption in the field that forward RF remapping can account for peri-saccadic biphasic mislocalization. Much additional work, both physiology aimed at clarifying the how, when, where and how much of RF remapping as well as psychophysics and computational modeling clarifying the exact phenomenology of peri-saccadic mislocalization, is needed before the relative importance of various mechanisms in the phenomenology of trans-saccadic processing can be elucidated. 

Acknowledgments

Funded by a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Research program (RGPIN-2022-05399) and Supplement (DGECR-2022-00321), and a computing resources grant from Calcul Quebec and the Digital Research Alliance of Canada to BSK, as well as a Vision Sciences Research Network (VSRN) MSc Recruitment Scholarship, a VSRN PhD Recruitment Scholarship, a Unifying Neuroscience and Artificial Intelligence - Québec (UNIQUE) MSc Excellence Fellowship, a Centre for Applied Mathematics in Bioscience and Medicine (CAMBAM) Fellowship and a McGill Department of Physiology Max and Jane Childress Entrance Fellowship to Y.E.B. The funders of this research played no role in any aspect of the research, decision to publish, or manuscript preparation. 

Commercial relationships: none. 

Corresponding authors: B. Suresh Krishna; Yohaï-Eliel Berreby. 

Emails: [email protected]; [email protected]. 

Address: Room 1229, McIntyre Medical Building, 3655 Promenade Sir-William-Osler, Montreal, QC H3G 1Y6, Canada. 

Appendix: Method details

Feedforward input

The normalized feedforward input I₀(x, t) received at time t by the neuron coding for position x is defined as the outer product of a temporal envelope (I_time) and a spatial envelope (I_space):  

\begin{equation} I_0(x,t) \equiv (I_{\rm{space}}\otimes I_{\rm{time}})(x,t) = I_{\rm{space}}(x) I_{\rm{time}}(t). \end{equation}

(1)

 

The temporal envelope I_time is defined by:  

\begin{eqnarray*} && I_{\rm{time}}(t_{\rm{LIP}})\\ && \equiv \left\lbrace \begin{array}{@{}l@{}}0, \quad \text{if } t_{\rm{LIP}}\lt 0, \\ \displaystyle \frac{t_{\rm{LIP}}}{\tau _{\rm{FF}}^{\rm{on}}}\, \frac{e^{-\frac{t_{\rm{LIP}}}{\tau _{\rm{FF}}^{\rm{on}}}}}{e^{-1}}, \quad \text{if } 0 \le t_{\rm{LIP}}\le \tau _{\rm{FF}}^{\rm{on}}, \\ \displaystyle (1-b_{\rm{FF}})\, e^{-\frac{(t_{\rm{LIP}}-\tau _{\rm{FF}}^{\rm{on}})^2}{2\,{\tau _{\rm{FF}}^{b}}^2}} + b_{\rm{FF}}, \\ \;\;\, \text{if } \tau _{\rm{FF}}^{\rm{on}}\lt t_{\rm{LIP}}\le \tau _{\rm{FF}}^{\rm{on}}+ \tau _{\rm{FF}}^{d}, \\ \displaystyle \left[(1-b_{\rm{FF}})\, e^{-\frac{{\tau _{\rm{FF}}^{d}}^2}{2\,{\tau _{\rm{FF}}^{b}}^2}} + b_{\rm{FF}}\right] \,\, e^{-\frac{t_{\rm{LIP}}- \left(\tau _{\rm{FF}}^{\rm{on}}+\tau _{\rm{FF}}^{d}\right)}{\tau _{\rm{FF}}^{\text{off}} }},\\ \;\;\, \quad \text{if } t_{\rm{LIP}}\gt \tau _{\rm{FF}}^{\rm{on}}+ \tau _{\rm{FF}}^{d}. \end{array}\right.. \end{eqnarray*}

 

It rises to its peak over a duration of \(\tau _{\rm{FF}}^{\rm{on}}\) upon stimulus onset, stabilizes to a baseline level b_FF with time constant \(\tau _{\rm{FF}}^{b}\) as the stimulus remains on, and ultimately decays back to 0 with time constant \(\tau _{\rm{FF}}^{\text{off}}\) after stimulus offset. The length of the stimulus is represented by \(\tau _{\rm{FF}}^{d}\). 

For short flashes such as those we consider, there is no time for stabilization to a baseline level to occur. Nevertheless, we chose this input formulation to make the model easy to extend in the future when considering longer-lasting stimuli. 

We model the propagation delay of signals from the retina to the model population as Δ_FF, and, for a stimulus onset on screen of \(t_{\rm{on}}^{\rm{scr}}\), we set \(t_{\rm{LIP}}= t - t_{\rm{on}}^{\rm{scr}}- \Delta _{\rm{FF}}\). Thus, I_time begins increasing Δ_FF after stimulus onset on screen. 

The spatial envelope I_space(x), with x given in oculocentric coordinates, is defined as  

\begin{equation} I_{\rm{space}}(x) \equiv e^{-\frac{(x-x_0(t_{\rm{on}}^{\rm{scr}}))^2}{2\sigma _{\rm{FF}}^2}} \end{equation}

(2)

with \(x_0(t_{\rm{on}}^{\rm{scr}})\) being the position on the retina of the image of a flash whose onset is \(t_{\rm{on}}^{\rm{scr}}\), and σ_FF controlling the spatial spread of the feedforward input. If the screen position fixated at time t is \(x_{\rm{eye}}^{\rm{scr}}(t)\) (in screen coordinates), then \(x_0(t_{\rm{on}}^{\rm{scr}}) = -x_{\rm{eye}}^{\rm{scr}}(t_{\rm{on}}^{\rm{scr}})\) (in retinal coordinates). 

Temporal CD signal

The unnormalized temporal component of the CD signal is the product of a rising and a falling sigmoid,  

\begin{equation} \eta _{\rm{un}}(t) \equiv \sigma \left(\frac{t - T_{\eta }^{\rm{start}}}{\tau _{\eta }^{r}}\right) \sigma \left(-\frac{t - T_{\eta }^{\rm{end}}}{\tau _{\eta }^{f}}\right) \end{equation}

(3)

with \(T_{\eta }^{\rm{end}}- T_{\eta }^{\rm{start}}= \Delta T_{\eta }\), and \(\frac{T_{\eta }^{\rm{start}}+ T_{\eta }^{\rm{end}}}{2} = T_{\eta }^{c}\). 

To freely allow for changes in the shape of the temporal CD signal without requiring re-tuning of other parameters to obtain accurate remapping of early flashes, we normalize this signal such that its integral is always 1:  

\begin{equation} \eta (t) \equiv \frac{\eta _{\rm{un}}(t)}{\int _{-\infty }^{+\infty } \eta _{\rm{un}}(u) \, du}. \end{equation}

(4)

 

Spatial CD signal

Given a stable bump of activity in the population, with no further feedforward input, and a sufficiently-dense population of model neurons, the product βη(t) is proportional to the instantaneous speed at which this response’s barycenter drifts (which is 0 in the absence of remapping). Thus, the distance travelled by a bump that is exposed to the entirety of η’s non-negligible support is proportional to the integral of βη, which is simply β, as β is constant over time and η is chosen to have an integral of 1. 

In the limit of a dense population of model neurons, there is thus a linear relationship between β and the remapping amplitude (Figure A1). 

The saccade amplitude A_sac, and thus the desired remapping amplitude, is communicated to the model through the parameter β = A_sac · β₀, which constitutes the spatial component of the CD signal. 

With the other model parameters set as described in other sections, β₀ = 20.0 deg⁻¹ allows for a practically exact correspondence between the saccade amplitude and the remapping amplitude of early flashes. 

Lateral connectivity

The time-dependent connectivity kernel W is defined as W(x, t) ≡ W₁(x) + η(t)W₂(x), with:  

\begin{equation} W_1(x) = \frac{w_{\rm{exc}}\cdot e^{-\frac{x^2}{2 \sigma _{\rm{exc}}^2}} - w_{\rm{inh}}\cdot e^{-\frac{x^2}{2 \sigma _{\rm{inh}}^2}}}{w_{\rm{exc}}- w_{\rm{inh}}} \end{equation}

(5)

 

\begin{equation} W_2 = \beta \frac{dW_1}{dx}. \end{equation}

(6)

 

W ₁ (Figure 1F) mediates persistence. It is a Mexican-hat interaction kernel, which is excitatory at small offsets and inhibitory at larger ones, normalized such that its peak has a value of 1. 

W ₂ (Figure 1F) mediates drifting. In a two-dimensional setting, drifting in an arbitrary direction can be obtained by taking the appropriate directional derivative instead of the derivative with respect to the positive x coordinate, or, equivalently, applying a rotation to the template W₂ obtained with the above formula. 

Figure A1.

 

Relationship between β and remapping amplitude for an early flash (at t = −300 ms before saccade onset) and a model population size of N = 101, 201, or 1001 neurons. As the model population size N is increased, it becomes possible to code for positions with greater precision, and the relationship between β and remapping amplitude becomes more linear. In this figure, only N and β were varied, with all other parameters set to the values described in Table A1.

Membrane potentials and firing rates

The evolution of the population’s membrane potentials, u, is governed by the differential equation:  

\begin{eqnarray} \tau _{m}\frac{\partial u}{\partial t}(x,t) \equiv -u(x,t) + h+ (\alpha W * r)(x,t) + I(x,t)\nonumber\\ \end{eqnarray}

(7)

with I(x, t) ≡ κ · I₀(x, t), and \(\alpha \equiv \frac{\alpha _0}{N}\). 

The time constant τ_m sets the timescale of relaxation in the absence of input or persistence. 

The deviation of the firing rate from baseline is a pure function of the membrane potential:  

\begin{equation} r(x,t) \equiv \sigma \left(\frac{u(x,t) - u_0}{\mu }\right) \end{equation}

(8)

with \(\sigma (x) \equiv \frac{1}{1+e^{-x}}\) being the standard logistic function. 

The parameter α₀ controls the overall strength of the interaction between neurons in the population. All else being equal, sufficiently low values of α₀ will fail to produce persistent activity. Beyond this threshold, increasing α₀ has relatively little effect on the model’s behavior, up until a second threshold, beyond which the model will display additional activity bumps. Any value of α₀ between these two thresholds will produce a single, stable bump of activity in response to a single brief flash. 

Analytical approximation of the mislocalization pattern

When considering brief flashes, as we do here, the mislocalization pattern produced by our model can be analytically approximated without simulating the full neural population dynamics. 

This approximation relies on three key insights: 

• (1) The drift speed of a stable bump of activity is proportional to βη(t),
• (2) The total distance traveled by such a bump depends on the integral of η(t) over the time window during which the bump exists, and
• (3) The perceived position of a flash can be computed by combining this remapping distance with the flash’s retinal position and the final eye position.

To formalize this, we consider a flash occurring on the screen at time \(t_{\rm{on}}^{\rm{scr}}\). This flash will first elicit a response in our model population at time \(t_{\rm{on}}^{\rm{pop}}= t_{\rm{on}}^{\rm{scr}}+ t_{\rm{crit}}= t_{\rm{on}}^{\rm{scr}}+ \Delta _{\rm{FF}}+ \tau _{\rm{FF}}^{\rm{on}}\). This response will then be decoded at some time t_dec after the saccade. 

Let \(R(t_{\rm{on}}^{\rm{pop}})\) be the proportion of total remapping effectively applied to a response first appearing in the population at time \(t_{\rm{on}}^{\rm{pop}}\), before it is decoded at t_dec. This can be expressed as the ratio of two distances:  

\begin{equation} R(t_{\rm{on}}^{\rm{pop}}) \equiv \frac{d(t_{\rm{on}}^{\rm{pop}}, t_{\rm{dec}})}{d(-\infty , t_{\rm{dec}})}, \end{equation}

(9)

where d(t₁, t₂) represents the distance traveled by a bump drifting from time t₁ to time t₂. 

Because the drift speed is proportional to η(t), and η is normalized to have an integral of 1, we have:  

\begin{equation} R(t_{\rm{on}}^{\rm{pop}}) = \int _{t_{\rm{on}}^{\rm{pop}}}^{t_{\rm{dec}}} \eta (t) \, \mathrm{d}t \end{equation}

(10)

 

\begin{equation} \hphantom{R(t_{\rm{on}}^{\rm{pop}})} = \int _{t_{\rm{on}}^{\rm{scr}}+ t_{\rm{crit}}}^{t_{\rm{dec}}} \eta (t) \, \mathrm{d}t. \end{equation}

(11)

 

The effective distance traveled by the response is then \(-A_{\rm{sac}}\cdot R(t_{\rm{on}}^{\rm{pop}})\), where A_sac is the saccade amplitude. Combined with the retinal position of the flash (\(-x_{\rm{eye}}^{\rm{scr}}(t_{\rm{on}}^{\rm{scr}})\), where \(x_{\rm{eye}}^{\rm{scr}}(t)\) is the eye position on screen at time t) and the final eye position (A_sac), this yields the perceived position:  

\begin{equation} x_{\rm{perc}}^{\rm{scr}}= -x_{\rm{eye}}^{\rm{scr}}(t_{\rm{on}}^{\rm{scr}}) - A_{\rm{sac}}\cdot R(t_{\rm{on}}^{\rm{pop}}) + A_{\rm{sac}}. \end{equation}

(12)

 

The mislocalization error is then simply \(x_{\rm{perc}}^{\rm{scr}}- x_{\rm{true}}^{\rm{scr}}= x_{\rm{perc}}^{\rm{scr}}\), where \(x_{\rm{true}}^{\rm{scr}}\) is the true flash position on screen, which is 0 here. 

This analytical approximation captures the essential features of the mislocalization pattern produced by the full model (Figure A2). It provides a direct link between the profile of the temporal CD signal η(t) and the resulting pattern of mislocalization. 

Saccade duration scaling

In Figure A3, we show the mislocalization patterns produced by the model when varying saccade amplitude and duration. For this figure, we derive the saccade duration T_sac from the saccade amplitude A_sac using the relationship reported in figure 3C of Guadron, van Opstal, and Goossens (2022). From this data, we read the pairs (5°, 30 ms) and (25°, 60 ms), and set by linear interpolation:  

\begin{eqnarray*} \theta (A_{\rm{sac}}) &\;=& \frac{A_{\rm{sac}}- 5^{\circ }}{25^{\circ } - 5^{\circ }} \\ T_{\rm{sac}}(A_{\rm{sac}}) &\;=& (1 - \theta (A_{\rm{sac}})) \times 30\,\mathrm{m}\mathrm{s} + \theta (A_{\rm{sac}}) \times 60\,\mathrm{m}\mathrm{s}. \end{eqnarray*}

 

Comparison with experimental data

To compare the mislocalization curve predicted by our model with experimental data, we used WebPlotDigitizer (Marin, Rohatgi, & Charlot, 2017) to extract the data listed in Table A2. In each of the experiments considered, a flash of duration 2 ms was presented, and the saccade amplitude was 8°. The reported perceptual errors that we digitized had been binned and averaged by the authors in all figures, except figure 3 from Honda (1991). In order to enable more meaningful comparison of this figure’s data with the rest of the data, we post-processed the digitized data points for each target position by averaging them within 20 ms bins from −200 ms to 200 ms. To compute the underlying trend shown in Figure 3, we pooled data points from each digitized curve (and binned values, in the case of Honda (1991) figure 3), and used locally-weighted scatterplot smoothing, as implemented in the Python package statsmodels (Seabold & Perktold, 2010), with a fraction of 0.15 and 3 iterations. The 95% confidence intervals were computed using 1,000 bootstrap resamples. For each bootstrap iteration, we resampled the data points with replacement and performed locally-weighted scatterplot smoothing with the parameters above, then took the 2.5th and 97.5th percentiles across bootstrap samples to form the confidence interval. 

Figure A2.

 

Effect of η and t_crit on the mislocalization patterns produced by the model. Each row represents a distinct set of η parameters; all other parameters are set to their values in Table A1. First column (A, D, G): η shapes chosen to peak around saccade onset (t = 0), while having their area distributed roughly equally around t_crit (A), mostly before t_crit (D), or mostly after t_crit (G). t_crit is denoted by the blue dashed line. The decoding time t_dec is indicated by the purple dashed line. Second column (B, E, H): Integral of η from population input peak time to decoding time t_dec (black curve, left y axis) as a function of flash onset time on screen (x axis), and mislocalization pattern predicted using this integral by analytical approximation (purple curve, right y axis). Third column (C, F, I): Mislocalization patterns predicted by the full model (black curves) or the analytical approximation (purple curves) for each η shape. Depending on how the area under the curve of η is distributed on either side of the critical time t_crit, the resulting mislocalization pattern will be either balanced (A–C), positively-skewed (D–F), or negatively-skewed (G–I), in a manner that is largely consistent between the full model and the analytical approximation.

Figure A3.

 

Effect of saccade amplitude on the mislocalization pattern produced by the model. (A) Mislocalization increases with saccade amplitude when only saccade amplitude (A_sac) and, correspondingly, saccade duration (T_sac, see Saccade duration scaling), are varied, with all other parameters kept fixed to their values in Table A1. The peak forward mislocalization for a 35° saccade is around 13.3°, similar to the mislocalization reported by van Wetter and van Opstal (2008a) for 35° saccades. (B) When the parameters of the temporal corollary discharge signal η are additionally allowed to be varied, this effect can be partially compensated, with the model displaying a more modest increase in maximum mislocalization with increasing saccade amplitude. (C) Colored curves: η curves used to produce B, color-matched to the corresponding saccade amplitude. The blue curve corresponds to the standard η parameter set, as presented in Fig. 1C. (D) Maximum forward mislocalization amplitude as a function of saccade amplitude. Black dashed line: data from A (standard η parameters). Red dashed line: data from B (modified η parameters). We note that the thought-provoking study of van Wetter and van Opstal (2008a) came with many limitations: a small number of subjects (4, with only 3 naive authors), a high degree of inter-subject variability (e.g. their Figure 9B shows two subjects with a strong bias for stimuli well before the saccade, and their Figure 10B shows the reported general pattern of a saturating mislocalization only for two of the four subjects), a blocked design where each first saccade amplitude was tested separately in a block (with block order unknown), and a restricted range of second saccade target locations so that subjects could have used a strategy where their reported percept was biased towards an a priori expected location once the rough target location (e.g. upper or lower hemifield) was determined. As mentioned in the main text, this study also did not find biphasic mislocalization; this is why we plot the peak forward mislocalization from the model in D. The reason for the lack of biphasic mislocalization in this data is not known. A follow-up study with 6 subjects (3 naive) performing head-free gaze shifts (van Wetter and van Opstal, 2008b) of around 40° to 60° showed a similar peak mislocalization amplitude of around 10°. Additional work is needed to clarify these issues.

Figure A4.

 

Influence of decoding time on the mislocalization patterns predicted by the model. Flash mislocalization curves are shown for t_dec = 150, 200, or 350 ms. Curves are truncated when the barycenter of the population response at t_dec becomes ill-defined due to insufficient activity, i.e. when a flash is presented so late that the corresponding feedforward population input is negligible at t_dec. When decoding occurs before remapping is complete, pre-saccadic flashes are mislocalized forward, even when presented abundantly early. Conversely, the backward mislocalization of post-saccadic flashes presented close to saccade onset is reduced when decoding occurs earlier, as decoding happens before their representations had time to be inappropriately remapped to a significant extent.

Table A1.

 

Model parameters. Unless otherwise specified, the values are those used to produce figures.

 

Table A2.

 

Digitized experimental data.

 

Jump-drift equivalence for barycenter trajectories

The main conclusion drawn from our model is that forward remapping can produce a flash-onset-dependent mislocalization curve similar to those observed in psychophysical experiments, as shown in Figure 3. We mentioned in the Discussion that this conclusion is robust to the consideration of jumping RFs, as opposed to the drifting RFs that we explicitly consider. Here, we explain why this must be true. 

The predicted perceptual localization error for a given flash onset time depends on the final location of the population activity’s barycenter at decoding time. In the case of drifting RFs, at any given time, the population barycenter corresponds to the location of the population response peak. With jumping RFs, there are two co-occurring bumps of activity in the response, one at the retinotopic location where the response to the flash was first elicited, and one at the remapping target location (post-saccadic retinal location corresponding with the flash’s position on screen). 

The key insight is that for any two symmetric bumps of activity at locations a and b, any desired barycenter \(\hat{C}\) in the interval (a, b) can be produced by appropriately adjusting the relative heights of the two bumps, in a smooth manner with respect to \(\hat{C}\). Furthermore, this can be achieved without pathological behavior such as extremely high activities, which would be unphysiological. 

We formalize as follows: 

Let \(f : \mathbb {R} \rightarrow \mathbb {R}^+\) be a symmetric, positive, integrable function (such as a Gaussian) with mass \(M = \int _{-\infty }^{+\infty } f(x) \, \mathrm{d}x\). 

Let \(a,b \in \mathbb {R}\) such that a < b. We consider two f-shaped bumps, centered at a (resp. b) and of amplitude A₁ > 0 (resp. A₂ > 0). 

Together, they form the response profile g:  

\begin{equation} g(x) = A_1 f(x - a) + A_2 f(x - b). \end{equation}

(13)

Its total mass is \(\int _{-\infty }^{+\infty } g(x) \, \mathrm{d}x = (A_1 + A_2)M\). 

Given a target location \(\hat{C} \in (a,b)\), can we find amplitudes A₁ and A₂ such that the barycenter of g is \(\hat{C}\)? 

The barycenter C of g is:  

\begin{eqnarray*} C &\;=& \frac{\int _{-\infty }^{+\infty } x g(x) \, \mathrm{d}x}{\int _{-\infty }^{+\infty } g(x) \, \mathrm{d}x} \\ &\;=& \frac{A_1 \int _{-\infty }^{+\infty } x f(x - a) \, \mathrm{d}x + A_2 \int _{-\infty }^{+\infty } x f(x - b) \, \mathrm{d}x}{(A_1 + A_2)M}. \end{eqnarray*}

 

Applying the change of variable x′ = x − a:  

\begin{eqnarray*} &&\int _{-\infty }^{+\infty } x f(x - a) \, \mathrm{d}x = \int _{-\infty }^{+\infty } (x^{\prime } + a) f(x^{\prime }) \, \mathrm{d}x^{\prime } \\ &&\qquad = \int _{-\infty }^{+\infty } x^{\prime } f(x^{\prime }) \, \mathrm{d}x^{\prime } + a \underbrace{\int _{-\infty }^{+\infty } f(x^{\prime }) \, \mathrm{d}x^{\prime }}_{M}. \end{eqnarray*}

 

Since f is symmetric, \(\int _{-\infty }^{+\infty } x^{\prime } f(x^{\prime }) \, \mathrm{d}x^{\prime } = 0\), thus \(\int _{-\infty }^{+\infty } x f(x - a) \, \mathrm{d}x = a M\). Likewise, \(\int _{-\infty }^{+\infty } x f(x - b) \, \mathrm{d}x = b M\). 

We can express the barycenter C in terms of the amplitudes and positions of the activity bumps:  

\begin{equation} C = \frac{A_1 a M + A_2 b M}{(A_1 + A_2)M} \end{equation}

(14)

 

\begin{equation} \hphantom{C} = \frac{A_1 a + A_2 b}{A_1 + A_2}. \end{equation}

(15)

 

Solving for \(C = \hat{C}\), we find:  

\begin{equation} \frac{A_1}{A_2} = \frac{b - \hat{C}}{\hat{C} - a}. \end{equation}

(16)

 

This ratio varies smoothly with \(\hat{C}\), tending to 0 (bump at a vanishes) when \(\hat{C} \rightarrow b\), and to +∞ (bump at b vanishes) when \(\hat{C} \rightarrow a\). 

Any such ratio of amplitudes will yield the desired barycenter; both amplitudes can be normalized, such that none exceeds 1. 

Numerical simulations

All simulation and plotting code was written in Python, using the JAX (Bradbury et al., 2018) and diffrax (Kidger, 2022) libraries to solve differential equations numerically. 

Our code and the supporting digitized data are publicly available at: https://github.com/m2b3/jov-2025-forward-remapping-mislocalization-code.