Saccade-induced changes in ocular torsion reveal predictive orientation perception

Abstract

Natural orienting of gaze often results in a retinal image that is rotated relative to space due to ocular torsion. However, we perceive neither this rotation nor a moving world despite visual rotational motion on the retina. This perceptual stability is often attributed to the phenomenon known as predictive remapping, but the current remapping literature ignores this torsional component. In addition, studies often simply measure remapping across either space or features (e.g., orientation) but in natural circumstances, both components are bound together for stable perception. One natural circumstance in which the perceptual system must account for the current and future eye orientation to correctly interpret the orientation of external stimuli occurs during movements to or from oblique eye orientations (i.e., eye orientations with both a horizontal and vertical angular component relative to the primary position). Here we took advantage of oblique eye orientation-induced ocular torsion to examine perisaccadic orientation perception. First, we found that orientation perception was largely predicted by the rotated retinal image. Second, we observed a presaccadic remapping of orientation perception consistent with maintaining a stable (but spatially inaccurate) retinocentric perception throughout the saccade. These findings strongly suggest that our seamless perceptual stability relies on retinocentric signals that are predictively remapped in all three ocular dimensions with each saccade.

Introduction

We move our eyes all the time, and with every movement we induce massive shifts of the retinal projection in all three dimensions: horizontal, vertical, and torsional. Despite this motion, we can keep track of both the locations and features (e.g., orientation) of objects in space. To achieve such stability in the presence of sensorimotor delays, the perceptual system is thought to compensate for each eye movement using predictive remapping, but if or how predictive remapping accounts for changes in the torsional state of the retinal image when interpreting spatial orientation is unclear. Further, the visual system needs to simultaneously keep track of features of objects (e.g., orientation) and their physical location in order to achieve spatial constancy during eye movements. 

Previous remapping work has only considered two-dimensional (2D) motion on the retina when in fact, shifts in the third, torsional dimension (i.e., around a rotation axis parallel to the line of sight) is also present during almost any eye movement and is a key component of ocular orienting. For example, retinal torsion can be induced by ocular counter-roll during head roll (Blohm & Lefèvre, 2010; Murdison, Paré-Bingley, & Blohm, 2013), by the natural tilt of Listing's plane (Blohm, Khan, Ren, Schreiber, & Crawford, 2008), or by simply manipulating the geometry of the retinal projection using oblique gaze orientations (Blohm & Lefèvre, 2010; which, importantly, does not require any mechanical torsion of the eyeball, and is sometimes termed “false torsion”). 

Orientation perception is not only influenced by e.g., head roll, but also by static head or body orientation in space. As an example, the oblique effect, i.e., the finding of smaller just-noticeable differences (JNDs) for orientation along the horizontal or vertical meridian as compared to oblique directions, is rather fixed to the head than to external space (Buchanan-Smith & Heeley, 1993). Differentiating between perceptual effects occurring in retinal or spatial coordinates has been impossible without linking remapping to exogenous factors such as the motion after-effect (Turi & Burr, 2012), the tilt after-effect (Melcher, 2007), or object features (Golomb, L'Heureux, & Kanwisher, 2014; Harrison & Bex, 2014). Conveniently, torsion provides a natural misalignment between retinal and spatial coordinates for which the perceptual system must directly compensate. Here, we geometrically induced torsional shifts by projecting a fronto-parallel stimulus onto the retina during movements to and from oblique eye orientations (oblique orientation-induced retinal torsion, ORT; Figure 1A). Past work has found that ORT influences orientation perception in a retinally predicted way during fixation (Haustein & Mittelstaedt, 1990; Nakayama & Balliet, 1977), yet no study has examined how ORT affects orientation perception during ongoing eye movements. 

Figure 1

 

Geometry of predictive remapping models. (A) Example ORT during oblique-to-oblique horizontal saccades. The real-world geometry, which is tilted on the retina due to ORT, does not correspond to a veridical representation of “vertical.” Note that ORT magnitude is exaggerated for illustration purposes. (B) Retinocentric representation of the retino-spatial predictive model and (C) of the purely retinal predictive model. Solid lines represent the actual retinal projection of the stimulus while dotted lines represent the corresponding percept.

Separate recordings from distinct retinotopic areas have revealed that receptive fields (RFs) presaccadically modulate their spatial tuning. In their seminal paper, Duhamel, Colby, and Goldberg (1992) showed that, before the onset of a saccade, some neurons from the lateral intraparietal area (LIP) became responsive for locations in the visual field which corresponded to receptive-filed locations only after the eye had landed. In this study, Duhamel and colleagues only tested the current and future receptive field location for responsiveness. Only more recently, Wang and colleagues (2016) could show that in such case LIP neurons expand their RF along the saccade trajectory. Neurons in the macaque frontal eye field (FEF) have been shown to converge towards the target location (Zirnsak et al., 2014). Consequently, these presaccadic RF modulations are assumed to be involved in the maintenance of perceptual stability, though how is unclear. 

Different potential explanations that have garnered some recent debate (Burr, Tozzi, & Morrone, 2007; Duhamel, Bremmer, Ben Hamed, & Graf, 1997; Harrison & Bex, 2014; Harrison, Mattingley, & Remington, 2012; Melcher, 2005; Morris, Bremmer, & Krekelberg, 2016; Morris & Krekelberg, 2019; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011; Turi & Burr, 2012; Zimmermann, Burr, & Morrone, 2011; Zimmermann, Morrone, Fink, & Burr, 2013; Zirnsak & Moore, 2014) are that either these RF modulations predictively remap a retinocentric representation purely in compensation for the upcoming retinal motion or they are involved in constructing a stable spatial map of the visual scene. Importantly, the remapping theoretically could be solved by two different mechanisms: either by tilting the orientation tuning towards the final ORT (similar to shifting spatial tuning across saccades; Duhamel et al., 1992), or by tilting the tuning away from the final ORT (similar to remapping neural activation in the direction opposite to the saccade; Rolfs et al., 2011). In the presence of torsional motion of the retinal image, these two models produce different predictions (Figure 1). In the one scenario (Figure 1B), the representation is remapped according to the preprogrammed spatial saccadic endpoint and accounting for the retino-spatial 3D geometry, such that the perception updates ballistically ahead of the eye. Under this hypothesis, there is a presaccadic remapping stage at which orientation perception is tilted in the direction of the saccade endpoint. While the eye is in flight, orientation perception leads the actual retinal projection. Therefore, at the midpoint of a symmetric trajectory, the perception of a spatially vertical tree is tilted towards the upcoming saccadic endpoint retinal projection. In the other scenario (Figure 1C), the perception is predictively remapped according to the vector difference between the initial retinal projection and that at the next time-step, such that the perception dynamically follows the eye. Under this hypothesis, there is a presaccadic remapping stage at which orientation perception is titled away from the saccade endpoint, allowing the perceptual system to account for visuomotor delays during motion to maintain a retinally accurate perception. Because of this predictive compensation, while the eye is in flight, orientation perception continuously matches the retinal projection. Therefore, near the midpoint of the same symmetric trajectory, the perception of a spatially vertical tree matches its projection onto the retina. 

There are three possible perceptual outcomes of torsional shifts of the retinal image during eye movements. First, there might be no predictive remapping, with orientation perception mostly adhering to ORT (Haustein & Mittelstaedt, 1990; Nakayama & Balliet, 1977) throughout the movement (null model). Second, the perceptual system might use an estimate of the future retino-spatial geometry based on the preprogrammed saccade endpoint to predictively tilt perception towards the final ORT, ahead of the eyes (retino-spatial model, Figure 1B). Third, the perceptual system might presaccadically tilt perception away from the final orientation, allowing a retinocentric perception to move with the eyes (purely retinal model, Figure 1C). Here we provide strong evidence in support of the purely retinal model using ORT during a perisaccadic orientation perception task. 

Materials and methods

Participants

Eight adults with normal or corrected to normal vision performed the experiment (five males, three females; age range 20–30 years). Participants were paid for their participation and were all naïve to the purpose of the experiment, and all had previous experience with psychophysical experiments involving video eye tracking. Each participant gave informed written consent prior to the experiment. All procedures used in this study conformed to the Declaration of Helsinki. 

Materials

Stimuli were computer-generated using the Psychophysics Toolbox (Brainard, 1997) within MATLAB (MathWorks, Natick, MA), and were projected onto a large 120 cm (81°) × 90 cm (65.5°) flat screen by means of a DS+6K-M Christie projector (Christie Digital, Cypress, California) at a frame rate of 120 Hz and a resolution of 1152 × 864 pixels. Participants sat in complete darkness 70 cm away from the screen, and a table-mounted chin rest supported their heads. The complete darkness was required to prevent participants to perceive a compression of space, which might have confounded our data by causing all orientations being perceived closer to vertical than they were (Krekelberg, Kubischik, Hoffmann, & Bremmer, 2003; Lappe, Awater, & Krekelberg, 2000; Morrone, Ross, & Burr, 1997). Eye movements were recorded using an infrared video-based Eyelink II (SR Research, Ottawa, Ontario) that was attached to the chin rest, providing a table-fixed head strap that kept each participant's head in a constant position throughout each experimental session. The screen was viewed binocularly, and eye position was sampled at 500 Hz. Prior to each block, participants performed a 13-point calibration sequence over a maximum eccentricity of 25°. The eye to which the perceptual stimulus was fovea-locked for each block was selected based on calibration performance. Drift correction was performed offline every 10 trials, based on a central fixation position. To ensure precise temporal measurement of trial start and stimulus presentation, we positioned a photosensitive diode over the lower left corner of the screen, where we flashed a white patch of pixels both at the start of each trial and at the presentation of the oriented bar stimulus (at the current on-screen gaze position of the participant). This part of the experimental apparatus was occluded from the view of the participant. After calibration for constant data acquisition delays, the photosensitive diode's voltage spikes provided reliable estimates of each trial's time-course (within a precision of approximately 2 ms). 

Procedure

Participants performed a two-alternative, forced choice (2AFC) perceptual task in which they made large horizontal saccades between targets 40° apart either along a 20° vertically eccentric horizontal axis (test trials) or along the horizontal meridian of the screen (control trials, Figure 2A). Importantly, test trials induced ORT throughout the eye movement. Participants began each trial by fixating the initial 0.3° diameter dot on the left side of the screen (at −20°) and indicated with a key press that they were prepared to start the trial (Figure 2B). Three hundred milliseconds later, a 0.3° diameter target was illuminated 40° to the right on the opposite side of the screen (at +20°). After a randomly selected duration (400–600 ms), the initial target was extinguished, representing the participant's “go” cue. At some point in time, either immediately before saccade onset (∼250 ms prior), during the saccade (average saccade duration ∼120 ms) or after the saccade, we presented an oriented bar stimulus in one of seven different orientations (from −8° to +8° rotated from vertical). For each trial, the exact time at which we presented the stimulus was chosen randomly from one of four 200 ms-width Gaussians, linearly spaced from the average reaction time (based on a 10-trial moving window) to 100 ms after, approximating the end of the movement. After the participant's eyes had landed on the saccade target, they were asked to respond with a key press representing their perception of the stimulus orientation (counterclockwise or clockwise perceptions). The trial ended after participants made their selection. This paradigm allowed us to reliably compute each participant's psychometric function with a fine time resolution throughout a saccade. 

Figure 2

 

Paradigm and task timing. (A) Illustration demonstrating rotational effects induced on retina due to oblique eye orientations while participants do task in either the test or control condition. Note that these retinal rotations are exaggerated for illustration purposes. Lower panel in (A) shows the geometrically predicted retinal torsion as a function of horizontal screen position for the Test (red) and control conditions (green). (B) Schematic showing task timing and stimulus presentation frequency distributions (inset). Bidirectional arrows represent 200 ms time window within which we randomly varied “go” cue.

Participants also performed a fixation version of the same task in which they fixated one of six randomly selected locations (−20°, 0° or +20° horizontal along either the 0° or 20° screen meridian) and we flashed the identical stimulus at the fixation location for a single frame. After the stimulus flash participants responded with a key press indicating their perception of its orientation, identically to the first experiment. In all conditions (fixation, test, and control condition), oriented bar stimuli were presented for a single frame (8.3 ms). 

Identifying Listing's plane for each participant

Finally, to correctly compute the retinal model predictions, we measured each participant's individual Listing's plane (Table 1) using photographs taken during fixation at each of 10 orientations on the screen (rectangular grid in the upper half of the screen along 0° and 20° meridians, with five equally spaced orientations along each horizontal and 20° eccentricity). Listing's Law states that with the head stationary, upright, and the eyes fixating an eccentric target, it can be assumed that this eccentric gaze can be achieved by a single rotation, starting from straight ahead gaze. Importantly, for all eccentric positions, all these rotation axes lie in the same plane, i.e., Listing's plane (Blohm & Crawford. 2007; Hepp 1990; Tweed, Cadera, & Vilis, 1990; von Helmholtz, 1867; Westheimer, 1957). From these photographs we extracted the natural ocular torsion based on the irises compared between the central orientation (0°, 0°) and eccentric locations, using an algorithm developed by Otero-Millan and colleagues (Otero-Millan, Roberts, Lasker, & Zee, 2015) modified for still images and implementation in MATLAB. 

Table 1

 

Identified Listing's plane tilt for each participant.

 

Analysis

All analyses were performed using custom MATLAB code (MathWorks, Natick, MA) and psychometric functions were fit using the Psignifit toolbox (Wichmann & Hill, 2001). Each participant performed 2080 trials in total, following a Gaussian distribution of presented stimulus orientations. Each performed a minimum of 221 repetitions for each of the most extreme bar orientations (±8°); conversely, for 0° bar orientations, they performed a maximum of 369 repetitions. These repetitions allowed us to be confident in our psychometric fits while not extending the sessions by oversampling easy trials. Trials containing blinks, loss of eye tracking, no saccades, hypometric, or inaccurate saccades (<25° amplitude or beyond 10° radius from target), or with reaction times greater than 1.5 s were all removed from the dataset (20% of all trials). Group-level statistics were computed using paired Student t tests, and participant-level and pooled analyses were performed using the bootstrapped 95% CI determined from Monte Carlo simulations during the psychometric curve fitting. 

We generated predictions for retinal torsion using the quaternion algebraic formulation developed in previous work from our lab (Blohm & Crawford, 2007; Blohm & Lefèvre, 2010; Leclercq, Blohm, & Lefèvre, 2013; Leclercq, Lefèvre, & Blohm, 2013; Murdison, Leclercq, Lefèvre, & Blohm, 2015). Briefly, this consisted of finding the torsional difference between screen coordinates and retinal coordinates, based on the orientation of Listing's plane for each participant with measured tilt α₀ (see Table 1):  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}{q_{LP}} = \left[ {\matrix{ 0 \cr 0 \cr {\cos \left( {{\alpha _0}} \right)} \cr { - sin\left( {{\alpha _0}} \right)} \cr } } \right]\end{equation}

 

Then, the gaze orientation vector q_ES was multiplied by q_LP to obtain the eye-in-head orientation q_EH observing Listing's Law:  

\begin{equation}\tag{2}{q_{EH}} = {q_{LP}}{q_{ES}}\end{equation}

where  

\begin{equation}\tag{3}{q_E} = \left[ {\matrix{ 0 \cr {{\theta _x}} \cr {{\theta _y}} \cr {{\theta _z}} \cr } } \right]\end{equation}

with θ_x, θ_y, and θ_z corresponding to the angular orientation of the eye.  

Finally, we computed the rotation V′ of a vertical unit vector V on the retina using standard quaternion rotation:  

\begin{equation}\tag{4}V^{\prime} = {q_{EH}}V{q_{EH}}^{ - 1}\end{equation}

 

Results

We directly investigated how ORT influences orientation perception across saccades using a novel retinal feature remapping paradigm in complete darkness. This paradigm allowed us to reliably compute each participant's psychometric function with a fine time resolution throughout the saccade. A fixation version of the task in which we presented the same stimulus at one of six possible fixation locations (three along each test or control trajectory—left, center, and right) allowed us to account for any perceptual effects during stable fixation. We also measured each participant's natural Listing's plane to account for any natural ocular torsion when making predictions using our retinal model (Table 1 in Methods). As can be seen, deviations of Listing's plane from the expected fronto-parallel plane was very small (< 1°). 

We examined the performance of each participant as a function of trial time (aligned to saccade onset) revealing orientation perception throughout any given trial, and we compared perceptions to the prediction of a retinal model. Participants had clear perceptual differences (Figure 3) between the start (light shades) and end (dark shades) of the saccade, but these differences were most pronounced for test trials. As the eyes moved across screen, we found that the perceptual changes were captured by the retinal model predictions during test trials (pooled regression analysis across 4° on-screen bins, n = 12, slope = 0.87, R² = 0.7, p < 0.01), and matched perceptions during fixation at the extreme time points: paired t test for points of subjective equality, t(15) = 1.52, p = 0.15, and for just-noticeable differences, t(15) = 0.47, p = 0.65, indicating that they behaved consistently during periods when the eyes were stationary, regardless of the behavioral context (fixation vs. saccade trials). 

Figure 3

 

Psychometric curves for all participants. Early (trial start until 50 ms presaccade; light shades) and late (100 ms postsaccade until trial end; dark shades) psychometric function fits are shown alongside the fixation experiment results for left (dotted) and right (dashed) targets. Color-matched arrows represent retinal predictions for ORT, corresponding to PSEs, during each time bin.

After pooling the data across participants, we were able to attain a time resolution of 15 ms for which we could compute the bin-wise psychometric functions, extracting the points of subjective equality (PSEs) to quantify the psychophysical biases and the just-noticeable differences (JNDs) to quantify the corresponding precision. These time-resolved biases (PSEs with 95% CIs) are shown alongside the retinal predictions (dashed lines) for both control (green) and test trials (blue) relative to saccade onset (Figure 4A). Psychophysical biases depended on whether participants performed control or test trials. Throughout control trials, perceptual biases followed the retinally predicted perception, with excursions from the retinal prediction occurring upon, but not prior to, saccade onset. Throughout test trials, however, orientation perception was biased towards the retinal prediction throughout the movement, with the exception of a significant perceptual rotation immediately prior to the movement onset. Using the pooled data, the effect began approximately 50 ms prior to the movement (gray-shaded window; inset), consistent with the timing of both attentional (Harrison et al., 2012; Rolfs et al., 2011) and RF shifts observed in retinotopic areas (Wang et al., 2016; Zirnsak et al., 2014). Furthermore, this deviation went in the direction opposite to the upcoming shift in ORT in a manner consistent with maintaining the retinocentric orientation throughout the upcoming movement, matching the purely retinal model. 

Figure 4

 

Pooled and participant-level biases. (A) Pooled PSEs (left column) and JNDs (right column) for control (green) and test trials (blue), plotted alongside the retinal predictions (color-matched dashes) over time. Gray shaded regions represent presaccadic (−50 ms to 0 ms) time bin. Inset reveals significant presaccadic perceptual rotation for test PSEs (asterisks). (B) Participant-level PSEs and JNDs, binned into early (t < −50 ms), presaccadic (−50 ms < t < 0 ms), perisaccadic (0 ms < t < 100 ms), and postsaccadic time bins (t > 100 ms), aligned to saccade onset for control (top row) and test trials (bottom row). Participant-level significant effects are shown by color-matched bars crossing bin thresholds (vertical dotted lines), and group-level significant effects are shown by bold black crossing lines and black asterisks. Insets (center column) reveal direct comparisons between PSEs in presaccadic (ordinate) and early time bins (abscissa). Within the test inset, shaded quadrant represents the retinal hypothesis for either time epoch, and arrows represent direction of retino-spatial (black) or purely retinal (red) remapping. Black circles and error bars represent across-participant means and standard deviations.

We next determined if this observed effect during test trials was simply a phenomenological effect of pooling the data across participants (Figure 4B). We separated each participant's data into four separate time bins representing characteristic time epochs during any given trial: (a) early fixation (trial start to 50 ms prior to onset); (b) Presaccadic (50 ms prior to saccade onset); (c) Perisaccadic (saccade onset to 100 ms later); and (d) Postsaccadic (100 ms post saccade onset until trial end). Using these binned data, we observed the same presaccadic bias shift on the group level for test trials, paired t test, t(7) = −4.33, p < 0.01, indicating that it was not due to pooling data across participants. We varied the presaccadic bin size as much as participants' time resolutions allowed and found qualitatively identical group-level presaccadic remapping effects up to 40 ms prior to onset (not shown here). Finally, as these bias shifts could potentially be simply explained by a less precise perception, we also examined the time-resolved changes in precision. We did this with JNDs in an identical way (Figure 4A and B, right column), and found that they only increased perisaccadically (paired t tests, all transsaccadic p < 0.01), as expected from retinal blurring and/or saccadic suppression (Bremmer, Kubischik, Hoffmann, & Krekelberg, 2009; Burr, Morrone, & Ross, 1994), but presaccadic precision was not different from precision during fixation. Thus, presaccadic perceptual shifts could not be explained by a decrease in perceptual precision. 

Discussion

We found that ORT, which is only partially corrected for during fixation (Haustein & Mittelstaedt, 1990; Nakayama & Balliet, 1977), is predictively remapped across saccades in an orientation perception task. Instead of updating the perception ahead of the eye movement using an estimate of the spatial geometry at the final gaze location (retino-spatial model), the presaccadic shifts we observe instead are compensatory for the future ORT, allowing the retinocentric orientation to be maintained while the eyes move (purely retinal model). This key finding agrees with recent psychophysical work (Golomb et al., 2014; Rolfs et al., 2011). 

Importantly, these behavioral results are in line with the earliest neurophysiological data from the macaque monkey. In their seminal paper, Duhamel et al. showed that neurons in the lateral intraparietal area (area LIP) respond completely retinocentric when tested before and after a saccade (1992). And even across saccades, these neurons anticipate the new location of their RF in eye-centered coordinates. Neurons in area LIP are also implicated in eye-movement control, which, by definition, is oculocentric. In follow-up studies it was argued by the same authors (Colby et al., 1995; Colby & Goldberg, 1999), that such an encoding is advantageous since it facilitates the programming of appropriate eye-movements without the need of transforming visual signals into a nonretino-centric frame of reference (Bremmer, Pouget, & Hoffmann, 1998; Boussaoud & Bremmer, 1999; Zipser & Andersen, 1988). Instead, visual signals are directly transformed into motor output. In this sense, sensorimotor processing is facilitated, since the receptor (eye, with the retina as receptor epithelium) is identical with the effector (eye). 

In the control condition, the average PSE for times ranging from 100 ms to 200 ms postsaccadically did not return to baseline values as observed long before the saccade. Although this might appear surprising at first glance, it is well in line with results from previous studies on perisaccadic visual perception. These studies have shown that effects of e.g., saccadic suppression, perisaccadic compression of perceptual space, as well as of compression of heading perception, outlast the end of a saccade by approximately 50–100 ms (Bremmer et al., 2009; Bremmer, Churan, & Lappe, 2017; Diamond et al., 2000; Ross, Morrone, & Burr, 1997). In our study, saccades were rather large and, hence, had an average duration of roughly 120 ms. Accordingly, the postsaccadic data as shown in Figures 3 and 4 are still in a temporal context in which visual perception is supposed to return to normal. Importantly, the studies mentioned above all employed visually guided saccades along the horizontal or vertical meridian, i.e., as in our control condition. Remarkably, also in the control condition we found a (comparably small) negative deflection of the PSE at saccade onset. As shown in Table 1, the Listing's planes of our observers were all slightly off the ideal case of being fronto-parallel (Bockisch & Haslwanter, 2001; Haslwanter, Straumann, Hess, & Henn, 1992). Accordingly, the saccade in the control condition might have been not exactly along the HM of the observer and hence most likely induced a minimal torsion. The observed modulation of the PSE could be indicative of this slight misalignment and of a smaller, predictive orientation remapping. 

Our study complements previous studies on transsaccadic orientation perception (Ganmor, Landy, & Simoncelli, 2015; Wolf and Schütz, 2015). In both studies, oriented stimuli were presented before, after, or before and after a saccade along the horizontal meridian (thus not introducing different torsional values before and after the saccade). Best performance was found when an oriented stimulus was visible both before (peripherally) and after a saccade (foveally), suggesting that humans integrate both signals. A detailed analysis further revealed that humans combined the two views close to optimal, using a weighted sum, with weights assigned based on the relative precision of foveal and peripheral representations. It would be interesting to apply this approach of stimuli being presented before and after a saccade to our paradigm. Similar to the study by Wolf and Schütz (2015), the analysis would allow determining the time-course of the weighting of foveal and peripheral information in case of saccades inducing retinal torsion. 

Our psychophysical results predict that the activity of orientation-selective neurons involved in predictive remapping should also exhibit torsion-induced modulations. Importantly, we assume that the predictive remapping of orientation perception concerns the whole visual field and not only an area around the fovea. Accordingly, we do not assume single neurons to rotate their orientation selectivity perisaccadically. Instead, we assume that orientation selective responses are combined with estimates about the rotational angle of the eye. This process would be very similar to perisaccadic processes combining RF location with eye position signals as have been shown for extrastriate and parietal areas (Morris et al., 2012, 2016) but recently also for primary visual cortex (Morris & Krekelberg, 2019). 

To our best knowledge, no such recordings have been made so far to test for a representation of the rotational angle of the eye in the ongoing or perisaccadic activity of neurons along the visual pathway, as has been done for eye position signals (e.g., Bremmer, Distler, & Hoffmann, 1997; Bremmer, Ilg, Thiele, Distler, & Hoffmann 1997; Bremmer, Graf, Hamed, & Duhamel, 1999; Bremmer, Duhamel, Ben, & Graf, 2000) and corollary discharge concerning upcoming saccades (e.g., Sommer & Wurtz, 2004; Zimmermann & Bremmer, 2016). Accordingly, we can only speculate that such signals might exist. 

The implication that the brain expends computational energy with each eye movement to predictively remap a (spatially incorrect) retinal perception is seemingly paradoxical; after all, in theory the brain has access to all the self-motion signals required to compensate for retinal blurring and/or retino-spatial misalignments. However, compensating for self-motion requires either updating of a nonspatial (e.g., retinal) representation (Henriques, Klier, Smith, Lowy, & Crawford, 1998; Medendorp, Van Asselt, & Gielen, 1999; Murdison et al., 2013) or subjecting sensory signals to reference frame transformations (Blohm & Crawford, 2007; Blohm & Lefèvre, 2010; Murdison et al., 2015) to achieve spatial accuracy. As both updating (Medendorp et al., 1999) and reference frame transformations appear to be stochastic processes (Alikhanian, Carvalho, & Blohm, 2015; Burns & Blohm, 2010; Burns, Nashed, & Blohm, 2011; Schlicht & Schrater, 2007; Sober & Sabes, 2003), eye-centered signals might provide high acuity sensory information on which to base working memory (Golomb, Chun, & Mazer, 2008), perception (Burns et al., 2011; Rolfs et al., 2011) and movement generation (Schlicht & Schrater, 2007; Sober & Sabes, 2003) explicitly requiring a reference frame transformation. 

The apparent dominance of retinocentric signals we observed during saccades is consistent with a growing body of psychophysical (Golomb et al., 2014; Murdison et al., 2013; Rolfs et al., 2011; Zirnsak, Gerhards, Kiani, Lappe, & Hamker, 2011) and electrophysiological (Colby, Duhamel, & Goldberg, 1995; Duhamel et al., 1992; Duhamel et al., 1997; Wang et al., 2016; Zirnsak et al., 2014) evidence. Indeed, participants are better at recalling the locations of stimuli across saccades in eye-centered coordinates compared to their spatial locations, which are degraded with each subsequent eye movement (Golomb & Kanwisher, 2012). Additionally, attention appears to be allocated in retinocentric coordinates (Golomb et al., 2008; Golomb, Nguyen-Phuc, Mazer, McCarthy, & Chun, 2010; Yao, Ketkar, Treue, & Krishna, 2016) and there is evidence that its locus shifts to the retinocentric target of upcoming saccades (Rolfs et al., 2011). Memorized targets for movement also appear to be encoded retinocentrically, as observed during saccades (Inaba & Kawano, 2014), smooth pursuit (Murdison et al., 2013), and reaching (Batista, Buneo, Snyder, & Andersen, 1999; Henriques et al., 1998; Medendorp et al., 1999). Together with this past work, our findings indicate that reliable retinal signals are paramount to maintaining a stable world percept during self-motion. 

Conclusions

For the first time, we have shown the orientation-specific perceptual consequences of shifts in the torsional dimension during saccades. Together with previous work (Wang et al., 2016; Zirnsak & Moore, 2014; Zirnsak et al., 2014), our current findings imply that the perceptual system faithfully maintains an eye-centered representation by predictively remapping across both translational and torsional retinal shifts. In the midst of motion on the retina with each exploratory eye movement, it appears that this predictive remapping underlies the seamless stability that is a hallmark of our perceptual experience. 

Acknowledgments

The authors want to thank Dr. Dominic Standage for his helpful comments on the manuscript, as well as the participants for their kind participation. This work was supported by DFG (IRTG/CREATE-1901) and CRC/TRR-135 (project number 222641018), Germany), NSERC (Canada), CFI (Canada), the Botterell Fund (Queen's University, Kingston, ON, Canada) and ORF (Canada). TSM was also supported by DAAD (Germany). 

Commercial relationships: none. 

Corresponding author: T. Scott Murdison. 

Email: . 

Address: Facebook Reality Labs (FRL), Redmond, WA, USA.