Weighted integration of short-term memory and sensory signals in the oculomotor system

Abstract

Oculomotor behaviors integrate sensory and prior information to overcome sensory-motor delays and noise. After much debate about this process, reliability-based integration has recently been proposed and several models of smooth pursuit now include recurrent Bayesian integration or Kalman filtering. However, there is a lack of behavioral evidence in humans supporting these theoretical predictions. Here, we independently manipulated the reliability of visual and prior information in a smooth pursuit task. Our results show that both smooth pursuit eye velocity and catch-up saccade amplitude were modulated by visual and prior information reliability. We interpret these findings as the continuous reliability-based integration of a short-term memory of target motion with visual information, which support modeling work. Furthermore, we suggest that saccadic and pursuit systems share this short-term memory. We propose that this short-term memory of target motion is quickly built and continuously updated, and constitutes a general building block present in all sensorimotor systems.

Introduction

Over the last two decades, Bayesian integration of different signals (i.e., the weighted summation based on the respective reliability of each signal) has been widely applied to the study of cognitive processes. Its intrinsic ability to handle the uncertainty of different signals and combine them makes it a particularly useful tool to model how the brain handles the imperfect sensory representations of the external world. Indeed, be it relative to movement (Tassinari, Hudson, & Landy, 2006; Yang, Lee, & Lisberger, 2012), learning (Nassar, Wilson, Heasly, & Gold, 2010), or estimation (Stocker & Simoncelli, 2006), numerous studies have highlighted behaviors that exhibit reliability-based integration. Furthermore, when several senses give information about the same event or object, reliability-based integration allows their (near) optimal combination (Ernst & Banks, 2002; Jacobs & Fine, 1999; Landy, Banks, & Knill, 2011). It has recently been suggested (Bogadhi, Montagnini, & Masson, 2013; Orban de Xivry, Coppe, Blohm, & Lefèvre, 2013) that internal predictive and sensory afferent information is combined in a Bayes-optimal fashion during smooth pursuit eye movements; this prediction has not been tested explicitly in humans. 

The oculomotor system is well studied and offers many typical examples of sensorimotor processes relying on both noisy sensory inputs (Bogadhi et al., 2013; Osborne, Lisberger, & Bialek, 2005; Spering, Kerzel, Braun, Hawken, & Gegenfurtner, 2005) and prior experience (Kowler, Martins, & Pavel, 1984; Madelain & Krauzlis, 2003; Yang et al., 2012) to produce accurate movements. For example, visual tracking has to cope with sensory delays (Osborne, Bialek, & Lisberger, 2004) and internal noise (Osborne et al., 2005). By integrating visual inputs with past experience and cues, the oculomotor system can overcome sensory delays and noise to produce eye movements matching current target movement. In the pursuit system, this allows, for example, anticipatory smooth eye movements (Dodge, Travis, & Fox, 1930; Hayhoe, McKinney, Chajka, & Pelz, 2012; Kowler, Aitkin, Ross, Santos, & Zhao, 2014; Westheimer, 1954) or zero-lag smooth pursuit tracking of a sinusoidal target motion (Dodge et al., 1930; Orban de Xivry et al., 2013). In the saccadic system, predictive saccades can be observed when tracking a target jumping at a fixed frequency (Shelhamer & Joiner, 2003), or a bouncing ball (Diaz, Cooper, Rothkopf, & Hayhoe, 2013). Thus predictive and sensory information interact to drive oculomotor behavior. 

The process of integration of sensory and predictive signals is still up to debate. Several models have been proposed to explain how sensory inputs and predictive signals might be integrated during smooth pursuit: Some have used a switching mechanism between predictive and sensory-feedback processes (Barnes, 2008; Bennett & Barnes, 2004), but in recent years several authors turned to reliability-based integration (Bogadhi, Montagnini, Mamassian, Perrinet, & Masson, 2011; Dimova & Denham, 2009; Freeman, Champion, & Warren, 2010; Montagnini, Mamassian, Perrinet, Castet, & Masson, 2007). In 2013, Orban de Xivry et al. proposed a model that, for the first time, was able to simulate the integration of a continuous flow of sensory information with past experience to drive motor behavior. This model used reliability-based integration (Kalman filtering) both to predict target movement and to simultaneously build a dynamic memory of it. Specifically, the model predicts that the estimated target movement is the reliability-weighted (reliability = 1/variance) average of visual and predictive target motion signals, in agreement with Bayes-optimal cue integration. This model managed to reproduce a large repertoire of pursuit behaviors. The same year, Bogadhi et al. (2013) proposed a similar model that used a static memory of target velocity and two recurrent Bayesian networks for sensory and predictive signal integration. However, while there are several studies on the influence of prior information on oculomotor tracking (Kowler, 2011; Kowler et al., 2014; Wende, Theunissen, & Missal, 2013; Yang et al., 2012), few if any investigated the mechanisms of integration of such prior information with sensory information, and how the reliability of the memory and the sensory inputs can influence this integration, until very recently, when this was done with monkeys by Darlington, Tokiyama, and Lisberger (2017). 

Here, we present two target eye-tracking experiments, with 13 human participants, in which we independently manipulated the uncertainties of visual information and short-term memory. As predicted by the model of Orban de Xivry et al. (2013), we observe reliability-based integration of a short-term memory of target motion with visual information during movement. Furthermore, we extend the results of Darlington et al. (2017) by showing that, in addition to smooth pursuit, catch-up saccades also exhibit features of this reliability-based integration. 

Given the similarities of smooth pursuit with other cortical sensorimotor systems (Lisberger, 2015; Lynch & Tian, 2006) and recent studies of memory updating (Gershman, Radulescu, Norman, & Niv, 2014; Nassar et al., 2010), we believe that our results validate an important prediction; continuous reliability-based integration of current sensory information with working memory signals is a general principle that is likely to be part of all sensorimotor processes. 

Methods

Participants

Because of the absence of previous literature on the topic, a power analysis could not be used beforehand to determine the number of participants. We therefore decided to refer to what is typically done in eye movement research and targeted a pool of more than 10 participants per experiment. Twenty participants between 18 and 30 years old were recruited to participate in our experiments. Thirteen participants (four female) participated in the first experiment and 13 (six from the first) in the second experiment (five female). 

Participants had normal or corrected-to-normal vision. After being given a full description of the experiment, informed consent was given by the participants. The procedures were approved by the Université catholique de Louvain Ethics Committee and were in accordance with the Declaration of Helsinki. 

Protocol

Participants were seated in a dark room, and looked at a 197 × 150 cm screen at 151 cm in front of them, spanning ±40° of their visual field. Head movements were restrained with chin and forehead rests. The stimuli were projected onto the screen with a cine8 Barco projector (Barco Inc., Kortrijk, Belgium) at a refresh rate of 100 Hz, and the eye movements were recorded at 1,000 Hz using an Eyelink 1000 (SR Research, Ottawa, Ontario, Canada). The display of visual stimuli was handled by an in-house toolbox, while interactions with the Eyelink were handled by the Psychtoolbox (Kleiner et al., 2007). Calibrations trials were first performed at the start of the experiments, then every 30 trials (±2 min). Breaks were allowed before every calibration (every 2 min), and the total duration of an experiment was around 30 min. 

In the design of the protocol, we wanted to make sure that behaviors could only be related to the current block—that is, that there was no transfer of information between blocks. As such, block duration and features were randomized, direction changed after each block, and each of them started with a passive trial meant to wash out previous block-related memories. 

Two types of stimuli were used: a red fixation target (uniform disk, diameter 0.8°) and a red pursuit target. The protocol was identical for both experiments except for the pursuit target. In Experiment 1, the pursuit target was a two-dimensional (2-D) Gaussian spot (σ = 1.27°; hereafter the noisy target, cf. Supplementary Figure S1). In Experiment 2, the pursuit target was a uniform disk (0.8° of diameter; hereafter the standard target, cf. Supplementary Figure S2). The overall luminance of the stimuli was the same for all stimuli; however, given the difference in the distribution (the pixel at the center of the noisy target has 5% of the luminance of any pixel of the standard target), the noisy target was harder to perceive. 

For both experiments, there were three types of trial: passive, training, and test. All trials started with the display of the fixation target at the center of the screen. After 500 ms, the pursuit target appeared at the center and immediately moved in one out of six possible directions (−20°, 0°, 20°, 160°, 180°, or 200°) at a constant velocity (15°/s or 20°/s for training trials) during 650 ms. In passive trials, the fixation target remained on for the whole trial, and participants were instructed to keep looking at the fixation target while inhibiting movements toward the pursuit target. In the other trials, the fixation target disappeared at the onset of the pursuit target and participants were instructed to follow the pursuit target with their eyes (see Figure 1). 

Figure 1

 

Experimental protocol. (A) Time course of trials (in one of the six possible directions). (B) Template of blocks (regardless of direction). Each block starts with a passive trial and ends with a test trial (red/green), and direction remains constant within a block. Percentages indicate the % of blocks containing trials of the same type (color) and number (#) at one of the two available velocities (e.g., 100% of the blocks have one passive trial). Black rings give an example of two trials that can be compared to highlight the effect of previous trials. Black diamonds give an example of trials comparisons highlighting visual effect. Dashed arrows indicate a 15°/s prior target velocity. Solid arrows indicate a 20°/s prior target velocity. Note that the noisy target shown here has been made more salient in Panel (A) to ensure that it is visible on the figure. Its actual appearance is shown in Supplementary Figure S1, with Supplementary Figure S2 showing the standard target as reference.

The trials were presented in blocks: Each block consisted of one passive trial followed by one to five trials with 850 ms between trials. To warn participants of the start of a new block, an auditory cue (440 Hz, 80 ms) was given 200 ms before the appearance of the fixation target. Target direction and velocity remained constant throughout a block, except for the last trial, hereafter named test trial, for which velocity was reduced or increased by 5°/s with respect to previous trials of the same block. Training trials are the trials displayed between the passive trial and the test trial. To summarize, each block started with one passive trial and ended with one test trial, with up to four training trials in between (Figure 1B). 

As examples of two possible blocks, we will describe the three-trial block ending at the black-ring marked (red) test trial, and the six-trial block ending with the (green) test Trial 6 at 20°/s (cf. Figure 1B). The three-trial block begins at Trial 1 with a passive trial with a target at 20°/s. It is followed by one training trial (Trial 2), also at 20°/s. The last trial (Trial 3) is a test trial that, in this case, has a lower (red arrow) target velocity of 15°/s. The six-trial block also begins with a passive trial (Trial 1), this time at 15°/s. Knowing that, we can then follow the dashed arrows (indicating a prior target velocity of 15°/s) and identify the ensuing four training trials (Trials 2–5) that have the same target velocity of 15°/s. The final trial is the test trial (Trial 6) that has a higher (green arrow) target velocity of 20°/s and concludes the block. 

There were 120 conditions in total (6 Directions × 2 Training Velocities × 5 Block Lengths × 2 Test Velocities). The average number of trials per condition was four, yielding 480 trials per participant. 

In this protocol, we expected the noisy target to elicit less reliable visual information, and the repetition of training trials to elicit more reliable short-term memory of target motion. Within the framework of the model (Orban de Xivry et al., 2013), the reliability of each of these inputs directly affects its weight in the integration of target motion (according to Bayesian or more generally reliability-based integration). As a consequence, the mismatch in target velocity created by test trials will lead to different eye velocity outcomes depending on the relative reliabilities of short-term memory and visual information. 

Comparisons between trials

To observe the influence of previous trials on the oculomotor response, we compared trials that had the same current target velocity, but different past target velocities (different prior trials). For example, we compared test trials having a target velocity of 15°/s (thus preceded by training trials at 20°/s) with training trials having a target velocity of 15°/s (thus preceded by training trials at 15°/s). In this situation, visual information (target motion) is the same for both test and training trials, but prior information is different, effectively highlighting its effect (cf. black rings in Figure 1B). 

In a similar way, the impact of visual information was estimated by comparing test trials to training trials that had the same past target velocity, but different current target velocities. (cf. black diamonds in Figure 1B). 

We always made comparisons between trials with the same trial number (in the same column on Figure 1B). 

Data processing

Data were processed using the MATLAB software (RRID:SCR_001622; MathWorks, Natick, MA). Blinks were detected based on missing values in the Eyelink output (when the pupil cannot be detected) and subsequently removed from the data, including a safety margin before and after the blink, up to the first local minimum in the y-coordinate. Eye position signals were low-pass filtered at 40 Hz. Eye velocity and acceleration were obtained from position signals using a central difference algorithm on a ±10-ms interval. For the analyses, we pooled data across all directions. 

Saccade onset and offset were detected using a 500°/s² threshold on the acceleration data. Saccades were thereafter removed from smooth eye velocity data and replaced by linear interpolation. 

In order to remove abnormal trials from the data set while limiting visual inspection of the data, we set a few criteria: (a) during the last 100 ms of fixation, eye position within 3° of the fixation target, (b) no missing data (blinks) in the first 450 ms of pursuit, (c) lower limit on eye displacement of at least 40% of target displacement, (d) no eye velocity over 40°/s during pursuit epochs. Based on these criteria, we set aside less than 3% of the trials. When analyzing eye velocity during pursuit epochs, we included only trials for which steady state pursuit velocity reached 33.33% of target velocity (98% of trials). 

Measures of pursuit features

In this protocol, we took no step to reduce the occurrence of catch-up saccades in the early phases of pursuit, as we wanted to guarantee continuous target display (no gap) and coherent target direction within a block (no step-ramp paradigm; Rashbass, 1961). As a consequence of this approach, most trials contain saccades during the early phase of pursuit. This, combined with the lower eye velocity (in particular for the noisy target) makes measures of pursuit features more difficult during this phase. Looking at the very start of the initiation, around 100 ms, would avoid most saccades; however, our interest is in the integration process of visual information with prior information, the former of which is barely accessible at that time. 

Therefore, we concentrated our analyses on the steady-state phase of smooth pursuit, during which eye velocity is higher, eye acceleration is lower, and saccades are more uniformly distributed, which allows us to make more robust measures of eye velocity. In parallel, we were able to study the integration of visual and prior information in the early pursuit through the analysis of catch-up saccades. 

Fitting algorithm

Pursuit onset and eye movement velocity during steady-state were obtained for each trial by fitting a piece-wise linear regression on the eye velocity data (least square regression, using the lsqcurvefit function of the MATLAB Optimization Toolbox), as follows (Figure 2):  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}f\left( t \right) = \left\{ {\matrix{ {P1\ {\rm{if\ }}t \le T1} \cr {P1 + B \times \left( {t - T1} \right){\rm\ {if\ }}T1 \lt t \le T2} \cr {P2\ {\rm{if\ }}T2 \lt t} \cr } } \right.\end{equation}

Where t is time in seconds, P1 and P2 are velocity plateaus (representing initial and steady-state velocities, in degrees per second), T1 and T2 are the reaction time and the pursuit steady-state onset time (seconds), and B is the initial acceleration, connecting the plateaus between times T1 and T2 (degrees per second squared). P1, P2, T1, and B were the free parameters of this fit. The fitted interval spanned 650 ms, from 50 ms before target onset up to 600 ms after.  

Figure 2

 

Typical fit and its initial parameters. The orange dashed line illustrates the initial parameters for the fit. The continuous green line shows the resulting fit of the eye velocity data.

Initial parameters for P1, P2, T1, and B were determined as follows: P1 was set to the average velocity in the first 100 ms of the interval, T1 to the time after which the eye velocity exceeded 20% of the target velocity. The variable T2 was defined as the start of the subinterval during which eye velocity exceeded 33% of the target velocity for at least 125 ms. P2 was then set to the average value of the eye velocity during the interval. Finally, B was determined from the previous parameters, such that it didn't exceed 150°/s². If any of the conditions couldn't be met, or if a suitable interval (>33% of target velocity for 125 ms) couldn't be found, the initial parameters were set to default values: T1 to 150 ms, T2 to 350 ms, and P2 to 80% of the target velocity. 

Since we wanted the fitting algorithm to measure smooth pursuit eye velocity, we gave less weight to the eye velocity data interpolated during catch-up saccades, setting it to 0.3 (compared to 1 for pursuit data). 

After applying the fitting algorithm, trials whose steady-state velocity plateau (P2) duration was under 50 ms were analyzed again. They went through a second step of fitting, using the same function, but with T2 as a free parameter instead of B. This allowed the use of a different set of initial values, with a steady-state velocity plateau lasting at least 50 ms. After the second fitting, any trial whose fitted steady-state plateau (P2) was still less than 50 ms long was rejected (±6% of trials), as it meant that the algorithm could not find the steady-state of the smooth pursuit. 

Computation of smooth pursuit eye velocity gain

The eye velocity gain of training trials was computed simply by dividing the eye velocity during steady state by the target velocity. Test trials, by definition, always have a target velocity that differs from the previous trial. To be able to compare them with training trials, we computed the eye velocity gains of test trials by dividing the eye velocity during steady state by the target velocity of training trials with the same trial number. The training trials used were the ones the test trials were being compared to, and therefore depended on the type of comparison (prior or visual, as defined below). 

When studying the influence of prior information, the eye velocity gains were computed with respect to the training trials having the same target velocity as the test trials (cf. black rings in Figure 1B). For example, the data in the red error bar of Figure 4B correspond to test trials having a prior target velocity of 20°/s and a current target velocity of 15°/s. Since these test trials are compared to training trials with a target velocity of 15°/s, the eye velocity is divided by 15 and the mean of training gains is subtracted from it. The result is bigger than zero, revealing an effect of prior information. In a similar way, the data in the green error bar of Figure 4B corresponds to test trials with a prior velocity of 15°/s and a current target velocity of 20°/s, whose gain is therefore computed by dividing eye velocity by 20, and from which the average gain of training trials at 20°/s is also subtracted. 

When studying the influence of visual information, the eye velocity gains were computed with respect to the training trials having the same prior target velocity as the test trials (cf. black diamonds in Figure 1B). For example, the data in the green and red dashed error bars of Figure 5B correspond to test trials having a prior target velocity of 15°/s (indicated by the dashes) and either 20°/s (green) or 10°/s (red) of current target velocity. We compare them to training trials having the same prior target velocity, 15°/s, and therefore divide the eye velocity by 15 before subtracting the mean gain of training trials, highlighting the effect of the change in target velocity. 

Normalization of gains across the two training target velocities

Our protocol is such that test trials target velocities matched training trials target velocities at 15°/s and 20°/s. As such, the sign of the prior effect on eye velocity was positive at 15°/s (prior at 20°/s) and negative at 20°/s (prior at 15°/s). To be able to compare across the two training target velocity conditions, we normalized, for each participant, the difference of eye velocity between training and test trials by the difference between their prior target velocities according to the following Equation 1:  

\begin{equation}\tag{1}NormEffec{t_{prior}} = {{mean(Eye\ Velocity\ of\ Training)\atop - Eye\ Velocity\ of\ Test} \over {Prior\ Target\ Velocity\ of\ Training\atop - Prior\ Target\ Velocity\ of\ Test}}\end{equation}

When the comparison across velocities had to be made for the effect of visual information, we normalized, for each participant, the difference of eye velocity between training and test trials by the difference of their target velocities using the following Equation 2:  

\begin{equation}\tag{2}NormEffec{t_{visual}} = {{mean(Eye\ Velocity\ of\ Training)\atop - Eye\ Velocity\ of\ Test} \over {Target\ Velocity\ of\ Training\atop - Target\ Velocity\ of\ Test}}\end{equation}

Both equations allowed us to report the effects of visual or prior information on eye velocity as a percentage of the change in target velocity (5°/s).  

Saccade metrics

We also studied the amplitude of the first (catch-up) saccade occurring between 100 and 400 ms after target onset (94% of first saccades made after target onset). The amplitude of the catch-up saccades made during training trials was used as a reference to build a linear model of the saccadic behavior of each participant (cf. Figure 3). 

For each of the participants, we computed the ideal amplitude of each saccade (difference between eye position at the onset and target position at the offset), and its actual amplitude. Then, the baseline relationship between those two parameters was obtained from training trials by fitting a linear regression (robustfit function, Statistics Toolbox, MATLAB) on the saccades data. Finally, saccades made during test trials were compared to this regression line by computing the mean of the residuals (vertical distances between test saccades data points and the regression line). Given this method, a mean value greater than zero implied that saccades made during test trials had larger amplitudes than those made during training trials. 

To compare saccades made during test trials across experiments (standard and noisy targets), we had to take into account differences of saccade latency between the two experiments. On average, the first catch-up saccade elicited in the noisy target experiment was made 50 ms later than in the standard target one. To keep our analysis clear and limit the number of parameters to account for, we decided to only compare saccades of similar latencies, for which participants had the same exposure to the visual stimuli. 

Figure 3

 

Computation of saccade residuals. Blue dots correspond to saccades made during training trials, red dots to saccades made during test trials with the same target velocity, but whose prior trials had a higher target velocity. The fit made on the training data is shown by the black line. Dashed lines and arrows show the measure of the residual of one of the test trials.

Figure 4

 

Prior information effect on eye velocity gain. (A) Averaged eye velocity traces of Training15 and Test15, for all participants (of the first experiment, noisy target), for all trials allowing direct comparison between a test and a training condition with a current target at 15°/s. Blue traces are the average eye velocity of training trials, while red traces are those of the test trials. The surrounding hue indicates the standard error of the mean. (B) Averages of participant's average differential gains of the smooth pursuit eye velocity response to a noisy target moving at 15°/s (full red line) or 20°/s (dashed green line) that was preceded by trials with the same target either at a higher (20°/s, full red line) velocity or a lower (15°/s, dashed green) velocity (test trials). The error bars indicate the standard error of the mean. (C) Same as panel (B), but for the second experiment with a standard target. Dashed lines indicate a 15°/s prior target velocity.

Figure 5

 

Visual information effect on eye velocity gain. (A) Averaged velocity traces of all participants (noisy target), for all trials with the same target velocity prior—15°/s—and all comparable trial numbers (all but the first and the last). The conditions Training15 (blue trace), Test10 (decrease of velocity, red trace) and Test20 (increase in velocity, green trace) have the same prior at 15°/s. The hue around each trace is the standard error of the mean. (B) Averages of participant's average differential gains of the smooth pursuit eye velocity response to trials of a noisy target moving at 10°/s or 20°/s (respectively red and green dashed lines) or at 15°/s or 25°/s (respectively red and green solid lines) that were preceded by trials with the same target either at a higher velocity (all red lines) or a lower velocity (all green lines). The x-axis indicates the trial number and the error bars the standard error of the mean. (C) Same as Panel (B), but for the second experiment with a standard target.

Therefore, we used a bootstrap procedure to create N = 10,000 samples of saccades from the two experiments, such that, for each sample, the latency distributions of the two experiments would be the same. This was done for each training target velocity. For each participant, the average residuals (one per training velocity) for each experiment were then obtained by averaging the average residuals (Display Formula\({\bar R_i}\)) obtained for each of the 10,000 bootstrapped samples (Display Formula\(R = {1 \over N} \cdot \mathop \sum \limits_i {\bar R_i}\)). For example, considering the saccades depicted in Figure 3, the bootstrap procedure might give several subsets of the red dots (test trials), which will then be compared to the regression model built from all blue dots (training trials) to obtain one average residual per subset. 

Data analysis

The effect of prior information (see previous section) on eye velocity or saccade amplitude was analyzed by comparing trials with the same target velocity but different priors of target velocity. For each training target velocity there was only one corresponding test target velocity condition (test at 20°/s – 5°/s for a training at 15°/s, 15°/s + 5°/s for a training at 20°/s). For each of those two conditions separately, we compared eye movement measures using a repeated measures analysis of variance (rANOVA), using the trial number (four levels) and the type of trial (training or test) as within-subjects factors. 

The effect of visual information on eye velocity or saccades was analyzed by comparing trials with the same target velocity during the previous trials (15°/s or 20°/s) but with different target velocities in the current trial (three levels: −5°/s, 0°/s, +5°/s with respect to target velocity during the previous trials, which correspond to three types of trial: test −5°/s, training, and test +5°/s). Eye movement measures were compared using repeated measures ANOVA with the following within-subject factors: target velocity in previous trials (two levels), type of trial (three levels), and trial number (four levels). 

When comparing data across the two types of target to examine differences in the magnitudes of the effects, we used a mixed-design ANOVA with the type of target as between-subjects factor. Comparisons on the effect of the prior were made for the two velocity conditions (after normalization with training data), and included the trial number as within-subject factor. The comparison was also made with the velocity condition as an additional within-subject factor. Comparisons on the effect of visual information were made with target velocity in previous trials, type of trial and trial number as within-subject factors. 

We analyzed the influence of the number of trials on the magnitude of the effects using linear regressions on the means per participant and per trial number. 

ANOVAs and linear regressions were performed with the R software (R Core team 2016, RRID:SCR_001905; ez package 2015). Sphericity assumptions were verified through Mauchly's Test for Sphericity. If sphericity assumptions were violated, we only reported results that were significant after Huynh-Feldt sphericity correction. When appropriate, we also reported the generalized eta-squared (ges) as a measure of effect size (Bakeman, 2005). 

Results

In this study, we varied the reliability of sensory and prior information independently in order to see how each of these information channels can influence the oculomotor response. 

Our first hypothesis was that the noisy (Gaussian) target would be harder to track for the participants. This was confirmed by measures of the pursuit onset (respectively to target onset) in the second trial (mean of 145.5 ms for the noisy target, 87.4 ms for the standard target; F(1, 24) = 18.37, p = 0.0003, ges = 0.38), and lower visual smooth pursuit gains overall (0.75 for the noisy target, 0.89 for the standard target; F(1, 24) = 11.98, p = 0.0026, ges = 0.31). These overall differences are not expected to affect further results, which are always relative to training trials done with the same target. 

From this, we expected to observe a hallmark of Bayesian integration on measures of eye movements: dynamic weighting of the effects of prior and sensory information as a function of their reliability. More precisely, we expected to see stronger effects of prior information—and weaker effects of visual information—with the noisy target, thus highlighting a weighting dependent on visual reliability. At the same time, we also expected a gradual increase of the effect of prior information with each trial repetition, highlighting the increase of the reliability of prior information. 

Prior information about the target velocity biases smooth pursuit eye velocity

We measured the effect of prior information on smooth pursuit by comparing eye velocity for a selection of test and training trials (Figure 1B). The selected trials had identical target velocities (for example 15°/s) during the current trial but different target velocities during the previous ones (for example 15°/s for training and 20°/s for test trials). In addition, these trials were matched by trial number. Therefore, any difference in behavior between these test and training trials has to be attributed to the influence of prior information on the oculomotor response. This comparison is illustrated on Figure 4A, which shows average eye velocity profiles (noisy target) for training and test trials for different trial numbers. On this figure, a clear and long-lasting bias (up to 500 ms after the target motion onset) can be seen. Furthermore, this bias appears to increase with the number of previous trials, which is further discussed in the last part of the results section. 

To evaluate the influence of the prior information on the smooth pursuit response, we computed the steady-state smooth pursuit gain for each trial (see Methods) and subtracted the gain of the training trial from the gain of the corresponding test trial (Figure 4B and C). For example, the test data from Panel A corresponds to the solid red error bar in Panel B. For all types of test trials, the smooth pursuit response was biased towards the velocity of the preceding trials (main effect of trial type; noisy target, test15: F(1, 12) = 51.81, p < 0.0001, ges = 0.17, test20: F(1, 12) = 53.11, p < 0.0001, ges = 0.1; standard target, test15: F(1, 12) = 21.38, p = 0.0006, ges = 0.07, test20: F(1, 12) = 24.95, p = 0.0031, ges = 0.06). That is, on a given trial, the steady state smooth pursuit gain was higher (respectively lower) if this trial was preceded by trials with higher (respectively lower) target velocity. This demonstrates the effect of prior information on smooth pursuit eye velocity. This effect was true for both target types. 

Visual information about the target velocity affects smooth pursuit eye velocity

The impact of visual information on the smooth pursuit eye movements was measured by comparing test and training trials that had the same prior information (same number of previous trials with the same target velocity), but different target velocities (see Figure 1B). 

Looking at eye velocity, this comparison (see Figure 5A) showed that visual information had a clear effect: Conditions having a higher target velocity showed, when compared to training conditions, a bias towards higher eye velocities and conditions having a lower target velocity showed a bias towards lower eye velocities. This effect was already present during the first active trial (Trial 2), persisted during later trials, and could be observed as late as 500 ms after the target onset. 

To highlight the effect of visual information on smooth pursuit eye velocity, we computed the difference of pursuit gains (see Methods) to obtain the change in pursuit gain between test and training trials. The differential gains show that the visual information modulated the smooth pursuit gain for all conditions of each experiment (Figure 5B and C), which was confirmed by statistical analyses. 

We found that the steady state eye velocity gains of test trials were significantly different from those of training trials, both for the noisy target (main effect of trial type: F(2, 24) = 37.03, p < 0.0001 Huynh-Feldt corrected, ges = 0.23) and the standard target (main effect of trial type: F(2, 24) = 124.29, p < 0.0001 Huynh-Feldt corrected, ges = 0.53). Note that the interaction with the trial number is discussed in the last section of the Results. 

For both types of target, the magnitudes of the biases were higher for the conditions for which prior target velocity was at 15°/s (interaction effect between trial type and prior target velocity; noisy target: F(2, 24) = 29.34, p < 0.0001 Huynh-Feldt corrected, ges = 0.03; standard target: F(2, 24) = 30.08, p < 0.0001 Huynh-Feldt corrected, ges = 0.05). Thus overall, visual information affected pursuit gain. 

Saccade amplitude is biased by prior information about the target velocity

It is known that saccade and pursuit share many inputs. In particular, catch-up saccade amplitude is dependent on the difference between target and eye velocity (de Brouwer, Missal, Barnes, & Lefèvre, 2002). Therefore, if the internal representation of target velocity is biased by prior information, we would expect that this bias also results in alterations of catch-up saccade amplitude. Thus, we tested if the prior information about the target motion would also influence the catch-up saccades. To do so, we compared the amplitude of saccades in test and training trials by computing the difference between the amplitude of saccades made during test trials and the amplitude of saccades made during training trials of the same target velocity (see Methods and Figure 3). 

We found that saccades were biased by prior information about target velocity; the amplitude of saccades made during test trials was more likely to be larger than during training trials if the previous trials had a higher target velocity and to be smaller if the previous trial target was slower (main effect of trial type; noisy target, test15: F(1, 12) = 34.47, p < 0.0001, ges = 0.28, test20: F(1, 12) = 18.62, p = 0.001, ges = 0.18). When the standard target was presented on the screen, we found a similar effect of the prior on saccade amplitude (standard target, test15: F(1, 12) = 6.82, p = 0.023, ges = 0.02, test20: F(1, 12) = 17.1, p = 0.0014, ges = 0.16). Figure 6A illustrates this effect on some representative trials from one participant. Another way to look at this effect is presented in Figure 3, where it can be observed that the data points from test trials—the red dots—tend to be located above the regression line, indicating a bias towards larger saccades coherent with an effect of a higher target velocity prior. 

Figure 6

 

(A) Effect of prior information on concomitant saccades. Position and velocity profiles of saccades made at the same time (15-ms window) in representative training and test trials. Left panel: Saccades made around 220 ms after the onset of the target, during training trials with a prior of target velocity at 15°/s (blue traces) and test trials with a prior of target velocity at 20°/s (red traces). Right panel: Saccades made around 220 ms after the onset of the target, during training trials with a prior of 20°/s (blue traces) and test trials with a prior of 15°/s (green traces). (B) Effect of prior information on saccades (noisy target). Averages of participant's average residuals (vertical distance from fit on training data) of the amplitude of saccades made during test trials, with respect to the trial number. The blue dotted line indicates the training reference (that has been subtracted from all data), red/green traces show residuals of saccades made during test trials. The error bars indicate the standard error of the mean. (C) Same as Panel (B), but for the second experiment with a standard target.

Because saccades can sometimes be adjusted online via internal feedback, we also studied the peak velocity of the saccades, as it is a good marker of saccade planning (Chen-Harris, Joiner, Ethier, Zee, & Shadmehr, 2008). Similarly to saccade amplitude, we also found peak velocity to be biased towards prior target velocity. Peak velocity tended to be higher, compared to training trials, during test trials with higher prior target velocity, while it tended to be lower when prior target velocity was lower (main effect of trial type; noisy target, test15: F(1, 12) = 24.05, p = 0.0004, ges = 0.26, test20: F(1, 12) = 12.95, p = 0.0037, ges = 0.10; standard target, test15: F(1, 12) = 5.36, p = 0.04, ges = 0.05, test20: F(1, 12) = 14.5, p = 0.0025, ges = 0.17). 

Saccade amplitude is also biased by visual information about the target velocity

As a sanity check for our analysis of saccades, we compared saccades made within conditions having the same prior information and different visual information about the target velocity. As expected, saccade amplitude was also influenced by visual information (Figure 7), meaning that catch-up saccades accounted for the velocity change. Test trials with a higher target velocity than training trials (green traces) had larger saccade amplitude, and conversely, test trials having a lower target velocity (red traces) than training trials exhibited smaller saccades. Once more, this effect was present for all velocities, and in both experiments (main effect of trial type, noisy target: F(2, 24) = 69.39, p < 0.0001 Huynh-Feldt corrected, ges = 0.4; standard target: F(2, 24) = 211.96, p < 0.0001 Huynh-Feldt corrected, ges = 0.7). We found no influence of the training (previous) target velocities (15°/s and 20°/s) on the magnitude of the effect (cf. dashed lines vs. full lines in Figure 7). 

Figure 7

 

(A) Effect of visual information on saccades. Averages of participant's mean residuals (vertical distance from fit on training data) of the amplitude of saccades made during test and training trials, with respect to their trial number. Red/green traces show residuals of saccades made during test trials (the training residuals have been subtracted from all data). The x-axis indicates the trial number and the error bars the standard error of the mean. (B) Same as Panel (A), but for the second experiment with a standard target.

Lower visual information reliability induces stronger prior information effect

By comparing the normalized effect of prior information across the two types of target (Figure 8), we found that the influence of the prior was significantly stronger for the noisy target than for the standard target. As such, the magnitude of the effect on eye velocity was significantly different between the two targets (main effect of target type: F(1, 24) = 4.43, p = 0.046, ges = 0.08). Furthermore, we observed significant differences of the modulation of saccade amplitudes by prior information (main effect of target type: F(1, 24) = 7.17, p = 0.0132, ges = 0.08). In other words, when confronted to a noisy, less reliable target, participants gave more weight to prior information than when confronted to the standard target. This effect of prior information also appeared to vary depending on the participants, as shown by the vertical spread of the individual averages of the effects. 

Figure 8

 

Magnitude of the effect of prior information on eye velocity (left panel, % is relative to the target velocity change: 5°/s, cf. Equation 1) and on saccade amplitudes (right panel). Dots are participant's averages across velocities. Error bars are centered on the averages of participant's averages across both comparable velocities and indicate standard errors of the mean. Significance (* indicates p < 0.05) refers to a mixed design ANOVA with within-factors “target velocity” and “trial#” and between-factors “type of target”.

Higher visual information reliability induces stronger visual information effect

As a complement to the previous analysis, comparing the two types of target (Figure 9), we found that the standard target elicited a stronger effect of the visual information on both smooth pursuit and saccadic eye movements. Indeed, the effect on eye velocity was stronger (main effect of target type; F(1, 24) = 20.77, p = 0.0001, ges = 0.26), and saccade amplitude was more strongly modulated (main effect of target type; F(1, 24) = 8.64, p = 0.007, ges = 0.1) when the target was standard rather than noisy. 

Figure 9

 

Magnitude of the effect of visual information on the eye velocity (left panel, % is relative to the target velocity change: 5°/s, cf. Equation 2), and on saccade amplitude (right panel). Dots are participant's averages across comparable velocities. Error bars are centered on averages of participant's averages across velocities and indicate standard errors of the mean. Significance (* indicates p < 0.05) refers to a mixed design ANOVA with within-factors “training target velocity” and “trial#”, and between-factors “type of target”.

This behavior, opposite to that of prior information, is coherent with the hypothesis that the reliability of visual information is higher for the standard target than for the noisy target, which would give more weight to visual information when pursuing the standard target. Again, this effect appeared to vary depending on the participants, as can be seen in the spread of the individual averages of the effects. 

Higher prior reliability induces stronger effect of the prior, but doesn't seem to affect visual effect

Postulating that the reliability of the prior information increases with the number of repetitions (Orban de Xivry et al., 2013), we analyzed how the effect of prior information evolves with trial number. To do so, we computed a simple linear regression on normalized measures of the smooth pursuit gain (see methods) to predict the effect magnitude based on the trial number. 

Across target types, we found that the magnitude of the effect of the prior on the eye velocity gain significantly increased with the trial number (significant regression slope of 4.9%/trial: F(1, 206) = 11.39, p = 0.0009), supporting the hypothesis that the weighting of prior information is dynamically updated to reflect its reliability (Figure 10A). We also computed the linear regression separately for each target type, and observed that for both targets there was a significant increase of the effect with the trial number (noisy target: slope of 4.7%/trial, F(1, 102) = 5.1, p = 0.026; standard target: slope of 5.2%/trial, F(1, 102) = 6.76, p = 0.011). 

Figure 10

 

Effect of trial number. (A) Influence of trial number on the magnitude of the effect of prior information on eye velocity (averages of participant's averages). (B) Influence of trial number on the magnitude of the effect of visual information on eye velocity (averages of participant's averages). Percentages are relative to the target velocity change (% of 5°/s). Error bars indicate the standard error of the mean.

In contrast, we found no indication (p > 0.39) of an effect of the number of repetitions on the impact of visual information on eye velocity (Figure 10B). 

Note that we could not perform the analysis of interaction of the number of repetitions on saccades because, since we had to match saccade latency distributions to compare saccades across targets (experiments), the number of comparable saccades was reduced by almost 50%. Consequently, the number of comparable saccades per (test) trial number was too low for such an analysis. 

Discussion

To investigate the dynamic reliability-based weighting of visual information and memory of target motion during eye movements, we asked participants to visually track either a noisy (Gaussian) target or a standard target for a variable number of identical trials. Then, we tested how the nervous system weighted new sensory information and prior expectations by presenting a trial with a different target velocity. 

Our results show that previous trials had a greater effect on eye velocity and catch-up saccades amplitude when the target was a Gaussian blob, in a manner consistent with a reliability-weighted integration of visual information with short-term memory. It was also observed that the effects of previous trials on eye movements appeared as early as after one trial and increased with the number of trials, hinting at a short-term memory of perceived target motion that is dynamically built and updated. 

Eye movements use a reliability-based representation of target motion

It has been known for a long time that extraretinal and sensory signals can both drive smooth pursuit (Barnes & Asselman, 1991; Dodge et al., 1930; Kowler & Steinman, 1979); however, how these two signals are integrated within the pursuit system has remained elusive. The first models including predictive smooth pursuit had a separate predictive pathway, activated by switches (Barnes, 2008). Such models often struggled to explain the transition between predictive and reactive pursuit and why predictive signals are observed despite the randomization of target features (Kao & Morrow, 1994; Kowler & McKee, 1987). 

In recent years, new models of smooth pursuit able to combine sensory and extraretinal signals without the need for a switch between pathways have appeared. The models of Orban de Xivry et al. (2013) as well as Bogadhi, Montagnini, and Masson (2013) both included two separate recursive Bayesian/Kalman filters (one for visual inputs, one for extraretinal inputs) and could reproduce many aspects of smooth pursuit behavior in different contexts of visual information. They differed in a few aspects, including the type of memory of the target and assumptions about the visual reliability of the target. The recent model of Darlington, Tokiyama, and Lisberger (2017) took a different approach, in which a prior (extraretinal input) directly enhances the gain of the visual-motor transmission of sensory inputs of similar direction and velocity, hinting at a role of the smooth eye movement region of the frontal eye fields (FEF_SEM) in the representation of the prior. 

Here, we mainly refer to the model of Orban de Xivry et al. (2013) because we can compare its predictions of the repeated tracking of a noisy target to our experiments. With a few modifications, we believe that similar predictions could be made by the model of Bogadhi et al. (2013). The model of Darlington et al. (2017) can predict the effect of prior on smooth pursuit without modification. However, its implementation of priors is oriented towards smooth pursuit gain modulation, and would need to be extended to saccades to explain the effect of prior on saccade amplitude that we observed. In contrast, the models of Orban de Xivry et al. (2013) and of Bogadhi et al. (2013) are less specific and could explain this effect through the influence of the prior on their prediction of the retinal slip, which is known to be taken into account by the saccadic system to produce catch-up saccades (Daye, Blohm, & Lefèvre, 2014; de Brouwer, Yüksel, Blohm, Missal, & Lefèvre, 2002). 

Compared to the predictions of the model of Orban de Xivry et al. (2013), we found that our results supported them. Indeed, the modulation of the effect of previous trials on eye velocity by both the reliability of visual information (noisy or standard) and the reliability of the memory (number of repetitions) are strong indicators of a reliability-based integration of visual information with a memory of target motion. Furthermore, these results reproduce, with humans, the results of Darlington et al. (2017) regarding eye velocity and highlight the gradual increase of the reliability of prior information. 

Importantly, our results go further than these studies by showing that similar processes are at work for catch-up saccades, which fits well with the fact that saccades and smooth pursuit are influenced by common inputs (Krauzlis, 2004; Krauzlis, 2005; Orban de Xivry & Lefèvre, 2007). 

While these common inputs have been suggested to be sensory signals from position and motion pathways (Blohm, Missal, & Lefèvre, 2005; de Brouwer et al., 2002; Krauzlis, 2004), or outputs from the forward model (Ego, Yüksel, Orban de Xivry, & Lefèvre, 2016; Orban de Xivry, Bennett, Lefèvre, & Barnes, 2006), we show here that an internal representation of target motion, stored in short-term memory, is also shared by the two systems. Hence, catch-up saccade amplitude was affected both by the target velocity of previous trials and by the velocity of the ongoing target. Moreover, the magnitude of the effect of previous trials was greater when the visual target was the (less reliable) Gaussian blob, which is fully compatible with reliability-based integration and shows that prior information influences gaze planning upstream from the separation between saccadic and smooth movement pathways. 

Therefore, we show in the present study that the whole oculomotor system has access to a single short term memory of target motion, and that reliability-based integration captures well the process by which this memory is combined with sensory signals. 

Nature of the representation: Short-term memory or prior

Here, we made the assumption that the extraretinal signals consist of an internal representation of target motion (Orban de Xivry, Missal, & Lefèvre, 2008) and not a prior on target velocity (Bennett & Barnes, 2004; Bogadhi et al., 2013; Yang et al., 2012). These two possibilities differ in the sense that an internal representation of target motion is a memory of the target motion paired to a measure of uncertainty while a prior is a Gaussian distribution with a mean and standard deviation (see Tassinari et al., 2006, for a study on the ability of humans to do reliability-based integration). 

The rapid change in the weight of the extraretinal signals with trial number (Figure 10) first appears to be at odds with a prior on target velocity, which is commonly slowly built and gradually updated. However, while many trials have usually been required to obtain a prior distribution (Yang et al., 2012: several days of training in monkeys; Kording & Wolpert, 2004: 1,000 trials in humans), there is now more and more literature describing shorter timescales for the building of certain sensorimotor priors—for example of target position: In 2008, Izawa and Shadmehr suggested continuous prior integration with sensory information; in 2011, Verstynen and Sabes reported on “fast adapting priors” built within 10 trials; in 2012, Rao, De Angelis, and Snyder suggested rapidly varying priors downstream of the sensory representation. Priors of target velocity and timing were also suggested to be continuously integrated with other sensorimotor signals by authors such as Heinen, Badler, and Ting (2005), and it can be argued that this trend has culminated with the rapidly adapting, one-trial priors proposed by Darlington, Tokiyama, and Lisberger, (2017). 

At this point, the gap between these two concepts of extraretinal signals is narrowing. Still, an internal representation of target motion can also drive anticipatory pursuit (Orban de Xivry et al., 2013), and there is previous evidence in favor of an internal representation of target motion (Bennett et al., 2010; Orban de Xivry et al., 2008). 

Although an internal representation of target motion is assumed in this study, it should be noted that the protocol used cannot differentiate between the two, and that our conclusions could be reframed in the context of priors on target velocity. 

Reliability-based integration of a short-term memory with new information

In recent years, reliability-based integration of sensory inputs with other signals has provided an elegant framework to explain how the brain can process noisy and changing visual information to produce appropriate movements. Reliability-based integration has already been observed for inputs from different sensory systems (Ernst & Banks, 2002; Landy et al., 2011), for visual position information and ongoing motor commands (Kording & Wolpert, 2004), for visual motion information and a low-velocity prior (Jogan & Stocker, 2015; Stocker & Simoncelli, 2006), and for sensory predictions and sensory feedback (Vaziri, Diedrichsen, & Shadmehr, 2006). However, there have been few studies about the reliability-based integration of a short-term memory with new information in a sensorimotor context (Brouwer & Knill, 2007, 2009; Darlington et al., 2017). Of those, most, including ours, suggest that the brain is continuously integrating and updating this memory with new information. Nevertheless, there is a need for more research to better explain how a memory of target motion can be created and continuously updated in the broad range of contexts encountered in sensorimotor behaviors, including smooth pursuit, saccades and reaching movements. 

Outside of the oculomotor context, several authors have studied the dynamics of memory and learning and found processes that relate to reliability-based integration of working memory with new information. They have suggested different models for which a memory is updated on the basis of the outcome of a previous action. The core principle is as follows: The brain maintains states/beliefs about the environment (in our case the target motion), and updates the existing beliefs on the basis of prior information and recent outcomes for the next trials. In addition, these models particularly focus on whether a new state (a memory) should be formed or the previous one updated, depending on whether a fundamental change in the system is thought to have occurred or not. The states are therefore updated or created depending on the probability of a fundamental change and on the reliability of the signals, namely memory or new sensory information, for example by means of an approximately Bayesian delta-rule model (Nassar et al., 2010; Wilson, Nassar, & Gold, 2013), or a similar optimal filtering model (Gershman et al., 2014). Such belief-updating models have also been applied to the motor adaptation domain (Kording, Tenenbaum, & Shadmehr, 2007; Wei & Kording, 2010). 

In all these studies, memory updating is influenced by the reliability of the new incoming information but could entirely take place during the intertrial interval. In contrast, the present study and the study of Darlington et al. (2017) demonstrate that the integration of the memory content and the sensory information takes place during the movement (when visual information about target velocity is available) and directly influences it. 

Integration of short-term memory and visual signals in other contexts

Several studies investigated the influence of the memory of a target position on reaching arm movements. Brouwer and Knill (2007, 2009) showed that reaching movements toward a visual target were biased towards its last known position and that the effect was stronger if the contrast of the target was low. Verstynen and Sabes (2011) also showed the presence of a bias toward previous positions of the target during reaching movements despite the fully predicable nature of the next positions. In addition to the position of the target, the spatial structure of the environment itself has also been shown to be memorized during visuomotor tasks (Aivar, Hayhoe, Chizk, & Mruczek, 2005; Hayhoe, Shrivastava, Mruczek, & Pelz, 2003). 

Those studies clearly highlight that the reliability of a visual target, and previous information about it, can affect hand movements in a way that is similar to their effect on eye movements (Issen & Knill, 2012). However, these studies mainly focused on the integration of a memory of target position with the visually presented one. Such memories are limited to a position signal and a spatial representation of the scene and do not contain the evolution of the signal over time. Given that we know that the oculomotor system does not restrict this memory to a measure of target velocity but also includes its time course (Bennett et al., 2010; Orban de Xivry et al., 2008), we believe that the content of the working memory used for reliability-based integration during movement is much more complex than the ones previously described (Song & Nakayama, 2009). 

Finally, while the integration of the position signals from visual information and memory occurs mostly before movement onset, we observe here a continuous integration of working memory and sensory information during eye movements. Given the similarities between the oculomotor system and other sensorimotor systems (Lisberger, 2015; Lynch & Tian, 2006), we may expect to find a similar process of continuous short-term memory updating in those systems. 

Potential for further investigations of this short-term memory

In our protocol, only one test trial is shown per block and always concludes it. While appropriate for our purposes, several variations could answer further questions about the characteristics of the short-term memory. Future studies should explore the effect this change in target velocity has on the memory—for example, through protocols that do not stop at the test trial but show additional trials after. The target velocity of these trials might then go back to training trials velocities, or simply repeat the test trial velocity. 

In all those potential protocols, we believe that the relative reliabilities of the short-term memory and the visual information will be paramount. Based on the model of Orban de Xivry et al. (2013), we would expect that the effect a change in target velocity has on subsequent trials will be negatively correlated with the reliability of the short-term memory at the first test trial and positively correlated with the reliability of the visual information. As such, we expect that a single test trial inserted in a sequence of seven training trials will induce stronger perturbations in subsequent trials if it is inserted in Trial 2, rather than in Trial 6 when the reliability of the short-term memory has been strengthened by observing more trials. 

Conclusion

In this study, we report experimental evidence in the context of human oculomotor behaviors, in both smooth pursuit and catch-up saccades, that short-term memory can be quickly built, constantly updated, and continuously integrated in a reliability-based manner with incoming visual information. We believe that this constitutes a general principle of dynamical updating of working memory, one that is consistent with recent studies (Darlington et al., 2017; Gershman et al., 2014; Nassar et al., 2010), and that is likely to be present in other sensorimotor systems (Lisberger, 2015). 

Acknowledgments

Support for N. Deravet and P. Lefèvre was provided by: the Belgian Federal Science Policy Office, IAP VII/19 DYSCO; the European Space Agency (ESA), PRODEX C90232; and the Communauté française de Belgique, ARC 13/18-053. Support for J.-J. Orban de Xivry was provided by: Katholieke Universiteit (KU Leuven), STG/14/054 and the Fonds Wetenschappelijk Onderzoek (FWO), 1519916N. Support for G. Blohm was provided by the Gouvernement du Canada | Natural Sciences and Engineering Research Council of Canada (NSERC). The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication. 

Commercial relationships: none. 

Corresponding author: Nicolas Deravet. 

Email: . 

Address: Institute of Information and Communication Technologies, Electronics, and Applied Mathematics and Institute of Neuroscience, Université catholique de Louvain, B-1348 Louvain-La-Neuve, Belgium.