Dynamic interaction between retinal and extraretinal signals in motion integration for smooth pursuit

Abstract

Abstract

Abstract:

Abstract  Due to the aperture problem, the initial direction of tracking responses to a translating bar is biased towards the direction orthogonal to the bar. This observation offers a powerful way to explore the interactions between retinal and extraretinal signals in controlling our actions. We conducted two experiments to probe these interactions by briefly (200 and 400 ms) blanking the moving target (45° or 135° tilted bar) during steady state (Experiment 1) and at different moments during the early phase of pursuit (Experiment 2). In Experiment 1, we found a marginal but statistically significant directional bias on target reappearance for all subjects in at least one blank condition (200 or 400 ms). In Experiment 2, no systematic significant directional bias was observed at target reappearance after a blank. These results suggest that the weighting of retinal and extraretinal signals is dynamically modulated during the different phases of pursuit. Based on our previous theoretical work on motion integration, we propose a new closed-loop two-stage recurrent Bayesian model where retinal and extraretinal signals are dynamically weighted based on their respective reliabilities and combined to compute the visuomotor drive. With a single free parameter, the model reproduces many aspects of smooth pursuit observed across subjects during and immediately after target blanking. It provides a new theoretical framework to understand how different signals are dynamically combined based on their relative reliability to adaptively control our actions. Overall, the model and behavioral results suggest that human subjects rely more strongly on prediction during the early phase than in the steady state phase of pursuit.

Introduction

Tracking eye movements use visual motion information to stabilize the image of an object of interest onto the retina so that its properties can be scrutinized with high-resolution spatial vision mechanisms (Rashbass, 1961). To control such movements, the brain must therefore compute a reliable and dynamic velocity estimate that accurately describes the trajectory of the selected object independent of its shape, its own internal kinematics (e.g., movement of a bird in a flock of birds), or some large variations in the retinal input due to sudden changes in illumination levels or the presence of occluding surfaces. In primates, the near perfect pursuit behavior relies on a complex visual motion processing using not only the different visual cues available from the retinal image but also the internal representations of objects properties and trajectories (see Ilg, 2002; Krauzlis & Stone, 1999; Masson & Perrinet, 2012, for reviews). However, the effect of uncertainty in speed and direction estimates on the dynamics of pursuit eye movements is not yet fully understood (Bogadhi, Montagnini, Mamassian, Perrinet, & Masson, 2011; Osborne, Lisberger, & Bialek, 2005). 

The time course of tracking responses carries the signatures of both visual motion processing and the subsequent visuomotor transformation stages (see Lisberger, 2010, for a review). For instance, because of the aperture problem faced by the early visual motion processing stage, the initial direction of tracking responses to a tilted bar stimulus moving horizontally is largely driven by the ambiguous unidimensional local motion signals. As a consequence, tracking responses are biased in the direction orthogonal to the orientation of the bar (Born, Pack, Ponce, & Yi, 2006; Masson & Stone 2002; Montagnini, Spering, & Masson, 2006b; Wallace, Stone, & Masson, 2005), similar to the perceived direction of motion. This directional bias is reduced in the subsequent 200 ms of pursuit, so that pursuit is fully aligned with the two-dimensional target motion when reaching steady state tracking. This pattern of temporal dynamics is consistent (after some temporal filtering) with that of the neural solution to the aperture problem (Masson, Montagnini, & Ilg, 2010; Pack & Born, 2001). However, accurate tracking is a product of both sensory evidence processing and internal representations about target motion. For instance, solving the aperture problem is best understood as the dynamical solution of an inference problem where sensory evidence and prior knowledge about smooth visual motion are recurrently integrated (Bogadhi et al., 2011; Montagnini, Mamassian, Perrinet, Castet, & Masson, 2007; Weiss, Simoncelli, & Adelson, 2002). 

A transient disappearance of the target during pursuit results in a drop in the eye velocity for about 250 ms before stabilizing at about 70% of preblanking target velocity throughout the blanking period (Becker & Fuchs, 1985; Mitrani & Dimitrov, 1978; Pola & Wyatt, 1997). In the absence of retinal image motion during steady state tracking, it has been proposed that extraretinal signals drive pursuit. The role of an internal positive feedback during steady state tracking is a feature of several oculomotor models (Churchland, Chou, & Lisberger, 2003; Krauzlis & Lisberger, 1994; Krauzlis & Miles, 1996b; Robinson, Gordon, & Gordon, 1986). However, the nature of the drive for the eye velocity in the absence of a visual target is still debated (see Kowler, 2011, for a review). Two different mechanisms have been proposed. An efference copy of the eye velocity command could act through a positive feedback loop (Churchland et al., 2003) that might depend on the behavioral context (Madelain & Krauzlis, 2003). Alternatively, pursuit could be driven by a leaky working memory signal fed by a static sample eye velocity command at the time of occlusion (Bennett & Barnes, 2004) or a dynamical representation of the target trajectory (Orban de Xivry, Missal, & Lefèvre, 2008). 

The two above mentioned examples clearly indicate that the interaction between retinal and extraretinal signals determines pursuit both under full vision and during occlusion epochs. A comprehensive model of primate smooth pursuit shall accommodate variable weights associated with both visual inputs and internal signals, and these relative weights must be dynamically modulated depending on the behavioral context. The focus of this study is on those dynamical interactions. Our previous work has indicated that the aperture problem can be used to probe the relative weighting of retinal and extraretinal signals (see Masson et al., 2010, for a review). For instance, during the fixation period before target motion onset, predictive signals related to the two-dimensional target motion can fully explain the unbiased anticipatory pursuit. However, these predictive cues seem to be disregarded as soon as the visual ambiguous input is available (Montagnini et al., 2006b). In addition, in contrast to the robustness of the aperture bias at pursuit initiation, the impact of reintroducing the aperture problem during steady state pursuit (for instance by changing target orientation or transiently hiding the target) is highly dependent upon the visual context and the predictability of target motion properties both in human and nonhuman primates (Masson, Fleuriet, Montagnini, & Mamassian, 2008; Masson & Stone, 2002). 

In the present study, we investigated how extraretinal and retinal signals interact using a translating tilted bar as the pursuit target and blanking it at different times during the pursuit. When the stimulus reappears there is a finite retinal image slip, which could result in the recomputation of local motion and thus again in the aperture problem with a consequent direction bias in the tracking responses. Rotating the target orientation by 90° during the blank would lead to a bias in a different direction (with opposite vertical component). If retinal signals dominate the control of smooth pursuit, the aperture-based pursuit bias shall thus be seen at target reappearance. On the contrary, if pursuit is entirely dominated by an internal representation of target motion (or eye motion) no bias shall be observed at target reappearance. Thus, it becomes possible to titrate the relative contribution of retinal and extraretinal signals during pursuit from the dynamics of its horizontal and vertical components. To fully account for such interactions, we need to consider the different phases of pursuit eye movements. It is known that the dynamics of the extraretinal drive during a transient blank is different in the early (Barnes & Collins, 2008b) and steady state phases of pursuit (Becker & Fuchs, 1985; Bennett & Barnes, 2003). We conducted two experiments where the tilted bar stimulus was blanked during the steady state of pursuit (Experiment 1) or during the early stage (blank starting in the first 100 ms) of pursuit (Experiment 2). We will show that the dynamics of the interactions between retinal and extraretinal signals change with the time at which the stimulus is blanked relative to pursuit onset. We report the counterintuitive observation that extraretinal signals dominate retinal signals during blanking in the early phase of pursuit, but that retinal signals dominate when transient blanking occurs during steady state pursuit. We further tested this observation by simulating a hierarchical inference model implementing an optimal cue combination rule to combine retinal and extraretinal signals for driving pursuit. 

Integrating different signals and weighting them based on their respective levels of uncertainty is a classical problem of motor control in artificial (Kalman, 1960) and biological (Fetsch, Pouget, DeAngelis, & Angelaki, 2011) systems. We have previously shown that a recurrent Bayesian inference framework can explain the temporal dynamics of motion integration for open-loop smooth pursuit (Bogadhi et al., 2011; Montagnini et al., 2007). Now, we extend our previous open-loop recurrent Bayesian model to a closed-loop two-stage recurrent Bayesian model. This model can explain the dynamic integration of retinal and extraretinal signals for pursuit control using an optimal cue combination rule whereby each signal is weighted by a coefficient proportional to the inverse variance of the signal itself. The model is simulated for different levels of prediction (high and low) in both steady state and early pursuit blanking conditions. Importantly, with a single set of parameters, the model captures both the drop of horizontal velocity during blank and the transient component of vertical eye velocity at target reappearance. With the help of the model-based simulations, we confirm the observation that extraretinal signals must dominate retinal signals during blanking in the early phase of pursuit (consistent with a high-prediction mode) and that retinal signals dominate during blanking in the steady state of pursuit (consistent with a low-prediction mode). This study opens the door to a new way of thinking about the interactions between retinal and extraretinal signals for eye movements, within the framework of Bayesian inference and dynamic optimal cue integration. 

Methods

Subjects

A total of six subjects participated in the study (mean age 31.5 years; SD 7.06). Three of them were completely naive to the present study as well as to eye movement research in general. All subjects were healthy and had normal or corrected to normal vision and had no relevant medical and psychiatric history. The experiments were conducted in accordance with Centre National de la Recherche Scientifique (CNRS) ethical regulations for behavioral research. All subjects participated after having given an informed consent. 

Visual stimulus and apparatus

The visual stimulus used in the experiments described below was a 17° long and 0.126° wide tilted bar with different orientations. All stimuli were presented on a 21 in. CRT monitor at refresh rate of 100 Hz against a gray background. The luminance of the gray background was 25 cd/m² and that of the white stimulus was 68 cd/m². The spatial resolution of the screen was set to 1280 (H) × 1024 (V) pixels. 

Experimental design

We conducted two experiments to investigate the interactions between retinal and extraretinal signals in motion integration for smooth pursuit at different stages of pursuit. The main difference between the two experiments was the time of blanking onset, which was either in the steady state (Experiment 1) or in the early stage (first 100 ms) of pursuit. Details of each experiment are described below. 

Experiment 1: Blanking during the steady state phase of pursuit

The moving stimulus was either a 45° or a 135° tilted bar translating horizontally to right or left at a constant velocity of 8.4°/s. In 4/6 of the trials, the stimulus was blanked for 200 ms or 400 ms, starting 600 ms after stimulus onset. Thus, blanking occurred during the steady state of pursuit. In half of the blanking trials, the target reappeared with a 90° change of orientation after blanking. In half of the no blanking trials, an instantaneous 90° change of orientation occurred at 600 ms after stimulus onset. Responses were compared to a control condition, where bar orientation remained constant during the whole stimulus duration. Duration of all trials was constant at 1500 ms and all conditions were randomly interleaved. Fifty trials per condition were recorded. The pursuit responses to instantaneous 90° change of orientation condition are not included in further analysis. The reason to exclude this condition with no blank from our analysis is that it doesn't provide an independent assessment of the strength of the extraretinal signal as in the blank condition. Thus, in order to compare the integration of different motion cues in similar conditions, we included only the blanking conditions in the subsequent analyses. 

Experiment 2: Blanking during the early phase of pursuit

Stimuli were either a 45° or a 135° tilted bar translating horizontally to the right or left at a constant velocity of 8.4°/s. In 5/6 of the trials, the stimulus was blanked at various times starting at 100, 120, 140, 160, or 180 ms after target onset (and within 100 ms from pursuit onset on average). In the remaining trials the stimulus was not blanked and this was used as a control condition. All the conditions were randomly presented. This experiment was repeated for two blanking durations in different blocks. In one block, the duration of the blank was 400 ms and the total duration of the trial was 1100 ms. In the second block, the duration of the blank was 200 ms and the total duration of the trial was 900 ms. In all cases 50 trials per condition were recorded. Notice that beside the time of blank onset, the experimental conditions for Experiments 1 and 2 differ in another important aspect: The two blanking durations are randomly interleaved in Experiment 1, thus making the end of the blank unpredictable in time until the first duration (200 ms). In contrast, in all other conditions (longest blank duration in Experiment 1 and both duration condition in Experiment 2), the time of target reappearance was fully predictable. All quantitative comparisons (unless explicitly stated) were performed between equally predictable blanking conditions. 

Experimental protocol

Subjects were sitting in front of the monitor at a viewing distance of 43 cm with their head stabilized by chin and forehead rests. Each trial started with subjects fixating a dot (diameter 0.33°) at the center of the screen for a random duration of 400 to 600 ms. Then the fixation dot was turned off leaving a completely dark screen for 100 ms. This introduces a gap between fixation target offset and moving target onset. The tilted bar was presented at 1° of eccentricity and moved in the opposite direction similar to the step-ramp paradigm introduced by Rashbass (1961). Subjects were instructed to track the center of the bar and to maintain pursuit during the blank period since the target would always reappear at the end of it. 

Data recording and treatment

Eye movements were recorded using an Eyelink1000, an infrared video-based eye tracker (SR Research). Horizontal and vertical positions of the right eye were recorded at a sampling rate of 1 KHz. Eye position time series were low-pass filtered with a cutoff frequency of 50 Hz. The resultant eye position data were differentiated to obtain the velocity traces. The velocity traces were low-pass filtered with a cutoff frequency of 50 Hz to remove the noise from the numerical differentiation. The velocity traces were further smoothed using the csaps spline function in MATLAB with a spline coefficient of 0.0001. All the above-mentioned filtering and smoothing operations are equivalent to a Butterworth (acausal) filter of order two with cut-off frequency of about 40 Hz. During the visual inspection of single trials in MATLAB, we used an automatic conjoint acceleration and velocity threshold to detect and remove catch-up saccades (Krauzlis & Miles, 1996a). An objective method was used to compute the smooth pursuit latency in each trial and the method is based on the intersection between the two linear regression lines with a threshold criterion for slope increase (Krauzlis & Miles, 1996a; Masson & Castet, 2002). Oculomotor traces were aligned to stimulus onset. Outlier trials (less than 5%) were eliminated using an offline inspection. The outlier trials are those in which saccades could not be eliminated without excluding the majority of the trial or in which high levels of noise exist during fixation and persist during pursuit. 

Data analysis

For the quantitative analyses, smooth pursuit responses for different orientation and motion direction conditions (e.g., 45° oriented bar translating leftward or 135° oriented bar translating rightward) were flipped as needed and realigned so that all responses are comparable to the responses for 45° oriented bar translating rightward. This resulted in a total of ∼200 trials (i.e., ∼50 trials for each of the four conditions) for each blanking condition. Mean smooth pursuit responses (over 200 trials) to a translating bar briefly blanked for 200 ms starting at 600 ms are shown in Figure 1a. Horizontal and vertical components of eye velocity will be referred to as ė_h and ė_v, respectively. Figure 1a shows the different parameters that were extracted from the horizontal velocity profile on each trial for quantitative analysis. δt_drop corresponds to the duration for which eye velocity dropped, starting at ∼80 ms (mean latency of pursuit responses) after blank initiation and before reaching a plateau velocity during the blank. V_min denotes the minimum eye velocity reached during the blanking period (the plateau). V₈₀ is the velocity observed 80 ms after blank initiation and V_ext is the velocity measured 80 ms after the reappearance of the target (Figure 1a). δV_drop is the drop in the eye velocity because of the blank and is computed as the difference between V_min and V₈₀ i.e., (V_min − V₈₀). δV_ant is the anticipatory rise in the eye velocity before target reappearance and is calculated as the difference between V_ext and V_min i.e., (V_ext − V_min) as shown in Figure 1a. 

Figure 1

 

Different measures quantifying the dynamics of pursuit responses during and after a 200 ms target blanking. (a) A schematic representation of the different measures performed on the horizontal component of eye velocity (ė_h). V_min is the minimum eye velocity reached during the blanking period. V₈₀ is the velocity observed 80 ms after blank initiation and V_ext is the velocity measured 80 ms after the reappearance of the target. δV_drop is the drop in the eye velocity during the blank (V_min − V₈₀) and the corresponding time duration is δt_drop. δV_ant is the anticipatory rise in the eye velocity before target reappearance (V_ext − V_min). (b) Sliding average vertical eye velocity ( in black) computed to highlight change in vertical eye velocity (ė_v) on target reappearance (shown with black arrow) using a 140 ms wide sliding window initially centered at 210 ms on vertical eye velocity (ė_v, in blue).

Although the stimulus was translating to the right, because of the aperture problem the pursuit started in a direction biased towards the orthogonal to the orientation of the bar. This can be seen from the transient vertical eye velocity component in Figure 1a. This vertical eye velocity peaked at ∼210 ms after target motion onset and then started to decelerate towards zero for the next ∼150 ms. By this time the horizontal eye velocity reached target velocity as shown in Figure 1a. Throughout the present study, this vertical eye velocity component was used as a measure of the impact of the aperture problem upon the sensorimotor transformation and its dynamics during pursuit initiation is known to reflect the dynamics of motion integration (see Masson et al., 2010). Note that, with our experimental conditions, peak vertical eye velocity on individual trials was always rather small. In addition, vertical eye velocity following target reappearance was close to the level of velocity fluctuations on individual trials. In order to be able to measure any changes correlated with the aperture problem and its relatively slow dynamics after the blank, the visually driven vertical component of velocity has to be highlighted. For this purpose, a sliding window average of the vertical eye velocity component was computed using a 140 ms wide window sliding at steps of 20 ms and initially centered at ∼210 ms after blank offset, where vertical eye velocity peaks (see Figure 1b for a 200 ms blank ending at 800 ms). The moving average described above on the vertical eye velocity component is equivalent to a Butterworth (acausal) filter of order two and cut-off frequency of about 3.2 Hz. Figure1b also shows the slide average vertical eye velocity in black (note the magnified scale). The black arrow indicates the observed bias in the vertical component following target reappearance. 

Statistical analysis

We conducted an ANOVA on the different quantities (δt_drop, δV_drop, δV_ant) measured from the horizontal eye velocity component with the blank starting time as the factor in both open loop and steady state blanking experiments. We conducted a paired t test on the slide average vertical eye velocity component to measure the significance of its change upon target reappearance from the baseline during the blank. 

Results

Experiment 1: Target blanking during the steady state phase of pursuit

The mean horizontal (ė_h) and vertical (ė_v) eye velocity traces across different conditions are shown in Figure 2 for a naive subject (Subject 4) and a nonnaïve subject (Subject 2). The velocity traces in red are responses to a no blank condition. In all blanking conditions, the target disappeared at 600 ms (shown by the black dotted line). At ∼690 ms the horizontal component of eye velocity began decelerating for ∼180 ms and ∼280 ms (average across subjects) for blanking durations of 200 and 400 ms, respectively (Figure 3a). After the deceleration, horizontal eye velocity reached a stable value (V_min) for the 400 ms blank duration as seen in Figure 2a. Some subjects show a brief anticipatory acceleration starting before 80 ms of target reappearance (target reappearance indicated by solid colored vertical line) and later responding to the target reappearance with higher acceleration as seen in Figure 2a (green and black traces). 

Figure 2

 

Mean smooth pursuit responses for blanking during steady state pursuit. (a) & (b) Mean horizontal (ė_h) and vertical (ė_v) eye velocity traces (across 200 trials) for a naïve subject (Subject 4, shown in [a]) and for a nonnaïve subject (Subject 2, shown in [b]). Dotted vertical line (black) indicates blank onset at 600 ms and the solid colored vertical lines indicate end of blank duration for the corresponding blanking conditions (in blue for 200 ms and black for 400 ms duration). Panels (c) and (e) indicate the vertical eye velocity component (ė_v) across all conditions for 300 ms duration starting from 100 ms after stimulus onset. Panels (d) and (f) indicate the vertical eye velocity component (ė_v) across all conditions for 300 ms duration starting from 100 ms after blank offset (i.e., on target reappearance) for blanking conditions. For no blank condition (in red), vertical eye velocity component is taken for the same epoch as for the 600–1000 ms blanking condition. Stimulus is blanked at 600 ms after stimulus onset, for 200 ms or 400 ms duration.

Figure 3

 

Blanking during steady state pursuit: Horizontal eye velocity component. (a) δt_drop for two blanking durations shown for all subjects (thin and colored lines) and average across subjects (thick black line). (b) δV_drop and δV_ant for two blank durations shown for all subjects (thin and colored lines) and average across subjects (thick black line).

Once the target had reappeared at the end of the blank duration, the horizontal eye velocity accelerated until it matched target velocity, as illustrated in Figures 2a, b. After target reappearance, the vertical eye velocity increased before it peaked at ∼210 ms and gradually reduced or stayed constant as shown in the panels d and f of Figure 2. These changes in vertical eye velocity depended on the change in line orientation at reappearance as shown in the same panels of Figure 2. At target reappearance, if the orientation of the line stayed constant at 45°, the vertical component increased only marginally as shown by the blue and black traces. If the orientation was flipped by 90°, the vertical eye velocity changed direction as illustrated by the magenta and green traces. This change in direction of the vertical component corresponds to the effect of the aperture problem that is now introducing a downward bias in line motion perception. 

The horizontal eye velocity component (ė_h)

The quantitative changes in the horizontal eye velocity component during the blank are shown in Figure 3. δt_drop is plotted for different blank durations in Figure 3a. All subjects showed a similar increase in δt_drop with longer blank durations (p < 0.001). In Figure 3b, the amplitude of the drop in the horizontal eye velocity (δV_drop) is plotted against blank durations. This drop was marginal but significantly higher for the longer blank duration (p < 0.001). Subjects with larger pursuit experience (red and blue lines) showed a smaller drop. 

The anticipatory rise in eye velocity was quantified as δV_ant and shown in Figure 3b for 200 and 400 ms blank durations. The anticipatory pursuit was higher with a 400 ms blank duration as compared to the 200 ms condition (p < 0.001 for all subjects). This can be explained by the fact that expectation of target reappearance was higher for trials with long blank durations. 

The vertical eye velocity component (ė_v)

To quantify the changes elicited in the vertical pursuit component by either target reappearance or a sudden change in target orientation, we evaluated the average vertical eye velocity Display Formula using a sliding window as described in the Methods section. The mean (across subjects) value is plotted in Figure 4a for all conditions. This sliding window was initially centered at ∼210 ms after stimulus onset corresponding to the peak of vertical bias introduced in pursuit direction by the aperture problem. This gives a higher starting slide average vertical velocity Display Formula as shown in Figure 4a for all conditions. As the window slides through the vertical eye velocity component, slide average vertical eye velocity Display Formula decreases due to motion integration and gradually falls off to a constant during the steady state as shown for the no blank condition (red curve) in Figure 4a. For all the blanking conditions, the slide average vertical eye velocity Display Formula began to decrease at 620 ms towards zero and stabilized at a negative value near zero. This was followed by an increase at target reappearance when the bar orientation was 45° (blue and black curves). The vertical eye velocity Display Formula reached a peak or a plateau after target reappearance at 1010 ms (200 ms blank) and 1210 ms (400 ms blank) i.e., ∼210 ms after target reappearance (see the colored arrows in Figure 4a). Such timing is similar to the latency of the maximum for the initial vertical bias. When the target orientation rotated by 90° after the blank, the vertical eye velocity Display Formula reduced to negative. Arrows in Figure 4a show a clear demarcation between responses consecutive to a bar rotation after blanking and those obtained when the bar orientation remained constant after blanking, for both 200 ms (compare blue and magenta curves) and 400 ms (compare black and green curves) durations. The change in the vertical component on target reappearance (indicated by the length of the arrows) was calculated as the difference between slide average vertical components taken at ∼210 ms after target reappearance ( Display Formula , colored arrows in Figure 4a) and ∼150 ms after blank initiation ( Display Formula_, gray arrow in Figure 4a). This change ( Display Formula − Display Formula ) was used as a measure of vertical bias on target reappearance and quantified as a percentage of the initial peak slide average vertical component ( Display Formula ) due to the aperture problem. The percentage change in the vertical component is calculated as: [( Display Formula − Display Formula )/ Display Formula ] × 100. A window centered at 150 ms would include an average from 80 ms to 220 ms after target disappearance (that is 40 ms before eye responds to target reappearance in a 200 ms blank condition). Thus, we can define a baseline measurement that is free from any retinal influence and therefore be consistent across blanking conditions. This is illustrated in Figure 4b plotting the mean percentage change (± SE) for each subject. An equivalent quantity for the change ( Display Formula − Display Formula ) was also computed in the no blank condition at the same time points that are used for computing the ( Display Formula − Display Formula ) in each blank condition (see each panel in Figure 4b). 

Figure 4

 

Blanking during steady state pursuit: Vertical eye velocity component. (a) Mean across subjects of the slide average vertical eye velocity ( ) for all conditions (200 trials per condition). The colored arrows point to the slide average vertical eye velocity for the respective conditions at ∼210 ms after target reappearance ( ). The gray arrow points to the slide average vertical eye velocity during the blank ( ). The length of the arrow indicates the change in the slide average vertical eye velocity on target reappearance relative to slide average vertical eye velocity during the blank ( − ), which is used as a measure of vertical bias on target reappearance. (b) The change in the sliding average vertical eye velocity at target reappearance (indicated by the length of the colored arrows in Panel [a]) is expressed as a percentage of the initial peak velocity that is due to the aperture problem, for all blanking conditions. The same is computed for a no blank condition. Each panel reports the mean values for one subject. Asterisks indicate statistically significant change (t test; p < 0.05). Error bars represent SE.

To test whether the vertical component of the eye velocity on target reappearance ( Display Formula_, colored arrows in Figure 4a) was statistically different from the vertical eye velocity during the blank ( Display Formula_, gray arrow in Figure 4a), we conducted a paired t test between the two velocities ( Display Formula and Display Formula ). The t test revealed a significant change (p < 0.05) in the slide average vertical component on target reappearance for all blanking conditions in three subjects (Subjects 2, 3, and 4). Subject 6 showed a significant change for all conditions except when the target reappeared with a bar rotation after a 400 ms blank. Subject 1 showed no significant change except when the target reappeared with same orientation after a 400 ms blank. Subject 5 showed a significant change only when target reappeared identical after 400 ms blanking and when it rotated after a 200 ms blank. A similar t test was conducted in a no blank condition. Except for Subject 1, all the subjects showed no significant change. Overall, we found a change in the vertical bias following target reappearance across all conditions, with some variability across subjects and conditions, as expected from our previous study (Montagnini et al., 2006a). 

Experiment 2: Target blanking during the early phase of pursuit

Figure 5 plots both mean horizontal (ė_h) and vertical (ė_v) eye velocity when target motion was blanked for 400 ms during the early phase of smooth pursuit, meaning with the blank onset within the first 100 ms of smooth pursuit. Figures 5a, b illustrates mean eye velocity profiles obtained with the same naive (Subject 4) and nonnaïve (Subject 2) participants already plotted in Figure 2. The no blank condition is plotted in red. Dotted lines indicate blank onset and solid lines indicate blank end, with different conditions distinguished by different colors. The initial increase in the vertical pursuit component was clipped off when the moving stimulus disappeared. This is evident when the moving line was blanked 140 ms after target motion onset or earlier. The horizontal component started to decrease down to a plateau and then re-accelerated before target reappearance. Given the latency of ∼80 ms for the bar stimulus, this brief phase of pursuit represents an anticipatory acceleration to catch up with the target at its expected reappearance. Panels c and e zoom on the vertical eye velocity profiles for two 300 ms epochs, starting 100 ms after stimulus onset. Panels d and f zoom on the vertical eye velocity profiles for two 300 ms epochs, starting 100 ms after target reappearance. 

Figure 5

 

Mean smooth pursuit responses for early blanking experiment. (a) and (b) Mean horizontal (ė_h) and vertical (ė_v) eye velocity traces (across 200 trials) for a naïve subject (Subject 4) shown in (a) and for a nonnaïve subject (Subject 2) shown in (b). Dotted colored vertical lines indicate blank onset and the solid colored vertical lines indicate end of blank duration for the corresponding blanking condition depicted with a different color. Panels (c) and (e) indicate the vertical eye velocity component across all conditions for 300 ms duration starting from 100 ms after stimulus onset. Panels (d) and (f) show the vertical eye velocity component across all conditions for 300 ms duration starting from 100 ms after blank offset. For the no blank condition (in red), the vertical eye velocity component is taken for the same duration as the 140 ms target occlusion condition. Stimulus is blanked for 400 ms duration at different times during the first 100 ms after pursuit initiation.

The initial horizontal eye velocity component (ė_h)

The decaying time of horizontal eye velocity (δt_drop) is shown in Figures 6a and 6c for 200 ms and 400 ms blank durations, respectively, across different blank onset times. A significant dependence upon the timing of stimulus blanking can be appreciated (p < 0.001). When the moving target was blanked at either 100 or 180 ms for 200 ms, mean deceleration phases (across subjects) lasted ∼90 or ∼140 ms respectively. Similarly, blanking the moving target at either 100 or 180 ms for 400 ms resulted in mean deceleration phase (across subjects) that lasted for ∼160 or ∼250 ms, respectively. Thus, earlier target blanking resulted in shorter phases of horizontal pursuit deceleration. 

Figure 6

 

Blanking during early pursuit initiation: Horizontal eye velocity component. (a) and (c) δt_drop for all blanking conditions shown for all subjects (thin and colored lines) and average across subjects (thick black line) for 200 ms blank duration (a) and 400 ms blank duration (c). (b) and (d) δV_drop and δV_ant for all blanking conditions shown for all subjects (thin and colored lines) and average across subjects (thick black line) for 200 ms blank duration (b) and 400 ms blank duration (d).

The drop in eye velocity during blanking (δV_drop) is plotted against timing of target disappearance in Figure 6b and 6d (solid lines) for 200 ms and 400 ms blank durations, respectively. An early blanking (starting at 100 ms after stimulus onset) for 200 ms duration resulted in a small reduction in eye velocity with a mean ± SE across subjects of 0.55 ± 0.22°/s while a later blanking (starting at 180 ms after stimulus onset) resulted in a larger drop with a mean ± SE across subjects of 2.3 ± 0.29°/s. Similarly, blanking at 100 ms with a 400 ms duration resulted in a small reduction in eye velocity with a mean ± SE across subjects of 1.61 ± 0.35°/s while blanking at 180 ms resulted in a larger drop with a mean (± SE) across subjects of 3.75 ± 0.31°/s. The drop in eye velocity increased with later blanking, as shown by the significant effect of the timing of target blanking upon δV_drop (p < 0.001). 

Similar to the steady state conditions, pursuit started to re-accelerate in expectation of target reappearance for early blank conditions with both 200 ms (Figure 6b) and 400 ms (Figure 6d) blank durations. Such anticipatory rise in eye velocity was stronger for early blank conditions with a mean ± SE across subjects equal to 2.9 ± 0.13°/s and 2.76 ± 0.22°/s for blanking onset time of 100 and 120 ms, respectively, (Figure 6b) for the 200 ms blank. A similar trend is observed for 400 ms blank duration (Figure 6d) where anticipatory eye velocity was higher when compared to the same blanking onset timing but a duration of 200 ms. For a 200 ms blank, except for Subject 2 (blue line in Figure 6b) all subjects show a statistically significant (p < 0.001) relationship between δV_ant and blanking onset timing. For 400 ms blanking duration, except for Subjects 1, 2, and 5 (red, blue, and magenta lines in Figure 6d) all the subjects show a statistically significant (p < 0.001) dependence of δV_ant upon the time at which blanking starts. 

The initial vertical eye velocity component (ė_v)

The slide average vertical component of the eye velocity is shown in Figures 7a and 8a for a blanking duration of either 200 or 400 ms, respectively. Note that in these figures, the no blank traces are taken from different datasets (see Methods section). The no blank condition in Figures 7a and 8a is shown as a reference to highlight the effect of blanking on vertical eye velocity component at target reappearance. The clipping of the vertical bias for different blank onset times can be well distinguished from these plots. For a 100 ms blank onset, the vertical velocity ( Display Formula ) peaked well before ∼210 ms after target reappearance as shown by the colored arrow in Figure 7a (100–300 ms blank conditions) and Figure 8a (100–500 ms blank conditions). For all other conditions with the 200 ms blank duration, the slide average vertical velocity peaked around ∼210 ms after target reappearance but was clearly much noisier than observed during steady state blanking (Figure 7a). By comparison, a 400 ms blank resulted in a clear change of vertical velocity, peaking at ∼210 ms after target reappearance (Figure 8a). The change in the sliding average of the vertical component at target reappearance was calculated as the difference between the measurements taken at ∼210 ms after target reappearance ( Display Formula ) (colored arrow in Figures 7a and 8a) and at ∼150 ms after blank initiation ( Display Formula ) (gray arrow in Figures 7a and 8a). This change ( Display Formula − Display Formula ) is a better estimate of the vertical bias at target reappearance rather than the absolute value ( Display Formula ) in order to avoid possible contamination from individual oculomotor biases. Such change was quantified as a mean percentage change in vertical component with reference to the initial peak vertical component Display Formula due to the aperture problem as: [( Display Formula − Display Formula )/ Display Formula ] × 100. The mean percentage change in vertical component (± SE across subjects) is shown in Figures 7b and 8b for 200 and 400 ms blank durations, respectively. An equivalent quantity for the change ( Display Formula − Display Formula ) was also computed in the no blank condition at the same time points that are used for computing the ( Display Formula − Display Formula ) in each blank condition (see each panel in Figures 7b and 8b). Asterisks in Figure 7b indicate significant change in vertical eye velocity at target reappearance for the 200 ms blanking duration block (paired t test, p < 0.05). When blanking started at 100 ms, only one subject (Subject 1) showed a significant change. When blanking started at 120 ms, only one subject (Subject 2) showed a significant change. When blanking started at 140 ms, three subjects (Subjects 2, 3, and 5) showed a significant change. For the other conditions, Subjects 2, 3, 4, and 5 showed a significant change. In a similar way, asterisks in Figure 8b indicate significant change in vertical eye velocity at target reappearance for the 400 ms blanking duration block. When blanking started 100 ms after pursuit onset, only two subjects (Subjects 3 and 6) showed a significant change. When blanking started at 140 ms, three subjects (Subjects 1, 3, and 6) showed a significant change. When blanking started at 160 ms, three subjects (Subjects 1, 2, and 3) showed a significant change. For the remaining two conditions, only one subject (Subject 3) showed a significant change. All the subjects show a significant reduction in the vertical eye velocity component in the no blank condition. 

Figure 7

 

Short blanking during pursuit initiation: vertical eye velocity component. (a) Mean across subjects of the slide average vertical eye velocity ( ) for all conditions (200 trials per condition). The blue arrow points to the slide average vertical eye velocity at ∼210 ms after target reappearance ( ) for the 100–300 ms blanking condition. The gray arrow points to the slide average vertical eye velocity during the blank ( ) for the same blanking condition. The colored horizontal segments help to visualize the blank→ ← interval in each condition. The change in the slide average vertical eye velocity on target reappearance relative to the same quantity evaluated during the blank ( − ) is used as a measure of vertical bias on target reappearance, for all blanking conditions. (b) The change in slide average vertical eye velocity on target reappearance is expressed as a percentage of the initial peak due to the aperture problem for all blanking conditions. The same is computed for a no blank condition. Asterisk indicates statistically significant change (t test; p < 0.05). Each panel shows data for one subject. Duration of the blank is 200 ms. Error bars represent SE.

Figure 8

 

Long blanking during pursuit initiation: Vertical eye velocity component. (a) Mean across subjects of the slide average vertical eye velocity ( ) for all conditions (200 trials per condition). The colored arrow points to the slide average vertical eye velocity at ∼210 ms after target reappearance ( ) for the 100–500 ms blanking condition. The gray arrow points to the slide average vertical eye velocity during the blank ( ) for the same blanking condition. The colored horizontal segments help to visualize the blank interval in each condition. The change in the slide average vertical eye velocity on target reappearance relative to slide average vertical eye velocity during the blank ( − is used as a measure of vertical bias on target reappearance, for all blanking conditions. (b) The change in slide average vertical eye velocity on target reappearance is expressed as a percentage of the initial peak due to the aperture problem for all blanking conditions. The same is computed for a no blank condition. Asterisk indicates statistically significant change (t test; p < 0.05). Each panel shows data for one subject. Duration of the blank is 400 ms. Error bars represent SE.

In this second experiment, we found that with early blanking there was little or no vertical bias at target reappearance. With shorter duration blanks, we found a significant change in the majority of our subjects, but its direction was not consistent with the direction bias introduced by the aperture problem. The lack of a significant clear effect across all subjects and conditions indicate that the interactions between retinal and extraretinal signals must vary greatly across subjects. 

Summary of results

In summary, results from Experiment 1 show that pursuit responses at the end of a blank occurring during steady state tracking show a clear bias resulting from the aperture problem. Results from Experiment 2 show that pursuit responses at the end of a blank occurring within the first 100 ms after pursuit initiation show little or no bias due to the aperture problem. Second, the amount of the drop in horizontal velocity increases with the blank onset time and is always larger in the steady state blanking conditions (coherent with results of previous studies). Finally, anticipatory eye velocity at the end of blanking during steady state pursuit (Figure 3) is lower compared to the anticipatory eye velocity observed at the end a blank occurring during early pursuit (Figure 6). To understand these results in the context of different levels of prediction between steady state and early pursuit, we designed a hierarchical inference model for pursuit and simulated it for two different levels of prediction (low and high) for both steady state and initial blanking conditions. This hierarchical model of the pursuit system is fully described in the next section. 

A hierarchical inference model of smooth pursuit

We have previously shown that a recurrent Bayesian inference framework can be used to model the temporal dynamics of pursuit eye movements for complex and ambiguous visual motion (Montagnini et al., 2007). Such optimal motion integration combines sensory likelihoods with a prior distribution favoring slow and smooth trajectories (Hurlimann, Kiper, & Carandini, 2002; Stocker & Simoncelli, 2006; Weiss et al., 2002). The resulting posterior distribution of sensory inputs is then recurrently used as a prior distribution for the next computing step and its maximum a posteriori (MAP) estimate provides a dynamical retinal slip signal. Cascaded with a rudimentary smooth pursuit model referred to as the oculomotor plant that converts the target velocity estimate into eye velocity (Goldreich, Krauzlis, & Lisberger, 1992), the inference model produces smooth pursuit traces that are very similar to human smooth pursuit responses to a translating tilted bar (Bogadhi et al., 2011). For the sake of simplicity, our first model operated in open-loop conditions with no physical feedback from the moving eye to the retinal module encoding visual motion. 

Here we extend that model to a closed loop version by including both a physical negative feedback and the positive efference copy feedback needed to maintain pursuit when retinal slip decays to zero (Yasui & Young, 1984). We kept the probabilistic framework to define a hierarchical Bayesian model with two recurrent stages implementing optimal sensorimotor control (Todorov, 2004). The block diagram illustrated in Figure 9 summarizes the information flow. The first loop implements the dynamics of visual motion integration as described above. The inputs to the model are 1D and 2D (see Barthelemy, Perrinet, Castet, & Masson, 2008) likelihood distributions for target velocity given by Equations 1 and 2 The sensory prior and likelihood distributions of the model are assumed to be Gaussian (Weiss et al., 2002) and their variance can be deduced from an independent set of oculomotor data (see Bogadhi et al., 2011; Montagnini et al., 2007). The second stage forms an extraretinal recurrent Bayesian network. It is fed with the full probability distribution computed by the first stage. As per the sensory prior, the extraretinal prior is initially centered on zero in the two-dimensional velocity space. Retinal and extraretinal posterior distributions are combined based on their respective reliabilities. Such weighted combination is equivalent to the Kalman filtering approach used for optimal motor control (Kalman, 1960). Below, we detail each computational step. 

Figure 9

 

A hierarchical recurrent Bayesian model of smooth pursuit. The 1D and 2D motion likelihoods are delayed by δ1D and (δ1D + δ2D), respectively, and combined with a prior to obtain a posterior as shown in the retinal recurrent Bayesian network. The prior is updated every δt_ret ms. The posterior from the retinal recurrent Bayesian network and prior from the extraretinal Bayesian network are combined to form the post-sensory output (P_out) (see Equations 10, 11). The maximum a posteriori of the target velocity in space (P_T) serves as an input to both the positive feedback system as well as the oculomotor plants. The output of the oculomotor plant is subtracted from the target velocity to form the image retinal velocity (physical feedback loop shown as broken line). During the transient blank when there is no target on the retina, the physical feedback loop is not functional so that the retinal recurrent block doesn't decode any motion. The output of the positive feedback system (shown in broken line) is added to the post-sensory output (P_out) only when the physical feedback loop is functional. The probability distribution of target velocity in space (P_T) is provided as an input to the extraretinal recurrent Bayesian network where it is combined with a prior to obtain a posterior which is used to update the prior every δt_ext ms.

Likelihood and prior distributions

The visual stimulus contains 1D (edge related) and 2D (terminator related) motion information. We assume both of them to be independent and Gaussian distributions. If ν₀ is the stimulus velocity, the likelihood function L₁ for edge related information (1D) in the velocity space (v_x, v_y) is given by:  

 

where Z is the partition function (the same symbol is used for all distributions), θ is the orientation of the line relative to the vertical, taken as positive in anticlockwise direction, and σ₁ is the standard deviation of the speed in the orthogonal direction to the line. The likelihood function L₂ for the terminator related information in the velocity space (v_x, v_y) is given by:  

 

where σ₂ is the standard deviation of the velocity of the image 2D cues. The overall likelihood function is the product of the two (1D and 2D) likelihoods since, they are both assumed to be independent: 

 

 

 

Assuming a prior distribution favoring slow speeds (mean centered at origin) and directionally unbiased (i.e., normally distributed with a diagonal covariance matrix and variance σ₀), the initial sensory prior distribution P₀ can be written in the velocity space (v_x, v_y) as follows:  

 

The likelihood function (L) is combined with the initial prior (P₀) using Bayes's rule to obtain the initial posterior distribution (Q₀)  

 

To obtain a read out of the distribution that can be used for the later stages, a decision rule called maximum a posteriori (MAP) is implemented as:  

 

The posterior distribution at every instant t is used to dynamically update the prior (i.e., generating a recurrent Bayesian computation) used at the next iteration:  

 

Finally, at any time t, the retinal Posterior can be expressed as:  

 

The square-root variance terms (or standard deviations) σ₀ (2.87°/s), σ₁ (1.38°/s), and σ₂ (4.22°/s)_, for the prior, 1D, and 2D likelihood, respectively, were estimated from an independent set of oculomotor data, applying Bayes's rule to pure 1D and pure 2D motion stimuli as described in detail in Montagnini et al. (2007). The delay in the retinal block δ_ret is estimated to be 65 ms based on the physiology literature (see Bogadhi et al., 2011). 

The initial prior in the extraretinal block (P_ext) is assumed to be Gaussian and centered on zero.  

 

For the sake of simplicity, the initial variance in the extraretinal prior is assumed to be same as the variance in the retinal prior (σ_ext = σ₀ in Equation 9). The extraretinal prior is combined with the probability of target velocity in space (P_T, see below) and updated with the resultant posterior as shown in Figure 9. This recurrent operation changes the extraretinal prior before blank onset. 

As illustrated in Figure 9, the initial post-sensory output P_out corresponds to the output of the sensory estimation Q₀ and its weighted combination with the extraretinal prior distribution P_ext as well as its evolution in time is described in the next section. 

Mixing retinal and extraretinal signals to drive pursuit

The role of the retinal recurrent Bayesian module is to integrate visual motion information over time, following a temporal dynamics that is comparable with that of motion integration in macaque area MT (Pack & Born, 2001; see Masson et al., 2010; Tlapale, Masson, & Kornprobst, 2010). When the stimulus disappears, the prior in the retinal recurrent block is gradually reset to its initial distribution centered on zero in the two-dimensional velocity space. Note that if the prior in the retinal block wasn't reset, the target speed and direction as estimated at the moment of blank onset would continue to be decoded from the sensory posterior distribution. This would not be consistent either with the observed dynamics of pursuit eye movements (i.e., a slowing down of tracking velocity), or with the neurophysiological findings that MT neurons stop encoding target velocity in absence of retinal image motion (Ilg & Thier, 2003; Newsome, Wurtz, & Komatsu, 1988). In physiological terms, the resetting of the prior to its initial distribution might be seen as a gradual reduction in the mean and increase in the variance of the neuronal population activity towards the level of spontaneous activity. When the target disappears, the likelihood representing the blank on the retina can be modeled as a Gaussian centered on zero with infinitely large variance (or equivalently as a uniform distribution over the possible velocities). When this likelihood is combined with the prior, the resultant posterior is equal to the prior. The MAP of the posterior is therefore zero and hence the retinal velocity that is estimated during blank is null. The role of the extraretinal Bayesian module is therefore to maintain smooth pursuit in absence of retinal inputs (or zero-centered, noisy retinal velocity). 

Figure 9 illustrates how retinal and extraretinal information are dynamically weighted and combined. The two variables, prior from the extraretinal block P_ext(v_x, v_y) and the posterior from the retinal recurrent block Q(v_x, v_y) are represented in the same velocity space and both have the same units. The extraretinal prior P_ext(v_x, v_y) and the retinal posterior Q(v_x, v_y) are weighted according to their respective reliabilities (i.e., they are inversely proportional to their variance) and combined linearly to form the dynamic post-sensory output (P_out in Equation 10 and Figure 9, see detailed description below) both during the blank and at target reappearance. The probability distribution of target velocity in space P_T is obtained by the summation of the post-sensory output with the positive feedback (see Figure 9 and below). A copy of P_T is given as an input to the extraretinal block (Freeman, Champion, & Warren, 2010) and its role here is analogous to that of the efference copy of the premotor drive. The resultant posterior is then used to update the extraretinal prior with a constant delay (δ_ext). Imposing a conditional update of the extraretinal (prediction) component similar to a Kalman filter is beyond the scope of this study. Instead, we decided that the extraretinal prior is updated only when the retinal weight (W_ret) is greater than 0.99 indicating that the extraretinal block stores a visuomotor drive triggered by reliable retinal motion information. This delayed update of prior with posterior can be viewed as a sample and hold mechanism, with the prior storing the memory signal, equivalent to the indirect loop of the model described by Bennett and Barnes (2004). The delay δ_ext (∼97.5 ms) was arbitrarily set at 1.5 times the update delay in the retinal block δ_ret (∼65 ms). The MAP of the probability distribution of target velocity in space (i.e., the target velocity estimate) serves as an input to both the oculomotor plants and the positive feedback system. 

Since there is no target on the retina during the transient blank, the physical feedback loop is not functional (broken line, Figure 9). Having this feedback being nonfunctional during the blank keeps the likelihood distributions centered at zero. Without such disabling, the retinal recurrent Bayesian block would encode a negative velocity due to the physical negative feedback. The output of the positive feedback system is added to the post-sensory output (P_out) only when the physical negative feedback is functional. The horizontal (v_x) and vertical (v_y) components of the MAP are delayed by δ_p and passed through low-pass filters with gain, K = (K_x, K_y), and cut-off frequencies, ω = (ω_x, ω_y), as shown in Figure 9. For simplicity, both horizontal and vertical components of the positive feedback system are shown as one single entity in Figure 9. The delay δ_p and the filter parameters were kept constant for both v_x and v_y across all conditions, δ_p = 110 ms; K_x = 3 (9.54 dB); ω_x = 1 Hz; K_y = 2 (6 dB); ω_y = 1 Hz. The values of the positive feedback system were set to compensate for the combined dynamics of the horizontal and vertical oculomotor plants in generating retinal slip, along with the retinal recurrent Bayesian block in processing the retinal slip to estimate (P_out). The parameters of the retinal recurrent Bayesian network as well the oculomotor plant (K_d, K_p) were taken from our previous study (see table 2 in Bogadhi et al., 2011). 

The retinal posterior, Q_t(v_x, v_y), and the extraretinal prior, P_ext(v_x, v_y), are weighted and combined to give the resultant post-sensory output, P₀(v_x, v_y), as described here. Following the idea proposed by recent models for sensory cue combination (Ernst & Banks, 2002; Kersten, Mamassian, & Yuille, 2004; Oshiro, Angelaki, & DeAngelis, 2011), the retinal and extraretinal weights were modeled as a function of their respective reliabilities (i.e., the inverse of their variance) as shown in Equation 10:  

 

and  

 

where W_ret and W_ext are dynamical retinal and extraretinal weights, respectively, and the symbols ( Display Formula , Display Formula ) correspond respectively to the variances of the retinal posterior, Q_t(v_x, v_y), and extraretinal prior, P_ext(v_x, v_y). 

Thus, the mean of the retinal posterior (μ_ret) and the mean of the extraretinal prior (μ_ext) are linearly weighted and summed to give the mean of the post sensory output ( Display Formula ) as shown in Equation 11a. The variance of the post sensory output ( Display Formula ) as a function of the retinal posterior variance ( Display Formula ) and the extraretinal Prior variance ( Display Formula ) is given by Equation 11b.  

 

 

 

During the blank, the variance of the prior in both retinal and extraretinal blocks increases towards their initial values (given by Equation 12). Thus, the dynamics of the retinal and extraretinal weight functions is a result of the dynamics of the increase in the retinal and extraretinal variances during the blank. For the retinal variance increase during blank, the value of c is kept constant across all conditions (c = 1). For the extraretinal variance increase during the blank, the value of c determines the strength of prediction:  

 

where t_b is the blanking onset time, c is the constant determining the rate at which variance increases, and Display Formula is the initial variance of the prior. Note that the particular functional dependence (hyperbolic) of the prior variance on t we have chosen here is coherent with the natural time course of the prior variance in a simple Bayesian recurrent model as the one described by Equation 8 (See also equations 9 and 10 in Montagnini et al., 2007). However, the constant c depends on mechanisms (e.g., the resetting of the prior) that we cannot really model in detail at this stage and thus we will keep it as a free parameter. 

Model simulations

Eye velocity profiles were simulated for different blanking conditions together with a control condition where the target was fully visible (no blank condition). These simulated smooth pursuit responses (both horizontal and vertical) are shown in Figures 10a, b together with the corresponding temporal dynamics of retinal and extraretinal variances (Figures 10e, f), and the resulting weights (Figures 10c, d). We simulated two sets of trials, corresponding to different sensitivities to the aperture bias, as well as two levels of pursuit gain during blanking phases and we categorized them as high prediction and low prediction, respectively. Our goal was to produce a family of responses corresponding to the diversity in human behavior that was reported in the results section. The simulated eye velocity profiles displayed in Figures 10a, b are similar to the smooth pursuit responses to a tilted bar stimulus in the steady state blanking condition (Figure 2) and early blanking condition (Figure 5). Note that, the simulated model responses do not reproduce the complete pattern of eye movement recordings of any given subject, but the two different levels of prediction (high and low) mimic the experimental data observed, respectively, for the early blanking and the steady state blanking conditions. 

Figure 10

 

Model simulations and temporal dynamics of retinal and extraretinal weights. Simulated smooth pursuit responses in different blanking conditions (a) and (b) and temporal dynamics of the retinal and extraretinal weights (c) and (d) as determined from their respective variances (e) and (f), for two different strengths of prediction. High- and low-prediction settings are illustrated in left and right columns, respectively. Model simulations are qualitatively similar to the smooth pursuit responses obtained in different subjects as shown in Figures 2 and 5, respectively. Panels (g) and (i) show the simulated vertical eye velocity for the same conditions during a 300 ms epoch starting 100 ms after stimulus onset. Panels (h) and (j) show the vertical eye velocity for all conditions, during a 300 ms epoch starting 100 ms after the target reappearance following a 400 ms blanking. In these panels, the control vertical eye velocity is taken during the same epoch in a no blank condition.

The evolution of both retinal and extraretinal variances are shown in Figures 10e, f. At target motion onset, the retinal variance decreases as a result of the Bayesian recurrent computation performed in the retinal block. After a delay of δt_ext ∼ 97.5 ms, the variance of the extraretinal signal starts decreasing as a result of the Bayesian computation in the extraretinal network. When the blank is introduced, both retinal and extraretinal variances increase gradually back to their initial value (see Equation 12). The dynamics of increase in the retinal variance is constant in both low- and high-prediction cases. What determines if the prediction is low or high is the dynamics of increase in extraretinal variance, determined by the constant c in Equation 12. In the high-prediction case (c = 10,000 for the example in the left panels of Figure 10), the extraretinal variance stays low during the blank (Figure 10e) and hence higher weight is given to extraretinal signals whereas in the low-prediction case (c = 50 for the example in the right panels of Figure 10), the extraretinal variance increases to a higher value during the blank (Figure 10f) and hence relatively lower weight is attributed to the extraretinal signals. The c values were chosen such that model pursuit responses capture the low- and high-prediction categories qualitatively. At the end of the blanking period, when the target reappears, the retinal variance starts decreasing again as a result of Bayesian computation in the retinal block whereas the extraretinal variance remains constant for some more time since our model assumes that the extraretinal signal is updated only when retinal information is reliable (i.e., W_ret ≥ 0.99). 

Some subjects show a small eye velocity drop during blanking in both steady state and early pursuit stages (e.g., Subject 1; see also Figures 3, 6). In our model, this behavior corresponds to the high-prediction case as set by the high value of parameter c. When the stimulus is blanked during the early phase of pursuit, the extraretinal variance changes little during the blank and the retinal variance increases towards its initial value (Figure 10e). Therefore the extraretinal weight is high during the blank resulting in a small drop in the horizontal eye velocity. At the end of the blank, once the target reappears the extraretinal weight decreases and the retinal weight increases. Since the extraretinal weight is still higher than the retinal weight at the time of reappearance, no bias is seen in the vertical eye velocity (Figure 10h). A similar pattern of dynamics of the weights and the resulting pursuit responses can be seen for the steady state blanking condition shown in Figures 10a, c, e (black line). Note that, in the high-prediction case we had to decrease the gain of the positive feedback (K_x = 1.8) to avoid motor behavior instability in the steady state blanking condition only. For all other blanking conditions, we kept this parameter identical (K_x = 3). 

Some subjects showed a much larger eye velocity reduction during blanking in both steady state and early pursuit stages (e.g., Subject 3; see Figures 3, 6). This would correspond to the low-prediction setting of our model (c = 50). Under these conditions, when the stimulus is blanked during the early phase of pursuit, the extraretinal variance increases more dramatically (Figure 10e) while the retinal variance increases towards its initial value (Figure 10f). As a result, the extraretinal weight is lower during the blank and hence a larger drop in eye velocity is seen during target occlusion. Once the target reappears, the extraretinal weight decreases and conversely, the retinal weight increases. Note that the weight given to retinal signal is much higher at target reappearance, as compared to the high-prediction setting (Figure 10c). This results in a significant bias in the vertical eye velocity since now the pursuit behavior carries the signature of the aperture problem affecting the visual motion computation (Figure 10j). Similar dynamics of the weights and the resulting pursuit responses can be seen for the steady state blanking condition shown in Figures 10b, d, f (black line). 

A source of interindividual variability in the pursuit behavior can thus be captured in the model simulations by tuning the constant c and thus the strength of the prediction. This variability and its dependence on the prediction level can be appreciated by comparing the black curves in Figure 2a and 2b with the corresponding black curves in Figure 10b and 10a, respectively. 

In Figure 10, Panels g, i show the vertical eye velocity at pursuit initiation for both high and low prediction, respectively. As expected, the initial vertical eye velocity component is similar to the no blank condition (compare with Figure 2). When blanking occurs during pursuit initiation, the initial vertical component is clipped off and its profile is very similar to that reported with human subjects (see Figure 5). At target reappearance (Panels h, j), the simulated peak vertical bias is slightly higher than observed (Figure 2) because of the overall scaling of model responses (see Bogadhi et al., 2011 for details). 

In summary, the model simulations show that the level of prediction and the dynamic interaction of retinal and extraretinal cues largely determine the dynamics of pursuit during the blank and the aperture bias on target reappearance, irrespective of whether the target is blanked in steady state or during early pursuit. Given these predictions, the majority of the subjects' responses can be classified as low prediction in steady state blanking condition and high prediction in early blanking condition. 

Furthermore, the model makes an important specific prediction that can be tested from our behavioral results. By varying the parameter c, the model predicts that a larger drop of horizontal velocity (higher retinal slip) during blank is accompanied by a higher vertical bias at target reappearance, as inferred from Figures 10b, d, and j, and this irrespective of whether the target is blanked in steady state or during early pursuit. We estimated the correlation between the observed horizontal eye velocity drop (absolute value) during blank and the observed aperture bias on target reappearance when the stimulus is blanked during steady state for either 200 ms or 400 ms blanking durations. Figure 11 plots the mean horizontal eye velocity drop during the blank against the mean percentage change in vertical bias at target reappearance for each subject and for two blanking durations. The line with the positive slope in Figure 11 is the line of best fit in a least squares sense. We tested the correlation in the steady state blanking conditions since this condition showed a significant aperture bias after the blank. Absolute eye velocity drop and the aperture bias were indeed positively correlated (Pearson correlation, r = 0.74; p < 0.01). Because the inclusion in the dataset of different blanking durations might have added some spurious covariance in the data, we also estimated the partial correlation between absolute velocity drop and vertical bias controlling for the variable blanking duration: also in this case the Pearson correlation turns out to be significant and with a similar correlation coefficient (Pearson correlation, r = 0.75; p < 0.01). Overall, this observation from data suggests that the level of prediction (reflected in the dynamics of cue interaction) largely determines the dynamics of pursuit during the blank and the aperture bias on target reappearance as predicted by the model. 

Figure 11

 

Testing a model prediction. Our model predicts a correlation between the eye velocity drop that is observed during blank and the aperture bias that will be see on the vertical eye velocity shortly after target reappearance. For six subjects and two blank durations, vertical bias of pursuit responses at target reappearance (expressed as a percent of the initial bias) is plotted against the eye velocity drop observed when occluding the target during steady-state pursuit. The line with the positive slope is the least squares regression line.

Limitations of the model

Our model borrows important elements of two classes of previous models, namely the notion of a static (and leaky) memory of target velocity (Barnes & Collins, 2008a; Bennett & Barnes, 2004; here implemented by the extraretinal module) and the positive feedback to the pursuit (Churchland et al., 2003; Madelain & Krauzlis, 2003). However, what is new in the model proposed here is the Bayesian probabilistic framework implemented both in the retinal block, to compute target velocity estimate, and in the extraretinal block, to sample and hold the target velocity estimate. Importantly, we use such probabilistic framework to implement a cue combination rule based on the respective reliability (i.e., the inverse of a cue variance) of retinal and extraretinal information. Overall, this inferential framework, integrated with the more standard elements of velocity memory and positive feedback, allowed us to qualitatively reproduce the observations in the blanking paradigm. 

However, in order to simulate pursuit responses in the blanking paradigm that are comparable to the observed responses, we had to incorporate a restricted number of ad hoc features. First, we assume that pursuit response starts with a retinal weight equal to one and which remains the same unless retinal motion information undergoes a perturbation, as for instance a transient blank. Ideally, when retinal and extraretinal signals are combined based on their respective reliabilities, one would like to see the same computation to be implemented throughout the entire trial, regardless of the presence of a blanking event. The reason to impose this sort of “dual regime” is twofold: First, we do not have, with the current set of data, enough evidence to constrain the exact dynamics of the interaction of retinal and extraretinal cues for long pursuit epochs and fully define the Bayesian recurrent framework accordingly. Second, we have the more general problem of explaining pursuit behavior during steady-state visual tracking. In other words, it is still unclear to us how one can replace the positive feedback loop during steady state visual tracking with our Bayesian model, because in that condition one would have to reconcile an almost-zero retinal slip and a virtually very high retinal weight with almost perfect pursuit. Second, we reasoned that a positive feedback could be functional only when the outer negative feedback is functional i.e., when there is a stimulus on the retina. If, instead, the positive feedback was operational during the blank, the resultant of the weighted retinal and extraretinal signals would be added to itself, resulting in an overestimate of the target velocity during the blank. Note that the two above raised features of the model could be avoided if one could integrate the positive feedback into the inferential extraretinal module. Third, we assumed that the extraretinal recurrent loop is functional only when retinal information is reliable (i.e., W_ret ≥ 0.99). On the contrary, if the extraretinal block was to be functional when the retinal weight is much less than one, like for instance during the blanking period, the eye velocity memory would be updated with the zero velocity information during the blank and this would lead to a rapid degradation of the target velocity information. In the current version of the model, the extraretinal information is indeed somehow degraded, as expressed by the increase of the extraretinal prior variance, but it is not biased toward zero. Finally, we assumed an ad hoc fast dynamic transient for resetting both retinal and extraretinal prior variances after the blank onset. Although such assumption is neither derived from some general properties of the Bayesian framework nor from the physiological literature, this reset is a biologically plausible feature of the model. Clearly, each of these specific assumptions needs to be specifically addressed in future studies. 

We don't claim that our model provides a complete and accurate representation of the physiological mechanisms underlying pursuit control, given all the limitations described above. In addition, the model in its current version, with a single free parameter, clearly fails to mimic the pursuit behavior of a large number of subjects across the whole duration of a pursuit trial: most experimental eye velocity curves correspond well to a high-prediction model curve for the early-phase blanking conditions but they are better fitted by the low-prediction model for the steady-state blanking condition. 

Nevertheless, the probabilistic nature of the model gives an opportunity to understand the mechanisms for combining different sources of information for pursuit, within a theoretical background, which has encountered a growing success in the cognitive neurosciences (Kersten et al., 2004; Weiss et al., 2002). In addition, the ability of the model to predict a correlation between the drop of horizontal velocity during the blank and the transient vertical bias on target reappearance (Figure 11) shows the interest of our approach when trying to understand experimental data as well as promoting a new view of the visuomotor transformation for pursuit that is based on inference rather than on deterministic signals. 

Discussion

In the present study, we investigated the effects of target blanking on pursuit eye velocity under different conditions. We used simple targets and ambiguous motion conditions that result in well-identified biases in vertical eye velocities due to the dynamics of visual motion integration (Bogadhi et al., 2011; Masson & Stone, 2002; Wallace et al., 2005). Visual and nonvisual signals are necessary to control tracking eye movements in responses to normal, fully visible (Robinson et al., 1986; Yasui & Young, 1984), as well as transiently occluded, moving targets (Barnes & Collins, 2008b). Our objective was to probe the weighted contribution of predictive signals in the presence of ambiguous target motion directions (i.e., at target reappearance) as well as in the absence of visual inputs (i.e., during blank) by comparing the pursuit biases seen after target first appearance or at reappearance (Masson & Stone, 2002). The time course of these biases allowed us to titrate the dynamics of such weighting. Overall, our behavioral results combined with the model simulations show that when the extraretinal signals play a prominent role in driving pursuit (e.g., during early blanking), the ambiguous motion signals from the retina are barely reflected in the pursuit trajectory (e.g., at target reappearance). By blanking the target at different times after motion onset, we were able to measure the temporal dynamics of these biases and therefore the relative contribution of visual and predictive signals from the magnitude of the vertical eye velocity (Masson & Stone, 2002; Montagnini et al., 2006b). 

A transient disappearance of the target during smooth pursuit results in an exponential decay of eye velocity if there is no expectation of target reappearance (Mitrani & Dimitrov, 1978; Pola & Wyatt, 1997). When the target is expected to reappear after a blank lasting more than 400 ms, the eye velocity stops decaying after about 250 ms and stabilizes at a constant level at about 70%–80% of target speed (Becker & Fuchs, 1985). Interestingly, learning improves the residual level of the velocity during the blank (Madelain & Krauzlis, 2003). This dynamics in the eye velocity during the blank is well investigated at steady state (Bennett & Barnes, 2003) and open loop stages of pursuit (Barnes & Collins, 2008a; Churchland et al., 2003). To explain this behavior, most of the models use an efference copy of the desired eye velocity to create a positive feedback signal that is continuously relayed to drive the pursuit in the absence of the visual stimulus (Churchland et al., 2003; Krauzlis & Lisberger, 1994; Madelain & Krauzlis, 2003). Moreover, several studies have reported a considerable influence of cognitive and voluntary factors arguing for the need for an extraretinal signal carrying more information than the instantaneous eye velocity. Bennet & Barnes have proposed that the anticipatory rise in eye velocity observed before the end of target blanking is explained by the existence of an indirect loop that samples the visuomotor drive during pursuit (Bennet & Barnes, 2004; 2006). They reasoned that, if eye velocity at the time of the blank is lower compared to the sample of the visuomotor drive acquired by the pursuit system during early pursuit, then eye velocity after an initial drop due to the blank increases to match the sample of the visuomotor drive during blank but with a lower acceleration. 

Our results give some insights into the dynamics of such predictive, extraretinal signals and its contextual modulation. The behavior of tracking responses to both early and late blanking is coherent with previous reports (Bennett & Barnes, 2004, 2006; Churchland et al., 2003). We show that target blanking during steady state disrupted pursuit gain by about 30% during the time of the blanking and by a smaller amount in the early blanking condition. Importantly, in many cases, and especially during early blanking, the increase in eye velocity needed to catch up with the target at its reappearance started well before the actual target reappearance. By extensively probing the dynamics of eye movements during early blanking, we show that early blanking yielded more predictive tracking. These results are consistent with a larger contribution of extraretinal signals during early pursuit initiation rather than during steady state tracking. This can be explained by the larger variance of visual signals in the pursuit initiation phase (Osborne et al., 2007). Because visual signals are intrinsically noisy and less reliable during the eye acceleration phase, the oculomotor system must rely more heavily on internal, predictive signals. Conversely, less weight is given to sensory evidence at this early stage. By contrast, during steady state tracking, more weight is given to visual input at target reappearance. The amplitude of the vertical eye velocity component supports this view. Remember that the vertical response component reflects the biased sensory estimation resulting from the aperture problem (Masson & Stone, 2002; Wallace et al., 2005; see Masson, 2004, for a review). A large vertical deflection of tracking direction would therefore reflect a large contribution of the sensory signal to the visuomotor transformation. Coherent with our interpretation, we found that early blanking results in weak vertical biases while late blanking produced stronger vertical biases at target reappearance. For the first time, our approach allows us to titrate the relative contribution of predictive signals and sensory evidences in the dynamic control of pursuit eye movements. 

Note that in the present experiments, we used a blocked design with a blanking duration of either 200 or 400 ms. This could have reinforced the role of expectation signals and reduced the role of visual signals and eye velocity memory (Barnes & Collins, 2008b; Montagnini et al., 2006b). However, Masson and Stone (2002) found that target reappearance after a 200 ms blanking resulted in almost no vertical biases. In their experiments, conditions were strictly interleaved, reducing the role of expectation. Coherently, they found no evidence for eye acceleration prior to target reappearance as reported here with a block design. At first glance, the complete absence of a vertical bias after target reappearance during steady state tracking found by Masson and Stone (2002) seems contradictory with the present study reporting a small but significant vertical bias in pursuit direction. A subject and context dependent weighting of predictive signals and sensory evidence can easily reconcile our two results. First, in the present study, the target orientation changed on half of the trials, regardless the occurrence of a target blanking. This could have increased the weight given to sensory evidence. Second, Masson and Stone (2002) used perifoveal tracking as subjects were instructed to follow the center of an empty line-drawing diamond. Under these conditions, the strength of extraretinal signals must have been higher than in the present experiments (Ilg & Thier, 2003; Masson et al., 2010), reducing the impact of retinal direction bias at target reappearance. Finally a slightly larger pool of subjects was tested in this study, which seems to have some importance, given the large interindividual variability that we observed. In summary, our hierarchical Bayesian model with different dynamics for the mixing weights of retinal and extraretinal information can accommodate the discrepancy between our current and past results. 

Comparison with previous modeling work

Traditionally, computational studies of pursuit eye movements have focused on either the dynamics of the visual motion processing (e.g., Krauzlis & Lisberger, 1989; Montagnini et al., 2006b) or the extraretinal signals used to predict the future target trajectory (e.g., Barnes & Collins, 2008b; Bennett & Barnes, 2004; Orban de Xivry et al., 2008) For instance, the models proposed by Bennet & Barnes can explain the anticipatory rise in the horizontal component of eye velocity during an early target blanking but cannot process ambiguous sensory inputs as they lack a well-defined motion integration stage. Therefore, they cannot simulate the context-dependent impact of blanking on two-dimensional eye movements tracking locally ambiguous motion signals (Bogadhi et al., 2011; Montagnini et al., 2007; Wallace et al., 2005). These previous models (Barnes & Collins, 2008a; Bennett & Barnes, 2004) explain the dynamics observed in smooth pursuit during blanking by switching between direct and indirect loops and changing the gain value (β) depending upon expectation about target reappearance. If such a switching mechanism were true, we should have observed a significant vertical bias at target reappearance in the early blanking conditions, very similar to the bias observed in the steady state blanking conditions for all subjects. This was clearly not the case. Our results suggest that the transfer of weights from retinal to extraretinal signals during blanking is not regulated by an ad hoc, instantaneous switching process but rather follows a well-defined, dynamical change. 

Furthermore, the dynamical change in the retinal and extraretinal weights is determined by the reliability of the competing retinal and extraretinal signals. This competition between retinal and extraretinal signals can explain the dynamics observed during the blank as well as the aperture bias on target reappearance. This proposed mechanism is largely different from arbitrarily changing the gain value depending upon expectation about target reappearance as proposed by previous models (Bennett & Barnes, 2004; Churchland et al., 2003; Madelain & Krauzlis, 2003). The current model indicates that the level of prediction largely determines both the dynamics of pursuit during the blank and the aperture bias on target reappearance. Importantly, this indication leads to a new specific prediction concerning the correlation, across subjects, between the amount of drop in horizontal velocity during the blank and the vertical bias on target reappearance. This prediction was confirmed by the data collected in the steady-state condition. Previous models based on gain modulation do not directly predict any such correlation between dynamics of pursuit during the blank and the aperture bias on target reappearance. 

The theoretical framework proposed here is based on Bayesian dynamic inference processes on both retinal and extraretinal computation levels and on optimal cue combination for the integration of the two types of signal. Differences between subjects can be explained with different dynamics in setting the weights (Figures 10a, b) and therefore by the dynamics of the variance of the extraretinal signal during the blanking period, as modeled by the parameter c in Equation 12. Our model does not predict the full pattern of eye movements for each subject, in particular the anticipatory velocity at the end of blank in the early blanking conditions and the relative difference observed for the post-blank bias in vertical velocity between the early and late blanking conditions. Nonetheless, this model provides a more complete theoretical framework for smooth pursuit, which is fully dynamical and can incorporate some interindividual differences, as well as the constraints imposed by both retinal and extraretinal inputs to the premotor drive. 

Physiological relevance of the hierarchical inference model

During the blanking of the stimulus, the retinal recurrent Bayesian block has been set not to show any sign of activity encoding target direction or speed as expected from the physiological equivalent of area MT (Ilg & Thier, 2003; Newsome et al., 1988). The extraretinal recurrent Bayesian network along with its input can be seen as implementing a possible contribution of the frontal eye fields (FEF) and supplementary eye fields (SEF), cortical areas which are known to provide an extraretinal drive to pursuit (Barborica & Ferrera, 2003; de Hemptinne, Lefevre, & Missal, 2008; Fukushima, Yamanobe, Shinmei, & Fukushima, 2003; Missal & Heinen, 2004; Xiao, Barborica, & Ferrera, 2007). Our model suggests a weighted mixing of the retinal and extraretinal signals, which could be done at the level of the lateral portion of the medio-superior temporal area (MSTl). Several studies have shown that area MSTl receives both retinal and extraretinal inputs (e.g., Newsome et al., 1988). This supports the view that there are two cortical networks; one processing retinal information (V1-MT-MSTl) and the other extraretinal information (MSTl-FEF-SEF) and both are combined at the level of area MSTl (see Thier & Ilg, 2005). We have proposed that these two loops implement a hierarchical recurrent model that can infer the most optimal integration for sensorimotor integration in ambiguous conditions (Masson et al., 2010). 

Conclusion

In the present study, we used a tilted bar stimulus as a pursuit target in a blanking paradigm to probe the interactions between retinal and extraretinal signals on target reappearance after the extinction of the blank. The target was blanked at different times during pursuit both in the open loop and steady state phases. The results show similar dynamics in the horizontal component of the eye velocity during the blank as reported by the previous studies (Barnes & Collins, 2008b; Becker & Fuchs, 1985; Bennett & Barnes, 2003; Churchland et al., 2003; Madelain & Krauzlis, 2003). However, the dynamics of vertical eye velocity at target reappearance provides new evidence in favor of a stronger role of extraretinal signals when a moving target reappears after a transient blank occurring during initial eye acceleration as compared to steady state tracking. These findings suggest that a simple switching from extraretinal to retinal signals at target reappearance as proposed by previous models (Barnes & Collins, 2008a) is not sufficient. Rather, they argue for a dynamic tradeoff of weight between retinal and extraretinal signals. We propose a physiologically plausible hierarchical recurrent Bayesian model with retinal and extraretinal recurrent Bayesian blocks whereby the retinal and extraretinal signals are dynamically weighted and combined to form the visuomotor drive. The size and the dynamics of the weights are determined by the reliabilities of the respective signals. The model simulations suggest that retinal and extraretinal signals are dynamically weighted for pursuit and corroborate the behavioral observations that extraretinal signals play a larger role during early stages of pursuit and that retinal signals play a larger role during the steady state of pursuit. 

Acknowledgments

ARB was funded by the CODDE ITN Network from the European Community's Seventh Framework Programme FP7/2007-2013 under grant agreement number 214728-2. AM and GM were supported by the FACETS (Sixth Framework, IST-FET-15879) and the BrainScales (VIth Framework, IST-FEF-269921) European IP Projects. We thank A. DeMoya, J. Baurberg, and T. Iherti for their excellent technical support. We thank Laurent Perrinet, Laurent Madelain, Andrew Meso, and our colleagues of the InViBe team for their helpful suggestions. We would also like to express our gratitude to the two anonymous reviewers who have consistently contributed, with thoughtful and focused remarks, to the improvement of this manuscript. 

Commercial relationships: none. 

Corresponding author: Amarender R. Bogadhi. 

Email: [e-mail]. 

Address: Institut de Neurosciences de la Timone, CNRS & Aix-Marseille Université, Marseille, France; Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health, Bethesda, MD, USA.