Reinforcement effects in anticipatory smooth eye movements

Abstract

When predictive information about target motion is available, anticipatory smooth pursuit eye movements (aSPEM) are consistently generated before target appearance, thereby reducing the typical sensorimotor delay between target motion onset and foveation. By manipulating the probability for target motion direction, we were able to bias the direction and mean velocity of aSPEM. This suggests that motion-direction expectancy has a strong effect on the initiation of anticipatory movements. To further understand the nature of anticipatory smooth eye movements, we investigated different effects of reinforcement on aSPEM. In a first experiment, the reinforcement was contingent to a particular anticipatory behavior. A monetary reward was associated to a criterion-matching anticipatory velocity as estimated online during the gap before target motion onset. Our results showed a small but significant effect of behavior-contingent monetary reward on aSPEM. In a second experiment, the proportion of rewarded trials was manipulated across motion directions (right vs. left) independently from participants' behavior. Our results indicate that a bias in expected reward does not systematically affect anticipatory eye movements. Overall, these findings strengthen the notion that anticipatory eye movements can be considered as an operant behavior (similar to visually guided ones), whereas the expectancy for a noncontingent reward cannot efficiently bias them.

Introduction

Natural environments offer myriad potentially relevant visual targets at the same time. Target selection is, therefore, one of the main functions of voluntary eye movements. To do so, humans and primates, in general, use saccades to quickly foveate objects of interest. In the case of smoothly moving objects, the oculomotor system can also elicit smooth pursuit eye movements that stabilize the retinal image of the selected target on the fovea, minimizing the motion blur that would otherwise alter the perception of the moving object. 

Historically, the pursuit system has been regarded as a servo system in which the delayed retinal slip first drives smooth eye movements throughout an open-loop phase (Robinson, 1986), i.e., before the visual feedback has time to affect the ongoing movement. Then, during a closed-loop phase, information about the actual eye movement is integrated into the oculomotor controller through a positive feedback loop, virtually enabling the eye movement to exactly match the target movement (Lisberger, Morris, & Tychsen, 1987; Tychsen & Lisberger, 1986). However, several observations suggest that such simple servomechanisms do not capture the whole richness of smooth pursuit (e.g., G. R. Barnes, 2008; Bogadhi, Montagnini, & Masson, 2013; Madelain & Krauzlis, 2003a, 2003b; Orban de Xivry et al., 2013; Perrinet, Adams, & Friston, 2014). In particular, it has been established that smooth pursuit eye movements can be strongly modulated by perceptual and cognitive factors (for reviews, see G. R. Barnes, 2008; Spering & Montagnini, 2011), such as visual attention (Berman et al., 1999; Chen, Holzman, & Nakayama, 2002; Culham et al., 1998; Ferrera & Lisberger, 1995; Keller & Khan, 1986; Khurana & Kowler, 1987; Souto & Kerzel, 2008), object-related processing (Ilg & Churan, 2004; Stone, Beutter, & Lorenceau, 2000) or illusory motion perception (Beutter & Stone, 2000; Lorenceau, 2012; Madelain & Krauzlis, 2003b; Montagnini, Mamassian, Perrinet, Castet, & Masson, 2007; Stone et al., 2000; Spering, Kerzel, Braun, Hawken, & Gegenfurtner, 2005; Wallace, Stone, & Masson, 2005). Importantly, and contrary to the conventional notion that some visual motion input (or, to a lesser extent, some motion input from another sensory modality) is necessary to initiate smooth pursuit, the predictability of the target's motion can lead to anticipatory initiation of pursuit before target motion onset or to the maintenance of smooth eye movements during transient occlusions of the moving target (Badler & Heinen, 2006; G. R. Barnes & Asselman, 1991; Becker & Fuchs, 1985; Bennett & Barnes, 2006; Collewijn & Tamminga, 1984; Freyberg & Ilg, 2008; Kowler, 1989; Kowler, Martins & Pavel, 1984; Kowler & Steinman, 1979a, 1979b; Landelle, Montagnini, Madelain, & Danion, 2016; Madelain & Krauzlis, 2003a; Orban de Xivry, Missal, & Lefèvre, 2008; Wells & Barnes, 1999). 

It has been recently proposed that oculomotor control strongly depends on reinforcement contingencies such that eye movements may adapt to environmental variable constraints (see Madelain, Paeye, & Darcheville, 2011, for review). There are numerous studies on reinforcement and saccadic eye movements, but only a handful of studies probed the sensitivity of smooth pursuit to reinforcement contingencies. In newborn infants (1–7 days), Darcheville, Madelain, Buquet, Charlier, and Miossec (1999) found that smooth eye movements can be enhanced using an auditory reinforcer contingent to the eye velocity. Schütz, Lossin, and Gegenfurtner (2015) showed that, in presence of two moving textures, pursuit direction choice is mainly determined by the target's visual salience at initiation, but 300–400 ms after motion onset, pursuit steered toward the rewarded direction, and the salience effects gradually disappeared. On the other hand, Joshua and Lisberger (2012) reported an influence of precued reward contingencies already on vector average pursuit initiation in monkeys. Finally, using a single moving target, Brielmann and Spering (2015) have shown that human participants' smooth pursuit is affected by reward expectation across the whole time course, whether the stimulus–reward association is precued or not. Moreover, reinforcement can also affect pursuit in the absence of a visual target. During the transient blanking of a moving target, human subjects can still elicit smooth tracking but with a natural reduction by 40%–60% of the normal eye velocity (Becker & Fuchs, 1985). Madelain and Krauzlis (2003a) successfully trained participants to maintain high gain pursuit while the visual moving target was transiently occluded with contingencies such that a reward (signaled by a contingent continuous auditory tone during pursuit) was given, depending on whether the eye velocity accurately matched the hidden target velocity. 

Here we investigate the effects of learning on a particular phase of human smooth pursuit, namely the anticipatory smooth pursuit eye movements (aSPEM) that can be observed ahead of target motion onset under particular conditions. Indeed, when temporal or spatial cues before target motion onset make the forthcoming motion properties predictable to some degree, anticipatory smooth tracking movements are generated. Several studies have previously looked at the influence of cognitive factors on anticipatory smooth pursuit (G. R. Barnes & Asselman, 1991; de Hemptinne, Lefevre, & Missal, 2008; Heinen, Badler, & Ting, 2005; Kowler, 1989; Kowler et al., 1984; Kowler & Steinman, 1979a, 1979b; Van den Berg, 1988; Wells & Barnes, 1999). It is now commonly accepted that anticipatory pursuit is strongly impaired if the expected motor response has not been previously experienced (G. Barnes, Grealy, & Collins, 1997) and that the efficiency in eliciting anticipation can vary dramatically depending on the type of cues (Kowler, Aitkin, Ross, Santos, & Zhao, 2014). Regularities in the target motion properties also strongly affect anticipatory smooth pursuit (Kowler et al., 1984; Kowler & Steinman, 1979a, 1979b; Montagnini, Souto, & Masson, 2010; Santos & Kowler, 2017). However, none of these studies has probed the extent of the effects of reinforcement contingencies on anticipatory smooth eye movements. 

In this paper, we used eye-movement recordings as behavioral measures in an eye velocity–contingent operant conditioning paradigm specifically designed to target aSPEM. We report effects of monetary reinforcement contingencies on anticipatory smooth eye velocity and confirmed these with a yoked-control procedure (Wasserman, 2010). We also aimed at further clarifying the role of operant versus nonoperant components of reward-related behavioral changes by implementing an associative learning procedure (Pavlov, 1927) and comparing it to results from the operant conditioning paradigm. Overall, our results suggest that anticipatory eye movements are affected by reinforcement contingencies in addition to other cognitive and predictive processes. 

Methods

Subjects

Nineteen human subjects (10 females and nine males, aged 23–43) participated in Experiment 1; two of them were excluded from further analyses due to over-noisy eye-tracking recordings. The remaining 17 participants were also assigned to three other experiments. Some of them participated in several experiments, depending on their availability. Seven participants were involved in Experiment 2, four in the yoked control condition, and six in Experiment 3. Two of the participants were authors of the study for Experiment 1 and one for Experiment 3, whereas all other participants were naïve as to the experimental conditions and goals of the studies. Naïve subjects received a fixed amount of money for their participation. However, for Experiment 2 (both for the main condition and the yoked control), they were instructed at the beginning of the experiment that the sum they would receive would depend on their performance. All subjects were healthy and had normal or corrected-to-normal vision and had no relevant medical or psychiatric history. The experiments were conducted in accordance with the ethical regulations of the Centre National de la Recherche Scientifique after obtaining the approval of the ethical committee of Aix-Marseille Université (approval no. 2014-12-3-05) for behavioral noninvasive research and in accordance with the Declaration of Helsinki. Written informed consent was obtained from all participants. 

Apparatus

Stimuli were presented on a 21-in. CRT monitor with a refresh rate of 100 Hz. Stimuli were generated using the Psychophysics Toolbox extension for MATLAB (Brainard, 1997; Pelli, 1997) running on a Mac Pro (first-generation) computer and displayed at a viewing distance of 57 cm against a gray background. The luminance of the gray background was 42 cd/m². To minimize measurement errors, the subject's head movements were restrained using a chin and forehead rest so that the eyes in primary gaze position were directed toward the center of the screen. Monocular eye movements were measured continuously with an eye-tracking system (Eyelink 1000, SR Research Ltd., sampled at 1,000 Hz), and data were transferred, stored, and analyzed off-line using programs written in MATLAB or Ipython notebooks. 

Data analysis

Recorded horizontal and vertical gaze positions were low-pass filtered using a Butterworth (acausal) filter of order 2 with a 30-Hz cutoff frequency and then were numerically differentiated to obtain velocity measures. We used an automatic conjoint acceleration and velocity threshold method to detect saccades (Krauzlis & Miles, 1996) and further inspected all individual traces visually to exclude aberrant trials. Mean eye velocity traces were obtained by trial averaging. For this purpose, saccades were excluded from the data by tagging them as “nonvalid,” and averaging was computed over valid data points. We evaluated the effects of the experimental manipulations on anticipatory velocity at the individual level by comparing the mean eye velocity during a temporal anticipation window between −50 and +50 ms around target motion onset (as highlighted in Figure 2), using individual one-way ANOVAs (with the probability bias or reinforcement type as three-level single factors). We also computed all post hoc pair-wise comparisons using Tukey's honestly significant difference (HSD) test. Even though the normality assumption of our data was fairly justified given the sample's size (always more than 60 trials per condition), we also computed nonparametric statistics (Kruskal–Wallis test) for comparison, and results were similar. All tests and analyses have been realized using Python 3.5 with the Numpy and Scipy libraries. 

Experiment 1: Motion-direction bias task (baseline)

The visual stimulus used in Experiment 1 was a white ring (0.30° outer diameter and 0.23° inner diameter) with a luminance of 102 cd/m² that moved horizontally on a gray background. Each trial started with a central fixation point displayed for a random duration drawn from a uniform distribution ranging between 300 and 600 ms. Then a fixed-duration 300-ms gap occurred between the offset of the fixation point and the onset of the moving target, which was presented at the fixation location and immediately started moving horizontally at a constant speed of 7.7°/s either to the right or the left for 1,000 ms. Across experimental sessions, the probability P of rightward trials was manipulated in order to create a direction bias and favor the buildup of direction expectancy (See Figure 1A). The experiment included three sessions. The first session had P = 0.50, hence about 200 rightward trials out of 400 trials; the second one had P = 0.75 (about 300 rightward trials out of 400 trials); the last one P = 0.90 out of a total of either 400 (360 rightward) or 600 trials (540 rightward; respectively, nine and 10 participants). The rationale for having a larger number of trials in the P = 0.90 session was to collect enough trials for the least frequent direction, namely the left one, but after running the first participants, we realized that the detailed analysis of the least-frequent trials did not provide, per se, a major advantage for the present study because the anticipatory behavior was clearly observable with the standard 400 trial-sessions. As we wanted to test a condition without any direction uncertainty, four participants also completed an additional session of 400 rightward trials only (P = 1.00). With the exception of the P = 1 condition, target motion direction was pseudorandomized across trials once for each session type, such that all participants were presented with the exact same sequence of randomly alternating directions. Participants were instructed to track the target as accurately as possible. 

Figure 1

 

Presentation of the experimental conditions. (A) Experiment 1, motion-direction bias task (baseline). After a random-duration fixation followed by a 300-ms gap interval, a small ring-shaped target moved from the center of the screen toward the left or the right side with constant velocity for 750 ms. The probability P of rightward motion direction was manipulated across experimental blocks. (B) Experiment 2, operant-conditioning pursuit tasks. Similar to Experiment 1, but the color of the moving target was dependent on the comparison of the online estimation of the eye velocity during the critical comparison window (between −100 ms and motion onset, see schematic illustration in the figure inset, which also summarizes the different rules for reward delivery) with an adaptive criterion velocity. The color of the target also informed about the money gain or loss (green target = gain, red target = loss) for the current trial. (C) Experiment 3, associative learning pursuit task. Similar to Experiment 1, but a given proportion of trials in each direction was associated with a monetary gain (green target, rewarded trial) regardless of the participant's oculomotor behavior, and the remaining fraction of trials (white target) was neither rewarded nor punished. In this experiment, the target motion direction was never biased.

Figure 2

 

Average horizontal smooth eye velocity across trials for rightward and leftward target motion direction (solid and dashed curves, respectively) and for different direction-bias sessions (color-coded; see figure legend). The solid vertical black line marks target motion onset. The dashed vertical lines mark the limits of the anticipation window, which was used to estimate the mean anticipatory velocity in each trial.

Experiment 2: Operant conditioning of aSPEM

The visual stimuli used in Experiment 2 were two rings (0.30° outer diameter and 0.23° inner diameter), one was green (luminance of 90 cd/m²) and the other red (luminance of 22 cd/m²). For each reinforcement condition (see below), there were three sessions in which the target direction bias and the timeline of each trial was very similar to the baseline experiment, but here we applied an eye velocity–contingent reinforcement schedule based on anticipatory smooth pursuit. We defined a 100-ms critical time window for anticipatory pursuit starting 100 ms before motion onset. Note that this window (which we refer to as a critical comparison window) is slightly different from the previously defined anticipation window (between −50 and 50 ms) as the comparison between the eye velocity and the criterion has to occur strictly before target motion onset in order to enforce a causal association with the reward cue (coincident with the target's color). The mean online eye-velocity estimate in the critical comparison window was compared to a velocity criterion. The criterion velocity was defined as the median of the anticipatory velocity estimated over the 20 previous trials. The criterion was, therefore, constantly evolving and depended on the participant's anticipatory behavior during the recent past. However, depending on the actual experimental condition, the criterion was further constrained to only increase or decrease (see below). For the first 20 trials of each experimental session, the criterion was defined as the median velocity of each participant during the corresponding (bias-matched) baseline session. We implemented two reinforcement conditions that were administered to each of the seven participants in separate sessions (see details on the order of presentation below): 

•  
  The first condition was the “anticipation booster,” whereby we rewarded increased anticipatory velocities. If the online anticipatory eye velocity estimate was higher than the criterion, then the green ring stimulus was displayed as a moving pursuit target. If it was smaller than the criterion, then a red ring target was displayed (see Figure 1B). The instructions given to participants were to accurately track the moving target regardless of its color and that participants were going to obtain 0.30 euros for each trial with a green ring and lose 0.10 euros for each trial with a red ring. Every 20 trials, a text message appeared on the screen stating whether the cumulative earnings were increasing or decreasing. Importantly, the velocity criterion was, by construction, only allowed to increase or remain constant but could never decrease in this condition.
•  
  The second condition was the “anticipation braker” in which decreased anticipatory velocities were rewarded. When the online anticipatory velocity estimate was smaller than the criterion in a given trial, we gave a reward (green moving target and +0.30 euros), and if it was higher, we gave a punishment (red moving target and −0.10 euros). In this experimental condition, the criterion could either decrease or remain constant but could never increase.

It should be noted that every saccade during the critical comparison window (defined on the basis of online estimated mean velocity exceeding 70°/s) was followed by a red target (thus, “punished”) whether in the booster or braker condition. 

We further tested a third condition, namely a yoked-control procedure, whereby a reward or punishment was provided on each trial regardless of the actual smooth pursuit performance. In practice, the yoked procedure was used to isolate any effects generally due to rewards but not to the contingency between anticipatory behaviors and rewards or punishments. Similarly to the baseline, we used the three sessions with a probabilistic bias for the target motion direction. All of the four participants of the yoked-control group had already participated in Experiment 1 and two of them also in the reinforcement conditions. Following the standard procedure for yoked control, the rate of reward for the participants of the yoked condition was randomly selected among the reward rates estimated for four other randomly matched participants of the reinforcement conditions. The instructions given to the participants were identical as in the booster/braker conditions despite the fact that, this time, the outcome was not causally related to the anticipatory behavior. Again, saccades occurring in the critical comparison window were punished. To closely stick to the rate of reward given in the matched reinforcement condition regardless of the individual number of anticipatory saccades, we readjusted in an online fashion the number of rewarded and punished trials of the yoked condition accounting for the saccade-based punished trials. 

Experiment 3: Associative learning and aSPEM

To investigate in more detail the nature of reward-related effects on anticipatory smooth pursuit and, in particular, to clarify the role of the operant-conditioning procedure, we implemented an associative learning procedure in which a specific motion direction was associated with the reward. Participants were presented with three 400-trial sessions of the standard pursuit experiment without any direction bias, namely with P = 0.50 for rightward and leftward trials. We introduced a bias in the rate of rewarded trials for each target motion direction: In the first session, we pseudorandomly rewarded a proportion P_rew = 0.50 of rightward trials (and 1 − P_rew = 0.50 of leftward trials); in the second session, we pseudorandomly rewarded P_rew = 0.75 of rightward trials (and 1 − P_rew = 0.25 of leftward trials), and in the third session, we pseudorandomly rewarded P_rew = 0.90 of rightward trials (and 1 − P_rew = 0.10 of leftward trials; see Figure 1C). The instructions were the same as in Experiment 2. 

Experimental conditions: Order of presentation

The order of presentation of the different direction-bias conditions was fixed for Experiment 1. All participants were first tested with P = 0.5, then P = 0.75, and finally P = 0.90. Out of the seven participants in Experiment 2, four (s5, s6, s7, and s8) were first presented with the booster condition followed by the braker condition. Three participants (s16, s17, s18) experienced the braker condition first, followed by the booster condition. In Experiment 2, the order of direction bias was fixed for most participants (first P = 0.5, then P = 0.75, then P = 0.9, and finally P = 1, when this last condition was implemented) with the exception of participants s7 and s8 who underwent a different order of presentation: first P = 0.90, then P = 0.5, and finally P = 0.75 for s7; first P = 0.75, then P = 0.5, and finally P = 0.90 for s8. Finally, for Experiment 3, the order of presentation of the different conditions was fixed; namely participants were first provided with P_rew = 0.50 of rightward rewarded trials, then P_rew = 0.75 and last P_rew = 0.90. 

Results

Experiment 1: Direction-bias task (baseline)

In this experiment, we introduced an increasing proportion of rightward trials throughout the sessions. Figure 2 shows the average smooth (and “desaccaded”) eye velocity trace across trials for each direction-bias condition for a representative participant (s6) who experienced all direction-bias conditions. 

Figure 2 reveals a gradual tendency to use predictive information about future motion as the smooth velocity traces show increasing anticipatory behavior when the uncertainty about the direction of future motion decreases. These results nicely replicate previous findings obtained from our and other groups under similar experimental conditions (Montagnini et al., 2010; Santos & Kowler, 2017). Figure 3a shows the distribution of anticipatory eye velocity for participant s6 highlighting the gradual shift of the whole distributions across direction-bias sessions. We compared the mean velocities recorded during the anticipation window by means of individual one-way ANOVAs (with the probability bias as three-level single factor) and post hoc all pair-wise comparisons using Tukey's HSD test. Boxplots in Figure 3b illustrate the results at the group level (with interquartile range 0.25–0.75 and outliers marked as a +) summarizing the anticipatory velocity distributions for each session. We also plotted the individual mean anticipatory velocity across sessions for each participant. The general tendency observed for participant s6, i.e., the increase of anticipatory velocities with the increase of direction probability, holds for all subjects. We found a systematic significant effect of the direction bias (p < 0.05) with large size effect (given by Cohen's d statistic), except for participant s11. 

Figure 3

 

(a) Histogram and trial average (vertical line) of anticipatory smooth eye velocity for each direction-bias session for participant s6. (b) Boxplots of anticipatory eye velocity for the whole group of participants of Experiment 1 in all direction-bias sessions. Individual mean velocities and 95% confidence interval are also presented with black symbols connected by thin lines. The individual data points' abscissae are randomly jittered on the x-axis in order to avoid the superposition of the symbols.

Previous work (Heinen et al., 2005; Kao & Morrow, 1994; Kowler et al., 1984; Kowler & McKee, 1987) has highlighted the role of local sequential effect upon anticipatory smooth pursuit. One could speculate, for instance, that the simplest mechanism underlying the observed effect of direction-bias could be that anticipatory eye velocity is determined by the target direction in the previous trial. In the P = 0.9 bias condition, for instance, there are many more trials preceded by a rightward target motion than by a leftward one. Hence, if a rightward trial is systematically followed by rightward anticipatory pursuit (and the other way around for leftward trials), anticipation could be, on average, strongly biased, in the P = 0.9 bias condition, in favor of rightward velocity. A first observation that makes this possibility unlikely is that the distributions of anticipatory velocity (see Figure 3a for a representative participant) look consistently unimodal for all direction-bias conditions, whereas a bimodal shape would have been expected (with increasingly uneven peaks for increasing bias) if the previous trial's direction had completely determined the anticipatory pursuit velocity. Nonetheless, we tested whether the direction of target motion of the immediately preceding trial could at least partly explain the observed bias in anticipatory velocity by comparing the latter for trials pooled depending on the previous trial direction. Figure 4 shows the group boxplots for aSPEM velocity together with the individual data (thin black lines and symbols): An effect of the preceding trial on aSPEM appears, indeed, when the direction bias is above 0.5. However, the difference in anticipatory velocity due to the previous trial's direction (illustrated by the difference between light and dark blue boxes) is consistently smaller than the probability-bias effect across experimental blocks. A two-way repeated-measures ANOVA with previous trial direction (right/left) and direction bias (P = 0.5, 0.75, and 0.9) as factors corroborated this observation, indicating that both main factors and their interaction affect significantly the anticipatory velocity (p < 0.05). We come back to a systematic, model-based analysis of the trial sequence effects later in this manuscript. 

Figure 4

 

Boxplots of anticipatory eye velocity for the whole group of participants of Experiment 1 in all direction-bias sessions. Data are grouped depending on the target direction in the preceding trial. Individual mean velocities and 95% confidence interval are also presented with black symbols connected by thin lines. The individual data points' abscissae are randomly jittered on the x-axis in order to avoid the superposition of the symbols.

We examined the main parameters of visually guided smooth pursuit (visual pursuit latency, pursuit acceleration, steady-state velocity) and did not find any systematic effect of the direction-bias manipulation. The rate of visually guided saccades during target tracking did not systematically change across direction-bias conditions either. In contrast, the latency of the first visually guided saccade increased significantly with the direction bias. This finding may be explained by the fact that when the target motion direction is coherent with the direction of anticipatory smooth pursuit, the retinal slip at the initiation of visually guided pursuit is reduced, and so is the retinal spatial distance between the moving target and the fovea. The reduction of retinal velocity and position error is more pronounced when pursuit anticipation is stronger, and this may naturally increase the delay for the execution of the first catch-up saccade. 

Experiment 2: Operant conditioning paradigm

This experiment aimed at probing the effects of reward contingencies on anticipatory eye movement velocity. Figure 5 illustrates the mean ± 95% confidence interval of anticipatory eye velocity for each direction-bias session and the corresponding reinforcement condition for participant s6. Much as in Experiment 1, anticipatory eye velocities increased with the motion-direction probability. However, it appears that these changes also depend on the reward condition, in particular with the P = 0.9 and P = 1 direction bias. 

Figure 5

 

Mean and 95% confidence interval of the mean anticipatory smooth eye velocity in each direction-bias session and reinforcement condition of Experiment 1 for participant s6.

Even though the direction-bias effect seems to be dominant and the reinforcement effects are strongly variable across direction-bias conditions, aSPEM is generally larger in the booster versus the braker condition as summarized in Figure 6. We performed a one-way ANOVA analysis to test for statistical differences between reinforcement conditions for each direction-bias condition and for each individual participant. Significant differences were found in the P = 0.50 direction bias for all participants (p < 0.05). Post hoc, all pair-wise Tukey HSD analysis revealed that anticipatory velocities were significantly higher in the booster condition than in baseline condition for all participants (p < 0.05). However, and surprisingly, for participants s6, s16, and s17, the mean anticipatory eye velocity in the braker condition was significantly higher than in the baseline condition (p < 0.05). In the P = 0.75 direction-bias condition, no significant differences were found between reinforcement conditions for participant s6 (F = 0.307, p = NS), s17 (F = 2.393, p = NS), and s18 (F = 2.964, p = NS). Significant differences were found for participant s5 (F = 13.939, p < 0.05), s7 (F = 23.144, p < 0.05), s8 (F = 17.435, p < 0.05), and s16 (F = 8.328, p < 0.05). Post hoc, all pair-wise Tukey HSD analysis revealed higher anticipatory mean velocities in the booster compared to both the baseline and braker conditions for participants s5, s7, and s8 (p < 0.05) and braker anticipatory mean velocities significantly below baseline and booster velocity for participant s16 (p < 0.05). The other pair-wise comparisons were not significant. Finally, for P = 0.90 direction bias, we found statistically significant differences for all participants. Post hoc, both all pair-wise Tukey HSD and 95% confidence interval analyses showed anticipatory mean velocities systematically below baseline velocities in the braker condition (p < 0.05). We did not find significant differences between booster and baseline velocities for participants s8, s16, s17, and s18, but an effect of the booster condition was found for participants s5, s6, and s7. It is important to note, however, that, contrary to the direction-bias effect, the effect size values for the operant-conditioning modulation (given by the eta square value) would conventionally be considered as within a small size range. 

Figure 6

 

Boxplots of anticipatory eye velocity for the whole group of participants in all direction-bias sessions and reinforcement conditions of Experiment 1. Individual mean velocities and 95% confidence interval are also presented with black symbols connected by thin lines. The individual data points' abscissae are randomly jittered on the x-axis in order to avoid the superposition of the symbols.

The parameters of visually guided smooth pursuit did not differ significantly across reinforcement conditions. For instance, the rate of visually guided catch-up saccades did not vary systematically between the baseline and the booster or braker sessions. The latency of the initial catch-up saccade, in contrast, increased slightly in the booster (mean increase by 15 ms) and even more in the braker condition (mean increase by 40 ms) with respect to baseline. 

To further probe the effects of reward contingencies on smooth eye movement velocity during the anticipation window and rule out the possibility that those effects would be instead related to unspecific reward delivery, we implemented a yoked-control procedure in which rewards and punishments were not contingent on anticipatory smooth pursuit. If the operant nature of the anticipatory behavior is critical, then we would expect anticipatory velocity in the yoked group to be reduced with respect to the booster condition and enhanced with respect to the braker condition at least for the P = 0.90 direction bias. In other words, only small differences are expected between the data of the yoked-control group (in which reward was delivered randomly across trials) and the baseline condition. Figure 7 shows the mean ± 95% confidence interval of anticipatory eye velocity for each direction-bias session in all reinforcement conditions for participants s8 and s18 who participated in all conditions of Experiment 2. 

Figure 7

 

Mean and 95% confidence interval of the mean anticipatory smooth eye velocity in each direction-bias session for the two participants (s8 and s18) that experienced all reinforcement and yoked conditions.

Overall, we observe a decrease of aSPEM in the yoked condition with respect to the booster condition for all direction-bias sessions for both s8 and s18. For participant S8, the yoked anticipatory velocity was also reduced with respect to the baseline condition for all direction biases (P = 0.50: T = −4.953, p < 0.05; P = 0.75: T = −4.042, p < 0.05; P = 0.90: T = −11.646, p < 0.05). Participant s18 shows a significant reduction for P = 0.75 (T = −3.975, p < 0.05) but not for P = 0.50 (T = 0.893, p = NS) and P = 0.90 (T = 0.525, p = NS). Figure 8 allows us to compare anticipatory eye velocity in the yoked condition with the baseline condition for each of the four participants of the yoked procedure, highlighting a general mild reduction in aSPEM with some variability across participants. 

Figure 8

 

Boxplots of anticipatory eye velocity in the baseline and yoked-control condition for the group of yoked participants across all direction-bias sessions. Individual mean velocities and 95% confidence interval are also presented with black symbols connected by thin lines. The individual data points' abscissae are randomly jittered on the x-axis in order to avoid the superposition of the symbols.

In summary, the results of the yoked-control procedure suggest that the random delivery of reward does not increase anticipatory eye velocity: aSPEM velocities were systematically lower compared to the velocities in the booster condition. On the other hand, the present results suggest also an unexpected generic “attenuating” effect, slightly reducing anticipatory velocity compared to the baseline condition. Interestingly, somewhat similar findings of a possible negative effect of random delivery of reward have been reported in the learning literature (Thompson & Iwata, 2005; Twining, Bolan, & Grigson, 2009); we come back to this point in the discussion. 

Experiment 3: Associative learning

This experiment was designed to probe the effects of a response-independent reinforcement, whereby the reward is simply associated to the stimulus condition and independent from the oculomotor behavior. In other words, we test the hypothesis that anticipatory smooth eye velocity is modulated by the relative expectation to be rewarded when the target moves in a specific direction. Here, we had three sessions with equal probability of rightward and leftward target motion direction (P = 0.50) but different reward probabilities. One might expect to observe a proportional increase of anticipatory smooth eye velocity in a given direction as a function of the probability to obtain a reward when the target moves in that direction. The higher the directional bias for reward likelihood, the higher the expected anticipatory velocity. This effect has been observed only in three participants out of six but to different extents. Participant s19 had aSPEM velocities significantly above the baseline (P = 0.5) velocities in all conditions P_rew = 0.50 (T = 5.965, p < 0.05), P_rew = 0.75 (T = 6.147, p < 0.05), P_rew = 0.90 (T = 4.872, p < 0.05). The other two participants had a significant difference only in the P_rew = 0.90 condition (s5, T = 3.704, p < 0.05; s15, T = 2.353, p < 0.05). For the three other participants, no significant difference from the baseline condition was detected. These results suggest that the reward expectancy, per se, can indeed shape the anticipatory smooth pursuit in some participants but with a weak effect and stronger interindividual differences than operant conditioning. Figure 9 summarizes all results of Experiment 3 by means of boxplots of the mean anticipatory velocity recorded for all participants in both conditions. 

Figure 9

 

Boxplots of anticipatory eye velocity in the baseline and associative learning condition for the whole group of participants of Experiment 3 as a function of the probability of reward for rightward trials P_rew. Individual mean velocities and 95% confidence interval are also presented with black symbols connected by thin lines. The individual data points' abscissae are randomly jittered on the x-axis in order to avoid the superposition of the symbols.

Discussion

Our data has shown that subjects are able to use statistical regularities of motion features to produce consistent aSPEMs. We found a monotonic relation between the mean aSPEM velocity and the bias applied to motion direction. This robust effect was found in all participants in the baseline experiment with no exception. We probed the operant nature of this behavior by implementing an eye velocity–contingent paradigm in which rewards depended on aSPEM velocities. We observed an influence of the reward contingencies on aSPEM, albeit with smaller and less robust effects than the direction-bias, per se. We further probed the operant nature of aSPEM by implementing a yoked-control condition, delivering reward randomly with no systematic association to oculomotor behavior. This manipulation resulted in a slight decrease of aSPEM velocities. Finally, we implemented an associative learning paradigm by systematically associating a specific target direction to reward probability. Our data showed a much smaller effect and greater intersubject variability than in the operant paradigm. Overall, these results lead us to conclude that, although sensorimotor probabilistic information about target motion seems to be dominant in controlling aSPEM during the course of our experiments, anticipatory pursuit can also be modulated by reward, especially within an operant-conditioning framework. 

Integration of statistical regularities: The direction bias

Throughout the first experimental condition, we investigated the integration of environmental statistical regularities in a smooth tracking task with a family of direction biases for the target motion. We found a global and robust effect of direction bias on anticipatory smooth pursuit. These results are coherent with previous oculomotor findings by our and also other groups (Montagnini et al., 2010; Santos & Kowler, 2017; Souto, Montagnini, & Masson, 2008). Typically, aSPEM is observed after a temporal cue and ahead of target motion onset (Kowler & Steinman, 1979a, 1979b; Kowler et al, 1984), and it depends on the previous trials experience whether or not the latter provides statistically reliable information (Heinen et al., 2005; Kao & Morrow, 1994; Kowler & McKee, 1987). Importantly, even in highly predictable situations, the occurrence of fixation trials interleaved with tracking trials can perturb pursuit anticipation (Watamaniuk, Bal, & Heinen, 2017). Some classical experiments have demonstrated the existence of prediction-based smooth pursuit during the transient disappearance of a moving target (Becker & Fuchs, 1985). Overall, it is now clear that smooth pursuit behavior can be modulated even in the absence of online sensory stimulation. We have chosen to study aSPEMs in order to probe the sensitivity of expectancy-based oculomotor behavior to global (i.e., on a long time scale, several hundreds of trials) and local (on a short time scale, few trials) regularities and to investigate how sensorimotor expectancy interacts with reward contingencies in shaping oculomotor behavior without direct sensory feedback. In previous studies (Montagnini et al., 2010; Souto et al., 2008), we have analyzed how forthcoming motion properties, such as target speed or direction, can be predicted and anticipated with coherent orienting eye movements. We found a graded effect of both the speed and the direction bias with mean anticipatory eye velocity linearly related to the probability of motion's speed or direction. Here, we replicated part of those results, using a limited number of direction probability biases and strengthened them by generalizing them on a large pool of participants. These results imply that the probability bias over a target's direction is one additional factor beyond other physical and cognitive cues (Kowler et al., 2014; Santos & Kowler, 2017) that modulate the common predictive framework driving anticipatory behavior to optimize a rapid and precise foveation of the target on its most expected future path. 

Direction bias and reward contingencies

The plasticity of pursuit eye movements during tracking of a visual target has previously been studied, on both humans (K. Fukushima, Tanaka, Suzuki, Fukushima, & Yoshida, 1996; Ogawa & Fujita, 1997; Optican, Zee, & Chu, 1985; Takagi et al., 2000; Takagi, Trillenberg, & Zee, 2001) and monkeys (Bourrelly, Quinet, Cavanagh, & Goffart, 2016; Kahlon & Lisberger, 1996). Here, we wanted to investigate the reinforcement-based plasticity of predictive smooth pursuit. In particular, with Experiment 2, we aimed at strengthening and expanding an already established result by Madelain and Krauzlis (2003a) to the case of expectancy-based aSPEM. Madelain and Krauzlis (2003a) have established that the natural smooth pursuit deceleration observed during the transient disappearance of the tracked target could be considerably reduced by reinforcement contingencies in the absence of any visual feedback. Overall, we found differences between individual aSPEM velocities in operant-conditioning sessions compared to the corresponding baseline sessions. However, those differences become mostly significant with a P ≥ 0.90 direction-bias session, i.e., for the highest level of predictability we tested. Importantly, and contrary to our expectation, the anticipatory velocity was sometimes larger in the braker condition than in the baseline when P was lower than 0.90 (e.g., for participants s6, s16, and s17). One possible explanation of this can simply be an effect of training carried over across successive sessions. Indeed, all participants experienced the baseline condition prior to the other experimental conditions. This might have trained participants and increased their sensitivity to direction biases or modified their prior expectation of a target going rightward. Results obtained for the P = 0.90 direction-bias session confirmed our hypothesis regarding the effect of reinforcement on anticipation as all participants had lower aSPEM velocities in the braker condition compared to baseline, strengthening the notion that a potential task learning–related boost of aSPEM is not always sufficient to counteract the reduction of aSPEM due to reinforcement contingencies. Concerning the booster conditions, we generally found an increase of anticipatory velocity with respect to the baseline. However, such increase was significant only for participants s5, s6, and s7 in the P = 0.9 condition. Several factors may explain the lack of a more robust positive effect of reward contingencies in the booster condition for the strongest direction bias: First, the adaptive criterion for reinforcement was constantly evolving toward more demanding levels, such that, at some point, the required aSPEM velocity to obtain a reward became extremely high, plausibly close to the maximum of the anticipatory velocity range. Second, this increase in task demand might also have affected our participants' attention and motivation. Third, because trials with saccades in the critical comparison window were punished, at some point participants had presumably to actively inhibit excessively strong anticipation to avoid these saccades. When we looked at the rate of trials with anticipatory saccades across participants, we found indeed a larger rate in the baseline condition than in the booster condition with about 18% less anticipatory saccade trials in the latter. If this decrease suggests that the punishment criterion for saccades has been taken into account, participants might have had to cope with somehow conflicting aims: the need to anticipate accurately to provide fast foveation and get rewarded and also the need to keep anticipatory velocity below the threshold for saccade detection and punishment. 

Nonetheless, our results demonstrate overall that aSPEMs are sensitive to the rewarding value of a stimulus even when the reward is related to the nonvisually guided phase of pursuit rather than the visually guided phase (Brielmann & Spering, 2015; Joshua & Lisberger, 2012; Schütz et al., 2015) and that reinforcement contingencies have to be taken into account in addition to attentional factors (Khurana & Kowler, 1987) and prediction (G. R. Barnes, Barnes, & Chakraborti, 2000; G. R. Barnes & Donelan, 1999; Santos & Kowler, 2017; Wells & Barnes, 1998) for the buildup of prior expectation about a target's direction. 

Integration of environmental regularities across trials: A model-based analysis

Integrating information about the regularities in our environment is essential to build a general knowledge of it whether the nature of that information is statistical or reward-related. To accurately build up this integration, one must rely on the previous trials, but one might prioritize recent information with respect to very remote events. In a study on the sequential effects of target speed, Maus, Potapchuk, Watamaniuk, and Heinen (2015) proposed a simple model in which anticipatory smooth pursuit eye velocity is a linear function of the mean target speed across the recent trial history. In a similar way, we developed a model-based analysis that aims at elucidating the relation between anticipatory smooth pursuit eye velocity and local estimate of the direction bias. As we can see in Figure 10 (in which we pooled all our participants' data together), for an arbitrary 10-trial memory, there is indeed a noisy but significant linear relation between the estimated probability of rightward direction (defined as the mean of a binary function associating one to rightward trials and zero to leftward trials across the previous 10 trials) and the recorded anticipatory velocity. The analysis of the linear regression for all individual data pooled together and a memory size of 10 trials yields P = 0.50: R = 0.13, p < 0.05; P = 0.75: R = 0.21, p < 0.05; and P = 0.90: R = 0.24, p < 0.05, where R is the Pearson's correlation coefficient. 

Figure 10

 

Scatterplots of mean anticipatory velocity (data pooled from all subjects) and linear regression lines between anticipatory velocities and the estimated direction bias over the 10 past trials for the different direction-bias sessions. For an easier visualization, data corresponding to different direction-bias sessions are slightly shifted with respect to each other. Within-condition data points are further jittered on the x-axis in order to avoid massive superposition of points.

Similar to Maus et al. (2015), we repeated this analysis for several memory sizes to test how Pearson's correlation coefficient varies when taking into account longer chunks of the recent history. With our analysis of the trial-sequence effects, we also wanted to address the complementary question of whether the weight of recent trials is affected by the global direction probability or by the reinforcement contingency. One possibility is, for instance, that the association between reward and anticipatory behavior potentiates the effect of the trial sequence. We, thus, analyzed the linear regression across different direction-bias and reward conditions. As we can see in Figure 11, we replicated, with lower R however, the general tendency found by Maus et al. (2015) of a higher Pearson's R for smaller memory sizes. A very similar dependence of the correlation coefficient upon tau was observed across all direction-bias and reinforcement conditions. For memory size below 20, the direction bias does also systematically affect the Pearson's R with higher R for the stronger direction bias. 

Figure 11

 

Pearson's correlation coefficient R for the linear regression of anticipatory velocity (group-pooled data) and the local estimation of direction bias across all direction-bias and reinforcement conditions and for different memory-storage sizes (Tau).

Altogether, anticipatory eye velocity seems to be weakly correlated to the local estimate of the target direction bias, which relies on a short-term memory integration (∼10 trials). Further work and larger statistical power would be needed to address the question of whether reinforcement contingencies interact with this relation. At this stage, this linear regression explains only a small fraction of the variability in our data, and this analysis does not allow any firm quantitative conclusion. A more advanced dynamical model for the integration of statistical regularities in target motion direction might be better suited to explain the experimental data. 

Reward unpredictability

To further confirm that the effect observed in Experiment 2 was truly due to reinforcement contingencies we used the well-established yoked-control procedure (Wasserman, 2010). To our knowledge, the study by Madelain and Krauzlis (2003a) is the only one that reported a yoked-control procedure in a smooth pursuit task with reinforcement. Similar to Madelain and Krauzlis, the yoked condition did not result in any enhancement of anticipatory smooth pursuit velocity compared to the baseline condition, and the measured anticipatory velocity was reduced compared to the booster condition of the operant-conditioning procedure. However, we would have expected the distribution of aSPEM velocities in the yoked group to be statistically similar to the one in the baseline (like for the Madelain & Krauzlis study), but on three out of four participants, we obtained yoked-control mean anticipatory velocity slightly below the baseline. 

Although unexpected in the context of our study, similar counterintuitive results have been reported in the literature in other contexts. Some examples of the effects of delivering rewards independently from the organism's behavior can be found in the so-called superstitious experiments (Skinner, 1948): When putting food-deprived pigeons in mere food delivery situations (a 5-s access to food every 15 s), they may exhibit superstitious behavior, e.g., repeating specific responses (wings spreading, head shifting, etc.) that had been temporally correlated with food delivery as the training progressed. In a study with human participants, Ono (1987) used a single response independent schedule of reinforcement with variable times. He found that random reinforcers influenced human behavior and, in some cases, observed stable behavioral patterns. In summary, reinforcers in yoked-control conditions may influence the participant's behavior. Although this remains a speculative explanation, the variability of observed behaviors when reward delivery is independent from the response seems to be a plausible ground for the variability of the effects of our yoked manipulation. 

Stimulus-associated reward expectancy

To separate the effects of the association between target direction and reward from those produced by operant conditioning, we systematically delivered rewards in different proportions across the two target motion directions in the absence of a contingency between the motor action and the reward occurrence. This response-independent reinforcement paradigm addresses the effects of reward delivery associated to a particular stimulus condition separately from those of the response contingency. What we observed in our associative learning experiment is a higher variability and an overall smaller size of the effect on aSPEM than in an operant-conditioning paradigm. Interestingly, and similar to the present study, in a recent article on reward-cued saccadic eye movements, different and uncorrelated effects have been reported for response-contingent and noncontingent reinforcement (Manohar, Finzi, Drew, & Husain, 2017). Several factors can have influenced these results. In our operant paradigm, a punishment or a reward is delivered on each trial, depending on the aSPEM velocity. In our associative-learning paradigm, a reinforcer is given from time to time independently from the oculomotor response. This form of learning depends on a low-level substitution of one unconditioned stimulus that triggers the response by a conditioned one (Pavlov, 1927). Given the results we obtained in the operant and yoked procedure, one interpretation could be that aSPEMs rely on a higher cognitive process rather than a low-level associative process. Another explanation might be related to the absence of punishment as only rewards were given in the associative-learning experiment. Participants always finished the experiment with the highest amount of cumulative reward they can get regardless of their behavior, and this could have reduced their motivation according again to the Yerkes–Dodson law (Yerkes & Dodson, 1908) stating that a not enough challenging task will be passively executed. Although our results on the associative-learning paradigm cannot allow us to fully disentangle the operant and Pavlovian processes involved in aSPEM, they, however, suggest the dominance of operant components. 

Neural bases of contextual contingencies modulation in smooth eye movements

Extensive research has addressed, in the past, the neuronal bases of reward modulations of motor (and oculomotor) control. The basal ganglia have been described as specifically related to the effects of reward on general behavior (Joshua, Adler, Mitelman, Vaadia, & Bergman, 2008; Lauwereyns, Watanabe, Coe, & Hikosaka, 2002; Schultz, 1998) and on eye movements in particular (Hikosaka, 2007; Yoshida & Tanaka, 2009). Importantly, the basal ganglia have strong anatomical connections with the frontal eye fields (FEFs), a major visuo-oculomotor area in the prefrontal cortex and, in particular, with the subregion of the FEF that is selectively involved in smooth eye movement (FEFsem; Cui, Yan, & Lynch, 2003; Tian & Lynch, 1997). In addition, one of the major outputs of the basal ganglia is the superior colliculus, which is known for its critical role for saccadic control and which has been recently shown to be involved in smooth pursuit initiation and target selection (Nummela & Krauzlis, 2010, 2011). Thus, the cortico-subcortical network, including the FEF, the superior colliculus, and the basal ganglia (and possibly other regions not discussed here), is probably a good candidate as a major substrate of the reward-based modulations of smooth eye movements observed here and in previous studies. 

The neuronal pathways involved in anticipatory eye movements have also been studied, in both monkeys and humans. In nonhuman primates, the correct execution of cued, expectancy-based pursuit is related to selective neuronal activity in the supplementary eye fields (SEFs) in the dorso-medial frontal cortex (de Hemptinne et al., 2008; Missal & Heinen, 2004) as well as in the FEFsem (Keating, 1993; Raghavan & Joshua, 2017). Different neurons in the FEFsem seem to contribute to different phases of pursuit and learning in pursuit processes (Li & Lisberger, 2011; Schoppik, Nagel, & Lisberger, 2008), and local modulations of neuronal activity related to pursuit learning of instructed motion properties have been also reported in the cerebellum (Li, Medina, Frank, & Lisberger, 2011; Yang & Lisberger, 2010). The FEF is also involved in processing extraretinal pursuit components, during both transient target-blank tasks (K. Fukushima, Sato, Fukushima, Shinmei, & Kaneko, 2000; K. Fukushima, Yamanobe, Shinmei, & Fukushima, 2002; Tanaka & Fukushima, 1998) and memory-based pursuit (J. Fukushima et al., 2011). In humans, Gagnon, Paus, Grosbras, Pike, and O'Driscoll (2006) applied transcranial magnetic stimulation (TMS) to the FEFsem and SEF during sinusoidal pursuit to assess its effects on the pursuit response to predictable changes in target motion rather than unexpected ones. For the FEFsem, they found that TMS applied immediately before the target reversed direction increased eye velocity in the new direction, whereas TMS applied in midcycle (immediately before the target began to slow down), decreased eye velocity. For the SEF, TMS applied at target reversal increased eye velocity in the new direction but had no effect on eye velocity when applied at midcycle. 

In conclusion, understanding the neuronal bases of the anticipatory smooth eye movements is still an open challenge, and the important issue that is addressed by recent studies is to identify the internal signals that have access to anticipatory control (Bosco et al., 2015; Wolpert & Landy, 2012; Zhao & Warren, 2015), including reward signals. 

Conclusion

When moving stimuli are sequentially presented according to direction bias, observers adapt to the current bias by producing aSPEM in a range of velocity that helps in minimizing the foveation delay. When we add eye velocity–contingent reward beyond the sensorimotor contingencies, we manage to slightly but significantly modulate aSPEM following the eye velocity–contingent punishment/reward schedule in force. In case of inconsistency between oculomotor behavior and reward, aSPEM behavior undergoes a little reduction. Finally, an associative learning paradigm showed smaller effects and larger intersubjects variability than the operant conditioning. Taken together, these results suggest that aSPEM are a flexible behavior, relying primarily on perceptual and statistical cues, but are also modulated by reward-contingencies; smooth pursuit might be optimized for action consequences rather than passive delivery of reward. 

Acknowledgments

This work was supported by Agence Nationale de Recherche grant ANR-13-APPR-008 and EU Marie-Sklodowska-Curie Grant No 642961. We thank Jeremie Jozefowiez for commenting on an earlier version of the manuscript. 

Commercial relationships: none. 

Corresponding author: Anna Montagnini. 

E-mail: [email protected]. 

Address: Aix Marseille Université, CNRS, Institut de Neurosciences de la Timone UMR 7289, Marseille, France.