Movements of the eyes and hands are coordinated by a common predictive strategy

Abstract

Although attempts to intercept a ball in flight are often preceded by predictive gaze behavior, the relationship between the predictive control of gaze and the effector is largely unexplored. The present study was designed to investigate the influence of the spatiotemporal demands of the task on a switch to the predictive control. Ten subjects immersed in a virtual environment attempted to intercept a ball that disappeared for 500 ms of its parabolic approach. The timing of the blank was varied through manipulation of the post-blank duration prior to the ball's arrival, and the shape of the trajectory was manipulated through variation of the pre-blank duration. Results reveal that the gaze movement trajectory during the blank was curvilinear, appropriately scaled to the curvature of the invisible moving ball, and the gaze vector was within 4° of the ball upon reappearance, despite 10° to 13° of ball movement. The timing of the blank did not influence the accuracy of predictive positioning of the paddle at the time of ball reappearance, indicated by the distance of the paddle relative to the ball's eventual passing location. However, analysis of trial-by-trial covariations revealed that, when the gaze vector more accurately predicted the ball's trajectory at reappearance, the paddle was also held closer to the ball's eventual passing location. This suggests that predictive strategies for paddle placement are more strongly mediated by the accuracy of gaze behavior than by the observed range of trajectories, or the timing of the blank.

Introduction

Visually guided action involves both online and predictive components of control. For example, when attempting to catch a ball in flight, the control of hand position can be understood as an online coupling to visual sources of information available throughout the ball's trajectory (Zhao & Warren, 2014). However, investigations of the gaze behavior of individuals attempting to intercept a ball as it moves in depth suggest that this online component of control is accompanied by eye movements made in prediction of the ball's future trajectory (Cesqui, Mezzetti, Lacquaniti, & D'Avella, 2015; Diaz, Cooper, & Hayhoe, 2013; Diaz, Cooper, Rothkopf, & Hayhoe, 2013; Hayhoe, McKinney, Chajka, & Pelz, 2012a; Land & McLeod, 2000; Mann, Spratford, & Abernethy, 2013). Under certain conditions, the predictive component can have a strong influence on factors leading to a successful interception of a target moving in two dimensions (Fooken, Yeo, Pai, & Spering, 2016), perhaps due to extraretinal contributions from smooth pursuit before visual feedback about the moving target's position is removed (e.g., through occlusion or target blanking). 

This study has been designed to investigate the factors that mediate the strength of the relationship between visual prediction and movements of the hand and body when attempting to intercept an object moving in depth. Although prediction may be accomplished through a “prospective” coupling of behavior to visual information that forecasts a likely future state (Montagne, 2005), this study more specifically focuses on predictive behavior that is separated in time from the sensory information that informed the control strategy. Our approach to studying prediction was to immerse subjects in a virtual reality (VR) environment in which the task was to use a handheld, motion-tracked badminton paddle to intercept a launched ball moving along a parabolic flight over a distance of approximately 20 m to a location within the subject's reach (Figure 1). Upon successful interception, the virtual ball would stick to the paddle for a brief duration before the end of the trial, providing visual feedback about the accuracy of paddle placement before the ball would disappear, and the next trial would begin. To promote predictive control strategies for gaze and the hand/paddle, the ball was made invisible (or “blanked”) for 500 ms on each trial during its parabolic flight toward the subject. The duration between the ball's reappearance and its arrival at the subject or, post-blank duration was limited to 500, 400, or 300 ms. 

Figure 1

 

(Left) Subjects wore an Oculus DK2 with integrated SMI eyetracker with a sampling rate of 60 Hz. (Right) The experimenter's desktop view of what the subject saw inside the helmet. The lines receding in depth represent the left and right gaze vectors, and the text in the upper left encodes trial number, block number, and timestamps. The red disc represents the face of the paddle, on which can also be seen the red virtual ball.

Our first hypothesis is that participants will engage in visual prediction through the blank period. This assumption is supported by several studies of gaze behavior prior to interception in naturalistic conditions (Barnes & Collins, 2011; Diaz, Cooper, & Hayhoe, 2013; Diaz, Cooper, Rothkopf et al., 2013; Hayhoe & Ballard, 2014; Hayhoe, McKinney, Chajka, & Pelz, 2012b; Land & McLeod, 2000). The accuracy of the visual prediction will compare the position and velocity of the subject's gaze behavior at the end of the 500 ms blank to the position and velocity of the ball at the same frame: at the time of reappearance, prior to the availability of post-blank visual information, and shortly before the attempted interception. 

Our second hypothesis is that the participant's strategy for positioning the paddle during interception will transition from an online mode of control to a predictive mode of control when the post-blank duration is shortest, and there is very little time for online control after the ball reappears. This hypothesis is motivated, in part, by the finding that, when visual feedback is removed upon the initiation of a movement to intercept, subjects' transition between online and closed loop control strategies is modulated by the quality of prediction (Faisal & Wolpert, 2009; Mazyn, Savelsbergh, Montagne, & Lenoir, 2007). In the context of natural(istic) interception tasks, the quality of prediction degrades quickly when the target is occluded for the final 280 ms of its flight or longer (Mazyn et al., 2007; Sharp & Whiting, 1974). Within the context of the present study, a prediction of the post-blank ball trajectory would have to be made on the basis of pre-blank visual information that is at least 500 ms old, and thus performance based on a prediction through the blank period is expected to be degraded relative to natural, online control. It follows that it would be advantageous to reestablish an online control strategy after the blank if the post-blank duration is sufficiently long for motor planning and execution on the basis of post-blank visual information. If this is not possible, subjects may adopt a more predictive mode of control on the basis of pre-blank visual information. This would be evident if more dramatic paddle movements occur during the blank, and if the relative distance of the paddle from the ball's eventual passing location at the end of the blank period is lower when the blank occurs later in the trial. 

Our third hypothesis is that visual and motor strategies are driven by shared resources. If so, we predict that predictive movements of gaze and the paddle will covary on a trial-by-trial basis. Similar signs of visuomotor coordination were apparent in a real-world interception task involving a carefully calibrated ball-launching device (Cesqui et al., 2015). Although the duration of tracking covaried with interception performance on a trial-by-trial basis, the effects were lost when averaging within conditions. Although the authors speculated that the influence of visual prediction on the subsequent movement to intercept is contingent upon the spatiotemporal demands of the task, the experiment was not designed to specifically test this hypothesis. In the present experiment, the spatiotemporal task demands are modulated by varying the timing of the blank relative to the ball's arrival, which is accomplished through manipulation of post-blank duration. Therefore, if visual and motor strategies involve shared resources, we would expect errors to be coupled at low values of post-blank duration, when the task places high temporal demands upon the participant, and elicits a more predictive mode of control for both gaze and the paddle. 

It is notable that, because the blank duration is fixed, reductions in post-blank duration will also have the effect of reducing the ball's overall time of flight, and bring about a straighter (less curved) ball trajectory. As a result, any conclusions concerning the role of visual prediction would be confounded by qualitative changes in the ball trajectory. To test the contribution from changes in the ball's trajectory to visual prediction, the duration between launch of the ball and the onset of the blank period, or pre-blank duration, varied between three values, between trials. These changes to the pre-blank duration modify the characteristics of the ball's trajectory independently of post-blank duration. Combinations of trajectories are presented in Figure 2. 

Figure 2

 

A side-view of the trajectories used in the experiment, which were comprised of a blank period of 500 ms (thick regions), a pre-blank duration of 600, 800, or 1,000 ms, and a post-blank duration of 300, 400, or 500 ms before the ball's passage over the X axis upon which the subject was standing. Thick regions along the trajectories represent the timing of the blank period. Overlapping blank periods reflect the fact that some trajectories have a common overall time-of-flight despite differences in pre-blank duration and post-blank duration, resulting in two possible times at which the blank may occur. Although ball trajectories in the experiment could have approached from a variety of angles, this representation assumes a single approach angle for visual simplicity.

In summary, this experiment has been designed to investigate the role of visual prediction in guiding the subsequent paddle movement in a VR interception task in which the ball is “blanked” for 500 ms as it moves in depth. We have suggested the following hypotheses: (1) that subjects will engage in visual prediction through the blank period, (2) that subjects will engage in predictive movements of the paddle only when the spatiotemporal demands of the task prevent online control, (3) that when subjects are engaged in both visual and motor control, their movements will demonstrate correlated errors between the modalities. 

Methods

The ten participants (seven male, three female) were between 19 and 30 years of age and had normal vision in the absence of visual correction. This study was approved by the Institutional Review Board at the Rochester Institute of Technology. 

Experimental apparatus

Stimuli were delivered by an Intel i7-based PC (Intel, Santa Clara, CA) with an NVIDIA GTX 690 connected to the Oculus Rift DK2 head-mounted display (Oculus VR, Irvine, CA) at 75 Hz, and an NVIDIA GTX 760 connected to the experimenter's desktop display (Acer, San Jose, CA). The computer ran Windows 7 (Microsoft, Redmond, WA), and the virtual environment was rendered using the Vizard Virtual Reality toolkit by WorldViz (Santa Barbara, CA). Physics were simulated using the OpenODE physics engine so that ball trajectories matched those expected within a real-world environment in the absence of wind resistance (Figure 1). Collisions between the ball and the paddle or other surfaces was also detected using the physics engine. To improve the fidelity of the physical simulation and collision detection, the physics engine was updated 10 times between frames (i.e., 750 Hz). 

The Oculus Rift HMD has an approximate field of view of 100°, and an approximate angular resolution of 10 to 15 pixels/° (dependent upon the position of the eye inside the head-mounted display). Head and paddle position/orientation were sampled and recorded at 75 Hz using a 14 camera PhaseSpace X2 motion capture system (Phasespace Inc, Pleasant Hill, CA), with a measured latency between the time a sensed movement would be reflected on the display of less than 30 ms. Eye movements were recorded with an SMI binocular eye tracker (SensoMotoric Instruments, Teltow, Germany) sampled at 75 Hz. A post-hoc correction was applied, as described in (Binaee, Diaz, Pelz, & Phillips, 2016), to correct for helmet slippage and other sources of spatial error in the eye tracking signal. The average eye tracking accuracy after calibration and correction was 0.53° for the central visual field [field of view (FOV) < 10°] and 2.51° in the periphery (10° < FOV < 30°). 

Experimental design

Participants were instructed to perform the interception task using their dominant hand, and all participants self-selected to hold the paddle with their right hand. All subjects conformed to the verbal instructions to catch, and not to hit the ball, using the paddle. Upon collision, the ball would stick to the virtual paddle for a brief duration before the end of the trial, providing visual feedback about the accuracy of paddle placement before the ball would disappear, and the next trial would begin. 

The red virtual ball was launched from a plane that was 6 m wide × 1.5 m high and parallel to both the virtual room's X-axis and the vertical axis. The launched ball's trajectory was calculated so that, if the ball were not intercepted, it would pass through a location randomly selected from a 1 m × 1 m plane near the subject, also parallel to the room's X-axis and the vertical axis. Trajectories consisted of a pre-blank duration of flight (600, 800, or 1000 ms), a 500 ms blank duration, and a post-blank duration of 300, 400, and 500 ms before the time at which the unimpeded ball would pass the X axis at which the subject was standing. This design produced nine combinations of pre- and post-blank duration, and seven possible flight-durations (Figure 2). Each subject performed 135 ball catching trials. 

Data preparation and analysis

Data preparation and non-statistical analysis of gaze and paddle movement data was conducted with Python 2.7 and 3.4, with modules Numpy, Pandas for computation, and a mixture of Bokeh and Plotly for figure generation. 

Analysis began by averaging the left/right eye-in-head vectors provided by the SMI eye tracker to produce a single unit vector that represents the cyclopean gaze direction. The transformation matrix of the motion-tracked head was applied to the vector to cast its appropriate location in front of the head in the virtual world. Data was passed through a median filter (length 3). The smoothed velocity signal was calculated with a third-order Savitzky-Golay filter with a window size of 5. 

Catching error

Catching error was calculated as the distance of the ball from the center of the paddle at the moment that the ball either collided with the paddle, or the moment that the ball passed by the vertically oriented plane located at the face of the paddle and orthogonal to the ball's trajectory. 

The influence of the timing of the blank period on catching behavior

Special steps were taken during the design phase to facilitate an investigation of whether the timing of the blank period within a particular trajectory affected behavior. This was made possible because both flight distance and blank duration are constant across all trial types. As a result, the shape of the ball's trajectory is determined by overall time of flight, which is equal to the sum of the pre-blank duration, blank duration, and post-blank duration. The design was such that there were two “paired” conditions that were identical in time of flight (and thus ball trajectories), but that differed in the timing of the blank period. These two pairs are both indicated in Figure 2 by the third and fifth lines from the top; for each pair, the partially overlapping bold portions of the trajectory indicate the two possible timings of the blank period within the otherwise identical trajectories. The first pair shared an overall time of flight of 1,800 ms, and included (1) the condition with a pre-blank duration of 1,000 ms and a post-blank duration of 300 ms, and (2) the condition with a pre-blank duration of 800 ms, and a post-blank duration of 500 ms. The second pair shared an overall time of flight of 1,600 ms, and included (1) the condition with a pre-blank duration of 800 ms and a post-blank duration of 300 ms, and (2) the condition with a pre-blank duration of 600 ms, and a post-blank duration of 500 ms. 

Visual prediction error

To measure the accuracy of the subject's prediction of the ball's displacement at the time of reappearance, we measured the angular distance between the gaze vector and the unit vector extending from the eye to the ball (i.e., the eye-to-ball vector) at the last frame of the blank period. 

Pursuit gain

If subjects adopted the strategy of engaging in visual pursuit/tracking of the ball across the blank period and leading up to an attempted interception, the speed of rotation of the gaze vector around the cyclopean eye would be well matched to the speed of rotation of the vector extending from the cyclopean eye to the ball around the cyclopean eye at frame of ball reappearance. Pursuit gain represents the ratio of these two values, where a ratio less than one would indicate that the gaze vector was rotating more slowly than the ball. The angular velocities of the gaze-in-world and eye-to-ball vectors were measured by calculating the vector distance between subsequent samples of the filtered signal, and then dividing the duration per frame, yielding a measure of velocity in degrees per second. 

Polynomial complexity of the gaze and ball's trajectory during the blank

Predictive strategies proposed in the literature span a continuum from those that rely upon complex internal models of object dynamics, to simplistic heuristic strategies that operate in the absence of stored priors (Zhao & Warren, 2014). In the present task, the complexity of the predictive gaze behavior demonstrated during the blank can provide evidence that may help disambiguate between these strategies for prediction. For example, if gaze trajectories are linear even in the presence of non-linear ball trajectories, it would suggest that either a simple heuristic strategy is sufficient to guide behavior given the constraints of the task. In contrast, a complex, non-linear gaze movement that is well matched to that of the (invisible) ball would suggest the influence of a learned model of target movement. 

To assess the complexity of predictive gaze behavior during the blank, first- through fourth-order polynomial models were fit to the trajectory of the gaze vector through spherical space during the blank. Models were constrained so that the intercept aligned with the azimuth and elevation of the initial gaze sample at the time that the ball disappeared. To assess whether increasing the polynomial order was justified, the winning model was selected on the basis of adjusted R², a measure of goodness of fit that avoids overfitting by incorporating a penalty for additional model terms. To assess whether the curvature of the gaze vector was well suited to that of the ball, the same procedure was applied to the ball's trajectory through head-centered spherical space, and the polynomial orders of the best fitting models for gaze and the ball were compared on a trial by trial basis. 

Paddle velocity ratio

A measure of relative paddle velocity during and after the blank is used to assess whether subjects guided paddle positioning in an online manner on the basis of post-blank visual information, or whether the paddle is positioned in a predictive manner, during the blank period. The mean paddle velocities for the two periods are calculated and then expressed as a ratio. The post-blank period is defined as the region from when the ball reappeared, until the earlier of two events: the moment that the ball either collided with the paddle, or the moment that the ball passed by the vertically oriented plane located at the face of the paddle and orthogonal to the ball's trajectory. 

Predictive catching error

The measure of predictive catching error is used to assess the accuracy of the positioning of the paddle at the moment of ball reappearance relative to the ball's post-blank trajectory. Paddle position is sampled at the time of ball reappearance. The ball's trajectory is then extrapolated until it collides with the vertical plane that passes through the center of the paddle and that is orthogonal to the ball's trajectory. The predictive catching error is the two-dimensional distance along this plane from the center of the ball to the center of the paddle, measured in meters. 

Statistics

Univariate analyses of variance were conducted using JASP version 0.9.1, and violations of sphericity were subjected to Greenhouse-Geisser correction. Prior to analysis, normality of the residuals was assessed through use of the Shapiro-Wilk test, and no violations were found. Significance was assessed using an alpha-level of 0.05 with Bonferroni correction for family-wise error rates. 

Mixed linear models were used to assess the covariation of the continuous variables of predictive catching error (pce) and visual prediction error. The mixed linear model is presented in equation 1:  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}\sqrt {pi{e_{si}}} = {\beta _0} + {\beta _1}{\rm{vp}}{{\rm{e}}_{si}} + {\beta _2}{\rm{pr}}{{\rm{e}}_{si}} + {\beta _3}{\rm{pos}}{{\rm{t}}_{si}} + {\beta _4}{\rm{vp}}{{\rm{e}}_{si}} \times {\rm{pos}}{{\rm{t}}_{si}} + {S_{0s}} + {\varepsilon _{si}}\end{equation}

 

Heteroskedasticity and normality were assessed through visual inspection of the residuals (Zuur, Ieno, Walker, Saveliev, & Smith, 2009), and these violations were mitigated via box-cox analysis (Box & Cox, 1964) and square-root transformation of the latent response variable, predictive interception error (pie). Model terms represent the categorical fixed effects of pre- and post-blank duration (pre and post), the continuous predictor visual prediction error (vpe), and the interaction of post-blank duration with visual prediction error. All values varied with item i and subject s. Mixed model analysis was conducted in R Studio version 1.136. The model was implemented using function lmer of the package LME4. Significance tests for fixed effects were run using the package lmerTest, and the related figures were generated using the library ggplot. 

An identical model was used to evaluate the relationship between predictive catching error and the pursuit gain (pg), and this model is presented in Equation 2. In response to similar signs of heteroskedasticity, the response variable received the same square-root transformation prior to model fitting.  

\begin{equation}\tag{2}\sqrt {pc{e_{si}}} = {\beta _0} + {\beta _1}{\rm{p}}{{\rm{g}}_{si}} + {\beta _2}{\rm{pr}}{{\rm{e}}_{si}} + {\beta _3}{\rm{pos}}{{\rm{t}}_{si}} + {\beta _4}{\rm{p}}{{\rm{g}}_{si}} \times {\rm{pos}}{{\rm{t}}_{si}} + {S_{0s}} + {\varepsilon _{si}}\end{equation}

 

The contribution of each fixed effect was evaluated through a comparison between models of increasing complexity. For example, the first test of the model presented in Equation 1 involved a comparison of a model which included only the random effect of subject to a model that also included the fixed effect of visual prediction error. The significance of the fixed effect was evaluated using a chi-squared likelihood ratio test. In addition, we required that, for a model to be accepted as superior, it must also reduce the Akaike information criterion (AIC)—a widely accepted metric used to assess whether the increase in model fit is justified by the inclusion of the additional terms. If the term passed these criterion its effect was considered significant, and the term was incorporated into the final model. 

Results and Discussion

Interception rates

Interception was more difficult at shorter post-blank durations

Interception rates are represented in Figure 3. Rates were lowest at shorter post-blank durations, when the ball appeared latest. A comparison between “paired” conditions that were identical in trajectory (as indicated by time of flight), but that differed in the timing of the blank period, reveals that subjects had a more difficult time intercepting the ball when it reappeared later. Figure 3B presents two of these pairs. Within the pair that shared a time of flight of 1,800 ms, the condition in which the ball reappeared later (pre: 1,000 ms, post: 300 ms) had an interception rate near 50%. In contrast, the condition with an earlier blank period (pre: 800 ms, post: 500 ms) had an interception rate near 80%, t(9.00) = 3.43, p < 0.004, r = 0.75. Within the pair that shared a time of flight of 1,600 ms, subjects caught approximately 40% of balls with a later blank period (pre: 800 ms, post: 300 ms) and above 70% for balls with an earlier blank period (pre: 600 ms, post: 500 ms; t(9.00) = 2.93, p < 0.01, r = 0.7). Overall, this suggests that subjects had a more difficult time intercepting the ball when the blank period occurred closer to its arrival, and thus the ball reappeared closer to the participant's head (Figure 3A). A comparison between identical trajectories that differ only in the timing of the blank suggested that this effect could not be attributed to the shape of the trajectory or the ball's time of flight (Figure 3B). 

Figure 3

 

(A) Interception rate by pre- and post-blank duration. Error bars represent 95% confidence intervals reflecting within subject variability. (B) A comparison of interception rate for the two pairs of conditions within which the trajectory is constant, but timing of the blank differs (see also Figure 2).

Gaze behavior

Figure 4 presents trajectories of the ball and gaze through head-centered spherical space in which the azumithal plane passed through the subject's eye and parallel to the ground plane, and the elevation plane passed through the gaze vector the world's vertical axis. This figure is meant to be representative of the ball trajectories and typical subject behavior. During and shortly after the blank period, gaze is characterized by a combination of smooth pursuit and catch-up saccades. Visual tracking behavior ends prior to the ball's arrival, during the ball's high-velocity movement through spherical space. 

Figure 4

 

The trajectory of the ball and gaze vector through head-centered spherical space. These representative trials are from one subject for each combinations of pre-blank duration of 600 or 1,000 ms, and post-blank duration of 300 or 500 ms. The visible portion of the ball's trajectory is represented in blue, and the portion of the ball's trajectory for which it is invisible is in cyan. Gaze is represented by the red trajectory, and the portion of the gaze trajectory when the ball is invisible is in pink. Gray lines connect samples of gaze/ball trajectory taken on the same frame. Trajectories move from left to right and end at the moment the ball either hits the paddle, or passes by a plane at the paddle's face.

Subjects compensated for the ball's angular displacement during the blank

Figure 5 presents the angular distance between the gaze vector and the ball's position throughout the last portion of the ball's trajectory, broken down by the post-blank duration. Regardless of condition, mean error remained below 4° throughout the blank period. Analysis of the ball's displacement (Figure 6A) reveals that this error reflects partial compensation for movement of the ball during the blank. The magnitude of angular ball displacement during the blank period ranged from 10.4° to 12.6°. Figure 6B presents the angular distance between the gaze vector and the edge of the ball at time of its reappearance (i.e., visual prediction error). This measure suggests that movement of the gaze vector during the blank resulted in a mean end-point accuracy of approximately 2.5° to 4° upon the ball's reappearance (Figure 6B). Neither main effects of pre-blank duration, post-blank duration, nor the interaction reached statistical significance [pre: F(2, 18) = 1.17, p = 0.33, Display Formula\(\eta _{\rm{p}}^2\) = 0.12, post: F(2, 18) = 0.64, p = 0.64, Display Formula\(\eta _{\rm{p}}^2\) = 0.05, Interaction: F(2.12, 19.03) = 1.02, p = 0.41, Display Formula\(\eta _{\rm{p}}^2\) = 0.10]. 

Figure 5

 

Mean angular distance between the gaze vector and the ball over the course of each trial type. Trials within each condition are aligned by the start and end of the blank duration, as indicated by vertical dashed lines. Shaded regions indicate 95% confidence intervals with between-subjects error removed.

Figure 6

 

(A) Angular displacement of the ball as it moved around the subject's head during the 500 ms blank period for which the ball was invisible. (B) Angular distance between the gaze vector and the ball at the time of ball reappearance. Error bars indicate 95% confidence intervals with between-subjects error removed.

Pursuit gain dropped below 1 for balls that were moving quickly upon reappearance

Angular velocity of the ball around the head is presented in Figure 7A. This figure shows that, at lower values of post-blank duration, the ball was moving at higher angular velocities. This is reasonable when one considers that, even if a ball that is moving at a fixed velocity through Euclidean space, if it reappears later and closer to the subject's head it will be moving more quickly through head-centered spherical space (measured in degrees azimuth/elevation). Consequently, and as is shown in Figure 7B, subjects had a more difficult time pursuing the ball at lower values of post-blank duration. In contrast, subjects were generally able to scale pursuit velocity to the ball's velocity at higher values of post-blank duration, when the ball reappeared earlier and farther away from the subject's head, F(1.25, 11.27) = 19.86, p < 0.001, Display Formula\(\eta _{\rm{p}}^2\) = 0.69. Following Bonferroni correction for family-wise error, neither the main effect of pre-blank duration nor the interaction reached significance, F(2, 18) = 4.23, p = 0.031, Display Formula\(\eta _{\rm{p}}^2\) = 0.32; F(4, 36) = 1.35, p = 0.24, Display Formula\(\eta _{\rm{p}}^2\) = 0.15. Post-hoc tests revealed differences between 300 ms and 400 ms (p < 0.011), 300–500 ms (p < 0.003), and 400–500 ms (p < 0.01). 

Figure 7

 

(A) Angular velocity of the ball around the head upon reappearance. (B) The ratio of angular velocity of the gaze vector and the eye-to-ball vector upon ball reappearance, or pursuit gain. Values reflect the average ratio within a 100 ms window centered upon the time of ball reappearance. The horizontal line indicates unity gain. Error bars represent 95% confidence intervals, with between-subject variability removed.

Curvature of gaze during the blank was tailored to the stimulus

Polynomial models of fit to the trajectory of the gaze vector during the blank reveal that prediction was not a simple linear extrapolation of the ball's movement before the blank. Figure 8 presents an example of two trials in which models varying in their polynomial complexity were fit to the gaze trajectory. For each trial, the “winning” model was selected on the basis of adjusted R², a measure of goodness of fit that penalizes for the incorporation of additional terms, and overfitting. For example, because the fourth order polynomial model presented in the top row of Figure 8A provides a better overall fit to the gaze data, one would expect a higher R²; however, the adjusted R² is slightly lower than the second-order polynomial model. 

Figure 8

 

The trajectory of the ball and gaze through spherical space for two representative trials. Several polynomial fits that vary in complexity are overlaid upon the gaze trajectory. (A) A trial in which gaze was best approximated by a second-order polynomial. (B) A trial in which gaze was best approximated by a third order polynomial.

To assess whether the curvature of the gaze trajectory was appropriately matched to the curvature of the ball, the same procedure was subsequently applied to the ball's trajectory through spherical space, and trials were binned in a two-dimensional histogram according to the complexity of the polynomial models needed to account for both gaze and ball trajectory (Figure 9). For example, consider that for a trial in which the pre-blank duration was 600 ms, and the post-blank duration was 300 ms, the trajectory of the ball was best approximated by a first-order polynomial model, and the trajectory of gaze by a fourth-order polynomial. This trial would contribute one count to the top-right bin of the upper left subplot. Brightness reflects the cumulative sum of all counts within a single bin, normalized within each combination of pre- and post-blank duration. 

Figure 9

 

A 2D histogram that represents the complexity of the polynomial models needed to model the trajectory of the gaze vector and the ball through spherical space (as in Figure 8). Cell brightness reflects the probability of each bin's associated polynomial pair, normalized within each combination of pre- and post-blank duration. In addition, numerical insets show the median and 95% confidence interval of adjusted R² for a subset of bins.

The concentration of data points along the second-order bin on the vertical (ball) axis in all subplots of Figure 9 reveals that the ball's trajectory through spherical space is best fit by a second-order polynomial in all conditions. Similarly, the curvature of the gaze trajectory is best explained by a second-order polynomial model in all conditions. Together, this reveals that that the gaze trajectory during the blank period was non-linear and appropriately complex given the non-linear trajectory of the ball as it moved through spherical space. 

Summary of gaze behavior

Analysis of the stimulus and of gaze behavior as balls moved through head-centered, spherical space revealed that subjects regularly engaged in visual prediction during the blank period. Prior to and during the blank period, the subject maintained less than 4° of distance between the gaze vector and the vector extending from the eye to the ball. During the blank period, this low-error was made possible by accurately accounting for the curvature of the ball's path on the basis of pre-blank visual information. Although positional accuracy was maintained across conditions, pursuit gain dropped from values near to 1 to values near to 0.7 when the ball reappeared later in the trial, as the result of a later blank period. This likely reflects increased task demands; when the blank period occurs later in the trial, when the ball is closer to the head, it will undergo a greater displacement during the blank, and be moving at a greater velocity upon reappearance. 

Paddle placement and movement kinematics

The speed of the paddle relative to its arrival for a subset of conditions is represented in Figure 10. Visual inspection of these velocity profiles suggests that the speed of paddle movement was greater after the blank when the blank occurred earlier relative to the ball's arrival (i.e., at longer post-blank durations; right column of Figure 10). In addition, paddle speed was greatest during the blank for the condition with the shortest time-of-flight, when the ball disappeared later in its trajectory (pre = 600 ms, post = 300 ms, top-left in Figure 10). These figures indicate that the subject chose to move the paddle during the blank, possibly on the basis of a prediction formed from pre-blank visual information. 

Figure 10

 

Paddle velocity relative to the ball's original time to contact. Vertical dashed lines indicate the onset/offset of the blank period. Shaded regions indicate 95% confidence intervals with between-subjects error removed.

To investigate whether mid-blank paddle movements were more dramatic at low values of post-blank duration, when conditions expected to elicit more predictive behavior, we investigated the average ratio of paddle velocity during the blank and post-blank period (Figure 11A). In all situations except the condition with the shortest time-of-flight (pre: 600 ms, post: 300 ms), the mean velocity was higher during the post-blank period than during the blank period (i.e., the ratio was <1). Although the main effect of pre-blank duration was not significant, F(1.3, 11.7) = 4.10, p < 0.058, Display Formula\(\eta _{\rm{p}}^2\) = 0.728, the main effect of post-blank duration was significant, F(2, 18) = 24.115, p < 0.001, Display Formula\(\eta _{\rm{p}}^2\) = 0.728, as was the interaction of pre and post-blank duration, F(1.8, 16.5) = 8.59, p = 0.003, Display Formula\(\eta _{\rm{p}}^2\) = 0.488. Post-hoc tests reveal that the value of pre-blank duration of 600ms and a post of 300 ms was significantly different than all other data points (p < 0.001 for all, except (p < 0.015 pre/post of 800 ms/300 ms, assessed using Tukey's honestly significant difference). This effect suggests that, when subjects foresaw that a late blank period would leave little time for post-blank adjustments to paddle position, they initiated more dramatic movements of the paddle during the blank period. 

Figure 11

 

Measures of paddle positioning and movement timing indicate the relative role of predictive movements, and predictive placement of the paddle. (A) The trial-by-trial ratio of mean paddle velocity during the blank to the post-blank velocity. Values greater than one represent a higher mean paddle velocity during the blank period. (B) Predictive interception error represents the distance from the paddle, sampled at the time of ball reappearance, to the ball's calculated passing point.

Figure 11B presents the distance of the paddle, sampled at the time of the ball reappearance, from the ball's eventual passing location. Neither main effects nor their interaction were significant following Bonferroni correction [Pre: F(1.27, 11.38) = 5.329, p = 0.034, Display Formula\(\eta _{\rm{p}}^2\) = 0.372, Post: F(2, 18) = 4.41, p = 0.028, Display Formula\(\eta _{\rm{p}}^2\) = 0.329, F(4, 36) = 1.151, p = 0.348, Display Formula\(\eta _{\rm{p}}^2\) = 0.113]. This suggests that the parameters of the ball's trajectory did not affect the overall accuracy of the paddle's positioning at the time of ball reappearance. 

Error bars represent 95% confidence intervals, with between-subject variability removed. 

Summary of paddle placement and kinematics

Analysis of paddle velocity during the blank and post-blank portions of the trial indicate that participants generally moved the paddle more quickly after ball reappearance than during the blank. The only time in which subjects moved the paddle more quickly during the blank was when the blank period occurred later in the trajectory (a pre-blank of 1,000 ms, and a post-blank of 300 ms). Analysis of paddle positioning at the time of ball reappearance did not reveal an effect of post-blank duration on predictive paddle placement. Thus, analyses provide only weak support for the second hypothesis that subjects will engage in more accurate prediction when the spatiotemporal demands of the task prevent online control. 

Predictive movements of gaze and the paddle covary

A linear mixed model was used to investigate hypothesis 3: that when subjects are engaged in both visual and motor control, they will demonstrate correlated errors between the predictive placement of the eyes and paddle on a trial-by-trial basis. If so, this would be indicative of shared resources, such as a shared predictive representation. Model fits are overlaid upon the data in Figure 12, and model parameters are presented in Table 1. 

Figure 12

 

Each subplot presents the relationship between visual prediction error (abscissa), and predictive interception error (ordinate) by post-blank duration (column) for each subject (row). Each point represents a single trial. Colors represent pre-blank duration, and solid lines present the fit from the model presented in Equation 1 for the range of actual values of visual prediction error.

Table 1

 

Effect of fixed and random effects for two models that included a random effect of subject, and fixed effects of pre-blank duration, post-blank duration, and the interaction of post-blank duration. In addition, Model 1 included the continuous predictor of visual prediction error (vpe), and model 2 pursuit gain (pg). Notes: Std. = standard; Num. obs. = number of observations; Num. groups = number of groups.

 

Comparison of nested linear models fit to predictive interception error revealed a main effect of visual prediction error, χ²(1.00) = 570.69, p < 0.001, dAIC = −58.49; post-blank duration, χ²(2.00) = 650.07, p < 0.001, dAIC = −154.75; pre-blank duration, χ²(2.00) = 680.92, p < 0.001, dAIC = −57.71; and the interaction of post-blank duration with visual prediction error, χ²(2.00) = 696.11, p < 0.001, dAIC = −26.36. As such, all terms were included in the final model. 

Although the quality of the model fit may vary by subject and condition, the trends indicated by model slopes are greatest at the post-blank level of 300 ms. In contrast, model slopes at 500 ms (right-most panels) indicate a weaker relationship between visual and motor prediction errors. These results support the hypothesis that visual-motor coordination is greatest when the ball reappears later in the ball's trajectory (at a post-blank duration of 300 ms). 

A similar linear mixed model was used to investigate for a relationship between visual pursuit gain and predictive interception error, where both measures were sampled at the time of ball reappearance. Comparison of nested linear models revealed a main effect of pursuit gain, χ²(1.00) = 570.69, p < 0.001, dAIC = −58.49; post-blank duration, χ²(2.00) = 615.72, p < 0.001, dAIC = −86.05; pre-blank duration, χ²(2.00) = 629.34, p < 0.001, dAIC = −23.24; and the interaction of post-blank duration with pursuit gain, χ² (3.00) = 705.25, p < 0.001, dAIC = −145.83. As such, all terms were included in the final model. Model fits are presented in Figure 13. Slopes were negative, indicating a drop in pursuit gain was accompanied by an increase in predictive interception error (Table 1). The higher gain values reflect lower ball speeds at ball reappearance and an increase in pursuit speed toward the end of the blank period. 

Figure 13

 

Each subplot presents the relationship between pursuit gain (abscissa), and interception error (ordinate) by post-blank duration (column) for each subject (row). Each point represents a single trial. Colors represent pre-blank duration, and solid lines present the fit from the model presented in Equation 2 for the range of actual values of interception error.

Summary of hypotheses and results

Interception rates suggest that the task is most difficult at low values of post-blank duration, and that this effect cannot be attributed to the shape of the trajectory. An inspection of gaze behavior offers a possible explanation. 

Consistent with hypothesis 1, subjects moved gaze in prediction of the ball's trajectory across the blank, in all conditions. The trajectory of gaze during the blank was curvilinear, appropriately scaled to the curvature of the ball as it moved around the head, and brought the gaze vector within 4° of the ball upon reappearance, despite 10° to 13° of ball movement. When the blank period occurred later in the trial, the ball would undergo greater displacement during the blank, and would be moving at a greater velocity upon reappearance. This resulted in a drop in pursuit gain that may explain the subject's difficulties in intercepting the ball. 

Analysis of paddle kinematics revealed weak evidence for the second hypothesis that subjects would switch to a more predictive mode of control at shorter values of post-blank duration, when there was little time to resume online control after ball reappearance. This conclusion is based upon the findings that mid-blank adjustments to the paddle were greatest in magnitude when the blank occurred later in the trial. 

The results supported hypothesis 3, that when participants were engaged in predictive control strategies for both gaze and the paddle (i.e., when post-blank duration was 300 ms), the two modalities would demonstrate correlated errors. Indeed, linear models reveal that visual prediction error and pursuit gain both covary with predictive interception error. This is indicative, but not conclusive evidence, for shared mechanisms behind the predictive control strategies guiding movements of gaze and the paddle. 

General discussion

This study provides valuable new insights into the role of prediction in naturalistic control of gaze and hand movements, and builds upon a growing body of literature suggesting that, although predictive control strategies may not be readily apparent in the analysis of navigation or effector placement (Zhao & Warren 2014), predictive mechanisms are fundamentally involved in the control of eye movements in visually guided action in natural or naturalistic contexts (Diaz, Cooper, & Hayhoe, 2013; Diaz, Cooper, Rothkopf, et al., 2013; Land & McLeod, 2000; Mann et al., 2013; McKinney, Chajka, & Hayhoe, 2008; Tuhkanen et al., 2019). What is less clear, however, is how predictive gaze control is related to predictive control strategies for controlling the effector (e.g., the paddle). The present task was designed specifically to elicit prediction across both gaze and the effector, and as a result it provides new evidence concerning the relationship between the two modalities. Unsurprisingly, we found that gaze was consistently controlled in prediction of the ball's curvilinear trajectory through space while it was “blanked” for a brief duration leading up to an interception. Although predictive placement of the paddle was not mediated by the timing of the blank duration within the ball's trajectory, predictive control of paddle placement was apparent in the analysis of trial-by-trial covariations between the positioning of the gaze vector and the paddle: when the gaze vector accurately predicted the ball's location at reappearance, the paddle was also held closer to the ball's eventual passing location. Because co-variation of gaze and paddle placement was present at the time of ball reappearance, it cannot be explained by the availability of post-blank visual information. Because this trial-by-trial co-variation was present within a single condition, its presence cannot be explained by changes in the timing of the blank. Together, these findings reveal that predictive strategies for paddle placement were more strongly mediated by the accuracy of gaze behavior than the range of variations in ball's trajectory adopted in this experiment. 

What does the observed coupling between the predictive control of gaze and paddle position tell us about their underlying relationship? Because co-variation between gaze and paddle positioning was found when both modalities were sampled simultaneously, at the time of ball reappearance, it is not possible that the accurate visual prediction provided better (post-blank) visual information that causally led to better paddle placement. The alternative explanation is that the covariation is the result of a common cause, however the present data cannot distinguish between the possibility that the observed covariation reflects a shared upstream neural mechanism, or two largely independent mechanisms that rely upon common sources of visual information. The data reveal that the strength of predictive covariation between the two modalities is mediated by the spatiotemporal demands of the task. This seems to suggest a transition from online to predictive modes of control that is mediated by post-blank duration, and thus presumably by the motor noise inherent in last-minute adjustments to paddle position. The reliability-mediated switching between online and predictive control strategies has also been demonstrated experimentally by coupling target disappearance to the initiation of movement in a two-dimensional (2D) disc interception task (Faisal & Wolpert, 2009) or in an interception task with real balls (Mazyn et al., 2007). In both contexts, subjects had the choice to make a slow controlled movement to the predicted location of the invisible target, or to initiate movement late in the ball's trajectory, with increased motor noise. Subjects demonstrated a preference for a ballistic visually guided movement that maximized the duration of the visible portion of the ball's flight, and minimized the role of prediction. Thus, predictive strategies appear to be adopted only when online control is prevented or degraded by experimental conditions. A similar proposal has been made by Belousov et al. (Belousov, Neumann, Rothkopf, & Peters, 2016), who modeled transitions between predictive and online control strategies when controlling a run to intercept a target, parameterized on the basis of observation noise, reaction time, and task duration. 

The finding that interception rate is heavily influenced by the timing of the blank period contradicts a recent study by López-Moliner, which suggests no effect of the timing of an occlusion upon catching performance (López-Moliner, Brenner, Louw, & Smeets, 2010). Gaze behavior was not measured. The difference can likely be attributed to differences in the types of trajectories used. In the study by López-Moliner, trajectories were thrown gently by the experimenter from a distance of about 75 cm to the approximately location of the catcher's hand, and the visual scene was entirely occluded for 250 ms of the ball's flight. In contrast, the trajectories experienced in our study spanned 20 m, balls were in the air for approximately 1.7 seconds, and passing location was randomized within a 1 × 1 m plane. These differences suggest our task may have made it more difficult for subjects to predict the ball's final arrival location and, as a result, exaggerated the impact of the timing of the blank window upon interception performance. Because balls in our experiment were not thrown, subjects did not benefit from advanced visual information present in the movement of the thrower. Alternatively, differences in trajectory may have affected the visual information upon which subjects relied to predict the ball's future trajectory (Malla & Lopez-Moliner, 2015). 

It is interesting to note that visual prediction may be more evident in data from subjects who had a more difficult time with the task. The role of individual differences is consistent in studies on visually guided behavior, and may provide a potential explanation for athletic expertise (Yarrow, Brown, & Krakauer, 2009). In fact, differences in eye movements have been found between individuals that vary in levels of expertise in multiple ball sports, including a simple interception task (Cesqui et al., 2015), cricket (Land & McLeod, 2000; Mann et al., 2013), baseball (Bahill, Adler, & Stark, 1975), juggling (Dessing, Rey, & Beek, 2012; Huys, Daffertshofer, & Beek, 2004), table tennis (Rodrigues, Vickers, & Williams, 2002), and tennis (Ripoll & Fleurance, 1988; Singer et al., 1998). Fooken et al. (2016) found that the experience level of baseball players predicted skill level in a 2D target interception task in which subjects pointing at discs that disappeared while moving across the fronto-parallel plane. Although the present study was not designed to address the factors contributing to the individual differences in gaze behavior and prediction, it is an area that may be addressed in the future. 

Acknowledgments

The authors would like to thank Dejan Draschkow for lending his expertise in the area of mixed effects modeling. 

Commercial relationships: none. 

Corresponding author: Kamran Binaee. 

Email: . 

Address: Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY, USA.