Temporal integration of focus position signal during compensation for pursuit in optic flow

Abstract

Observer translation results in optic flow that specifies heading. Concurrent smooth pursuit causes distortion of the retinal flow pattern for which the visual system compensates. The distortion and its perceptual compensation are usually modeled in terms of instantaneous velocities. However, apart from adding a velocity to the flow field, pursuit also incrementally changes the direction of gaze. The effect of gaze displacement on optic flow perception has received little attention. Here we separated the effects of velocity and gaze displacement by measuring the perceived two-dimensional focus position of rotating flow patterns during pursuit. Such stimuli are useful in the current context because the two effects work in orthogonal directions. As expected, the instantaneous pursuit velocity shifted the perceived focus orthogonally to the pursuit direction. Additionally, the focus was mislocalized in the direction of the pursuit. Experiments that manipulated the presentation duration, flow speed, and uncertainty of the focus location supported the idea that the latter component of mislocalization resulted from temporal integration of the retinal trajectory of the focus. Finally, a comparison of the shift magnitudes obtained in conditions with and without pursuit (but with similar retinal stimulation) suggested that the compensation for both effects uses extraretinal information.

Introduction

Forward locomotion results in an expanding pattern of optic flow of which the focus coincides with one's direction of motion (Gibson, 1950). When smooth pursuit eye movements are made at the same time, this one-to-one relation between focus position and heading direction disappears. The velocity of the eye adds a component of motion to the optic flow field that shifts the focus of expansion in the direction of pursuit. We call this the velocity summation effect because it depends on the sum of the instantaneous velocities in the flow field and that caused by the pursuit (Figure 1). Human observers accurately estimate their simulated heading direction from displays of optic flow, even while making concurrent smooth pursuit eye movements (for reviews, see Lappe, Bremmer, & van den Berg, 1999; Warren, 1998). This suggests that the visual system compensates for the effects of smooth pursuit. 

Figure 1

 

Two separate effects of smooth pursuit on optic flow. The instantaneous velocity field of optic flow that is viewed during smooth pursuit appears warped on the retina because of the addition of retinal slip velocity. We call this the velocity summation effect of smooth pursuit. This effect shifts the focus (C) of the flow field in a direction that depends on the type of flow, e.g., in the direction of the pursuit for expansion and orthogonal to the pursuit for rotation. In addition, the pursuit displaces the gaze over time (color coding), which results in a drift of the pattern over the retina. This gaze displacement effect of pursuit is in the same direction for all types of optic flow.

Many mechanisms of heading detection from optic flow during smooth pursuit have been proposed (for a review, see Hildreth & Royden, 1998). These can be distinguished by the source of information that is used to estimate the pursuit component of the flow: purely retinal information such as motion parallax, i.e., the relative retinal motion of elements at different depths (e.g., Gibson & Carmichael, 1966; Longuet-Higgins & Prazdny, 1980); extraretinal information about the eye movement, such as corollary discharge (Banks, Ehrlich, Backus, & Crowell, 1996; Royden, Crowell, & Banks, 1994; von Helmholtz, 1867; von Holst & Mittelstaedt, 1950; Wurtz, 2008); or a combination of retinal and extraretinal information (e.g., Beintema & van den Berg, 1998; Lappe, 1998). 

Common to these explanations is the use of the instantaneous velocity field as visual input, thus ignoring that during pursuit any world-fixed target (be it an optic flow field focus or otherwise) drifts over the retina with the same speed but opposite to the pursuit. Because the magnitude of this drift increases over the course of the pursuit, contrary to that of the velocity summation effect, we call this drift the gaze displacement effect (Figure 1). van den Berg (1999) showed that such a gaze displacement effect indeed leads to mislocalization of the focus of expanding optic flow in the direction of the smooth pursuit, i.e., opposite the direction of the retinal focus drift. He hypothesized that the mislocalization resulted from underestimating the amount of retinal focus drift. This underestimation was consistent with either a processing lag (latency) or a temporal integration of the position signal on the order of half a second. 

In the present study, we expand on those findings using a novel method that allows us to directly measure the velocity summation and the gaze displacement components of focus mislocalization. We will use this to investigate the roles of retinal and extraretinal information in the compensation and investigate whether a processing lag or temporal integration underlies the mislocalization resulting from gaze displacement. 

We performed a focus localization experiment in which not only expanding patterns were shown during smooth pursuit but also contracting and rotating ones. ¹ Rotating patterns are particularly interesting in this context because the instantaneous shifts that result from velocity summation are orthogonal to the pursuit direction (Bradley, Maxwell, Andersen, Banks, & Shenoy, 1996; Duijnhouwer, van Wezel, & van den Berg, 2008; Pack & Mingolla, 1998). In contrast, the incremental effect of gaze displacement is parallel to the pursuit direction for all types of flow pattern. We found a pattern of localization errors that was consistent with roles of both the velocity summation and the gaze displacement effects. 

To establish that mislocalization in the direction of pursuit was indeed due to gaze displacement, we varied the presentation duration of rotational optic flow patterns (Experiment 2). Because prolonged pursuit results in more gaze displacement, we predicted that localization errors in the direction of the pursuit would increase with increasing stimulus duration (Experiment 2). Conversely, increasing the flow speed (Experiment 3) should decrease the vertical mislocalization because it reduces the relative impact of the pursuit velocity on the retinal flow field. In addition, the hypothetical duration dependence of the mislocalization in the direction of pursuit in rotational flow may shed light on the distinction between the proposed lag and temporal integration mechanisms. With increasing presentation duration, a pure lag would result in a linear increase of mislocalization followed by a marked ceiling at presentation durations exceeding the lag. The temporal integration hypothesis, on the other hand, predicts a shallower increase as a function of presentation duration with a more gradual saturation. As a further test of the temporal integration mechanism, we added motion noise to the stimuli (Experiment 4). We reasoned that since the visual system needs more integration time when the signal is noisy (e.g., Huk & Shadlen, 2005), increasing the noise level should increase the component of mislocalization in the direction of pursuit if that component is indeed due to temporal integration. 

In summary, we found that the velocity summation and the gaze displacement effects of smooth pursuit both give rise to separate components of focus mislocalization. The magnitudes of these errors suggest that the brain partially compensates for both these effects of pursuit on optic flow. Finally, we argue that the mislocalization resulting from the gaze displacement effect is consistent with temporal integration. 

Methods

Observers

A total of six observers participated in the experiments. Five were unaware of the purpose of the study, one was an author. Two were female, four were male. Their age range was 20 to 36 years. All were right handed and had normal or corrected-to-normal vision. 

Stimulus generation

The visual stimuli (Figure 2) were generated with in-house software (Neurostim) that uses OpenGL for rendering. The software ran on a Pentium 4 PC with an ATI Radeon X550 video card that produced luminance-calibrated 14-bit grayscale images using a Bits++ device (Cambridge Research Systems) on a 19.8″ cathode-ray tube display (Sony GDM-C520) in 1024 × 768 pixels at 120 frames s⁻¹ mode. The viewing distance was 57 cm. No lighting other than the monitor screen was present in the room. 

Figure 2

 

Schematic of a real pursuit trial (left column) and a simulated pursuit trial (right column) with rightward pursuit and clockwise optic flow. The following sequence of events was used in all real pursuit trials. (A) The fixation marker and the stimulus appeared concentrically at the left side of the screen. (B) After fixating for 0.5 s, the fixation marker and the stimulus aperture accelerated linearly toward the center of the screen for 0.5 s, reaching a top speed of 5 deg/s. (C) The rotation of the optic flow pattern started 0.25 s after the onset of constant speed pursuit (but was variable in Experiment 2). (D) The pursuit and the optic flow continued until the fixation marker and the aperture reached the center of the screen. The constant speed pursuit phase lasted 1 s (1.5 s in Experiment 2). (E) The observer indicated the final perceived location of the optic flow field focus with a mouse. In the simulated pursuit condition, the fixation marker and stimulus were always at the screen center, but the dots moved such as to create an almost identical retinal stimulation as during real pursuit.

We programmed the optic flow stimuli as follows. One thousand nine hundred dots were randomly positioned on a square plane (the “canvas”) that was viewed by the OpenGL camera at a distance at which the width and height of the canvas were 44° (on average 1 dot per deg²). The dots were gray (5.0 cd m⁻²) on a black background (0.3 cd m⁻²) and had a diameter of 5 pixels, corresponding to 0.20° diameter at the center of the screen. OpenGL's anti-aliasing option was used for all rendering. 

Flow was created by updating the position of the dots on the canvas at frame rate. We used four types of flow fields: expansion (EXP), contraction (CON), clockwise rotation (CW), and counterclockwise rotation (CCW). The speed distributions of these flow fields were identical, but the local motion directions differed (in steps of 90° in “spiral space” (Graziano, Andersen, & Snowden, 1994) going from EXP to CCW to CON to CW). The local flow speed increased linearly from the focus of the optic flow field outward. This speed was 5° s⁻¹ at 3.78° from the focus unless stated otherwise. For rotating flow, this corresponds to 0.21 rotation per second; for radial flow, this corresponds to moving at 2.8 m s⁻¹ toward or away from a wall of dots positioned 5 m in front of the observer. On each trial, the focus of the optic flow pattern was located at a random position within a square 7° × 7° region at the center of the canvas. The dots had asynchronous limited lifetimes of thirty frames (250 ms) before being randomly repositioned in the canvas. To minimize the decrease (increase) of dot density near the focus of EXP (CON) over the course of a presentation, not the starting positions but the halfway points of all dot trajectories were uniform randomly distributed over the canvas. 

We presented the stimuli in two ways: during real pursuit and during simulated pursuit. In the real pursuit conditions, a fixation marker (a gray 30 cd m⁻² annulus with an outer diameter of 0.59° and an inner diameter of 0.20°) moved at 5° s⁻¹ toward its final position at the center of the screen. In the simulated pursuit conditions, the fixation marker was always displayed at the center of the screen, and the optic flow field shifted with 5° s⁻¹ toward its final position at the center of the screen. This shift was implemented by rotating the OpenGL camera around its vertical axis and projecting its image onto the physical screen. At the end of each trial, the observer's gaze was directed straight ahead and the canvas had a frontoparallel orientation. Any response differences found between the real and simulated pursuit conditions would suggest an influence of extraretinal factors, because the retinal stimulation in both conditions is identical except for imperfections in pursuit and fixation performance, and for the fact that in the periphery the dim edges of the monitor were moving on the retina in the real but not in the simulated pursuit condition. 

An important point is that not all dots on the canvas were visible. A circular aperture of 25° diameter that was centered on the fixation marker occluded all but about 500 dots. Thus, the aperture of the visual stimulus was always fixed in retinal coordinates and centered on the foveae in all conditions and over the course of a trial. This meant that the focus moved within the stimulus aperture during real and simulated pursuits. In the real pursuit condition, the focus was static on the physical monitor screen and with respect to the stabilized head of the observer. In the simulated pursuit condition, the focus moved at the simulated pursuit speed in retinal and world-centric coordinates. 

Procedure and stimulus timing

Observers sat in front of the center of the screen and used a bite bar for stability. Viewing was binocular and both eyes were tracked using an infrared video system (Eyelink II, S.R. Research) at 500 samples s⁻¹. An observer started a trial by fixating within 1.0° of the fixation marker that was on the left or right of the screen in real pursuit conditions, or at the center of the screen in simulated pursuit conditions. In a real pursuit condition, the fixation marker remained stationary for 0.5 s before moving toward the center of screen, accelerating linearly for 500 ms to 5° s⁻¹, and then continuing at that speed for either 1 or 1.5 s depending on the experiment. The pursuit phase ended when the fixation marker stopped at the center of the screen. The same timing was used in the simulated pursuit conditions, but the fixation marker was always at the center of the screen. One hundred milliseconds after the pursuit phase, a gray circular pointer (0.27° diameter; 30 cd m⁻² luminance) appeared at the center of the screen. The observers were instructed to align this pointer with the remembered final focus location using a computer mouse. A mouse key press ended the trial. The observers were required to maintain gaze within 1.0° from the fixation marker from the moment they fixated it at the beginning of the trial to the moment they clicked the mouse, otherwise the trial was aborted and the data discarded. 

The stimulus dots were visible throughout the trial. During most of each trial, the dots remained static on the canvas until they jumped to a new location at the end of their 250-ms lifetime. Note that dots that are static on the canvas still moved on the screen and on the retina when the OpenGL camera or the fixation dot moved relative to the canvas. Only during the final part of the pursuit phase, during the optic flow phase, did the dots move over the canvas according to the trial's optic flow settings. The optic flow phase ended at the same moment the fixation marker stopped moving. The optic flow phase ranged from 500 to 1250 ms in Experiment 2 and was 750 ms in all other experiments. During the part of the pursuit phase that preceded the optic flow phase, the stimulus dots swept uniformly across the retina in the direction opposite to the pursuit, both in the real and in the simulated pursuit conditions. During the final phase, the response phase, the dots were stationary on the canvas (and the retina) for the duration of their lifetime. Keeping the dot stimulus visible throughout the trial prevented abrupt global luminance changes, thus minimizing pupil reflexes that the eye tracker could spuriously register as shifts of gaze. 

Data analysis

Trials in which saccades occurred during the optic flow phase were discarded. Eye movements were considered saccades if the speed exceeded 22° s⁻¹ and the acceleration was more than 4000° s⁻². For each remaining real pursuit trial, the speed of the performed pursuit was determined. This was done by fitting a line to the horizontal coordinates of the gaze trace of each eye over the interval corresponding to the optic flow phase and averaging the slopes of both fits. (We performed all analyses in this study also without discarding trials with saccades, and by removing the saccades from the eye traces prior to determining the pursuit gain. All three methods yielded highly similar results.) 

We analyzed the horizontal (X) and vertical (Y) components of the focus localization responses separately (Figure 3). To quantify the effect of pursuit on focus localization, we performed multiple linear regression on the data by least squares fitting. The dependent variable of this regression was the “indicated focus X (Y) position”. The two independent variables were “real focus X (Y) position” (i.e., the final position of the focus on the screen without the eye velocity vectors added in) and “performed pursuit speed”. In the simulated pursuit trials, we set the speed of the performed pursuit to the simulated pursuit speed (always ±5° s⁻¹). We used the slope of the regression plane in the “performed pursuit speed” direction as the measure of the effect of pursuit on locating the focus. X and Y localization errors presented in this paper are defined as the coefficients of these slopes multiplied by the mean of the absolute “performed pursuit speed” values used in the regression. 

Figure 3

 

Example of focus localization data (observer JT's counterclockwise optic flow with real pursuit condition of Experiment 1). The left panel shows the horizontal (X) coordinates of the real (abscissa) and indicated (ordinate) focus position; the right panel shows the vertical (Y) coordinates. Data obtained during rightward pursuit are shown in green and leftward pursuit in black. The vertical distances between the regression lines show that large systematic mislocalizations were found in both the X and Y directions.

In Experiments 2, 3, and 4, we used CW and CCW flows, but results are presented as if the flow was always CW. To achieve this, we first applied linear regressions as described above to the CW and CCW separately and subtracted the constant terms yielded by these regressions from the localization responses. Thus, pursuit unspecific response biases were removed from the data. Then, the CCW data set was transformed by multiplying the vertical coordinates of the response and the veridical focus by minus one. Finally, the data were pooled and analyzed with the multiple linear regression method. 

Results

Across all experiments and subjects, we recorded a minimum number of 47 successful responses for use in a single regression as described in the Data analysis section. The median number was 94 for the conditions with simulated pursuit and 124 for those with real pursuit. In most runs of the experiments, we included more real pursuit than simulated pursuit conditions to compensate for the higher saccade rejection rate during real pursuit. The offline rejection rate was 59% (SD = 17) during real pursuit and 25% (SD = 11) for simulated pursuit. Note that this rate applies to trials that were brought to completion and does not include trials that were aborted online because of incorrect pursuit or blinks. As an example, the regressions shown in Figure 3 were based on the X and Y components of 125 responses. 

Experiment 1: Locating the focus of EXP, CON, CW, and CCW

We asked whether both the velocity summation and the gaze displacement effect play a role in locating the focus of optic flow patterns. To answer this question, we presented EXP, CON, CW, and CCW patterns to six observers. In terms of the vector sum of instantaneous velocities, viewing these patterns during pursuit shifts the focus in different directions for each of these four patterns: in the direction of the pursuit for EXP; in the direction opposite the pursuit in CON; and orthogonal to the pursuit in CW and CCW. The retinal drift of the focus resulting from the gaze displacement effect is always opposite to the pursuit direction for all four flow types. 

The mean focus localization errors for six subjects are shown in Figure 4. The data for leftward and rightward pursuit were pooled and analyzed as if pursuit was always to the right (positive X direction). All participants showed similar patterns of XY-mislocalization that bear the signatures of both the velocity summation and the gaze displacement effects. As predicted by the velocity summation effect, the mislocalization was in different directions for EXP, CON, CW, and CCW. The effect of gaze displacement was most clearly visible in the CW and CCW data, which were not only shifted up and down as predicted by the velocity summation effect but also parallel to the pursuit. More specifically, the mean horizontal mislocalization of the center of rotation across observers was 0.88° (SD = 0.39) with real pursuit (one-sample t ₁₁ = 7.74; p < 0.001) and 1.24° (SD = 0.27) with simulated pursuit (one-sample t ₁₁ = 15.9; p < 0.001). These shifts in the direction of (simulated) pursuit are consistent with a perceptual underestimation of the amount of retinal focus drift due to (simulated) incremental gaze displacement (van den Berg, 1999). However, the gaze displacement effect seems to also have played a role in the EXP and CON data: pooled over real and simulated pursuits, the magnitude of the X-mislocalization in CON (2.10°, SD = 0.47) was smaller than in EXP (2.71°, SD = 0.61; paired t ₁₁ = 3.62; p = 0.004). We interpret this to be the result of the gaze displacement effect causing mislocalization in the pursuit direction, thus counteracting the velocity summation effect in CON and adding up to it in EXP. 

Figure 4

 

Mislocalization of the focus of expanding (EXP), contracting (CON), clockwise (CW), and counterclockwise (CCW) optic flows after seeing it during real (red) and simulated (blue) smooth pursuits (plotted as if pursuit were always to the right). This resulted in a similar pattern of horizontal (X) and vertical (Y) mislocalizations for six observers (mean and SEM are shown). The pattern is consistent with concurrent contributions from the velocity summation and the gaze displacement effects. The velocity summation effect made the focus of EXP shift in the direction of pursuit; that of CON in the opposite direction; and those of CW and CCW up and down. Dashed lines indicate each flow type's baseline velocity summation effect for real (red) and simulated (blue) pursuits (see Compensation for the effects of smooth pursuit section). In addition, underestimating the focus drift resulting from incremental gaze displacement caused mislocalization in the pursuit direction for all flow types, hence the marked horizontal shifts for CW and CCW and the more hidden mislocalization magnitude asymmetry in EXP and CON.

Finally, comparing the differences in XY-mislocalization magnitudes for simulated (3.05°, SD = 0.62) and real (2.45°, SD = 0.53) pursuits revealed that the world-centric focus location was more accurately indicated with real pursuit (paired t ₂₃ = 7.02; p < 0.001). This suggests that extraretinal signals played a role in compensating for the effect of smooth pursuit on localization. However, it should be noted that these differences can at least be partially explained by the reduced pursuit gain in the real pursuit (group mean 0.93, SD = 0.05) compared to that of the simulated pursuit conditions (1 by definition). This issue will be addressed in the Compensation for the effects of smooth pursuit section. 

Experiment 2: The effect of optic flow phase duration

Experiment 1 showed that when viewed during pursuit, the focus of CW and CCW was systematically mislocalized in both the horizontal and the vertical directions. We wished to establish that the focus mislocalization in the direction of smooth pursuit indeed resulted from the incremental gaze displacement effect. To this end, we varied the duration of the optic flow phase. The mislocalization resulting from gaze displacement is expected to increase with longer presentations because gaze displacement increases over time, unless the mislocalization is already at ceiling level at the low end of the duration range. The velocity summation effect, being instantaneous, should be impervious to this manipulation. 

We presented stimuli similar to the ones used in Experiment 1 to five subjects. Here, only CW and CCW stimuli were used because in rotating flow the velocity summation and the gaze displacement effects were most easily separated as they work in orthogonal directions, i.e., respectively, in the X and Y directions. The optic flow phase durations lasted for the final 0.5, 0.75, 1.0, and 1.25 s of the pursuit phase interval, which was 1.5 s for all trials. Saccades occurring during the final 1.25 s of the pursuit phase lead to offline rejection of the trial. 

Figure 5 shows the mean X-mislocalization (left panel) and Y-mislocalization (right panel) across observers. Data obtained with real pursuit are red; data obtained with simulated pursuit are blue. We found that, indeed, X-mislocalization increased with presentation duration and the Y-mislocalization did not. More specifically, we used a least squares method to fit lines to the aggregate, non-averaged data of the five observers. The resulting slope and offset values are shown in Figure 5 with 95% confidence intervals. The confidence intervals of the X-mislocalization slopes do not include zero, whereas those of the Y-mislocalization do. Furthermore, the confidence intervals of the slopes for X- and Y-mislocalizations did not overlap. 

Figure 5

 

Results of Experiment 2. The horizontal (X, left panel) and vertical (Y, right panel) mislocalizations of the focus of rotating optic flow, after viewing it during smooth pursuit, as a function of optic flow phase duration. The data were pooled to represent the effect of rightward pursuit on clockwise optic flow and averaged over five subjects. Positive X means rightward; positive Y means up. Red represents the real pursuit condition, and blue represents the simulated pursuit condition. Error bars are SEM. We fitted lines with slope a and offset b to these data (solid lines). The fit parameters, shown with 95% confidence intervals in the plots, indicate that X-mislocalization increased with duration and Y-mislocalization did not. This is consistent with the idea that X-mislocalization results from incremental gaze displacement and Y-mislocalization from instantaneous velocity summation. Dashed lines indicate the retinal focus shift due to velocity summation (see Compensation for the effects of smooth pursuit section).

We ask why incremental gaze displacement leads to mislocalization in the pursuit direction. van den Berg (1999) reported a similar finding and found that a perceptual lag of about half a second could explain his results. Such a lag results in a perceptual underestimation of the retinal focus drift, thus causing the mislocalization in the direction of pursuit. Those data did not allow one to differentiate between a pure lag of the localization system (a latency) and an integration of the optic flow information over a temporal window. 

To test these two explanations with our data, we fit a pure lag and a temporal integration model to our X-mislocalization as a function of duration data. The pure lag was modeled as a linear increase of mislocalization with duration followed by a constant mislocalization when the duration exceeded the lag. This model had two parameters: the lag and the slope as a function of duration. The alternative model integrated the retinal focus trajectory with a leaky integrator that weighed recent locations more heavily than earlier locations (Krekelberg & Lappe, 2000; Roulston, Self, & Zeki, 2006). This model also had two parameters: the time constant of the exponentially decaying weighting filter and a gain. Separate fits were made to the 20 data points obtained with real pursuit and those obtained with simulated pursuit. 

For the real pursuit data, the lag model had an initial slope of 1.11 ± 0.12 degrees of mislocalization per second of stimulus duration, followed by a constant mislocalization after 1.13 ± 0.19 s (ranges are 95% confidence intervals). The r ² value of this model was 0.769. The alternative, temporal integration model had a gain of 1.66 ± 0.78 and a time constant of 0.45 ± 0.18 s. The r ² value was 0.773. 

Fitting the lag model to the simulated pursuit data resulted in a slope of 1.33 ± 0.46° s⁻¹, a lag of >1.25 s (i.e., the mislocalization did not level off within our range of durations), and an r ² of 0.698. The temporal integration model yielded a gain of 1.85 ± 1.06, a time constant of 0.50 ± 0.26 s, and an r ² of 0.711. 

Based on their relative fit quality in terms of sum of square residuals, we cannot decide between the lag and temporal integration model, neither in the real pursuit conditions (F _19,19 = 1.017, p = 0.49) nor in the simulated pursuit conditions (F _19,19 = 1.048, p = 0.46). However, according to the lag model, the visual system processes optic flow with an inconceivably long latency of over a second. On the other hand, the temporal integration window had a time constant of half a second. This seems more plausible and matches earlier results of van den Berg (1999) and integration time constants found in the flash-lag literature (Krekelberg & Lappe, 2000). 

Experiment 3: The effect of optic flow speed

If the Y-mislocalization of the focus of rotation is the result of the velocity summation effect, it relies on the relative speeds of the pursuit and the optic flow. This predicts small mislocalizations when the optic flow speed is high relative to the pursuit speed, i.e., when the local flow velocity that cancels the pursuit is close to the focus, and vice versa. If the X-mislocalization, on the other hand, stems from the gaze displacement effect, it depends on the product of optic flow phase duration and pursuit speed and should be invariant to optic flow speed. 

To test these predictions, we performed an experiment similar to Experiment 2 in which the optic flow speed was varied. Rotation rates were 0.11, 0.21, 0.42, 0.63, 0.84, 1.26, and 1.68 Hz, corresponding to a range of 0.66 to 10.56° s⁻¹ at a 1° distance from the focus. 

The group result of five subjects is shown in Figure 6. As predicted, the observed Y-mislocalization (right panel) decreased with increasing flow speed. This decline was roughly the reciprocal of the optic flow speed, consistent with the velocity summation account, i.e., the data fit well with a straight line in log–log space. The slope of this line was significantly negative, both for real pursuit (red) and simulated pursuit (blue). The fit values with 95% confidence are shown in Figure 6. 

Figure 6

 

The effect of optic flow speed on X- and Y-mislocalizations, plotted with the same graphical conventions as used in Figure 5. The logarithm of the Y-mislocalization decreases linearly with the logarithm of the rotation rate, which was expected on the basis of instantaneous velocity summation. However, a significantly negative dependence of X-mislocalization on rotation rate was also found, which is inconsistent with the idea that the X-mislocalization depends exclusively on gaze displacement.

However, the observed X-mislocalization (left panel) also showed a significant decrease with increasing flow speed, contrary to the prediction of invariance. In addition to this main trend, a reduction of the X-mislocalization can be observed at the very low end of the speed range, giving rise to a peak at a rotation rate of around 0.2 Hz. 

A potential explanation for the main trend, the reduction of X-mislocalization with increasing flow speed, is that the center of a rapidly rotating flow is much more conspicuous than that of a slow one. In terms of temporal integration as a means to increase sensitivity in the face of noisy inputs (e.g., Huk & Shadlen, 2005), less integration time may be needed when the perceptual uncertainty of the focus location is reduced, thus leading to reduced underestimation of the retinal focus trajectory and, hence, mislocalization. 

To test this idea, we performed an additional analysis of the data from Experiment 3. We defined target uncertainty as the standard deviation of the residuals of the regression that was used to obtain the localization errors (as described in the Data analysis section). Figure 7A shows that horizontal target uncertainty (HTU) decreased linearly with optic flow speed in log–log space for both real and simulated pursuits. This means that the focus of faster rotating flow was indeed more precisely located than that of slower optic flow. Next we plotted X-mislocalization as a function of log HTU (Figure 7B). Using linear regression, positive relations were found for real and simulated pursuits (the 95% confidence intervals of the slopes do not include zero). Apparently, larger uncertainty about the focus location was accompanied by larger horizontal localization errors. This agrees with the idea that the gaze displacement effect leads to mislocalization through a mechanism of temporal integration. 

Figure 7

 

Additional analysis of the data of Experiment 3. (A) Increasing the rotation rate of the optic flow stimulus decreased the horizontal target uncertainty (HTU), showing a negative linear relation in log–log space. Data points are the average for five subjects; error bars are SEM. (B) X-mislocalization increased linearly with the logarithm of HTU. Each data point represents one subject's localization error obtained at a single rotation speed. The results are similar for real (red) and simulated (blue) pursuits. These observations link the dependence of X-mislocalization on optic flow speed that was unexpectedly found in Experiment 3 (Figure 6, left panel) to target uncertainty.

We think that the reduction of X-mislocalization at very low speeds was the result of the focus being so unclear that subjects often resorted to guessing its location. Because purely adventitious responses are not systematically biased, localization errors were reduced. Consistent with this idea is that the Y-mislocalization was also slightly reduced at this speed. 

Experiment 4: The effect of stimulus uncertainty

In Experiment 3, we found a positive linear relation between the logarithm of horizontal target uncertainty (HTU) and observed X-mislocalization. However, the variation in HTU depended on variation in optic flow speed. Thus, it is possible that the decrease of the X-mislocalization magnitude with increasing flow speed was caused by some unknown effect of flow speeds other than a correlation with HTU. 

To circumvent this confound, we used a stimulus scrambling method to more directly influence HTU while keeping the optic flow speed similar for all conditions (Figure 8A). Each dot's trajectory was kept intact but was spatially offset within the stimulus aperture. The horizontal offset of the ith stimulus dot was dx _i = R _i cos(A _i ) and its vertical offset was dy _i = R _i sin(A _i ), where A _i is a random value between 0 and 360° and R _i is the square root of a random value between 0 and the square of R _max. Thus, the endpoints of the offset vectors (dx, dy) were homogeneously distributed in a circular area with radius R _max. We used R _max values of 0, 2, 4, and 8°. 

Figure 8

 

Results of Experiment 4. (A) To test the idea that temporal integration of the retinal focus trajectory (resulting from gaze displacement) leads to mislocalization, we scrambled the optic flow by adding a random offset to each dot trajectory. The focus position consistent with each trajectory is indicated with a green dot. When the maximum offset magnitude (R _max) was zero (left panel), these positions overlapped with the mean of all positions, i.e., the pattern's global focus (red dot). With large R _max (right panel), the spread was large but the global focus stayed in the same location on average. (B) Increased levels of R _max led to increased mean horizontal target uncertainty (HTU; left panel) and vertical target uncertainty (VTU; right panel) in 5 observers. (C) X-mislocalization increased with HTU (left panel). This is consistent with the idea that noisier stimuli require longer temporal integration, which leads to larger underestimation of the gaze displacement, which in turn explains the increase of X-mislocalization. No significant effect of VTU (right panel) on Y-mislocalization was found, which is consistent with the idea that Y-mislocalization results from velocity summation. (Graphical conventions of (C) as in Figure 5.)

To compensate for the increase in task difficulty due to stimulus scrambling, a relatively high rotating flow speed of 0.84 Hz (compared to 0.21 Hz in Experiments 1 and 2) was used in all conditions. The HTU values in this experiment were calculated on the basis of an additional non-pursuit condition in which both the fixation marker and the focus position were stationary. The X- and Y-mislocalizations resulting from real and simulated pursuits were obtained as in the other experiments. 

Figure 8B shows that the mean HTU and vertical target uncertainty (VTU) of five subjects increased linearly with R _max. Based on the idea that gaze displacement causes mislocalization in the direction of pursuit by means of temporal integration, and that uncertain targets require longer processing, we expected X-mislocalization to increase with increasing HTU. This is what we found (Figure 8C, left panel). Both in the real and simulated pursuit conditions, the X-mislocalization increased linearly with increasing HTU. Note that although in the figure the HTU values on the horizontal axis are binned in 0.25° bins for clarity of display, the lines were fit to the aggregate, non-averaged X-mislocalization and non-binned HTU of the five observers. For completeness, the right panel of Figure 8C shows the relation between VTU and vertical target mislocalization. No systematic bias of the velocity summation effect was expected with increasing VTU. Indeed, the slopes of the lines fitted to these data were not significant (their 95% confidence intervals included zero). 

We conclude that the effect of optic flow speed on X-mislocalization that we found in Experiment 3 is related to target uncertainty and not to some unknown effect of optic flow speed per se. 

Compensation for the effects of smooth pursuit

It is interesting to compare localization errors obtained during real and simulated pursuits. Because the retinal stimulation was nearly identical in both conditions, differences in mislocalization magnitudes implicate extraretinal signals in the compensation for pursuit. Although in all experiments the mean X and Y localization errors in the simulated pursuit conditions exceeded those of the corresponding real pursuit conditions, the descriptive linear fits we applied did not reveal significant differences between the slopes or the offsets of the lines. However, paired t-tests applied to the aggregate data of all observers and experiments (Figure 9) showed that the X-mislocalization following real pursuit was smaller than in the corresponding simulated pursuit conditions (paired t ₈₆ = 8.60, p < 0.001). The Y-mislocalization was also smaller after real compared to simulated pursuit (paired t ₈₆ = 6.80, p < 0.001). 

Figure 9

 

Mean absolute horizontal (X, left panel) and vertical (Y, right panel) mislocalizations of the focus of rotation across all experiments and subjects. Both the X and Y localization errors were larger when the pursuit was simulated (blue markers) as opposed to real (solid red markers). This remained the case in an alternative analysis that used an idealized pursuit gain of 1 for the real pursuit condition (open red markers). This indicates that extraretinal information was used to compensate for the effect of velocity summation as well as the effect of gaze displacement. Error bars are SEM, N = 87; stars indicate paired t ₈₇, p < 0.001.

However, as pointed out in Experiment 1, these differences were slightly inflated by the fact that the gain of real pursuit was less than one (on average 0.94 (SD = 0.036) across subjects and experiments). In other words, there was less pursuit to compensate for. We corrected for ocular following performance by dividing the localization errors of the real pursuit conditions by their corresponding pursuit gains. Localization errors with real pursuit remained significantly smaller (X: paired t ₈₆ = 6.19, p < 0.001; Y: paired t ₈₆ = 4.66, p < 0.001). These results suggest that extraretinal signals were used to compensate for the effects of both velocity summation and gaze displacement. 

Apart from comparing the localization errors obtained with real and simulated pursuits, it is possible to compare the errors caused by the velocity summation effect to an absolute level that would occur if no compensation took place. This level is simply the distance between the focus and the point in the flow field that has equal speed but a direction opposite to the pursuit component of the retinal flow field. The predicted Y-mislocalization for each condition is shown as dashed lines in Figures 4, 5, 6, and 8. The blue lines correspond to the simulated pursuit conditions and the red line to real pursuit. The offset between the lines are the result of differences in pursuit gain. Expressed as a percentage of these predictions, and averaged across all subjects, the four experiments, and the different duration, speed, and noise conditions, the mislocalization magnitude for rotational flow was 62% (SD = 19) during real pursuit and 71% (SD = 20) during simulated pursuit. This again shows that real pursuit yielded smaller errors (paired t ₈₆ = 5.43, p < 0.001), but it also means that significant compensation did occur in the simulated pursuit condition (one sided t ₈₇ = 13.74, p < 0.001). This finding will be addressed in the Discussion section. 

Discussion

Two components of shift

The visual system faces at least two separate challenges when estimating the world-centric location of an optic flow field's focus during smooth pursuit. First, the velocity summation effect instantaneously shifts the focus in retinal coordinates. Many models have been developed that address this issue (Beintema & van den Berg, 1998; Hildreth, 1992; Lappe & Rauschecker, 1994; Perrone & Stone, 1994, 1998; Rieger & Toet, 1985; Royden, 1997; Royden & Picone, 2007). Second, the focus incrementally drifts across the retina opposite to the pursuit. The experiments presented here show that this second issue leads to mislocalization too, possibly resulting from temporal integration of the retinal trajectory of the focus. 

We asked observers to locate the focus of expanding, contracting, and rotating optic flows (Experiment 1) and found that the smallest errors were made in contracting flow. This is counterintuitive because one might have expected compensation for pursuit to be strongest in expanding flow because it results from the common mode of human locomotion. However, this finding can be explained if one considers that the mislocalization resulting from velocity summation and gaze displacement are in the same direction in expanding, and in opposite directions in contracting optic flow. One problem with this reasoning is that it predicts the asymmetry between the localization errors observed with EXP and CON to be twice the size of the horizontal shifts in the CW and CCW conditions, but we found a much smaller asymmetry. We think this is related to the finding in Experiments 3 and 4 that the mislocalization due to gaze displacement was larger when the uncertainty about the focus location was larger. In Experiment 1, after per subject normalization of the HTUs of EXP, CON, CW, and CCW by division with the HTU obtained with EXP, the median HTU index was 1.00 for radial flow and 1.28 for rotational flow (Mann–Whitney U = 133, n ₁ = n ₂ = 12, p < 0.001). Improved motion discrimination in radial compared to rotational flow has been reported earlier (Beardsley & Vaina, 2005). 

Partial compensation for pursuit in rotating flow has been mentioned before in the context of human psychophysics (Bradley et al., 1996), but only the component of mislocalization orthogonal to the pursuit (the velocity summation direction) was measured. We showed that significant mislocalization occurred both orthogonal and parallel to the direction of horizontal pursuit. The effect of varying stimulus duration on the X-mislocalization and the lack thereof on the Y-mislocalization (Experiment 2) supported the idea that the mislocalization parallel to the pursuit direction resulted from temporal integration. Increasing the stimulus speed (Experiment 3) strongly decreased the mislocalization orthogonal to the pursuit in a way that matched the geometric interpretation of the velocity summation effect. 

Compensation for pursuit has been studied extensively in simulated heading experiments using expanding flow. It has been shown that heading judgments remain accurate during real pursuit, presumably because the visual system uses extraretinal eye position and velocity signals. Accurate heading estimates during simulated pursuit have also been found, even though extraretinal signals in that case would indicate fixation. For this retinal compensation to work optimally, depth cues need to be available (e.g., such as in ground plane and 3D cloud stimuli or stimuli containing reference objects; Li & Warren, 2000; Royden, Banks, & Crowell, 1992; Warren & Hannon, 1988, 1990) and the pursuit speed low (<1.5° s⁻¹; Royden et al., 1992, 1994). One reason depth cues help compensate is that the motion of distant elements is dominated by the pursuit and that of nearby dots by the forward translation (e.g., Longuet-Higgins & Prazdny, 1980; van den Berg & Brenner, 1994). In our experiments, we eliminated depth cues by using a vertical wall of dots and we reduced the effectiveness of simulated pursuit compensation by using a relatively high pursuit speed of 5° s⁻¹. However, we found only small differences between mislocalizations observed during real and simulated pursuits, both in the X and Y directions. 

Why was the compensation for simulated pursuit so similar to that observed during real pursuit despite the lack of depth cues and the high pursuit speed? Grigo and Lappe (1999) showed that purely visual compensation for pursuit is possible in 2D stimuli using the cue that, during horizontal pursuit, the flow field lines curve outward in the far high and lower periphery. However, the stimuli used in the present study were probably not sufficiently large for this to play a role. Alternatively, the relatively strong compensation during simulated pursuit may have resulted from the lateral motion of the stimulus dots that were visible prior to the optic flow phase interval. This could trigger pursuit compensation before the onset of the optic flow stimulus by suppression of the optokinetic nystagmus that is elicited by large field laminar motion (cf. Chaudhuri, 1991; Duffy & Wurtz, 1993; Freeman, Sumnall, & Snowden, 2003). Another possibility is that the lateral motion caused a motion aftereffect (Anstis, Verstraten, & Mather, 1998; Mather, Pavan, Campana, & Casco, 2008) that perceptually reduced the simulated pursuit component in the combined rotational and simulated pursuit flow. However, the role of such an aftereffect was probably limited by the random interleaving of leftward and rightward pursuit conditions, which reduces buildup over time. 

We found a level of compensation for real pursuit in expanding flow of 61% (SD = 10) across the six observers in Experiment 1; 66% (SD = 14) after normalization for pursuit gains. This is low compared to the nearly perfect compensation for real pursuit reported earlier (Banks et al., 1996; Royden et al., 1992; van den Berg, 1996; Warren & Hannon, 1990). This discrepancy possibly resulted from a number of stimulus properties that reduced the sense of self-motion conveyed by our stimuli, which may have negatively impacted the level of compensation. Our stimuli were small (so that they could move with the pursuit without clipping at the screen edges), consisted of dots with limited lifetimes (to minimize density differences across flow types), and had comparatively low flow speed. Another difference with previous studies is that the stimulus aperture was always centered on the fixation direction, also during real pursuit. Finally, we did not instruct the observers to estimate their heading direction but asked them to indicate the final perceived location of the focus. The impact of task instructions on the perception of comparable stimuli has been demonstrated (Li & Warren, 2004; Royden, Cahill, & Conti, 2006). 

Temporal integration

In our experiments, observers were instructed to indicate the final perceived position of a moving target after it abruptly disappeared at the end of the optic flow phase. Other studies of the localization of abruptly disappearing moving targets have yielded mixed results. In the flash terminated condition (FTC) used in some studies of the flash-lag effect, the perceived location at which a moving dot disappeared relative to a briefly flashed stationary stimulus was probed using two-alternative forced-choice paradigms. Eagleman and Sejnowski (2000) found no mislocalization in the FTC. On the other hand, Fu, Shen, and Dan (2001) and Kanai, Sheth, and Shimojo (2004) found an overshoot in the direction of motion. The perceived endpoint of the vertical trajectory of a dot that was viewed during horizontal smooth pursuit was biased opposite to the pursuit direction (Souman, Hooge, & Wertheim, 2006). Finally, Roulston et al. (2006) reported a “flash-lead”, a shift opposite the retinal motion of the target, similar to the findings reported here. 

The lag effects found in the FTC were contingent on the moving targets being spatially uncertain, which was manipulated by applying a Gaussian luminance window (Fu et al., 2001) or increased eccentricity (Kanai et al., 2004). Correspondingly, the sharply defined stimuli used by Eagleman and Sejnowski (2000) were not mislocalized. This is reminiscent of a previous study that showed that indistinct optic flow focuses undergo more illusory displacement in the direction of transparently overlapping laminar flow than conspicuous ones (Duijnhouwer et al., 2008). This may be related to the influence of spatial target uncertainty on the magnitude of mislocalization that we found in the present study, although our main effect is opposite to the direction of retinal motion. 

We showed that the X-mislocalization in Experiment 2 is consistent with temporal integration that weighs recent locations more heavily than earlier locations (Krekelberg & Lappe, 2000; Roulston et al., 2006). We fitted such a leaky temporal integrator with an exponentially decaying weighting filter to the X-mislocalization data of Experiment 2 and found that the time constant of the filter was on the order of half a second. 

While this temporal integration seems long, it is consistent with estimates of the time needed to locate the focus of expansion without pursuit, which ranges from 228 ms to 430 ms (Crowell, Royden, Banks, Swenson, & Sekuler, 1990; Hooge, Beintema, & van den Berg, 1999; Te Pas, Kappers, & Koenderink, 1998) or even up to 3 s (Burr & Santoro, 2001). Similarly, temporal integration in the flash-lag effect has also been estimated to range from 100 ms to approximately 500 ms (Krekelberg & Lappe, 2000; Roulston et al., 2006). It also corresponds well to the heading stimulus processing durations that ranged from 300 to 600 ms in the study of van den Berg (1999). In that study, it was impossible to determine whether the mislocalization due to the gaze displacement effect resulted from a pure delay of the processing, or to a process of slow temporal integration. However, in the present study we found that the X-mislocalization continued to increase well beyond optic flow phase durations of half a second (Experiment 2), which fits better with the idea of temporal integration with a time constant in the order of half a second than with a pure delay of that duration. 

The range of integration times can at least in part be attributed to the uncertainty associated with the location of the object; as our Experiments 3 and 4 confirm, greater uncertainty is associated with longer integration times. This might reflect a general strategy of the visual system to increase integration duration when visual signals are weaker (Huk & Shadlen, 2005). 

Conclusion

The problem of heading detection during smooth pursuit is usually phrased in terms of instantaneous velocities, i.e., the velocity summation effect. However, we found that the incremental retinal shift of the focus resulting from pursuit also leads to focus mislocalization. This localization error was in the direction of pursuit and occurred because of underestimation of the amount of focus drift on the retina. The underestimation was consistent with temporal integration of the retinal trajectory of the focus. In this study, we found an integration time constant of half a second and we presented evidence that this duration increases with stimulus properties that increase target uncertainty, such as low flow speed and added noise. Compensation for pursuit occurred for both the velocity summation and the gaze displacement effects, especially when extraretinal information could be used. Both effects of pursuit should be taken into account in future experiments and models of heading detection during smooth pursuit. 

Acknowledgments

This work was supported by a VIDI Grant from The Netherlands Organization for Scientific Research (NWO), a High Potential Grant from Utrecht University awarded to RvW, and an NIH Grant (R01 EY017605) awarded to BK. 

Commercial relationships: none. 

Corresponding author: Jacob Duijnhouwer. 

Email: [email protected]. 

Address: Aidekman Building, 197 University Ave., Newark, NJ 07102, USA. 

Footnote

Footnotes

1  In this paper, a rotating optic flow pattern is a pattern of circular motion trajectories that are centered on a point in the visual field, not the laminar flow resulting from rotating the eye.