The vector combination underlying the double-drift illusion is based on motion in world coordinates: Evidence from smooth pursuit

Abstract

If a Gabor pattern drifts in one direction while its internal texture drifts in the orthogonal direction, its perceived direction deviates strongly from its true direction and is instead some combination of its real external motion and its internal motion (Tse & Hsieh, 2006). In the first experiment, we confirm that, for the stimuli used in our experiment, the direction shifts on a gray background were explained by a vector combination of the internal and external motions whereas for the Gabor on a black background, we find no illusory shifts. These results suggest that the internal motion contributes to the perceived direction but only when the Gabor's positional uncertainty is high. Next, we test whether the vector combination is based on motions on the retina or motions in the world. When participants track a fixation point that moves in tandem with the Gabor, keeping it roughly stable on the retina, the illusion is undiminished. This finding indicates that the vector combination of internal and external motion that produces the double-drift illusion must happen after the eye movement signals have been factored into the stimulus motions to recover motions in the world, in particular, in areas V3A, V6, MSTd, and VIP.

Introduction

Most cells in the visual system are characterized by receptive fields, organized in retinotopic maps. The location specificity of a receptive field provides a straightforward positional coding of any stimulus that activates the cell. It might seem that there is little further to be discovered concerning the computation of position, but that would be wrong. Dramatic errors in perceived location are found when the target (Nijhawan, 1994; Eagleman & Sejnowski, 2007; Cavanagh & Anstis, 2013) or the eyes (review by Ross, Morrone, Goldberg, & Burr, 2001) are in motion. These results indicate that perceived location must instead be computed by a sophisticated, predictive system that constructs target locations based on their motion. We show here that this predictive position system can combine two motion vectors and that it lies at or beyond the level at which eye movements are discounted to recover motions in the world. 

The double-drift stimulus (also known as the infinite regress and curveball illusion; Tse & Hsieh, 2006; Shapiro, Lu, Huang, Knight, & Ennis, 2010; Gurnsey & Biard, 2012; Kwon, Tadin, & Knill, 2015; Lisi & Cavanagh, 2015), leads to extreme misjudgments of an object's location (Figure 1, left). In this stimulus, a Gabor patch moves over an equiluminant background in one direction while its internal texture moves orthogonally to the Gabor's path. The perceived direction of this double-drift stimulus (Lisi & Cavanagh, 2015) may differ by more than 50° in direction from its physical direction. Moreover, the effect is seen not only for the direction but also for the judged position of the target (Lisi & Cavanagh 2015, experiment 2) which may deviate by as much as several degrees of visual angle from its true location. 

Figure 1

 

Left. A large shift in perceived direction and perceived position is produced when the internal texture of a moving Gabor drifts orthogonally to the Gabor's path. Right. If the fixation point moves in tandem with the Gabor, then the position of the Gabor on the retina is roughly stable. Here we test if the illusion survives the rough stabilization of retinal motion that arises during this pursuit condition. See Movie 1 and Movie 2.

Tse and Hsieh (2006) proposed that perceived direction is produced by a vector combination of its internal and external motions. Kwon et al. (2015) proposed that this illusory integration would occur when the Gabor's positional uncertainty was high. In our first experiment we test this integration model by varying the external speed while participants hold steady fixation (Figure 1, left). We test the effect of external speed on a gray background matched to the Gabor's mean luminance for maximum position uncertainty and on a black background for minimum uncertainty. 

Second we examine whether the illusion persists when the participants smoothly pursue a fixation point that is yoked to the motion of the Gabor (Figure 1, right). With roughly accurate pursuit, the Gabor will have little motion on the retina even though its motion in the world is clearly seen. The visual system recovers motion in the world during pursuit by compensating for the motion of the eye either based on retinal or extraretinal inputs (Wallach, 1959; Rieger & Lawton, 1985; Brenner & van den Berg, 1994; Freeman, 2001; Souman, Hooge, & Wertheim, 2005; Freeman, Champion, Sumnall, & Snowden, 2009). If the double-drift illusion fails during smooth pursuit (as it did for saccades, Lisi & Cavanagh, 2015), then the motion computation that produces the illusory direction must occur prior to the correction for eye movement. If the illusion is maintained without loss during pursuit, the vector combination must be at or after the point at which motions in the world have been recovered (Movies 1 and 2). 

 

Movie 1

 

Movies 1 and 2. Left, Movie 1. Fixate on the static point on the left and notice the apparent direction of the Gabor on the right. It may appear almost vertical even though its true direction is oblique, as can be seen by looking directly at it.

 

Movie 2

 

Right, Movie 2. Now track the moving fixation point and again pay attention to the apparent direction of the Gabor on the right. Does it also appear almost vertical or does it appear to follow its physical, oblique path?

Experiment 1: Eyes fixed, changing speed and background

This experiment evaluated the effect of the speed of the Gabor along its path on the shift in its perceived direction. This was tested with the Gabor on a midgray background with the same mean luminance as the Gabor and on a black background (the internal speed of the Gabor was constant throughout). If the perceived direction is some fixed combination the internal and external speeds, the perceived direction should follow a simple function of the two speeds. We varied the speed in two ways: We changed the rate at which the motion reversed (every 200, 400, or 800 ms) and we changed the path length—3.125 degree of visual angle (dva) and 6.25 dva. As before, participants adjusted the orientation of the Gabor's path until it appeared vertical. We then ran separate sessions with the same speed variations for a Gabor on a black background. 

We can derive the path orientation, θ, that will appear vertical, assuming that the orthogonal internal and external motion vectors, v_i and v_e, are combined with weights of w_i and w_e :  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}\theta = {\tan ^{ - 1}}{{{w_i}{v_i}} \over {{w_e}{v_e}}}\end{equation}

 

In Figure 2, the weights for the internal and external motions are equal, as they are for a vector sum (w_i = w_e = 1) or a vector average (w_i = w_e = 0.5). In our experiments, we evaluate only the perceived angle so we cannot estimate the separate weights but only their relative value, k = w_i / w_e. That is what we will recover from our measurements of the angle, θ, that appears vertical:  

\begin{equation}\tag{2}\theta = {\tan ^{ - 1}}k{{{v_i}} \over {{v_e}}}\end{equation}

 

Figure 2

 

Here we assume that the perceived motion direction is a weighted combination of the internal and external motion vectors. In the experiment, the internal and external vectors are orthogonal so the physical path that appears vertical is given by tan⁻¹(internal speed/external speed) when the internal and external speeds are weighted equally, as they would be for a vector sum or a vector average (see Equation 2 in the text).

If we also measured the perceived speed, we could recover the individual weights where the speed, the amplitude of the perceptual vector, Display Formula\(\vec P \), is given by  

\begin{equation}\tag{3}P = \sqrt {{{({w_i}{v_i})}^2} + {{({w_e}{v_e})}^2}} \end{equation}

 

Method

Participants

Six healthy adults took part in the experiment (two males, four females; mean age = 36.2 years, SD = 22.1, with a range of 16 to 66 years), including the two authors and four experienced participants naïve to the purpose of the experiment. All participants in this and the following experiment reported normal or corrected-to-normal vision. All participants gave informed consent in writing prior to participation, and the protocols for the study were approved by the Dartmouth College Review Board in accordance with the principles of the Declaration of Helsinki. 

Stimuli and apparatus

The experiment took place in a darkened room. Stimuli were presented on a gamma-corrected CRT monitor (75 Hz, 1152 × 870 resolution covering 36 × 27 cm). Participants were seated 57 cm from the monitor with their heads resting on a chin- and headrest. 

The stimulus was a 75% contrast Gabor with a sigma of 0.625 dva and spatial frequency of 1.56 cycles/dva moving at 5 Hz orthogonally to the Gabor's path. This resulted in an internal speed of the Gabor's bars of 7.8 dva/s. The Gabor moved back and forth over one of two path lengths (3.125 dva and 6.25 dva), and reversed direction with one of three durations, 200 ms, 400 ms, or 800 ms. In the main condition, the Gabor was presented against a midgray background (10.2 cd/m²) that matched its mean luminance. In the black condition, the background was black (2.1 cd/m²) and the Gabor was 75% contrast with a spatial frequency of 1.56 cycles/dva moving at 5 Hz but it was now in a hard aperture of 1.875 dva diameter with a sigma of 1.875 dva, so it appeared as almost a uniform grating within a circular aperture (see Movie 3). The black (2.1 cd/m²) fixation dot enclosed in a white outline circle remained on the screen throughout the experiment and did not move. It was centered vertically on the screen and 9.375 dva to the left of the screen's midpoint, so a total of 18.75 dva from the Gabor path's midpoint. 

 

Movie 3

 

Fixate on the static point on the left and notice the apparent direction of the Gabor on the right. Even though this oblique physical path appeared vertical in Movies 1 and 2, here with a black background, it may now appear to follow its physical path.

Procedure and design

Participants performed 24 method of adjustment trials in the main experiment and 24 in the black background experiment, four trials in each of the six conditions (two path lengths and three reversal rates). During each trial, the Gabor appeared initially at the midpoint of its path. It then moved up to the left, toward its upper endpoint where it reversed both its external and internal motion directions and returned through the midpoint to the bottom extent of its path where it again reversed internal and external directions. These back-and-forth transits repeated throughout the trial. Participants were instructed to keep their gaze on the black fixation dot and to adjust the orientation of the Gabor's path, using a track pad, until it appeared vertical. When the participant was satisfied with the setting, he or she pressed the space bar to end the trial. The duration of each trial depended on the participant but averaged about 5 s. The starting orientation of the Gabor's path was set randomly on each trial. The six speed conditions were randomly intermixed. The main and black background experiments were run separately. 

Results

The magnitude of the illusion was calculated as the angular offset from vertical required to make the path appear vertical. In Figure 3, the results for the gray background tests are shown in red and blue and reveal that the illusion strength decreased smoothly with the external speed of the Gabor (the internal speed was always 5 Hz). Note that the combinations of short and long paths and three reversal rates resulted in two pairs of conditions having the same external speeds but different path lengths and durations. Here the red and blue data overlap very closely, suggesting that the controlling factor is the external speed and not the path length or reversal rate. We also plot the curve of the direction derived from the internal and external motions (Equation 2) as two dashed lines on the plot, one for a relative weight, k, of 1.0, the weight for a standard vector sum, and one for the relative weight of 0.81 which gives the best fit to the data. The data fall very close to this second line (r² = 0.997) where the internal motion is weighted about 80% of the external motion. 

Figure 3

 

Illusion strength as the angular offset from vertical required to make the path appear vertical. The angle is plotted as a function of the external speed of the Gabor and that is given by the path length/reversal time. The model of direction from Equation 3 is given by the two dashed lines, the light one for k = 1, equivalent to a vector sum or vector average, the darker one for k = 0.81, the best fitting value, indicating that the external motion is given slightly more weight than the internal motion. The vertical bars show ±1 SE.

The results for the black background (green and purple plots on Figure 3) are clear as well—the illusion is significantly weakened or eliminated when the background luminance is no longer matched with the mean luminance of the Gabor. 

Experiment 2: Eyes fixed versus smooth pursuit

This experiment measured the perceived shift in the orientation of the Gabor's path when the fixation was either static or moving. The Gabor moved continuously back and forth along an oblique path with its internal motion reversing direction at each endpoint, so the apparent orientation of the path appeared stable (Lisi & Cavanagh, 2015). Participants adjusted the orientation of the Gabor's path until it appeared vertical. When the fixation point was moving, it followed the path of the Gabor, offset horizontally to the left by 18.75 dva so that the Gabor would fall approximately18.75 dva to the right of the fovea during the smooth pursuit. 

Method

Participants

The participants of Experiment 1 all took part in this experiment. 

Stimuli and apparatus

The stimulus and apparatus were the same as in the previous experiment with the following differences. In both static fixation and pursuit trials, the Gabor moved back and forth over a path 4.69 dva long, reversing direction every 1.0 s for an external speed of 4.69 dva/s. The midpoint of the Gabor's path was 9.375 dva to the right of the center of the screen. The Gabor was presented against a midgray background (10.2 cd/m²). On the static fixation trials, as in Experiment 1, the fixation point was centered vertically on the screen and 9.375 dva to the left of the screen's midpoint, so a total of 18.75 dva from the Gabor path's midpoint. On the moving fixation trials, the fixation followed the path of the Gabor, offset horizontally 18.75 dva to its left. 

Procedure and design

Participants completed 24 trials, 12 with static fixation and 12 with pursuit. Participants were instructed to keep their gaze fixed on the black dot whether it was stationary or moving and to adjust the orientation of the Gabor's path, using a track pad, until it appeared vertical. The participants, all experienced psychophysical observers, reported that they were able to pursue the fixation, when it was moving, without difficulty throughout the trial. The static and pursuit trials were randomly intermixed. The procedure was otherwise similar to that of Experiment 1. 

Results

The magnitude of the illusion was reported as the angular offset from vertical required to make the path appear vertical. Figure 3 shows the results where the average value of the illusion across the six participants was 53.45° ± 2.46° for the static fixation condition and 55.54° ± 3.54° for the smooth pursuit condition. A t test showed no significant difference between the two conditions, revealing that the strength of the double drift illusion is unaffected by concurrent smooth pursuit that roughly stabilizes the Gabor on the retina. This suggests that the point at which the two motion vectors are combined must be at or following the point where the pursuit motion is combined with retinal motion to recover motions in the world. In the following experiment, we examine how the two vectors, internal and external, combine. 

In Figure 4, we plot the two data points from this experiment (the red and blue outline discs) along with the results from Experiment 1 and note that the new data fall on the same line as the results with the gray background from the first experiment. This confirms that with the internal speed and other parameters fixed, the external speed controls the illusion strength and not the factors of path length or reversal rate. The first experiment had a path length (4.69 dva) and reversal time (1,000 ms), both differing from the corresponding values in this second experiment. Nevertheless, the resulting external speed places the new data on the same function as the data from Experiment 1. Thus, the speed that governs the illusion strength here must be that of the external motion, in world coordinates, as that is the only frame of reference in which the external speed is matched for both the static fixation and pursuit conditions here. For comparison, the retinal speed of the external motion is dramatically different in the two conditions, being approximately 0 in the pursuit case. 

Figure 4

 

Illusion strength for the Pursuit and Static conditions (red and blue outline discs, respectively) plotted together with the data for the gray background of Experiment 1. The angular offset from vertical required to make the path appear vertical is plotted as a function of the external speed of the Gabor, and that is given by the path length/reversal time. Both the Pursuit and Static conditions had the same speed. The model of direction from Equation 3 is given by the dashed line for k = 0.81. The vertical bars show ±1 SE.

General discussion

We have found that for the parameters used in our experiment, the perceived direction of the double drift illusion is driven by a vector combination of its two motions, internal and external, as long as its positional uncertainty is high. When the drifting patch had high contrast with its surround (black background) giving it high positional certainty, the effect disappeared. Critically, the external motion vector that contributes to the effect is the motion in world coordinates, not retinal coordinates. When the external motion was nulled on the retina by pursuing a yoked fixation point, the illusion was undiminished. 

Incremental accumulation model

The double-drift effect has two components, first, an angle deviation where the Gabor's external direction is influenced by its internal motion and second, a continuous accumulation of position offsets where the next perceived position is offset from the current one in the illusory direction. Our experiment has addressed only the angle deviation and we will discuss it first before passing on to the question of the position shifts that accompany the effects on angle. 

The angle settings here are consistent with the replacement of the external motion of the Gabor with a combination of its internal and external motions as proposed by Tse and Hsieh (2006). The physical angle that makes a double-drift Gabor appear move vertically is tan⁻¹(k × v_i / v_e) where k is the relative weight of the internal, v_i, and external, v_e, motions in the vector combination (see Equation 2 above and the schematic in Figure 2). This function is plotted twice in Figure 3, once for equal weights of the internal and external motions (k = 1.0) as would be the case for a standard vector sum or average, and once for the best fitting value of k, 0.81. Tse and Hsieh (2006, Figure 5) found that k increased nonlinearly with increasing v_i, reaching ∼0.75 at the highest v_i tested, 6 dva/s, close to our k of 0.81 here for a v_i of 7.69 dva/s. Tse and Hsieh also found that changes in the external motion, v_e, had no effect on k, and that there was no interaction between v_i and v_e on the value of k. 

Figure 5

 

Continuous accumulation model. When the Gabor is equiluminant to the background, it has a noisy position signal, and the motion dominates the updating of position. From each current location, the next location is determined by extrapolating along the direction of the Gabor's motion. That direction has been shifted by the inappropriate combination of the true external motion and the internal motion so the extrapolated, perceived path drifts farther and farther away orthogonally from the physical path.

Even though this simple model accounts for perceived motion direction, it is not a model of positional mislocalization. In this case, we assume that when the position information from the Gabor is noisy, as it is on an equiluminant background, position judgments are dominated by predictions based on the Gabor's motion. From each current perceived location, the next location is an extrapolation along the perceived direction (e.g., Nijhawan, 1994; Whitney & Cavanagh, 2000; see Figure 5 here). What is unusual is that the incremental position offsets in the perceived direction continue to accumulate in the direction orthogonal to the physical path (the direction of the internal motion), moving the perceived location further and further from the physical path. The extrapolation in the direction of the Gabor's external motion simply matches the Gabor's physical displacements in that direction. 

One critical factor to generate the double-drift effect is the match between the mean luminance of the Gabor and that of the background. With the background maximally different from the mean luminance of the Gabor, as in black background condition of our Experiment 1, the illusion is eliminated. These results suggest that the high degree of positional uncertainty of the Gabor on an equiluminant background lets the motion signals contribute to the computation of position. Note that the Gabor may have high uncertainty for its position, but the motion of its envelope is registered with sufficient certainty to drive the illusion—with a higher relative weight than the internal motion. With the black background, the positional certainty of the grating patch is sufficient that there is little or no contribution of the internal motion in the computation of location—the value of k in Equation 2 is 0 when the background is black. Gurnsey and Biard (2012) reported a similar loss of the illusion when they kept the background midgray but rectified the Gabor, replacing the dark bars with the same background gray and leaving only the light bars. Again, with a loss of the match in the mean luminance, the Gabor was seen to follow its actual path, indicating that its motion was governed only by the external motion and the internal motion was ignored. 

Kwon et al. (2015) proposed a tracking model for the perceived deviations of the Gabor's path where the integration of the Gabor's motion would contribute to its perceived location when the Gabor's positional uncertainty was high. Their model is similar to our accumulation model above except that they have a saturation of the orthogonal offset that we do not observe in our stimulus conditions. They also proposed that the internal motion that is erroneously attributed to the Gabor's external motion would be lost or subtracted from the perceived speed of the internal drift. In other words, if all the internal motion was attributed to the external direction (k = 1), the Gabor should appear to have no internal drift. We did not evaluate the apparent velocity of the internal motion, but according to their proposal, it should be about 1/5 of its actual speed, since our value of k was about 0.8, indicating that about 80% of the internal motion was attributed to the external motion of the Gabor. Movie 1 can allow an informal judgment of whether this degree of slowing is seen. Its actual internal speed is 5 Hz, but Kwon et al.'s model predicts it should appear to be about 1 Hz. Compare this to Movie 3 with the black background to notice how the internal speed appears without the illusion. 

An alternative source for the illusory direction may lie in the paths of the individual features of the Gabors (Figure 6) rather than in a combination of two vectors that are encoded separately. This alternative predicts the same direction deviations and allows some additional inferences about the effects of eccentricity and Gabor size. Specifically, the light or dark bars of the Gabor themselves actually travel physically along the direction defined by the two motion vectors. If the Gabor were extended to fill the screen as a simple drifting grating, motion selective cells would signal the direction orthogonal to the grating bars—the internal motion direction. However, in the standard Gabor, the bars of the grating are short and somewhat blob-like. If the Gabor's bars are short enough, some cells may respond to the oblique displacement of the light or dark feature and this motion response is in the direction that combines the internal and external motions. If this is correct, the illusion should be reduced for Gabors with more bars as then there will be competing light and dark segments falling in the same receptive field, cancelling out the motion response. Moreover, the receptive fields of the responding cells need to be large enough to capture the diagonal traversal of a luminance peak, or trough predicting that the illusion strength should increase with eccentricity where receptive fields are larger and decrease for larger Gabors. This is indeed what Gurnsey and Biard (2012) report. However, recall that these local motion responses in the illusory direction would still be given by neurons that fall along the actual path (Figure 6, right); there is no deviation in the position of the stimulus from its actual location. The recovery of the combined vector from the direction of the Gabor features cannot alone account for the shift in perceived position that accompanies the shift in direction (Lisi & Cavanagh, 2015). We need to invoke an iterative extrapolation of position by the local combined motion direction (Figure 5) or local feature direction to explain the position offset. 

Figure 6

 

Local feature model for illusory direction. In (A), the schematized Gabor moves up and to the left, and its internal motion moves up to the right. The individual light and dark bars within the Gabor, however, are drifting straight up (shown as blobs for convenience). The actual path of individual light or dark bars follows the vector combination of the external and internal motion and pass through motion selective receptive fields (red ellipse) that signal that direction. In (B), the individual features make a series of short upward motions along the physical path as the Gabor sweeps up to the left. This is a space-time version of spatial effects like the Frasier spiral where a set of local segments oriented one way are arrayed in a series oriented another way (Morgan, 2015). Note also on the right that the local motions are all oriented upward, but there is no position shift. That must arise from accumulating shift that updates position at each moment based on motion direction.

One remarkable property of the double-drift illusion is the duration over which the illusion accumulates: a second or more. This accumulation does not occur when saccades are made to the Gabor (Lisi & Cavanagh, 2015) where it appears that the saccade plan either ignores the internal motion of the Gabor or only integrates it over a short interval. Unlike saccades, smooth pursuit did not reduce the illusion, letting the position shifts accumulate for a second or more. This is a striking increase over the 100 ms or so of accumulating position shift seen for a static Gabor with moving internal texture (Chung, Patel, Bedell, & Yilmaz, 2007). The direction of individual light or dark features of the static Gabor is also consistent with the position shift seen there. Nevertheless, the accumulation of position information from the static Gabor envelope appears to quickly override the motion cue and stops the position shift. There are other examples of corrective signals arriving with short time delays. For example, when an oblique grating is moving horizontally within a rectangular aperture, its motion is initially seen perpendicular to the orientation of the bars (Lorenceau, Shiffrar, Wells, & Castet, 1993). However, within 200 ms, the horizontal motion of the terminators propagates inward, changing the perceived orientation to horizontal. Similarly, MT neurons initially respond to the direction of motion that is perpendicular to a moving line (Pack & Born, 2001), but over a period of about 60 ms, shift their response properties so that they respond to the true motion of the line independent of its orientation, suggesting that the unambiguously moving endpoints of the line quickly generate a veridical motion solution (see also Pack, Gartland, & Born, 2004). In contrast, any such rapid correction seems to be lost for the double-drift illusion. Possibly, with the moving Gabor, the envelope is not stabilized long enough to contradict the position shift suggested by the motion. Our results with pursuit indicate that if the motion of the envelope is the key to avoiding a rapid termination of the accumulating position shift, then that critical motion is in world coordinates, not in retinal coordinates. The Gabor is roughly stabilized on the retina during the pursuit condition, and yet the position shift continues to accumulate. 

Coordinate frame of vector combination

In Experiment 2, we instructed participants to keep fixation on a spot that moved in tandem with the Gabor, keeping it roughly stable on their retina. They then adjusted the perceived path to be vertical. Their settings show that a similar offset from vertical was required to make the path appear vertical whether fixation was static or yoked to the moving Gabor. 

During smooth pursuit, objects that are stationary in the world are displaced across the retina, but this spurious retinal motion is mostly discounted to recover the true object motions in world coordinates (Wallach, 1959; Rieger & Lawton, 1985; Brenner & van den Berg, 1994; Freeman, 2001; Souman et al., 2005; Freeman, et al., 2009). In the case of the double-drift illusion with pursuit of the yoked fixation in Experiment 2, the Gabor will be roughly stable on the retina—its average motion will be near zero. Discounting of the retinal motion caused by the pursuit will recover, at some level, a perception of the Gabor's true displacement on the display screen. As we discuss above, the double-drift illusion arises from an inappropriate addition of the Gabor's internal motion to its real external motion. Experiment 2 suggests that this vector combination occurs along with or after discounting the consequences of the smooth pursuit eye movement. If the vector combination occurred before compensating for the pursuit, the Gabor envelope would be roughly static on the retina, and we would expect only the much smaller position offset typically seen for the static illusion (De Valois & De Valois, 1991; Ramachandran & Anstis, 1990). Previous research has shown that a static Gabor with internal drift is perceived to shift only a small amount, less than the width of the Gabor's aperture (Chung et al., 2007) and this position shift accumulates only over the first 100 ms of the presentation (Chung, et al., 2007). In contrast, the position shift for a moving Gabor (the double-drift stimulus) is large, several times the width of its aperture (Lisi & Cavanagh, 2015), and it accumulates over the entire duration of the presentation, here up to 1 s. If the vector combination occurs with or after the compensation for the smooth pursuit motion, then the Gabor recovers its external motion, and the illusion should be similar in magnitude to the double-drift with static fixation. This is what we found. 

Anatomical locus of vector combination

Our results in Experiment 2 showed that smooth pursuit has little effect on the perceived path of the double-drift stimulus. This suggests that the illusion arises at or beyond the neural populations that compensate for self-generated motions, taking retinal slip, image motion, eye and head motion into account to recover object motion in the world. There are several areas with such cells, specifically MST, V3A, V6, and posterior parietal (VIP). Galletti and colleagues originally reported these “real motion” cells in areas V6 and V3A in monkey as well as in early visual cortex (see review by Galletti & Fattori, 2003). However, Erickson and Thier (1991) and Thier and Erickson (1992) reported that the direction-selective cells they found in VI, V2, and V3 showed no compensation for eye movements. They also found that most cells in V4 and MT showed no compensation either. In contrast, cells in dorsal MST did responded preferentially to externally induced motion, suppressing responses to retinal motion caused by eye movements. This finding of cells in MST that responded to motion in the world independently of motion on the retina was supported by several subsequent studies (Dukelow et al., 2001; Ilg & Thier, 2003; Inaba, Shinomoto, Yamane, Takemura, & Kawano, 2007). Cells that compensated for eye movements were also reported in parietal areas (Duhamel, Bremmer, BenHamed, & Graf, 1997; Ilg, Schumann, & Thier, 2004). 

Results from human fMRI support the results from monkey physiology showing responses to real world motion in V6, V3A, and VIP (Fischer, Bülthoff, Logothetis, & Bartels, 2012; Nau, Schindler, & Bartels, 2018; Schindler & Bartels, 2018). Nau et al. (2018) also report real world motion responses in early areas V1 and V2 but suggested that this was due to feedback from higher areas. Such feedback may also have accounted for Galletti, Squatrito, Battaglini, and Grazia Maioli's (1984) initial reports of “real motion” cells in early visual cortex. 

These results indicate that the double-drift illusion must emerge at a later stage than early cortices V1, V2, V3, or MT for the illusion to persist undiminished during pursuit. Our results do not help us differentiate among the several candidate areas that do compensate for eye movements, principally, V3A, V6, MST, and VIP and of course any higher level areas that share this coding of real-world motion. Interestingly, individual cells in these areas can show responses to motion in world coordinates, but across each area, the receptive fields of the cells themselves are organized in retinotopic maps for position despite changes in eye position (Golomb & Kanwisher, 2012). The motion responses compensate for the retinal consequences of eye movements, but the position representation of the cells is not converted into world or spatiotopic coordinates. 

Finally, although the physiological and imaging results indicate that cells in these areas do compensate for the motion caused by pursuit eye movements, these findings do not reveal how that compensation comes about. Some authors have suggested that the common motion vector across the retina gives an estimate of the motion created by eye movements (Wallach, 1959; Rieger & Lawton, 1985; Brenner & van den Berg, 1994). Others propose that extraretinal signals, efference copy, from the motor system would provide the direction and speed of the motion that needs to be discounted (Freeman, 2001; Souman et al., 2005). Neither the studies reviewed above nor our findings with pursuit here differentiate between these alternatives. 

Conclusion

The double-drift illusion offers us a radical mismatch between perceived and retinal positions and between perceived and retinal motion directions. The illusion is affected by many factors: the eccentricity, contrast, size, internal speed, internal spatial frequency, and external speed of the Gabor, among others. Here we focus on the role of the external motion speed, and our results support the vector combination model for the perceived direction (Tse & Hsieh, 2006). The finding that the motion persists even when smooth pursuit has nulled the Gabor's motion on the retina suggests that the recovery of the combined motion vector occurs at a site where there are cells responding to real-world motion (e.g., V3A, V6, MST, VIP). The motion of an object in world coordinates is available in those areas, either computed locally, or biased by top-down feedback from higher level areas. However, our data do not address the question of the anatomical site where the combined motion vector then displaces perceived position continuously, creating an accumulating position offset, but, again, it must happen at or after the locus where the combination with eye movement signals occurs. 

Acknowledgments

The research was support by funding from the Department of Psychological and Brain Science, Dartmouth College (PC), a National Science Foundation Grant 1632738 (PUT), and a grant from NSERC of Canada (PC). The authors would like to thank Becky Zagans for running the participants. 

Commercial relationships: none. 

Corresponding author: Patrick Cavanagh. 

Email: . 

Address: Department of Psychology, Glendon College, CVR York University, Toronto, ON, Canada.