Attention updates the perceived position of moving objects

Abstract

The information used by conscious perception may differ from that which drives certain actions. A dramatic illusion caused by an object's internal texture motion has been put forward as one example. The motion causes an illusory position shift that accumulates over seconds into a large effect, but targeting of the grating for a saccade (a rapid eye movement) is not affected by this illusion. While this has been described as a dissociation between perception and action, an alternative explanation is that rather than saccade targeting having privileged access to the correct position, a shift of attention that precedes saccades resets the accumulated illusory position shift to zero. In support of this possibility, we found that the accumulation of illusory position shift can be reset by transients near the moving object, creating an impression of the object returning to near its actual position. Repetitive luminance changes of the object also resulted in reset of the accumulation, but less so when attention to the object was reduced by a concurrent digit identification task. Finally, judgments of the object's positions around the time of saccade onset reflected the veridical rather than the illusory position. These results suggest that attentional shifts, including those preceding saccades, can update the perceived position of moving objects and mediate the previously reported dissociation between conscious perception and saccades.

Introduction

Visual motion signals can strongly affect the perception of position (see Whitney, 2002, for a review). For example, texture motion within a stationary object can shift that object's perceived position in the direction of the texture motion (De Valois & De Valois, 1991; Ramachandran & Anstis, 1990). More recently, it was discovered that this shift can become much larger if such an object moves in a trajectory orthogonal to its texture motion. Then, the illusory position shift can steadily accumulate over seconds, giving a sensation of a trajectory oriented between the directions of texture and object motions. This is the infinite regress, curveball, or double-drift illusion (Shapiro, Lu, Huang, Knight, & Ennis, 2010; Tse & Hsieh, 2006). 

Lisi and Cavanagh (2015) reported that when participants are asked to shift their gaze to a double-drifting object, their eyes land in approximately the veridical position rather than the illusory shifted position. Lisi and Cavanagh concluded that the information used by conscious perception is different from that which drives the saccadic eye movement. If so, this would constitute an important case of a dissociation between conscious perception and action (Goodale & Milner, 1992). 

As for other phenomena that suggest a perception-action dissociation, an essential question is whether the perceptual and motor tasks are truly comparable (e.g., Franz, Gegenfurtner, Bülthoff, & Fahle, 2000). Because the participants in the experiments of Lisi and Cavanagh (2015) judged the position perceived in different trials than they made saccades, the existence or nature of a shift of attention to the stimulus may have differed. While previous work has shown that attention is typically deployed at the saccadic goal position prior to a saccade (Deubel & Schneider, 1996; Kowler, Anderson, Dosher, & Blaser, 1995), this may not have occurred in the perceptual trials or involved a different kind of attentional shift, or one with different timing. 

Attention can improve the spatial resolution of visual processing (Yeshurun & Carrasco, 1998, 1999), and here we suggest that a shift of attention can update the position estimate used by perception, resetting the accumulated position shift. This is our Hypothesis 1. If attention shifts do indeed reset the illusory position accumulation, it is possible that the reason saccades target the veridical position is that they occur after a shift of attention has reset the illusion (Hypothesis 2). 

Our first two experiments find evidence for Hypothesis 1, that attention shifts can reset the illusory position shift. Indeed, the results suggest that attention can have a dramatic effect on the perceptual integration of position and motion information. Our third and last experiment more directly addresses Hypothesis 2, that the apparent perception-action dissociation is instead a result of a presaccadic attention shift. 

First, Experiment 1 finds that stimulus transients near the moving object can result in reset of the accumulation of illusory position shift, creating an impression of the object returning to its actual position. The results of Experiment 2 demonstrate that repetitive luminance and color changes to the object can also trigger reset of the accumulation. When participants were required to do a concurrent digit identification task while viewing the color change display, the illusion was restored. These results favor our Hypothesis 1 that attention can reset accumulated position shifts. 

In Experiment 3, participants were asked to saccade to the moving object at the time of a signal (disappearance of the fixation point). Just as Lisi and Cavanagh (2015) found, the participants’ eyes landed close to the veridical position of the object. However, we also flashed a probe object 500 ms after the saccade began. Participants judged the position of the probe relative to the moving object. By aggregating these judgments across trials, we reconstructed the average perceived position of the moving object around the time of saccade initiation. The results indicate that the perceived accumulated position shift was reset. This finding is consistent with our Hypothesis 2 that a shift of attention toward the moving object resets the illusory position accumulation, which results in the veridical position used for saccades. 

Experiment 1

If a shift of attention to the object eliminates the accumulated position shift, irrelevant transients close to the stimulus, thought to capture attention (Jonides, 1981; Remington, Johnston, & Yantis, 1992), should reset the illusion and result in a trajectory “jump” toward the veridical object location. 

To characterize the trajectory participants perceive, we asked them to directly draw the path they perceived after each trial of viewing a double-drift object. We used a drawing task to characterize the phenomenology because standard (e.g., nulling) methods need assumptions (that the illusion has a very particular spatiotemporal character, such that the manipulation will null it) that had not yet been justified for this illusion. For this reason, we see these data, as the first we know of in which participants can rather freely indicate what they see, as being a contribution on its own. 

Methods

Participants

Nine observers (four females, one author) participated. They had normal or corrected-to-normal vision and provided written informed consent. All experiments were conducted in accordance with the Declaration of Helsinki (2003) and were approved by the ethics committee of the University of Sydney. 

Apparatus

Images were displayed on a gamma-corrected 22-inch CRT screen (Mitsubishi Diamond Pro 2070SB; 1,280 × 1,024 pixels) with a frame rate of 75 Hz. The CRT resolution was 1.8 min/pixel at a viewing distance of 57 cm. Mean luminance was 45.8 cd/m² and CIE 1931 xy chromaticity was (0.61, 0.35) in the R, (0.29, 0.62) in the G, and (0.15, 0.07) in the B channel, respectively. Eye movements of the right eye were monitored at a rate of 1,000 Hz (EyeLink 1000 version 4.56; SR Research Ltd, Mississauga, Ontario, Canada). 

Stimuli

Visual stimuli were circular objects (1.8˚ in diameter) consisting of sinusoidal grating texture (0.8 cycle/deg) enveloped by blurry edges. The contrast profile for the envelope included a central constant region (plateau) with radius r₀ (1.2˚), surrounded by a sloping region where contrast falls by a cosine gradient to reach a zero contrast level (mean background luminance) by d (0.3˚). Mathematically, the contrast modulation (C) for each distance r from the center was  

\begin{equation}\begin{array}{@{}l@{}} C\left( r \right) = \left\{ {\begin{array}{@{}*{2}{l}@{}} {{C_0},\ }&{r \le {r_0},}\\ {{C_0}\cos \left( {\frac{{r - {r_0}}}{d}\pi } \right)0.5 + 0.5,\ }&{{r_0} < r \le {r_0} + d,}\\ {0,\ }&{r > {r_0} + d,} \end{array}} \right.\\ \end{array}\end{equation}

(1)

with \(r = \sqrt {{x^2} + {y^2}} \), and the object luminance distribution (for vertical grating) was  

\begin{equation}L\left( {x,y} \right) = {L_0}\left( {1 + C\left( r \right)\cos 2\pi fx} \right).\end{equation}

(2)

 

A pair of these circular objects moved for 1.8 s at 6.7˚/s diagonally (diagonally down and outward at an angle of 30˚ from straight down), one in the left and one in the right upper visual fields (Figure 1A; also see Supplementary Movies S1–S3). The center of the trajectory was 20˚ away from a black fixation point (0.2˚ in diameter) with a polar angle of 60˚ and 120˚. The orientation of each grating was parallel to the motion path, its maximum luminance contrast (C₀) was 0.5, and its initial spatial phase was randomized. The grating within the double-drifting objects drifted inward at 3.0˚/s, while that of the single-drifting objects was stationary relative to the envelope. 

Figure 1.

 

Schematic stimulus display and results of Experiment 1. (A) Two circular objects moved diagonally for 1.8 s: double-drifting (with orthogonal drift of the internal pattern) or single-drifting (no drift of the internal pattern) objects (see Supplementary Movie S1). Solid arrows (not shown during experiment) indicate the physical motion directions of envelope and grating for the double-drifting objects. White squares were flashed for one screen frame between 0.7 and 1.1 s after the motion began, except there was no flash in half of the double-drift trials (Supplementary Movie S2). Dashed arrows indicate possible perceived trajectories of double-drifting objects with flashes (Supplementary Movie S3). (B) Object trajectories sketched by two participants (P1 is an author, P8 is a naive participant). Rows correspond to participants and columns correspond to stimulus conditions. Display colors correspond to trial numbers and bold curves denote paths classified as jumps by maximum likelihood. Black diagonal lines indicate physical trajectories and pink-shaded bands indicate spatial locations of flash presentations. (C) Blue horizontal bars show average proportion of perceived trajectories classified as jumps (circles for individuals). (D) Blue horizontal bars show average likelihood ratio or T (circles for individuals). Error bars represent ± 1 SE across participants.

Procedure

All experiments were conducted in a dark room. During each single-drift trial and in half of the double-drift trials, white squares (0.7˚ × 0.7˚) were flashed on both sides of each of the two moving objects at a horizontal offset of 5.4˚ for 13.3 ms (one movie frame), which occurred at a random time between 0.7 and 1.1 s after stimulus motion onset. In each trial, participants maintained gaze on the fixation point throughout the moving object presentation. After the objects disappeared, the participant traced the object trajectories they perceived with a computer mouse. Participants could redraw their response as many times as they liked before confirming their response. Ten trials (except for Participant 9, who performed 20 trials) per stimulus condition were randomly interleaved. We removed trials that contained blinks or saccades (identified as eye movements over 30˚/s or 8,000˚/s²) in the interval from 0.7 to 1.5 s after the start of each trial. On average, 81% (SD = 13%) of trials were used in subsequent analyses. 

Results

Figure 1B shows the trajectories reported for double-drifting objects (left column), double-drifting objects accompanied by flashes (middle column), and single-drifting objects accompanied by flashes (right column). Pink-shaded bands indicate intervals wherein flashes were presented. It can be seen that the internal grating motion of the double-drifting objects biased the trajectories perceived toward fixation. The appearance of the flashes (middle column) frequently resulted in a sharp trajectory change, sometimes described by participants as involving a jump, toward the true position of the object. On the other hand, the single-drifting objects’ trajectory was relatively unchanged. In summary, the luminance flash frequently reset the accumulation of illusory position shift induced by double drifts. 

The variance of the trajectories drawn for different trials was nontrivial, even within a participant (see Supplementary Figure S1 panels for the trajectories by all participants). To quantitatively and reproducibly assess the proportion of trials in which the drawings indicated a trajectory change, we classified them as “jump” and “no-jump” trials by a statistical comparison of which trajectory shape fit best, using the likelihood ratio test (Neyman & Pearson, 1933). The no-jump trajectory model was represented by a quadratic curve (x = ay² + by + c; x and y denote horizontal and vertical positions; a, b, and c are free parameters) fit to the trajectory the participant drew, and the jump trajectory used the quadratic with two additional parameters specifying a horizontal jump (corresponding to the timing and the size of the jump). These were fit by maximum likelihood using the MATLAB fminsearch function assuming that the model's errors are normally distributed.  

\begin{equation}Likelihood = \mathop \prod \limits_{i = 1}^n {\rm{exp}}\left( {\frac{{ - {{\left( {{x_i} - {\mu _i}} \right)}^2}}}{{2{\sigma ^2}}}} \right).\end{equation}

(3)

 

In Equation 3, μ and σ denote the model prediction of horizontal position (x) and standard deviation of horizontal positions, respectively. The fits of the curves were very good—on average, the jump/no-jump model r² = 0.956/0.948 (SE = 0.017/0.018) for double-drifting objects without flashes, 0.958/0.912 (SE = 0.008/0.019) with flashes, and 0.985/0.980 (SE = 0.005/0.006) for single-drifting objects with flashes (also see Supplementary Figure S2 panels for the best-fit curves). The likelihood ratio test was used to decide whether the jump model fit significantly better than the no-jump model. This test used the maximum likelihood for the jump and no-jump models and the following test statistic (T), which follows a chi-squared distribution (degrees of freedom = 2).  

\begin{equation}T = 2log\left( {\frac{{Likelihood\ with\ jump}}{{Likelihood\ without\ jump}}} \right).\end{equation}

(4)

 

Perceived (drawn) trajectories were classified as “jump” paths if T values exceeded 5.99 (corresponding to p < 0.05), which are drawn in bold in Figure 1B. Figure 1C shows the proportion of the “jump” paths in each condition. Jumps were much more common for double-drifting objects with flashes compared to the other two conditions (one-way analysis of variance [ANOVA], F_{(2, 16)} = 23.96, p < 0.001, partial η² = 0.60; between double-drifting objects with vs. without flashes, t₍₈₎ = 6.18, p < 0.001, Cohen's d_z = 2.06; between double- vs. single-drifting objects with flashes, t₍₈₎ = 5.79, p < 0.001, Cohen's d_z = 1.93), while there was no significant difference between double-drifting objects without flashes and single-drifting objects with flashes, t₍₈₎ = 0.39, p = 0.70, Cohen's d_z = 0.13. As an alternative analysis, instead of using the T value to classify the trials and comparing their proportion, we compared T values between conditions directly. Figure 1D plots average and individual T values. Similar to the classification-based analysis, the T value is significantly larger for double-drifting stimuli with flashes (one-way ANOVA, F_{(2, 16)} = 17.35, p < 0.001, partial η² = 0.55; t₍₈₎ = 5.15, p < 0.001, Cohen's d_z = 1.72; t₍₈₎ = 5.05, p < 0.001, Cohen's d_z = 1.69; t₍₈₎ = 0.10, p = 0.92, Cohen's d_z = 0.03 in comparisons of the same order as above). The perceived jumps occurred at approximately the time of the flashes (relative to the flashes’ time, –0.04 ± 0.22 s, 95% CI), based on the curve fits after scaling the vertical distance of the sketched trajectory to that of the actual trajectory. 

These results indicate that presentation of the luminance flashes resets the accumulated illusory position shift to near zero. Why do the flashes lead to a position reset? Here are three possible explanations. First, the first-order motion energy of the flashes (Adelson & Bergen, 1985; Watson, 1986) might cause apparent motion percepts between the moving stimuli and the flashes, which might be mistaken by the participants for a trajectory change. Second, the flashes might create a gap in the perceived trajectory because of the fact that flashes can suppress certain percepts (Wolfe, 1984), although this account may find it hard to explain the change in trajectory before and after the flashes. As an alternative to the above explanations dependent on the stimulation of low-level luminance motion detectors, the flashes might capture the participants’ attention and somehow cause a change in the perceived trajectory. 

Experiment 2

If an attention shift caused the apparent resetting of the illusion in Experiment 1 (Hypothesis 1), an attentional load at fixation should reduce the occurrence of position resets, both without transients and in the presence of transients. To test this prediction, in Experiment 2, the participants were given an additional task of identifying digits embedded in a central stream of letters. This rapid serial visual presentation (RSVP) task restricts observers’ attentional resources, including for attention-based motion processing (e.g., Motoyoshi, 2011; R. Nakayama & Motoyoshi, 2017). 

Based on the results of Experiment 1, we expected that the repetitive transient changes to the object would regularly reset the accumulation and therefore the perceived path would be closer to the physical tilt than to the tilt of the perceived trajectory when transients were not presented. Additionally, if an attentional shift is necessary for the occurrence of position resets, an attentional load at fixation would restore the illusory shift even in the presence of transients. A simple judgment of motion path tilt was used for this experiment as it might be more robust to attentional demands than the previously used drawing task. 

Methods

Participants

Eight observers (six females, one author) participated both in the pretest and the main experiment. A different nine observers (seven females, one author) participated in the control experiment. They had normal or corrected-to-normal vision and provided written informed consent. 

Apparatus

Images were displayed on a gamma-corrected 22-inch CRT screen (Viewsonic G225f; 1,280 × 1,024 pixels) with a frame rate of 75 Hz. The CRT resolution was 1.8 min/pixel at a viewing distance of 57 cm. Mean luminance was 74.8 cd/m² and CIE 1931 xy chromaticity was (0.60, 0.33) in the R, (0.29, 0.60) in the G, and (0.16, 0.07) in the B channel, respectively. 

Stimuli

A circular object (as described in Equations 1 and 2) moved for 1.5 s at 6.7˚/s in either the left or right visual field (Supplementary Movie S4). The center of the motion path was on the horizontal meridian 10˚ away from a fixation point, and the path orientation was variable across trials. The grating orientation was always parallel to the path orientation, its luminance contrast was 0.3, and its initial spatial phase was randomized. The grating drifted either inward or outward at 3.6˚/s. 

Procedure

As a pretest to make the trajectory of the baseline illusion appear vertical and thereby make any resets even more conspicuous, we used a staircase to achieve subjective verticality of motion path orientation. In the pretest trials, participants judged the left/right tilt of the motion path after the presentation of the moving object. The physical orientation of the path was adjusted by a staircase with a step size of 8˚ and a one-up, one-down rule, which targets a 50% proportion of “right” tilt responses. A hundred trials per stimulus condition combining leftward/rightward texture motion and left/right visual field were randomly interleaved. For each condition, we estimated the point of subjective equality (PSE) as the vertical path corresponding to chance reporting of the tilt by fitting a logistic curve via maximum likelihood. 

In the main experiment, a double-drifting object moving at the path estimated as vertical in the pretest underwent repetitive color changes between magenta and cyan at 1.5 Hz (Supplementary Movie S5). The color changes consisted of a combined chromaticity and luminance modulation with one of three possible amplitudes for the luminance component—0% corresponded to subjective equiluminance with the background, meaning that only chromatic changes occurred. Mathematically, the object luminance (RGB intensity) distribution L_RGB was described by  

\begin{equation}{L_R}\left( {x,y} \right) = {L_0}\left( {1 + C\left( r \right)\cos 2\pi fx \pm C\left( r \right)} \right),\end{equation}

(5)

 

\begin{equation}{L_G}\left( {x,y} \right) = {L_0}\left( {1 + C\left( r \right)\cos 2\pi fx \mp IC\left( r \right)} \right),\end{equation}

(6)

 

\begin{equation}{L_B}\left( {x,y} \right) = {L_0}\left( {1 + C\left( r \right)\cos 2\pi fx} \right).\end{equation}

(7)

 

The temporal modulation corresponds to switching the plus minus signs (±) out of phase in Equations 5 and 6. The amplitude of the luminance component is controlled by the I parameter. If I_0% indicates subjective equiluminance, I_50% is \(\frac{{( {{I_{0\% }} + 1} )}}{2}\) while I_100% is 1. Subjective equiluminance was determined by flicker photometry (Ives, 1912) before the experiment started—participants minimized the subjective flicker by adjusting the green intensity (I) of the objects, positioned at the centers of the trajectories in both sides, alternating between the magenta and cyan colors at a rate of 8 Hz. 

From 0.2 s before to 0.2 s after the stimulus presentation period (1.9 s in total), a RSVP display appeared in the center of the screen instead of the fixation point. A sequence of 12 capital alphabetical letters (drawn from the alphabet but excluding I, O, Q, Y, and Z) was presented with a stimulus duration of 80 ms and separated by a blank interval of 80 ms (6.3 Hz). Each letter was drawn in Arial font in black and subtended approximately 0.75˚ × 0.75˚. Two of the letters were replaced by two digits chosen at random between 1 and 9. One of the two digits appeared at a random serial position between 2nd and 5th inclusive, while the other appeared between 7th and 11th inclusive. 

The single- and dual-task conditions were tested in separate blocks. In single-task trials, participants viewed the display while fixating the central RSVP letters and judged the left/right tilt of the motion path. Participants were instructed to concentrate on the moving object while keeping their gaze on the central letters. In dual-task trials, participants were first asked to identify the two digits in the central RSVP display by typing them on the keyboard. If the participants identified both digits in the correct order (auditory feedback was given), they then judged the left/right tilt of the motion path. On average, participants were able to respond within a few seconds after the stimulus presentation. Participants were instructed to keep digit identification performance as high as possible. 

Each task block began with practice trials, and in the dual-task block, only trials in which participants correctly identified the central digits were used in subsequent analyses. The average proportion of digits correctly identified was 92.6% (SD = 3.7%). 

Each task block contained, randomly interleaved, 30 trials per cell of the factorial design crossing luminance component amplitude, leftward/rightward texture motion, and left/right visual field. Dual-task trials for which participants’ digit identification was incorrect were not counted in this number—the experiment was continued until 30 trials per cell was achieved. 

Results

As a result of the pretest (Figure 2A), the opposite texture motions, shifting the perceived path orientations in their directions, produced 33.5˚ (SE = 3.0˚) difference in the subjective verticality on average. Note that the PSEs showed some centrifugal bias, which may reflect a general preference for motion moving away from the fovea (Albright, 1989; Ball & Sekuler, 1980). 

Figure 2.

 

Results of Experiment 2, including those for the pretest (A) and for the main experiment (B). (A) The path orientation found by the pretest to be subjectively vertical (the orientation for which participants were equally likely to give the left/right path tilt response) for each combination of leftward/rightward texture motion and left/right visual field (because of the likely occasional occurrence of resets in the pretest, this likely underestimated the orientation needed for subjective vertical in trials without resets). The result was used for the stimulus of the main experiment. (B) Blue bars show the average proportion of trials in which the correct tilt was perceived for the subjectively vertical (according to the pretest) trajectory (other markers show individual participants). The abscissa is the amplitude of the luminance component of the 1.5-Hz chromaticity and luminance modulation. The stimulus display was the same for the single- and dual-task trials, as was the primary task—path tilt judgment. The dual-task trials further required participants to identify two digits from a central stream of letters, which substantially decreased the proportion of correct reports, suggesting it reduced the incidence of resets. Data are collapsed across texture motion and visual field conditions. Error bars represent ± 1 SE across participants.

For the main experiment, Figure 2B shows the proportion of single- and dual-task trials in which the true path tilt was reported. Chance performance (0.5) would correspond to the trajectory on average being perceived as vertical, as in the pretest. We found that the correct tilt was perceived more often in the single-task trials than in the dual-task trials, suggesting reset occurred more often in the single-task trials, and the correct tilt also was reported more often with large luminance components (two-way ANOVA, F_{(1, 7)} = 15.78, p = 0.005, partial η² = 0.51 for task mode; F_{(2, 14)} = 27.04, p < 0.001, partial η² = 0.36 for modulation strength; F_{(2, 14)} = 2.11, p = 0.16, partial η² = 0.03 for interaction, after transforming proportion correct values into angular values \(arcsin( {\sqrt p } )\)). This suggests that an attentional load reduces resets, preserving the illusory trajectory that might otherwise be eliminated by transients. 

Strikingly, at subjective isoluminance (0% luminance amplitude) in the dual-task trials, participants reported the incorrect tilt significantly more often than chance (z = 3.10, p = 0.002). Why did participants so often report the incorrect tilt? If resets occasionally occurred even in the pretest, as we expected because Experiment 1 revealed occasional jumps even without transients, then the pretest staircase would have somewhat underestimated the orientation needed for subjective verticality for trials without a reset. This is because participants likely perceived close to the correct tilt in the trials with resets, biasing the staircase toward a smaller tilt. 

The reduction in resets relative to the pretest in the dual-task condition yields greater average accumulation of the position shift than in the pretest, driving the perceived orientation beyond the subjective vertical assessed in the pretest, resulting in report of the incorrect tilt. Again consistent with our hypothesis, this account implies that the availability of attention results in spontaneous resets, which are reduced by an attentional load. 

In summary, the results of this experiment indicate that it is a shift of attention toward a moving object that triggers the position reset, not something else about the effects of a luminance change. 

Control experiment for the orientation discrimination

An alternative explanation for our dual-task finding of markedly lower proportion correct judging the true path orientation is that discrimination thresholds increased (ability to do the task decreased) so much that participants were unable to discriminate the orientation change resulting from resets. 

To assess this possibility, we conducted a control experiment to compare discrimination thresholds for path tilt in the single- and dual-task conditions. Experimental stimuli and procedure were same as the pretest except that the texture grating was always a single-drifting object (grating vertical and stationary relative to the envelope). As in the main experiment, half of trials were single task and half dual task. The average proportion correct of digit identification was 90.0% (SD = 7.3%). For each visual field condition, we estimated half of the difference between path tilts at 20% (below chance) and 80% accuracy levels by fitting a logistic curve via maximum likelihood as the discrimination threshold. 

In the main experiment, resets likely changed the perceived tilt from approximately vertical to the true physical tilt used (determined by the pretest), which across participants averaged 15.6˚ and 17.9˚ for the left and right visual fields (SD = 4.9˚ and 4.0˚), respectively. But the control experiment (Figure 3) indicates that the impairment to path discrimination thresholds by the dual-task demand was only about 1.6˚ and 1.3˚ (SD = 2.5˚ and 1.8˚), respectively, while larger than zero (two-way ANOVA, F_{(1, 8)} = 5.64, p = 0.04, partial η² = 0.09 for task mode; F_{(1, 8)} = 1.13, p = 0.32, partial η² = 0.02 for visual field; F_{(1, 8)} = 0.16, p = 0.70, partial η² = 0.001 for interaction), which is too small to eliminate perception of the orientation change (15.6˚ and 17.9˚) caused by the resets. 

Figure 3.

 

Results of the control experiment for Experiment 2. Thresholds for discriminating motion path tilt (mean of nine participants). Error bars represent ± 1 SE across participants. In each trial, a single-drift vertical grating object was presented in the same manner as the pretest. Single task (tilt judgment only) and dual task (tilt judgment with digit identification same as the main experiment) were tested in separate blocks. The dual task had only little effect on the motion tilt thresholds.

Experiment 3

This experiment investigated how saccades interact with the accumulated position shift in perception. If the attention shift thought to precede saccades causes resets, the veridical position should be perceived around the time of saccades targeting double-drifting objects, supporting a different explanation than that of Lisi and Cavanagh (2015) that saccades have an independent, illusion-free representation of position. 

Methods

Six observers (four females, one author) participated. They had normal or corrected-to-normal vision and provided written informed consent. The same CRT and settings as in Experiment 1 were used. The circular object of Experiment 2 was used except that its texture drifted inward on every trial. 

Procedure

The double-drift object (which was gray-scale, identical to that of Experiment 1) trajectory was always at the perceptually vertical orientation estimated by a pretest (the pretest was done in the same way as in Experiment 2). 

A trial started after participants maintained gaze on the fixation point for 1 s. Participants were instructed to make a saccade to the moving object as soon as the fixation point disappeared. The fixation point disappeared at one of three times: 0.3, 0.6, or 0.9 s after the stimulus started to move. Sixty trials per stimulus condition combining left/right visual field and saccade-signaling time were randomly interleaved. 

As soon as the gaze position was detected to be 2˚ or further from fixation, the object was removed from the screen so that participants received no feedback about the accuracy of their saccades. The average saccade latency was 300 ms (SD = 25 ms) with saccade duration 60 ms (SD = 6 ms) and the object was removed 25 ms (SD = 1 ms) after the saccade onset. Five hundred milliseconds after the removal of the object, a white rectangle (0.3˚ wide × 1.8˚ high) was presented for 250 ms, horizontally offset from the object's last position. The horizontal offset was adjusted by the staircase one-up, one-down rule, with a step size of 0.9˚. Participants judged the left/right offset of the rectangle relative to the perceived position of the moving object. 

Gaze position was recorded at 1,000 Hz and monitored online; trials in which participants shifted gaze or blinked before the disappearance of the fixation point and trials with saccades not between 0.1 and 0.6 s after the disappearance were determined online and not included in further analysis, making the experiment longer by that many trials. 

After the experiment, we removed any trials with saccade landing points closer than 4˚ to fixation, which averaged 2.9% (SD = 2.3%) of trials. For each condition, we estimated the PSE for the perceived position as the object's last position corresponding to chance reporting of the offset by fitting a logistic curve via maximum likelihood. 

Taking the last perceived position (PSE) for each of the three saccade-signal times, we created a nominal perceived trajectory by computing the angle (slope) of the perceived positions against the three different saccade-signaling times via linear regression where horizontal coordinates were estimated as PSEs and vertical coordinates were actual last positions (at the object disappearance) averaged across trials. 

To calculate the corresponding nominal trajectory for the saccade targeting, for each participant, we fitted two separate linear models, one with the horizontal saccade amplitudes as the dependent variable and the horizontal coordinate of the object at the moment of saccade onset as the predictor, and one with the vertical saccade amplitude as the dependent variable and the object's vertical coordinate as the predictor (Lisi & Cavanagh, 2015). Then we used the fitted models to calculate predicted saccade amplitudes for each of the positions along the object's path. Finally, we computed a linear regression of the vertical on the horizontal predicted saccade amplitudes between trials and derived the trajectory's angle of deviation from vertical based on the regression. 

We tested the null hypothesis that this saccade-based path orientation is equivalent for the physical and the inferred trajectories with the Bayesian paired samples t test provided in the JASP statistical software (JASP Team, 2019). BF₁₀ is the ratio of the likelihood of the data under the alternative hypothesis divided by that under the null hypothesis. Values smaller than 1 indicate the data favor the null hypothesis over the alternative hypothesis. 

Results

As can be seen in Figure 4A, our experiment replicated the finding of Lisi and Cavanagh (2015) in that the path orientation represented by the saccade landing points (dots and dashed lines) was approximately the same as the physical path orientation (black lines). The two do not differ significantly from each other, and there is weak support for the null hypothesis (t₍₅₎ = 0.22, p = 0.83, Cohen's d_z = 0.09, BF₁₀ = 0.38 for rightward motion; t₍₅₎ = 0.43, p = 0.69, Cohen's d_z = 0.18, BF₁₀ = 0.40 for leftward motion) and they are significantly closer to the physical orientation than the subjectively vertical orientation found in the pretest (z = 8.17, p < 0.001 for rightward motion; z = 4.74, p < 0.001 for leftward motion). Another finding not as relevant to the critical issues is that, as Lisi and Cavanagh found, for some participants, the saccades fall short of the physical positions, while for others, they do not (see Supplementary Figure S3 for the results of all participants). 

Figure 4.

 

Results of Experiment 3. (A) Solid lines indicate physical path orientations (estimated without saccades in the pretest as subjectively vertical). Perceived positions (circles) and estimated motion paths (dashed lines) based on saccade landings (dots) for two of the six participants (P2, P4). The perceived positions were inferred from judgments of a probe presented soon after saccade initiation (signaled at one of three different times). Error bars represent 95% CI (determined by the bootstrap percentile method with 10,000 bootstraps). Dots indicate saccade landing points. Dashed lines are linear regressions of the saccade landings. Red/blue denotes left/right visual field (corresponding to rightward/leftward texture motion). (B) Average orientation of physical path (left), path estimated from perceived positions (middle), and path estimated from saccade landings (right). Error bars represent ± 1 SE across participants.

Figure 4A also plots the perceived positions (PSEs) as circles. Just as for the saccade landing points, the orientation of the nominal perceived trajectory (linear regression) indicated by these positions is similar to that of the physical trajectory (solid lines). These orientations are summarized in Figure 4B (physical vs. perceptual panels) and are not significantly different from each other (t₍₅₎ = 0.89, p = 0.41, Cohen's d_z = 0.36, BF₁₀ = 0.51 for rightward motion; t₍₅₎ = 0.55, p = 0.61, Cohen's d_z = 0.22, BF₁₀ = 0.42 for leftward motion). 

The resemblance of the perceived positions to the actual physical trajectory implies that they differ from the vertical orientation perceived during the pretest where there was no saccade task, and indeed they are statistically significantly different from vertical (z = 10.23, p < 0.001 for rightward motion; z = 32.60, p < 0.001 for leftward motion). 

These results suggest that the accumulation of illusory position shift is not reflected in perception around the time of the saccade. Combined with our previous results indicating that a shift of attention triggers the position reset, these results support the theory that the attention shift preceding saccades eliminates the accumulated position shift. 

An alternative explanation for the results of Experiment 3 is that participants used their postsaccadic gaze location as a proxy for the last location of moving object. This could result in the judgments being veridical regardless of whether the perceived location before the saccade was veridical or not. 

Although this alternative seems difficult to completely exclude, we note that under this account, the amount of saccadic overshoot arguably should affect participants’ responses trial by trial. Specifically, if participants used their postsaccadic gaze location as a proxy for the final object location, their judgments of the probe offset should have been biased toward the saccade direction or the opposite, depending on whether saccades relatively undershot or overshot. Since the staircase we used (one up, one down) converged at 50.11% “right” responses (SD = 0.81%) across trials, which means that the probe offset was often close to the overall mean value, raw responses (“left” and “right”) should have been associated with the amount of saccadic overshoot. 

Figure 5 compares across participants the amount of saccadic overshoot between “left” and “right” response trials. If participants used their postsaccadic gaze location as a proxy for the last location, the data for the left visual field (red) should be shifted toward the left top (larger overshoot for “right” trials), while the data for the right visual field (blue) should be shifted toward the right bottom (larger overshoot for “left” trials). Instead, the data approximately follow the diagonal or at least the tendency is inconsistent across participants (average perpendicular distance from the diagonal is 0.018˚ [SD = 0.13˚] for red and 0.015˚ [SD = 0.14˚] for blue toward the right bottom), thereby showing no or little bias associated with the amount of saccadic overshoot. 

Figure 5.

 

Saccade horizontal overshoot for trials in which a participant reported the probe was offset to the right (vertical coordinate) plotted against overshoot for trials in which the participant reported the probe was offset to the left (horizontal coordinate). Different markers are different participants, split into left visual field (rightward texture motion) trials in red and right visual field (leftward texture motion) trials in blue. Error bars show the 95% CI across trials.

Discussion

We found that an accumulated illusory position shift is often reset to near zero by luminance or chromatic transients in a moving object (Experiment 2). This reset, we suggest, is a result of attention being attracted by a transient. The perceptual position shift was also eliminated if the participant made a saccade to the moving object (Experiment 3). These findings are consistent with the proposition that a shift of attention, including those thought to precede saccades (Deubel & Schneider, 1996; Kowler et al., 1995; Rizzolatti, Riggio, Dascola, & Umiltá, 1987), can reset the illusory accumulation of the double-drift illusion. 

Lisi and Cavanagh (2015) found that saccades to a double-drifting object were unaffected by the illusion. In an important contribution to the long-running debate over how perception and action differ in their use of sensory signals, they suggested that this constituted the first clear and large difference between perceived position and saccade targeting. However, the present study suggests that the reason saccades are made to the correct position is explained by the associated allocation of attention (Corbetta, 1998) rather than a dissociation between perception and action. 

While saccades go to approximately the veridical position, Lisi and Cavanagh (2017) found that hand movements to point at a double-drifting object were biased toward the perceived illusory location. This is seemingly inconsistent with our proposition that a shift of attention eliminates the illusion because goal-directed actions should result in the allocation of attention toward the goal (Baldauf & Deubel, 2010). But hand movements usually take much longer than saccades, during which an illusory accumulation might recur enough to bias hand movements even after the object disappeared. In fact, a similar accumulation was reported for saccades and perceptual judgments made delayed from the object disappearance (Ueda, Abekawa, & Gomi, 2018). High-level motion and position signals might persist and continue the accumulation process. 

The present study is consistent with the possibility that rather than saccades and other actions having privileged access to the correct position, perceptual and oculomotor systems rely on the same position information (Van Heusden, Rolfs, Cavanagh, & Hogendoorn, 2018). Several previous studies have investigated stationary objects with internally drifting texture, which elicits a small position shift and relatively little accumulation (Chung, Patel, Bedell, & Yilmaz, 2007; De Valois & De Valois, 1991; Nishida & Johnston, 1999; Whitney, 2002). One study, while showing that the size of the shift varies with the stimulus used to compare position to, also found that it is smaller if compared to flashed lines than if judged by actions, specifically pointing and saccades (Kerzel & Gegenfurtner, 2005). We suggest that this result may reflect the flashed lines resetting the position shift close to zero, whereas the actions are guided somewhat by information after the attention shift, during which position shifts may accumulate (Chung et al., 2007; Nishida & Johnston, 1999). 

While an eyetracker was used to exclude trials with saccades in Experiment 1, smaller eye movements, termed micro-saccades, may be a concern as abrupt stimulus changes induce micro-saccades (Engbert, 2006). However, as can be seen in Supplementary Movie S6, the double-drift illusion still occurs when a fixation target in motion is pursued with the eyes (Cavanagh & Tse, 2019). Smooth pursuit regularly contains small saccades to catch up to a moving target tracked, but such small jitters in retinal image seem not to reset the accumulation of position shift. 

Although the results of Experiment 2 suggest attention contributes to reset, an attention shift may not be sufficient—both stimulus transients and attention might be required. This would be reminiscent of the possible joint role of transients and attention for binding changing visual features (Fujisaki & Nishida, 2010; Holcombe, 2009). If so, a presaccadic shift of attention would not be enough to cause the reset. In dual-task trials, however, an attentional load made the incorrect tilt perceived more likely (above chance at subjective isoluminance), thereby implying that in the pretest, availability of attention resulted in spontaneous resets (without transients), yielding underestimation of the subjectively vertical orientation. This finding on the role of top-down attention helps to adequately link the present results to our proposition that a presaccadic shift of attention can eliminate the illusion even without transients. 

Another possible explanation for the results of Experiment 3 is that participants compare the probe position with object position after it is updated postdictively after saccades, although we do not know of previous work suggesting this type of postdictive updating occurs. Another possible concern is that participants might use the display edges and corners as a reference for their position judgment and/or saccade tasks. 

The objects in the present experiments were likely attentionally tracked by the participants throughout most trials except the dual-task trials, since the participants’ task was to draw or judge the tilt of their trajectories. Diluting attentional tracking resources among four such objects (Alvarez & Franconeri, 2007; Holcombe & Chen, 2013) appears to have little effect on the double-drift illusion (Haladjian, Lisi, & Cavanagh, 2018). But the transients that accompany sudden visual changes have long been suggested to engage attention to a greater extent than typically occurs with top-down attention (e.g., K. Nakayama & Mackeben, 1989) and may even engage qualitatively different attentional processes than does top-down attention (Holcombe, 2009; Nishida & Johnston, 2002), such as resetting the phase of ongoing oscillations (Wood, Gu, Corneil, Gribble, & Goodale, 2015). However, because we found similar effects here with the saccade execution (or perhaps preparation) that involved no transients, we suspect that top-down attention can cause reset, although perhaps only when it is high in amplitude. 

Why might high intensity of attention to an object reset the position perceived during the double-drift illusion? Attention to an object likely brings additional resources to bear for estimating object position, possibly including detection of prediction error (e.g., Rao & Ballard, 1999) and activation of the primary visual cortex neurons stimulated by that object (Roelfsema, Lamme, & Spekreijse, 1998), with their greater spatial precision (Fischer & Whitney, 2009). For the optimal integration of position and motion signals, greater precision of position signals should reduce the weighting of the texture motion (Kwon, Tadin, & Knill, 2015) that otherwise creates the illusion. Additionally, an effect of attention could be to change the relative weight of current and past visual information, thus resetting the position error by decreasing the weight of past visual input. 

More concretely, if attention results in greater use of primary visual cortex representations, that may be sufficient to cause reset. Consider that the large illusory position offset that occurs without reset appears to reflect position coding by anterior areas with large receptive fields or perhaps no retinotopy at all (Liu, Yu, Tse, & Cavanagh, 2019). Although the primary visual cortex representations whose use improves position precision may themselves be shifted in the direction of object motion (Harvey & Dumoulin, 2016), because the amount of shift in retinotopic areas is proportional to the size of the receptive fields (Harvey & Dumoulin, 2016), the use of early visual cortical areas should greatly reduce the size of the illusion relative to the use of later cortical areas. 

This theory may also explain why saccades delayed until longer after object disappearance reflect the illusory rather than veridical positions (Massendari, Lisi, Collins, & Cavanagh, 2018; Ueda et al., 2018). The high-precision early visual cortical representations required by reset are largely driven by sensory input and thus should not be strong shortly after object disappearance. 

Acknowledgments

Supported by JSPS KAKENHI Grant JP18J01398 to RN. This study was carried out when RN was a JSPS research fellow; a visiting researcher at CiNet, NICT; and a visiting researcher at the University of Sydney. A portion of the study was presented at the 2019 annual meeting of the Vision Sciences Society, St. Pete Beach, Florida, USA (21/5/2019). 

Commercial relationships: none. 

Corresponding author: Ryohei Nakayama. 

Email: [email protected]. 

Address: National Institute of Information and Communications Technology, Japan. 

Supplementary material

View all videos in loop mode maintaining gaze at a fixation point. Repeated presentations seem to enhance the perception of illusion. 

Supplementary Movie S1. Single-drifting objects accompanied by flashes. Diagonal trajectories will be perceived as they are mostly regardless of luminance flashes (control condition). 

Supplementary Movie S2. Double-drifting objects. Texture motion will accumulate over seconds into a large illusory position shift of objects, whose diagonal trajectories will be perceived as straight down or slightly slanted in periphery (double-drift illusion). 

Supplementary Movie S3. Double-drifting objects accompanied by flashes. Accumulated position shift will be reset to near zero by luminance flashes, i.e., if one tracks a separation between two moving objects, a flash presentation will apparently increase the separation, changing object trajectories. 

Supplementary Movie S4. An example of double-drifting objects used in the pre-test. Accumulated position shift will result in perception of the incorrect path tilt. 

Supplementary Movie S5. Accumulated position shift will be reset regularly by a combined color and luminance modulation, allowing observers to perceive the correct path tilt. However, if attention to the object is reduced by a second task at fixation (digit identification), the incorrect path tilt will be perceived despite the presence of transients. 

Supplementary Movie S6. Double-drift illusion still occurs during smooth pursuing with the eyes a fixation target moving in parallel with the object, showing that eye movement is not a direct cause of resets.