Looking ahead: The perceived direction of gaze shifts before the eyes move

Abstract

How do we know where we are looking? Our direction of gaze is commonly thought to be assigned to the location in the world that falls on our fovea, but this may not always hold, especially, as we report here, just before an eye movement. Observers shifted their gaze to a clock with a fast-moving hand and reported the time perceived to be on the clock when their eyes first landed. The reported time was 39 ms earlier than the actual time the eyes arrived. In a control condition, the clock moved to the eyes, mimicking the retinal motion but without the eye movement. Here the reported time lagged 27 ms behind the actual time on the clock when it arrived. The timing of perceived fixation in our experiment is similar to that for the predictive activation observed in visual cortex neurons at the time of eye movements.

Introduction

As the eyes move around, the point in the visual world that lands on our fovea shifts. It is commonly assumed that our perceived direction of gaze moves to that new point. This sense that it is our direction of gaze that shifted and not the world is an important component in seeing the world as stable despite displacement of the retinal image caused by eye movements. 

Cell recordings at the time of eye movements in a wide range of visual areas, including striate and extrastriate cortex (Nakamura & Colby, 2002; see also Merriam, Genovese, & Colby, 2007), Lateral Interparietal cortex (LIP, Duhamel, Colby, & Goldberg, 1992), and the Frontal Eye Fields (FEF, Sommer & Wurtz, 2002, 2006), have demonstrated that a proportion of visual cells in these regions respond to stimuli that will fall in their receptive fields before the eyes move to bring them there. The majority of these cells begin responding at the time of the saccade, or just following it, showing responses that are too early to be driven by the actual arrival of the stimulus in their receptive field (Kusunoki & Goldberg, 2002). In other words, the consequence of each eye movement is simulated in many visual areas ahead of the actual movement. The signal to remap receptive fields arises, at least in part, from a pathway to FEF that originates in the Superior Colliculus, which is closely associated with eye movement control (Sommer & Wurtz, 2002, 2006) Together these findings have supported the idea that a pre-saccadic signal from the oculomotor system allows the visual system to predict and discount spatial changes produced by an eye movement (see Helmholtz, 1890; Sperry, 1950; von Holst & Mittelstaedt, 1950). 

If the predictive activation of receptive fields is part of the processes that stabilize the visual world, we may also see a shift in the perceived direction of gaze just before an eye movement. This would suggest that the sensed direction of gaze is attributed to the predicted location of the fovea. Previous work is suggestive; Deubel, Irwin, and Schneider (1999) found that a probe flash had to precede a saccade by as much as 250 msec to be judged as simultaneous with the arrival at the saccade target. This large delay is far outside the range of predictive remapping, but the experiment required subjects to judge the relative timing of a flashed probe and an eye movement, and the difficulty of shifting attention between the probe and the saccade events may have contributed to the size of the effect. We used a more direct measure of the perceived time of arrival at a saccade target and we find a delay more consistent with the range found for remapping. We presented a large clock with single hand rotating at 1 rps (see Carlson, Hogendoorn, & Verstraten, 2006) as a saccade target. Observers made an 8° saccade to the clock, noted the time they arrived, and then adjusted a second clock to match this perceived arrival time (see Figure 1). Observers reported that it was easy to tell when they began looking at the clock as opposed to still viewing it at 8° in the periphery. The physical arrival time was taken as the position of the clock hand at the moment the eye monitor showed the eyes had landed on the clock. Thus we can observe the time that observers judged they were looking at the clock relative to when they actually did arrive. To control for various sources of lead or lag in reported arrival time, in a second condition the task and stimuli were the same, but the eyes remained steady and the clock moved to fixation, mimicking the retinal motion of the clock without an eye movement. 

Figure 1

 

A. Observer saccades to a clock with a spinning hand. B. Observer moves the hand to the position it was in when the eyes landed on the clock face.

Main clock experiment

Methods

Each of the 8 observers (one was the first author, the others were recruited from the Harvard Vision Lab) was seated in a dimly lit room, viewing a 20-inch, 85 Hz monitor from a chin rest positioned 37 cm away. Movements of the right eye only were monitored using a 120 Hz Eyelink I monitor (SR Research, Canada). Experimental sessions consisted of a block of 80 experimental and block of 80 control trials, with order counterbalanced across observers. Each block began with a 9-point calibration sequence. The initial display was a fixation circle (a black annulus) on a 50% gray background, 4° to the left of the center of the screen, and a clock face with a 4.4° diameter, with its center positioned 4° to the right of the center of the screen. The clock had a single clock hand pointed in a randomly selected direction. After the observer pressed the space bar and a stable fixation was detected, the hand began spinning in a clockwise direction at a rate of 1 rps. After 330 ms, a beep signaled the observer to make an eye movement to the clock face. The clock continued to spin for 670 ms, and then the screen cleared and a new clock was presented at the center of the screen with the hand pointed in a randomly selected direction, and the words “use l/r arrows to adjust and press space” printed above, instructing the observer to use the left and right arrow keys to move the clock hand to the time they remembered to be on the clock when the eyes arrived on its face (see Figure 1). Trials were rejected if the first eye movement was not executed in the interval between the saccade signal and the termination of the clock (10.8% of trials), or if the first eye movement did not land on the face of the clock (11.2% of remaining trials). The remaining saccades had a mean landing position 2 mm to the right and 4 mm below the actual center of the clock, with a standard error of less than 1 mm. The actual arrival time of the eyes was defined as the time on the clock when the eye movement velocity dropped below the saccade detection threshold of 30° per second (and the trial was only included in the analysis if the eyes were on the clock when this threshold was crossed). 

The control condition was the same, except that the observer remained fixated and the clock face moved to fixation. In a pilot version of the main experiment, saccade latencies to execute an eye movement to the clock had a mean of 235 ms, a standard deviation of 114 ms, and a duration of 42 ms. Therefore, the latency from the onset of the beep to the onset of the clock's movement to fixation was selected on each trial from a distribution with this mean and standard deviation to approximate the same timing parameters as the saccade condition. The duration of the motion was 47 ms (i.e., 6 frames). The observer was instructed to remain fixated throughout each trial, and to judge the time that was on the clock when it arrived at fixation. Trials were rejected if a saccade was detected before the clock arrived at fixation. These accounted for 5% of trials. 

Results

Results are shown in Figure 2. Average perceived arrival time in the saccade condition was 39 ms earlier than the actual time on the clock when the eyes arrived, and reported arrival times that were earlier than the actual arrival time comprised 67% of reports (20% of reports were times that occurred when the eyes were actually in motion). In the control condition, perceived arrival lagged behind the actual arrival by 27 ms, a delay likely related to the flash-lag effect (that is, the perceived position of a moving stimulus at the time of a nearby flash lies further along its motion trajectory (Nijhawan, 1994; Whitney, Murakami, & Cavanagh, 2000). 

Figure 2

 

Frequency of estimates of clock time when the eyes arrived (black) or when the clock arrived at fixation (gray), relative to the actual time on the clock. The vertical gray bar is the period during which the eyes were in motion.

The frequency of reporting arrival times that were earlier than the actual arrival times was reduced in the control condition (32%). The 66 ms difference in mean reported arrival times between the two conditions was significant [ t(7) = 3.16, p < .05, d = 1.2]. The results for individual subjects are depicted in Figure 3. 

Figure 3

 

Markers indicate the mean perceived time on the clock when each individual observer judged they were looking at it (red stars, saccade condition), and the time each observer perceived to be on the clock when it moved to fixation (control condition, green squares).

Variable clock speed and direction

A second experiment was conducted to determine if perceived arrival time varied as a function of clock hand speed or direction, and if the effects would change when the speed and direction were less predictable. Six observers were tested with clock hand speed (0.5/1 rps) and direction (clockwise/counterclockwise) varying from trial to trial. The experiment was the same as the main clock experiment above, except each of the 6 observers (including the first author) completed 120 trials of a saccade condition and 120 trials of a control condition (2% of trials were excluded). The speed (1 rps or 0.5 rps) and direction (clockwise or counterclockwise) of the clock hand motion varied from trial to trial in a pseudorandom order (30 trials of each type). Overall, results show anticipation of arrival time compared to their respective control conditions (see Figure 4). While the distributions are noisier, probably due to the smaller number of observations per cell as well as the reduced predictability of the hand motion, the pattern is similar to the main clock experiment, with the saccade distribution consistently shifted to the left of the control condition distribution. The proportion of reports of arrival times that were earlier than the actual time on the clock was higher for the saccade condition (74% of reports) than the control condition (36% of reports) [ t(5) = 2.97, p < .05, d = 1.5], and the means are consistent with this (35 ms of anticipation in the saccade condition, 72 ms of lag in the control condition) [ t(5) = 2.17, p < .1, d = 1.3]. Three observers were further tested with a faster clock hand, rotating at 2 rps. The experiment was the same as the main clock experiment above, except a 15-inch, 120 Hz monitor was used, and the clock hand rotated the face of the clock every 500 ms. Observers completed 60 trials of a saccade condition, and 60 trials of a control condition, in which the clock moving at 2 rps shifted to fixation as above (6.1% of trials were rejected). A similar result was obtained, with mean perceived arrival times in the saccade condition that were 42 ms earlier than the actual arrival time, and the control condition about equal to actual arrival times [ t(2) = 3.12, p < .1]. Of the reported times, 72% were earlier than the actual time in the saccade condition, and around 50% in the control condition (see Figure 5). 

Figure 4

 

Clock time estimates in saccade (black) and control (gray) conditions for varied clock hand speed and direction: clockwise (left column) or counterclockwise (right column); 1 rotation per second (top row) or half that speed (bottom row).

Figure 5

 

Clock time estimates in saccade (black) and control (gray) conditions, relative to the actual time on the clock, when the clock speed was 2 rps.

Clock hand appears on arrival

To determine if arrival time judgments were based on pre-saccadic information, we repeated the experiment with a clock hand that did not appear until the eyes arrived at the clock. Each trial began with a blank clock face, and the hand was presented as soon as the eye monitor detected that the eyes had crossed onto the clock face (or in the control condition, as soon as the clock arrived at fixation). The hand appeared at a random time position and began moving in a clockwise direction at 1 rps. In the control condition, the hand was presented, again at a random time position, on the last frame of the clock's motion to the fixation point. The experiment was otherwise the same as the main clock experiment. Five observers completed 80 trials of each condition (8.7% of trials were rejected). Here the perceived arrival time was 34 ms later than the actual arrival time, and the saccade and no-saccade conditions were similar ( Figure 5, [ t(4) = 0.34]). Observers did not reconstruct arrival time by retracing the clock's motion, visible after landing, backward in time. 

Digit stream

To test whether our conclusions would hold for an unpredictable stimulus with no motion, we used a rapid stream of digits presented at a rate of 20 Hz in place of a clock. Observers reported which digit was present when the eyes landed on the stream. The setup was the same as the main clock experiment, except for the following. A 15-inch, 120 Hz monitor was used, and each trial began with a fixation point 4° to the left of the screen center, and a plus sign 4° to the right of the screen center. Three observers (including the first author) were recruited from the Harvard vision lab. The observer pressed the spacebar to begin, and if a stable fixation was detected the stream of digits (in 64-point Helvetica font) would begin, each presented for 25 ms, with a 25 ms blank between digits. Five randomly selected digits were presented (a total duration of 250 ms), and then a beep instructed the observer to saccade to the stream, and then each of the digits 0–9 were presented in a random order (a total duration of 500 ms). The stream ended with a plus sign, which remained until the observer responded by using the number pad to key in which digit was present when their eyes arrived on the stream. In the control condition, the observer remained fixated and the stream shifted to fixation, generally following the same timing parameters as the clock motion. There were 80 trials in each condition. In addition to the inclusion criteria applied in the previous experiments, trials were also rejected if the saccade landed on the first digit in the stream, leaving no possibility for the report of an earlier digit. 18.1% of trials were removed from analysis. 

Here the most frequently reported digit was the one that was actually present when the eye movement arrived, and pre-saccade digits comprised 19.8% of responses (see Figure 6). While this may seem inconsistent with the clock results, it is important to consider the control condition, in which the digit stream shifted to fixation. Here observers usually reported the digit one or two positions (i.e., 50–100 ms) later in the stream than the one that was present when the stream arrived at fixation; they almost never reported digits from before the stream shifted to fixation (1.3% of responses). This difference in proportions was significant [ t(3) = 5.3, p < .05, d = 5.2]. On the whole, therefore, the pattern is similar to the clock results, though not as dramatic, underscoring the extra precision available in judgments of moving stimuli; motion deblurring (Morgan & Benton, 1989) allows sharper temporal estimates than is possible for a stationary sequence like the digits, in which successive items mask each other. The loss of precision in stimulus timing (50 ms steps with digits, versus as low as 7 ms with the clock hand) and measurement precision (standard error was 10.2 ms for the digit stream and 4.6 ms for the clock) with digit stream stimuli likely contributes to differences relative to the clock results. Not much of the difference between the two conditions could be due to the predictability of the clock hand movement, because this predictability is also present in our control conditions (where the clock face moved to fixation) and in standard flash-lag displays where the reported position always lags, not leads as it does in the clock condition (Figure 7). 

Figure 6

 

Clock time estimates in saccade (black) and control (gray) conditions, relative to the actual time on the clock, when the hand did not appear until the eyes arrived on the clock face.

Figure 7

 

Presentation time of the digit judged to be present when the stream was first fixated relative to the presentation time of the digit that was actually present, in the saccade (black) and control (gray) conditions.

Conclusions

When observers move their eyes to a clock, they report seeing a time that, on average, pre-dates their eye movement. This result suggests that the subjective experience of fixating a saccade target precedes the moment that the eyes actually arrive there. The effect is about 40–60 ms and is robust across clock hand speeds and directions. Pre-saccade information appears to be necessary for pre-dating saccade arrival time, because when no hand information is available before the eye movement begins, reported time lags behind the actual time in both saccade and control conditions (as the flash lag effect would predict). Finally, when an unpredictable stimulus (a stream of digits) was used instead, observers still experienced the moment of the eyes landing on the saccade target as occurring earlier, on average, than when the target shifts to fixation. 

Although presaccadic remapping provides a plausible explanation for this anticipatory percept of fixating the clock before actually arriving there, other explanations are possible. One alternative is that visual attention shifts to the saccade target before the eye movement (Deubel & Scheider, 1996), and this shift of attention is confused with the actual eye movement. This explanation was previously proposed to explain why a probe flash was reported as happening after an eye movement, even when it occurred as much as 250 ms before the eyes began moving (Deubel et al., 1999). But shifts in attention alone are clearly insufficient to produce the sensation of the eyes moving; attention moves around the visual field with no subjective impression that our eyes have moved. We therefore think it is unlikely that the initial shift of attention to the saccade target is the critical step; rather, we propose that remapping the saccade target back to the fovea just before the eye movement is the key. This remapping moves activity from its current location to the location it will have after the saccade (i.e., the fovea), to maintain a representation of its position across the eye movement. It is this step, and its timing, that appears to have the elements required to explain the mislocalization of the direction of gaze, as seen in our results. 

Other possible explanations are based on the distortions of duration around the time of a saccade. For example, when a saccade is made to an analog clock, the second hand appears to be stopped for longer than a second before moving on (chronostasis or the “stopped clock” illusion, Yarrow, Haggard, Heal, Brown, & Rothwell, 2001; but see Hunt, Chapman, & Kingstone, 2008). The proposed explanation for this distortion was that the post-saccadic signal is post-dated back to fill in the missing visual data from the saccade duration itself. This effect predicts that the clock in our experiment will appeared briefly stopped as the saccade lands on it. According to the chronostasis story, the observer may experience fixating the clock already from the beginning of the saccade (before landing on it) but the time seen on this back-dated clock must be a post-saccadic time. This is opposite to the effect we find here where the time seen on the clock, on average, pre-dates the onset of the saccade. Although a brief slowing of the clock's progression might be noticeable in our stimulus (we did not test this), it would not explain the report of an earlier time than the post-saccadic value. A post-hoc reconstruction of perception that fills in the “gap” in experience caused by an eye movement is not consistent with our current results and the neurophysiological data, which suggest that a predictive, rather than postdictive, process subserves perceptual continuity. 

Another suggested explanation for our results is a delay in perception at the time of an eye movement, so that observers report an earlier time on the clock because at the moment they arrive, the perception of the hand motion lags behind its actual position. Durations of events during a saccade are perceived as shorter than they actually are (Morrone, Ross, & Burr, 2005), which should produce an acceleration rather than a delay. Nevertheless, there are unavoidable delays in perception, and these delays may be accentuated during a saccade. However, these delays would also apply to the perception of where we are looking, so that at the time the saccade lands, perception would be showing an earlier time on the clock that was still 8° in the periphery. There would be no reason to report this time, because in this delayed percept, our gaze has not yet fallen on the clock. The fact that we see the hand in an earlier position at the moment when we land indicates different delays in our perception of the clock hand and the perception of our direction of gaze. Some function, we suggest remapping, must be acting on our perception of direction of gaze in the scene independently of our perception of the stimuli in the scene. 

Remapping allows locations to be continuously represented across the eye movement by maintaining both current and expected locations simultaneously, facilitating the transition between the two. Our findings show that visual experience may be influenced by this predicted representation, with the consequence that we think we are looking at our new target before our eyes arrive there. Our direction of gaze is not irrevocably tied to what is actually falling on the fovea, but may instead be attribute briefly to locations where the fovea will be once a saccade is completed. 

Acknowledgments

NIH EY09258 and Chaire d'Excellence, ANR (both Patrick Cavanagh's). 

Commercial relationships: none. 

Corresponding author: Amelia R. Hunt. 

Email: [email protected]. 

Address: School of Psychology, King's College, University of Aberdeen, Aberdeen AB24 2UB, UK.