Depth perception during saccades

Abstract

A number of studies have investigated the localization of briefly flashed targets during saccades to understand how the brain perceptually compensates for changes in gaze direction. Typical version saccades, i.e., saccades between two points of the horopter, are not only associated with changes in gaze direction, but also with large transient changes of ocular vergence. These transient changes in vergence have to be compensated for just as changes in gaze direction. We investigated depth judgments of perisaccadically flashed stimuli relative to continuously present references and report several novel findings. First, disparity thresholds increased around saccade onset. Second, for horizontal saccades, depth judgments were prone to systematic errors: Stimuli flashed around saccade onset were perceived in a closer depth plane than persistently shown references with the same retinal disparity. Briefly before and after this period, flashed stimuli tended to be perceived in a farther depth plane. Third, depth judgments for upward and downward saccades differed substantially: For upward, but not for downward saccades we observed the same pattern of mislocalization as for horizontal saccades. Finally, unlike localization in the fronto-parallel plane, depth judgments did not critically depend on the presence of visual references. Current models fail to account for the observed pattern of mislocalization in depth.

Introduction

Humans and other primates perform fast ballistic eye movements (saccades) on average three times a second. These saccades are accompanied by large changes in direction of gaze and ocular vergence. Even pure version saccades, i.e., saccades from one point of the horopter to another, induce transient changes in vergence caused by slight differences in the velocity profiles between the two eyes (Collewijn, Erkelens, & Steinman, 1988a). A typical horizontal saccade is accompanied by a brief divergent and a subsequent convergent component. Upward saccades show the same pattern, while downward saccades show an inverted pattern, i.e., an initial convergent and a subsequent divergent component (Collewijn, Erkelens, & Steinman, 1988b). 

A large number of studies have investigated the localization of perisaccadically flashed stimuli to test how the visual system accounts perceptually for the large changes in gaze direction (e.g., Cai, Pouget, Schlag-Rey, & Schlag, 1997; Honda, 1989, 1991; Lappe, Awater, & Krekelberg, 2000; Morrone, Ross, & Burr, 1997; Ross, Morrone, & Burr, 1997). Two distinct patterns of mislocalization in the fronto-parallel plane were observed which seem to depend on the presence of visual references and/or the ambient lighting conditions (Lappe et al., 2000; Morrone, Ma-Wyatt, & Ross, 2005a). In the dark, i.e., in the absence of visual references, perisaccadically flashed stimuli are uniformly mislocalized in the direction of the saccade vector when flashed prior to saccade onset, and in the opposite direction shortly thereafter (perisaccadic shift: Cai et al., 1997; Honda, 1989, 1991). In ambient light conditions and in the presence of visual references, perisaccadically flashed stimuli are mislocalized toward the endpoint of the saccade (perisaccadic compression: Morrone et al., 1997; Ross et al., 1997). 

So far, it is not known whether depth judgments are affected perisaccadically in a way similar to judgments about the location in the fronto-parallel plane. In the present study we examined perisaccadic depth judgments of stimuli flashed around the onset of horizontal and vertical saccades—both in the dark and in ambient light conditions. Our experiments were designed to test whether explanations put forward to explain the mislocalization in the fronto-parallel plane also apply to localization in depth. 

Methods

Experimental setup

Stimuli were presented binocularly on a Wheatstone stereoscope. Subjects were seated in front of a mirror system, head movements were restricted with a chin rest. The mirrors of the stereoscope were positioned 5 cm in front of the subjects' eyes. A PC with a 100 Hz dual head graphics card was used to present the stimuli on two identical 15″ monitors with a resolution of 1024 × 768 pixels each. Monitors were positioned 75 cm left and right of the mirror system, amounting to a total viewing distance of 80 cm. Pixel size was 1.3′. Positions of both eyes were measured with an infrared eye tracker (EyeLink2, SR Research Inc.) with a temporal resolution of 2 ms. 

Stimuli

Subjects initialized a trial by pressing a key on the computer keyboard. A fixation target which consisted of a long vertical bar (size 13′ × 90′) and a short horizontal bar (30′ × 13′) appeared either 3.25° to the left or to the right of the center of the display on the horizontal meridian ( Figure 1). After a fixed temporal interval ( t = 500 ms) an identical saccade target appeared at the opposite side with respect to the vertical meridian. The fixation cross remained on the screen and subjects were required to maintain fixation for a variable temporal interval ( t = 1000 ms–2000 ms). The subject's 6.5 degree right- or leftward saccade was induced by the disappearance of the small horizontal bar of the fixation target. 

Figure 1

 

Methods. Subjects performed 6.5 degree left and rightward saccades (A) while performing a disparity or a vernier task (B) of a briefly flashed bar. The bar was in front of, in the same depth, or behind the reference in the depth task. In the vernier task the bar was left, right, or at the same lateral position as the reference. The flashed bar could appear either above or below the fixation or the saccade target (C) which served as reference for the tasks in the corresponding trials. The experiment was performed either in presence or absence of additional visual references (D). Stimuli were presented on a Wheatstone stereoscope.

At 8 different times relative to this go-signal a small vertical bar (13′ × 90′) was flashed for a single frame (10 ms) either above or below the fixation or the saccade target. Depending on the position of the flash, either the fixation or the saccade target served as reference for the relative position task (see below). On any given trial, either the fixation or saccade target was pseudo-randomly chosen to serve as the reference. The choice of the reference did not depend on the timing of the flash relative to the saccade go signal. Similarly, the vertical position of the flash either above or below the reference was chosen pseudo-randomly. In different blocks of trials either the horizontal position or the depth of the flashed bar had to be judged relative to the reference. 

Vertical saccades

Conditions in experiments with vertical saccades were kept as similar as possible to those for horizontal saccades. However, a few changes were inevitable. The fixation target appeared either 3.25 degree above or below the center of the display. The flash appeared either to the left or to the right of the fixation or the saccade target, respectively. The flashed bar was slightly larger (45′ × 90′) than in the experiments with horizontal saccades. 

Depth judgments

In the blocks of trials which required depth judgments, the bar was flashed with either crossed, uncrossed or zero disparity. In a two-alternative forced-choice (2AFC) procedure subjects judged whether the flashed bar was in front of or behind the reference which defined the plane of fixation, i.e., zero disparity. The disparity offsets were chosen well above each subject's individual threshold as determined during steady fixation. For the horizontal saccades, disparity values of ±2.6 and 0′ were used for two subjects. For one subject with slightly larger disparity thresholds we presented disparity values of ±5.2 and 0′. A larger range of disparity values of ±7.8, ±2.6, and 0′ was used for the vertical saccades (see Results). 

To avoid subjects from being able to solve the task monocularly, the horizontal position of the flash was given a random lateral offset. The amount of this jitter was in the same order of magnitude as the horizontal disparity value. Accordingly, the horizontal position of the flash relative to the reference in one eye alone could not be mapped to a given horizontal disparity. 

Vernier judgments

In the blocks of trials which required horizontal vernier judgments, the bar was flashed either to the left, to the right or at the same horizontal position as the reference. In a 2AFC-procedure subjects judged whether the flashed bar was to the left or to the right of the reference. The magnitude of the lateral shift (±21′) was chosen well above each subject's individual threshold. 

Background illumination

Experiments 1 and 2 were performed under two different lighting conditions. Experiment 1 was performed in a completely dark room with a black monitor background that lacked any visual references. In order to minimize phosphor persistence effects and to reduce the background luminance below detection threshold, tinted panes were placed in front of the monitors in this condition. Under such conditions, perisaccadic shift can be observed reliably (Cai et al., 1997; Honda, 1989, 1991; Lappe et al., 2000). In Experiment 2, a gray background and two white horizontal rulers were permanently visible 4.3° above and below the fixation and saccade target. Under these conditions, perisaccadic compression can be induced reliably (Lappe et al., 2000; Ross et al., 1997). 

Luminance and phosphor decay in the two conditions were measured using a photometer (Photoresearch Spectraspot SPRD) and a fast photo diode (Siemens BPX63). Background luminance was 130 and 0.1 cd/m ² in the compression and shift condition, respectively. Stimulus luminance was 280 and 25 cd/m ², respectively. Phosphor decay times were measured as the decay time to 10% of the peak value after presentation of a stimulus. From this data we calculated the time it took for the stimulus luminance to drop below the luminance of the uniform background as 4 ms and 3 ms for the compression and shift condition, respectively. 

Data processing

Times of saccade onset were calculated offline with in-house software. Saccades were defined as contiguous epochs with a velocity above 54 degree per second and the maximum velocity surpassing a threshold of 162° per second. Saccade amplitude was defined as the displacement in the relevant direction during the saccade. Only trials with saccades of an amplitude larger than 3.2° were used for the analysis. The delay between stimulus presentation and eye-position signal was estimated to be below 5 ms. 

Responses of the subjects were coded as plus or minus one, corresponding to the right and left mouse-button, respectively. A moving average of the responses with respect to the time between flash and saccade onset was calculated offline for each stimulus condition separately. Moving averages were calculated using a Gaussian kernel with a standard deviation of 8 ms. In a bootstrap-procedure 100 identical moving averages were calculated, using re-sampled responses randomly drawn from the pool of responses acquired more than 50 ms before or after saccade onset. The boot-strapped moving averages were based on identical trial-times as the original moving averages. Only the responses for each trial were randomized. Time-resolved 5% and 95% quantiles of the 100 bootstrapped moving averages were used as lower and upper bounds to determine epochs during which subjects' responses were significantly different from what would be expected if depth judgments were not affected perisaccadically. The width of the confidence interval is mainly a function of the number of trials in a certain time bin. Time bins with many trials will yield a narrow confidence interval. 

Whenever five different relative disparity values were presented, time-resolved psychometric functions were determined. At each time point, cumulative Gaussians with two parameters were fit to the five moving averages. The same process was repeated 100 times for the 100 bootstrapped moving averages which were calculated as described above. This procedure rendered confidence intervals for the parameters of the psychometric function under the assumption that depth judgments were not affected perisaccadically. As for the individual moving averages, the width of the confidence intervals for the psychometric parameters decreases with the number of trials in that time bin. 

Results

Experiment 1: Depth and vernier judgments in the absence of visual references

In Experiment 1, we tested depth perception during left and rightward horizontal saccades in the absence of visual references. Such conditions have previously been shown to elicit a shift-like pattern of mislocalization in the fronto-parallel plane (Cai et al., 1997; Honda, 1989, 1991; Lappe et al., 2000). Three human observers performed vernier and depth judgments of flashed stimuli relative to a continuously present reference, while performing 6.5 degree left- and rightward saccades. Figures 2A and 2B show the results of the vernier and depth task, respectively. Responses pooled over subjects and saccade directions are shown as a function of time relative to saccade onset. In the case of correct performance, the green and the orange data lines would run at the bottom and the top of the panel, respectively. However, differing from veridical perception and confirming previous results (Cai et al., 1997; Honda, 1989, 1991; Lappe et al., 2000) vernier judgments revealed a biphasic shift pattern (see Figure 2A). Prior to saccade onset, stimuli were mislocalized in direction of the saccade vector (orange and blue line in Figure 2A, −50 to 0 ms relative to saccade onset). In contrast, stimuli were mislocalized in direction opposite to the saccade direction when flashed after saccade onset (green and blue line in Figure 2A, 0 to 30 ms after saccade onset). The timing of the shift pattern is slightly earlier than has been reported in previous studies (Cai et al., 1997; Honda, 1989, 1991; Lappe et al., 2000). 

Figure 2

 

Vernier (A) and depth judgments (B) in the absence of visual references. Moving averages of the results of the 2AFC vernier (A) and depth (B) judgments as a function of time relative to saccade onset at t = 0 ms. Both panels show moving averages with data pooled across all subjects. Individual subjects show similar results (data not shown). Different colors correspond to different stimuli. In the vernier task, the blue color represents stimuli with the same horizontal position as the reference. The green and orange color corresponds to stimuli, which are positioned relative to the reference in the opposite and the same direction of the saccade vector, respectively. For example, for rightward saccades, the green color corresponds to stimuli right of the reference. For leftward saccades, the green color corresponds to stimuli left of the reference. In the depth task, orange corresponds to crossed disparity, blue to zero disparity, and green to uncrossed disparity. Dotted lines demarcate the confidence limits (see Methods). Bright dots on the solid lines denote periods when the moving averages were outside of these confidence limits. In the lower panel, mean and standard deviation of perisaccadic changes in gaze-direction (A) and ocular vergence (B) are depicted by the blue line and blue shaded area, respectively. (A) In the vernier task, subjects judged the flashed stimulus as being either left or right of the reference. For the analysis, these judgments were coded as being either in the same or opposite direction of the saccade vector. For example, during rightward saccades a ‘right’-response corresponded to a ‘same’-judgment. Vernier judgments reveal the expected biphasic pattern of mislocalization in direction of the saccade vector before saccade onset (orange and blue line from −50 to 0 ms relative to saccade onset) and contrary to its direction after saccade onset (green and blue line from 0 to 30 ms after saccade onset). (B) Depth judgments reveal a distinct pattern of mislocalization in depth. Stimuli flashed from −20 to 30 ms relative to saccade onset were mislocalized to a closer depth plane (green and blue line), while stimuli shortly before and after this period tended to be mislocalized to a farther depth plane (orange and blue line).

Figure 2B shows the results of the 2AFC depth judgments. Stimuli flashed more than 100 ms before or more than 50 ms after saccade onset were correctly perceived to be in front of the reference cross when presented with crossed disparity (orange line in Figure 2B). In the same time interval, stimuli with uncrossed disparity were reliably judged as being behind the reference (green line). However, almost all stimuli flashed from −20 to +30 ms relative to saccade onset were perceived to be in front of the reference, even if they were presented in the same or a farther depth plane (blue and green lines in Figure 2B). In contrast, stimuli flashed around 100 to 50 ms before or 30 to 70 ms after saccade onset had a tendency to be mislocalized to a farther depth plane (blue and orange line). 

For vernier as well as the depth judgments we observe a decrease in the width of the confidence interval toward the time of saccade onset. Time bins long before or long after saccade onset typically have wider confidence intervals. This is accompanied by a general increase in variability of the moving averages which is especially evident for the zero shift and zero disparity condition. Both effects are due to differences in sampling density in different time-bins. The higher sampling density in the time window from −50 to 50 ms after saccade onset decreases the variability of the moving averages which in turn leads to the narrower confidence intervals. This effect is especially evident in the zero-shift and zero-disparity conditions because the variance of a binomial distribution peaks at a probability p of 0.5 which is close to the response probability for these conditions. 

Experiment 2: Depth and vernier judgments in the presence of visual references

Localization in the fronto-parallel plane depends critically on the presence of visual references and/or the background lighting conditions (Lappe et al., 2000; Morrone et al., 2005a). In the second experiment we tested whether localization in depth shows a similar dependency. Three observers performed 2AFC vernier and depth judgments in conditions known to elicit perisaccadic compression (Lappe et al. 2000; Morrone et al., 1997; Ross et al., 1997). Figures 3A and 3B show the results of the vernier and depth task, respectively. 

Figure 3

 

Vernier (A) and depth judgments (B) in the presence of visual references. Conventions as in Figure 2. (A) Vernier judgments provide evidence for mislocalization of flashed stimuli toward a point close to the saccade target (saccadic compression). (B) Depth judgments reveal a similar pattern of mislocalization in depth as observed in the absence of visual references. In this case, however, the flanking regions, i.e., the periods with mislocalization to a farther depth plane, are less prominent.

The vernier judgments no longer showed the biphasic shift pattern observed in the data of Experiment 1. Instead, they revealed a pattern, which was compatible with perisaccadic compression. Stimuli flashed beyond the saccade target cross were mislocalized opposite to the direction of the saccade when flashed around saccade onset (green line, upper panel of Figure 3A). All other stimuli were mislocalized in direction of the saccade (blue and orange lines). Compression occurred slightly earlier than would be expected from previous studies (Lappe et al., 2000; Morrone et al., 1997; Ross et al., 1997). Overall, this pattern is consistent with compression toward a point in space close to the saccade target. 

Figure 3B shows the results of the depth judgments during conditions which cause perisaccadic compression. A similar pattern of mislocalization in depth could be observed as in the condition without visual references. Further, there was no apparent difference between stimuli flashed at fixation and saccade target. However, while the effect of mislocalization to a closer depth plane around saccade onset was equally pronounced, the temporal flanking periods with mislocalization toward a farther depth plane seemed less pronounced than in the condition without visual references. Finally, strongest mislocalization was observed at saccade onset, i.e. at the same time as in Experiment 1. 

Timing of mislocalization towards a closer depth plane

So far, our results suggest that mislocalization toward a closer depth plane starts well before saccade onset. It is critical to rule out that phosphor decay or temporal smoothing used to calculate the moving averages are responsible for the early onset (see Methods). We measured phosphor decay times in the range of 4 ms (see Methods). Thus, it seems unlikely that phosphor decay might account for the early onset of the mislocalization which starts about 25 ms before saccade onset. 

In our previous analyses we had used a symmetric Gaussian kernel with a standard deviation of 8 ms to smooth the data in time. This smoothing might have contributed to the early onset of the effect. To rule out this possibility we repeated the analyses using a Gaussian with a smaller standard deviation (3 ms), a rectangular kernel with a total width of 10 ms as well as an asymmetric exponential kernel with a half-height decay time of 3.5 ms. Using the bootstrap method described above we determined time points where the moving average deviated from what would have been expected based on the distribution of responses to stimuli flashed more than 50 ms before saccade onset. 

Pooling all responses to uncrossed stimuli from Experiments 1 and 2, we found evidence for mislocalization toward a closer depth plane as early as 25, 22, and 19 ms before saccade onset for the three kernels, respectively. For the zero disparity stimuli the values were 20, 19, and 16 ms, respectively. Thus, our results clearly support the notion that the effect of mislocalization toward a closer depth plane starts well before saccade onset. 

Similarly, we estimated the end of the period of mislocalization to a closer depth plane. In this case, the asymmetric exponential kernel was inverted in time. For the stimuli with crossed disparity we found values of 29, 25, and 23 ms, respectively, for the three kernels. For the zero disparity stimuli we found values of 37, 32, and 30 ms, respectively. Given an average saccade duration of 33 ms, our data suggest that the effect of mislocalization probably does not persist beyond saccade offset. 

As mentioned previously, perisaccadic compression and perisaccadic shift appeared earlier than would be expected from the literature. If this discrepancy were to be caused by delays in the system this would also affect our estimate of the time of onset of mislocalization toward a closer depth plane. However, the delay was estimated to be below 5 ms and hence does not change our interpretation that the effect starts before saccade onset. 

Experiment 3: Depth judgments during vertical saccades

The mislocalization in depth observed in Experiments 1 and 2 might have been caused by perisaccadic suppression or changes in attentional load imposed by saccade planning and execution. From these experiments it is not clear whether the transient changes of ocular vergence contribute to the mislocalization. To investigate this possibility we tested whether the pattern of mislocalization depends on the sign of the vergence changes during saccades. To that end subjects performed up- and downward saccades which have been shown to give rise to inverted pattern of vergence changes (Collewijn et al., 1988b): upward saccades are accompanied by an initial divergent and a subsequent convergent component, much like horizontal saccades. Downward saccades, in contrast, are accompanied by an initial convergent and a subsequent divergent component. 

Two observers performed 2AFC depth judgments of flashed stimuli relative to a continuously present reference while performing 6.5 degree up- and downward saccades. Judgments were performed in the presence of visual references. Vernier judgments were not measured for vertical saccades, because it had been shown previously that there are no differences in mislocalization in the fronto-parallel plane between horizontal and vertical saccades (Honda, 1991). 

Pilot experiments had revealed strong effects of perisaccadic suppression during vertical saccades. A considerable fraction of perisaccadic flashes were not seen during vertical saccades, while during horizontal saccades subjects had detected almost all flashed stimuli. To account for this difference, the width of the flashed bars was increased (see Methods). Further, we used a wider range of disparity values in order to fit time-resolved psychometric functions to the data. 

The results of the depth judgments during vertical saccades are presented in Figure 4. The pattern of mislocalization during upward saccades was similar to that observed for horizontal saccades. Stimuli flashed around saccade onset tended to be perceived in a closer depth plane, while stimuli flashed shortly before or after this period tended to be mislocalized to a farther depth plane. However, for downward saccades, a clearly different pattern emerged. Although perisaccadic depth judgments were impaired, there was no tendency to perceive perisaccadically flashed bars in front of the reference. Additional analyses revealed that for downward saccades, perisaccadic performance for large crossed and uncrossed disparities was significantly above chance-level. 

Figure 4

 

Depth judgments during vertical saccades. (A) Conventions similar to Figure 2B. However, green and cyan colors correspond to large and small uncrossed disparities, blue corresponds to zero disparity, and red and orange correspond to large and small crossed disparities. (A) During upward saccades, a similar tri-phasic pattern of mislocalization in depth can be observed as for the horizontal saccades. (B) Depth judgments during the downward saccades reveal a different pattern. Depth judgments appear to be slightly impaired during saccades but show no clear indication of a shift in depth.

Experiment 4: Comparison of depth judgments during horizontal and vertical saccades

In Experiments 1 and 2, only a single very small disparity offset was used. Hence, we performed an additional experiment with horizontal saccades using the same stimuli and range of disparities as for the vertical saccades in Experiment 3. This enabled us to directly compare psychometric functions for upward, downward, and horizontal saccades. Figure 5 shows the results for the left- and rightward saccades separately. The results are consistent with Experiments 1 and 2. 

Figure 5

 

Depth judgments during horizontal saccades. Conventions as in Figure 4. Background lighting conditions as in Experiments 2 and 3. Results with 5 different disparity values are similar to the ones reported with 3 disparity values in Figures 2B and 3B. Furthermore, leftward (A) and rightward (B) saccades reveal a similar pattern of mislocalization in depth.

We separately fit time-resolved psychometric functions to the data for rightward, leftward, upward and downward saccades (see Methods for details). Furthermore, the data were split according to the position of the flash at the fixation or saccade target. Figures 6A and 6B show the estimated parameters for flashes at the fixation and saccade target, respectively. The panels in Figure 6A show the mean of the fitted cumulative Gaussian (point of subjective equality, PSE). PSE values above 7.8 min or below −7.8 min arc depend on extrapolation, as the largest disparity offsets were ±7.8 min arc. Thus, the precise values of parameter estimates close to or outside this range are uncertain. The panels in Figure 6B display the width of the fitted Gaussian (just noticeable difference, JND) as a function of time of the flash relative to saccade onset. Data are displayed separately for rightward (blue), leftward (green), upward (orange), and downward (red) saccades. Time points at which the parameter estimate is outside the confidence interval (see Methods) are indicated by a bright dot in the corresponding color. 

Figure 6

 

Time-resolved psychometric functions for stimuli flashed during leftward (green), rightward (blue), upward (orange), and downward (red) saccades. (A) For horizontal and upward saccades PSE shifts toward uncrossed disparities around saccade onset. In all cases the PSE shifts beyond the range of displayed disparities (±7.8 min arc) and hence depend on extrapolation. Further, shortly before and after this period the PSE shifts toward crossed disparities. Mislocalization for the downward saccades (red line) shows a clearly different pattern. First, the amplitude is much smaller and the pattern differs for stimuli presented near the fixation and the saccade target. (B) Estimates of the JND are significantly elevated from about 40 ms before saccade onset at the fixation cross and 50 ms before saccade onset at saccade target. The JND returns to values within the confidence limit about 30 to 40 ms after saccade onset.

We observe a significant increase in JND around saccade onset. The increase starts between 50 and 40 ms before saccade onset and lasts until 30 to 40 ms after saccade onset. In most cases the PSE shifts beyond the range of displayed disparity values (see below). In those time-bins, both the PSE and JND depend mainly on extrapolation and the precise values should be interpreted with caution. 

The main effect of mislocalization toward a closer depth plane around saccade onset is reflected as a shift of the PSE toward positive disparity values (farther depth plane) for the horizontal and upward saccades. For these cases, the PSE reaches or exceeds the critical value of 7.8 min arc. In contrast, a clearly different pattern is observed for downward saccades: for stimuli flashed at the fixation target, the amplitude of the effect is greatly reduced and the timing is less precise. For stimuli flashed at the saccade target, the direction of the shift in PSE is even reversed: perisaccadic flashes are perceived in a farther depth plane. 

The analysis further reveals that for horizontal and upward saccades, the PSE is often significantly smaller than zero shortly before and after the period of mislocalization toward a closer depth plane. This effect is not quite as consistent as the main effect and seems to be stronger at larger eccentricities: the effect is larger for stimuli flashed at the saccade target before saccade onset and for stimuli flashed at the fixation cross after the saccade. For flashes at the fixation target a significant shift in PSE toward negative disparities can be observed prior to and after the mislocalization toward a closer depth plane. Also, a significant shift toward a farther depth plane can be observed for stimuli flashed at the saccade target before saccade onset. Only for stimuli flashed at saccade target after the saccade the effect does not reach significance. Overall, the analysis of the moving psychometric functions supports the notion of three distinct phases of mislocalization (farther–closer–farther) for stimuli flashed during horizontal and upward saccades. 

Eye-movement analysis

The most important characteristics of the eye movements, i.e., time-resolved mean version and vergence in the conditions, which require vernier and depth judgments, respectively, are plotted in the lower panels of Figures 2, 3, and 4. Figures 2A and 3A show that the average amplitude of horizontal saccades was in the expected range (6.2 ± 0.8°), confirming that subjects performed the saccades as instructed. Saccades had slightly longer amplitudes in the shift condition as compared to the compression condition (6.22 ± 0.78° versus 6.08 ± 0.73°). Average saccade duration was 32 ± 4 ms. In line with the larger amplitude, we also found a slightly longer duration for the saccades in the shift condition (33.5 ± 3.8 ms versus 32.3 ± 4.1 ms). Despite the small size of the effects they reached significance in a linear model with the factors ‘Subject’ and ‘Background condition’. 

Figures 2B and 3B show the typical biphasic pattern of transient vergence responses with an initial divergent and a subsequent convergent component. Consistent with reports in the literature (Collewijn et al., 1988a), the amplitude of the vergence movement was in the range of 0.5° visual angle (mean cumulative divergent eye movements in the first 20 ms after saccade onset: 0.67 ± 0.44°). Figures 4A and 4B show that this pattern is reversed for both upward and downward saccades. Here we observed an initial convergent movement followed by a subsequent divergent movement. The amplitude of this pattern was larger for the downward saccades. 

The results of Experiment 3 have revealed striking differences between upward and downward saccades. Some of these differences might be due to differences in saccade parameters such as amplitude, duration, peak velocity, and amount of vergence. Thus, we performed a more detailed analysis of the upward and downward saccades, which might help to understand the different mislocalization patterns in the two conditions. A linear model with the factors ‘Subject’, ‘Saccade Direction’ and their interaction term was fit to the data. We found a significant effect of saccade direction on amplitude, duration, peak velocity and amount of convergence. As we were only interested in effects which were uniform across subjects, we performed post-hoc t-tests within subjects and screened for significant effects with identical sign for both subjects. This reduced the potential candidate parameters to saccade duration and amount of convergence. In order to account for the large qualitative differences in mislocalization between the upward and downward saccades the effects should not only be significant, but also substantial in size. Saccade duration was 3.5 ms longer for downward as compared to upward saccades. Given the mean duration of 36.8 ms, this amounts to a modulation of 9%. Amount of cumulative convergence in the first 20 ms after saccade onset was 0.6° larger for the downward as compared to the upward saccades. Given a mean cumulative convergence of 0.6°, this amounts to a modulation of 90%. 

Discussion

We examined depth judgments for stimuli flashed around the onset of horizontal and vertical saccades relative to continuously present references. For all saccade directions we observed an increase in disparity thresholds beginning about 30 ms before saccade onset and lasting until saccade offset. 

Stimuli flashed ±25 ms around the onset of horizontal saccades were systematically mislocalized to a closer depth plane. In contrast, stimuli flashed shortly before or after this period tended to be mislocalized to a farther depth plane. While a similar pattern was observed during upward saccades we found a clearly different pattern of mislocalization during downward saccades. 

Perisaccadic suppression and depth perception

In the following we discuss three different mechanisms which might contribute to the observed threshold elevations. Image smear caused by the fast changes in gaze direction may account in part for the observed threshold elevations. However, two observations back the assumption that other mechanisms might be involved. First, image smear is perpendicular to the relevant dimension for the vertical saccades and parallel for the horizontal saccades. Thus, we might expect a bigger effect of image smear for horizontal saccades. However, thresholds were similar during horizontal and upward saccades. Second, the early onset of the effect prior to saccade onset clearly indicates that other factors such as perisaccadic suppression are likely to play a role. This assumption is backed by the similar time courses for increases in depth thresholds and perisaccadic suppression (Diamond, Ross, & Morrone, 2000; Michels & Lappe, 2004). 

Further, shifts of attention associated with saccade execution might also affect disparity thresholds. If such shifts of attention are involved, we might expect differences between fixation and saccade target as attention shifts from the fixation point toward the saccade target shortly before saccade onset (Castet, Jeanjean, Montagnini, Laugier, & Masson, 2006; Gersch, Kowler, & Dosher, 2004). For horizontal and upward saccades, estimates of JNDs were very variable, and consequently such an effect could not be detected. The large variability might be due to the large shifts of the PSE, which cause the estimates of JND to rely on extrapolation. For downward saccades, however, the shift in PSE was considerably smaller, and thus estimates of disparity thresholds were much more reliable. For the downward saccades we indeed found smaller perisaccadic disparity thresholds at the saccade target, which might be caused by shifts of attention. We conclude that the perisaccadic increases in depth thresholds are likely to be caused by a combination of perisaccadic suppression, high attentional load, and image smear. 

While increases in disparity thresholds can not explain the systematic error in depth judgments, they might do so in combination with a perceptual bias to perceive stimuli closer than their retinal disparity would suggest. Regardless of how such a prior probability might be implemented neuronally, we would expect it to gain influence over depth estimates in conditions when estimates of retinal disparity are noisy. As this is likely to be the case perisaccadically, we would expect a tendency to perceive perisaccadic flashes in a closer depth plane. 

We conclude that a depth bias in combination with additional mechanisms such as perisaccadic suppression or high attentional load during saccade preparation and execution can explain part of our results. This is especially true for the early onset of the mislocalization toward a closer depth plane. However, this explanation fails to account the clear differences between upward and downward saccades, as well as the flanking phases of mislocalization toward a farther depth-plane. 

Damped disparity and persistent flash hypothesis

In the following we discuss two mechanisms which are conceivable to understand the flanking phases with mislocalization toward a closer depth plane as well as the differences between upward and downward saccades. 

Persistent flash hypothesis

Compared to brisk response onsets, neuronal responses due to stimulus offset decay only slowly over time. Consequently, a briefly flashed stimulus will elicit neuronal activity for a period which is clearly longer than the actual stimulus presentation (Pola, 2004). Pola has argued convincingly that during this decay period, encoded features like retinal position or disparity are expected to stay constant, regardless of changes in gaze direction or ocular vergence. Thus, while a flashed stimulus may coincide with a continuously present stimulus for only a single monitor frame, their corresponding neuronal activities will overlap for a considerably longer time in the visual system. Due to the ensuing temporal uncertainty, relative disparity judgments may have a unique solution in the outside world but not in visual system (Figure 7A). Such a situation is expected to arise if the disparity of the continuously present stimulus rapidly changes, as is the case during saccades. 

Figure 7

 

Two hypotheses for systematic errors in relative depth judgments between flashed and continuously present stimuli during fast vergence changes. Typical horizontal version saccades are accompanied by transient changes in vergence. Retinal disparity of a continuously present reference (green line) is plotted as a function of time from saccade onset. (A) Persistent flash hypothesis. Following an idea proposed by Pola (2004), we assume that neuronal activity elicited by flashed stimuli slowly decays and lasts for a considerably longer period than the stimulus was actually present on the screen (red-orange lines). During the entire period of slow decay, the neuronally signaled disparity is expected to stay constant. Thus, relative disparity comparisons (difference between blue and red/orange/yellow line) give rise to different results at different time points during the decay period. It was suggested that the final depth judgment might be an average over the entire decay period weighted with the corresponding strength of the decaying neuronal activity. The persistent flash hypothesis would give rise to two distinct periods of mislocalization, one toward a farther depth plane for stimuli flashed prior to divergent eye movements and one toward a closer depth plane for stimuli flashed prior to convergent eye movements (black line). (B) Damped disparity hypothesis. Based on the results of Nienborg et al. (2005), we hypothesize that neurons in primary visual cortex cannot follow fast changes in disparity. Hence they do not signal the correct retinal disparity (green line), but a low-pass filtered version thereof (blue line). For reasons of simplicity neuronal delays were disregarded. If we further assume that neurons correctly signal disparity of flashed stimuli (red dots), we would expect mislocalization in depth when the blue and green lines do not match. The damped disparity hypothesis predicts one epoch of mislocalization toward a closer depth plane for stimuli flashed during the first part of the saccade, and one epoch with mislocalization toward a farther depth plane in the second half and extending for a short while after saccade completion (black line). (C) Taken together the two hypotheses predict three epochs of mislocalization, first toward a farther depth plane, then toward a closer depth plane and finally toward a farther depth plane again (black line).

Damped disparity hypothesis

The study by Nienborg, Bridge, Perker, and Cummings (2005) suggests that neuronal responses in V1 do not track rapid changes in retinal disparity. These authors found that disparity-selective neurons in primary visual cortex (V1) do not follow fast oscillations in retinal disparity (10 Hz high-frequency cutoff measured at half peak height). Thus, while neurons in area V1 are expected to signal the correct disparity of a flashed stimulus, they might fail to do so for a continuously present stimulus when its retinal disparity is subject to rapid changes as for example during brisk changes in ocular vergence. Consequently, we might expect systematic errors of relative depth judgments in such situations (Figure 7B). 

For horizontal saccades, the persistent flash hypothesis predicts a period with mislocalization toward a farther depth plane for stimuli flashed just prior to saccade onset and toward a closer depth plane for stimuli flashed later during the saccade (black line in Figure 7A). Under the same conditions, the damped disparity hypothesis predicts mislocalization toward a farther depth plane for stimuli flashed just after saccade offset and toward a closer depth plane for stimuli flashed later during the saccade (black line in Figure 7B). For horizontal saccades the two hypotheses predict a central period with mislocalization toward a closer depth plane as well as two flanking periods with mislocalization toward a farther depth plane (black line in Figure 7C). Further, the sign of the mislocalization predicted by these two hypotheses depends on the sign of the disparity/vergence changes. Thus, they would predict an inverted pattern of mislocalization for downward saccades which show an inverted pattern of vergence changes compared to horizontal saccades. 

There are three points, however, which argue against the idea that the damped disparity and persistent flash hypothesis alone can account for the observed pattern of mislocalization. First, both hypotheses would predict mislocalization toward a closer depth plane only for stimuli flashed after saccade onset. However, we observe mislocalization toward a closer depth plane for stimuli flashed well before saccade onset. Second, while the pattern of mislocalization in depth during downward saccades is clearly different from the one observed for horizontal and upward saccades, it is not the predicted inverted version. Third, contrary to our expectations (Collewijn et al., 1988b), we found vergence changes during upward saccades to be very similar to the ones observed during downward saccades, but with smaller amplitude. Thus, both hypotheses would predict a similar pattern of mislocalization in depth for upward and downward saccades. However, the pattern of mislocalization in depth during upward saccades is more similar to the one observed for horizontal saccades. 

Relation to mislocalization in the fronto-parallel plane

In the following we will briefly relate our findings of mislocalization in depth to three mechanisms which can explain mislocalization in the fronto-parallel plane: the damped eye-position model (Honda, 1989; Matin, 1976), Pola's model (Pola, 2004) and the compressed time model which to our knowledge has not been described in the literature. 

Damped eye-position model

This model assumes that the mislocalization of flashed stimuli is exclusively due to an erroneous estimate of current eye-position. Under this assumption, it is possible to infer the brain's estimate of eye-position as the sum of actual eye-position and the localization error (see, for example, Honda, 1989; Matin, 1976). The observed pattern of mislocalization was taken to suggest that the brain's estimate of eye-position is a damped version of the actual eye-position. 

Pola's model

This model explains mislocalization as a combination of two mechanisms (Pola, 2004). First, it exploits the mechanisms of neuronal persistence (cf. above, the persistent flash hypothesis). Second, it assumes an erroneous estimate of eye-position. However, due to the interaction with the first mechanism, the erroneous eye-position estimate has a different form than the one predicted from the damped eye-position model. 

Compressed time model

This model is based on similar simplified assumption as the damped eye-position model. However, it assumes that the time assigned to the flash is erroneous while estimated eye-position is accurate. The model assumes that the correct retinal position estimate is assigned the wrong time and consequently paired with the correct eye-position estimate for the wrong time. The pattern of distortion in time which would replicate the empirical results corresponds to a compression of time around the midpoint of the saccade. In the light of recent experiments reporting perisaccadic compression of time this might not seem entirely unrealistic (Morrone, Ross, & Burr, 2005b). 

If we substitute ocular vergence for eye-position all three models can be extrapolated to predict mislocalization in depth. For horizontal saccades, the damped eye-position model as well as Pola's model predict the same tri-phasic pattern of mislocalization as the combination of the persistent flash and damped vergence hypothesis (see above). Thus, they fail to account for the observed data in the same points as outlined in the previous paragraph. 

The compressed time model would predict a different pattern of mislocalization in depth. For horizontal saccades it would predict two distinct epochs of mislocalization toward a farther depth plane with a short period of veridical depth perception about halfway into the saccade. However, the observed pattern of mislocalization in depth does not match this prediction. 

In summary, none of the three explanations of mislocalization in the fronto-parallel plane generalizes to predict the precise pattern of mislocalization in depth. 

Conclusions

Our study provides new insight into depth perception during saccadic eye movements. We show that, despite retinal slip, perisaccadic suppression, and large retinal eccentricities, depth perception is not abolished during saccades. The accuracy of depth judgments seems to depend on saccade direction. The best results were observed during downward saccades. For horizontal and upward saccades, we found a pronounced pattern of mislocalization in depth: stimuli flashed ±25 ms around saccade onset were systematically mislocalized to a closer depth plane. Further, stimuli flashed just before and after this period were mislocalized toward a farther depth plane. None of the mechanisms considered so far gives a completely satisfactory account of the data. 

Acknowledgments

This research was supported by EU-MEMORY, DFG FOR-560, and DFG GRK-885-NeuroAct. 

Commercial relationships: none. 

Corresponding author: Tobias Teichert. 

Email: [e-mail]. 

Address: 1051 Riverside Drive, New York, NY 10032, USA.