Depth cue combination in spontaneous eye movements

Abstract

Where we look when we scan visual scenes is an old question that continues to inspire both fundamental and applied research. Recently, it has been reported that depth is an important variable in driving eye movements: the directions of spontaneous saccades tend to follow depth gradients, or, equivalently, surface tilts (L. Jansen, S. Onat, & P. König, 2009; M. Wexler & N. Ouarti, 2008). This has been found to hold for both simple and complex scenes and for a variety of depth cues. However, it is not known whether saccades are aligned with individual depth cues, or with a combination of depth cues. If saccades do follow a combination of depth cues, then it is interesting to ask whether this combination follows the same rules as the well-studied case of depth cue combination in conscious perception. We showed subjects surfaces inclined in depth, in which perspective and binocular disparity cues specified different plane orientations, with different degrees of both small and large conflict between the two sets of cues. We recorded subjects' spontaneous saccades while they scanned the scene, as well as their reports of perceived plane orientation. We found that distributions of spontaneous saccade directions followed the same pattern of depth cue combination as perceived surface orientation: a weighted linear combination of cues for small conflicts, and cue dominance for large conflicts. The weights assigned to the cues varied considerably from one subject to the next but were strongly correlated for saccades and perception; moreover, for both perception and saccades, cue weights could be modified by manipulating cue reliability in a way compatible with Bayesian theories of optimal cue combination. We also measured vergence, which allowed us to calculate the orientation of the plane fitted to points scanned in depth. Contrary to perception and saccades, vergence was dominated by a single cue, binocular disparity.

Introduction

We spontaneously scan our three-dimensional environment by rapidly shifting our gaze both across the visual field and across depth planes. These gaze shifts can be divided into two components. First, we shift the overall direction in which the two eyes are looking; this conjugate component of gaze shifts is called a saccade. Second, we modify the relative alignment of the two gaze directions, so that they meet in different depth planes; this disjunctive component is referred to as vergence. Most studies of the effect of depth cues on eye movements have, understandably, focused on vergence, either vergence accompanying saccades (Both, van Ee, & Erkelens, 2003; Enright, 1987a, 1987b; Sheliga & Miles, 2003; Wismeijer, van Ee, & Erkelens, 2008), or fixational vergence (Hoffmann & Sebald, 2007; Ringach, Hawken, & Shapley, 1996; Wagner, Ehrenstein, & Papathomas, 2009). A persistent question has been whether it is depth cues or depth perception that guides vergence. Using illusions or perceptual depth reversals, several of these studies have claimed that vergence is correlated with perception per se, at least to some degree (Both et al., 2003; Hoffmann & Sebald, 2007; Ringach et al., 1996; Sheliga & Miles, 2003; Wagner et al., 2009). However, a recent study using stimuli with conflicting depth cues, whose perception observers could modify at will in accordance to one or the other depth cue, has shown that disparity dominates vergence responses and found no correlation between vergence and reported perceptual state (Wismeijer et al., 2008). 

Recently, another link has been found between 3D perception and eye movements: spontaneous saccade directions have been shown to be highly aligned with the direction of the depth gradient, or tilt, of planes inclined in depth (Wexler & Ouarti, 2008). ¹ This effect was shown to hold for different depth cues (disparity, perspective, texture gradients, presented without cue conflict) and to generalize to objects composed of multiple planes. However, as for vergence, it is not known whether saccades are aligned with merely the depth cues present in the stimulus, or with the depth gradient as it is experienced in visual perception. A way to answer this question is to study both the perception and spontaneous saccades elicited by conflicting depth cues. In the present study, we ask whether spontaneous saccades, as well as vergence, show any evidence of cue combination and if so, what is the relation between cue combination in eye movements and in perception? 

In the domain of perception, two decades of work have revealed the detailed ways in which depth cues interact. The visual system uses different strategies depending on the congruency and reliability of available depth cues. If the conflict between cues is small, depth perception is based on a weighted combination of the available cues (Bruno & Cutting, 1988; Dosher, Sperling, & Wurst, 1986; Johnston, Cumming, & Parker, 1993; Rogers & Collett, 1989; van Ee, Adams, & Mamassian, 2003; Young, Landy, & Maloney, 1993). The weights given to each cue are based on their respective reliabilities and in a sense this weighing is optimal: it reduces the total variance of the estimated variable (Greenwald, Knill, & Saunders, 2005; Hillis, Watt, Landy, & Banks, 2004; Landy, Maloney, Johnston, & Young, 1995). On the other hand, if the conflict between cues is large, the visual system usually exhibits either cue switching, in which perceptual switches in time occur between multiple depth percepts based on single depth cues (van Ee, van Dam, & Erkelens, 2002), or cue dominance, where depth perception is based on a single cue to depth (usually, the most reliable one, Bülthoff & Mallot, 1988; van Ee et al., 2003). 

For eye movements, on the other hand—both for saccades and vergence—the question of cue combination remains open, as we have seen: both whether and how depth cues are combined, and if so, whether the combination is performed by the same mechanism as in perception. The question is an interesting one because answers to it can give insight into the unicity or multiplicity of visual mechanisms subserving perception and its most intimately coupled action, namely eye movements. It would also help us pinpoint the site of cue combination in visual processing. Moreover, if depth cues are, indeed, combined in eye movement programming—either in the same way as in perception, or differently—this would provide either a new way of measuring perceptual cue combination, or a different measure to test theories of cue combination. 

It is possible to imagine several sorts of answers to the question of how depth cues are combined in eye movements. First, it may turn out that depth cues are not combined at all in eye movements in cases where they are combined in perception, and instead we find cue dominance or switching. In this case, we would conclude that cue combination is a mechanism that occurs late and only in the perceptual, but not in the sensorimotor, stream. Second, depth cues may be combined in eye movements, but in a different way than in perception. This would imply that depth cue combination is a late process that occurs in multiple, distinct versions, in both perceptual and sensorimotor streams. Third, it may turn out that depth cues are combined in the same way in eye movements as in perception. A signature of this outcome would be that relative cue weights, which vary in a reliable way from observer to observer, would be correlated for eye movements and perception across observers. However, even if this were so, it would not be proof of a single cue combination mechanism subserving both perception and eye movements, because there could be several such mechanisms operating with similar parameters. The signature of a single mechanism would be the presence of detailed, trial-by-trial, correlations between perception and eye movements. Finally, the answers to these questions may vary between different eye movements, namely saccades and vergence. 

Here, we study the correlation between perception and spontaneous eye movements while observers viewed inclined plane stimuli in which perspective and binocular disparity cues conflicted to varying degrees. We compare three different types of responses to the 3D stimuli: explicit responses of perceived surface tilt, the direction of spontaneous saccades made while viewing the surface, and the tilt of the “scanning” plane defined by the vergence angles of the fixations. 

Methods

Stimuli

Stimuli consisted of slanted planes with monocular (perspective) and binocular (disparity) cues independently specifying surface orientation (see Figures 1a and 1b). The inclined planes were seen through a simulated circular aperture with a radius of 10° visual angle (with the origin positioned at screen center). Surface texture was defined by a grid of square cells (size of 1° × 1°, before rotation or projection of the surface) and was rotated by an angle (which we call the texture angle) in the stimulus plane, so as to prevent any of the grid lines from being aligned with surface tilt. To parameterize surface orientation, we will use the polar slant and tilt angles (Stevens, 1983b). The fronto-parallel surfaces were inclined by first rotating them about the vertical axis (by the perspective-defined slant) and then about the straight-ahead axis (by the perspective-defined tilt). To create cue conflict, the resulting images were then projected onto a second surface, whose orientation was specified by disparity-defined slant and tilt (polar projection from the point halfway between the two eyes). Finally, the image for each eye was polar-projected onto the monitor, assuming that the eyes were located at the same height in the frontal plane 57.3 cm from the monitor, with interocular distance of 6.4 cm. 

Figure 1

 

Examples of stimuli. The stereo images are for uncrossed viewing. (a) Small cue conflict stimulus with a 15° conflict between the perspective- (135°) and disparity-defined (120°) tilts. (b) Large cue conflict stimulus with a 90° conflict between the perspective- (135°) and disparity-defined (45°) tilts. Perspective- and disparity-defined slants are 45°. The stimuli have a texture angle of 37.5°.

The stimuli were drawn in red (the color that resulted in the least cross-talk between the two eyes' images; CIE chromaticity coordinates of 0.607, 0.376) with a luminance of 0.85 cd/m², on a black background (0.16 cm/m²). Luminance was measured in stereo mode, for each eye, across both the active and passive filters. Cross-talk between the two eyes' images was not measurable above the black background. 

The perspective-defined tilt took one of six values equally spaced around the unit circle: 30°, 90°, …, 330°. Disparity-defined tilt differed from perspective-defined tilt by a tilt-conflict angle, which took the values of 0°, ±15°, ±30°, ±45° (all of which we called small-conflict stimuli: see Figure 1a for an example), or ±90° and 180° (which we called large-conflict stimuli: see Figure 1b). 

In addition, we varied the reliability of both the disparity and the perspective signals individually. The discriminability (and reliability) of surface tilt varies with the surface slant, with larger slants giving rise to more precise perceptual tilt estimates (see Koenderink, van Doorn, and Kappers, 1992 and Stevens, 1983a for perspective and van Boxtel, Wexler, and Droulez, 2003 for structure-from-motion cues). This is not surprising: if a plane's normal is misestimated by a small δ in a random direction, the mean tilt error will be on the order of δ/sinσ, where σ is the slant. We varied the slant of each cue between 15°, 45°, and 60°, while the slant of the other cue remained constant at 45°. 

Procedure

Each trial consisted of the presentation of a fixation cross (subtending 1.3°, presented with a random duration between 1.3 and 1.7 s), followed by the main stimulus described above, which was presented for 3 s, shown without any fixation mark. Observers were instructed to fixate the cross while it was visible, but to look where they wished during the other phases of the trial. Following the disappearance of the main stimulus, a visual probe of 3D plane orientation appeared; observers adjusted the orientation of this probe by inclining a joystick, in order to report the perceived orientation of the stimulus plane. The visual probe was a binocular planar object consisting of concentric circles and radial spokes, whose radial size was 5° when fronto-parallel. 

Two observers performed 990 trials in a factorial design (3 texture angles (7.5°, 22.5° and 37.5°) × 6 perspective tilts × 11 tilt conflict angles × 5 slant combinations), while the rest performed 660 trials (2 texture angles (15° and 30°) with the rest of the factors the same). The total duration of the experiment was about 3–4 h for each observer. At the beginning of session and then after every 25 trials, observers were guided through the standard EyeLink 9-point calibration and validation procedures. In order to continue, fixation errors of both eyes had to be below 1.5°. 

Apparatus

Stimulus images were generated by specially written software in C++ using OpenGL for graphical display. The experiment ran on a PC computer (Dell Precision Workstation 390 Series, Intel Core 2 CPU, 2.13 GHz, 2-GB Ram, Microsoft Windows XP) with an Nvidia Quadro FX 3500 video card. Images were displayed on a 19-inch Dell Ultrascan P991 CRT monitor with a flat Sony Trinitron tube (at 1024 × 768 resolution, 120-Hz refresh rate) using the quickly decaying red phosphor only, for the main stimuli. Stereo separation was obtained using a polarizing screen (ZScreen, StereoGraphics), which covered the entire surface of the monitor and was synchronized with its vertical refresh rate. Observers wore passive polarizing filters mounted on an eyeglass frame. The resulting refresh rate for each eye was 60 Hz. Observers were seated and their head movements restrained using a chin rest, with their eyes approximately at 57.3 cm from the monitor. 

Binocular eye movements were recorded with an infrared video eye tracker (EyeLink II, SR Research), sampling at 500 Hz in pupil-only mode with head tracking. The EyeLink cameras had no trouble detecting the pupil through the polarizing filters worn by the observers. 

Data analysis

Eye movements

Eye movement data obtained during the presentation of the stimulus were analyzed offline. Left and right eye data of individual trials were corrected by offsetting by the average position of each eye during the first 50 ms of the initial fixation during stimulus presentation. Saccades were detected when version (mean of left and right eye gaze directions) speed exceeded 30°/s. In between saccades, we defined fixations as contiguous periods of at least 100 ms in which version speed was below 10°/s. Velocity profiles were obtained using the first derivative of a Savitzky–Golay filter (with a 5th-order polynomial fit and a kernel size of 21 samples, i.e., a 40-ms window). Only saccades with an amplitude greater than 1° were used in further analyses. 

In addition, we calculated the orientation of the vergence or “scanning” plane, defined by all the fixations in each trial in 3D space. We first calculated the position in depth where left and right eye gaze directions came closest to crossing (the point halfway between the points on each ray that had minimal Euclidean distance from each other). For these calculations, we assumed an interocular distance of 6.4 cm, that the two eyes were both in a plane parallel to the monitor and 57.3 cm away from it, had the same vertical position, and that the point halfway between the eyes was directly across from the center of the monitor. The orientation (or plane normal) of the vergence plane was obtained via a least-squares fit to these points in 3D space, provided there were at least three fixations in a given trial. 

All response types

We analyzed three response types: explicit reports of perceived surface tilt, saccade directions, and the tilt of the “scanning” plane defined by vergence. For all three response types, we collapsed the data from the (six) perspective-defined tilts by rotating such that the perspective-defined tilt always pointed upward. In addition, we flipped trials with negative tilt differences between the disparity- and perspective-defined tilts (clockwise direction of tilt conflict), so that the disparity-defined tilt was always counterclockwise from the perspective-defined tilt. These transformations are illustrated in Figure 2. We also collapsed trials across the different values of texture angle. Absolute tilt difference and slant value of each cue were the resulting variables in the analysis. 

Figure 2

 

Alignment of different tilt directions. We collapsed data from different tilt directions by rotating the data such that the perspective-defined tilt always pointed upward. Furthermore, we flipped certain trials, so that the disparity-defined tilt always deviated in the counterclockwise direction from the perspective-defined tilt.

The three response measures all reflect (tilt) directions and are thus directional data. We therefore used circular statistics (mean

$\bar{q}$

, the Rayleigh R measure of non-uniformity, and variance V), defined for a set angular variables q ₁, q ₂, … by (Mardia, 1972) 

$\bar{C},\bar{S}=\frac{1}{n}\sum _{i=1}^n\cos q_i,\frac{1}{n}\sum _{i=1}^n\sin q_i,$

(1)

 

$\bar{q}=\arctan (\bar{C},\bar{S}),$

(2)

 

$R=\sqrt{{\bar{C}}^2+{\bar{S}}^2},$

(3)

 

$V=\sum _{i=1}^{n}1-\cos (q_i-\bar{q}{)}_{\mathrm{mod}p},\mathrm{s}\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}0\le V\le 1.$

(4)

 

We used the Rayleigh test to test for uniformity of circular distributions. For these tests and all other significance test, we applied a Sidak correction (similar to the better-known Bonferroni correction). To test whether differences between conditions were significant, we used bootstrap procedures (1000 resamples; see Efron & Tibshirani, 1993). When responses showed peaks separated by 180°, as they did for saccade directions, we used axial versions of the above circular statistics: we collapsed the two opposing peaks into one by multiplying all angles by 2, calculating the statistic, and then dividing by 2. 

Observers

All six observers (2 males, aged between 25 and 60 years) were naive as to the purpose of the study and had no prior exposure to experiments on vision. The reason we used inexperienced observers is that we found in pilot studies that experienced psychophysical observers were so bent on fixating—even when no fixation mark was visible—that they often either continued to fixate for the duration of the stimulus, or when told not to, performed idiosyncratic but stereotypical saccades, independent of any stimulus features. ²  

All observers had normal or corrected-to-normal vision with a stereo acuity of at least 60 arcsec (Stereo Optical RanDot test) and were paid for their participation. 

Results

The angular distributions of the three response types, explicit perceptual reports of tilt, saccade directions, and the tilt of the scanning plane defined by the vergence, are displayed in Figure 3 for the six degrees of cue conflict, collapsed across observers (distributions for two observers, one who showed more disparity-based responses and another who showed more perspective-based responses, can be found in Figures S1 and S2 in Supplementary data). The perceptual response distributions show one pronounced peak lying between the perspective- and disparity-defined tilts, for small conflicts. For large conflicts, on the other hand, two peaks can be observed; this bimodality is overwhelmingly due to between-observer variations and is only occasionally seen in individual observer data. ³ Saccade distributions show two peaks separated by 180°, one of which is roughly in the direction of the peak observed for perception. ⁴ Vergence distributions show wide peaks roughly in the direction of disparity-defined tilt. 

Figure 3

 

Perceptual and eye movement responses for each cue conflict condition. The response distributions are depicted as circular histograms: the distribution of responses in each direction (smoothed using a 30° window) is shown as the radial distance between the central yellow circle and the black curve. Data have been transformed (see Figure 2) so that the perspective-defined tilt (dashed gray line) always points upward, whereas the disparity-defined tilt (solid gray line) varies with the degree of cue conflict and is oriented counterclockwise relative to the perspective tilt. The angular means are shown in red (for saccades, the bidirectional lines reflect the axial nature of the distributions). For large conflicts, the angular means are not displayed for either the perceptual or saccade responses, because the observed bimodality of the response distributions were mostly due to between-observer variations, making the overall means meaningless; angular means for individual observers are shown in Tables S1, S2, and S3 in Supplementary data.

All distributions were non-uniform and showed pronounced peaks. Since the distributions are aligned with the tilt of the stimulus plane, non-uniformity implies an effect of 3D surface orientation on perceptual and oculomotor responses. To test whether these distributions were indeed non-uniform, we used the Rayleigh test, which is based on the vector sum of the unit vectors corresponding to each angle (the Rayleigh R, see Equation 3 in Mardia, 1972). We found that the distributions were significantly non-uniform for the perceptual responses for all observers and all values of tilt conflict (36 tests). Saccade direction distributions were non-uniform except for one observer for the 90° tilt conflict. Vergence tilt distributions were significantly non-uniform except for three different observer-cue conflict combinations. The results of these significance tests are shown for individual subjects in Tables S1, S2, and S3 in Supplementary data. 

As we have seen, perception and vergence responses had one peak for individual observers, whereas saccade directions showed two peaks. The bimodality observed for saccades is almost absent for first saccades, which are concentrated in the direction lying between the perspective- and the disparity-defined tilts (see Figure S3 in Supplementary data, which shows distributions for first and subsequent saccades), although some subjects do show bimodality in the direction of the first saccade (see Figures S1 and S2 in Supplementary data, which show data of two individual observers). Since we found no other significant differences between first and later saccades, we will report the effects for all saccades combined. In the 180° conflict condition, because the direction of the second peak in the saccadic distribution coincides with the direction of disparity-defined tilt, we cannot distinguish saccades that follow disparity from those that follow perspective tilts. Therefore, we excluded this condition from most subsequent analyses. 

We calculated the mean angular direction (see Equation 2 in Mardia, 1972) for the three different response types. Figure 3 shows the mean directions collapsed across observers (red lines). Both the mean perceived tilt direction and the saccade directions were located in between the perspective-defined tilt (vertical up) and the disparity-defined tilt for small cue conflicts (0°–45°). Large cue conflicts, on the other hand, gave rise to cue dominance: each response was based on one or the other tilt cue. In contrast to perception and saccades, vergence responses were largely aligned with the disparity-defined tilt for all cue conflicts. Angular means for individual observers are given in Tables S1, S2, and S3 in Supplementary data. 

Mean angular directions for the three response types of individual observers are shown in Figure 4 as a function of cue conflict. If observers based their responses (perceptual, saccades, vergence) on the perspective-defined tilt only, then angular means should be zero for all values of cue conflict. On the other hand, if observers based their responses on the disparity tilt only, then angular means would lie on the diagonal (red dashed lines). In perceptual and saccade responses, observers clearly fell into two clusters: one group that based their responses mainly on perspective tilt and one group whose responses were based mainly on disparity tilt. Vergence responses, however, can once again be clearly seen to be clustered around the diagonal in all observers, implying that they are based on disparity alone. 

Figure 4

 

Change in response tilt with varying degrees of cue conflict, for individual observers. The mean angular response directions per cue conflict condition are shown for perceptual, saccade, and vergence responses.

We wondered whether for small conflicts (0°–45°) perceptual and saccadic responses were completely dominated by one or the other cue, or whether the two cues were combined. We performed two tests that suggested that the two cues were, in fact, combined. First, we fitted the mean angular responses as linear functions of cue conflict angle. The resulting slopes, which we will call fusion slopes, reflect the relative cue weights. Thus, the higher the slope, the higher the weight of disparity with zero indicating a pure perspective-based response and one a pure disparity-based response. We tested whether the fusion slopes were significantly different from both zero and one. A bootstrap test revealed that in four out of the six observers, fusion slopes were significantly greater than zero and less than one. In one observer, the perceptual fusion slope was not significantly different from one (while the saccade fusion slope was), whereas in another observer, the saccade fusion slope was not significantly different from one (while the perceptual fusion slope was). On the other hand, none of the vergence slopes was significantly different from unity. 

Even though most fusion slopes assumed intermediate values, this does not provide conclusive proof of cue combination, optimal or otherwise. For example, if responses were drawn from separate distributions corresponding to each cue, we could still obtain unimodal distributions with intermediate slope values for small conflicts. However, in this case variance of the responses would show a quadratic increase with increasing cue conflict. In the case of optimal cue combination, on the other hand, variance should remain constant as cue conflict increases (Muller, Brenner, & Smeets, 2009). We therefore calculated the angular variance for different cue conflicts between 0° and 45° for all response types, in individual subjects. We used the standard definition of angular variance (Mardia, 1972), but in order to take into account bimodal distributions we performed an axial transformation to force all angles to lie within 90° of the angular mean. ⁵ The results are shown in Figure 5. To test for quadratic growth in variance, we fitted it as a quadratic polynomial depending on cue conflict, for all three response types in every subject. Using a bootstrap to calculate 95% confidence intervals of the polynomial coefficients, we found no significantly positive quadratic coefficients for any measure in any subject (one subject had a significantly negative quadratic coefficient for saccades). An additional between-subject bootstrap test on the means of the quadratic coefficients revealed that the perception and saccadic variances had no significant quadratic growth (and that the vergence variance showed a significant quadratic decrease). We thus have converging evidence for optimal (i.e., constant variance) cue combination for small cue conflicts in perceptual and saccadic responses. 

Figure 5

 

Variance of perceptual, saccadic, and vergence responses as a function of the degree of cue conflict, for small conflicts. Angular variance was calculated for individual observers and degrees of cue conflict. Here the means and standard errors across observers are depicted. Variance did not vary significantly with the degree of cue conflict, providing evidence for optimal cue combination. Note that zero means no angular variance and one is equal to the maximum angular variance (see Data analysis section for details).

Although perceptual and saccade responses as a function of cue conflict showed a similar pattern (see Figures 3 and 4), we wished to test whether they were correlated across observers: for example, whether observers who put a higher weight on disparity in their perception also tended to do so in their saccade directions. In Figure 6a, we plot the saccade fusion slopes versus the perceptual fusion slope for each observer for small cue conflicts (see Table 1 for values). The correlation between the two fusion slopes was positive (r = 0.89), and a bootstrap test revealed that this correlation was significant (95% confidence interval [0.73, 1.00]). We used a different procedure for the data from the 90° conflict condition as these data showed either cue dominance or cue switching. We calculated the mean angular direction and normalized it by 90°, so that zero corresponded to perspective- and one to disparity-based responses. In Figure 6b, we plot these measures for saccades relative to perception. The correlation between the cue dominating perception and the cue dominating saccade direction was extremely high (r = 0.99), and a bootstrap procedure revealed that it was significant (95% confidence interval [0.90, 1.00]). The underlying distribution of the data in the 90° condition may violate the normality assumption of the Pearson's correlation coefficient. We therefore confirmed these findings by two measures of rank correlation, which do not require normally distributed data sets: Kendall's τ = 0.76, [0.47, 1.00]_95% and Spearman's ρ = 0.89, [0.71, 1.00]_95%. On the other hand, there was no significant correlation between vergence and the other two response types (perception, saccades). Thus, saccade directions and perceptual responses were based on either the same weighted cue combination (small conflicts) or the same dominant cue (90° conflict); vergence, on the other hand, followed disparity and was independent of the other response types. 

Figure 6

 

Correlation between perceptual and saccadic responses, in individual observers, for small (0–45°) and large (90°) conflicts. For small conflict stimuli, the weights given to each cue in the combination is reflected by the change in the response for varying degrees of conflict, as depicted in Figure 4. Here, we plot the fusion slope values, the slopes of the linear fit on the data (0°–45°) shown in Figure 4 of the saccadic response relative to the perceptual response, for individual observers. The dashed lines show the points where the two weights are equal. For large conflict stimuli (90°), we calculated the angular mean normalized by 90°.

Table 1

 

Fusion slopes. Values in bold were significant.

	Perception	Saccades	Vergence
al	0.24	0.45	1.03
al1	0.84	0.93	1.26
av	0.70	0.70	1.07
lp	0.71	0.64	1.03
og	0.89	0.83	1.20
tj	0.33	0.21	1.19

The fact that there was a correlation between perception and saccade directions across observers does not prove that saccades and perception obtained their information from a common source. If they did, there would also be trial-by-trial correlations of saccade directions and perception, independently of their correlation to disparity- and perspective-defined tilts. On the other hand, if each response type obtains its information from a different channel, then trial-by-trial correlation would be absent. We calculated trial-by-trial correlation separately for each observer and each cue conflict, to exclude correlations to stimulus tilts. Furthermore, we protected our calculation from the spurious effect of angular discontinuity by removing all trials in which one of the two measures differed by more than 90° from midway between perspective and disparity tilt, for small conflicts. For large conflicts, we removed trials that differed more than 90° from the dominant cue (for individual observers). Because cue switching could occur for saccade directions for the large conflicts, we only used first saccades (for all cue conflicts). 

One observer showed reliable, positive trial-by-trial correlations between perceived tilt and saccade directions, for five out of six tilt conflicts; another observer showed such correlations in half the conditions, while a third did so in only one of the six conditions. On the other hand, another observer's only reliable correlations were negative, in two of the conditions. The remaining observers had mixed positive and negative values among their reliable correlations (Table 2). 

Table 2

 

Trial-by-trial correlation of perception and saccade directions. Values in bold were significant.

	0	15	30	45	90	180
al	−0.20	0.14	0.11	0.15	0.45	0.07
al1	−0.16	−0.04	−0.19	−0.15	−0.22	−0.36
av	0.30	0.40	0.46	0.43	0.40	0.59
lp	0.13	0.21	0.44	0.34	0.34	0.11
og	0.39	−0.33	0.22	0.11	0.02	0.18
tj	−0.27	0.08	0.06	0.25	0.21	0.23

Thus, there is little evidence for reliable, detailed correlation between trial-by-trial variations in saccade directions and perceived tilt. This would seem to imply separate tilt estimation streams for perception and saccades (but both showing similar mechanisms of cue combination, with similar cue weights in each observer). However, uncorrelated noise in the final response-related component of each of the two streams may very well mask any correlations. We will return to this point in the Discussion section. 

We independently varied the reliability of each cue for determining surface tilt, by varying the corresponding slant: larger slants provide more reliable tilt estimates. In the results presented up to this point, we have collapsed the data across all slants. By separating the data for different slants, we can investigate whether saccade directions put more weight on cues that are more reliable, as does perception. In Figure 7, we plot the fusion slopes of both saccades and perception as a function of slant (either of the perspective cue while disparity-defined remains fixed, or vice versa), collapsed across observers. With increasing perspective-defined slant, the relative weight given to perspective increases; similarly, for increasing disparity slant, the weight given to disparity increases. These changes in relative weights were significant for both saccade directions and perception: a bootstrap test revealed that the slope of the linear regressing of fusion slopes versus slant was significantly negative for variations in disparity slant and significantly positive for variations in perspective slant (p < 0.05). These results suggest that, for both perception and saccade direction, cue weights are based on cue reliability, a result compatible with Bayesian models of cue combination. 

Figure 7

 

Effect of cue reliability on weights assigned to each cue (small cue conflicts only). The change in fusion slopes as a function of slant values, of both perspective (red) and disparity (brown), is shown for saccadic and perceptual (dashed) responses. Cue reliability was individually modified via slant (15°, 45°, and 60°). A fusion slope value of zero corresponds to a pure perspective-based response, while one corresponds to a pure disparity-based response. Each point reflects the (fitted) fusion slope value across observers with the error bars denoting standard errors (note that the bars are slanted as to prevent them from overlapping). The increase or decrease in cue weight with cue reliability was significant for all four cases depicted.

Discussion

We have investigated the combination of two depth cues, disparity and perspective, for both small and large conflicts, using three different response measures: explicit reports of perceived surface tilt, the directions of spontaneous saccades when scanning the stimulus, and the orientation of the plane defined by the vergence of the spontaneous eye movements. Both saccadic and perceptual responses showed evidence of cue combination. For small conflicts, tilt responses (the mean saccade direction, or the reported tilt for perceptual responses) lay between the disparity and perspective tilts, with variance that did not increase with tilt conflict. Moreover, when we made either one of the cues more reliable for tilt perception by increasing its slant, the relative weight of that cue increased in both perceptual and saccadic measures, which is compatible with Bayesian theories of optimal cue combination. For large conflicts, both saccadic and perceptual responses were dominated by one or the other cue—both responses dominated by the same cue within each observer, and varying across observers. Vergence responses, on the other hand, were always dominated by the disparity cue, for both small and large conflicts, showing no evidence of cue combination. 

We analyzed whether perceptual and saccadic responses, which both showed evidence of cue combination, reflected the same combination of cues. For each subject, we calculated the relative weights of the two cues, for perceptual and saccadic responses. For both small and large conflicts, we found that the weights for the two response measures were highly correlated. However, neither measure was correlated with the vergence measure in this way. However, we found no evidence of more detailed trial-by-trial correlations between perceptual and saccadic responses. Taken together, these results indicate that both saccades and perception use similar cue combination mechanisms, with weights that are consistent within subjects: those who weigh disparity highly for perception, for example, also tend to weigh it highly in their saccades. However, the possible absence of detailed trial-by-trial correlation indicates that the two cue combination mechanisms, though functionally similar, are distinct. 

Our results leave open two possibilities. The first possibility is that there are detailed trial-by-trial correlations between perceived tilt and saccade directions, but we do not observe these correlations because they are washed out by noise. Imagine a model in which saccade direction and perceived tilt are given by T _s = T ₀ + R _s and T _p = T ₀ + R _p, respectively, where T ₀ is a common mechanism for extracting plane orientation, incorporating the combination of multiple depth cues that we have observed, and R _s,p is the noise specific to saccadic and perceptual responses, respectively. This model would explain the subject-by-subject correlations that we have found in cue weights between perceptual and saccadic responses and does not contradict the apparent absence of trial-by-trial correlations. This last statement, although surprising at first glance, is due to the fact the correlation between T _s and T _p depends on the variance of the underlying processes. For instance, if the standard deviation of R _s and R _p is three times that of T ₀, then the expected coefficient of correlation T _s and T _p is about 0.1, which cannot be excluded by our results. If this model holds, we can imagine that perception and saccades are either both derived from a common signal (T ₀) as parallel processes, or work in series: saccade directions depend on perception, or vice versa. 

The second possibility is that our failure to find significant trial-by-trial correlations between tilt responses and saccade directions is not due to noise but is really due to two independent processes going on in parallel. A remarkable aspect of this possibility is the presence of subject-by-subject correlations in cue weights. In other words, it implies that although there are two separate processing streams, leading to perception and action—a phenomenon that frequently occurs in the nervous system (Milner & Goodale, 1996)—these streams function in the same way, and in particular they weigh disparity and perspective cues in a similar fashion from subject to subject. One possible way to account for a duplication of similar mechanisms could be that the perceptual and motor streams take as inputs the same single cue depth estimates, with their respective noise levels. Each stream then combines the cues but does so using generic neural mechanisms that have been shown to give rise to optimal cue combination (Denève, Latham, & Pouget, 2001). Thus, according to this account, a subject who weighs disparity highly in both perception and saccades does so because her estimate of disparity-defined tilt is less noisy than her estimate of perspective-defined tilt. However, without further data (perhaps with many more trials, to increase the signal with respect to noise), we cannot determine whether detailed trial-by-trial correlations exist between tilt perception and saccade directions. 

Our results for saccades extend those recently published by Wexler and Ouarti (2008), who showed that spontaneous saccade directions follow the tilt axis of an inclined plane, for a variety of depth cues that were presented singly (other recent work that reports compatible results is Jansen, Onat, & König, 2009). The fact that spontaneous saccade directions are aligned with the surface depth gradient of a purely disparity-defined plane as shown by Wexler and Ouarti (2008) excludes the possibility that the luminance gradient, always correlated with the perspective tilt axis, guided saccade directions in that and the current study. Earlier, Vishwanath and Kowler (2004) reported that saccade landing positions were related to the 3D geometric cues of the saccade targets revealing that depth cues are important in driving saccades, whereas luminance has been shown to have at the most a moderate effect on saccade landing positions (Melcher & Kowler, 1999; Spering, Montagnini, & Gegenfurtner, 2008). Here we have shown that when two depth cues are present, the axis of spontaneous saccades combines the tilts of the two cues in the same way, and with the same weights, as perception does. These new results therefore open the possibility of studying cue combination in a new way, by measuring the directions of spontaneous saccades. The technique could prove valuable in cases where explicit perceptual responses are unavailable, such as in infants or non-human animals. 

Contrary to saccade directions, our second oculomotor measure, vergence, seems to depend exclusively on the disparity cue. It shows no evidence of cue combination and is uncorrelated with perception or saccade directions. This implies that vergence is not guided by perception and that vergence and saccades are programmed separately and combined in a later stage of the motor pathway. The absence of correlation between vergence and perception contradicts a number of earlier studies that reported such a correlation to varying degrees and under varying experimental conditions such as vergence accompanying a saccade under monocular (Enright, 1987a, 1987b) and binocular viewing conditions (Sheliga & Miles, 2003) or fixational vergence under monocular (Ringach et al., 1996) or binocular viewing conditions (Hoffmann & Sebald, 2007; Wagner et al., 2009). Another study that examined vergence accompanying saccades under binocular viewing conditions reported a moderate correlation between vergence and perception after a period of free viewing, but without a preceding free viewing period this correlation was absent (Both et al., 2003). Moreover, another recent study went a step further by using a stimulus paradigm in which the depth cues were uncorrelated with depth perception, thus in that sense being distinctive from the previous studies (Wismeijer et al., 2008). This study reported that vergence was uncorrelated with perceived depth as such but was determined by depth cues, predominantly disparity. 

How can we reconcile our results with the very diverse aforementioned studies? In most of these studies, perception was correlated with the various stimulus cues and only partially correlated with vergence (Both et al., 2003; Enright, 1987a, 1987b; Ringach et al., 1996; Sheliga & Miles, 2003). These results dovetail nicely with the assumption that for vergence the statistically optimal cue combination is different from that for perception or saccades (as previously suggested by Wismeijer et al., 2008); depending on the response measure, different weights were assigned to individual depth cues. For example, while under monocular viewing conditions, perspective may be the most reliable cue for vergence (Enright, 1987a, 1987b), under binocular viewing conditions, disparity is by far more reliable (current results and those of Wismeijer et al., 2008). Yet, in three studies, depth perception was dissociated from stimulus cues by using perceptual rivalry stimuli: the hollow mask illusion (Hoffmann & Sebald, 2007) and slant rivalry stimuli (Wagner et al., 2009; Wismeijer et al., 2008). Both Hoffmann and Sebald (2007) and Wagner et al. (2009) reported an influence of perception on vergence estimated during prolonged viewing periods (fixational vergence). Whereas Wismeijer et al. (2008) reported no effect of perception vergence accompanying saccades (a measure similar to the one used in this study). Apparently, the measure of vergence used is of importance: fixational vergence is different from the vergence component in disjunctive saccades and vergence in response to a stimulus moving in depth (Erkelens & Collewijn, 1985; Erkelens & Regan, 1986; Regan, Erkelens, & Collewijn, 1986; Wismeijer & Erkelens, 2009, all reported that vergence and perception of motion in depth were uncorrelated). That leaves one discrepancy still unexplained: Wismeijer et al. (2008) did report an effect of perspective on vergence, which we could not replicate in the current work. Wismeijer et al. (2008) reported a very small effect of perspective on vergence (about 14% of vergence was attributed to perspective); the measure of vergence used here, the tilt of the “scanning” plane based on multiple fixations in between saccades, may have washed out any effects of perspective on vergence. More broadly, we believe that more work is needed to determine why vergence follows disparity, or a weighted combination of depth cues, in some cases, and conscious perception in others. 

There is ample, experimental and computational, evidence that saccadic and vergence systems interact, and not necessarily linearly, when directing gaze across different depth planes (Busettini & Mays, 2005a, 2005b; Chaturvedi & Gisbergen, 1998; Collewijn, Erkelens, & Steinman, 1995, 1997; Enright, 1984, 1986; Erkelens, Steinman, & Collewijn, 1989; Erkelens, Van der Steen, Steinman, & Collewijn, 1989; Kumar et al., 2006; Ramat, Das, Somers, & Leigh, 1999; Zee, Fitzgibbon, & Optican, 1992). Whether this indicates that vergence accompanying saccades is distinctly different from fixational vergence, because the movements are generated via different visuo-motor pathways, remains to be seen. Although, the visuo-motor pathways for the generation of conjugate saccades and those for the generation of pure disjunctive eye movements have been studied extensively (for reviews see Gamlin, 1999; Mays & Gamlin, 1995; Moschovakis, Scudder, & Highstein, 1996; Scudder, Kaneko, & Fuchs, 2002), the encoding of disjunctive saccades has received little attention. The few studies on non-human primates investigating the origin of disjunctive saccadic movement commands have thus far focused on the superior colliculus (SC; Klier, Wang, & Crawford, 2001; Walton & Mays, 2003) and the brainstem (Horn & Cullen, 2009; Horn, Sylvestre, & Cullen, 2008; Sylvestre, Choi, & Cullen, 2003; Sylvestre & Cullen, 2002; Sylvestre, Galiana, & Cullen, 2002). The SC has been shown to predominantly encode signals related to the generation of conjugate saccades in 2D coordinates, and not to encode signals specifically related to the generation of disjunctive saccades or vergence (Klier et al., 2001; Walton & Mays, 2003). Thus signals leading to disjunctive saccades, those related to vergence and conjugate saccadic eye movements, must be combined beyond SC. Recently, it has been shown that the brainstem encodes signals related to disjunctive saccades (Horn & Cullen, 2009; Horn et al., 2008). Further research is needed to ascertain whether this is the first area in the visuo-motor pathway to encode both conjugate and disjunctive eye movement-related signals, making it a likely site for the combination of the two. 

Viewed from the other direction of the visuo-motor pathway, our results suggest that saccade target selection should occur after the stage of cue combination. Combined estimates of surface structure are already encoded in areas V3/V3a (Orban, Janssen, & Vogels, 2006; Tsutsui, 2002) and areas it projects to: the lateral interparietal area (LIP) and the frontal eye fields (FEFs). Both these areas have been shown to engage in saccadic target selection with targets encoded in three dimensions (Gnadt & Mays, 1995). Chaturvedi and Gisbergen (1998) reported that target selection is combined for both saccades and vergence, and because of the retinotopic encoding in SC, areas LIP and FEF are more likely candidates for saccadic target selection in 3D visual stimuli. How the vergence system obtains information about the new target location remains an open question. 

Finally, the psychophysical literature on the existence of separate pathways for perception and action (Aglioti, DeSouza, & Goodale, 1995; Bridgeman, Kirch, & Sperling, 1981) is rife with contradictions, even concerning particular movements, such as ocular saccades (de Grave, Franz, & Gegenfurtner, 2006; de Grave, Smeets, & Brenner, 2006; Knox & Bruno, 2007; McCarley, Kramer, & DiGirolamo, 2003). Here we shed new light on the issue by showing that in the case of depth cue conflict, of two oculomotor measures of depth, one—saccade direction—is strongly correlated perception and the detailed way in which it combines cues, while the other—vergence—is correlated only with one of the two cues, binocular disparity. Vergence is the slower of the two movements, and slower movements have been found to be more correlated with perception. Thus, the interplay between action and perception looks as complex as ever, if not more so. 

Conclusions

We have shown that saccades and perception combine disparity and perspective cues to depth in a similar way. For small conflicts, cues are averaged with correlated weights in each observer, with the weights depending on cue reliability in a way compatible with Bayesian theories of optimal cue combination. For large cue conflicts, one cue dominates the other, with the dominant cue varying from one observer to another but the same for perception and saccades in each observer. The mechanism subserving vergence is different from those for saccades and perception and shows dominance of disparity. 

Supplementary Materials

Supplementary Data - Supplementary Data 

Acknowledgments

We would like to thank R. van Beers for some helpful comments on the data analysis. This project was supported by a grant from the Dutch Association for Biophysics and Biomedical Technology awarded to D. A. Wismeijer. 

Commercial relationships: none. 

Corresponding authors: D. A. Wismeijer and M. Wexler. 

Email: [email protected]; [email protected]. 

Address: Physics of Man, Helmholtz Institute, Utrecht University, Padualaan 8, 3584 CH, Utrecht, The Netherlands. 

Footnotes

Footnotes

¹  Other links between 3D perception and saccades have also been found: for instance, landing positions of saccades are affected by the 3D geometric cues of the target (Vishwanath & Kowler, 2004).

Footnotes

²  We believe that the difference between inexperienced and experienced observers is due to the latter having learned not only to maintain strict fixation but to perform visual judgment tasks under fixation. The fact that experienced observers made stereotypical saccades, when instructed to fixate, probably means that this instruction was interpreted as an additional task. All inexperienced observers, both in the current study and in that of Wexler and Ouarti (2008), showed a robust correlation between their spontaneous eye movements and the 3D structure of the stimulus.

Footnotes

³  Between-subject variability is rather commonly observed in these types of tasks (Oruç, Maloney, & Landy, 2003; Vishwanath & Kowler, 2004) and even across modalities (Ho, Serwe, Trommershäuser, Maloney, & Landy, 2009).

Footnotes

⁴  The 90° conflict is an exception: it has four modes corresponding to the perspective- and disparity-defined tilts and their opposite directions.

Footnotes

⁵  In other words, if an angle differed by more than 90° from the angular mean, we added 180° to it.