Coarse-to-fine interaction on perceived depth in compound grating

Abstract

To encode binocular disparity, the visual system uses a pair of left eye and right eye bandpass filters with either a position or a phase offset between them. Such pairs are considered to exit at multiple scales to encode a wide range of disparity. However, local disparity measurements by bandpass mechanisms can be ambiguous, particularly when the actual disparity is larger than a half-cycle of the preferred spatial frequency of the filter, which often occurs in fine scales. In this study, we investigated whether the visual system uses a coarse-to-fine interaction to resolve this ambiguity at finer scales for depth estimation from disparity. The stimuli were stereo grating patches composed of a target and comparison patterns. The target patterns contained spatial frequencies of 1 and 4 cycles per degree (cpd). The phase disparity of the low-frequency component was 0° (at the horopter), –90° (uncrossed), or 90° (crossed), and that of the high-frequency components was changed independent of the low-frequency disparity, in the range between –90° (uncrossed) and 90° (crossed). The observers’ task was to indicate whether the target appeared closer to the comparison pattern, which always shared the disparity with the low-frequency component of the target. Regardless of whether the comparison pattern was a 1-cpd + 4-cpd compound or a 1-cpd simple grating, the perceived depth order of the target and the comparison varied in accordance with the phase disparity of the high-frequency component of the target. This effect occurred not only when the low-frequency component was at the horopter, but also when it contained a large disparity corresponding to one cycle of the high-frequency component (±90°). Our findings suggest a coarse-to-fine interaction in multiscale disparity processing, in which the depth interpretation of the high-frequency changes based on the disparity of the low-frequency component.

Introduction

Binocular disparity, or a difference in the locations of the corresponding points in the images projected to the left and right eyes, is one of the most reliable three-dimensional (3D) depth cues. Currently, the disparity energy model is the most widely accepted theoretical framework to explain binocular disparity processing in the human visual system (Fleet, Wagner & Heeger, 1996; Ohzawa, 1998; Ohzawa, DeAngelis, & Freeman, 1990; Qian, 1997; Qian & Zhu, 1997; Read, Parker, & Cumming, 2002). In this model, each disparity energy sensor, similar to the disparity-sensitive cells in the primary visual cortex, has a pair of bandpass filters, one for the left eye image and the other for the right eye image. These neurons respond to changes in binocular disparity based on either a position or a phase shift between the two receptive fields. When the position- or phase-shift between the two eye receptive fields matches the disparity between the two eye images, the neurons would have the best response. In line with such an assumption of the disparity energy model, much evidence indicates that the disparity information is mediated by multiple spatial frequency channels (Cormack, Stevenson, & Schor, 1993; Frisby & Mayhew, 1978; Hess, Kingdom, & Ziegler, 1999; Julesz & Miller, 1975; Mayhew & Frisby, 1976; Shioiri, Hatori, Yaguchi, & Kubo, 1994; Smallman & MacLeod, 1997; Stevenson, Cormack, Schor, & Tyler, 1992; Yang & Blake, 1991). One prevailing view for multiscale disparity processing is that disparity energy is computed independently within each subband. Because the filter scale varies with the preferred spatial frequency of the bandpass filter, different channels respond to different ranges of disparity. Low-spatial-frequency (coarse-scale) channels have a wider disparity response range and are sensitive to larger disparities, whereas high-spatial-frequency (fine-scale) channels respond to a narrower range of disparity and are tuned to a smaller disparity. Such a relationship between the spatial frequency and disparity selectivity is referred to as the size-disparity correlation (Felton, Richards, & Smith, 1972; Heckmann & Schor, 1989; Marr & Poggio, 1979; Prince & Eagle, 1999; Schor & Wood, 1983; Schor, Wood, & Ogawa, 1984a; Smallman & MacLeod, 1994). 

Local disparity measurements by bandpass mechanisms can be ambiguous in depth interpretation, especially when the input image contains repeated features. Based on the disparity energy model (Fleet et al., 1996; Ohzawa, 1998; Ohzawa et al., 1990; Qian, 1997; Qian & Zhu, 1997; Read et al., 2002), the disparity channel can have the largest response when the position or phase shift between its two eye receptive fields matches the binocular disparity in the two eye images. For a simple stereo display made of a sinusoidal grating, if there is a peak cross-correlation between the left and the right eye image at angle θ, there are also other peaks at θ ± 2 nπ, where n is an integer. All of the possible matches are equally valid; thus, there is no unique solution for disparity. Even with the more complex stimuli, bandpass filtering likely generates multiple peaks in the disparity response function. To resolve this ambiguity, the size-disparity correlation hypothesis assumes a half-cycle limitation (Qian, 1994), in which the smallest peak disparity within ±π phase offset is considered as the estimated depth at that scale. This rule excludes the peak responses outside the half-cycle range for disparity computation. However, the off-range peaks may well correspond to the real disparities at a fine scale. Although it may be possible to find such large real peaks with detectors having binocular receptive fields with large position shifts, resolving the phase ambiguity within a single scale is still a challenging computational problem. 

An alternative solution to the matching ambiguity is coarse-to-fine matching (Boothroyd & Blake, 1984; Chen & Qian, 2004; Farell, Li, & McKee, 2004; Marr & Poggio, 1979; Menz & Freeman, 2003; Quam, 1987; Smallman & MacLeod, 1997). At the real peak of cross-correlation between the two eye images, the disparity response of the channel is expected to peak across a range of scales. Thus, the disparity information of coarse scales can be used to constrain the solution of the peak in a finer scale and thus eliminate ambiguity. This coarse-to-fine interaction overcomes the half-cycle disparity range limitation and allows the fine-scale mechanisms to respond to a broader range of disparity. 

Psychophysical evidence supports the coarse-to-fine interaction in human stereopsis. It is shown that humans can detect large disparities beyond the half-cycle limit of the size-disparity correlation (Prince & Eagle, 1999; Schor & Wood, 1983; Schor, Wood, & Ogawa, 1984b). Farell et al. (2004) addressed the coarse-to-fine interaction in stereopsis by using compound grating stimuli made of spatial frequencies separated by 2 octaves (e.g., 1f + 4f). They measured disparity discrimination thresholds while changing the overall disparity of the compound stimuli. The disparity given to the two frequency components was always the same. Their results showed that the discrimination threshold was lower for the compound gratings than either of the component frequencies when the disparity pedestal was at or slightly higher than a half cycle of the higher frequency. Their results suggest that the presence of a low-spatial-frequency component enables a high-spatial-frequency component to improve stereo discrimination even when the disparity was outside the half-cycle limit, demonstrating the coarse-to-fine interaction in the stereo discrimination sensitivity. 

As to the perceived depth, Farell et al. (2004) also reported that the perceived depth of the 1f + 4f grating was very close to that of the 1f simple grating, even when the disparity was larger than the half cycle of the fine-scale (4f) component. They concluded that, although the coarse-to-fine interaction enhanced the stereo discrimination sensitivity, the presence of the additional fine-scale component had no discernable effect on the perceived depth of the multiscale stimuli compared to the single-scale stimuli. In their experiment, it should be noticed that the two components had the same disparities. One interpretation of this finding is that the visual system estimates the perceived depth through the coarse-to-fine interaction. That is, the information of the coarse scale is used to resolve the disparity ambiguity in the fine scale, thus constraining the depth estimation of the fine scale to a larger disparity that equals the disparity manipulated in the stimuli. However, because the two components share the same disparities, it is possible that the fine-scale component might just be ignored in depth matching. A possible scenario is that the fine-scale component is processed independently of the coarse-scale component but can contribute to depth discrimination accuracy because the accuracy is higher than the coarse-scale component; however, it does not contribute to depth magnitude because its minimum-phase depth is significantly different from that of the low-frequency component. We therefore do not believe that the findings reported by Farell et al. (2004) provide conclusive evidence that the visual system utilizes the coarse-to-fine interactions in estimating perceived depth. 

Here, we examined whether a coarse-to-fine interaction exists in estimating suprathreshold stereoscopic depth from disparities in multiscale stimuli. The experiments were designed based on the earlier study of Farell et al. (2004). The stimuli were stereograms with compound sinusoids separated by 2 octaves (1f + 4f). We first gave the compound grating a crossed or uncrossed binocular disparity corresponding to a quarter cycle of the 1f component. According to the earlier findings of Farell et al. (2004), we can expect the perceived depth of this compound grating to be consistent with the assigned disparity, although the effective disparity of 4f is zero. Our question is how the visual system processes disparities larger than a half cycle of a periodic pattern in estimating perceived depth. In other words, is the phase of the 4f component interpreted simply as zero disparity or as a one-cycle disparity via the coarse-to-fine interaction? To answer this question, we further modulated the disparity of 4f while keeping the disparity of 1f the same to see whether a change occurred in perceived depth. Figure 1 shows the prediction of the two possible mechanisms we considered for interpreting the depth measurement of Farell et al. (2004). The first mechanism is that, when the disparity indicated by the coarse-scale component is greater than a half cycle of the fine-scale component, the visual system ignores the fine-scale disparity information in estimating perceived depth. In this case, we would predict that, at larger disparities, the perceived depth of the multiscale image is determined only by the coarse-scale disparity and is independent of the disparity manipulated in the fine scale (the red and blue lines in Figure 1A). Thus, the additional disparity in the fine-scale should have no change in perceived depth, and the perceived depth should be the same as that indicated by the coarse-scale disparity. This hypothesis also predicts that the visual system can use fine-scale disparity when the disparity indicated by the coarse-scale component is zero (the green line in Figure 1A). Because the disparities indicated by the coarse and fine scales are similar, observers may perceive a depth between the coarse- and fine-scale disparities through disparity averaging (Parker & Yang, 1989; Rohaly & Wilson, 1994). 

Figure 1.

 

Predictions of the possible multiscale disparity processing mechanisms. The left diagram in each panel illustrates the possible depth indicated by the coarse scale and the fine scale in our stimulus manipulation for mechanisms ignoring the fine scale at larger disparities (A) and using the coarse-to-fine interaction (B). The right diagram in each panel depicts the predicted perceived depth of the multiscale image relative to the coarse-scale disparity. The red, green, and blue curves indicate the conditions with a −90°, 0°, and 90° phase disparity, respectively, in the coarse scale.

Alternatively, the visual system could adopt the coarse-to-fine interaction to reduce the matching ambiguity at fine scales based on the disparity information of coarse scales. In this case, we expect that the possible depth matches of the fine-scale component will be constrained by the phase disparity in the coarse-scale component and indicate depth signals around the coarse-scale disparity (see Figure 1B). The visual system can then integrate the depth signals across scales via disparity averaging. Thus, no matter whether the coarse-scale component contains zero, crossed, or uncrossed disparities, the perceived depth should change with the phase disparity of the fine-scale component. 

Methods

Apparatus

The stimuli were presented on a calibrated Display++ LCD monitor (Cambridge Research Systems Ltd., Rochester, UK) controlled by a ViSaGe x16 PCI express graphic card (Cambridge Research Systems Ltd.) on a Dell Precision 5820 Tower Workstation with the Windows 10 Pro for WorkStations operating system (Dell, Round Rock, TX). The monitor had a spatial resolution of 1440 (horizontal [H] × 1080 vertical [V]) and a refresh rate of 100 Hz. The calibration was done in the VSG Desktop software to achieve a linear grayscale with a mean luminance of 59.6 cd/m². At a viewing distance of 120 cm, 1 pixel on the screen subtended 0.017° × 0.017° (about 1.04 arcmin per pixel). 

The experimental control and the stimuli generation were written in MATLAB R2021a (MathWorks, Natick, MA) with the CRS toolbox. Left and right eye images were simultaneously presented side by side on the monitor, and a backboard was placed in the middle to separate the monitor into two halves. Observers viewed the stimuli through a four-mirror stereoscope in a dark room. The stereoscope mirrors reflected the left image to the left eye and the right image to the right eye, allowing the observers to fuse the left- and right-eye images effectively into a stereoscopic image. The observer's head was stabilized by a chin rest. 

Participants

Four observers participated in this study, including one of the authors of this article and three observers who were naïve to the purpose of this study. All of the observers had normal or corrected-to-normal visual acuity (20/20). The use of human participants was approved by the ethics committee of the Graduate School of Informatics, Kyoto University. Written consent was obtained from each observer before the experiment. 

Stimuli

The stimuli were stereograms made of gratings. A stereogram was composed of a target pattern and a comparison pattern for both eyes (see Figure 2). The target and the comparison patterns were randomly assigned to the upper and lower halves of the stereogram, separated by a sharp-edged horizontal gray area (0.35° in height). Observers judged whether the target or the comparison pattern looked closer to them. All of the stimuli had a Gaussian envelope extending 4.19° H by 4.21° V with a width parameter σ of 1.04°. Hence, the Gaussian envelopes centered at the middle of both two-eye images had a zero disparity. 

Figure 2.

 

Display configuration. The stimuli were stereo grating patches composed of upper and lower parts. In each trial, the target and the comparison pattern were randomly assigned to these two parts. In the demonstration here, the upper pattern is the target compound grating, which contains a 90° phase disparity in both 1-cpd and 4-cpd components. The comparison pattern in the lower half is a 1-cpd simple grating with a 90° phase disparity.

The target patterns were vertically oriented compound gratings consisting of sinusoids that were 1 cycle per degree (cpd) and 4 cpd. The phase of the luminance profiles for each frequency component was randomly assigned. We manipulated the disparity of each frequency component by a horizontal phase shift between the two eye images. The phase disparity of the low-frequency (1-cpd) component was −90°, 0°, or 90°, and that of the high-frequency (4-cpd) component ranged from −90° to 90° with a step size of 45°. The positive signs indicate the near direction, and the negative signs indicate the far direction. 

Two kinds of comparison patterns were used in separate experiments. In Experiment 1, the comparison patterns were compound gratings, in which a 4-cpd sinusoid was added to the 1-cpd sinusoidal luminance modulation like the target pattern. The phase disparity of the 1-cpd component was the same as its corresponding target, but the phase shift was fixed at 0° for the 4-cpd component. The comparison patterns in Experiment 2 were simple gratings, which consisted of a vertically oriented 1-cpd sinusoid. The phase disparity of the 1-cpd sinusoid was the same as that of the 1-cpd component in its corresponding target pattern. For Experiment 3, we used both compound and simple gratings as comparison patterns. The luminance contrast of each frequency component was 30% for the compound gratings, and the contrast for the simple gratings was 60%. 

Procedure

In each trial, the left- and the right-eye images were presented on a uniform gray rectangular background patch (7.68° H × 8.41° V) whose luminance matched the mean luminance of the stereo grating patches. The screen outside the background patches had a uniform luminance of 0.1 cd/m². Two nonius lines, a vertical line (0.14° in length) and a horizontal line (0.21° in length), were presented dichoptically at the center of both of the two-eye images. These lines formed a capital letter T in the right-eye image and an upside-down T in the left-eye image. The two-eye images had to be aligned properly for the observer to perceive a full cross (+). Figure 2 shows an example. 

Each trial began with an 800-ms uniform gray blank period followed by the stimulus presentation with a beep sound. The duration of the stimulus presentation varied in the experiments. In Experiment 1 and Experiment 2, the stimuli were presented on the monitor until the observers made a response. The nonius lines were flashed on and off during the stimulus presentation for the observers to stabilize their vergence. For Experiment 3, the stimuli were shown briefly for 300 ms, followed by a blank at the mean luminance of the stimuli. The observers’ task was to press one of two response buttons on a CT6 Push Button Response Box (Cambridge Research Systems Ltd.) to indicate which pattern, the upper or lower one, was perceived closer in depth. The observers’ response would initiate the subsequent trial in the run. 

Each run contained 50 trials. In each run, the phase disparity of the low-frequency (1-cpd) component was the same across trials, whereas the phase disparity of the high-frequency (4-cpd) component varied trial by trial. All conditions were randomly presented to the observers. We measured the percentage of perceiving the target pattern as the closer one in each run. Each data point was the average of the measurements across four runs. 

Results

Experiment 1: Effect of fine-scale phase disparity on perceived depth in compound gratings

In Experiment 1, we tested whether the additional phase disparity in the fine-scale (4-cpd) component affects the perceived depth of a multiscale (1-cpd + 4-cpd) pattern. Figure 3 shows the percentage of perceiving the target as closer than the comparison patterns for each observer (the upper and middle rows) and their averages (the bottom panel). The x-axis is the phase disparity of the 4-cpd component, with the positive value indicating a crossed disparity and the negative value indicating an uncrossed disparity. Symbols and curves in various colors indicate the phase disparity in the 1-cpd component. 

Figure 3.

 

The effect of phase disparity in fine scale on depth judgment for each observer (upper and middle panels) and the group data (lower panel). Positive/negative ordinate values indicate crossed/uncrossed phase disparities. Different colors represent the conditions with different phase disparities manipulated in the 1-cpd component. Error bars represent 1 standard error of the means.

When the 1-cpd element was at zero disparity (the green dashed curves), the perceived depth of the stimuli varied with the additional phase disparity in the 4-cpd sinusoid. For all observers, the target patterns were perceived as closer than the zero disparity comparison pattern when they contained a crossed 4-cpd component but were perceived as farther away with an uncrossed phase disparity. Such a result is consistent with the prediction that the mechanism would either ignore the fine scale at larger disparities or use coarse-to-fine interaction (see Figure 1). 

However, when the 1-cpd contained either a 90° crossed (blue curves) or uncrossed (red curves) phase disparity, the depth judgment was affected by the phase disparities in the 4-cpd component, and this effect was the same as that for the zero-disparity 1-cpd component. The results follow the prediction of coarse-to-fine interaction in multiscale disparity processing (Figure 1B) but not by mechanisms that simply ignore the disparity signals in the 4-cpd component (Figure 1A). 

A two-way repeated-measures analysis of variance (ANOVA) showed a significant main effect of the 4-cpd phase disparity, F(4, 12) = 105.89, p < 0.001. An interaction also occurred between the phase disparity of 1-cpd and 4-cpd sinusoids, F(8, 24) = 10.75, p < 0.001. Tests of simple main effects at each 4-cpd phase shift level indicated a significant simple main effect of 1-cpd phase disparity when the 4-cpd component contained −90° phase disparity, F(2, 6) = 8.88, p < 0.05. Post hoc comparison using Tukey's honestly significant difference test showed that, with a −90° disparity in the 4-cpd component, the depth judgment of the −90° 1-cpd disparity condition was significantly different from that of the 90° 1-cpd disparity condition (p < 0.05). Such a difference may occur because the disparities of the two components were in opposite directions and the difference between them was relatively large. In some cases, observers may interpret the −90° disparity in 4-cpd component as a +270° disparity, leading to inconsistent depth judgment responses across trials. 

In short, our data favor the mechanism using coarse-to-fine interaction, which predicts a similar effect of 4-cpd disparity when the 1-cpd component contains either a 0° or ±90° phase disparity. However, both the target and comparison patterns we used were compound gratings. One can argue that the observers might judge the relative depth between the target and the comparison pattern by directly comparing the phase shifts in 4-cpd components between the two patterns rather than integrating the disparity estimation across scales and reporting the overall depth orders of the multiscale patterns. It is necessary to confirm further whether the effects we found were due to the cross-frequency integration of disparity information or to simply looking at the phase shift of the 4-cpd component. 

Experiment 2: Depth comparison between simple and compound gratings

Experiment 1 showed that the additional phase disparity in the fine scale could modulate the perceived depth of the multiscale pattern in a manner consistent with the prediction of coarse-to-fine interaction in multiscale disparity processing. However, because the target and comparison patterns both consisted of 1-cpd and 4-cpd grating and differed only in the 4-cpd phase disparity, the results might be achieved by simply comparing the interocular phase shifts of the 4-cpd component between the two patterns, rather than the coarse-to-fine cross-frequency integration. 

Here, to investigate whether the effects we found in Experiment 1 were due to the cross-frequency integration of disparity information, we asked the observers to judge the depth of target patterns relative to simple gratings that contained only the coarse-scale component. Figure 4 shows the data. Similar to Experiment 1, regardless of the phase disparity of the 1-cpd component, the additional phase-disparity information in the 4-cpd component affected the perceived depth of the compound grating. The crossed 4-cpd phase disparities pulled the target pattern toward the observer, and the uncrossed 4-cpd phase disparities pushed the target pattern away from the observer. 

Figure 4.

 

The relative depth judgment between simple and compound sinusoidal patterns for each observer (upper and middle panels) and the averaged data (lower panel). The target patterns were compound gratings, whereas the comparison patterns were simple gratings. Positive/negative ordinate values indicate crossed/uncrossed phase disparities. Different colors represent the conditions with different phase disparities manipulated in the 1-cpd component. Error bars represent 1 standard error of the means.

Again, the result is supported by the two-way repeated-measures ANOVA on the group data, which revealed a significant main effect of the 4-cpd phase disparity, F(4, 12) = 27.00, p < 0.001. However, the interaction between the two phase-disparity manipulations was also significant, F(8, 24) = 3.91, p < 0.01, indicating that the effect of the 4-cpd phase disparity depended on the phase disparity of the 1-cpd component. Simple main effect tests at each 4-cpd phase shift level showed a significant simple main effect of 1-cpd phase disparity when the 4-cpd component contained −90° phase disparity, F(2, 6) = 7.09, p < 0.05. Post hoc comparison using Tukey's honestly significant difference test revealed a significant difference in depth judgment between the condition with −90° phase 1-cpd disparity and the condition with 90° phase 1-cpd disparity when the 4-cpd phase disparity was −90° (p < 0.05). In general, the effect of the 4-cpd phase disparity on perceived depth judgment was the same as in Experiment 1. These results were in accord with the prediction of coarse-to-fine interaction in multiscale disparity processing (see Figure 1B), indicating that the visual system does integrate different disparities across scales based on the coarse-to-fine interaction. 

Experiment 3: Multiscale disparity processing under brief stimulus presentation

In Experiments 1 and 2, the stimuli were presented on the screen until the observers made a response and the nonius lines were flashed on and off to stabilize observers’ vergence. Nevertheless, the observers might have moved their eyes during the trial. In Experiment 3, we shortened the stimulus presentation duration for better eye movement control. We repeated all of the conditions in Experiments 1 and 2 but limited the stimuli presentation to 300 ms in each trial. Notice that the 300 ms duration falls within the range of critical time required to discriminate the disparity in high- and low-spatial-frequency gratings (approximately 150 ms and 750 ms, respectively) (Lee, Shioiri, & Yaguchi, 2004). This operation also allowed us to gain insight into the temporal characteristics of the coarse-to-fine interaction in perceiving depth. 

Figure 5 shows the data for the four observers (top four rows) and their average (bottom row). The left column represents the data for conditions using compound gratings as comparisons, and the right column plots the data when comparing the target patterns to simple gratings. Generally, the percentage of judgments that the target patterns were closer increased with the phase disparity of the 4-cpd component in the target patterns. The result agreed with the prediction of the coarse-to-fine interaction in multiscale disparity processing, similar to our observations in the above two experiments using unlimited stimulus presentation duration. However, this 4-cpd phase disparity effect depended on the type of comparison pattern. The influence of the 4-cpd phase disparity on depth judgment was smaller when the target pattern was compared with the 1-cpd simple grating than the 1-cpd + 4-cpd compound grating. Such a reduction in the influence of the 4-cpd phase disparity was consistent with those we observed between Experiment 1 and Experiment 2 with unlimited presentation duration. These results are supported by a three-way repeated-measures ANOVA on the group data with a significant main effect of the 4-cpd phase disparity, F(4, 12) = 7.39, p < 0.01, and a significant interaction between the 4-cpd phase disparity and the type of comparison pattern, F(4, 12) = 7.97, p < 0.01. 

Figure 5.

 

The effect of phase disparity at fine scale on depth judgment with 300-ms stimulus presentation. The carriers for the comparison pattern were either 1-cpd simple gratings (right column) or 1-cpd + 4-cpd compound gratings (left column). Positive/negative ordinate values indicate crossed/uncrossed phase disparities. Different colors represent the conditions with different phase disparities manipulated in the 1-cpd component. Error bars represent 1 standard error of the means.

Although the brief-presentation dataset is in accord with the prediction of the coarse-to-fine multiscale disparity processing, it reveals a lack of strong interaction across frequencies in disparity processing. Compared to the individual performance in Experiment 1 and Experiment 2, the short presentation time greatly affected the depth judgment for observer SZT in all conditions and observer CPY in conditions with an uncrossed disparity in the 1-cpd sinusoid (red curves). This might indicate that the 300-ms stimulus presentation is too short for the visual system to integrate disparity information across scales based on the coarse-to-fine interaction. 

Discussion

In this study, we manipulated the phase disparity of the sinusoid components in stereo grating patches to investigate whether the visual system employs coarse-to-fine interaction to integrate disparity information across scales for perceiving depth. A 1-cpd + 4-cpd target pattern was presented with a comparison pattern for the observers to judge which one was closer in depth. We found that the relative depth judgment of the target pattern to the comparison pattern varied with the phase disparity of the fine-scale sinusoid. This effect occurred regardless of whether the comparison patterns were multiscale compound gratings (Experiment 1) or low-frequency simple gratings (Experiment 2). However, the effect was less obvious when presenting the stimulus briefly for 300 ms (Experiment 3). Such results favored the mechanism of coarse-to-fine multiscale disparity integration. 

Boothroyd and Blake (1984) also investigated how the disparities created by spatial frequency components affected the perceived depth. They used a variety of compound gratings (e.g., gratings made of 3f + 5f sinusoidal components) as stimuli, manipulated the interocular phase difference in each frequency, and then measured the perceived depth. They found that the perceived depth of the compound grating was determined by the overall disparity of the compound, even when it was larger than the half-cycle limitation of the individual components. Their results seem to be in line with our finding of the coarse-to-fine interaction in disparity processing. However, the selection of spatial frequency components was different between the study by Boothroyd and Blake and our study. In Boothroyd and Blake (1984), the ratio of scales for the compound gratings was always within the range of 2 octaves. It is possible that the larger depth perceived beyond the half-cycle range in their study might be mediated by the second-order stereopsis, which is a mechanism that responds to the second-order structure in the scenes, such as contrast modulations (Hess & Wilcox, 1994; Hess & Wilcox, 2008; Sato & Nishida, 1993; Wilcox & Allison, 2009). The standard model for processing second-order information is the linear–nonlinear–linear (LNL) model (Wilson & Richards, 1992). In the LNL model, the first-stage linear filter responses are nonlinearly transformed through a full-wave rectification and then pooled by a second-stage linear filter that has a larger receptive field than the first-stage bandpass filters. The process of the LNL model is different from the coarse-to-fine interaction. In contrast, we used compound gratings with spatial frequencies separated by 2 octaves, in which the coarse and fine scales are more likely to be processed by separate bandpass filters and the LNL model cannot detect second-order features effectively. Thus, we considered that our findings favored the coarse-to-fine interaction and may not be explained by the LNL model. 

Farell et al. (2004) also found evidence for a coarse-to-fine interaction in disparity processing. Their critical finding was that the discrimination threshold was lower for the 1f + 4f grating than for either the 1f grating or the 4f grating when the pedestal disparity was at the half-cycle of the 4f grating. Their result cannot be explained by independent processing of 1f and 4f components, because it would produce a discrimination threshold of compound grating equal to the lower threshold of the component gratings. Although their experiment used a set of stimuli similar to ours and addressed a similar research question, significant differences exist between their study and ours. First, whereas the 1f and 4f components always had the same disparity in Farell et al. (2004), the disparity of the 4f component could be different from that of the 1f component in our experiments. Second, although the study by Farell et al. demonstrated an improvement in the discrimination threshold, we showed an effect of the additional phase disparity in the 4f component on perceived depth. Although Farell et al. (2004) also measured the perceived depth of the 1f + 4f grating in a subsidiary experiment, the components of their compound gratings always had the same disparity. We not only replicated the conditions that Farell et al. (2004) measured on perceived depth (the middle point of each color curve in Figure 4) but also demonstrated that the perceived depth of the compound grating varied with the phase disparities in the fine scale (Figures 3–5). Third, although Farell et al. (2004) found a contribution of the 4f component at around the pedestal disparity of a half cycle of the 4f component, we found a depth modulation effect by the 4f component at the pedestal disparity at around one cycle of the 4f component. At this large disparity, Farell et al. (2004) did not find any notable effect (i.e., the threshold was approximately equal between the 1f grating and the 1f + 4f grating). Therefore, our study provides novel independent psychophysical evidence supporting coarse-to-fine processing in stereopsis. 

Similar to our procedure, Rohaly and Wilson (1994) superimposed two spatial frequency gratings separated by 2 octaves and assigned different disparities to each scale. They showed that the perceived depth was at a depth between the two disparity components carried by the compound gratings. Based on these results, they proposed a multichannel model in which each spatial scale processes a different disparity to generate separate disparity estimations, and the overall perceived depth of the stimulus is the weighted combination of these disparity estimates. However, in their study, they assigned disparities only within a quarter cycle of the fine-scale component (either zero disparity or a disparity of 112.29 arcsec) to the two scales to avoid the ambiguous disparity cues due to the periodic luminance waveform. Their results appeared to work, because the mechanism with independent processing channels and the disparity-averaging operation in their model is generally considered to be linked to the mechanism based on the size-disparity correlation to process disparity independently within each subband (Felton et al., 1972; Heckmann & Schor, 1989; Marr & Poggio, 1979; Prince & Eagle, 1999; Schor & Wood, 1983; Schor et al., 1984a; Smallman & MacLeod, 1994). Different from the disparity range that Rohaly and Wilson (1994) used, we made use of the periodic structure of gratings and had conditions with coarse-scale disparity beyond a half cycle of the fine-scale component (red and blue curves in Figures 3–5). In these conditions, if the depth estimation of each scale is limited within a ±π phase shift, the fine scale would always suggest a depth smaller than the depth indicated by a 90° phase disparity of the coarse-scale component. After disparity averaging, the model of Rohaly and Wilson (1994) would expect that the additional phase disparity in the fine scale reduces the perceived depth of the compound grating compared to the coarse-scale disparity. In this case, observers should always perceive the target pattern as being closer when the coarse-scale disparity is −90° but perceive the target pattern as being farther away when the coarse-scale disparity is 90° (Figure 6), which was not the case in our data (Figure 4). Thus, the model of Rohaly and Wilson (1994) is incompatible with our results. A coarse-to-fine interaction must exist to constrain the response of the fine scale based on the disparity information of the coarse scale. 

Figure 6.

 

Prediction of the model of Rohaly and Wilson (1994) for the conditions compared with 1-cpd simple gratings used in this study. Different color curves indicate the conditions with different phase disparities in the 1-cpd component.

Next, we consider the possible mechanisms for coarse-to-fine interaction. The possible mechanisms are not necessarily mutually exclusive. 

First, the coarse-to-fine interaction is generally considered a sequential process (Allen & Freeman, 2006; Marr, 1982; Watt, 1987). Several computational models suggest that the coarse scales first provide an initial estimation of disparities in a scene, and then the perceived depth is fine-tuned by the fine scales (Chen & Qian, 2004; Marr & Poggio, 1979; Nishihara, 1984; Quam, 1987). The physiological measurements of the temporal characteristics of the disparity-sensitive complex cells in a cat's striate cortex made by Menz and Freeman (2003) support the temporal coarse-to-fine processing. Their findings showed that disparity tuning of most binocular complex cells became sharper during the time course of responses, and cells tuned to coarse and fine scales had relatively shorter and longer temporal latencies. Further analysis of the simultaneously recorded cell pairs also showed a stronger connection for coarse-to-fine processing than for fine-to-coarse processing. In our Experiment 3, we tested the coarse-to-fine interaction under a short stimulus presentation. Although the effect of 4-cpd phase disparity was sustained when we reduced the stimulus presentation duration to 300 ms, we found that this effect was less pronounced than in the dataset using long stimulus presentations (Experiments 1 and 2). This result might be explained by the sequentially processed coarse-to-fine interaction, because it may predict that, under brief presentation, the sequential processing is not fully developed and the coarse-scale interaction is reduced. As far as we are aware, however, there is no independent evidence that the coarse-to-fine processing takes as long as several hundred milliseconds. 

Second, with coarse-to-fine processing, the visual system can use fine-scale mechanisms even when at large disparity. A possible mechanism encoding a large disparity at fine scale could be disparity channels that have binocular receptive fields with large position shifts that can be activated only by multiscale patterns. The existence of these position-based channels, whose disparity tuning does not change over time, may provide the opportunity for the visual system to resolve the phase ambiguity at fine scales. However, most neurophysiological studies measuring the binocular receptive fields and disparity tuning of neurons have used simple gratings rather than multiscale patterns as stimuli. It is unclear whether such disparity channels exist in the visual system. 

Finally, the visual system is known to use population coding to represent various stimulus properties, such as orientation and motion direction. Based on the population coding, Tsai and Victor (2003) proposed a multiscale neural model for representing binocular disparity based on a template-matching approach. Their model suggests that different channels first respond to the stimulus separately, then the disparity is recovered at the decoding stage by comparing the population response to a set of defined templates, each of which corresponds to a unique disparity. Instead of resolving the disparity ambiguity at fine scales at the encoding stage, this template-matching approach points out the possibility of solving the disparity-matching problem at the decoding stage. However, our data might be insufficient to examine this mechanism. Further research and analysis are needed to explore the underlying mechanism of the coarse-to-fine interaction in perceived depth. 

In conclusion, we have demonstrated that the visual system uses the disparity information at fine scales to estimate the perceived depth of the multiscale pattern even when the disparity is beyond the half-cycle limitation of the fine scale. We considered that it is because the visual system utilizes the coarse-to-fine interaction to resolve the phase ambiguity at fine scales to respond to larger disparities. 

Acknowledgments

Supported by grants from the Ministry of Science and Technology of Taiwan (110-2917-I-564-022), National Science and Technology Council of Taiwan (112-2423-H-002 -002), and Japan Ministry of Education, Culture, Sports, Science, and Technology (MEXT)/Japan Society for the Promotion of Science (JSPS) KAKENHI Grants-in-Aid (20H00603, 20H05957). 

Commercial relationships: none. 

Corresponding author: Pei-Yin Chen. 

Email: [e-mail]. 

Address: Department of Psychology, National Taiwan University, Taipei, Taiwan.