Motion aftereffects specific to surface depth order: Beyond binocular disparity

Abstract

Despite evidence for concurrent processing of motion and stereopsis from psychophysics and neurophysiology, the detailed relationship between depth and motion processing is not yet clear. Using the contingent aftereffect paradigm, we investigated how the order of surfaces presented across depth influenced motion perception. After having observers adapt to two superimposed populations of dots moving in opposite directions at different binocular disparities, we assessed how much of the motion aftereffect (MAE) was specific to absolute disparity and how much was specific to the depth order of the surfaces. The test contained two planes of moving dots at several different pairs of disparities and asked observers to report the MAE direction at one of the planes (the target). In addition to the disparity-contingent MAE (Verstraten, Verlinde, Fredericksen, & van de Grind, 1994), we found MAEs dependent on surface order. When the target surface was in front of another surface, observers more often reported the MAE in the direction opposite to the front adapting surface than the back. This effect was observed despite differences in absolute and relative disparity between the adapted and test surfaces. The results suggest that some motion information is represented in terms of surface depth order.

Introduction

Most of our visual experience in daily life consists of surfaces. “The surface is where most of the action is” (Gibson, 1979) because light reflected from the surfaces in front of our eyes contains all the visual information we get from the environment. The representation of surfaces is a stage of visual information processing posed between the early detection of visual features and later stages of object recognition (Nakayama, He, & Shimojo, 1995). Surface organization affects several aspects of visual processing, including search (Nakayama & Silverman, 1986), attention (Blaser, Pylyshyn, & Holcombe, 2000; Sohn, Papathomas, Blaser, & Vidnyánszky, 2004), and motion (Stoner & Albright, 1996). A surface can be described by its location and appearance. A single surface usually lies at a smooth and continuous plane rather than across discontinuous depths and consists of relatively homogeneous colors and textures. If our visual system processes visual features in the unit of a surface, the features such as color, texture, and motion comprising a surface may be associated together with the depth at which the surface is located. 

Conjunction of more than one visual property into a single representation has been investigated by studies of selective adaptation. Human vision shows selective adaptation not only to particular features of visual stimuli, such as color, depth, orientation, or motion, but also to the conjunction of such features. In the classic McCollough effect, selective adaptation to a red, vertical stimulus interleaved with a green, horizontal stimulus leads to a percept of green with subsequent exposure to a vertical black-and-white stimulus and of red with a horizontal stimulus (McCollough, 1965). Such contingent aftereffects also have been reported for color and motion (Favreau, Emerson, & Corballis, 1972) and for motion and depth (Anstis, 1974; Nawrot & Blake, 1989; Verstraten, Verlinde, Fredericksen, & van de Grind, 1994). These phenomena lead to questions about the locations in the visual processing stream in which multiple features are combined. Humphrey and Goodale (1998) argued that orientation-contingent color aftereffects (McCollough effects) are based in large part on neural adaptation in early visual area such as V1. On the other hand, Domini, Blaser, and Cicerone (2000) showed that the locus of combined processing of color and depth is likely to lie at a rather later stage, beyond the site of binocular matching. In their more recent study, these authors proposed that the representation of surfaces is a stage where features such as color and orientation are linked with particular depths (Blaser & Domini, 2002). 

Among other visual properties, conjunction of motion and stereopsis into a single representation has been extensively investigated. Anstis and Harris (1974) reported that the motion aftereffect (MAE) is contingent on binocular disparity. They had their observers adapt to two alternating textured discs rotating in opposite directions with different binocular disparity, one located in front of fixation and the other behind fixation. In the test phase, a static textured pattern was presented either in front or behind the fixation. The direction of the MAE was always opposite to the motion direction of the adapting disc at a given depth plane. Verstraten et al. (1994) showed the same disparity-contingent aftereffects with transparently moving dots. After adapting to a bidirectional transparent motion display at different depth planes, observers reported transparent MAEs that were contingent on the depth planes during adaptation. They also reported that the magnitude of the MAE decreased as the difference in disparity between adapting and test stimuli increased, implying adaptation of neurons sensitive to both a specific motion direction and specific disparity. Nawrot and Blake (1989, 1991a) reported the inverse phenomenon, a depth aftereffect contingent on motion directions. In their experiment, adaptation to transparent dot fields that were moving in opposite directions at different disparities generated a rotational aftereffect in a structure from motion (SfM) display with ambiguous directions. The SfM display consists of two groups of dots moving in opposite directions (left and right) that generate a three-dimensional form (Todd, 1984). The speed profile of the dots across space is such that the display appears to be a dotted transparent globe or cylinder rotating clockwise or counterclockwise around a vertical axis. Because the two groups of dots are both at zero disparity, the depth order of the leftward and rightward motion is ambiguous. Observers can see either leftward motion on the front surface of the globe or on the back surface of the globe. Nawrot and Blake (1989, 1991a) capitalized on this ambiguity to test whether adaptation to unambiguous motion could influence the perceptual interpretation. After observers adapted to the leftward motion at crossed (in front) and rightward at uncrossed disparities, they perceived that the front surface of an ambiguous SfM globe was moving rightward. These psychophysics studies demonstrate that motion and depth are co-registered at some level of visual processing. 

Association between motion and depth may be supported by concurrent coding of these properties in the motion-sensitive visual area, MT. The importance of area MT in motion processing is clear, as it contains neurons correlated with and a determinant of motion perception (for a review, see Britten, 2004). About two thirds of MT neurons are also selective for disparity, but they are found in discrete patches, which are separated by regions that have poor disparity tuning (DeAngelis & Newsome, 1999). However, this spatial map for disparity selectivity does not have a systematic relationship with that for motion direction, as one would expect for motion in three-dimensional depth if a particular direction was coded dependent on a particular disparity (Maunsell & van Essen, 1983). 

What then is the function of disparity in the motion-sensitive area MT? Disparity sensitivity may provide cues for motion segmentation and for the representation of surfaces. Suppression effects between opposite motion signals in MT become weaker if two directions are presented at different depth planes (Bradley, Qian, & Andersen, 1995). Therefore, transparent moving surfaces at different disparities are represented independently in MT whereas opposite motion signals from a same surface tend to cancel each other. This depth constraint is thought to provide a way of confining the motion integration to a particular surface, aiding motion segmentation (Born & Bradley, 2005). In fact, disparity sensitivity of MT cells is involved in depth perception of frontoparallel surfaces, as evidenced by biases in disparity discrimination from stimulation of disparity-selective MT neurons (DeAngelis, Cumming, & Newsome, 1998). The segmentation process may further play an important role in the perception of three-dimensional structure, for example, arranging surfaces in depth relative to one another. Bradley, Chang, and Andersen (1998) showed that MT responses can reflect the perceived depth order of surfaces. They had animals view an ambiguous SfM rotating cylinder at zero disparity. When animals reported a certain depth order, say leftward motion in front and rightward back, similar neural responses were generated as those when they viewed the leftward motion at a depth plane in front of rightward motion as defined by binocular disparity. When they reported the reverse surface order with the very same stimulus, the responses changed as if they were viewing the same directions at the reversed, disparity-defined depth order. This report implies that disparity-defined depth order and ordinal representation without disparities involve the same neural mechanism. The disparity coding in MT may potentially be a part of more general process of perceiving depth order, which surface is in front of which, rather than just absolute disparity (Grunewald, Bradley, & Andersen, 2002). 

In the present study, we report evidence for a process of perceiving depth order that occurs across a broad range of binocular disparities. Using the adaptation paradigm, we measured the amount of selective adaptation to motion information that is associated with absolute disparity and that with the surface depth order. The present study provides some new information about the types of depth information used in human motion perception. 

Experiment 1

In our first experiment, we investigated the relationship between motion and the order of surfaces in depth using the contingent aftereffect paradigm. After adapting to leftward motion at crossed disparity (in front) and rightward motion at uncrossed disparity (behind fixation), observers viewing two static planes at the same disparities report rightward MAE in front and leftward behind (Verstraten et al., 1994). Based on the idea that disparity coding in MT may be involved in perceiving depth order rather than just absolute disparity (Grunewald et al., 2002), we postulated that the association between motion and stereopsis in the reported contingent aftereffect may have its ground in rather higher-level organization, such as the representation of surfaces. 

We tested the specificity of the MAE to the adapted disparity. We modified experiments by Verstraten et al. (1994) to test MAEs at various disparities, some of which were the same as the adapted ones and some that were different. There were always two transparent depth planes during the adaptation and the test. Observers reported the MAE direction of one of the surfaces (called the target surface) cued by a tone. Both test surfaces contained randomly moving dots that have been shown to generate stronger MAEs compared to static patterns (Hiris & Blake, 1992). We replicated the depth-contingent MAE previously reported with the same disparities as the adapted ones. When the target surface was not at an adapted disparity, the direction of the MAE at the target depth plane was influenced by the relative position of the two surfaces during the test. 

Methods

Observers

Four observers including the two authors (AES and WYS) participated in the experiment. They had normal or corrected-to-normal visual acuity. 

Stimuli

During adaptation, two superimposed populations of dots were moving in opposite directions (left and right) within a square aperture (6.1 × 6.1 deg in visual angle). Each dot group contained 150 dots against black background. A single dot was a square subtending 3.6 × 3.6 arcmin and moved at 2.7 deg/s with limited lifetime of 112 ms. Each dot was actually presented with a stereoscopic pair of colored dots, one red and the other green, viewed through red/green stereo glasses, generating a percept of depth. During motion adaptation, one group of dots was presented at the disparity of 11.6 arcmin crossed (in front relative to the fixation point), and the other dot group was presented at 11.6 arcmin uncrossed (behind fixation). During the MAE test, two planes of randomly moving dots were presented with the same relative disparity (23.2 arcmin) but at various absolute disparities (−29.2 and −6; −23.2 and 0; −11.6 and 11.6; 0 and 23.2; 6 and 29.2 arcmin, where negative numbers indicate crossed disparity and positive uncrossed). Each group of dots was presented within a square aperture, and two apertures were misaligned by 0.3 deg to help observers distinguish two depth planes easily (Figure 1). 

Figure 1

 

A schematic drawing of stimulus in Experiment 1. Observers viewed two stereoscopic planes of dots, one in front of the fixation cross and one behind. Dots and frames shown in black actually consisted of red and green stereoscopic pairs, generating a percept of light yellow dots and frames against a black background seen through red/green glasses. Arrows and dotted lines did not appear in the visual display. During adaptation (top), the two groups of dots presented at crossed and uncrossed disparities moved in opposite directions. During the MAE test (bottom), two groups of randomly moving dots were presented at various disparities (see text). The same disparity pair as the adaptation is shown.

Procedure

Each trial consisted of two parts: adaptation and test intervals. During the 60 s of the adaptation interval, observers fixated the cross at the center of the stimulus without performing any task. Then, a 1-s blank screen was presented, followed by a test interval. Right before the test stimulus presentation, a brief beep sounded to cue the target surface, whose MAE direction was to be reported. If the beep was in a high tone, observers were supposed to report the direction of the MAE of the front surface of the two test surfaces, and if it was in a low tone, they reported the MAE direction of the back surface. The front and the back surfaces were cued equally often for every absolute disparity that was tested. After 4 s of the test stimulus presentation, observers reported the direction of the MAE in the target surface, left or right, by pressing a key. The next trial started 10 s after observers' responses. There were 20 trials within one session. Observers performed four sessions, for a total of eight repetitions per condition. The adapting direction at each disparity was counterbalanced across sessions. 

Results

Figure 2 shows the main results of Experiment 1, averaged across four observers. The results were recorded as the percentage of trials the observers reported the MAE in the opposite direction to the motion of the front adapting surface. When the test surfaces were at the same disparities as the adapted ones, observers reported the direction of the MAE that was complementary to the adapting surfaces, replicating previous work (Verstraten et al., 1994). Specifically, when the target surface was at the same depth as the front surface during adaptation, the MAE direction was reported more often in the direction opposite to that of the front adapted surface than the back, t(3) = 55, p < .0001 (Figure 2, left bar). 

Figure 2

 

Results of Experiment 1. Percentage of trials that observers reported the MAE in the direction opposite to the motion of the front adapting surface is shown against the test conditions. The data were pooled across both adapting directions (left and right) at a given depth plane. Depicted below the graph are the different disparities tested with values given at the right. At the left of the graph, lines show the disparities of the adapted planes. Below each data bar, thick lines show the disparities of the tested planes. Black thick lines represent the target surface, the surface of the reported MAE. Gray thick lines indicate the nontarget surface that functioned as a context. Data are averaged across four observers and the error bars indicate standard error.

As the disparity between adapting and target surfaces became larger, MAE direction was less affected by adaptation (not shown), consistent with the previous work (Verstraten et al., 1994). Reports of the MAE opposite to each adapting surface (±11.6 arcmin) were given on average 85% of the time for the target surfaces at ±23.2 arcmin, and 73% for those at ±29.2 arcmin. 

Interestingly, when the target surface was at zero disparity, which was exactly in between the two adapting surfaces, observers' responses differed depending on the relative positions of the two test planes. When the target surface was in front of the other surface, observers reported that the MAE direction was opposite to the motion in the front adapting surface, whereas they reported the MAE direction opposite to the back adapting surface when the target was behind the other surface, t(3) = 48, p < .0001 (Figure 2, middle). When the fixation surface was the context and the other surface was the target (the thick gray lines in Figure 2), complementary results were observed. Observers reported the MAE opposite to the front adapting surface for 78% of the time with the target surface in front at −23.2 arcmin, and 9% with the target surface in back at +23.2 arcmin. 

The effects of surface order even overpowered the proximity in disparity between adapted and target surfaces (Figure 2, right bar). For example, when the target surface was in the back but at a crossed disparity (e.g., −6 arcmin) that was closer to the front adapting surface (−11.6 arcmin) than the back (+11.6 arcmin), observers more often reported the MAE in the direction opposite to the motion of the back adapting surface than the front, t(3) = 13, p < .001 (Figure 2, right bar). Consistent with the prediction from surface order effects, the closest target surface at −29.2 arcmin resulted in 78% of responses to the direction of the front MAE. 

Across all conditions, the direction of the MAE was influenced by the depth order of the surfaces in the test, suggesting that the MAE is modulated by more than just disparity. 

Experiment 2

In addition to the disparity-dependent effects, the results of Experiment 1 showed that MAEs at unadapted disparities were influenced by surface depth order. In Experiment 2, we tested how general these effects were by measuring MAEs at the same disparities as those during adaptation but with a different surface depth order. One possibility is that the ordinal relation of surfaces affects motion perception only when the association between direction and disparity is ambiguous. Previous work has shown that the strength of the MAE is weaker at unadapted disparities than at the adapted disparity, with a monotonic decrease in the MAE with the distance from the adapted disparity (Verstraten et al., 1994). In Experiment 1, we observed surface depth order effects at test planes that were intermediate distances from the two adapting planes, where disparity-dependent MAE would have been rather feeble. If information about surface order affects motion interpretation only when neither directional aftereffects are dominant, then the surface order effects would only be observed at an unadapted disparity, not at the adapted disparity. Alternatively, if adaptation dependent on surface depth order reflects motion interpretation that is independent of the adaptation effects of absolute disparity, one should be able to observe the depth-order-dependent bias in MAE even at the same disparity as the adapted one. 

In Experiment 2, we tested whether surface depth order could bias the MAE direction at the same disparities as those during adaptation. For example, we tested whether the direction of the MAE was opposite to the motion of the front or back adapting surface if the target surface at the disparity of the front adapting surface was positioned behind the other surface. 

We also modified the experimental design in several ways. In Experiment 1, the MAE was indexed by the number of observers' binary responses (left or right) in the MAE direction, rather than a quantitative measure of the strength of the MAE. Also, the position of the target surface was explicitly cued during the MAE test period. Observers' awareness of the depth order of the two surfaces may have influenced their responses. To measure the MAE dependent on surface depth order more quantitatively and objectively in Experiment 2, we used a direction deflection method (Blaser, Papathomas, & Vidnyanszky, 2005). Using the phenomenon that MAEs alter the direction of coherent, unidirectional motion (Levinson & Sekuler, 1976), we measured the deflection in the direction of coherently moving dots during the test. Observers were tested with coherent motion in the direction orthogonal (upward) to the two adapting directions (left and right) and asked to report the perceptual deflection from the vertical direction. If the MAE was leftward, one would experience the upward motion tilted to the left and the amount of tilt indicates the strength of the MAE. The direction deflection is assumed to reflect the adaptation of the same motion mechanism that has been tested with random motion in Experiment 1 because both random motion (Hiris & Blake, 1992) and coherent motion (Levinson & Sekuler, 1976) integrate with MAEs such that the percept of this integrated motion is not distinguishable from real motion. The other plane presented during the test period contained flickering dots without any motion in a particular direction. Observers were asked to judge the tilt of the upward motion and ignore the flickering dots but were never cued to the location of the two planes in depth, as this was unimportant for the task. This direction deflection method provided a quantitative measure of the MAE magnitude, and moreover, it was an objective measure that avoided explicit indication of the relative positions of the target surface. 

Methods

Observers

Five observers including the same four observers from Experiment 1 participated in this experiment. 

Stimuli

The stimulus configuration was similar to the one in Experiment 1, except for the following. Moving dots with limited lifetime (90 or 168 ms) were presented within a square aperture (5.5 × 5.5 deg). A mirror stereoscope was used to generate various depth planes, and the dots and frame box were white on a black background. During adaptation, one group of dots was presented at the disparity of approximately −7.3, and the other dot group was presented at +7.3 arcmin (Figure 3a). We used smaller disparity difference (14.6 arc min) between the two surfaces than Experiment 1 (23.2 arc min) to ensure that, while maintaining the same distance between test surfaces, the largest disparity test plane would still be at a moderate disparity that could be fused by all our observers. The fixation marker was a binocular central square (0.1 × 0.1 deg) with a cross. The cross was made from horizontal and vertical monocular nonius lines to give observers feedback about their fixation accuracy. Additionally, white-and-grey poles were shown at the fixation plane to the left and right side of the dots to aid binocular fusion (Figure 3a). 

Figure 3

 

(a) Stimuli of Experiment 2. Black and gray were actually white and grey against a black background. Vertical gray and black poles, a central fixation square and a cross-aided fusion. During adaptation (top), two groups of dots moving oppositely were presented at crossed and uncrossed disparities. Also shown are the two horizontal lines presented at the upper and lower side of the fixation divided into segments for the gap detection task (see text). During the MAE test, coherent upward motion was presented at one disparity and flickering dots were presented at the other. Shown is the same disparity pair as adaptation. (b) A top view of disparities presented during the MAE test. Black thick lines represent the target surface where coherent upward motion was presented. Gray thick lines indicate the nontarget surface where flickering dots were presented. At the bottom of the figure, each condition is labeled. The first letter in each condition indicates the disparity, either zero (Z), crossed (C), or uncrossed (U). The second letter indicates the relative position of the target surface, either in front (F) or in back (B). Note that all disparities were tested at all relative positions.

During adaptation, observers performed a “gap detection” task to help them to maintain fixation at the appropriate depth plane and draw their attention equally away from both the front and back surface. Every 2 s, two horizontal lines were presented at the upper and lower side of the fixation cross for 500 ms. Each line was broken into eight segments, four of which were randomly selected to be presented to one eye and the other four to the other eye. With appropriate fixation on the cross, observers would have perceived these lines at the same depth as fixation plane. There was always one segment missing among 16 that generated a gap in either the upper or lower line. Observers reported which line contained the gap. If fixation moved in depth away from the cross, the line segments would partially overlap and leave more than one gap, leading to poor performance at gap detection. 

During the MAE test, two groups of dots were presented. One group of dots was asynchronously flickering with the limited lifetime, and the other moved coherently in one direction (target surface). The coherent motion was either upward (0 deg) or slightly tilted to left or right. In an initial session, five steps of the tilt angle were used for the all observers (−5.7, −4.1, 0, 4.1, and 5.7 deg, where negative numbers indicate leftward tilt and positive rightward). A subsequent session using seven steps of tilt angle was completed by observer AES, CLW, and WYS (−5.7, −4.1, −2.0, 0, 2.0, 4.1, and 5.7 deg) in order for the curve fitting algorithm to produce a better fit. The two groups of dots were presented either at the same disparity (−7.3, 0, or 7.3) or at different disparities (−21.9 and −7.3; −14.6 and 0; −7.3 and 7.3; 0 and 14.6; 7.3 and 21.9 arcmin, see Figure 3b). The same disparity condition was run only with five steps of the tilt angle for all observers. In this single-disparity condition, observers were still able to clearly segregate flickering dots and coherent motion without difference in disparity, but they reported that the coherent motion appeared in front about as often as in back (observer HMF and SCC). Due to this inconsistent percept of depth order, we did not include this condition in the analysis of surface order effects, but effects of absolute disparity could be assessed. With the two depth planes, all combinations were tested, such that the target plane was presented at each adapted disparity, either in front of the other plane or behind. 

Procedure

Each trial consisted of two parts; the adaptation and test intervals. Initial adaptation before the first trial in a block was 60 s, and in the following trials the top-up adaptation was 10 s. During the adaptation, observers fixated the cross at the center of the stimulus while performing the gap detection task. Observers' responses within 1.5 s after the lines appeared were considered as valid responses. The responses outside this window were considered as wrong answers. The performance for the gap detection task was taken as the index of how well observers fixated during adaptation. If observers' performance did not reach 75% accuracy, the whole session was repeated. The number of repeated sessions was not more than two per observer. The overall performance for each observer ranged from 80.6% to 96.9% correct. After adaptation, a 500-ms blank screen preceded the test interval. The test stimulus was presented for 2 s, after which observers reported the direction of the tilt in the vertical motion, left or right, by pressing a key. The next trial started 1 s after observers' responses. A total of eight repetitions were obtained per condition. 

Results

The data were tabulated as the frequency with which observers judged the direction of the upward motion toward the opposite direction of the front adapting surface (Figures 4a–d). The data were fitted with the logistic function, using “psignifit” version 2.5.6 for Matlab (see http://bootstrap-software.org/psignifit/), which implements the maximum likelihood method described by Wichmann and Hill (2001a). The 50% threshold value was obtained from the fitted logistic function for each condition and used as the point of subjective equality (PSE) for the 0 deg deflection from vertical motion. Observer AES and WYS completed both sessions with five and seven steps of tilt angles and showed similar results. The results from the seven-tilt session are presented in Figure 4. 

Figure 4

 

Results of Experiment 2. (a–d) Psychometric functions for a representative observer (WYS). The proportion of trials that the MAE direction was perceived as opposite to the motion of the front adapting surface is plotted against tilts in the coherent test motion. The solid and dashed lines are best fitting logistic functions for two conditions. (a) Results for the target surfaces at zero disparity, which were unadapted (ZF and ZB from Figure 3). Filled squares indicate trials with the target surface in front. Open squares indicate trials with the target surface in back. Error bars indicate 84% confidence intervals around the PSEs that were found by the bootstrap method based on 1999 simulations (Wichmann & Hill, 2001b). (b) Results from the nonzero disparity conditions averaged to show the absolute disparity effect. Filled squares indicate trials with the target surface at the crossed adapted disparity and open squares show the uncrossed. (c) Results from nonzero disparity conditions averaged to show the surface order effect. Filled squares indicate trials in which the target surface was in front and unfilled squares show in back. (d) Results from single-disparity conditions. Filled squares indicate trials with the test surface at the crossed adapted disparity, crosses zero disparity, and open squares the uncrossed disparity. The function could not be fit in the crossed adapted disparity, so data points were linearly connected. PSE was obtained by linear interpolation. (e–h) PSEs of the two conditions from all observers plotted against the tilts of the coherent motion in the target surface. (e) Effects at the zero disparity test conditions. (f) Effects of absolute disparity (g) Effects of surface depth order. (h) Effects of absolute disparity with test surfaces at a single disparity.

At the unadapted disparity at the fixation plane, similar surface-order-dependent effects as those in Experiment 1 were observed. Psychometric functions are shown from the two zero-disparity conditions where the target surface was in front (Figure 3b, ZF) and where it was behind (Figure 3b, ZB) for one observer (Figure 4a). Across the three observers whose data were fit by the logistic function, the PSEs for these two conditions differed by 3.1 deg of tilt, t(2) = 6.1, p < .05 (Figure 4e). This difference reflects the magnitude of the effect of surface order for zero disparity tests. 

At the other unadapted disparities, farthest from the adapted disparities, PSEs were modulated by the distance between the adapting and test disparities, consistent with the previous work (Verstraten et al., 1994). Relative to the adapting disparity of ±7.3 arcmin, the target surfaces at ±14.6 arcmin revealed PSEs biased opposite to the adapted direction by 0.39 deg, and the more distant target surfaces at ±22.0 arcmin revealed PSEs biased opposite to the adapted direction by 0.32 deg. The rather small adaptation effect in these conditions could have been caused by poor fusion of these high binocular disparity planes. A few observers reported that these surfaces sometimes appeared less coherent in terms of their depth. However, these results do not pertain to the purpose of Experiment 2, namely, to assess the surface order effect at adapted disparities. 

At the adapted disparities, the results were combined to estimate the two different effects; those due to absolute disparity and those due to surface depth order. The magnitude of the MAE that was contingent on absolute disparity was obtained by averaging results in the conditions that tested the same absolute disparity, namely, where the target surface was at −7.3 arc min disparity (Figure 3b, CF and CB: crossed disparity) and that where it was at +7.3 disparity (Figure 3b, UF and UB: uncrossed disparity). Psychometric functions for this comparison are plotted in Figure 4b. The difference between PSEs from these two conditions indicates the magnitude of the MAE dependent on absolute disparity, 1.8 deg of tilt averaged across five observers, t(4) = 3.3, p < .05 (Figure 4f). To assess the magnitude of the MAE dependent on surface organization from the same data set, we averaged results from conditions in which the target plane was the same ordinal surface, namely, where the target surface was in front of the other during the test regardless of its absolute disparity (Figure 3b, CF and UF: front) and where it was behind (Figure 3b, CB and UB: back). Psychometric functions for a representative observer and a summary of all observers are shown in Figures 4c and g, respectively. Four of five observers show clear MAE magnitude dependent on surface organization. The PSEs differed by 3.1 deg of tilt averaged across all observers, t(4) = 6.6, p < .005. Although this effect was larger in magnitude than the absolute disparity effect on average (3.1 vs. 1.8), variability between observers was high so that the difference was not statistically significant, t(4) = 1.3, p > .25 at a paired t test. 

An ANOVA that tested all four conditions (CF, UF, CB, and UB) across the five observers revealed a marginal but not statistically significant main effect of absolute disparity, F(1,4) = 4.7, p < .10, and a significant main effect of surface order, F(1,3) = 11, p < .05, but did not reveal any interaction between these two factors (F < 1) indicating that the two effects were independent. 

In the single-disparity condition, surface order is undefined because two groups of dots were presented at the same disparity during the test. The results from this condition are shown by absolute disparity (Figures 4d and h). The amount of absolute disparity-contingent aftereffect in this condition (1.3 deg) was similar as that in the two-surface condition (1.8 deg; Figures 4b and f), indicating that the MAE followed that predicted by adaptation of direction-selective cells sensitive to absolute disparity when only one surface disparity was tested. 

While the PSEs from conditions with the target surface at the crossed disparity clearly differ from those at the uncrossed, both PSEs appear to be shifted to the direction of the front surface for two observers (WYS and HMF in Figure 4f). The PSEs at crossed disparity were substantially above the zero point, but the uncrossed PSEs were close to zero, with the 84% confidence interval overlapping zero. For one observer (AES), however, two PSEs appear symmetric around zero. Variability in results like this was also observed for the surface order effect. Attentional allocation in three-dimensional space may be responsible for this variability among participants. Attention is naturally drawn to stimuli at surfaces in front of fixation (Andersen, 1990; Andersen & Kramer, 1993). Although we used the gap detection task to help observers sustain attention at the fixation plane throughout the adaptation interval, it is possible that this manipulation failed for some observers and they might have been attending more to the front surface during adaptation. 

Experiment 3

In the test stimulus of Experiments 1 and 2, we tested the effects of surface order across various absolute disparities, the distance between each surface and the fixation, but we always maintained the same relative disparity, the distance between the two surfaces. Separate neural responses related to absolute and relative disparities have been reported in the macaque (Cumming & Parker, 1999; Thomas, Cumming, & Parker, 2002) and human brain (Neri, Bridge, & Heeger, 2004). The fixed relative disparity in Experiments 1 and 2 leaves a possibility that mechanisms selective for relative disparities may be causing the effects we have attributed to surface depth order. Experiment 3 was designed to test whether surface order effects can be found across variation on relative disparity. If relative-disparity-selective mechanisms were entirely responsible for the surface order effects in the present study, then different relative disparities at adaptation and test would abolish the effects. We tested conditions where the relative disparity during the test is the same as the one during adaptation and where it was different. 

Methods

Observers

Four observers from Experiment 2 participated in Experiment 3. 

Stimuli

The stimulus during adaptation was identical to that used in Experiment 2. During the test, however, three relative disparities were used, 7.3, 14.6, and 29.2 arcmin. The two surfaces during the test were always located symmetrically relative to the fixation depth. Seven steps of the tilt angles were used for the coherent motion during the test. 

Procedure

Experimental procedure was the same as Experiment 2. 

Results

Psychometric functions were created in the same manner as in Experiment 2 for the three relative disparity conditions. Psychometric functions from the same relative depth condition (14.6 arc min) are shown for observer HMF in Figure 5a. The summary of the results for four observers is shown in Figure 5b. MAE magnitude dependent on surface depth order was calculated by comparing the conditions where the target surface was in front to the conditions where the target surface was behind. The surface order effects averaged across observers for three relative disparities (7.3, 14.6, and 29.2 arc min) were 4.24, 4.70, and 5.05 deg of tilt, respectively. An ANOVA tested all six conditions across the four observers and revealed no main effect of relative disparity, F(2,6) < 1, a significant main effect of surface order, F(1,3) = 23, p < .05, and no interaction, F(2,6) = 3.8, p < .10. These results suggest that surface order affects the MAE across differences of relative disparity. Therefore, the fixed relative disparity is unlikely to fully account for the surface order effects of Experiments 1 and 2. 

Figure 5

 

Results of Experiment 3. (a) Psychometric functions for a representative observer. The proportion of trials that the MAE direction was perceived as opposite to the motion of the front adapting surface is plotted against tilts in the coherent motion in the target surface. Filled squares indicate the condition where the target surface was in front of the other test surface. Open squares indicate the condition where the target surface was behind. The solid and dashed lines are best fitting logistic functions for two conditions. Only 14.6 ft of disparity condition is shown here. (b) The summary of the results for four observers. The x-axis indicates the tilts of the coherent motion in the target surface. Each bar represents each disparity condition (7.3, 14.6, and 29.2 ft).

Discussion

Throughout the three experiments, we have shown that the MAE is sensitive not only to the absolute disparities of adapting and test surfaces, but also to the depth orders of those surfaces. Previous work has shown that unambiguous depth order during motion adaptation biases the perception of rotational direction in a subsequent SfM stimulus at zero disparity with ambiguous direction (Feng & He, 2004; Nawrot & Blake, 1989, 1991a). The present study extends the knowledge of the relationship between motion and stereopsis, by showing that ordinal representation of transparently moving surfaces affects motion coding and interpretation at a broad range of disparities. 

Effects of fixation shift

In Experiments 2 and 3, we measured the performance for the gap detection task to examine the stability of observers' fixations during adaptation. Due to the brief time interval (2 s), we were unable to introduce the same task during the test. It is possible that observers might have shifted fixation from the nonius fixation cross to the point in space between the two test planes, where no stimulus was presented. Such a strategy would have resulted in visual stimulation of the same absolute disparities during the test as during the adaptation. If this was the case, the surface order effects we observed could be simply the result of disparity sensitivity of the MAE. However, it seems very unlikely that the observed surface order effects are solely due to fixation shift for three reasons. First, observers have difficulty sustaining vergence at depths that have no stimuli that can be fused. Shifting fixation to a depth between the two test surfaces in these experiments would have required fixation at a depth where no stimuli were presented. Second, for the test conditions in which the target surface was presented at zero disparity in Experiments 1 and 2, observers would have had to move fixation away from binocularly stable stimuli that were the focus of attention for that trial. Third, surface order effects were observed robustly in Experiment 3 where fixation shift is not likely to occur with the test planes that were symmetrically placed at both sides of fixation. 

Representation of surface depth order

Assuming that the MAE reflects responses of neurons or ensembles of neurons that are sensitive to test stimuli (Hammond, Pomfrett, & Ahmed, 1989; Mather, 1980; van de Grind, van der Smagt, & Verstraten, 2004), the current observations suggest that there are neurons specifically sensitive to surface depth order. A recent study by Nguyenkim and DeAngelis (2003) raises the possibility of a surface-based representation that is rather free from a specific disparity. The authors tested whether MT neurons could signal the three-dimensional orientation of planar surfaces defined by tilt and slant. They reported that tuning for the tilts of surfaces is insensitive to the mean disparity gradient of stimuli. In other words, some cells sensitive to the direction of tilt in the surface were not modulated by the absolute disparity of the tilted surface. Combining with this report, the results of the present study imply a surface representation that may be initially built from segmentation cues such as binocular disparity but then characterizes the surfaces more abstractly, with respect to the context of other surfaces. 

All the manipulations of surface order in the experiments presented here have been created with variation in binocular disparity. As noted in the introduction, many other features can also define surfaces. Cues to the order of surfaces in depth, in particular, can come from multiple pictorial cues such as occlusion, linear perspective, and contrast (O'Shea, Blackburn, & Ono, 1994). In some preliminary observations, we attempted to ascertain whether such two-dimensional cues to depth would produce the same effect of surface depth order on the MAE. We modified the adapting stimulus of Experiment 2 such that two surfaces of moving dots at zero disparity were perceived at different depths due to a pictorial cue. Multiple pictorial cues were tried, including linear perspective, occlusion, and luminance contrast. However, most cues produced an inconsistent perception of depth order over the 60-s adaptation interval, with depth assignment reversing spontaneously every few seconds. The inability to stabilize surface order perception long enough to produce adaptation, however, does not preclude the existence of representation of surface depth order defined by two-dimensional cues. Future investigation is needed to determine the role of two-dimensional cues and the extent to which surface order representations are independent of cue type. 

Relations to existing models

The supposition of such a surface order representation provides an extension of an existing model on perception of depth order in SfM and its interaction with stereopsis (Hiris & Blake, 1996; Nawrot & Blake, 1991b). The model posits units selective for both binocular disparity and direction of motion in a competitive network. At a given disparity plane, units preferring the same direction are facilitatory and units preferring opposite directions are mutually inhibitory. Across separate disparity planes, the units preferring the same direction are inhibitory and those preferring opposite directions are facilitatory. Given a typical two-dimensional SfM rotating cylinder with two opposing motions, units at zero disparity are initially activated. Due to small initial inequalities in activation of these units and their mutual facilitation and inhibition, the activation spreads to the broadly disparity-tuned units and eventually segregates into activation of a pair of units sensitive to separate disparity planes, one responding to each direction of motion. According to this model, perceptual reversals in rotational direction in SfM stimuli occurs because as one pair of units adapts, it will tend to lose the competition to the other pair of units over time. The model also explains the influence of adaptation at particular disparities on the perception of zero disparity stimuli that carries the same motion directions. Adaptation of units responding to motion at particular disparity pairs reduces the likelihood of their response to the subsequent zero disparity stimulus. As a result, the direction of rotation in the zero disparity SfM stimulus is likely to be perceived as opposite to the adapting stimuli (Nawrot & Blake, 1991a). In support of this notion, MT cells have the broad disparity tuning, sometimes including zero disparity, and large receptive fields that may be responsible for the propagation of activity to unspecified disparities (Bradley et al., 1998). 

Extending the model to account for the current observation demands sensitivity to surface order as well as disparity. In our experiments, depth order during adaptation influenced motion perception at various disparities during the test. The activity of adapted units sensitive to binocular disparity cannot fully account for the current observations. Instead, the results suggest that the model should include separate units sensitive to surface order. Binocular disparity between opposite motions during adaptation may influence not only the units that are selective to the motion directions at given disparities, but also the ordinal representation of surfaces irrespective of their absolute disparities. Surface order units could comprise their own competitive network independently of disparity-sensitive units, or the two networks could be closely connected. Addition of surface order units would bolster the model's ability to transfer activation from zero disparity to nonzero disparity stimuli, and vice versa (Nawrot & Blake, 1991a, 1991b, 1993), because surface order units would respond to both types of stimuli. 

The neural site of surface order representation

Although the brain area that serves the ordinal representation of surfaces remains to be discovered, it is clear that neurons responsible for this representation must be sensitive to and affect motion perception. Considering the high correlation between MT responses for depth order perception and those for disparity-defined depth order (Bradley et al., 1998) and the fact that some surface representation in MT is insensitive to disparity (Nguyenkim & DeAngelis, 2003), MT could have neurons that are sensitive to surface order and are responsible for the observed MAEs. Continued exposure to the opposing motion signals at different disparities would adapt both disparity-selective and surface-order-sensitive neurons to the specific motion directions at each plane. A following test with two transparent planes at different disparities would not elicit much net aftereffect from disparity-tuned cells but would drive responses of cells sensitive to surface order. Neurophysiological experiments to date have not tested for surface order sensitivity across changes in binocular disparity, leaving open the possibility that some subset of MT cells may be coding surface order. 

Although the cumulative evidence makes MT the most likely candidate, it is possible that other sites along the motion pathway are also involved in the representation of surface order. Another candidate for this is the visual area MST that receives most information directly from MT. Neurons in this area also exhibit direction selectivity but respond to larger visual field than those in MT and to complex motion such as rotation (Saito et al., 1986). Related to surface representation, MST shows similar direction selectivity that is dependent on disparity as MT (Roy, Komatsu, & Wurtz, 1992). MST neurons that were sensitive to either near or far stimuli showed changes in the preferred direction of stimulus motion as the disparity of the stimulus was changed. Upadhyay, Page, and Duffy (2000) investigated surface representation in MST during pursuit. When local motion components were identical, some MST cells show larger responses and stronger heading selectivity to multiple depth-plane motion stimuli than to single-plane ones. To speculate, MT may be primarily involved in representing ordinal information of surfaces, and this information is fed into MST responses for the purpose of guiding self-movement and action on visual objects. 

Implications on feature binding

The results of the present study also provide an insight on the stages where feature binding occurs. The observed MAE that is contingent on the absolute disparity may be explained by double-duty neurons (McCollough, 1965) that may be tuned simultaneously to disparity and motion. The effects of surface depth order cannot be explained by such neurons but instead require a higher-order stage of surface representation. In this stage, visual surfaces are the units to group visual features. Features may be organized based on surface order. The current study implies more than one stage for feature binding. First, visual features may be bound together based on local information such as spatiotemporal co-localization. This is when double-duty neurons may take part. Next, features may be grouped based on surfaces. Surface segmentation could be based on any number of visual cues, such as edge junctions, lightness discontinuities, or binocular disparity. In the case of transparent surfaces, the ordinal information of each surface can be formed by the aid of low-level features such as binocular disparity, although the relative order of surfaces is the basis of the representation. Then, location of the features belonging to each surface is encoded as being associated with “front” or “back,” instead of a particular binocular disparity. The adaptation dependent on depth order in the present study may occur at this stage. 

Conclusions

Considering the goal of human visual system to represent objects and surfaces in the world, how visual features should be combined to generate three-dimensional percepts is an important question. Objects often consist of multiple surfaces. The order of surfaces in depth is important information for understanding the three-dimensional structure of an object. Representing ordinal information between surfaces may be a necessary step between surface formation and perception of three-dimensional objects. The present study implies that this process appears to occur after initial uptake of visual features but early enough to influence on sensory adaptation in the case of motion. While the current investigation provides evidence for the existence of a surface order representation in the human motion system, further investigation on surface order effects is warranted, using different cue types, before a detailed theory can be proposed. 

Acknowledgments

This research was supported by the National Institutes of Health Grants EY-014984 to AES. We thank Randolph Blake and Duje Tadin for helpful comments on the manuscript. 

Commercial relationships: none. 

Corresponding author: Wonyeong Sohn. 

Email: [e-mail]. 

Address: 301 Wilson Hall, 111 21st Avenue South, Nashville, TN 37203.