Prior depth information can bias motion perception

Abstract

Previous studies of the motion aperture problem have shown that the direction of grating motion can be biased by using binocular disparity to designate borders of the aperture as intrinsic (belonging to the grating) or extrinsic (resulting from occlusion of the grating). Observers report motion in the direction of the extrinsic border, as if the grating was extended and moving underneath an occluding surface. Here we investigate whether prior information about depth ordering, given by structure-from-motion, can bias the perceived motion direction of a subsequent moving grating in a similar manner. We presented an aperture stimulus that rotated about its vertical and horizontal axes, revealing the depth relationships (intrinsic and extrinsic) of the aperture borders. The grating then translated within the aperture and observers reported the direction of perceived motion. The test stimulus contained no information about the depth ordering of the scene. We found that observers’ reported motion shifted toward the direction of the occluding edges, consistent with the intrinsic-extrinsic border predictions. These results indicate that prior scene information, not just depth information explicitly defined in the test stimulus, is used to help solve the motion aperture problem.

Introduction

The motion aperture problem refers to the observation that the direction of a translating contour within an aperture is ambiguous; the motion is consistent with an infinite number of motion directions (Wallach, 1935; Marr & Ullman, 1981; Adelson & Movshon, 1982). This is an intriguing problem for investigation because motion detection units in early visual cortical areas are often characterized as having relatively small receptive fields corresponding to a limited region of visual space: an aperture on the world (Hubel & Wiesel, 1962). Because physiology indicates that the aperture problem is a fundamental one to solve, the question of how the brain resolves locally ambiguous motion information is of some importance. This ambiguous motion can, in theory, be correctly interpreted by integrating motion estimates from several local detectors. Many models of visual motion detection, therefore, consist of two stages: one where local information is analyzed and a second stage where local detectors are combined (Adelson & Movshon, 1982; Welch, 1989; Weiss, Simoncelli, & Adelson, 1998). 

In this two-stage framework, the “barber pole” display of Wallach (1935; translated by Wuerger, Shapley, & Rubin, 1996) provides evidence that the shape of the aperture could influence the perceived direction of motion. Specifically, elongated apertures bias motion perception along the direction of the longest side. These results have been explained as the propagation of motion signals generated by the grating line terminators along the aperture border. These border terminators provide unambiguous motion information and disambiguate the local motion signals of the inner regions of the stimulus after integration across local apertures. Shimojo, Silverman, and Nakayama (1989) described two possible ways of classifying aperture borders in a real world situation, as either being intrinsic (belonging to the grating) or extrinsic (resulting from occlusion). By adding stereoscopic disparity information to the borders, Shimojo et al. were able to bias the perception of motion in the barber pole stimulus. When disparity was added to the border of the aperture so that the grating appeared in front of the aperture and thus all borders were intrinsic, the barber pole effect followed the aperture shape observations of Wallach (1935). When disparity was added to the border of the aperture so that the grating appeared behind the aperture and thus all the borders were extrinsic, the influence of the aperture shape was eliminated. In this configuration, the grating can be interpreted as being completed at a depth behind the occluder, following a process similar to amodal completion (Kanizsa, 1979). These results imply that real-world occlusion conditions can influence the perception of motion direction. 

Duncan, Albright, and Stoner (2000) carried out an additional test of the intrinsic and extrinsic border classification, pitting intrinsic borders against extrinsic borders in a stimulus configuration that they termed the “barber-diamond” stimulus. In the barber-diamond stimulus, two borders are oriented at 45° and two at −45° to the grating orientation. Disparity was used to designate two alternate borders as intrinsic (behind the grating) or extrinsic (in front). The bordering panels were composed of random dot textures. With this stimulus it was found that motion perception was consistently seen in the direction parallel to the intrinsic border, following the intrinsic-extrinsic predictions. Electrophysiological recordings in monkey area MT conformed with the human psychophysical data, finding cells that responded maximally to depth-motion conditions consistent with the perceived direction of surface motion under an occluder. 

Several studies have indicated that the results of Shimojo et al. (1989) and Duncan et al. (2000) might be explained as a result of half-occlusions, or unpaired monocular regions in the stimulus, introduced by the local displacement of the interocular positions of contour terminators (Anderson, 1999; Castet & Wuerger, 1997; Castet, Charton, & Dufour, 1999). Monocular occlusion cues in stimuli that have no disparity have also been shown to influence the perception in an aperture stimulus (Liden & Mingolla, 1998). 

The present study seeks to further investigate the effect of intrinsic and extrinsic border classification by using a different method to provide depth information to the observer. We introduced monocular structure-from-motion information to designate the aperture borders as being intrinsic or extrinsic prior to grating motion. The use of prior structure-from-motion information is interesting for a couple of reasons: it eliminates both monocular and binocular cues to depth ordering during the test phase of the stimulus, and it allows testing of the contribution of prior information to a relatively simple motion stimulus. In a second experiment, we tested the strength of this prior information using the barber pole stimulus. In a third experiment, we explored the time course of the influence of the prior information. 

General methods

Observers

Six observers took part in all three experiments: the three authors and three naïve observers. All subjects had normal or corrected-to-normal vision. 

Stimuli and procedure

Computer-generated images were presented on a 21″ Sony Trinitron monitor. Observers viewed the stimuli monocularly, wearing an eye patch over the left eye. Observers were positioned in a headrest apparatus, situated 114 cm from the monitor. Each trial began with a button press by the observer. A stationary stimulus (9° wide by 9° tall, depicted in Figure 1a) was presented for 250 ms. The stimulus consisted of nine, equally sized panels, where the center panel contained a square-wave grating oriented at 45° from vertical. After the button press, the stimulus began to rotate about the vertical axis centered on the vertical midline of the image. The relative motion of the grating and border regions clearly determined the depth ordering within the stimulus. For nonzero depth conditions, this rotation revealed six of the eight non-grating panels (e.g., three upper and three lower) to be in front of the grating and the other two panels at the same depth as the grating. The depth between these front panels and the grating was 1.5 cm. The grating and the two remaining panels were revealed to be part of a continuous surface, partially occluded by the panels presented in front. The stimulus rotated 22.5° in one direction, then back through fronto-parallel to 22.5° in the other direction, and then back to zero. The rotation was then repeated around a horizontal axis centered at the horizontal midline of the image. At the end of the rotation, which lasted approximately 3 s, the stimulus remained stationary for 100 ms. 

Figure 1

 

Depiction of a trial. (a). The stimulus rotated about the vertical axis, then the horizontal axis; following a 100-msec static period, the inner grating translated rightward (b) for 100 ms. The observer then used designated keys to move an arrow to indicate the perceived direction of motion (c).

Experiment 1: The effect of prior scene information on motion perception

Methods

Five conditions were used in Experiment 1 (Figure 2, right column). In the first two conditions, the top and bottom borders of the stimulus were revealed by structure-from-motion information to be in front of the grating, whereas the left and right borders were at the same depth as the grating. In the first condition (i), the grating was elongated under the near panels so that when the stimulus rotated, the grating was revealed as having a rectangular outline extending vertically under the near panels. In the second depth condition (ii), the grating was revealed to have a square outline, so that as the stimulus rotated, all four borders of the grating were visible. In a third condition (iii), the structure-from-motion information specified zero depth for all of the panels. The fourth (iv) and fifth (v) conditions were mirror identical to conditions one and two, except that the left and right panels were presented in front and thus the rectangular grating outline of condition (v) extended horizontally rather than vertically. Each of the five conditions was presented 16 times to each observer in a single block of trials. 

Figure 2

 

Results of Experiment 1. Plotted in the histogram is the proportion of responses for each response direction in each of the five conditions tested. The conditions from top to bottom are top and bottom panels in front (elongated grating), top and bottom panels in front (square grating), zero depth, left and right panels in front (square grating), and left and right panels in front (elongated grating). The reported values for each condition are the means of the distribution.

Results

Figure 2 shows the results of the first experiment. Each panel depicts a different stimulus condition. The data are averaged across the five observers, and plotted in a histogram with each bar representing the responses for the indicated direction as a proportion of responses to all directions. The mean value (μ) of the distribution is shown on each graph, where 0° corresponds to rightward motion and 90° corresponds to upward motion. It can be seen that the prior designation of aperture borders significantly influenced subjects’ subsequent motion perception. In general, when the top and bottom panels were presented in front, observers perceived the direction of motion to be more vertical than in the zero depth condition. When the left and right panels were in front, observers perceived the motion moving more often to the right. This influence is consistent with the predicted effect of the intrinsic-extrinsic border designations. This pattern was consistent across all six observers. The individual subject data is presented in Table 1. 

Table 1

 

Results of the six individual observers for Experiments 1–3. All values are in degrees (°) orientation, where 0 is rightward motion and 90 is upward motion. For Experiment 2, only the two extreme aspect ratio values are shown. For Experiment 3, only the shortest and longest time delay values are shown. T-B = top and bottom panels in front, L-R = left and right panels in front, ZD = zero depth condition, ex. = extended grating condition, and sq. = square grating condition.

	Experiment 1					Experiment 2						Experiment 3
						asp. ratio = 1.0			asp. ratio = 0.6			delay = 0.1 s			delay = 4.0 s
Subject	T-B ex.	T-B sq.	ZD	L-R sq.	L-R ex.	T-B	ZD	L-R	T-B	ZD	L-R	T-B	ZD	L-R	T-B	ZD	L-R
bc	46	51	45	32	31	45	45	38	71	66	58	49	45	41	50	45	43
eg	64	47	46	23	15	43	33	15	73	68	64	38	45	0	38	47	14
fs	72	73	45	13	11	79	45	19	86	60	24	77	47	22	71	45	21
ml	63	54	46	33	16	64	45	21	75	64	52	47	47	36	47	49	45
mw	80	84	44	11	7	64	43	24	86	60	0	86	45	4	77	45	17
wa	43	48	43	21	28	51	34	13	75	66	54	57	41	22	44	40	45

We conducted a two-way ANOVA to test whether the observed effects were significant and to determine whether a significant difference existed between the square grating and elongated grating conditions. The analysis revealed a significant main effect of stimulus configuration (top-bottom in front, zero depth, left-right in front) [F(1, 10) = 18.50, p = .006], (p values adjusted for heterogeneity, Huynh-Feldt). We tested all nonzero depth configurations against the zero depth condition as well as testing the effect of grating shape. Three of the four nonzero depth conditions were significantly different from the zero depth condition, with the top-bottom in front, the square grating condition being the exception [F(1,5) = 5.25, p = .071]. The interaction of shape and configuration was not significant [F(2,10) < 1.0], and we did not find significant contrast effects between the square and elongated gratings within each depth configuration. This result implies that the amodal continuation of the grating underneath the occluder was not as important as the depth relationship revealed by the structure-from-motion. If the amodal continuation had been important in determining boundary ownership, then the reported motion direction in the elongated grating condition should have been more upward in the top and bottom panel arrangement and more rightward in the right and left panel arrangement. 

The main result of Experiment 1 is that prior structure-from-motion information is sufficient to bias perception of a subsequently presented stimulus. We have also shown that neither binocular disparity nor the unpaired monocular regions of binocular stimuli are necessary to influence perception of motion direction. Additionally, we have shown that concurrent definition of boundary ownership in the test stimulus is not necessary to change the perceived motion direction. But how important is the prior depth ordering information? One way to test the influence of this prior information is to see how it interacts with other monocular information to bias perception. In Experiment 2, we used the same structure-from-motion stimulus as in the previous experiment, combined with the barber pole effect. 

Experiment 2: Intrinsic-extrinsic borders and the barber pole effect

The barber pole effect, as described previously, is a compelling stimulus where the perception of motion is strongly influenced by the shape of the aperture containing the motion. As the aspect ratio of the aperture is changed from 1, motion perception becomes biased toward the orientation of the longer side of the rectangle. In this experiment, we change the aspect ratio (width/height) of the grating in the stimulus, while providing the same depth ordering information of the boundaries and grating as in Experiment 1. 

Of interest is the way in which the two factors interact. The effect of aperture shape and prior depth ordering information may simply combine linearly, such that we would observe a constant difference between the depth conditions at all aspect ratios. It will then be possible to see how much change in aspect ratio is required to null the effects of the prior depth ordering information. However, the results of Shimojo et al. (1989) suggest that a different pattern of results might be expected. In their experiment, when disparity was added to make all boundaries of a grating extrinsic, the effect of aperture shape disappeared. This suggests that in their experiment, boundaries that have been classified as extrinsic had little or no effect on motion perception. If our prior depth ordering information works in the same way as disparity in the above experiment, we would expect that lengthening the extrinsic edges of our grating (Figure 3, right column) would not influence perceived motion direction. 

Figure 3

 

Results of Experiment 2. Plotted is the proportion of responses for each response direction in each of the 18 conditions tested. Each column represents a different depth condition, from left to right: top and bottom panels in front, zero depth, and left and right panels in front. Each row represents a different aspect ratio of the grating. The reported values for each condition are the means of the distribution.

Methods

Three depth conditions were used in Experiment 2: (1) left and right panels in front, (2) zero depth, and (3) top and bottom panels in front. for each depth condition, six different aspect ratios were used: 1.0, 0.89, 0.79, 0.71, 0.65, and 0.6. Aspect ratios less than 1 correspond to a grating that is taller than it is wide. The grating extended underneath the occluding boundary, so that two boundaries were intrinsic and two were extrinsic. Five trials of each of the 18 conditions were run per observer. 

Results

The averaged results for Experiment 2 are shown in Figure 3 and are presented as response proportions as in Experiment 1. The middle column represents the zero depth condition. Here the classic aperture shape observations of Wallach (1935) are replicated. Lower rows represent vertically elongated grating conditions, and it is apparent that the effect of decreasing aspect ratio in our stimulus is to bias perceived motion direction toward the vertical (90°). The left column represents the condition where the top and bottom panels were presented in front; the right column represents the condition where the left and right panels were presented in front. Comparing the data across the columns, the shift in observers’ responses shows the effect of the prior depth information. For our observers, changing the prior depth information has approximately the same magnitude of effect at all aspect ratios. An interesting case is shown in the bottom right panel, where the aspect ratio is 0.6. In this panel, the mean value of 45.3 deg indicates that the effects of the aspect ratio and the depth ordering effectively nulled each other. Individual observer data are provided in Table 1. 

In a two-way ANOVA we tested whether the observed differences between depth configurations were significant and whether a significant difference existed between aspect ratio conditions. There was a significant main effect of configuration [F(2, 10) = 18.95, p = 0.006] and of aspect ratio [F(1, 5) = 14.53, p = 0.012]. There was no significant interaction [F(10,50) = 1.19, p = .345] (p values adjusted for heterogeneity, Huynh-Feldt). 

Changing the aspect ratio of the boundary did bias the perceived motion direction in all three depth conditions. In other words, the assignment of boundaries as extrinsic did not eliminate their influence on motion perception. Shimojo et al. (1989) found that extrinsic boundaries eliminated the effect of aperture shape. However, in our study, increasing the length of an extrinsic boundary did result in motion being perceived more toward the direction parallel to that boundary. Our results suggest that prior depth ordering information may not work in an “all or none” fashion. Previous aperture motion studies where boundaries have been designated with disparity (Rubin & Hochstein, 1993) or monocular cues (Liden & Mingolla, 1998) have suggested that extrinsic boundaries work on a continuum, where terminators can be classified as more or less extrinsic. Similarly, we suggest that the prior information reduces rather than eliminates the motion information provided by terminators at extrinsic boundaries. 

Experiment 3: Time scale of the influence of prior information

In this experiment, we investigated the temporal extent of the effect of prior information. Of interest is whether the effect of the prior depth information can continue to influence motion perception several seconds after its presentation. To determine this, we varied the time between the stimulus rotation and the grating translation. 

Methods

Three different depth stimuli were used in Experiment 3: (1) top — bottom panels in front, elongated grating, (2) zero depth, and (3) right — left panels in front, elongated grating. The procedure was similar to the first two experiments, with the following change: After the rotation of the stimulus, the display was static for a delay period, after which the grating translated for 100 ms. Four time delays were used: 0.1, 0.5, 2.0, and 4.0 s. Each delay was presented 8 times to each subject in one trial block in random order. 

Results

The results of Experiment 3 are shown in Figure 4, where it can be seen by comparing the mean values of the three conditions for the last temporal delay condition (4 s) that a difference still exists between the three depth conditions. This data indicate that prior information has a lasting effect on motion perception. The individual subject data are presented in Table 1. 

Figure 4

 

Plotted is the proportion of responses for each response direction in each of the five conditions tested in Experiment 3. Each column represents a different depth condition, from left to right: top and bottom panels in front, zero depth, and left and right panels in front. Each row represents a different time delay between stimulus rotation and grating translation. The reported values for each condition are the means of the distribution.

A two-way ANOVA showed a significant effect of depth configuration [F(2, 10) = 7.46, p = .013], but not of temporal delay [F(3, 15) = 2.23, p = .175]. However, the interaction between configuration and temporal delay was significant [F(6, 30) = 4.58, p = .036] (p values adjusted for heterogeneity, Huynh-Feldt). The fact that the 0.1-s delay condition was not significantly different from the 4-s delay condition indicates that the effect of the prior information was still present at delays up to 4 s. 

Discussion

We have shown that depth information made explicit before stimulus presentation but not contained in the test stimulus can influence the perception of a motion stimulus. We have shown that this information is robust, having an influence when shown before the presentation of a barber pole stimulus, as well as lasting through a 4-s delay period. Next we comment on additional details found in the data. 

Non-stereo information and influence on motion

As discussed earlier, stereo depth information has been shown to influence motion perception (i.e., Shimojo et al., 1989). A portion of the influence of stereo information results from the presence of binocularly unpaired regions in the image (Anderson, 1999; Castet & Wuerger, 1997; Castet, Charton, & Dufour, 1999). 

Both motion and stereo rely on multiple views of a scene: two monocular views for stereo and two successive temporal images for motion. It has been suggested that a motion analogue of the unpaired stereopsis case exists, which can also influence motion perception (Anderson & Julesz, 1995; Anderson & Sinha, 1997). Temporally unpaired segments of motion aperture stimuli from successive frames lead to the accretion and deletion of contour segments, which can influence perception. Although any biasing effect of the accretion and deletion of stimulus elements does not exist in our test stimulus, the possibility does arise in the prior rotation phase of the stimulus. For example, during one of the directions of rotation (around a horizontal axis for the display where the top and bottom panels were near), the panels that are presented in front obscure part of the grating while revealing another previously unseen part of the grating. Could this be sufficient to influence perception? We conducted a control condition where we presented one rotation per trial (either about the horizontal or vertical axis) for one depth condition (top and bottom panels in front). Thus, while both rotations revealed the depth ordering in the stimulus, only one rotation led to accretion and deletion of contour elements. The results for three observers (not depicted) showed no difference between the two rotation conditions, indicating that the prior information was primarily responsible for our effects. 

Another possible influence on motion perception in our stimulus is nonvisual cognitive functioning. To test this, we ran a control study where we replaced the structure-from-motion phase of the stimulus with instruction to the observer to “imagine” the depth ordering of the stimulus while they viewed a zero-depth stimulus [as in Figure 1a (i)]. The grating then translated and observers responded with their perceived direction of motion. Three observers showed no effect of depth ordering information, indicating the importance of the structure-from-motion information in the main experiment. 

History-dependent motion effects

The data from Experiment 3 show that the depth ordering information presented before aperture motion is retained at some level for several seconds to influence perception. History-dependent motion effects have been previously noted with motion stimuli. Joseph and Nakayama (1999) showed that prior experience before occlusion could affect the amodal continuation of a surface behind an occluder, and, therefore, the direction of motion perception in an apparent motion task. They reported that after a delay of 2 s, four of their six subjects showed an effect of the prior information. While their study relied on amodal continuation of objects behind an occluder, in our experiment this was not a necessity. In fact, as reported in Experiment 1, we saw no statistically significant difference between the two different grating shape conditions, one where the subject had information that might be used to amodally continue the grating under the occluder and another where all four grating boundaries were clearly presented. Instead of showing amodal continuation over time, our study shows the effect of depth ordering information over time on motion perception. 

The possible involvement of feedback from higher visual areas

Our time course data from Experiment 3 indicate that our results are unlikely to be explained by traditional feed-forward neural architecture. It would need to be assumed that cells corresponding to regions with extrinsic terminators modify their output for up to 4 s before the onset of stimulus motion. Perhaps it is more plausible to explain our results as a consequence of feedback mechanisms where lower motion processing areas are influenced by the output of higher visual areas relating to occlusion relationships and scene configuration. Previous work investigating amodal completion shows that this might be reasonable, as several studies (Alais, van der Smagt, van den Berg, & van de Grind, 1998; Duncan et al., 2000; Lorenceau & Alais 2001; Shimojo et al., 1989) have shown that amodal completion can influence depth-motion interactions. As described by van der Smagt and Stoner (2002), amodal representation may introduce additional motion signals that could influence perception when combined with motion signals from the visible areas of the surface. We imagine the prior information from our stimulus as working in a similar manner by introducing motion signals at extrinsic boundaries due to the combination of the unambiguous prior depth information and the ambiguous information contained in the test stimulus. The additional unambiguous information is considered to be the result of feedback from scene and object segmentation and recognition areas in cortical regions higher in the processing stream than area MT. The feedback conceptualization is consistent with the current wealth of literature indicating the role of feedback mechanisms in visual processing (e.g., Lee, Mumford, Romero, & Lamme, 1998). 

Acknowledgments

This research was funded by NSF-IRFP (EWG), the Wellcome Trust (WJA), and EPSRC, UK (ML). Part of the work was presented at ECVP-Glasgow in August 2002. 

Commercial relationships: none. 

Corresponding author: Erich W Graf. 

Email: . 

Address: Department of Psychology, University of Southampton, Southampton, UK.