Temporal characteristics of depth perception from motion parallax

Abstract

Abstract

Abstract:

Abstract  Temporal characteristics of depth perception from motion parallax were examined by modulating parallax intermittently while observers moved their head side to side. In Experiment 1, parallax of a fixed value was introduced only for the central 1/6 to 5/6 portion of each component head movement. It was found that the perceived depth was proportional to the temporal average of parallax-specified depth. In addition, observers did not notice any abrupt temporal change of depth. In Experiment 2, parallax was increased or decreased once per trial either at the center or the end of one of the component head movements, and observers judged the direction of depth change. Again, observers did not notice any abrupt change of depth. The percentage of correct responses was almost constant for large change amplitudes. Reaction times to the change were over 1 s even for the largest changes, and it increased for smaller change amplitudes. These results indicate that the mechanism for depth from parallax has a configuration similar to that proposed for structure from motion, and that it involves a temporal integration process with a relatively long time-constant.

Introduction

When observers move their head while watching two points placed at different distances from them, a relative motion between the two points is produced on the retina. This relative motion caused by observers' self motion is called motion parallax, and it generates compelling subjective impressions of depth (Rogers & Graham, 1979). The main objective of this paper is to examine the temporal characteristics of depth perception from motion parallax. 

Relative motion on the retina is produced not only by an observer's motion but also by the object's motion. An observer sees relative motion when objects move with respect to the observer. The visual system has to resolve this ambiguity; that is, it has to sort out to two components, one caused by the observers' self motion with respect to the objects, and the other caused by the objects' motion relative to the observer. The visual system resolves this problem mainly by relying on knowledge about the observer's own motion. If self motion is known and can be related to retinal motion, the motion components caused by self motion and by object motion can be segregated. Depth structure can also be perceived even when objects are moving without an observer's self motion, a situation called “structure from motion” (SfM). In SfM, the motion component caused by an object's global motion such as rotation or translation has to be segregated from the relative motions caused by depth structure. To perform this segregation, the visual system supposedly relies heavily on an assumption that objects are rigid (rigidity assumption). In addition, it has been proposed that it is necessary to accumulate motion input for a relatively long period to obtain SfM information (e.g., Hildreth, Grzywacz, Adelson, & Inada, 1990; Ullman, 1984). Considering the similarity between motion parallax and SfM, it is plausible that depth from motion parallax also requires relatively long temporal accumulation. However, although significant knowledge has been accumulated on temporal characteristics of SfM, the temporal characteristics of depth perception from motion parallax have not been studied systematically. Therefore, we begin by reviewing SfM studies relevant to a consideration of the temporal characteristics of motion parallax. 

Treue, Husain, and Andersen (1991) manipulated the life time of dots for an SfM display representing a rotating cylinder and found a life-time threshold of 50 to 100 ms, which is similar to the threshold for velocity estimation. They claimed that this similarity suggests that velocity measurements are used to process SfM. At the same time, in an experiment in which the duration was manipulated, they found that the performance in distinguishing cylindrical and scrambled displays increased as a function of presentation time up to 1000 ms when the life time was 100 ms. They also found that the reaction time for detecting an SfM cylinder was about 1000 ms. In a similar study, Eby (1992) reported that although the depth from SfM could be perceived for durations as short as 100 ms, it was underestimated for such short durations. Perceived depth increased as the stimulus duration increased up to 500 to 1000 ms, at which point it reached a plateau. More recently, Domini, Vuong, and Caudek (2002) reported that perceived depth from SfM at a given moment is affected by stimulus changes presented up to 1 s prior to the moment. All these results suggest that although low-level motion parameters on which SfM perception relies are detected rather quickly, several such measurements have to be integrated over time to obtain reliable SfM perception. Based on these results, Caudek, Domini, and Di Luca (2002) proposed a model for SfM perception with two stages. In their model, it is assumed that the first stage calculates depth from local motion distribution within 150 ms, and the second stage calculates global consistency by integrating the output of the first stage over a period of approximately 1 s. 

It is plausible that the mechanism for motion parallax has a similar second stage structure to that for SfM, and thus has similar temporal characteristics. For depth detection from motion parallax, local motions and relative motions between adjacent local motions have to be detected before depth is reconstructed. This motion detection stage, probably together with some pooling process, might correspond to the first stage proposed for depth from SfM. In addition, for parallax depth, motion is not visible even when depth is clearly visible if parallax is very small (e.g., Ono & Ujike, 2005). These detected motion signals then have to be bound with information about self-motion to be converted to depth signals. 

Recently, Nawrot and Stroyan (2012) reported that reaction time for depth-order discrimination was as short as 32 ms. However, in this study, they did not employ head movement. They examined depth perception when observers moved only their eyes, and they demonstrated that it was possible to discriminate depth polarity (near/far) with a very short stimulus duration with this paradigm. They did not examine if amount of depth can be perceived or not. Their results, nonetheless, could be understood as evidence for a two-stage mechanism for depth from motion parallax. However, as already mentioned, the knowledge about temporal characteristics of depth perception from motion parallax is very limited, and more knowledge should be accumulated before we discuss the configuration of mechanisms in detail. Therefore, the main objective of this study was to examine the temporal characteristics of depth perception from motion parallax in more general terms. 

In the first experiment, we measured the perceived amplitude of a one-dimensional sinusoidal depth modulation induced by stimulus movement that was yoked to observers' side-to-side head movement. The motion parallax was presented, that is, the stimulus motion was yoked to the head movement for only a part of each component head movement. The yoke-ratio, which was the ratio of head to stimulus movements, geometrically corresponded to the depicted, or parallax-specified depth. Therefore, in other words, the parallax-specified depth was turned on and off in this experiment. The percentage of the yoked period and the yoke-ratio were systematically manipulated. If depth is detected by a mechanism with short temporal integration (i.e., high temporal resolution), perceived depth should appear to change and should be independent of the percentage of parallax presentation; however, if depth is determined by a slow integrating mechanism, perceived depth should be some sort of average when the percentage is lower. In the second experiment, to examine the response characteristics of parallax detection, we introduced an abrupt change in parallax during head movements to see if observers could detect the change. 

Experiment 1

The perceived depth from motion parallax was measured while having the observers move their head reciprocally. Motion parallax was presented for only a part of each component head movement, and the ratio of this moving period to the whole length of each component movement was manipulated. 

Methods

Participants

Four observers including one of authors (KH) participated in the experiment. All observers had normal or corrected to normal vision. All observers were naïve as to the purpose of the experiment. 

Apparatus

Stimuli were generated by an Apple Power Macintosh G3 and displayed on a gamma-corrected 21-inch monitor (Eizo FlexScan E76D) with a refresh rate of 75 Hz. The display resolution was 1024 × 768 pixels. Stimuli were viewed monocularly from a distance of 76 cm. Observers put their head on a chin rest that translated parallel to the CRT screen on a rail. The chin rest guided observers' head movements. Length of the rail was 20 cm. The position of chin rest was encoded by a potentiometer similar to that used by Rogers and Graham (1979). The value of the potentiometer was measured by an A/D converter (National Instruments DAQ Card1200) connected to the potentiometer at 1,000 signals per second. The head position was reflected to visual stimulus within 12 ms after acquisition. 

Stimulus

The stimulus consisted of two rectangular random-dot fields each subtending 3.5° × 5.0° in visual angle. They were placed horizontally with a separation of 7.0° between the centers (Figure 1a). Each dot subtended 4 (horizontal) × 10 (vertical) pixels, which corresponded to 7 × 17 arc min in visual angle. The dot density was 20%. Rectangular dots were used to realize the highest horizontal resolution within the limit of performance of the computer used for stimulus presentation. The luminance of each pixel within a dot was modulated to specify the dot position with subpixel accuracy by using a method similar to that used by Mitsudo and Ono (2007). 

Figure 1

 

(a) Schematic illustration of stimulus presented in Experiment 1. (b) Dots in the stimuli are moved when observer's head was in “concomitant interval.”

The dots moved horizontally when observers moved their head. The velocity of dots was modulated sinusoidally along the vertical axis as shown in Figure 1a. The spatial frequency of the sinusoidal function was 0.4 cycles/deg and the modulation had two cycles within a stimulus field. The peak amplitude of modulation was varied in three steps: 0.14, 0.20, or 0.27 cm per 1 cm of head displacement. These values correspond to 40, 60, and 80 arc min shifts for every 6.5 cm of head movement (equivalent disparity, Rogers & Graham, 1982). 

Throughout each trial, the movement of dots in one of the stimulus fields (reference stimulus) were yoked with the head movement of observers, while dots in the other field (target stimulus) were yoked only when the observers' head, i.e., chin rest, was within the central part of the rail with a predefined length (concomitant interval). The dots stayed stationary on the screen outside of this concomitant interval. The stationary stimulus instead of blank field was presented during the off period so that we could examine the effect of transformation of parallax-specified depth. The concomitant interval was varied in five steps: ±1.67, ±3.33, ±5.00, ±6.66, and ±8.33 cm from the center of the rail. The movement range of the chin rest in total was 20 cm. Thus, the ratios of concomitant interval to total head movement (concomitant ratios) were 1/6, 2/6, 3/6, 4/6, and 5/6 (Figure 1b). 

Procedure

Observers reciprocally moved their head over the full length of the rail. They were instructed to synchronize head movement to a tone that sounded every 1 s. That is, the mean velocity of head movement was ideally 20 cm/s, which exceeded full-depth threshold of head velocity for motion parallax (Ono & Ujike, 2005). In each trial, observers were instructed to compare the perceived depth in the two stimuli over at least three reciprocal head movements and to judge which stimulus had deeper corrugation. There was no fixation point, and observers could alternate their fixation between the two stimuli. The judgment that the two stimuli had equal depth was also allowed. The observers responded by pressing one of three keys designated for each judgment. The response was ignored and the trial was repeated when they responded before they completed three cycles of head movement. 

There were 15 different types of target stimuli: 3 parallax magnitudes × 5 concomitant ratios. Point of subject equality (PSE) about the depth for each target stimulus was determined by varying the parallax-specified depth (hereafter just parallax) of reference stimulus with a modified method of limit. There were ascending and descending series. In an ascending series, the parallax of reference stimulus was set to zero at the beginning. It was increased by 2.5% of the peak parallax of target when observers responded that target depth was larger than reference depth and by 1.25% when observers judged both stimuli had equal depth. The series was terminated when observers judged the reference had larger depth, and the reference parallax for the penultimate trial was designated as the PSE value for the trial. In a descending series, the initial value was set to twice the peak value of the target. Parallax was decreased by 2.5% of the peak target value when observers judged the reference had larger depth and by 1.25% when they judged both had equal depth. The series was terminated when the reference was judged to have smaller depth, and the reference parallax for the penultimate trial was designated as the PSE value for the trial. Both ascending and descending series were conducted twice for each observer for each stimulus combination. Therefore, four measurements were obtained from each observer for each stimulus, and the average of the four measurements was designated as the observer's PSE for the stimulus. 

Results and discussion

In Figure 2, the averaged PSE values for each parallax amplitude were plotted as functions of concomitant ratio of target stimuli. As can be seen from the graph, PSE values increased monotonically as concomitant ratio increased for all parallax conditions. The dotted lines in the graph indicate the mean parallax-specified depth over the stimulus presentation period. The concomitant interval is defined by distance of head movement. However, observers were instructed to move their head back and forth every 1 s, thus each component motion from one end to the other took approximately 1 s. Therefore, the line represents the predicted depth when averaging of parallax-specified depth over presentation period was assumed. The data from the experiment overlapped almost exactly with the predicted values in all amplitude conditions. 

Figure 2

 

Averaged PSE value as functions of concomitant ratio of target stimuli in Experiment 1. The dotted lines indicate the predicted depth values if linear averaging according to concomitant ratios was assumed. Error bars indicate the 95% confidence intervals of the mean bootstrap results by resampling data sets for 10,000 times.

A two-way ANOVA revealed significant main effects of both parallax amplitude, F(2, 39) = 197.50, p < 0.01, and concomitant ratio, F(4, 78) = 322.37, p < 0.01. The interaction between the two factors was also significant, F(8, 312) = 12.19, p < 0.01. 

These results indicate that perceived depth is not determined by the value of parallax when it is on. Instead, perceived depth is integrated over a certain temporal window. These results therefore suggest that depth from parallax is generated by a slow integration mechanism similar to the second stage proposed for SfM perception (Caudek et al., 2002). In this experiment we employed free viewing for easier comparison between the two stimuli. However, several studies have reported that eye movement affects the depth perception from motion parallax (Freeman & Fowler, 2000; Mitsudo & Ono, 2007; Naji & Freeman, 2004; Nawrot, 2003a, 2003b; Nawrot & Joyce, 2006). To examine whether the present results were affected by fixation, we conducted an additional experiment in which careful fixation was required. For this experiment, we presented the two stimulus fields side by side with an 1° gap in between and presented a fixation marker (0.5° × 0.5°) at the center of the gap. We asked observers to maintain fixation very carefully. We examined only the largest parallax condition of the main experiment (80 arc min shifts for every 6.5 cm). Since different set of observers participated in this control experiment, we also obtained the data for nonfixation condition in this experiment. The results are shown in Figure 3. It was found that PSE varied in proportion to the product of depicted depth and concomitant ratio, that is, the findings of the main experiment were not affected by viewing condition. 

Figure 3

 

Averaged PSE value as functions of concomitant ratio of target stimuli from the control experiment where careful fixation was required together with data from nonfixation condition obtained together with the control condition. Filled symbols indicate fixation and open symbols are nonfixation condition. Error bars are standard errors across observers.

There was an additional support for the averaging of parallax with a relatively long integration period. There were abrupt changes of depicted depth at the beginning and end of each concomitant period. We asked observers if they noticed any change in depth corresponding to these stimulus changes, and all observers reported that they did not. Together, these results indicate that the depth from motion parallax is detected by a mechanism with low temporal resolution. 

However, in this experiment, observers were not asked to detect abrupt changes in each trial. They were asked to report if they notice any abrupt change in depth during the experiment when all trials were finished. To examine whether it is possible to detect abrupt change of depth from motion parallax, in the next experiment, we measured the reaction time, together with the percentage of correct decisions, for an abrupt change of parallax that occurred only once at the center of a component head movement in which the head speed is approximately maximal. We compared the results from this experiment to those from trials in which the change was introduced at one of the end points. It should be noted that the parallax speed actually changed at the moment for the central condition, but it caused no stimulus change at the moment for the end condition, since the head was stationary and there was no stimulus movement at the moment. 

Experiment 2

The detection performance and reaction time for an abrupt change of parallax was measured in this experiment. The parallax change was introduced either at the center or one of the end points of reciprocal motion. 

Method

Participants

One of authors (KH) and three naïve observers who did not participate in the previous experiment participated in this experiment. All observers had normal or corrected to normal vision. 

Stimulus and procedure

Stimuli and apparatus were the same as for the previous experiment except for the points described as follows. In this experiment, only one stimulus was presented at the center of the monitor screen. Observers moved their head continuously throughout each trial. The stimulus first appeared when the observer's head reached one of the end points of head movement range. The parallax was fixed to 60 arc min at the beginning of the trial and it changed abruptly at a particular point described below. Observers were asked to press one of the two keys designated for depth increase or decrease when they detected a change in depth. The percentage of correct detection (increase/decrease) and the reaction time for the response after the stimulus change was recorded. They were instructed to keep moving their head until they detected the depth change, and the trial was terminated when they did not detect any change within 10 cycles of head movement. For these terminated trials, the response was designated as incorrect and the reaction time was not obtained. 

The amplitude of change was varied in 11 steps between 100% and −100% of the initial parallax value (60 arc min equivalent parallax). Each change value was repeated four times for each observer. The order of presentation was randomized. The parallax amplitude was changed at the point when observers had moved their heads 2, 4, 6, 2.25, 4.25, or 6.25 cycles after the trial was started. In 2.25-, 4.25-, and 6.25-cycle conditions, the parallax change occurred at the center of head movement where the speed of head movement was approximately maximal. Thus, the task of observers in these conditions was to detect a dynamic depth change. In contrast, the head was stationary at the 2-, 4-, or 6-cycle points since these were at the end of reciprocal motion, thus, there was no dynamic depth change at these points. Observers had to detect the difference between the two depth amplitudes presented before and after the change. 

Results and discussion

In Figures 4 and 5, correct rates and reaction times averaged over observers for different change timings are plotted as functions of change amplitude. In these figures, the data from 2-, 4-, and 6-cycle conditions (end conditions) are collapsed, and data from 2.25-, 4.25-, and 6.25-cycle conditions (center conditions) are collapsed. As for reaction times (Figure 5), only the data from trials with correct responses were used. 

Figure 4

 

Percentage of correct decisions as functions of amplitude change in Experiment 2. Filled squares indicate “end condition” and open circles are “central condition.” Error bars indicate the 95% confidence intervals of the mean bootstrap results by resampling data sets for 10,000 times.

Figure 5

 

Reaction time as functions of amplitude change in Experiment 2. Filled squares indicate “end condition” and open circles are “central condition.” Error bars indicate the 95% confidence intervals of the mean bootstrap results by resampling data sets for 10,000 times.

A two-way ANOVA on arcsine-transformed correct ratios revealed a significant main effect of change amplitude, F (9, 27) = 17.59, p < 0.01, but that both the main effect of change position, F(1, 3) = 0.33, ns, as well as interaction, F(9, 27) = 0.52, ns, were not significant. As for reaction times, a two-way nonorthogonal ANOVA (Overall & Spiegel, 1969; Overall, Spiegel, & Cohen, 1975) indicated that the main effects of change amplitude, F(9, 558) = 10.16, p < 0.01) and position, F(1, 558) = 11.12, p < 0.01, are both significant, but interaction, F(9, 558) = 0.83, ns, is not significant. 

The percentage of correct responses (Figure 4) is fairly high and constant except for the lowest change amplitude, and there was no statistically significant difference between the end and center conditions. That is, whether the change of parallax occurred when the head was stationary or moving does not affect the ability to detect the change. As for subjective impressions, none of the observers reported that they noticed any abrupt change. Instead, most of them reported that although they did not notice any sudden change, there was a difference in depth between two separate spells within each trial. These results indicate that observers can reliably notice the depth changes from parallax modulation. However, detection does not seem to rely on the sudden or transient changes. Rather, it depends on comparisons of sustained levels of perceived depth. 

Reaction times significantly increased as the change amplitude increased for both end and central conditions (Figure 5). These results indicate that the detection of change in depth from parallax requires a longer time when the magnitude of change in parallax is smaller. Reaction times increased slowly as the magnitude of the change decreased from 100% to 40%, and they were longest at 20%. The long reaction times at 20%, in addition to very low correct rates, suggest the changes were very difficult to detect at small amplitudes. 

The shortest reaction time found in this experiment was about 1 s, obtained for the highest amplitude when it was presented at the center. Dzhafarov, Sekuler, and Allik (1993) reported that simple reaction time for velocity change was within a range of 200 to 300 ms. As for stereoscopic depth, Norcia & Tyler (1984) reported that motion in depth defined by binocular disparity is perceived up to 6 Hz. Compared with these values, reaction times found in the present study are several times longer. These are similar to what had been found with SfM. Treue et al. (1991) reported that although the life-time threshold for random-dot SfM was 50 to 100 ms, reaction times for SfM perception were longer than 1 s. Therefore, reaction times for depth from parallax agree with those from SfM. These relatively long reaction times, together with the insensitivity to change positions of correct rate, suggest that perception of depth from motion parallax does not reflect instantaneous values of parallax. Rather, the instantaneous values have to be integrated over a certain time period that is probably close to 1 s. In other words, when the visual system detects changes in depth from motion parallax, it compares sustained depth values obtained by integrating instantaneous values from different time intervals instead of directly detecting transient changes themselves. This is probably why the observers in Experiment 1 did not see any depth changes during the period of each component head movement. 

In contrast to the percentage of correct responses, the effect of change position was found to be significant for reaction time. Reaction times were shorter when the change was at the center compared with at the end of the head movements. However, this positional effect found here is just apparent, since the abrupt change in parallax-specified depth (yoke-ratio) occurs at different timings between central and end condition. The change occurred when the speed was near maximum for the central condition, thus the change in parallax-specified depth was supposedly reflected immediately to the perceived depth. However, for the end condition, the change occurred when the head was stationary. Thus, in this condition, the change in parallax-specified depth is reflected in the perceived depth only when the head speed reached to a certain level. In short, the difference in timing of change most likely accounts for the positional effect for reaction time found in this experiment. 

General discussion

In Experiment 1 in which the yoking, or the parallax-specified depth was presented intermittently, observers perceived the depth approximately equal to the averaged parallax-specified depth over the head movement period or interval. In Experiment 2, in which the sudden change of parallax-specified depth was introduced at the center of head movements, the reaction time for the change was over 1 s. These results suggest that the mechanism detecting depth from motion parallax is essentially a low-pass filter and the processing requires a longer time. 

It has been reported that the mechanism for SfM without head movement has a slow temporal characteristic. It requires approximately 1 s of integration time (Domini et al., 2002). The present results are quite similar to the results of Domini et al., and this similarity suggests that the mechanisms for depth perception from motion parallax and those for SfM share some part in common. Based on their results, Domini et al. (2002) proposed a two-stage model in which the depth is first detected by a mechanism with a short integration time, and then the detection results are integrated by the second stage with a longer integration time. The existence of integration stage seems compatible with the proposition that sequential comparison is necessary for disambiguation in shape from motion (e.g., Hildreth et al., 1990; Ullman, 1984). 

Processes for depth perception from motion parallax and SfM are both involved in calculating depth structure from retinal motion. In this sense they are similar, thus, it is possible that the mechanism for depth from motion parallax has a two-stage structure similar to that of SfM. As mentioned in the Introduction, Nawrot and Stroyan (2012) recently reported that depth discrimination from parallax combined with eye movements was possible within a time period as short as 32 ms. These results suggest the existence of a stage with short integration time, but there was no clue in the present results to suggest the necessity of a stage with short integration time. The difference might have come from the difference in experimental procedure such as the involvement of head movements vs. eye movements, measurements of reaction time vs. manipulation of stimulus duration, or parallax values. Tasks are different, too. Nawrot and Stroyan (2012) asked observers to discriminate depth polarity; whereas, we asked observers to judge the amount of depth (Experiment 1) or to detect sudden change in depth (Experiment 2). In addition, Rogers (2012) reported that pursuit eye movements are neither necessary nor sufficient to disambiguate depth order. 

The detection of motion parallax is used not only to obtain depth information, but also to segregate motion per se and motion caused by depth and this ultimately contributes to the stability of the visual world (Ono & Steinbach, 1990; Ono & Ujike, 2005; Shimono, Tam, & Ono, 2007). Structure from motion detects relative motions within rigid objects caused by motion and interprets them as depth signals. It prevents unnecessary perception of deformation of moving objects and contributes to stability. Motion parallax also achieves this function. The relatively long integration time may be advantageous to fulfilling these functions, considering that we and our heads are always moving in an unpredictable fashion. 

Acknowledgments

Supported by a Grant-in-Aid for Scientific Research (21330167) to TS from the Ministry of Education, Culture, Sports, Science and Technology. We are grateful to Professor Gayle Sato for her assistance in text editing. 

Commercial relationships: none. 

Corresponding author: Takao Sato. 

Email: [e-mail] 

Address: Department of Psychology, Graduate School of Humanities and Sociology, The University of Tokyo, Bunkyo-ku, Tokyo, Japan.