Optimal combination of form and motion cues in human heading perception

Abstract

We examined what role motion-streak-like form information plays in heading perception. We presented observers with an integrated form and motion display in which random-dot pairs in a 3D cloud were oriented toward one direction on the screen (the form FOE) to form a radial Glass pattern while moving in a different direction in depth (the motion FOE). Observers' heading judgments were strongly biased toward the form FOE direction (weight: 0.78), and this bias decreased with the reduction of the salience of the global form structure in the Glass pattern. At the local level, the orientation of dot pairs in the Glass pattern can affect their perceived motion direction, leading to a shift of the perceived motion FOE direction in optic flow. However, this shift accounted for about half of the total bias. Using the measurements of the shifted motion FOE and the perceived form FOE directions, we found that at the global level, an optimal combination of these two cues could accurately predict the heading bias observed for the integrated display. Our findings support the claim that motion streaks are effective cues for self-motion perception, and humans make optimal use of both form and motion cues for heading perception.

Introduction

How do people perceive the instantaneous direction of their self-motion (heading)? Gibson (1950) has proposed that humans use the visual motion of the image of the environment during locomotion (optic flow) to determine self-motion. When traveling on a straight path with no eye, head, or body rotation (pure translation), the focus of expansion (FOE) of the resulting radially expanding optic-flow pattern indicates one's heading. Psychophysically, it has been shown that humans can locate the FOE in optic flow to estimate their heading within 1°–2° of visual angle during simulated translation (e.g., Crowell & Banks, 1996; Li, Peli, & Warren, 2002; Warren, Morris, & Kalish, 1988). 

Although the FOE is defined by the expanding global motion in one 2D velocity field of optic flow, it can also be implied by the global form structure of “motion streaks” (Figure 1a), which arise through temporal integration of motion of environmental points beyond a single velocity field (e.g., Burr, 2000; Geisler, 1999). Research over the last three decades has focused almost exclusively on what motion cues in optic flow humans use to perceive heading but ignored the potential influence of such form cues. 

Figure 1

 

(a) An illustration of motion streaks in a temporally integrated optic-flow pattern for translation through an outdoor scene. (b) A radial Glass pattern in which all dot pairs are oriented toward a common location (the blue “×”), an FOE implied by the global form structure of the radial Glass pattern.

It has been proposed that humans process motion and form using two separate visual pathways that originate from the primary visual cortex (V1) and project either dorsally to the parietal cortex for motion processing (also called “where” pathway), or ventrally to the inferotemporal cortex for form processing (also called “what” pathway, e.g., DeYoe & Van Essen, 1988; Mishkin, Ungerleider, & Macko, 1983). Area MST in the primate visual cortex receives inputs from area MT along the dorsal pathway (Maunsell & Newsome, 1987) and has been widely proposed as the site for the analysis of global motion in optic flow (e.g., Britten, 2008; Duffy & Wurtz, 1991). Human brain-imaging studies have revealed that a human homologue of primate MST is the MT complex (MT+; Huk, Dougherty, & Heeger, 2002; Morrone et al., 2000; Wall, Lingau, Ashida, & Smith, 2008). Neurons in area MST and MT+ have large receptive fields and exhibit a large extent of spatial pooling (Burr, Morrone, & Vaina, 1998; Tanaka & Saito, 1989), whereas MT and especially V1 neurons have much smaller receptive fields (Felleman & Kaas, 1984; Hubel & Wiesel, 1974, 1977). 

Despite some initial neuropsychological evidence from brain-damaged patients showing double dissociation of the dorsal and ventral streams (e.g., Benson & Greenberg, 1969; Goodale & Milner, 1992; Zihl, von Cramon, & Mai, 1983), many more recent studies have suggested that motion and form processing are closely linked (see Kourtzi, Krekelberg, & van Wezel, 2008, for a review). For example, motion can help people perceive form that could not be seen from a static display, such as the classical demonstration of the kinetic depth effect (Wallach & O'Connell, 1953) and biological motion (Johansson, 1973). Conversely, form can also dictate motion perception. For instance, people usually do not have any problem in extracting motion from the static “speed lines” (i.e., motion streaks) depicted in cartoons, and these speed lines have been shown to bias people's perceived motion direction (Burr, 2000; Burr & Ross, 2002; Geisler, 1999; Ross, 2004). 

Given the interaction between form and motion processing, it is natural to think that a form-implied FOE would integrate with the motion-defined FOE in optic flow to influence human heading perception. In a natural setting, these form and motion FOEs are often congruent. No study so far has systematically investigated whether and to what extent form information can affect human heading perception. Furthermore, if such influence exists, what could be the possible neural mechanisms responsible for this form and motion interaction? 

Due to the fact that Glass patterns (the moiré patterns of superimposed random dots, Glass, 1969) contain motion-streak-like form information (Burr, 2000; Ross, Badcock, & Hayes, 2000) and show resemblance to temporally integrated optic-flow patterns (Barlow & Olshausen, 2004), in the current study, we investigated how motion-streak-like form information affects heading perception by presenting observers with displays that contained a form-implied FOE from a radial Glass pattern (Figure 1b). Specifically, in our displays, the dot pairs of the Glass pattern were randomly positioned in a 3D cloud but coherently oriented toward one direction on the screen (the form FOE) while moving in a different direction in depth (the motion FOE, Figure 2a). This setup allowed us to put the form and motion information in conflict to investigate whether form information influences heading perception. We then systematically varied the number of signal dot pairs that were oriented to convey the form FOE in the radial Glass pattern to examine how the brain combines the form and motion signals for heading perception. 

Figure 2

 

A schematic illustration of the visual stimuli used in Experiment 1. (a) Integrated display. A 3D cloud composed of 100 white dot pairs all oriented toward the form-implied FOE at 10° (the blue “×”) to form a radial Glass pattern. Red arrowhead lines represent the velocity vectors of the centroids of the dot pairs in the flow field, generated by translation toward the center of the display, i.e., the motion FOE at 0° (the red “+”). (b) Non-integrated display. A static Glass pattern consisting of 100 white dot pairs all oriented toward the form FOE at 10° (the blue “×”) is superimposed on a 3D cloud consisting of 100 white dots all moving away from the motion FOE at 0° (the red “+”).

The detection of global form structure in a radial Glass pattern involves two stages. The first stage requires the observer to perform local grouping to find the orientation of the dot pairs, and the second stage requires the observer to perform global summation of the local orientation features for the extraction of global shape (Dakin & Bex, 2001; Wilson & Wilkinson, 1998; Wilson, Wilkinson, & Asaad, 1997). The two stages have been postulated to happen at cortical area V1 (Smith, Bair, & Movshon, 2002; Wilson & Wilkinson, 1998) and area V4 along the ventral pathway (Gallant, Braun, & Van Essen, 1993; Gallant, Connor, Rakshit, Lewis, & Van Essen, 1996; Ostwald, Lam, Li, & Kourtzi, 2008). As a result, the influence of the form information in the radial Glass pattern on heading perception can happen through local and global interactions between form and motion signals. At the local level, line orientation and motion direction detectors in area V1 have been shown to function together to determine local motion direction (Burr & Ross, 2002; Geisler, 1999; Geisler, Albrecht, Crane, & Stern, 2001), and pattern type neurons in area MT have been shown to respond to static bars oriented nearly parallel to their preferred motion direction (Albright, 1984). By means of such local level mechanisms, the perceived motion direction of each dot pair in the Glass pattern can be biased toward its orientation, and the global pooling of these biased local motion signals can shift the perceived heading (i.e., the motion FOE in optic flow) without involving the extraction of the form FOE in the Glass pattern. At the global level, findings from human brain-imaging studies have shown that the dorsal stream can be activated by global form as well as global motion (Braddick, O'Brien, Wattam-Bell, Atkinson, & Turner, 2000; Krekelberg, Vatakis, & Kourtzi, 2005). The form FOE conveyed by the global structure in the Glass pattern may thus directly affect neuronal responses to the motion FOE and thereby influence heading perception. 

To separate the local effect of the orientation of each dot pair on its perceived motion direction from the global effect of the form FOE on heading perception, we generated a stimulus that simulated the effect of local form–motion interactions on the flow field and tested observers' heading perception. If the visual system combines form and motion cues to optimize heading perception, following the Bayesian framework for cue integration of multiple sensory inputs (Knill & Pouget, 2004; Mamassian, Landy, & Maloney, 2002; Yuille & Bülthoff, 1996), we expect that heading performance would be predicted by a statistically optimal rule that applies weights to the form-implied FOE in the radial Glass pattern and the motion FOE in the radial flow field (which could be shifted due to local form and motion interactions) proportional to the reciprocal of the variances associated with their separate measurements. Consistent with our hypotheses, the findings from our current study show that human heading perception is not solely determined by the motion FOE in optic flow. Motion-streak-like form information is an effective cue for the perception of self-motion, and humans make optimal use of both such form and motion information in optic flow for heading perception. 

Experiment 1: Does form information influence heading perception?

In this experiment, we presented observers with an integrated form and motion display (Figure 2a and Movie 1) in which the random-dot pairs in a 3D cloud were oriented toward one direction on the screen (the form FOE) to form a radial Glass pattern while moving in a different direction in depth (the motion FOE) and a non-integrated display (Figure 2b and Movie 2) in which a static radial Glass pattern was superimposed on a translational optic-flow field. For both displays, on each trial, the motion FOE in optic flow randomly varied between −8° and 8° along the azimuth, and the conflict angle between the form-implied FOE in the Glass pattern and the motion FOE was set at 0°, ±5°, or ±10°. Observers were asked to use a mouse to indicate their perceived heading at the end of the trial. 

 

Movie 1

 

A schematic video of the display for the integrated condition. The centroids of all dot pairs in the 3D cloud move away from the center of the screen (the 0° motion FOE), while all dot pairs are oriented toward a form FOE at 10° to the right to form a radial Glass pattern. The dot size and the dot-pair offset are optimized for on-screen viewing.

 

Movie 2

 

A schematic video of the display for the non-integrated condition. All dots in the 3D cloud move away from the center of the screen (the 0° motion FOE), while the static dot pairs in the radial Glass pattern are oriented toward a form FOE at 10° to the right. The dot size and the dot-pair offset are optimized for on-screen viewing.

The logic is given as follows. If the motion-streak-like form information in the radial Glass pattern influences heading perception, observers' heading judgment should be biased toward the form FOE. For the integrated display, the influence of the form information on heading perception can happen at both the global level and the local level as form and motion information are coupled in each moving dot pair. For the non-integrated display, the radial Glass pattern was static and superimposed on a radial optic-flow pattern. As the oriented dot pairs in the Glass pattern did not move and were not coupled with the motion signals from optic flow, any shift of the perceived heading toward the form FOE is likely due to global interactions between the form and motion FOEs. 

Methods

Participants

Eight undergraduates (all naive as to the specific goals of the study; five males, three females) between the ages of 19 and 23 participated in the experiment at the University of Hong Kong. All had normal or corrected-to-normal vision. 

Visual stimuli

Two display conditions were tested. In the integrated condition (Movie 1), the display simulated observer translation at 3 m/s through a 3D cloud composed of 100 white dot pairs. The dot pairs were placed against a gray background (luminance contrast: 75%) and had a constant 1° center-to-center separation on the screen. The size of the dots in the dot pairs was also kept constant at 0.5° diameter. The 100 dot pairs were randomly placed in the depth range of 1.1–5 m so that the same number of dot pairs originated from each distance in depth. For each frame, all dot pairs were oriented toward one direction on the screen to form a radial Glass pattern. The stimulus thus offered two independently generated FOEs: the form-implied FOE given by the orientation of the dot pairs (blue “+” inFigure 2a) and the motion FOE given by the motion of the centroids of the dot pairs (red “×” in Figure 2a). It should be noted that while the centroid of each dot pair underwent rigid 3D motion, the individual dots did not as the dot pairs rotated in each frame to remain oriented toward the form-implied FOE (see Movie 1). However, as the center-to-center distance of the dot pairs was 1°, the amount of non-rigid motion introduced by the dot-pair rotation in the flow field was minimal and barely noticeable. In the non-integrated condition (Movie 2), the 100 white dot pairs in the 3D cloud were replaced with 100 white dots. A static radial Glass pattern consisting of 100 white dot pairs was then superimposed on the random-dot cloud display. The stimulus thus also offered two independent FOEs: the form FOE given by the orientation of the dot pairs in the static Glass pattern (blue “+” inFigure 2b) and the motion FOE given by the motion of the dots in the 3D cloud (red “×” in Figure 2b). 

For both display conditions, on each trial, the simulated observer translation direction (i.e., the motion FOE direction) was randomly varied between −8° and 8° along the azimuth of the center of the screen (negative values to the left and positive values to the right of the center of the display). The form FOE direction had five offset values (0°, ±5°, and ±10°) from the motion FOE direction (negative values representing the form FOE to the left of the motion FOE and positive values to the right). The maximum eccentricity of the form FOE away from the center of the screen was thus 18°. Throughout a trial, the total number of motion signals in the 3D cloud was kept constant, i.e., if a certain number of dots or dot pairs moved outside of the field of view (70.1° H × 70.1° V) in one frame, the same number of dots or dot pairs was regenerated in that frame with an algorithm that maintained the depth layout of the cloud. 

The displays were programmed in MATLAB using the Psychophysics Toolbox 3 (Brainard, 1997; Pelli, 1997) on a Dell Precision Workstation 670n with an NVIDIA Quadro FX1800 graphics card at the frame rate of 60 Hz. The displays were then rear-projected on a large screen with an Epson EMP-9300 LCD projector (native resolution: 1400 × 1050 pixels) at a 60-Hz refresh rate. Observers viewed the displays monocularly with their dominant eye from a chin rest at a distance of 56.5 cm from the large screen. 

Procedure

At the beginning of each trial, a cross appeared at the center of a blank display. Observers were asked to fixate on the cross and then click a mouse button to start the trial. The fixation cross then disappeared and the stimulus was displayed for 1 s. At the end of the trial, a horizontal line appeared at the center of the display, and observers were asked to use a mouse-controlled probe to indicate their perceived heading (i.e., their perceived instantaneous direction of translation) along the horizontal line. The final angle between the observer's perceived heading and the motion FOE direction, defined as the heading bias, was recorded. 

Each observer completed a total of 400 experimental trials (40 trials × 5 form FOE offset directions × 2 display conditions). The trials were blocked by the display conditions and were randomized within each block. The order of conditions was counterbalanced between observers. Before the commencement of the experiment, we screened observers using 30 training trials with displays simulating translation through a cloud composed of 100 random dots. The display duration was 2 s and the simulated translation speed was 1 m/s or 10 m/s, randomly interleaved. Observers were asked to use the mouse to indicate their perceived heading at the end of the trial. We allowed the observers who displayed stable heading performance (SD < 5°) on the training trials to participate in the experiment. 

Results and discussion

The mean heading bias averaged across eight observers is plotted against the form FOE offset in Figure 3. A horizontal line (dashed line at 0° heading bias) indicates that the perceived heading is solely determined by the motion FOE in optic flow, whereas a positive slope of one (dotted line) indicates that the perceived heading is solely determined by the form FOE in the Glass pattern. For the integrated display condition, individual regressions show that heading bias increases linearly with form FOE offset (r ² > 0.80, p < 0.003). For the non-integrated display condition, individual regressions show that heading bias increases linearly with form FOE offset for all but one observer (r ² > 0.82, p < 0.002). A paired t-test on the slopes of the regression lines reveals that the mean slope is significantly higher for the integrated than for the non-integrated display condition (0.78 vs. 0.27; t(7) = 10.61, p < 0.0001). A separate t-test also reveals that the mean slope for the non-integrated display condition is significantly higher than zero (t(7) = 3.96, p < 0.01). 

Figure 3

 

Mean heading bias against form FOE offset for the integrated and non-integrated display conditions. The dashed horizontal line indicates heading performance relying solely on the motion FOE direction, and the dotted line indicates heading performance relying solely on the form FOE direction. Error bars are SEs across eight observers.

These results indicate that in the presence of an FOE implied by the motion-streak-like form information in the radial Glass pattern, observers' heading perception is no longer determined solely by the motion FOE in optic flow. The significant linear trend with non-unity slopes for both the integrated and non-integrated displays suggests that the perceived heading is a weighted average of the form and the motion FOE directions. Despite the conflicting form and motion cues, observers consistently reported that they perceived one heading in the display when questioned at the end. 

Comparing the slopes from the integrated and non-integrated display conditions, the bias in heading estimates is about three times larger for the integrated than for the non-integrated condition. As the form FOE direction in the two display conditions is the same, the larger bias observed for the integrated display could be due to local interactions between form and motion signals, i.e., the orientation of dot pairs in the integrated display affects their perceived motion direction, leading to a shift of the motion FOE direction in optic flow and thus the perceived heading. On the other hand, in the non-integrated display where the Glass pattern is statically superimposed on the translating 3D random-dot cloud display, the perceptual salience of the form FOE conveyed by the global form structure of the static Glass pattern may decrease due to the motion “pop-out” effect. Thus, the larger heading bias observed for the integrated display could also be due to the increased form FOE salience. In the next experiment, by systematically varying the percentage of signal dot pairs oriented to convey the global structure of the form-implied FOE in the integrated display, we examined the extent to which the global form coherence of the radial Glass pattern affects the form-induced heading bias. 

Experiment 2: The effect of global form coherence

In this experiment, we varied the global form coherence of the radial Glass pattern by manipulating the number of dot pairs oriented toward the form-implied FOE. Four levels of global form coherence were tested, in which 87.5%, 75%, 50%, or 25% of the 100 dot pairs in the 3D cloud were oriented toward the form FOE, and the rest of the dot pairs were randomly orientated at least 15° away from the form FOE (Figure 4). As the centroids of all dot pairs in the 3D cloud moved outward from a single motion FOE, the signal strength of the global motion in optic flow was kept constant at 100%. Since the global form structure of the form FOE in the Glass pattern becomes less visible with increasing noise (Wilson & Wilkinson, 1998), we expect that the heading bias toward the form FOE would decrease as the global form coherence of the Glass pattern decreases. 

Figure 4

 

Sample displays in which (a) 87.5%, (b) 75%, (c) 50%, and (d) 25% of the 100 dot pairs in the 3D cloud are oriented toward the form FOE at 10° (the blue “×”), and the rest of the dot pairs were randomly orientated at least 15° away from the form FOE.

Methods

Participants

Twelve undergraduates and staff (all naive as to the specific goals of the study; 7 males, 5 females) between the ages of 18 and 28 participated in the experiment at the University of Hong Kong. All had normal or corrected-to-normal vision. 

Visual stimuli

In this experiment, the 3D cloud was composed of 100 white dot pairs as in the integrated display condition in Experiment 1, except that either 87.5%, 75%, 50%, or 25% of the dot pairs had orientations consistent with the form FOE at ±10°, ±5°, or 0° away from the motion FOE direction (Figure 4). The rest of the dot pairs were randomly orientated at least 15° away from the form FOE to reduce the signal strength of the global form structure in the radial Glass pattern. Regardless of their orientation, all dot pairs in the 3D cloud moved outward from a motion FOE randomly selected in the range of −8° to 8° along the azimuth of the center of the display as in Experiment 1. 

Procedure

Each observer completed a total of 400 trials (20 trials × 5 form FOE offset directions × 4 form coherence levels) in a fully randomized order. The testing and screening procedures were the same as in Experiment 1. 

Results and discussion

The mean heading bias averaged across twelve observers is plotted against the form FOE offset for the four global form coherence levels in Figure 5a. To depict heading performance change starting from the 100% global form coherence level, the mean heading bias for the integrated display condition from Experiment 1 is also plotted. Again, a horizontal line (dashed line at 0°) indicates that the observers' perceived heading is solely determined by the motion FOE in optic flow, whereas a positive slope indicates that the perceived heading is biased toward the form FOE in the Glass pattern. The slopes of heading bias at all five global form coherence levels appear to be positive. More importantly, the slope decreases with the reduction of the global form coherence. 

Figure 5

 

(a) Mean heading bias against form FOE offset for the five global form coherence levels. Data at the 100% coherence level are from the integrated display condition in Experiment 1. (b) Mean slope of heading bias against form coherence level. Error bars are SEs across 12 observers for the data from Experiment 2 and across eight observers for the data from Experiment 1.

To better depict the change in heading bias as a function of form FOE offset for the five global form coherence levels, Figure 5b plots the mean slope of heading bias as a function of form coherence level. A repeated-measures ANOVA on the slopes for the 87.5%, 75%, 50%, and 25% form coherence levels reveals that the main effect of global form coherence is significant (F(3,33) = 72.08, p ≪ 0.0001). Separate t-tests show that while the slope for the 100% form coherence level is marginally not different from that for the 87.5% form coherence level (t(18) = 2.02, p = 0.059), it is significantly larger than the slope for the 75% form coherence level (t(18) = 2.89, p < 0.01). 

As expected, the slope increases with global form coherence, indicating that observers' heading judgment was more biased toward the form FOE as its salience increased. Separate t-tests reveal that the slopes for all four form coherence levels are significantly larger than zero (t(11) > 3.44, p < 0.01), indicating that even at the lowest form coherence level (25%), observers still did not solely rely on the motion FOE in optic flow for heading judgment. 

In summary, the above results indicate that the heading bias toward the form FOE depends on the global form coherence of the radial Glass pattern. This supports the claim that observers use the form FOE conveyed by the global form structure of the Glass pattern in addition to the motion FOE in the radial flow for heading estimation. The caveat is that as we reduced the global form coherence of the Glass pattern, we might have also reduced the effect of local interactions between form and motion signals on heading perception. Thus, in the next experiment, we examined the extent to which heading perception is affected by the local interaction between form and motion signals in the integrated display. The goal was to first separate the global and local form effects and then explore how humans combine form and motion cues for the final heading estimate. 

Experiment 3: Optimal integration of form and motion cues

When multiple sources of information are available, how does the brain combine them to reach optimal perception? The general framework for cue integration of multiple sensory inputs proposes that the brain behaves like a maximum likelihood estimator by weighting each cue in inverse proportion to its reliability (Knill & Pouget, 2004; Mamassian et al., 2002; Yuille & Bülthoff, 1996). This framework of sensory cue combination follows Bayesian probability inference and has been supported by a wide range of studies on perception, some within (Hillis, Watt, Landy, & Banks, 2004; Jacobs, 1999; Knill & Saunders, 2003; Landy & Kojima, 2001) and some across sensory modalities (Alais & Burr, 2004; Ernst & Banks, 2002; Shams, Ma, & Beierholm, 2005; van Beers, Sittig, & Denier van der Gon, 1999). A recent study has also revealed neural correlates in macaque MST for combining visual and vestibular cues for heading discrimination in this statistically optimal manner (Gu, Angelaki, & DeAngelis, 2008). 

In the current experiment, we investigated whether the visual system combines form and motion cues for heading perception in a similar statistically optimal way. The first step was to find out the extent to which the bias in observers' heading judgment can be attributed to the shift of the perceived motion FOE direction in optic flow due to local form and motion interactions. It has been reported that for translational motion, the perceived motion direction of dot pairs in a Glass pattern is deflected about halfway toward their orientation for form–motion conflict angles between 0° and about 40° (Krekelberg, Dannenberg, Hoffmann, Bremmer, & Ross, 2003; Ross, 2004). For our integrated display at the 100% global form coherence level, 95% of the angles between the orientations of dot pairs and their motion directions are within this range. The motion of most dot pairs would thus undergo deflection about halfway toward their orientation causing a shift of the perceived motion FOE in the optic-flow field (see Figure 6a). 

Figure 6

 

Sample velocity fields of the visual stimuli used in Experiment 3. Red lines indicate the original dot motion directions in the 3D cloud, and the blue lines indicate the orientation signals used for deflecting the dot motion directions at the (a) 100% and (b) 50% global form coherence levels. Black lines represent the deflected dot motion presented in the display. As a result, the motion FOE in the flow field (the black “+”) is in between the simulated observer translation direction (the red “+”) and the form FOE of the deflecting orientation signals (the blue “×”).

To measure the amount of shift of the motion FOE in the deflected flow field, we generated a display that simulated observer translation through a 3D random-dot cloud. We used the data from Ross (2004) to deflect the motion direction of each dot as if it had an orientation signal from a dot pair coupled with it. Like the orientation signals provided by the dot pairs in the integrated display, the orientation signals used for deflecting the dot motions in the 3D random-dot cloud were oriented toward one direction on the screen to form a radial structure. The bias in observers' heading judgment with this optic-flow stimulus would thus reflect the shift of the motion FOE due to the pooling of local interactions between form and motion signals. 

To investigate whether a weighted linear combination of the form FOE in the radial Glass pattern and the shifted motion FOE in optic flow explains the bias in observers' heading judgment for the integrated display, we varied the global form coherence of the orientation signals in the 3D random-dot cloud display to match the 100%, 75%, 50%, and 25% coherence levels of the integrated displays in the two previous experiments. In a separate session, we asked observers to indicate the center of a static radial Glass pattern with the above four global form coherence levels and measured the associated variance as an estimate of the reliability of the form FOE cue in the radial Glass pattern. If the brain combines the form-implied and the motion-defined FOE cues in a statistically optimal manner for heading perception, we expect that the pattern of heading bias observed for the integrated displays at different form coherence levels in the two previous experiments would be predicted by a weighted linear combination of the perceived form and motion FOE directions, with their weights in inverse proportion to the variances associated with their separate measurements. 

Methods

Participants

Six students and staff (4 naive as to the specific goals of the study; all male) between the ages of 23 and 29 participated in the experiment at the University of Hong Kong. All had normal or corrected-to-normal vision. 

Visual stimuli

A standard 3D random-dot cloud display was used except that for each frame, the motion direction of each dot on the screen was deflected as if it had an orientation signal from a dot pair coupled with it. The deflected motion direction (α) was given by 

$\alpha ={\theta }_m+s_f*\phi ,$

(1)

where θ _m is the dot's original motion direction, φ is the angle between the dot's original motion direction and the deflecting orientation signal (θ _f − θ _m ), and s _f is a deflection strength factor. Based on the data from the study by Ross (2004), we used the following equation to set s _f : 

$s_f=0.5*\min (1,1-\frac{(\phi -40)}{50}).$

(2)

Thus, for φ up to 40°, s _f was 0.5, which thereafter linearly decreased to zero at the maximum φ of 90° (Ross, 2004). The speed of each dot motion was left unchanged, as motion streaks appear to affect the perception of motion direction only (Burr & Ross, 2002). The deflecting orientation signals coupled with the dots in the 3D cloud, like the dot pairs in the integrated display, were oriented toward one direction on the screen to define a form FOE. Four global form coherence levels of these orientation signals were tested, i.e., 100%, 75%, 50%, or 25% of the orientation signals were aligned with the form FOE. Figures 6a and 6b illustrate the velocity fields of the 3D random-dot cloud display with the coherence of the deflecting orientation signals at 100% and 50%, respectively. 

As in the previous two experiments, on each trial, the simulated observer translation direction was randomly varied between −8° and 8° along the azimuth of the center of the screen. The form FOE of the deflecting orientation signals had five offset values (0°, ±5°, and ±10°) from the simulated observer translation direction. 

Procedure

Each observer completed a total of 800 trials (40 trials × 5 form FOE offset directions × 4 global form coherence levels) in a fully randomized order. The testing and screening procedures were the same as in the two previous experiments. Note that there was no visible form signal in the 3D cloud display in this experiment, so observers' judged heading direction corresponded to their perceived motion FOE direction in optic flow. 

In a separate session, each observer was also asked to use a mouse-controlled probe to indicate the center of a static radial Glass pattern (i.e., the perceived form FOE direction) at the same four global form coherence levels of 100%, 75%, 50%, and 25%. The static radial Glass pattern was generated using the integrated display with the simulated observer translation speed set to 0 m/s. Each observer again completed 800 trials (40 trials × 5 form FOE offset directions × 4 global form coherence levels) in a fully randomized order. 

Results and discussion

The mean heading bias averaged across six observers is plotted against the form FOE offset of the deflecting orientation signals for the four coherence levels in Figure 7a. A horizontal line (dashed line at 0°) indicates that observers' perceived heading (i.e., the motion FOE in optic flow) coincides with the simulated observer translation direction, whereas a positive slope indicates that the perceived motion FOE in optic flow is shifted toward the form FOE due to local motion direction deflections. The mean slope of heading bias as a function of global form coherence level is plotted in Figure 7b, together with the mean slopes of heading bias from Experiments 1 and 2. A one-way repeated-measures ANOVA on the slopes reveals that the main effect of form coherence is significant (F(3,15) = 80.96, p < 0.001). The slope increases with the global form coherence of the deflecting orientation signals, indicating that the perceived motion FOE in optic flow was shifted more toward the form FOE as more deflecting orientation signals were aligned with it. 

Figure 7

 

(a) Mean heading bias against form FOE offset of the deflecting orientation signals for the four global form coherence levels. Error bars are SEs across six observers. (b) Mean slope of heading bias against form coherence level for the data from all three experiments. The shaded area indicates mean slope of predicted heading bias assuming optimal combination of the form and the shifted motion FOE directions (its height corresponds to SEs across six observers). (c) Mean predicted form and motion weights (w _f + w _m = 1) against form FOE offset for the four form coherence levels. Error bars are SEs across six observers. (d) Mean predicted heading bias against form FOE offset for the four form FOE levels (shaded areas, their height corresponds to SEs across six observers), along with the observed mean heading bias from Experiments 1 and 2.

Separate t-tests reveal that the slopes for the 100%, 75%, and 50% coherence levels are significantly larger than zero (t(5) > 7.05, p < = 0.001). The slope for the 25% coherence level is marginally not different from zero (t(5) = 2.55, p = 0.051), indicating that local motion direction deflections no longer lead to a significant shift of the perceived motion FOE direction when the global form coherence of the deflecting orientation signals is low. Furthermore, independent t-tests reveal that the slopes for all but the 25% coherence level are significantly lower than the slopes for the same coherence levels of the integrated displays from Experiments 1 and 2 (t(12) = 3.93, p < 0.01, t(16) = 2.88, p < 0.05, and t(16) = 2.56, p < 0.05 for 100%, 75%, and 50% coherence levels, respectively), indicating that while local motion direction deflections can shift the perceived motion FOE direction in optic flow, this local effect alone cannot explain the total heading bias toward the form FOE in the integrated displays in the two previous experiments. 

We now determine whether the heading bias toward the form FOE in the integrated displays can be explained by an optimal combination of the form FOE conveyed by the global form structure of the radial Glass pattern and the shifted motion FOE in optic flow due to the local interactions between form and motion signals. If the brain behaves like a maximum likelihood estimator in combining the form FOE (F _f ) and the motion FOE (F _m ) for heading perception, and assuming that the form and motion FOE direction likelihood distributions are independent and Gaussian, the perceived heading (H _p ) can be represented by 

$H_p=w_f{F}_f+w_m{F}_m.$

(3)

Assuming a uniform prior of the perceived heading and incorporating the normalizing assumption (w _f + w _m = 1), the weights for the form FOE (w _f ) and the motion FOE (w _m ) are in inverse proportion to the variances associated with their respective measurements, given by 

$w_f=\frac{{\sigma }_m^2}{{\sigma }_f^2+{\sigma }_m^2}\mathrm{a}\mathrm{n}\mathrm{d}{w}_m=\frac{{\sigma }_f^2}{{\sigma }_f^2+{\sigma }_m^2}.$

(4)

 

For each of the six observers in this experiment, we were thus able to use the variances associated with the form FOE and the shifted motion FOE measurements to compute the predicted weights of these two cues. We used the SDs of each observer's responses in our probe adjustment task to estimate the form and motion variances. To ensure that our estimated variances were not contaminated by the response variance and reflected the sensory variance, in a pilot experiment, we compared five observers' motion and form FOE responses in the 2AFC discrimination task with their responses in the probe adjustment task at the 100% coherence level. Indeed, for the range of the parameters used in the experiment, we found that the variance measured as the sigma of the fitted psychometric function in the discrimination task was similar to the SD of the responses in the probe adjustment task (mean ± SE across 5 observers: 2.43° ± 0.39° vs. 2.45° ± 0.15° for motion variances, and 2.18° ± 0.09° vs. 1.99° ± 0.27° for form variances). 

Figure 7c plots the mean predicted weights for the form and motion cues averaged across six observers as a function of form FOE offset for the four coherence levels. A 4 (global form coherence) × 5 (form FOE offset) repeated-measures ANOVA reveals that both the main effects of global form coherence and form FOE offset are significant (F(3,15) = 77.97, p ≪ 0.0001 and F(4,20) = 3.59, p < 0.05), as well as their interaction effect (F(12,60) = 2.89, p < 0.01). As expected, the predicted weights for the form FOE cue increase with global form coherence. Separate one-way repeated-measures ANOVAs show that while the weights at the 100% coherence level display a “U” shape (i.e., higher form weights at larger form FOE offset angles, F(4,20) = 7.21, p < 0.001), they are relatively flat at the other three coherence levels (F(4,20) ≤ 1.34, p ≥ 0.29). This “U” shape is mainly due to the larger variance associated with the shifted motion FOE estimation when the form FOE of the deflecting orientation signals had a ±10° offset. 

For each observer, we then used the predicted weights and the mean perceived form and motion FOE directions to compute the predicted heading bias at each form FOE offset and global form coherence level (Equation 3). Figure 7d plots the mean predicted heading bias averaged across six observers as a function of form FOE offset for the four coherence levels tested in this experiment, together with the mean observed heading bias for the same four coherence levels from Experiments 1 and 2. The predicted and observed heading biases are similar, suggesting that the visual system combines the form FOE and the shifted motion FOE in a statistically optimal way for heading perception. 

To better compare the predicted heading biases with the empirical data from the two previous experiments, Figure 7b plots the mean slope of predicted heading bias averaged across six observers as a function of form coherence level, along with the mean slopes of observed heading bias from the two previous experiments. Independent t-tests reveal that the predicted slopes at all four coherence levels are not significantly different from the observed slopes (100%: t(12) = −0.06, p = 0.957; 75%: t(16) = −1.18, p = 0.255; 50%: t(16) = 0.06, p = 0.950; and 25%: t(16) = 0.64, p = 0.533). Thus, a weighted linear combination of the perceived form and motion FOE directions accurately predicts the observed heading bias toward the form FOE in the integrated displays at all four coherence levels. This indicates that the visual system treats the motion FOE in the radial flow and the form FOE in the radial Glass pattern as independent cues for heading perception and behaves like a maximum likelihood integrator in combining them for the final heading estimate. 

General discussion

Combining the results from the three experiments, we have found for the first time that humans use the motion-streak-like form information in a radial Glass pattern for heading perception. This is contrary to the traditional proposal that humans rely mainly on motion signals in optic flow to perceive heading (Gibson, 1950). The data from Experiment 1 show that when the form FOE in the radial Glass pattern and the motion FOE in the radial flow are put in conflict in the integrated display condition, the perceived heading is significantly biased toward the form FOE direction (weight: 0.78). By manipulating the global form coherence of the radial Glass pattern, the data from Experiment 2 show that the global form percept mediates the biasing effect of the form FOE on heading perception. The data from Experiment 3 reveal, however, that local interactions between the orientation of each dot pair and its perceived motion direction in the integrated display also affect heading perception. By presenting observers with a 3D random-dot cloud display in which each dot motion direction was deflected as if it had an orientation signal from a dot pair coupled with it, we find that the perceived motion FOE direction in the optic-flow field is shifted. Nevertheless, the amount of shift of the motion FOE in optic flow accounts for about half of the total heading bias observed for the integrated displays in Experiments 1 and 2 (see Figure 7b), indicating a clear role for both local and global interactions between form and motion signals in heading perception. 

Neural mechanisms for local form–motion interactions

For the local level interactions, two neural mechanisms could be responsible for the bias of the perceived motion direction of dot pairs in a Glass pattern toward their spatial orientation (Krekelberg et al., 2003; Ross, 2004). First, at area V1, Geisler (1999) has proposed that motion streaks, generated by temporally integrating motion of fast moving objects, activate the finely tuned V1 orientation detectors, the signals of which are then combined with those from the more broadly tuned V1 direction-selective neurons to arrive at accurate estimates of the direction of object motion. Thus, when motion-streak-like form information (such as the oriented dot pairs in a Glass pattern) is not aligned with the actual motion direction, the perceived motion direction would be biased toward the form orientation. Second, at area MT, pattern type neurons respond to static bars oriented near parallel to their preferred motion direction (Albright, 1984). If a form signal in the pattern neuron's receptive field is not perfectly aligned with its preferred motion direction, the response of this MT pattern neuron would be shifted toward the form orientation. 

In the context of heading perception, motion signals in optic flow are first processed in areas V1 and MT before they input to area MST in the primate brain for global motion extraction (Duffy & Wurtz, 1991; Tanaka & Saito, 1989). Macaque MT and especially V1 neurons have much smaller receptive fields (V1: about 1–3°, Hubel & Wiesel, 1974, 1977; MT: about 2–15°, Albright & Desimone, 1987; Felleman & Kaas, 1984) than macaque MST neurons (>60°, Duffy & Wurtz, 1991) and process primarily local motion information. The small receptive fields of V1 and MT neurons may cause the aperture problem in detecting local motion direction, e.g., the perceived motion direction of an obliquely oriented line is biased toward the direction perpendicular to the line's orientation. However, due to the small size of the dot pairs (1.5° total extent) we used and spatial integration of motion signals, it is unlikely that the detection of local motion of dot pairs in our integrated displays suffers from the aperture problem. 

Neural mechanisms for global form–motion interactions

At the global level, it has been well established that area V4 along the ventral visual pathway is selective for global form patterns (Gallant et al., 1993, 1996; Ostwald et al., 2008), while area MST down the dorsal pathway responds to global motion patterns (e.g., Duffy & Wurtz, 1991; Tanaka & Saito, 1989). Despite the prevalent view that form and motion information processing is segregated along the ventral and dorsal pathways (see, e.g., DeYoe & Van Essen, 1988; Mishkin et al., 1983), many brain-imaging studies now show that motion and form processing are closely linked (see Kourtzi et al., 2008 for a review). The interaction between the two pathways is also supported by anatomical evidence showing a substantial number of connections between the downstream areas along the two pathways (e.g., Felleman & Van Essen, 1991). The findings from the current study show that the form FOE conveyed by the global structure in a radial Glass pattern together with the motion FOE in the radial flow field determines the final heading estimate, and thus are consistent with the view that the ventral and the dorsal pathways communicate with each other for form and motion processing. 

Optimal combination of form and motion cues

The data from Experiment 3 show that local interactions between the orientation of dot pairs and their perceived motion direction in the integrated display shift the perceived motion FOE direction in optic flow. Using the measurements of this shifted motion FOE direction and the perceived form FOE direction in the radial Glass pattern, we find that the heading bias observed for the integrated displays at four different global form coherence levels in Experiments 1 and 2 can be accurately predicted by be a weighted linear sum of these two cues with the weights in inverse proportion to the variances associated with their separate measurements. This indicates that the brain treats the motion signal in optic flow and the motion-streak-like form information in the radial Glass pattern as separate cues for heading perception and combines these two sources of information in such a way as to arrive at a most reliable (minimum variance) heading estimate. 

Note that, in the integrated displays, we only tested small conflict angles between the form and motion FOE directions (≤10°). Observers, when questioned during debriefing, consistently reported perceiving a single heading direction, indicating successful integration of the motion and form FOE cues to arrive at a unitary percept of heading. In a previous study, however, we found that when the conflict angle between motion and form FOE directions is larger than 30°, observers noticed the conflict and vetoed the form cue (Cheng, Khuu, & Li, 2008). This is consistent with studies showing that when discrepancies between two sensory cues are too large, the brain shows robust estimation and discounts information from a discrepant source (Girshick & Banks, 2009; van Ee, van Dam, & Erkelens, 2002). 

As to the potential cortical areas in charge of the optimal integration of the form and the motion FOE cues for heading perception, there are several possibilities. The activation of macaque MST neurons is known to be modulated by cues relevant to heading perception, such as eye and head movements (Bradley, Maxwell, Andersen, Banks, & Shenoy, 1996; Shenoy, Bradley, & Andersen, 1999). Along with the recent evidence showing neural correlates in macaque MST for optimal integration of visual and vestibular cues for heading discrimination (Gu et al., 2008), area MST could be the site for integrating all cues pertaining to heading perception. However, although area V4 and MT are interconnected (Maunsell & Van Essen, 1983), MST neurons do not seem to respond to form information (Geesaman & Andersen, 1996) and neural connections between areas V4 and MST are rarely found (Boussaoud, Ungerleider, & Desimone, 1990; Ungerleider, Galkin, Desimone, & Gattass, 2008), arguing against the possibility that global form information from area V4 is integrated with global motion information in area MST. Recent neurophysiological studies reveal that the ventral interparietal area (VIP) is also involved in heading perception (Bremmer, Duhamel, Hamed, & Graf, 2002; Zhang & Britten, 2010; Zhang, Heuer, & Britten, 2004). This area receives some inputs from area V4 but a much larger amount of input from area MST (Ungerleider et al., 2008). As the superior temporal polysensory area (STPa) responds to optic-flow stimuli (Anderson & Siegel, 1999) and receives extensive inputs from both form and motion processing areas (Boussaoud et al., 1990; Oram & Perrett, 1996), it appears to be the most likely candidate brain area for the integration of global form and motion signals for heading perception. 

In conclusion, our results indicate that human heading perception is not solely determined by motion signals in optic flow. Motion-streak-like form information in a radial Glass pattern strongly influences heading perception. This influence arises from both local and global level interactions between form and motion signals. At the local level, the perceived local motion direction is deflected toward the dot-pair orientation, leading to a shift of the perceived motion FOE direction in optic flow. At the global level, this shifted motion FOE is then optimally combined with the form FOE to determine the final heading estimate. The neural candidates for the site of local interactions between form and motion signals are known, but the site responsible for the integration of global form and motion information remains in question and needs further research. 

Acknowledgments

This study was supported by a grant from the Research Grants Council of Hong Kong (HKU 7478/08H) to L. Li. We thank Sieu Khuu, Bosco Tjan, Jeff Saunders, and two anonymous reviewers for their helpful suggestions and discussions. 

Commercial relationships: none. 

Corresponding author: Li Li. 

Email: [e-mail]. 

Address: Department of Psychology, The University of Hong Kong, Pokfulam, Hong Kong SAR.