Reliability and relative weighting of visual and nonvisual information for perceiving direction of self-motion during walking

Abstract

Abstract

Abstract:

Abstract  Direction of self-motion during walking is indicated by multiple cues, including optic flow, nonvisual sensory cues, and motor prediction. I measured the reliability of perceived heading from visual and nonvisual cues during walking, and whether cues are weighted in an optimal manner. I used a heading alignment task to measure perceived heading during walking. Observers walked toward a target in a virtual environment with and without global optic flow. The target was simulated to be infinitely far away, so that it did not provide direct feedback about direction of self-motion. Variability in heading direction was low even without optic flow, with average RMS error of 2.4°. Global optic flow reduced variability to 1.9°–2.1°, depending on the structure of the environment. The small amount of variance reduction was consistent with optimal use of visual information. The relative contribution of visual and nonvisual information was also measured using cue conflict conditions. Optic flow specified a conflicting heading direction (±5°), and bias in walking direction was used to infer relative weighting. Visual feedback influenced heading direction by 16%–34% depending on scene structure, with more effect with dense motion parallax. The weighting of visual feedback was close to the predictions of an optimal integration model given the observed variability measures.

Introduction

During walking, there are multiple sources of information about direction of self-motion. Moving through an environment produces optic flow, which is a rich source of information about self-motion. Previous studies have found that observers can judge heading from optic flow in an accurate and robust manner (e.g., Warren, Blackwell, Kurtz, Hatsopoulos, & Kalish, 1991; Warren, Morris, & Kalish, 1988). Nonvisual sensory information about self-motion is also available from vestibular and proprioceptive cues, and observers are capable of judging heading from this information alone (Butler, Smith, Campos, & Bülthoff, 2010; Ohmi, 1996; Telford, Howard, & Ohmi, 1995). When movement is self-initiated, as when walking, the motor system provides another potential source of information. The visual-motor system has some implicit knowledge about the relationship between motor commands and the expected consequences, as evidenced by our ability to initiate movement in the direction of a target from a standing position. This knowledge could potentially be used to predict the expected heading direction resulting from active movement, thereby providing another potential source of information about heading direction. 

The purpose of this study was to measure the reliability of visual and nonvisual cues to direction of self-motion during walking, and test whether these cues are integrated in a statistically optimal manner. An optimal model would integrate all sources of information about self-motion according to their relative reliability. For example, if visual-motor prediction of heading direction has low uncertainty and nonvisual sensory information is very noisy, it would be beneficial to rely heavily on optic flow to estimate heading direction. On the other hand, if predicted heading is highly reliable and nonvisual cues provide precise estimates of heading, optimal integration would predict less weighting of optic flow. Measurement of the reliability of heading perception from visual and nonvisual cues allows quantitative prediction of their expected contributions. 

An approach for testing for optimality is to measure the reliability of different cues and use these measures to predict optimal cue weights and variance reduction from multiple cues, which can then be compared to measured values. Using this paradigm, human sensory integration has been found to be near-optimal in a variety of contexts (e.g., Alais & Burr, 2004; Ernst & Banks, 2002; Hillis, Watt, Landy, & Banks, 2004; Jacobs, 1999; Knill & Saunders, 2003; Lovell, Bloj, & Harris, 2012; Shams, Ma, & Beierholm, 2005). 

Some recent studies have used this paradigm to test whether visual and vestibular cues to direction of self-motion are integrated in a statically optimally manner. Butler et al. (2010) measured heading discrimination thresholds from vision and vestibular cues in isolation, and tested whether the reliability of single cues predicted variance reduction and cue weighting in combined cue conditions. Thresholds in combined cue conditions were consistent with optimal integration, but cue weights were not. Both vision and vestibular cues affected judgments, but observers relied on vestibular cues more than predicted based on measured reliability. Fetsch, Turner, DeAngelis, and Angelaki (2009) observed similar results for trained monkeys: optimal reduction in variance, but apparent underweighting of vision cues. Another recent study by de Winkel, Weesie, Werkoven, and Groen (2010) tested for optimal variance reduction from visual and vestibular cues, and found that vision did not reduce thresholds as much as expected. Thus, there are some observations consistent with statistically optimal integration of vision and vestibular cues, but also evidence that visual information is underutilized. 

I used a similar approach to test whether visual and nonvisual information is optimally integrated for perception of self-motion during walking. The reliability of visual and nonvisual information was assessed by measuring the variability of walking with and without visual feedback, and the relative weighting was measured using cue conflict conditions. Optimal cue weights were computed from variability measures and compared to measured cue weights, as described later. 

Heading perception during active walking

During active walking, proprioception and motor prediction provide additional nonvisual information about self-motion that would not be available during passive movement. These sources of information have been shown to contribute to perception of distance traveled (Campos, Butler, & Bülthoff, 2012; Mittelstaedt & Mittelstaedt, 2001) and perceived speed of self-motion (Frissen, Campos, Souman, & Ernst, 2011). To the extent that proprioception and motor information also contribute to perceived direction of self-motion, the accuracy of heading perception from vestibular cues alone underestimates the full capabilities of the visual-motor system during walking. 

Only one previous study by Telford, Howard, and Ohmi (1995) has tested nonvisual heading judgments when proprioceptive and motor information are available. In their active walking conditions, observers walked along with a moving cart that was accelerated at rates below the threshold for the vestibular system, thereby isolating proprioceptive and motor information. Heading judgments based solely on this information had a variable error of 4.4°. In comparison, Telford et al. (1995) observed average variability of 9° for passive movement without vision, which isolated vestibular cues. This indicates that active walking does provide useful nonvisual information besides vestibular cues, and that this additional information can be more reliable. 

Typical heading judgment tasks are potentially problematic for the situation of active movement. To initiate self-generated movement, an observer must be given some cue to the goal direction of walking. If observers later indicate their perceived direction of self-motion with a visible probe, as in many heading perception studies, responses might be artificially biased by the initial cue. That is, observers might use the remembered visual direction of the goal as a direct cue to their walking direction. Telford and colleagues (1995) avoided this problem by having observers walk along a path guided by a rope, which provides minimal direct information about target direction. However, this also makes the task less naturalistic, and may not provide the same amount of information as normal, self-initiated walking toward a target. 

I used a heading alignment task to assess perception of direction of self-motion during active walking. Observers walked toward a distant target in a virtual reality environment, either with or without global optic flow. The target was simulated to be infinitely far from the observer, so that it had the same egocentric direction regardless of the position of the observer's head. The target therefore provided no direct feedback about direction of translation, and served as a static visual reference direction to which observers aligned their direction of self-motion. In one condition, the environment consisted of only the distant target and a homogeneous background. Accuracy would depend on ability to initiate movement in the correct direction and adjust based on nonvisual feedback. Movement was active and self-initiated, so observers would be able to take advantage of motor and proprioceptive cues as well as vestibular cues. Other conditions included visible environmental features that provided global optic flow. To the extent that global optic flow contributes to perceived self-motion, observers should be able to better align their direction of heading with the target. Differences in variability in conditions with and without optic flow would reflect the contribution of visual information about heading direction. 

Visual environments

The contribution of visual information to perception of self-motion likely depends on structure of the visual environment. Warren, Kay, Zosh, Duchon, and Sahuc (2001) observed that the influence of optic flow on feedback control of walking depended on the visual environment, with rich environments resulting in comparatively more influence. Environmental structure has also been shown to influence the rate of adaptation to conflicting visual and physical cues to self-motion (Bruggeman & Warren, 2010; Bruggeman, Zosh, & Warren, 2007; Saunders & Durgin, 2011). There is also evidence that the structure of the environment can improve visual judgments of self-motion from simulated optic flow (Ehrlich, Beck, Crowell, Freeman, & Banks, 1998; Li & Warren, 2000). 

In addition to varying the presence or absence of visual information about self-motion, I also varied the simulated visual environment. Figure 1 illustrates the two visual environments tested. In the ground condition, the target was on a textured ground that provided optic flow over a large portion of the field of view. In the ground and posts condition, there were also scattered posts throughout the ground plane. Both conditions provide global optic flow that would allow visual perception of heading direction. However, the posts provide more motion contrast than the ground plane alone and could serve as reference objects for perception of self-motion. The results of Warren and colleagues (2001) suggest that these environments will produce different weighting of vision relative to nonvisual information. 

Figure 1

 

Sample views of the two simulated visual environments: textured ground (left) and textured ground with scattered posts (right).

Predicting and measuring cue weights

In this section, I describe the predictions of an optimal integration model and how these predictions were tested empirically. Suppose that visual and nonvisual cues provide estimates of heading direction, H_v and H_n, that are corrupted by Gaussian noise with variance σ_v² and σ_n², and that the noise is statistically independent. For information derived from different modalities, sources of measurement noise would be largely distinct, so statistical independence is a reasonable assumption. The minimum variance estimate from combined cues would then be a weighted average of the visual and nonvisual estimates, with weights inversely proportional to variance of each cue:  

 

If information were used optimally, the variance of the integrated estimate from combined visual and nonvisual information would be lower than variance from individual cues, given by this equation:  

 

If visual and nonvisual cues were the only available sources of information, the weights would sum to one, and can be expressed as:  

 

 

To estimate optimal cue weights, I assumed that variance in walking direction is primarily determined by the noise in visual and nonvisual heading estimates. If so, the variance of walking without vision provides a measure of Display Formula , and the variance of walking with vision provides an estimate of Display Formula . With these two variances, the predicted visual weight w_v can be calculated from Equation 3. 

To measure the relative weighting of visual and nonvisual cues, I tested cue conflict conditions. In these conditions, the heading specified by optic flow did not match the physical direction of heading. Rather, the visual heading was 5° to the left or right of the physical heading (Figure 2). This type of manipulation has been used to study the influence of optic flow on visually guided walking (e.g., Rushton, Harris, Lloyd, & Wann, 1998; Warren et al., 2001). 

Figure 2

 

Illustration of a heading conflict condition and possible responses. The heading specified by optic flow is 5° to the left of the physical direction of the body. If visual information about self-motion is ignored, observers would be aim their physical heading toward the target (left). If observers relied entirely on visual feedback, they would walk with their physical heading 5° to the right of the target, thereby aligning the target with the visual heading (middle). If response is intermediate, the amount of shift of physical heading provides an indication of the influence of visual heading. In the example shown in the right panel, physical heading is shifted by 3°, corresponding to visual weight of 60%.

The walking direction in heading conflict conditions can be used to infer the relative weighting of visual and nonvisual information. Suppose that observers relied entirely on nonvisual information to perceive direction of self-motion, and ignored optic flow. Physical direction of walking would be expected to remain toward the target (Figure 2a). At the other extreme, suppose that observers relied entirely on optic flow to perceive direction of self-motion. To align the visual heading with the target, observers would have to physically walk in a direction 5° away from the visual direction of the target (Figure 2b). If visual and nonvisual information both contribute to perceived heading, one would expect an intermediate walking direction (Figure 2c). The offset in physical walking direction relative to the target, between 0° and 5°, provides a measure of the relative weighting of visual information. 

The main experiment had two components tested in separate sessions: measuring the variability of heading direction with and without visual feedback, and measuring of the relative weighting of visual and nonvisual information in heading conflict conditions. The sessions measuring reliability tested conditions with no visual feedback and with global optic flow that was consistent with the physical direction of movement. These results were used to measure Display Formula and Display Formula . The sessions measuring cue weights tested conditions in which the visual heading specified by optic flow differed from the physical direction of walking by ±5°. The proportional amount of adjustment was used to estimate the relative weighting of visual information, w_v. I also included interspersed trials with no optic flow to test for aftereffects from the heading conflicts, which could arise from fast visual-motor adaptation. 

Optimality can be evaluated from the results in two ways. First, the amount of variance reduction when visual feedback is available, Display Formula versus Display Formula , should be consistent with the variance in perceived heading from optic flow, Display Formula , following Equation 2. For example, if σ_n is much larger than variability in visual heading perception, then optimal integration predicts a large benefit from visual feedback. A comparatively large variability in walking with vision, σ_n+v, would indicate suboptimal use of visual information. Second, the visual weights measured in cue conflict conditions should be consistent with optimal weights predicted by Equation 3 using the measured estimates of Display Formula and Display Formula . For example, if observers responded to heading conflict conditions by largely ignoring the visual heading specified by optic flow, then the measured visual weight would be lower than predicted based on variability measures, indicating suboptimality. 

In addition to the main experiment testing walking toward a target, I also tested ability to judge heading from optic flow for simulated walking trajectories, in order to measure variability in visual heading perception (σ_v) in directly comparable conditions. This experiment is reported as Experiment 1. The main experiment, measuring variability in walking direction and cue weights for visual and nonvisual cues, is reported as Experiment 2. 

Experiment 1. Heading perception from simulated walking

Experiment 1 measured variability in judgments of heading direction from optic flow in conditions that are directly comparable to the walking conditions of Experiment 2. Observers viewed simulated self-motion along straight paths and adjusted a probe to indicate their perceived direction. Two visual environments were tested that provide different qualities of optic flow (Figure 1). The speed profile for simulated movement was constructed to match the average speed profile observed in walking conditions. 

One complication in creating matched visual conditions is that walking produces lateral oscillations of the head, which cause instantaneous heading to vary over a 6°–10° range over time. These lateral oscillations could potentially make it difficult to precisely estimate the overall direction of self-motion. During active walking, vestibular and proprioceptive information might contribute to analysis of optic flow and help to compensate for these oscillations. These nonvisual cues would not be present for simulated self-motion. To the extent that nonvisual cues contribute to analysis of optic flow, presenting a visual simulation of a walking trajectory would not provide equivalent information as during actual walking. To deal with this problem, I tested heading judgments both with and without simulated lateral sway. Performance in these conditions would provide an upper and lower bound on ability to perceived heading from optic flow during normal walking. 

Methods

Participants

Twelve undergraduate students from the University of Hong Kong were paid to participate in the experiment. Due to constraints of the head-mounted display (HMD), participants were required to have either normal vision or corrected-to-normal vision with contact lenses. Participants were also pre-screened for sensitivity to motion sickness. The procedures were approved by and conform to the standards of the Human Research Ethics Committee for Non-Clinical Faculties at University of Hong Kong. 

Apparatus and display

Simulated self-motion was presented with the same virtual reality apparatus used in the main walking experiment. Observers were seated and wore an NVis SX-111 HMD (NVis, Inc., Reston, VA). The HMD presented stereo images with 1280 × 1024 resolution for each eye, updated at 60 Hz. The visual field of view for each eye was 76° × 64° (H × V), with binocular overlap of 38° and total horizontal field of view of 90°. Participants wore a narrow black scarf to cover any gap between their face and the HMD. Stereo views of the virtual environment were rendered using OpenGL and an Nvidia Quadro FX 580 graphics card (Nvidia, Santa Clara, CA). 

Observers were instructed to keep their body stationary but were allowed free movement of their head. The position and orientation of the head was sampled at 180 Hz with an Intersense IS-1200 inertial tracking system (Intersense, Billerica, MA), and displays were updated in real time. To match conditions in the walking experiment, participants also wore active noise-cancelling earphones (Etymotic Research MC5, Elk Grove Village, IL). 

Two simulated environments were tested: textured ground only, or textured ground with scattered posts. The ground texture was a texture-mapped tileable image that had energy at low as well as high spatial frequencies. Although observers were seated during the experiment, the ground was simulated to be at a vertical distance equal to the observer's standing eye height. Standing eye height was used for consistency with the walking experiments. Scattered poles were gray and 0.1 m in diameter and 1.25 m tall, and had a density of 0.25/m². In both conditions, a homogeneous “sky” region was also rendered to visually indicate the horizon. In addition to the background environment, an adjustable probe was visible throughout simulated movement. The probe was a frontal disc with 2.8° diameter simulated to be at eye level and infinitely far from the observer. 

Simulated movement was 3.35 m along a straight path with 4.5 s total duration. The speed profile was constructed to be similar to the mean speed profile of walking trajectories observed in Experiment 2. Acceleration linearly increased from zero to 1.12 m/s² during the first 1 s, and linearly decreased back to zero during the second 1 s, resulting in a speed of 1.12 m/s after 1 m of movement. Speed remained constant until 2.7 m (3.5 s), after which deceleration was linearly varied from zero to −4.48 m/s² and then back to zero, bringing movement to a stop. For conditions with simulated lateral sway, the frequency of oscillation was 0.91 cycles/s and the peak-to-peak amplitude was 30 cm during the main “walking” phase from 1 m to 2.7 m. During the acceleration and deceleration phases of simulated movement, the amplitude of oscillation was smoothly modulated with an envelope that approximates the transitions observed in the walking experiment. 

Procedure

Participants viewed simulated self-motion and adjusted a probe disc with a mouse to be aligned with their perceived direction of self-motion. The simulated heading direction varied over a range from −25° to +25° around the central direction that their body was facing. The probe was visible during simulated movement and remained visible until the participant pressed a button to indicate their response. 

The experiment was conducted in a single one-hour session consisting of 20 practice trials followed by two blocks of experimental trials with 80 trials each. The four combinations of visual environment and lateral sway were randomly intermixed within blocks. Trials were self-paced and a break was provided between blocks. Practice trials were the same as experimental trials, and no feedback was given. Each participant performed a total of 40 trials per condition. 

Results

The results were analyzed to estimate center bias as well as heading variability. Previous studies have found that heading judgments tend to be systematically biased toward the center of the display (e.g., Johnston, White & Cumming, 1973; Saunders, 2010; Warren & Saunders, 1995). If such biases were present, heading errors for eccentric translation directions would not be solely due to variable error, which could inflate the apparent variability in visual heading perception. In this study, the HMD moved with an observer's head, so the displays did not provide a fixed reference frame. However, I found that there was still a general bias toward a central axis defined by the direction of the body. To distinguish variable error from errors due to a center bias, I used a regression analysis. For each observer and condition, I performed a linear regression fit of judged heading direction as a function of actual heading direction, with heading expressed relative to the central direction of the body. On average, heading judgments varied linearly as a function of actual heading, but with a slope less than one due to center bias. The residual errors indicate the variability in heading judgments after factoring out the effect of center bias. I used the slopes of regression fits as a measure of accuracy for each observer and condition, and the root-mean-squared residual error as a measure of heading variability. Constant bias was also fit for each observer and condition, but these showed no systematic deviations from zero and are not reported. To minimize the effect of outliers, I used the Theil-Sen estimator to obtain a robust estimate of the regression slope (Rousseeuw & Leroy, 2003), and the biweight midvariance for a robust estimate of the residual variability. 

The left graph of Figure 3 plots the mean slopes of the regression fits, averaged across observers, for the four conditions. The mean slopes were close to one, indicating that heading judgments were relatively accurate on average. However, there was a modest but significant center bias in all conditions (p < 0.001). An ANOVA on the slopes found a main effect of visual environment, F(1, 11) = 10.2, p = 0.009, corresponding to higher accuracy in the richer environment with ground and scattered posts. The presence of simulated lateral sway had no effect on center bias, F(1, 11) = 0.17, p = 0.70, and did not interact with visual environment, F(1, 11) = 0.90, p = 0.36. 

Figure 3

 

Accuracy and variability of visual heading judgments from Experiment 1. (a) Mean slope of regression fits of judged heading as a function of simulated heading. A slope of less than 1.0 corresponds to center bias. (b) Mean variability in heading judgments computed from the residual errors of the regression fits. Blue squares show results for the environment with only a textured ground plane, and red diamonds show results for the environment with a textured ground and scattered posts. Closed symbols show results for conditions with simulated lateral sway, and open symbols show results with no simulated sway. Error bars depict ±1 standard error of the mean.

The right graph of Figure 3 plots the mean variability of heading judgments, averaged across observers, for the four conditions. An ANOVA revealed main effects of both visual environment, F(1, 11) = 11.1, p = 0.007, and simulated lateral sway, F(1, 11) = 22.9, p < 0.001, and no interaction between these factors, F(1, 11) = 0.94, p = 0.35. Heading judgments were less variable overall for the environment with ground plane and posts than the environment with ground plane alone, and simulated lateral sway significantly increased variability. Pairwise comparisons found a significant effect of visual environment on variability for conditions with simulated lateral sway, t(11) = 2.97, p = 0.012, but not for the conditions without simulated sway, t(11) = 1.65, p = 0.127. The interpretation of the main effect of visual environment is therefore ambiguous, and might be solely due to conditions with simulated sway. 

Discussion

The main purpose of Experiment 1 was to measure variability in visual heading perception (σ_v) in conditions that are comparable to the walking experiments in Experiment 2. One complication in creating matched conditions, discussed previously, is the presence of lateral sway during walking. The results of Experiment 1 indicate that such sway would have a limited effect on performance: simulated lateral sway increased heading variability by 21% on average. The variability in conditions with and without simulated sway provide an estimate of the upper and lower bound on the variability of visual heading during actual walking. Because the difference between these conditions was small, the mean results for the two visual environments, σ_v = 3.77° and 3.35°, would provide a reasonable estimate of reliability of visual heading perception during walking. 

Some previous studies have measured variability in heading judgments from optic flow in comparable conditions. Telford et al. (1995) tested heading perception from passive translation with vision in a condition with very slow acceleration and deceleration, which effectively isolates visual information. Heading judgments were found to have variability of σ_v = 2.9° in this condition, which is close to the measures observed here. Other studies of visual heading perception have observed greater precision. Warren and colleagues observed 75% JND thresholds of around 1.5° for simulated movement along a ground plane at fast walking speeds (Warren & Hannon, 1988; Warren et al., 1988), which would correspond to a variability of σ_v = 2.2°. The larger variability observed in Experiment 1 might be due to the relatively slow simulated speed of movement, which was used to match walking trajectories in Experiment 2. Another possible factor is the amount of visible optic flow, particularly for the condition with only a textured ground. If observers tended to align their head toward the probe, which was at eye level, the visible region of the ground was limited to regions farther than 2.7 m. Such factors could explain the modest discrepancy between the present results and previous measures of the precision of visual heading perception. 

I found that the variability of heading judgments depended on the visual environment. A motivation for testing these environments was the result of Warren et al. (2001). They observed more influence of optic flow in an environment like the ground and posts condition than in an environment with only a textured ground. The results of Experiment 1 indicate that the richer environment provides more reliable visual information about heading direction. Optimal integration of visual and nonvisual information would therefore predict greater reliance on vision, which would be consistent with the findings of Warren and colleagues (2001). The predictions of optimal integration are directly tested in Experiment 2. 

Experiment 2. Heading perception during active walking

Experiment 2 tested walking toward a distant target with no visual feedback or with global optic flow from different visual environments (Figure 1). In one set of sessions, visual feedback was consistent with the physical direction of movement. Performance in these consistent-cue trials was used to estimate the variability of walking direction with and without visual feedback (σ_v+n and σ_n). In another set of sessions, the optic flow specified a heading direction that differed from the physical direction of movement by ±5°. Performance in these heading conflict conditions was used to estimate the relative weighting of visual information for perception of heading direction during walking (w_v). Results were analyzed to evaluate whether variance reduction and cue weights were consistent with optimal integration of visual and nonvisual information. 

Methods

Participants

Fifteen undergraduate students from University of Hong Kong were paid to participate in the experiment. Due to constraints of the HMD, participants were required to have either normal vision or corrected-to-normal vision with contact lens. Participants were also pre-screened for sensitivity to motion sickness. The procedures were approved by and conform to the standards of the Human Research Ethics Committee for Non-Clinical Faculties at University of Hong Kong. 

Apparatus and display

Observers walked in a virtual environment. The virtual reality environment was the same as in Experiment 1: an NVis SX-111 HMD to present visual displays and an Intersense IS-1200 inertial tracking system to record position and orientation of the head. To eliminate auditory cues to self-motion and position within the room, participants wore noise-cancelling earphones (Etymotic Research MC5). Participants also wore a narrow black scarf to cover any gap between their face and the HMD, ensuring that the physical ground was fully occluded. 

Three simulated environments were tested: target-only, target with ground, and target with ground and posts. The target was a frontal disc with 2.8° diameter simulated to be at eye level and infinitely far from the observer. The ground texture was a texture-mapped tileable image that had energy at low as well as high spatial frequencies. Scattered poles were gray, 0.1 m in diameter, and 1.25 m tall, and had a density of 0.25/m². In all conditions, a homogeneous “sky” region was also rendered to visually indicate the horizon. 

Procedure

The task was to walk straight toward a distant target at a natural walking pace until a stop cue was presented. Prior to the start of a trial, a participant aligned their body toward a visible marker. This marker disappeared and the initial position and direction of the head was recorded. The walking target was then presented in a random direction between −15° and +15° around the initial direction of the head. The appearance of the target cued the observer to begin walking. The target remained visible until the observer had walked 3 m from the initial position, at which point a stop cue was presented. After the stop cue, participants turned around and walked to a post indicating the starting location for the next trial. The starting location randomly varied over a 1.6 m region, and participants received no explicit feedback about their performance. 

Observers performed four experimental sessions on separate days, two sessions testing consistent-cue conditions and two testing conflicting-cue conditions. Consistent-cue and conflicting-cue sessions were presented in alternating order with the initial session counterbalanced across subjects. All sessions consisted of two blocks of 36 experimental trials, preceded by a block of 12 practice trials. Practice trials were the same as experimental trials, and no feedback was given. Observers tend to walk atypically slow when first experiencing a virtual environment. During the practice trials, they were reminded to try to walk at a natural pace. In experimental blocks, each of the three visual environments was presented 12 times in randomized order. In the consistent-cue blocks, the heading direction specified by optic flow was always consistent with the physical direction of motion. In the conflicting-cue blocks, the heading specified by optic flow was 5° to the left or right of the physical heading direction. Equal numbers of positive and negative conflicts were presented, with order randomized across trials to prevent long-term adaptation. The conflicting-cue blocks also included target-only trials, for which there was no conflicting optic flow, to assess whether heading conflicts on previous trials caused short-term aftereffects. For each participant, the four sessions yielded a total of 48 trials per condition. 

Throughout a session, an experimenter walked directly behind an observer to ensure safety and to hold the cables for the HMD and tracker. The experimenter held the cables so that there was approximately 1 m of slack hanging behind the observer, and attempted to keep their grasp point centered behind the observer's head. The experimenter was not aware of the exact target location on any given trial. 

Measurement of walking direction

A main goal of this study was to measure how reliably observers can direct their walking toward a target. Lateral oscillations of the head pose a difficulty for measuring the precision of walking direction over the course of a movement. Even if observers walked along a straight path, the instantaneous velocity of the head would have large variability in direction over time. To minimize the effect of lateral sway on measures of heading variability, I filtered the raw trajectories to attempt to remove this oscillatory component, and computed heading direction over 1 m intervals of walking. For each trial, the vertical oscillation of head position (head bob) was first used to estimate the time-varying frequency of the step cycle. This is advantageous because the frequency of head bob is twice the frequency of head sway, and vertical oscillations tend to be around a constant value. The vertical trajectory of the head was band-pass filtered near the expected frequency to isolate the approximate oscillatory component. This initial estimate of vertical oscillation was iteratively refined to get an estimate of the time-varying frequency of the step cycle. The frequency of lateral oscillation would be half the frequency of vertical oscillation, with some unknown phase offset. I estimated the phase of lateral oscillation by computing the cosine and sine components of horizontal position function along windows of one period width. At points where the phase of sway was ±90°, which correspond to midpoints between left and right steps, the lateral oscillatory component would be zero. The positions of the head at these points were used to smoothly interpolate intermediate positions. Figure 4 shows a sample raw and filtered trajectory. The filtered trajectories were parameterized by distance from the starting location in the direction of the target. The horizontal change in head position across distances of 0–1 m, 1–2 m, and 2–3 m were used to compute measures of heading direction at the start, middle, and end of each trial. To check the sway filtering procedure, I compared analysis results using either filtered or unfiltered trajectories. Removing the lateral oscillations reduced variability measures by about 5% but did not change the qualitative pattern of results. 

Figure 4

 

Trajectories of a sample trial to illustrate the filtering of lateral oscillations. The top graph shows the vertical position of the head as a function of distance from the starting location. This function was used to estimate the time-varying frequency of the step cycle. The lower graph shows the horizontal position of the head as a function of distance. The solid line indicates the original, unfiltered trajectory. The circles and gray lines show estimated points where the oscillatory component was zero (phase = π/2 + k·π). The lateral oscillation has half the frequency of the vertical oscillation, and an approximately constant phase offset. The dashed line shows the filtered trajectory obtained by interpolating the points with phase = π/2 + k·π. Estimates of heading direction were computed from these filtered trajectories.

Noise in measurement of heading position would also contribute to noise in estimates of walking direction. Based on manufacturer specifications, the 3D tracking system had RMS errors of 2–5 mm for position and 0.1° for orientation. Over 1 m of walking, a 5 mm RMS error in position measurement would contribute variable error of 0.4° in estimated walking direction. If σ_n is the true variability in walking direction, then the estimated variability including measurement noise would be Display Formula . If σ_n ≥ 1.6°, as suggested by the results, the noise due to the tracking system would increase estimates of variability by less than 0.05°, corresponding to an increase of 3% or less. 

Results

Variability of walking direction with and without vision

To measure accuracy and precision of walking direction across trials, I performed linear regression fits of heading direction as a function of target direction for the set of trials from each observer. The direction of the head at the start of a trial was used as a reference direction to encode target direction and heading direction in normalized coordinates. Heading measures from the start, middle, and end of movement were analyzed separately. For trials with no heading conflict, the regression model included an overall constant bias and three slope parameters representing center bias for the three visual feedback conditions. For trials with heading conflicts, the regression model included two additional bias parameters representing the effect of heading conflicts for the ground environment and the ground with posts environment. In the final interval of walking, the average slope parameter for the condition without visual feedback was 0.95, which was significantly less than one, t(14) = 2.89, p = 0.012. This indicates that final walking direction tended to vary around the correct target direction, except for a small bias toward the direction that the observer faced at the start of a trial. For the conditions with visual feedback, the average slopes in the final interval of walking were not significantly different from one: t(14) = 1.940, p = 0.073 (ground); t(14) = 1.977, p = 0.068 (ground and posts). 

Heading variability for each condition was computed by taking the RMS residual error of trials from a condition. Figure 5 shows variability of heading error as a function of walking distance for blocks of trials with consistent cues (left graph), and blocks of trials with heading conflicts (right graph). The graphs plot the standard deviation of heading error, averaged across observers, for the three visual feedback conditions. The variability over the first 1 m of walking was similar across conditions, averaging σ = 3.3°, and then reduced over the course of movement. The pattern of results was similar for consistent-cue and conflicting-cue conditions, with the main difference being that overall variability was somewhat higher in the conflicting-cue conditions, F(1, 14) = 11.5, p = 0.004. This small difference could have been due to the fact that observers typically made more heading adjustments in the conflict conditions, or possibly due to small aftereffects from preceding heading conflict trials. 

Figure 5

 

Variability in walking direction, averaged across subjects, as a function of walking distance. The three lines on each graph correspond to the target-only (black circles), ground (blue squares), and ground with posts environments (red diamonds). The left graph plots results from blocks of trials with consistent cues, and the right graph plots results from blocks with heading conflicts. Error bars depict ±1 standard error of the mean.

Heading variability at the end of movement (2–3 m) was the main measure of interest, as this reflects performance after adjustments based on visual and nonvisual feedback. The variability in final heading was low even for the target-only condition, which provided no visual feedback about direction of self-motion. To test for effects of the visual environment, the results from conditions with and without heading conflicts were combined to increase power. The presence of global optic flow significantly reduced variability for both the ground environment, F(1, 14) = 7.47, p = 0.016, and the ground with posts environment, F(1, 14) = 28.6, p < 0.001. However, the reduction in variability was modest: σ_n = 2.37° for the target only condition versus σ_n+v = 1.93° for the richer visual environment with ground and posts. Although there appears to be a trend toward a difference between the two visual environment conditions, this trend was not statistically significant, F(1, 14) = 3.63, p = 0.077. 

Figure 6 replots the variability in final heading for conditions with and without visual feedback along with predictions of an optimal integration model. The optimal predictions were computed from Equation 2 using the mean results from Experiment 1 as estimates of variability in visual heading perception, σ_v = 3.77° and 3.35°. Because variability in walking direction without visual feedback was comparatively low, optimal integration predicts relatively little benefit from visual feedback. The observed variability measures in Experiment 2 from walking with visual feedback were close to these predictions. Based on these results, a further prediction is that optic flow would have a relatively low influence on walking paths in heading conflict conditions. 

Figure 6

 

Variability in final walking direction in conditions without heading conflicts (solid symbols), and predicted variability based on results from Experiment 1 assuming optimal integration (dashed symbols).

Cue weighting in heading conflict conditions

Figure 7 shows the response to conflicting visual information on trials with heading conflicts. The graph plots mean heading error as a function of walking distance, averaged across observers, for conditions with positive and negative heading conflicts (solid and dashed), and for the two visual environment conditions (blue and red). On trials with a heading conflict, the error between the physical direction of motion and the target direction is not the same as the error between the visual direction of motion and the target direction. The graph plots the error between the physical direction of motion and the target direction. If visual information was ignored and observers simply directed their physical motion toward the target, the heading errors would be expected to remain at zero. If observers adjusted so that their visual heading from optic flow was directed toward the target, walking errors would be expected to converge to ±5°. The results indicate that observers did adjust their walking direction in response to conflicting visual information. Even during the first 1 m of walking, there was a significant difference between positive and negative conflict conditions, for both the ground environment, t(14) = 2.39, p = 0.032, and the ground with posts environment, t(14) = 2.26, p = 0.041. This difference increased over the course of movement, but remained less than expected if observers fully aligned their visual heading with the target direction. 

Figure 7

 

Effect of heading conflicts on walking direction, averaged across subjects, for trials with positive (solid) and negative (dashed) conflicts. The blue squares show results for the ground environment, and red diamonds show results for the ground and posts environment. Error bars depict ±1 standard error of the mean. If observers aligned their visual heading with the direction of the target, one would expect a heading bias of ±5° (dotted lines). If observers made no correction in response to visual feedback and aimed their physical direction of movement toward the target, the expected heading bias would be zero.

Figure 8 plots the relative weight of visual information at the end of movement, computed from the difference in heading error from positive and negative heading conflict trials over the last 1 m of walking. The visual weight was greater than zero for both visual environments: t(14) = 4.78, p < 0.001 (ground); t(14) = 8.52, p < 0.001 (posts), and was significantly larger for the ground and posts environment, t(14) = 4.09, p = 0.001. However, even in the richer visual environment, the average amount of correction was only about a third of the heading conflict. Figure 8 also shows predicted optimal weights (dashed line) computed from the variability measures using Equation 3. The optimal weights predicted by the mean variability measures are close to the mean weights derived from responses to heading conflict conditions. This indicates that the modest responses to conflicting visual information are generally consistent with the amount of variance reduction observed in conditions without heading conflicts. 

Figure 8

 

Relative weighting of visual information for the ground environment (blue square) and ground with posts environment (red diamond). Visual weights were computed by taking the difference between the mean heading in positive and negative conflict conditions at the end of movement and dividing by the difference expected if observers fully aligned their visual heading (±10°). The graph plots mean visual weights averaged across subjects. Error bars depict ±1 standard error. The dashed line shows predicted weights computed from mean variance measures using Equation 3, and dotted lines show ±1 standard error for the predicted weights.

I also analyzed individual observers' results to test the relationship between observed visual weights and weights predicted by variability measures assuming optimality. Figure 9 plots predicted versus observed weights from individual observers for the ground environment (blue squares) and the ground with posts environment (red circles). Note that there was considerable variability in predicted weights, and a large amount of overlap between the two visual environment conditions. One factor that contributes to this variability is that predicted weights are based on a ratio of variances (Equation 3), which amplifies the effect of measurement noise. Despite this variability, there was a significant correlation between predicted and observed weights (r = 0.48, p = 0.008). This indicates that the influence of conflicting visual information varied in way that was consistent with the observed reduction in heading variability when optic flow was present. There was no significant correlation between observed and predicted weights when the two visual environments were analyzed separately (ground: p = 0.056, posts: p = 0.465), which suggests that the overall correlation is primarily due to a difference between the two visual environments. Although I did not observe a significant difference between the heading variability for the two visual environments (Figure 6), this may have been due to lack of power. For the majority of observers (11 of 15), heading variability was lower in the condition with ground and posts, which could explain the overall correlation between predicted and observed visual weights. 

Figure 9

 

Visual cue weights from individual observers plotted as a function of predicted weights based on individual variability measures, for the ground environment (blue squares) and ground with posts environment (red circles). The dashed line shows the best fitting line from of an orthogonal regression analysis applied to the combined data.

Adaptation effect

Target-only trials with no visual feedback were randomly intermixed with trials with heading conflicts to test whether exposure to conflicting visual information produced aftereffects. This subset of trials was identified and sorted according to the sign of conflict and visual environment in the preceding trial. The heading biases on these trials were normalized for sign of heading conflict and then averaged across trials that were preceded by the same visual environment. Figure 10 plots the mean aftereffects, averaged across observers, for the two visual environments. A small but significant aftereffect was observed for the environment with ground and posts, t(14) = 3.24, p = 0.006, but not for the ground environment t(14) = 0.50, p = 0.628. The mean aftereffect in the ground and posts condition was −0.57°, corresponding to 11% of the conflict on the preceding trial. Note that the positive and negative heading conflicts were randomized across trials to prevent long-term adaptation, so this aftereffect indicates the average influence of a single previous trial. This result demonstrates that visual-motor adaptation in response to conflicting visual information can occur very rapidly. During a trial in which conflicting visual information was present, rapid adaptation might have affected performance over the course of movement. However, the adaptation effect was not large enough to fully explain the observed influence of optic flow on heading conflict trials (11% vs. 34%). 

Figure 10

 

Aftereffects caused by a preceding heading conflict trial. For the target-only trials that were preceded by a trial with conflicting visual information, heading biases were normalized for sign of conflict and averaged across trials to compute a measure of aftereffect for each condition. The graph plots mean aftereffects, averaged across observers, for the two visual environments. Error bars depict ±1 standard error. If an observer fully adapted to the preceding heading conflict, the expected aftereffect would be −5°.

Discussion

Walking without visual feedback

One goal of the present study was to measure the ability to guide walking toward a target without visual feedback. Although many previous studies have measured the accuracy of walking distance without visual feedback (e.g., Loomis, Da Silva, Fujita, & Fukusima, 1992; Mittelstaedt & Mittelstaedt, 2001; Rieser, Ashmead, Talor, & Youngquist, 1990; Thomson, 1983), directional errors when walking without vision have not been reported. I found that in the target-only condition, which provided only a static visual reference direction, walking direction was highly accurate (slope = .95) and had low variability (σ_n = 2.4). Observers are able to walk toward a target in an accurate and reliable manner even with no visual feedback about direction of self-motion. 

Telford et al. (1995) tested heading judgments for active walking with and without vision, which can be compared to the conditions here. For walking with vision, Telford et al. found that heading judgments had a variable error of 2.1°, which is close to the variability in walking direction that observed here in the conditions with global optic flow. For walking without vision, however, Telford et al. observed a variable error of 4.4°, which is larger than observed here. In their active walking conditions, observers were guided, which is not representative of normal walking, and walking speed was very low, which minimized vestibular information. Either of these factors could account for the higher variability in the walking-only condition of Telford et al. (1995). 

Variability in walking direction without visual feedback is much smaller than the variability in heading perception from vestibular cues alone. For passive translation without vision, Butler et al. (2010) observed an average just-noticeable-difference (JND) of 7.2°, and de Winkel et al. (2010) observed an average JND of 16.1°. The variability in heading judgments reported by Telford et al. (1995) for passive motion without vision would correspond to a JND of about 6°. In comparison, heading variability observed here for walking without vision would correspond to a 75% JND threshold of about 1.7°. The comparatively low variability observed from active walking suggests that proprioception and motor information are significant contributors to perceived direction of self-motion. This is consistent with findings of Telford et al. (1995), who observed more accurate heading judgments for active self-motion without vision than passive self-motion without vision. 

Contribution of visual feedback during walking

The availability of visual feedback during walking reduced variability in walking direction, but the benefit was relatively small. The proportional reduction in variance, ( Display Formula – Display Formula ) / Display Formula , was 19% for the ground condition and 33% in the ground and posts condition. The limited benefit is consistent with the relative reliability of visual and nonvisual information. Performance in the target-only condition indicates that observers can reliably walk toward a target even without visual feedback, and with more precision than heading judgments from simulated optic flow (Experiment 1). Consequently, one would expect relatively little benefit from visual information during active walking, as observed. 

The relative influence of visual information measured from the heading conflict conditions was also consistent with the reliability of visual and nonvisual information. In response to optic flow specifying a conflicting heading direction, observers adjusted their walking trajectories to correct for 16% of the conflict with the ground environment and 34% with the ground and posts environment. These visual weights are similar to the amount of variance reduction produced by visual feedback, 19% and 33%, as would be expected based on Equation 3. Furthermore, when individual results were analyzed, there was a correlation between the observed and predicted weights (Figure 9). Thus, both the observed cue weights and amount of variance reduction are consistent with optimal use of visual and nonvisual information. 

General discussion

Optimal integration of visual and nonvisual information about self-motion

The results reported here provide evidence that visual and nonvisual cues are optimally integrated for perception of direction of self-motion during active walking. Two predictions of optimal integration were supported. First, the variability of heading direction when walking with visual feedback was consistent with measures of the reliability of visual information (Experiment 1) and nonvisual information (Experiment 2, target-only condition). Second, the relative weighting of visual information measured from heading conflict conditions was consistent with the observed reliability of visual and nonvisual cues. The consistency across measures suggests that visual and nonvisual information about heading direction was integrated in an optimal manner. 

Butler et al. (2010) performed a similar test for optimal integration of visual and inertial cues to heading perception, and observed results suggesting suboptimal integration. Heading discrimination thresholds were measured from vision alone, passive translation without vision, and combined visual and inertial cues. The presence of both visual and inertial cues reduced variance by an amount consistent with optimal integration (i.e., consistent with Equation 2). However, judgments in the heading conflict conditions indicated that vision was weighted less than inertial cues, contrary to optimal predictions (observed 30%–40% vs. predicted 50%–60%). Visual cues appeared to be underweighted, given the relative reliability of visual and inertial cues to heading direction. 

The results presented here, consistent with optimal integration, would appear to conflict with the findings of Butler et al. (2010). One possible factor is active versus passive self-motion. Information may be integrated more effectively in the natural situation of active walking. However, the discrepancy in findings could also be due to errors in estimating reliability of different cues. For example, if Experiment 1 underestimated the true reliability of visual heading perception from optic flow, the present results would reflect suboptimal use of visual information. In the other direction, Butler et al. (2010) might have underestimated the ability to perceive heading direction from nonvisual cues. In a previous version of Experiment 2 (not reported here), the target disappeared after the onset of movement. This seemingly small difference in methodology was sufficient to cause the reliability of walking without visual feedback to be underestimated. Quantitative evaluation of optimality would be sensitive to any such errors in measurement of reliability. 

Another study by de Winkel et al. (2010) tested for optimal variance reduction from vision and inertial cues to heading direction, but did not predict or measure cue weights. In contrast to Butler et al. (2010), de Winkel et al. (2010) found that combined cues did not reduce variance in an optimal manner. In some cases, thresholds from combined visual and inertial cues were higher than from visual cues alone, which would be consistent with overweighting of inertial cues. However, this finding could also be explained by particulars of the methodology. In de Winkel et al. (2010), observers judged perceived direction of self-motion relative to straight-ahead, and heading was only varied in a rightward direction. Over the course of trials, perception of straight-ahead might have adapted toward the average direction of simulated motion. If such adaptation occurred, rightward translation would have appeared closer to straight ahead, resulting in larger apparent thresholds. Thresholds observed by de Winkel et al. (2010) were higher overall than in Butler et al. (2010), consistent with this possible explanation. Thus, the results of de Winkel et al. (2010) might indicate overweighting of inertial cues to heading direction, but could also be explained by adaptation of perceived straight-ahead. 

Implications for studies of visually guided walking

The results reported here could help to explain conflicting observations in previous studies of visually guided walking. Heading conflict conditions have been used to test the role of optic flow in online control of walking. A number of studies have found no apparent influence of conflicting visual heading (Harris & Bonas, 2002; Harris & Carre, 2001; Rushton et al., 1998; Saunders & Durgin, 2011). Other studies have observed a significant influence of visual heading (Bruggeman et al., 2007; Warren et al., 2001; Wood, Harvey, Young, Beedie, & Wilson, 2000), but corrections were not always large or immediate, and depended on the structure of the simulated environment. Given that heading can be perceived from optic flow in an accurate and robust manner, the weak influence of optic flow on visually guided walking might be taken as evidence that this information is not fully utilized for locomotor control. 

However, the present results indicate that a weak influence of optic flow might be expected even if visual information were used optimally. I found that observers are capable of high walking precision even without visual feedback, with variability as low as judgments of heading from optic flow. Unless optic flow provided very strong information, an optimal strategy would rely primarily on feed-forward prediction and nonvisual sensory information. This could explain why an influence of optic flow on visually guided walking has only been observed in some situations. Factors such as a small field of view or low density of environmental features might be sufficient to degrade optic flow to the point where it would not be advantageous to rely on visual information to perceive heading during walking. In the present experiment, I observed different contributions of visual information for the ground-only environment and the ground and posts environment, which is consistent with a difference observed by Warren et al. (2001). Although optic flow from a textured ground plane is sufficient for reliable judgments of heading, this information may not be reliable enough to override nonvisual information about self-motion available during active walking. Due to the high reliability of walking without vision, the contribution of visual information might be highly sensitive to the quality of optic flow. 

Visual-motor adaptation

Although the present results show that visual feedback from optic flow is not needed to accurately walk toward a target, it would still be required to maintain visual-motor calibration. Initiating movement toward a visible target and adjusting based on nonvisual sensory feedback both require knowledge about the mapping from physical to visual coordinates. Visual feedback during self-motion provides the information needed to learn these mappings and maintain calibration. 

Some recent studies have found that exposure to heading conflict conditions can induce rapid visual-motor adaptation (Bruggeman et al., 2007; Bruggeman & Warren, 2010; Herlihey & Rushton, 2012; Saunders & Durgin, 2011). Consistent these previous results, I found that a single heading conflict trial is sufficient to cause a small but detectable negative aftereffect in a subsequent trial with no visual feedback. This adaptation effect was observed only for the richer visual environment with ground and posts. The previous studies by Bruggeman and colleagues (Bruggeman et al., 2007; Bruggeman & Warren, 2010) and by Saunders and Durgin (2011) also found that the amount of adaptation depended on the structure of the visual environment. One important difference relative to previous studies is that I tested walking toward an infinitely distant target. There was therefore no target drift when observers walked in a direction offset from the target direction in heading conflict conditions. Target drift potentially provides a useful error signal for maintaining visual-motor calibration, and the results of Saunders and Durgin (2011) and Herlihey and Rushton (2012) indicate that this information can be sufficient to drive adaptation. In the conditions tested here, however, there was no target drift, so the observed adaptation effect was due to discrepant optic flow from the surrounding environment. The present results provide further evidence that optic flow is used to continuously recalibrate visual-motor mappings, and that adaptation can occur without direct feedback from the target. 

Rushton (2004) has suggested that the influence of optic flow on control of walking might be primarily due to fast visual-motor adaptation, rather than direct use of visual heading error. If adaptation to a heading conflict can occur during the course of an ongoing movement, this would tend to increase the apparent weighting of visual information on a trial. However, the amount of adaptation observed here, as indicated by the size of aftereffects, was smaller than the influence of visual feedback during the course of a heading conflict trial. On heading conflict trials with the ground and posts environment, observers corrected for an average of 34% of the discrepancy between visual heading and target direction, but the subsequent negative aftereffect averaged only 11%. For the environment with only a textured ground, observers made significant corrections during heading conflict trials but there was no detectable aftereffect in subsequent trials. While fast adaptation could have contributed to corrective adjustments in trials with heading conflicts, the results suggest that such adaptation would not fully explain the influence of optic flow. 

Conclusions

The goals of the experiments were: (a) to measure walking performance in conditions with and without visual feedback, (b) to determine how much visual heading from optic flow should be weighted if information was used optimally, and (c) to estimate the relative weighting of visual heading for control of walking and compare to the prediction of an optimal model. 

The results demonstrate that observers are capable of walking toward a target with high precision even with no visual feedback. Consequently, even if visual information from optic flow were weighted optimally, the expected contribution from vision would be limited. Performance in heading conflict conditions was consistent with optimal integration. Observers adjusted their walking direction in response to conflicting visual information, but the adjustment was modest (16%–34%) and comparable to predictions based on variability measures. The reduction in variability due to visual feedback was also consistent with optimal integration. Walking direction was more precise when global optic flow was available, but the benefit was relatively small, as would be expected given the relative reliability of heading judgments from optic flow (Experiment 1). Taken together, the results suggest that information about direction of self-motion during walking is integrated an optimal manner, and that the weak contribution of optic flow reflects the high reliability of motor and nonvisual sensory cues during active walking. 

Acknowledgments

This research was supported by a grant from the Hong Kong Research Grants Council, GRF HKU-750209H. I would like to thank Wayne Zhang and Xing Xing for assistance with the virtual reality environment, and Celina Wong and Sudharshan Ravindran for assistance with data collection. 

Commercial relationships: none. 

Corresponding author: Jeffrey A. Saunders. 

Email: [e-mail]. 

Address: Department of Psychology, University of Hong Kong, Hong Kong, Hong Kong SAR.