Integration of auditory and visual cues in spatial navigation under normal and impaired viewing conditions

Abstract

Auditory landmarks can contribute to spatial updating during navigation with vision. Whereas large inter-individual differences have been identified in how navigators combine auditory and visual landmarks, it is still unclear under what circumstances audition is used. Further, whether or not individuals optimally combine auditory cues with visual cues to decrease the amount of perceptual uncertainty, or variability, has not been well-documented. Here, we test audiovisual integration during spatial updating in a virtual navigation task. In Experiment 1, 24 individuals with normal sensory acuity completed a triangular homing task with either visual landmarks, auditory landmarks, or both. In addition, participants experienced a fourth condition with a covert spatial conflict where auditory landmarks were rotated relative to visual landmarks. Participants generally relied more on visual landmarks than auditory landmarks and were no more accurate with multisensory cues than with vision alone. In Experiment 2, a new group of 24 individuals completed the same task, but with simulated low vision in the form of a blur filter to increase visual uncertainty. Again, participants relied more on visual landmarks than auditory ones and no multisensory benefit occurred. Participants navigating with blur did not rely more on their hearing compared with the group that navigated with normal vision. These results support previous research showing that one sensory modality at a time may be sufficient for spatial updating, even under impaired viewing conditions. Future research could investigate task- and participant-specific factors that lead to different strategies of multisensory cue combination with auditory and visual cues.

Introduction

Successful navigation is a critical skill for everyday life. There are multiple sources of sensory information that could serve as cues for navigation, including vision, hearing, and the proprioceptive and vestibular senses associated with active movement (here, termed body-based self-motion cues). There is inherent uncertainty in these sensory systems in the form of noise, so it is logical that humans might benefit from combining sensory information to improve navigation precision. Some work suggests that humans combine sensory cues from multiple sensory systems optimally to effectively navigate (Bates & Wolbers, 2014; Chen, McNamara, Kelly, & Wolbers, 2017; Nardini, Jones, Bedford, & Braddick, 2008; Sjolund, Kelly, & McNamara, 2018). Optimal cue combination, in this context, is the combination of multiple sensory streams such that the multisensory percept has reduced variability (or noise) compared with each of the individual sensory percepts. Importantly, not all research demonstrates the optimal combination of sensory cues for navigation. Nardini et al. (2008), for example, found that children typically alternate which sensory systems they use to navigate, such that they are only using one sensory modality at any one moment in time, resulting in higher navigation error and variability. Nardini et al. (2008) defines a non-optimal probabilistic model for predicting the variability of multisensory performance based on performance with two unisensory cues called the Alternation model. Bates and Wolbers (2014) and Shayman et al. (2024b) both found that, although older adults may not alternate which systems they rely on, their navigation performance is sub-optimal, again resulting in decreased navigational performance compared with younger adults. Zhao and Warren (2015) found that cues may be integrated in some contexts of navigation but compete in others. 

The field of cue combination in navigation research has largely focused on the combination of visual landmarks and body-based self-motion cues; however, there is growing evidence that auditory cues, in the form of either multisensory home location redundancy (Shayman et al., 2024c) or landmarks (Jetzschke, Ernst, Froehlich, & Boeddeker, 2017; Zanchi, Cuturi, Sandini, & Gori, 2022), play a role in spatial updating. Shayman et al. (2024c) found that participants' spatial updating accuracy increased when a visual target was supplemented with a spatial auditory cue, and that participants with the worst spatial updating performance had the greatest benefit from sound. Although this study focused on supplementing the target with audition similar to the auditory beacon literature (Clemenson, Maselli, Fiannaca, Miller, & Gonzalez-Franco, 2021), several other studies have examined the contributions of auditory landmarks to successful navigation. Jetzschke et al. (2017) found that auditory landmarks function similarly to visual landmarks for spatial updating during navigation and that increasing the number of auditory landmarks facilitated updating even more. The authors used differing amplitude-modulated white noise as auditory landmarks while participants walked on a treadmill in virtual reality with the orientation of their head controlling their walking direction in the virtual environment. Participants' accuracy and variability of homing was measured with both visual and auditory landmarks and Jetzschke et al. (2017) found that performance was similar regardless of landmark type. Although this study discussed whether these cues might be combined optimally, they did not test the integration of audition and vision. 

More recently, Zanchi et al. (2022) used a probabilistic modeling approach to test auditory and visual cue integration for navigation. They found that the addition of auditory landmarks to visual landmarks did not improve navigation performance over visual landmarks alone across all participants. Furthermore, participants in Zanchi et al. (2022) showed large inter-individual differences in cue combination weighting strategy. Approximately one-half of the participants relied similarly on auditory and visual cues. For this subset of participants, variability was decreased when participants had access to two cues compared with when using each cue alone. This result shows that, for those using vision and audition to a similar degree, having both cues helped participants to be less variable with spatial updating during navigation. However, even when there was a benefit in decreased variability with two cues compared with one, multisensory homing accuracy did not improve relative to participants' performance with their just their best single cue (visual landmarks). Garcia et al. (2017) tested whether the integration of visual and auditory cues for cue localization in the yaw plane followed a model of optimal cue combination. Although the results are not directly comparable with the navigation literature because participants performed the task while seated, they found that participants with normal vision and peripheral low vision optimally integrated audiovisual spatial cues. However, participants with long-standing central low vision (mostly from Stargardt disease) did not optimally combine audiovisual cues. This experiment demonstrated that individuals diagnosed with low vision who have long-standing experience navigating without visual information may not optimally combine audiovisual cues. 

Understanding the role of landmark reliability in multisensory integration is of critical importance, especially for individuals with impairments in sensory acuity. Hundreds of millions of individuals worldwide have vision loss that cannot be fully corrected (Bourne et al., 2017). Research has shed some light on how low vision may or may not influence spatial updating and path integration (Creem-Regehr, Barhorst-Cates, Tarampi, Rand, & Legge, 2021). For example, Legge, Granquist, Baek, & Gage (2016b) found that low vision did not impact spatial updating. Their homing experiment showed similar performance between participants with normal acuity, low vision, and congenital blindness. They concluded that spatial updating may be amodal in nature. However, when auditory information was deprived, participants with congenital blindness (not low vision) seemed to be the least affected. Rand, Barhorst-Cates, Kiris, Thompson, and Creem-Regehr (2019) also investigated whether low vision influenced spatial cognition. In their study, participants with acutely and severely simulated degraded vision were less accurate at distance perception while navigating. However, these results hinged on participants learning a route with simulated degraded vision and then making a perceptual judgment with normal vision. Despite these findings, the role of audition for spatial updating in instances of low vision has not been well-characterized. Specifically, we do not know how audition is integrated with vision for navigation, particularly in cases of low vision, as well as how much audition is weighted relative to normal or impaired vision. 

Experimental framework and hypotheses

Here, we assess the role of audition in cue combination with vision during navigation across two experiments. In the first, we conceptually replicate the study of Zanchi et al. (2022) in immersive virtual reality instead of the real world with additional trials and target locations to increase the difficulty of the task. In the second, we introduce a simulation of profound visual impairment in an identical paradigm with a new group of participants to test how navigation performance, cue weighting, and sensory integration strategy may change with increased perceptual uncertainty in the form of acute visual impairment. Specifically, with impaired vision, participants' variability in performance with visual-only navigation will likely increase. But, because of the lack of experience with lower reliability of vision, they still may rely on visual cues more when both auditory and visual cues are available. In other words, the observed weighting of auditory cues may be less than what is predicted. Across these experiments, we hypothesize the following: 

• 1. Participants will receive multisensory (auditory + visual) benefits to homing accuracy and variability (revealed by reduced error and variability) over auditory-only or visual-only navigation regardless of whether they are navigating under conditions of normal or impaired visual acuity.
• 2. During conditions of normal visual acuity, participants will weight, or rely on, auditory cues less than visual cues for spatial updating during navigation.
• 3. Participants will optimally integrate vision and audition for navigation according to principles of maximum likelihood estimation when they are navigating with normal visual acuity, but will suboptimally integrate their senses when navigating under conditions of acute impaired visual acuity because of the unfamiliarity with unreliability of visual cues.
• 4. Participants' navigation performance (accuracy and variability) will be worse with acute impaired visual acuity.
• 5. Participants will still weight, or rely on, auditory cues less than visual cues for spatial updating during conditions of acute impaired visual acuity, but will demonstrate increased auditory weighting relative to the group navigating without impaired visual acuity.

Experiment 1

Method

Participants

An a priori power analysis was based on the sample size used in Bates and Wolbers (2014) and Zanchi et al. (2022) and medium effect sizes (\(\eta _{\text{G}}^{2}\) = 0.11−0.20) reported in similar studies with young adults (Bates & Wolbers, 2014; Chen et al., 2017; Sjolund et al., 2018). We determined that 20 to 24 participants would be sufficient to find within- and between-group effects at 80% power, so our recruitment target was set at 24 participants in Experiment 1 (and later in Experiment 2) with replacement as needed for participants whose data were excluded owing to noticing the covert conflict condition manipulation as is standard practice in studies of multisensory integration. In the end, 29 participants performed the study and 5 participants were excluded from Experiment 1 for this reason. Here, 24 participants (17 women, M_age = 22.5 years, SD = 5.8) were included. The study protocol was approved by the university’s institutional review board, and all participants gave written, informed consent to participate. All procedures strictly adhered to the Declaration of Helsinki. Participants formed a sample of convenience, recruited from the Department of Psychology’s participant pool and by word of mouth. Participants were compensated for their 2 hours of participation with either course credit or an hourly wage ($20/hour). 

All participants reported normal hearing and vision. Further, all participants completed Snellen vision chart screening and had either normal or corrected-to-normal (20/40) visual acuity. Participants all passed a cognitive screen, the Mini-Mental Status Exam (normal > 24/30), and were able to stand on foam with their feet together and eyes closed for 30 seconds (Romberg test). All participants answered questions about their hearing including, “Do you hear equally out of both of your ears?”, “Have you ever been concerned about your hearing?”, and “Have you ever been diagnosed with a hearing problem?” No participant indicated a known problem or asymmetry with their auditory acuity. As an additional measure of spatial hearing ability, all but two of the participants, owing to a technical error, completed an auditory walking task in which they were asked to walk toward the auditory stimuli (described later) that were used as spatial landmarks in the experiment. Auditory stimuli were played one at a time in the absence of spatial visual cues, and participants were asked to take three steps toward the auditory stimulus. In total, participants completed eight trials of this task (walking toward both of the auditory stimuli played out of each of the four speakers that were used for localized sound). Unsigned angular error, calculated as the angular deviation from the correct path formed between the starting location and the target stimulus, was averaged for each participant for all eight trials. All participants were able to walk toward the stimuli within 15° of error (mean, unsigned angular error = 4.0°, SD = 2.3°, range: 0.3°–9.2°). Fifteen degrees was used as a cutoff to determine that participants had sufficient sound localization ability to participate in the study because the spatial conflict (described elsewhere in this article) was exactly 15°. 

Homing task procedures, materials, and design

All testing was performed in a room 8 m × 9 m in size. Participants were brought into the room with their eyes closed and never saw the physical space in which the task took place. All participants put on the head-mounted display (HMD, HTC Vive Pro with wireless capabilities) before opening their eyes. The ground plane of the virtual environment was a grassy field that extended infinitely with a horizon and no cast shadows for objects. A rendering of this environment can be seen in Figure 1. Visual landmarks consisted of two poster signs (dimensions: 36 cm wide × 50 cm tall): a dog and a duck. The top of the landmark board to the ground plane measured 1.45 m. From the origin of the environment (waypoint 2; see Figure 2) the dog landmark (left) was 4.5 m away at an angle of 15° to the left and the duck landmark (right) was 4.5 m away at an angle of 30° to the right. Fourteen possible target and waypoint 1 locations were located either 3 or 1.5 m from the origin at the following angles: 60° (left), 40° (left), 20° (left), 0°, 20° (right), 40° (right), and 60° (right). The second waypoint was always in the same location and was both the origin and calibration location of the environment. Of many possible triangular homing configurations, 40 such configurations were chosen such that path lengths ranged from 1.5 m to 4.0 m. Some of the configurations were repeated once as there were 68 total trials of the homing task. All participants began each homing trial from 1 m behind the origin. 

Figure 1.

 

Virtual environment as seen from the perspective of a participant wearing the HMD. The home (target) location can be seen in black on the ground plane. Waypoints 1 and 2 are red and blue shapes, respectively. Visual landmarks are visible as posters on stands in the distance in the environment.

Figure 2.

 

An overview of the target/way point/landmark layout (left) and trial progression (right) viewed from an overhead perspective. Though the icons for auditory and visual landmarks are stacked in the figure for display purposes, they were spatially co-located in the environment (except when provided in conflict).

Auditory landmarks were co-located with the visual landmarks and consisted of semantically congruent auditory stimuli as per Zanchi et al. (2022) playing out of two distinct speakers. Specifically, the height of the center of the speakers was calibrated to align with the top of the visual landmarks at 1.45 m above the ground. Orb Audio speakers (Orb Audio, CA) powered by a 50-W four-channel amplifier (Behringer, Germany) were used. Ableton Live 11 (Berlin, Germany) was used with an external 4-channel sound card (Motu M4, Cambridge, MA) plugged into a Macbook Pro (Apple, Cuppertino, CA) to play the audio files. Auditory landmarks consisted of two identical sets of a duck “quack” sound (right) and a dog “bark” (left) sound, similar to a previous auditory-landmark navigation study (Zanchi et al., 2022). Sounds were downloaded from a royalty-free archive (https://freesound.org) and were modified to have distinct spectral peaks, with the bark sound ranging in frequency from 40 to 4 kHz (peak: 500 Hz) and the quack sound ranging from 500 to 10 kHz (peak: 1.5 kHz). Auditory streams were looped such that they provided a steady signal (except for when the landmarks were disabled during the waiting period of the homing task, or during the response phase of the vision condition). Speakers were calibrated using a sound-level meter (REED Instruments, Wilmington, NC). The bark sound was set to 65.6 dB(A) at 1 m and the quack sound to 65.2 dB(A) at 1 m, such that they were perceptually equal in loudness. The sound field signal-to-noise ratio (comparing the brown noise background and both auditory landmarks to the brown noise alone) at the origin of the homing task was 4.9 dB(A). The second set of speakers used for the shifted landmarks in the conflict condition was calibrated identically. Additional speakers were used to mask room sounds and provide continuous sound to participants, even when auditory landmarks were disabled. Overhead brown noise (calibrated to 60 dB(A)) was played through an array (4.5 m × 5 m) of six ceiling-mounted speakers (JBL, Los Angeles, CA). The overhead noise was centered on the origin of the homing task and constantly played for the duration of the experiment. It prevented participants from hearing the experimenters interact with the computer or walk around the room (i.e., potential aids for localization). 

The homing task was run as in classic cue combination paradigms (Bates & Wolbers, 2014; Chen et al., 2017; Nardini et al., 2008; Zanchi et al., 2022). Participants walked from the start location to the target (home) location, then via two other path segments (via waypoint 1 and waypoint 2) forming a triangular path. At the second waypoint, one of four condition-specific manipulations was performed while the HMD was blackened and auditory landmarks were extinguished. Four conditions were used as listed in Table 1. In all of the conditions, body-based self-motion cues were dissociated from auditory and visual landmark-based cues by spinning participants in a swivel chair for 10 seconds before them homing to the encoded target. Participants were spun slowly and equally in both directions for two full rotations to prevent confounding dizziness from velocity storage, although the spinning was broken up into smaller segments such that participants could not ascertain that they were being spun equally in both directions. In all conditions, the auditory landmarks were disabled during this segment while participants were being spun. In the audition condition, visual landmarks were not turned back on. In the vision condition, auditory landmarks were not turned back on. In the consistent condition, both auditory and visual landmarks were enabled after the period of spinning. In the conflict condition, the primary auditory landmarks remained off and a second, identical set of auditory landmarks (shifted 15° counterclockwise) were turned on following the self-motion cue dissociation. 

Table 1.

 

A description of each of sensory cue condition manipulations. ^*Self-motion cues, while not entirely removed, were dissociated from landmark-based cues in every condition.

 

Cue combination modeling and weighting calculations

As described in the Introduction, cue combination studies have used homing tasks where participants navigate using either unisensory or multisensory cues. This is done with either spatially congruent or spatially conflicting stimuli (Newman, Qi, Mou, & McNamara, 2023). Optimal integration, here, refers to the decrease in variability in performance with multisensory cues according to the principles of maximum likelihood estimation. Given that sensory inputs exhibit noisy distributions, if the brain operates optimally, there should be greater reliability (less variability) with multiple sensory cues compared to performance with a single sensory cue, and the cues should be combined following each individual cue’s reliability (Ernst & Banks, 2002). 

The predicted optimal weights for auditory (a) and visual (v) cues are determined using the relative variance (σ²) of the single cue conditions:  

\begin{eqnarray} w_{a(predicted)} = \frac{1/\sigma ^2_{a}}{1/\sigma ^2_{a}+1/\sigma ^2_{v}} = \frac{\sigma ^2_{v}}{\sigma ^2_{v}+ \sigma ^2_{a}}\quad \end{eqnarray}

(1)

Because only two sensory systems are available during the task, the weights of the two sensory cues sum to unity so that:  

\begin{eqnarray} w_{v(predicted)} = 1 - w_{a}\quad \end{eqnarray}

(2)

The predicted optimal weights are used to predict the variance when both auditory and visual landmarks are available:  

\begin{eqnarray} \sigma ^2_{a+v ({predicted\; optimal})} = w^2_{a}\sigma ^2_{a}+w^2_{v}{\sigma ^2_{v}} \quad \end{eqnarray}

(3)

Evaluating this maximum likelihood estimation model necessitates the incorporation of two multicue conditions. First, the consistent condition entails auditory and visual cues indicating the same target location. Second, the conflicting condition introduces spatial conflict, where one cue (audition) is covertly shifted, thus specifying a different target location compared with the target location encoded by the visual landmark. The conflicting condition offers insight into the relative proximity of the mean response location in the conflicting condition (μ_conflict) to the mean location of the single-cue auditory condition (μ_a) and the single-cue vision condition (μ_v). In this case, d_a is defined as the distance between the auditory single cue mean response location and that of the conflict, and d_v, the distance between the vision mean response location and the conflict.  

\begin{eqnarray} d_{a} =| \mu _{a} - \mu _{conflict} |\quad \end{eqnarray}

(4)

 

\begin{eqnarray} d_{v} =| \mu _{v} - \mu _{conflict} | \quad \end{eqnarray}

(5)

 

The observed weights are computed in the following equations:  

\begin{eqnarray} w_{a(observed)} = \frac{1/d^2_{a}}{1/d^2_{a}+1/d^2_{v}} = \frac{d_{v}}{d_{v} + d_{a}} \quad \end{eqnarray}

(6)

 

\begin{eqnarray} w_{v(observed)} = \frac{1/d^2_{v}}{1/d^2_{v}+1/d^2_{a}} = \frac{d_{a}}{d_{a} + d_{v}}\quad \end{eqnarray}

(7)

 

Alternatively, if participants switch sensory cue reliance on a trial by trial basis, known as the Cue Alternation model (Nardini et al., 2008), multisensory variance should not decrease compared with that of the single cues. This is modeled with Equation 8, where p_a is the probability of following the auditory cue and is equivalent to the auditory weight (w_a(predicted)) predicted by Equation 1. In addition, p_v is the probability of following the visual cue and is equivalent to the weight of the visual cue (w_v(predicted)) as shown in Equation 2.  

\begin{eqnarray} &&\sigma ^2_{a+v ({{predicted\; alternation}})} \nonumber\\ && = p_{a}(\mu ^2_{a}\sigma ^2_{a}) + p_{v}(\mu ^2_{v}\sigma ^2_{v}) - (p_{a}\mu _{a} + p_{v}\mu _{v})^2 \quad \end{eqnarray}

(8)

 

Data cleaning and analysis

All data and analyses are available on Open Science Framework at the following link: https://www.doi.org/10.17605/OSF.IO/8VURM. Before analysis, 65 of 1,536 trials were excluded from the dataset due to experimenter error, such as accidentally mismarking the location to which participants homed, and technical errors, such as the loss of VR HMD tracking. In total, the removed trials account for 2.41% of all trials. We also excluded trials in which participants reached the boundary of our physical space and had to be stopped for their own safety (1.82% of all trials). Analyses were run in R (v 4.1.3) (R Core Team, 2022). Mixed-effects models for accuracy and variability were performed using the analysis of variance formatting with lme4 (Bates, Mächler, Bolker, & Walker, 2015) and lmerTest (Kuznetsova, Brockhoff, & Christensen, 2017) packages. Mixed models were chosen owing to their ability to account for the nested structure of the data. Accuracy was defined as the distance between the response location in each trial to the actual target location (reported as error in cm) to preserve individual trials.¹ The waypoint 2 (see Figure 2) in each trial was treated as the origin of the coordinate system in which the y-axis was the correct walking direction. Variability was calculated as the standard deviation of the distances between each response and each participant’s mean response location (the centroid of their responses). 

To understand how auditory and visual cues were used for homing accuracy and variability, data were analyzed using mixed-effects models with random intercepts for each participant. Regression coefficients were used to understand how the single cues (auditory or visual) compared with multisensory performance. To determine whether auditory and visual cues were combined following an maximum likelihood estimation “optimal” cue-combination model, the following analyses were performed. Predicted optimal weights were calculated following Equations 1 and 2. Pearson correlation was performed with the predicted optimal consistent multisensory variability and the variability observed in the consistent condition (Equation 3). Planned paired t-tests were performed with these same sets of data. Given that frequentist statistics are unable to provide evidence in favor of the null hypotheses, JZS Bayes factors (BF₀₁) were also computed (Morey & Rouder, 2023). BF₀₁ indicates the probability, as an odds ratio, that a null hypothesis is more likely than an alternative hypothesis. Although previous studies of optimal cue combination have typically used BF₀₁’s of 3 and higher in the comparison of predicted optimal weights to observed weights to indicate strong evidence that optimal integration is occurring (Chen et al., 2017; Newman & McNamara, 2021; Zanchi et al., 2022), BFs can stand on their own as odds ratios. Our prior was a Cauchy distribution with r scale set to 0.707, which is commonly used throughout cue combination studies (Chen et al., 2017; Newman & McNamara, 2021; Shayman et al., 2024b; Zanchi et al., 2022). 

Results

Accuracy

Accuracy was examined on a trial-by-trial basis using a mixed model with a random intercept by participant to account for between-subject variability. There was a main effect of condition on accuracy, F(2,932.35) = 18.05, p < 0.01, \(\eta _{\text{p}}^{2}\) = 0.04, BF₀₁ = 1.64e-08. Accuracy in the consistent cue condition was 65.80 cm from the intended target location, β₀ = 65.80, SE = 4.66, p < 0.01. As seen in Figure 3, accuracy was significantly worse in the audition condition as response error increased by 22.15 cm compared to the consistent cues condition, β₁ = 22.15, SE = 3.97, p < 0.01. Accuracy did not differ between the vision and consistent cue conditions, β₂ = 3.59, SE = 3.96, p = 0.37. 

Figure 3.

 

Accuracy (left) and variability (right) with normal vision for three sensory cue conditions. ***p < 0.001; ns, p > 0.05. Standard error of the mean error bars are attached the mean, labeled as a bold horizontal line.

Variability

Variability was also examined using a mixed model with a random intercept by participant. There was no main effect of condition, F(2,46) = 2.32, p = 0.11, BF₀₁ = 5.25. However, planned comparisons (Figure 3) show that variability in responses was greater in the audition condition, β₁ = 6.95, SE = 3.38, p = 0.04 relative to the consistent cue condition, β₀ = 39.01, SE = 4.00, p < 0.01. Variability did not differ between the vision and consistent cue conditions, β₂ = 1.56, SE = 3.38, p = 0.65. Because the pattern of condition-based differences is the same as accuracy, the lack of a significant main effect is likely due to the magnitude of between-condition comparisons. 

Cue weighting

Overall, the observed auditory weights derived from the conflict condition ranged from 0.05 to 0.57 (Figure 4). The mean auditory weight of 0.29, SD = 0.12, indicates that, on average, participants relied more on vision than audition when performing the homing task. To examine the relationship of cue weighting, a paired samples t-test was run between the optimal integration predicted auditory weights and the observed auditory weights. Observed auditory weights derived from the conflict condition, M = 0.28, SD = 0.12, were significantly less than predicted auditory weights, M = 0.41, SD = 0.17; t(23) = 4.21, p < 0.01, d = 0.86, BF₀₁ = 0.01. As an additional measure to test whether there was a relationship between predicted weights from the optimal integration model and observed weights on a within-subject level, correlational analysis was performed. A weak but significant relationship was observed, R² = 0.20, p < 0.05 (Figure 5). Additionally, paired t-tests compared the observed SD from the consistent cue condition to the predicted SDs from the optimal and alternation models. The observed SD from the consistent cue condition, M = 39.01, SD = 16.93, was significantly greater than the optimal integration predicted SD, M = 28.94, SD = 13.42; t(23) = −4.17, p < 0.01, d = 0.85, BF₀₁ = 0.01, Figure 6, left). Importantly, observed SD was not significantly different from the alternation predicted SD, M = 42.78, SD = 19.01; t(23) = 1.30, p = 0.21, JZS BF₀₁ = 2.20 (Figure 6, right). The t-tests, correlation, and Bayesian analyses taken together suggest that participants were alternating the cues on which they relied instead of integrating them according to a maximum-likelihood estimation model. 

Figure 4.

 

Observed auditory weight derived from the conflict condition. The bold line denotes the mean with standard error of the mean error bars extending from it. The dotted line represents equivalent weighting of auditory and visual cues. Because auditory and visual weights sum to 1 (Equation 2) higher visual weights are seen on the left half of the graph.

Figure 5.

 

Predicted vs. observed auditory weight. Each participant is a single point in the plot. The correlation best fit line is shown as a solid line with the 95% confidence interval shaded in red. A parity line is dotted in black.

Figure 6.

 

Left: Predicted optimal vs. observed auditory SD. Right: Predicted alternation vs. observed auditory SD. ***p < 0.001; ns, p > 0.05.

Experiment 2

Method

Participants

A new group of 24 participants was recruited with replacement for those reporting awareness of the conflict condition (of which there were 6). Participants were recruited in an identical manner to Experiment 1. A between-subject, as opposed to a within-subject, approach was taken here to prevent learning effects. The same power analysis was used as in Experiment 1. The 24 participants, 10 of whom were female, had a mean age of 23.0, SD = 3.9 years. Again, all participants gave written informed consent, and were screened for sensory acuity in an identical manner to the first experiment. Ability to walk toward an auditory target, as per above, was collected from each of the 24 participants and was indistinguishable in results from Experiment 1, mean, unsigned angular error = 3.7°, SD = 2.7, range 0.2–11.2. 

In addition, 10 participants (6 female and 4 male; M_age = 22.60 years, range 19–28, SD = 2.72) were recruited to test the perceptual degree of visual degradation created by our simulated low vision filter. The method of validating our visual blur filter is described elsewhere in this article. These participants, recruited by word of mouth, gave informed consent and received compensation in the form of an hourly wage or course credit. 

Materials and protocol

Experiment 2 was performed exactly as Experiment 1, with one change: the introduction of simulated low vision in the form of a blur filter. A rendering of the blur filter can be seen in Figure 7. The blur filter was implemented in the Unity post-processing environment, by using Unity post process profile and Unity post-process volume (version 3.0.3). Blur effect documentation can be found here: https://docs.unity3d.com/Packages/com.unity.postprocessing@3.0/manual/Depth-of-Field.html. The focus distance was set to 5.02, aperture to 6.8, focal length to 10.6, and max blur size set to “very large.” The separate group of participants who validated the blur filter, as described elsewhere in this article, completed three tests of visual acuity. First, participants completed standard Snellen visual acuity testing at 20 feet. All participants reported no known problems with visual acuity. Participants averaged 20/19.77² visual acuity in a real-world testing setting using a Snellen chart. Participants then put on the HMD used in both Experiments 1 and 2. These participants completed standardized early treatment diabetic retinopathy study (ETDRS) visual acuity testing as is standard in clinical testing of visual acuity (Lim, Frost, Powell, & Hewson, 2010; Rosser, Murdoch, Fitzke, & Laidlaw, 2003). Participants were tested in an identical virtual environment to that of the homing task where the lighting, ground plane, and horizon were equivalent. ETDRS testing was performed at 1 m distance from the screening chart with blurred vision first (with the appropriate logarithm of the minimum angle of resolution (logMAR) correction by adding 0.6 logMAR units to the obtained score given that testing with normal vision is typical at 4 m). Participants then tested again without the blur filter, this time at 4 m distance from the chart. LogMAR acuity scores with blur ranged from 1.34 to 1.58 with a mean of 1.52, SD = 0.09. This equates to a Snellen score of 20/662.26. Without blur, logMAR acuity scores averaged 0.30, SD = 0.07; range 0.20–0.42, or a Snellen equivalent of 20/39.91. 

Figure 7.

 

A rendering of the virtual environment from the view of the HMD with the blur filter turned on. Note that although the visual landmarks are indistinguishable from one another here, most participants could accurately identify them as either a dog or a duck when standing at the location of one of the far targets. This figure can be compared with Figure 1 to visualize the severity of the visual degradation.

Data cleaning and analysis

As per Experiment 1, some trials were excluded owing to participant boundary stops (2.54%), as well as experimenter errors and technology errors (0.98%). In total, 54 of 1536 non-training trials were excluded across all 24 participants in Experiment 2. Data in Experiment 2 were analyzed in the same manner as Experiment 1 with a combination of mixed-models, Pearson correlation, t-tests, and JZS Bayes factors (BF₀₁). 

Results

Accuracy

To examine the effect of blurred vision on homing accuracy, the same mixed model from Experiment 1 was run on this new group of participants. The main effect of condition on accuracy was significant, F(2,1874.09) = 24.25, p < 0.01, \(\eta _{\text{p}}^{2}\) = 0.05, BF₀₁ = 1.06e-07. Accuracy in the consistent cue condition was 66.96 cm from the intended target location, β₀ = 66.96, SE = 5.79, p < 0.01. Similar to Experiment 1, accuracy in the audition condition was significantly worse than consistent cues as response error increase by 25.25 cm, β₁ = 25.25, SE = 3.93, p < 0.01. There was no significant difference between accuracy in the vision-only and consistent cue conditions, β₂ = 3.58, SE = 3.92, p = 0.91. These results can be visualized in Figure 8. 

Figure 8.

 

Accuracy and variability for two unisensory conditions and one multisensory consistent condition for 24 participants navigating with visual blur degradation. *p < 0.05; ns, p > 0.05. Standard error of the mean error bars are attached the mean, labeled as a bold horizontal line.

Variability

The effect of blurred vision on homing variability was examined using the same mixed model from Experiment 1. The main effect of condition on variability was not significant (F(2,46) = 2.95, p = 0.06, BF₀₁ = 3.47). Overall, the standard deviation in homing performance with the blur vision was 40.18 (β₀ = 40.18, SE = 4.00, p < 0.01). The consistent cue condition was not significantly different from audition condition (β₁ = 6.77, SE = 3.54, p = 0.06) or vision-only (β₂ = −1.20, SE = 3.54, p = 0.74). Variability data can be seen in the right panel of Figure 8. 

Cue-weighting

Overall, the auditory cue weighting in the blurred vision condition ranged from 0.05 to 0.68 (Figure 9). The mean auditory weight of 0.27, SD = 0.18, indicates that participants, on average, relied more on visual cues than auditory cues when vision was blurred. The relationship of cue weighting for blurred data was examined using a paired t-test between the predicted auditory weights from the optimal integration model and observed auditory weights. For cue weighting with blurred vision, the observed auditory weights derived from the conflicting condition, M = 0.28, SD = 0.12, were significantly less than the auditory weights predicted by a model of optimal integration, M = 0.41, SD = 0.17; t(23) = 2.96, p < 0.01, d = 0.60, BF₀₁ = 0.15. To test whether there was a relationship between predicted weights from the optimal integration model and observed weights on a within-subject level, correlational analysis was performed. No significant relationship was observed, R² = 0.01, p = 0.59 (Figure 10). Additionally, the observed SD from the consistent cue condition, M = 40.18, SD = 19.81, was significantly more than the optimal integration predicted SD, M = 28.92, SD = 12.32; t(23) = −3.40, p < 0.01, d = 0.69, BF₀₁ = 0.06 (Figure 11, left). The observed SD was not significantly different from the alternation predicted SD, M = 43.10, SD = 18.72; t(23) = 0.76, p = 0.46, JZS BF₀₁ = 3.60 (Figure 11, right). Overall, these results suggest that participants followed a model of cue alternation as opposed to maximum-likelihood optimal integration. 

Figure 9.

 

Observed weight with blur. The bold line denotes the mean with standard error of the mean error bars extending from it. The dotted line represents equivalent weighting of auditory and visual cues.

Figure 10.

 

Predicted vs. observed auditory weight with blur. Each participant is a single point in the plot. The correlation best fit line is shown as a solid line with the 95% confidence interval shaded in red. A parity line is dotted in black.

Figure 11.

 

Left: Predicted optimal vs. observed auditory SD under visual blur degradation. Right: Predicted alternation vs. observed auditory SD (also under visual blur degradation). **p < 0.01; ns, p > 0.05.

Combined experiment results

To test the effect of degraded vision on homing accuracy, we ran the same mixed model as in Experiments 1 and 2, but also included visual degradation as a between-subject variable. The effect of condition on accuracy was previously discussed in the results of both experiments. Visual degradation (blur vs. no blur) was not a significant predictor of homing accuracy, F(1,45.64) = 0.11, p = 0.75, BF₀₁ = 14.34. The interaction between visual degradation and condition was not a significant predictor of accuracy, F(1,1874.09) = 0.21, p = 0.81, BF₀₁ = 305.37. These results suggest that visual degradation does not affect the accuracy of homing performance with auditory and visual landmarks. 

Similarly, to examine whether degraded vision increased variability in homing performance, condition and visual degradation (between-subjects) were regressed onto variability. The effect of condition on homing variability was previously discussed in the results of both experiments. Visual degradation was not a significant predictor of variability, F(1,46) = 0.001, p = 0.97, BF₀₁ = 6.37. Additionally, the interaction between visual degradation and condition was not significant, F(1,92) = 0.20, p = 0.82, BF₀₁ = 16.66. These results indicate that visual degradation does not increase variability in homing performance in this task based on blur condition. 

Finally, to determine whether weighting differed between the participants who experienced the task without visual degradation in Experiment 1 and those who received blurred vision in Experiment 2, we conducted an independent t-test. Visual degradation did not change how participants weighted auditory cues across experiments, t(46) = 0.22, p = 0.83, BF₀₁ = 3.41. 

General discussion

We tested the relative role of auditory and visual landmarks in spatial updating for navigation across two cue-combination experiments: normal and simulated degraded vision. We hypothesized that having multisensory landmarks as spatial references would allow for more accurate and less variable navigation performance compared with navigating with only one modality for the landmark (vision or hearing). However, we did not find evidence to support that hypothesis, as participants were no more accurate nor less variable with multisensory cues compared with visual cues alone. This was true regardless of whether navigation was happening under normal or degraded visual acuity. We also hypothesized that participants would weight auditory cues less than visual cues during navigation regardless of whether their vision was normal or degraded. Based on the evidence gleaned from the conflict condition across two experiments, we supported this hypothesis. Auditory cues were relied on less than visual cues regardless of normal or degraded vision condition. We did not see evidence that navigation performance was worse with impaired vision. In addition, we did not find that auditory cues were up-weighted for participants navigating with impaired visual acuity. Finally, we hypothesized that performance would be based on optimal integration for participants navigating under normal viewing conditions, but be suboptimal for degraded viewing conditions. Instead, we found suboptimal integration in both cases. In fact, participants' performance aligned with a model of cue alternation. Each of these findings are discussed in greater detail. 

In both experiments, participants were no more accurate with visual and auditory landmarks as opposed to their performance with just visual landmarks. These findings align with those found previously by Legge, Gage, Baek, and Bochsler (2016a), Legge et al. (2016b), and Zanchi et al. (2022). The main differences between Experiment 1 of our study and that of Zanchi et al. (2022) were the size and scale of the homing task, the number of trials per condition, the number of potential target (home) locations, and the number of landmarks available. We had hypothesized that our task would be more difficult, and therefore allow for a greater potential separation of performance by condition, allowing us to detect a possible multisensory benefit if there was to be one. Despite increasing the number of trials per participant from 16 (4 trials in each condition) in Zanchi et al. (2022) to 56 (14 in each condition) here, no audiovisual benefit was observed compared with that of vision. In our Experiment 2, we added in a simulation of blur and the results remained essentially the same. Consistent with these results, Legge et al. (2016b) and Legge et al. (2016a) found that individuals with low vision were just as accurate as those with normal vision on a spatial updating task in the real world. Their findings suggest that visual landmarks may not be necessary for accurate homing regardless of normal or impaired vision. That is, self-motion cues may be sufficient. Body-based (non-visual) self-motion cues were removed in both experiments run here. This left participants with either one or two modalities of external spatial landmarks (visual, auditory, or both) as well as visual self-motion cues from optic flow. It may be that, similar to Legge et al. (2016a), humans are relatively adept at using only one set of spatial landmark cues for spatial updating. This may be why Kinateder and Cooper (2021), who also attempted to assess the role of simulated low vision in spatial precision, did not find vision-specific differences in a spatial reorientation paradigm. It is also possible that the optic flow information aided spatial updating (Cardelli, Tullo, Galati, & Sulpizio, 2023; Riecke, Cunningham, & Bülthoff, 2007) and compensated for the blur despite a relatively minimal cue environment. Finally, it may be that our homing task was not sufficiently challenging enough to observe a multisensory benefit even with the increase in difficulty compared to Zanchi et al. (2022). 

We found that almost all participants weighted visual landmarks more strongly than auditory landmarks. Past work on cue combination with vision and self-motion cues has shown that vision predominates (Bates & Wolbers, 2014; Chen et al., 2017; Nardini et al., 2008; Sjolund et al., 2018). Further, recent studies with auditory multisensory cue combination have shown that auditory landmarks are weighted less than visual cues (Zanchi et al., 2022) and self-motion cues in the absence of visual cues (Shayman et al., 2024a). While it is common in the navigation cue-combination literature for audition to be weighted less than other modalities, this finding is not universal in the multisensory literature. For instance, Shayman et al. (2020) found that auditory information provides greater benefit than self-motion cues at low frequencies of yaw rotation in a self-motion direction recognition paradigm. However, Werkhoven, Van Erp, and Philippi (2014) did not use a probabalistic modeling approach with a conflict condition, they found that auditory and visual landmarks provided similar wayfinding precision. Their work was done in a virtual maze task in which participants had to explore a space and then recall and draw a map of the auditory, visual, or audiovisual landmarks. Thus, it may be that more complex tasks than homing could show benefits for different modalities of cues. 

We had hypothesized that simulated low vision in the form of a blur filter would cause participants to increase their reliance on auditory landmarks (as defined by where they walk in the conflict condition). Auditory weighting in Experiment 2 did not differ from that of Experiment 1 where no blur was introduced. This is somewhat surprising, given the compensatory spatial hearing improvements seen in individuals with low vision and blindness (Battal, Occelli, Bertonati, Falagiarda, & Collignon, 2020). Still, some work exploring depth-of-field blurs in virtual reality has shown that they generally do not disrupt distance perception regardless of the HMD type (Langbehn et al., 2016). Although the navigation literature with respect to homing accuracy does not necessarily show multisensory benefit with auditory and vision (Legge et al., 2016a; Legge et al., 2016b), we still expected to see an increased reliance on auditory cues in the form of increased auditory weight in Experiment 2 given degraded visual acuity. The lack of increase in auditory weighting with blur is not surprising given that performance (in terms of both accuracy and variability) did not change with the introduction of blur. We expected performance to change given a level of visual degradation comparable to that used in other studies (using real-world blur simulations) that affected memory for target locations on a route learning task (Rand, Creem-Regehr, & Thompson, 2015), perception of distance while locomoting (Rand et al., 2019), and combat sports interaction (Krabben et al., 2021). However, some results in the literature on spatial updating with comparable blur have shown little effects on performance (Legge et al., 2016a; Tarampi, Creem-Regehr, & Thompson, 2010). It is likely that both our performance and weighting results were partially due to the task being relatively easy and that the visual landmarks remained a salient cue despite the blur. Another important reason as to why we did not see a change in auditory cue reliance is the lack of lived experience with degraded vision. Participants in Experiment 2 all had normal visual acuity and no experience navigating under the simulated degraded vision condition. It is possible that individuals who have been diagnosed with low vision and have long-standing experience with it may rely more on auditory cues. In other words, potential changes in weighting could require a period of calibration or adaptation to the low vision to occur. While this adaptation is not seen for non-navigation localization (Garcia et al., 2017), auditory weighting differences may still present during navigation for individuals with long-standing low vision. 

Last, we had predicted optimal cue combination in normal vision, but suboptimal cue combination for those navigating under acute simulated low vision. Given the growing literature on optimal (Chen et al., 2017; Newman et al., 2023; Sjolund et al., 2018) and suboptimal integration during navigation (Bates & Wolbers, 2014; Chrastil, Nicora, & Huang, 2019; Nardini et al., 2008; Shayman et al., 2024b), even with auditory and visual cue combination (Zanchi et al., 2022), it was not entirely clear whether we would observe optimal integration. Our recent work using a similar paradigm with auditory landmarks and body-based self-motion cues (without visual landmarks) showed evidence for optimal cue combination, suggesting that auditory landmarks can be integrated during navigation in some circumstances (Shayman et al., 2024a). Our current finding of suboptimal integration generally aligns with Zanchi et al. (2022), although they were able to find a sub-group of participants whose equal weighting of cues resulted in optimal cue combination. Our results in both experiments are more indicative of cue alternation (Nardini et al., 2008). Although both experiments show evidence of cue alternation, the results with our simulated degraded vision condition show stronger evidence of cue alternation (with alternation being roughly three times more likely than the null finding of non-alternation) compared with the results with normal vision (where alternation is roughly two times more likely than the null finding). We believe that a model of cue alternation may make more sense when cues are spatially redundant and used as landmark-type references. Or, it may be that participants did not treat the auditory and visual landmarks as coming from the same location (Körding et al., 2007). There are many potential explanations for patterns of suboptimality in perceptual decision-making (see Rahnev and Denison [2018] for an extensive discussion and review). Given that participants had access to co-located, semantically congruent auditory and visual landmarks on the outbound path, regardless of trial condition type, it may be that participants have a salient spatial landmark representation for spatial updating even if one modality of landmark cues are removed during the homing path. Newman and McNamara (2021) tested whether cue presentation on the outbound path vs. homing path changes integration strategy, and found that the number of cues available generally did not make a difference for uni-sensory trial type. However, this work was done with visual and self-motion cue combination, which involves both allothetic (landmark based cues) and idiothetic (body-based) cues. Our results included only landmark based cues, in which the multiple sensory systems tested also differed with auditory cues used in place of self-motion cues. 

Conclusions, limitations, and future directions

Overall, we show across two experiments of auditory and visual cue combination: visual predominance in landmark use, a lack of multisensory performance benefit (accuracy/variability) when including auditory landmarks relative to visual landmarks alone, no change in auditory weighting in conditions of simulated low vision, and strong evidence of cue alternation as opposed to optimal, maximum-likelihood cue integration. Notably, although we did not find predicted differences with simulated visual impairment, we demonstrated a clear replication of the results regarding the combination of visual and auditory cues in a spatial updating task. Our results also hint at large inter-individual differences. This is not surprising given previous literature in both the navigation (Newcombe, Hegarty, & Uttal, 2023) and cue combination (Zanchi et al., 2022) realms. For example, one paper reported gender differences in cue combination (Kelly, McNamara, Bodenheimer, Carr, & Rieser, 2009). Determining what contributes to those differences was not the goal of the present study, and future work should collect additional measures of individual differences to better elucidate differences in cue weighting strategy. Additional work should be done with those diagnosed with low vision to test whether cue weighting is an adaptive process in cases of long-standing visual loss. Previous work suggests that our results may be limited by the number of landmarks available (Jetzschke et al., 2017), the type of blur filter used (Langbehn et al., 2016), or possibly even the titration of task difficulty afforded by the relative salience of auditory and visual landmarks (Scheller & Nardini, 2023). Further, our findings supporting visual and auditory cue alternation can only generalize to the conditions of the current experiment where visual and auditory landmarks were co-located. Future studies might also examine the contribution of auditory landmarks in circumstances where they are positioned separately from visual landmarks. Although there are a number of directions for future work, our results contribute a novel finding that auditory landmarks may not help navigators decrease uncertainty under circumstances of normal or impaired viewing. 

Acknowledgments

The authors thank the following individuals for their help with data collection: Andoni Katsanevas, Linden Carter, Hollyn Gottschalk, Marcy Morse, Misty Myers, Laurie Oldfield, Gabriel Holm, and Taylor Schmidt. We also thank Nathan Seibold for his help in designing our virtual environment and the blur filter. 

Supported by the National Institute on Deafness and Other Communication Disorders under Award Number 1F30DC021360-01 (Shayman), the American Otological Society in the form of a Fellowship Grant (Shayman), and the University of Utah via the 1U4U seed grant (Creem-Regehr, Fino, Stefanucci). 

Data and analyses are available on Open Science Framework at the following link: https://www.doi.org/10.17605/OSF.IO/8VURM. 

Commercial relationships: none. 

Corresponding author: Corey S. Shayman. 

Email: [e-mail]. 

Department of Psychology, University of Utah, 380 S. 1530 E., Salt Lake City, UT 84112, USA. 

Footnotes

1  An alternate way of analyzing accuracy is to calculate the difference between the mean response location (centroid) and the actual target location. Analyses of centroids indicated the same of pattern of results using this method.

Footnotes

2  Average Snellen scores were calculated by first converting into logMAR, averaging the logMAR visual acuity across participants, and then converting back into Snellen equivalency for acuity presentation.