The ventriloquist in periphery: Impact of eccentricity-related reliability on audio-visual localization

Abstract

Abstract

Abstract:

Abstract  The relative reliability of separate sensory estimates influences the way they are merged into a unified percept. We investigated how eccentricity-related changes in reliability of auditory and visual stimuli influence their integration across the entire frontal space. First, we surprisingly found that despite a strong decrease in auditory and visual unisensory localization abilities in periphery, the redundancy gain resulting from the congruent presentation of audio-visual targets was not affected by stimuli eccentricity. This result therefore contrasts with the common prediction that a reduction in sensory reliability necessarily induces an enhanced integrative gain. Second, we demonstrate that the visual capture of sounds observed with spatially incongruent audio-visual targets (ventriloquist effect) steadily decreases with eccentricity, paralleling a lowering of the relative reliability of unimodal visual over unimodal auditory stimuli in periphery. Moreover, at all eccentricities, the ventriloquist effect positively correlated with a weighted combination of the spatial resolution obtained in unisensory conditions. These findings support and extend the view that the localization of audio-visual stimuli relies on an optimal combination of auditory and visual information according to their respective spatial reliability. All together, these results evidence that the external spatial coordinates of multisensory events relative to an observer's body (e.g., eyes' or head's position) influence how this information is merged, and therefore determine the perceptual outcome.

Introduction

Through the integration of information acquired by the different senses, our brain is able to construct a unified and more robust representation of our environment. The respective reliability of distinct sensory inputs strongly influences how they are merged in a single coherent percept (Sumby & Pollack, 1954). Probably one of the best examples of how the perceptual system deals with intersensory reliability is the ventriloquist effect (Howard & Templeton, 1966), in which the localization of auditory information is biased in the direction of a synchronously presented, but spatially misaligned, visual stimulus. In this perceptual situation, because the visual system typically provides the more accurate and reliable spatial information, the brain attributes more weight to vision in localizing the audio-visual event, thus inducing a visual capture of acoustic space (Pick, Warren, & Hay, 1969). This effect explains why although a movie actor's voice comes from loud-speakers far away from the screen, our brain recalibrates this discrepancy to give us the false impression that sound is actually coming from his mouth. Such visual capture does not occur in a rigid, hardwired manner, but follows flexible situation-dependent rules that allow information to be combined with maximal efficacy. Some experiments demonstrated that when the reliability of visual input is reduced, for example when visual information is blurred (Alais & Burr, 2004), presented at a low perceptual threshold (Bolognini, Leo, Passamonti, Stein, & Làdavas, 2007), near the onset of a visual saccade (Binda, Bruno, Burr, & Morrone, 2007) or degraded with myopia-inducing lenses (Hairston, Laurienti, Mishra, Burdette, & Wallace, 2003), the relative reliability of the auditory spatial cues increases and more weight is attributed to sounds in the localization of bimodal stimuli. Therefore, the ventriloquist effect may be considered as a specific example of near-optimal statistical (Bayesian) combination of visual and auditory spatial cues, where each cue is weighted in proportion of its relative reliability, rather than one modality capturing the other (Alais & Burr, 2004; Battaglia, Jacobs, & Aslin, 2003; Ernst & Bülthoff, 2004). 

In daily life, a frequent situation inducing a decrease in visual and auditory reliability is when the same event appears in periphery rather than in the central spatial field relative to our sensory receptors. Auditory information is less reliably localized when it is presented in the periphery versus in front of the head (Barfield, Cohen, & Rosenberg, 1997; Makous & Middlebrooks, 1990; Nguyen, Suied, Viaud-Delmon, & Warusfel, 2009; Thurlow & Runge, 1967) due to change in interaural cues (Middlebrooks, 1991). Similarly, it is more difficult for a human observer to localize visual inputs when presented in the periphery compared to the center of the visual field because the central (foveal) part of retina contains a greater density of cone photoreceptors compared to the retinal periphery, leading to a more fine-grained representation of the information being processed (Baizer, Ungerleider, & Desimone, 1991; Dacey & Petersen, 1992; Stephen et al., 2002). 

Such systematic reduction in the ability of the human observer to locate peripheral auditory and visual targets therefore affords a unique ecological opportunity to investigate how the reliability of sensory stimulations impact audio-visual integration across space (Collignon, 2007). Surprisingly, if the impact of the reliability on audio-visual integration has been massively investigated in the central visual field, only a few studies explored this phenomenon throughout the frontal space. Hairston and collaborators (2003b) studied how the absolute location in space modifies the ability of an irrelevant visual signal to influence the localization of an auditory target. They showed that auditory localization is more affected by the simultaneous presentation of a visual input coming from the center rather than from the periphery of the visual field. However, because these authors did not explore the far periphery (+30°), and because their paradigm was not designed to jointly investigate the redundancy gain (presentation of spatially aligned multisensory sources) and sensory capture (presentation of spatially misaligned multisensory sources), the impact of eccentricity-related reliability on audio-visual spatial integration remains elusive. 

The goal of the present study was therefore to investigate further how spatial signals acquired by different sensory systems are integrated in the context of their position relative to the observer. To do so, we used a design allowing us to probe how the position of audio-visual targets in the entire frontal space (180°) affect the cross-modal capture and the multisensory integration (MSI) resulting from the presentation of spatially incongruent and congruent audio-visual events, respectively. 

Methods

Subjects

Thirty-two right-handed participants (16 males; mean age of 24 years ±3; range 20–30 years) were recruited and participated in this experiment. None of the subjects reported a history of neurological or psychiatric disorders. They all had normal hearing and normal or corrected-to-normal vision. The study was approved by the Comité d'Éthique de la Recherche de la Faculté des Arts et des Sciences (CÉRFAS) of the University of Montreal. All participants provided written informed consent and received financial compensation for their participation in the study. 

Apparatus and stimuli

The entire experiment took place in a darkened, double-walled audiometric room (3.49 m ×1.94 m × 1.89 m), which was insulated by 7.5 cm width wedges on five sides (the floor was made of carpeting). Stimuli were presented on a semi-circular perimeter of 180° with a diameter of 150 cm. A height-adjustable chair and a keypad with two response buttons were installed at the center of the semicircle. The distance between the participant's head and the perimeter was 75 cm. Thirteen white light-emitting diodes (LEDs) and 13 small loudspeakers (Beyer Dynamics, DT-48) were positioned on this unit. A LED was placed directly above each speaker at eye level of the participant. Each pair, composed of a LED and a speaker, was separated by 15° of visual angle. These pairs were located at 0°, ±15°, ±30°, ±45°, ±60°, ±75°, and ±90° relative to the central fixation point, with positive values representing areas to the right and negative values representing areas to the left of the fixation point (Figure 1A). An additional red LED was used as a fixation spot placed at 0°, just above the central target white LED. The whole perimeter was covered with a semi-transparent black fabric so that the participants could not see the speakers and the LEDs when they were turned off. 

Figure 1

 

(A) Schematic representation of the apparatus. Thirteen LEDs and 13 small loudspeakers were positioned on a 180° semi-circular perimeter. (B) Schematic example of an experimental trial. Pairs consisting of a LED placed above a speaker were located at 0°, ±15°, ±30°, ±45°, ±60°, ±75°, and ±90° relative to the fixation point of the subject. The probe could originate from seven different locations, either 0°, ±15°, ±45°, or ±75° from midline, while the comparison stimulus was presented either to the right (+15°) or/and to the left (−15°) of this reference.

Visual stimuli consisted in a 15-ms illumination of a white LED (48 cd/m²). Auditory stimuli were 15-ms bursts of white noise (5-ms rise/fall time, 5-ms plateau). The level of intensity was adjusted to 65 dB SPL at the position of the participants' head. Bimodal (audio-visual) stimuli consisted in the simultaneous occurrence of a visual and an auditory stimulus. Stimuli delivery was controlled with Presentation software (Neurobehavioral Systems, Inc., Albany, CA). 

Procedure

A pair of two stimulations (always consisting of a probe and of a target stimulus) was presented on each trial (see Figure 1B for an example). The first stimulation (probe) was composed of an audio-visual pair, in which the auditory and visual stimuli were always delivered simultaneously at the same location. The probe could originate from seven different locations: 0°, ±15°, ±45°, or ±75° of eccentricity from midline. The second stimulation (target) could be (a) a visual stimulus alone, (b) an auditory stimulus alone, or (c) both visual and auditory stimuli presented simultaneously in a congruent or incongruent fashion. For the congruent matching, the visual and auditory stimuli were presented at the same location. In contrast, for the incongruent matching, the visual and auditory stimuli came from separate eccentricities. The target stimulus was presented between 100 and 150 ms after the probe either to its right (+15°) or to its left (−15°). For the incongruent conditions, one component of the target stimulus (e.g., visual stimulus) was presented to the right (+15°) of the probe while the other (e.g., auditory stimulus) appeared to its left (−15°). Each pair of stimuli was followed by a 1000-ms interval for response production. The central red LED was lit for 500 ms preceding the occurrence of a stimulus to ensure central fixation. 

Participants were instructed to judge as accurately and as fast as possible whether the target stimulus, which could be an auditory, a visual, or an audio-visual stimulation, was located to the left or to the right of the probe by pressing the appropriate buttons of the keypad with the index (leftward response) or the middle fingers (rightward response) of their right hand. They were asked to localize the target stimulus based on their initial reaction to the stimulation even if they perceived a conflict between the senses. 

A total of 1,680 randomly interleaved stimuli [7 (position of the probe: 0°, ±15°, ±45°, ±75°) × 2 (target's location: right, left) × 4 (modality: visual, auditory, bimodal congruent, bimodal incongruent) × 30 repetitions] were presented to each subject. These stimuli were displayed in 15 blocks of 112 stimuli lasting approximately 4 min each. Breaks were encouraged between blocks to maintain a high concentration level and prevent mental fatigue. 

Data analysis

In order to investigate the participants' ability to localize auditory, visual, and congruent audio-visual stimuli across the external frontal space, we computed the sensitivity index (d′) to obtain a measure of the discriminability between leftward versus rightward targets relative to the probe and the position bias (β). The d′ were calculated as [(z_{proportion correct} − z_{proportion incorrect})/√2] and the β as [−(z_{proportion correct} + z_{proportion incorrect})/2] (Macmillan & Creelman, 1991). Correct mean reaction times (RTs; ranging from 150 ms to 1000 ms) were also analyzed. In experiments that place equal emphasis on accuracy and processing speed, it is possible that subjects adopt different response strategies by varying RT inversely with accuracy (and thus show speed/accuracy trade-off). Therefore, overall performance may be best reflected by a single variable that simultaneously takes into account speed and accuracy. We have therefore also calculated the Inverse Efficiency (IE) scores which constitute a standard approach to combine mean RTs and accuracy measures of performance (Townsend & Ashby, 1978) and can be considered as “corrected reaction times” that discount possible criterion shifts or speed/accuracy trade-offs (Collignon et al., 2008; Röder, Kusmierek, Spence, & Schicke, 2007; Spence, Shore, Gazzaniga, Soto-Faraco, & Kingstone, 2001). The IE scores were obtained by dividing response times by the rate of correct responses separately for each condition (thus, a higher value indicates worse performance). 

MSI was investigated by calculating the redundancy gain (RG) based on d′, RTs and IE scores. RG was defined as the difference (in percent) between the mean d′, mean RTs, or IE scores obtained in the multisensory condition and the mean d′, RTs, or IE scores obtained in the best unisensory condition. RG was measured separately for each participant and level of eccentricity. Different explanations have been put forward to account for the observation of the RG. The most commons are the race and the coactivation models. The race model proposes that each individual stimulus elicits an independent detection process. For a given trial, the fastest stimulus determines the observable RT. On average, the time to detect the fastest of several redundant signals is faster than the detection time for a single signal. Therefore, the speeding up of reaction time is attributable to statistical facilitation (Raab, 1962). When the race model's prediction is violated, the speedup of RTs cannot be attributed to a statistical effect alone but some kind of coactivation must have occurred. To account for violations of the race model's prediction, the coactivation model (Miller, 1982) proposes that the neural activations of both stimuli combine to induce faster responses. Testing the race model inequality is widely used as an indirect behavioral measure of neurophysiological integrative processes underlying RT facilitation (see for example Girard, Pelland, Lepore, & Collignon, 2013; but also see Otto & Mamassian, 2012). To further investigate MSI, the race model inequality was evaluated (Miller, 1982) using the RMITest software, which implements the algorithm described at length in Ulrich, Miller & Schröter (2007). This procedure involves several steps. First, cumulative distribution functions (CDFs) of the RT distributions were estimated for every participant, eccentricity, and condition (visual, auditory, and audio-visual conditions). Second, the bounding sum of the two CDFs obtained from the two unimodal conditions (visual and auditory) were computed for each participant. This measure provided an estimate of the boundary at which the race model is violated, given by Boole's inequality. Third, percentile points were determined for every distribution of RT, including the estimated bound for each participant. In the present study, the race model inequality was evaluated at the 2.5, 7.5, 12.5, … 97.5 percentile points of the RT distributions. Fourth, for each percentile, mean RTs from redundant conditions were subtracted to the mean RTs from the bound. If those scores were above 0, the results exceeded the race model prediction and therefore supported the existence of a facilitation process (Miller, 1982; but see also Otto & Mamassian, 2012). In order to quantify sensory dominance (the ”ventriloquist effect”) in incongruent conditions, the percentage of responses toward the sound (i.e., when a sound is coming from the right side and a flash is coming from the left side of the probe, and the participant reports the stimulus as coming from the right side) was subtracted from the percentage of responses toward the flash (i.e., when a sound is coming from the right side and a flash is coming from the left side of the probe, and the participant reports the stimulus as coming from the left side) for each eccentricity separately. A positive score would therefore indicate a visual capture over audition, while a negative score would suggest the reverse. 

The relative performances obtained in visual and auditory unisensory conditions were directly compared by subtracting visual from auditory d′, RTs, and IE scores and respectively dividing the score obtained by the auditory d′, RTs, and IE scores, (d′, RTs, or IE auditory − d′, RTs, or IE visual)/d′, RTs, or IE auditory. Positive scores therefore attest higher unisensory spatial performance in vision expressed in percent of the unisensory auditory results. These scores could then be correlated with the visual capture observed in the bimodal incongruent condition in order to investigate whether the relative superiority of vision over audition in unisensory conditions relates to the ventriloquist effect. 

Since there was no difference in performances for the eccentricities to the left (−15°, −45°, −75°) and to the right (+15°, +45°, +75°) of the central fixation point, these scores were combined (see Supplementary Material for the details of the statistical analyses), at the exception of the β analysis which differ depending on which side the stimuli were presented. Statistical analyses were therefore conducted on four levels of eccentricity, which enhance statistical power by reducing the number of multiple comparisons: |0°| (central); |15°| (mean: −15°, +15°), |45°| (mean: −45°, 45°), |75°| (mean: −75°, +75°). Each result's figure (Figure 2 through Figure 6) represents the performance obtained in the full frontal space as well as the figures for the combined eccentricities. 

Figure 2

 

Left panel: d′ scores (A), RTs (B), and IE scores (see Methods) (C) for the discrimination of auditory, visual, and audio-visual stimuli as a function of the eccentricity. Right panel: relative performance for visual over auditory targets as a function of the eccentricity based on d′ scores (D), RTs (E), and IE scores (F). Error bars denote standard errors of the mean corrected for between-subjects variability (Cousineau, 2005).

Figure 3

 

β scores for the discrimination of auditory, visual, and audio-visual stimuli as a function of the eccentricity. Positive values reflected a leftward response bias, while negative values represented a rightward response bias. Error bars denote standard errors of the mean corrected for between-subjects variability (Cousineau, 2005).

Figure 4

 

RG (in percent) based on d′ scores (A), RTs (B), and IE scores (C) as a function of the eccentricity. (D) RG in function of the performance for the best unisensory component based on d′, RTs, and IE scores. Error bars denote standard errors of the mean corrected for between-subjects variability (Cousineau, 2005).

Figure 5

 

Race model inequality. Test for violation of the race model inequality (Miller, 1982; Ulrich, Miller, & Schröter, 2007). The graph represents the difference in milliseconds (on the y-axis) between the model prediction computed from the RTs of each unisensory counterpart (the model bound) and the RTs obtained in the redundant conditions. Positive values on the graph refer to RTs that were faster than the race model prediction. Negative values on the graph refer to RTs that were slower than the race model prediction. The difference between the bound and the RTs of the redundant condition are computed for each percentile of the RT distribution (on the x-axis).

Figure 6

 

(A) Ventriloquist effect as a function of the eccentricity. This score is obtained by subtracting the percentage of responses guided by hearing to the percentage of responses guided by vision for each eccentricity. Error bars denote standard errors of the mean corrected for between-subjects variability (Cousineau, 2005). (B) Ventriloquist effect as a function of the relative reliability between visual and auditory targets. This figure shows a significant positive correlation between the ventriloquist effect and the visual reliability scores for each eccentricity.

Results

Localization of auditory, visual and audio-visual congruent stimuli: General performance

Differences in general performance were analyzed by submitting d′, β, RTs, and IE scores to repeated-measures ANOVAs (4 [eccentricities: |0°|, |15°|, |45°|, |75°|] × 3 [modalities: auditory, visual, bimodal congruent]). Based on significant F values, Bonferroni post-hoc analyses were performed when appropriate. 

d′ (Figure 2A):

We observed a main effect for the “modality” factor, F(2, 248) = 153.75, p ≤ 0.0001, η² = 0.55, reflecting inferior performance for auditory stimuli compared to both visual and bimodal stimuli and inferior performance for visual than that of bimodal stimuli. We also obtained a main effect for the “eccentricity” factor, F(3, 124) = 38.69, p ≤ 0.0001, η² = 0.48. Overall, this effect revealed that performance steadily decreased with eccentricity, with superior performance when comparing eccentricities |0°| and |45°|, |15°| and |45°| or |45°| and |75°|. Finally, an “eccentricity” by “modality” interaction was also found, F(6, 248) = 3.02, p ≤ 0.05, η² = 0.07, driven by the fact that auditory performance only (not visual or bimodal) decreased between eccentricities |0°| and |15°|. 

β (Figure 3):

We found a main effect for the “eccentricity” factor, F(6, 217) = 7.18, p ≤ 0.001, η² = 0.17), with a leftward bias for stimuli presented at the eccentricities −75°, −45°, −15°, 0°, and 15° compared to 75° and for the stimuli presented at the eccentricity −75° compared to 45°. There was no main effect for the “modality” factor, F(2, 434) = 0.99, p = 0.37, η² = 0.005), and no interaction was identified between the factors “modality” and “eccentricity,” F(12, 434) = 1.71, p = 0.06, η² = 0.05. 

RTs (Figure 2B):

First, a main effect was found for the “modality” factor, F(2, 248) = 114.94, p ≤ 0.0001, η² = 0.48, with higher RTs for auditory stimuli compared to both visual and bimodal stimuli and higher RTs for visual than bimodal stimuli. We also observed a main effect for the “eccentricity” factor, F(3, 124) = 2.76, p ≤ 0.05, η² = 0.06, revealing a general slowdown of RTs with eccentricity. Finally, an “eccentricity” by “modality” interaction was found, F(6, 248) = 10.11, p ≤ 0.0001, η² = 0.20, revealing that only RTs to auditory targets were not influenced by the eccentricity of the stimuli (p = 1.0 for all comparisons). 

IE scores (Figure 2C):

We observed a main effect of the factor “modality,” F(2, 248) = 204.58, p ≤ 0.0001, η² = 0.62, with worst performance for auditory stimuli compared to both visual and bimodal stimuli and a worst performance for visual than bimodal stimuli. We also found a main effect for the “eccentricity” factor, F(3, 124) = 20.52, p ≤ 0.0001, η² = 0.33, again showing a decrease of performance with eccentricity. No interaction was identified between the factors “modality” and “eccentricity,” F(6, 248) = 2.65, p = 0.13, η² = 0.04. 

The relative unimodal performance of vision over audition (see data analysis section for details) was compared across the eccentricities using one-way ANOVAs. When based on d′ scores, the relative unimodal performance of vision over audition changed across the eccentricities, F(3, 127) = 6.01, p ≤ 0.005, η² = 0.13, with lower visual dominance for eccentricity |0°| than |45°| (Figure 2D). If measured using the RTs, the relative performance of vision over audition significantly decreased in periphery, F(3, 127) = 14.59, p ≤ 0.001, η² = 0.26), with higher visual dominance when comparing eccentricities |0°| than |45°|, |15°| than |45°| and |45°| than |75°| (Figure 2E). No significant effect of eccentricity, F(3, 127) = 2.12, p = 0.10, η² = 0.05) was found when the relative performance of vision over audition was calculated based on IE scores, even if the scores tended to be lower at the largest eccentricity (|75°|; Figure 2F). 

Localization of spatially congruent audio-visual stimuli: Redundancy gain

There was no significant difference in RG throughout the different eccentricities when derived from the d′, F(3, 124) = 2.32, p = 0.08, η² = 0.05 (Figure 4A), or the IE scores, F(3, 124) = 0.23, p = 0.88, η² = 0.006 (Figure 4C). If measured using the RTs, RG showed fluctuations according to the degree of eccentricity, F(3, 124) = 4.55, p ≤ 0.005, η² = 0.10, with higher RG for eccentricity |0°| and |15°| compared to |75°| (Figure 4B). A positive difference (meaning a violation of the race model prediction) was observed between the redundant condition and the probabilistic bound for the first three percentiles of the RT distribution at eccentricity |0°| and for the first percentile at eccentricity |15°|. No violation of the race model estimation was found for the eccentricities |45°| and |75°| (Figure 5, see Supplementary Material for the figure representing the results obtained in the full frontal space). 

In order to further investigate the association between the RG and the localization performance for unisensory information, we correlated (using Pearson product-moment correlation coefficient) the RG with the mean d′, RTs, and IE scores for the best unisensory modality across eccentricities (Gingras, Rowland, & Stein, 2009). According to the principle of inverse effectiveness (IE), stating that the multisensory gain produced by the integration of separate sensory estimates is enhanced when the reliability of these stimuli is low (Stein & Meredith, 1993), these variables are generally inversely correlated; that is, lower accuracy in localizing single stimuli is associated with maximal RG when adding another stimulus (Stanford, Quessy, & Stein, 2005; Stein & Meredith, 1993). We obtained no significant correlation between the RG and the best unisensory d′ (r = −0.33, p = 0.47), RTs (r = −0.72, p = 0.07), or IE scores (r = −0.27, p = 0.55) (Figure 4D), meaning that RG was relatively constant regardless of the reliability of the best unisensory component. The marginal tendency observed with RTs was in opposite direction of the predictions made from the IE principle: Shorter RTs in the best modality (at more central location) were associated with a higher RG. 

Localization of spatially incongruent audio-visual stimuli: The ventriloquist effect

We observed a main effect of the factor “eccentricity,” F(3, 124) = 17.67, p ≤ 0.0001, η² = 0.30, revealing a decrease in the ventriloquist effect with eccentricity. Post-hoc comparisons demonstrated a higher visual capture for eccentricity |0°| compared to |45°| and |75°|, for eccentricity |15°| compared to |45°| and |75°|, and for eccentricity |45°| compared to |75°| (Figure 6A). 

In order to investigate the association between the ventriloquist effect and the relative reliability of unisensory visual over auditory stimuli, we correlated the relative reliability of visual over auditory unimodal information in d′ scores, RTs, and IE scores (reflecting both the discriminability and processing speed of the stimuli) with the visual capture effect separately for each eccentricity. For all eccentricities, we obtained a positive correlation between the ventriloquist effect and the scores of visual reliability based on IE scores [|0°| (r = 0.63, p ≤ 0.001), |15°| (r = 0.70, p ≤ 0.001), |45°| (r = 0.69, p ≤ 0.001), and |75°| (r = 0.77, p ≤ 0.001); Figure 6B], d′, [|0°| (r = 0.42, p ≤ 0.05), |15°| (r = 0.21, p = 0.24), |45°| (r = 0.32, p = 0.07), and |75°| (r = 0.18, p = 0.33); Supplementary Figure S2A] and RTs [|0°| (r = 0.55, p ≤ 0.001), |15°| (r = 0.72, p ≤ 0.0001), |45°| (r = 0.62, p ≤ 0.0001) and |75°| (r = 0.56, p ≤ 0.001; Supplementary Figure S2B], meaning that individuals having higher visual localization abilities relative to auditory ones also show an enhanced visual capture at every level of eccentricity. 

Control experiment

In the main experiment, the probe always consisted in an audio-visual pair of stimuli while the target could either be a unimodal (auditory or visual) or a bimodal (audio-visual) stimulation. Therefore, the unisensory trials were not purely unimodal as they contained a bimodal probe, and the amount of noise associated with the two types of multisensory pairs was not equal to that associated with the unisensory auditory and visual pairs. In order to test if the differences we observed between unisensory and multisensory stimulations was associated to the difference in the level of noise between those conditions, we conducted a control experiment in which the first stimulation of each pair was always presented in the same modality as the second stimulus of the pair (either auditory, visual, or audio-visual), instead of always being an audio-visual stimulation (Supplementary Figure S3A and S3B). 

Results (and related statistics) are presented in the Supplementary Material (Supplementary Figure S4 through S8) and are consistent with what was found in the main experiment. Therefore, using a constant level of noise for each pair of stimulation, we observed that despite a strong decrease in auditory and visual unisensory localization abilities in periphery (see Supplementary Figure S4A through S4C), the RG resulting from the congruent presentation of audio-visual targets was not affected by stimuli eccentricity (see Supplementary Figure S6). We also confirmed that the ventriloquist effect steadily decreased with eccentricity (see Supplementary Figure S8A), paralleling a lowering of the relative reliability of unimodal visual over unimodal auditory stimuli in periphery (see Supplementary Figure S4D through S4F), and that it positively correlated with a weighted combination of the spatial resolution obtained in unisensory conditions (see Supplementary Figure S8B). 

Discussion

By exploring how the spatial location of auditory and visual targets affects both the multisensory gain and the ventriloquist effect, the current study shed new lights on how the natural eccentricity-related changes in reliability of audio-visual stimuli impact their integration across the entire external frontal space. 

We first examined how the position of auditory-alone and visual-alone stimuli influences our spatial discrimination abilities. Regarding the localization of the visual stimuli, we observed higher discriminability (Figure 2A) and faster RTs (Figure 2B) when the information was presented in the center of the frontal field rather than in the periphery. This reduction in the reliability of visual information with eccentricity is consistent with the fact that, through its physiological properties (higher density of cones than rods in the fovea and the reverse in the periphery of the retina), the visual system has a better spatial resolution for central vision (Baizer et al., 1991; Dacey & Petersen, 1992; Stephen et al., 2002). When looking at the ability to localize the auditory stimuli, we also observed a severe reduction in the discriminability of information presented in the periphery (Figure 2A). This finding is in accordance with studies showing that higher auditory spatial resolution is achieved for information presented directly in front of the head, due to the spatial dependence of the interaural difference cues (Blauert, 1997; Middlebrooks, 1991). Interestingly, the RTs for the auditory targets remained similar across all the frontal space (Figure 2B), which therefore contrasts with the results obtained in vision where eccentricity similarly affected both the discriminability and the RTs of the stimuli. These results suggest that, at least in terms of RTs, the detrimental effect of peripheral stimuli appearance is lower in audition than in vision (see also Figure 2E–F). This outcome might be related to studies reporting that saccadic reaction time toward auditory (but not visual) stimuli accelerated as a function of target eccentricity, whereas accuracy decreased (as for visual targets) with increasing eccentricity (Gabriel, Munoz, & Boehnke, 2010; Yao & Peck, 1997), suggesting that there is a speed-accuracy trade-off in the localization of peripheral auditory targets. The mechanisms underlying this effect still remain elusive, but have been suggested to be related to spatial receptive field attributes of neurons in the superior colliculus (Zambarbieri, 2002; Zambarbieri, Beltrami, & Versino, 1995; Zambarbieri, Schmid, Magenes, & Prablanc, 1982), as it changes with eye position or when the fixation point disappear before the onset of the target. 

The first main objective of the current study was to explore the impact of the eccentricity-related reliability of auditory and visual information on the RG. Based on the inverse efficiency (IE) principle in MSI stating that the multisensory gain produced by the integration of separate sensory estimates coming from the same event is enhanced when the reliability of these stimuli is low (Stein & Meredith, 1993), one may have expected a positive relation between the RG and the eccentricities of the targets since the reliability of visual and auditory information dramatically decrease when presented in periphery (Figure 2C). Moreover, the observation of direct projections between caudal region of the auditory cortex and visual areas representing peripheral visual space in monkeys (Falchier, Clavagnier, Barone, & Kennedy, 2002; Rockland & Ojima, 2003) and humans (Beer, Plank, & Greenlee, 2011) provided a neurophysiological support to hypothesize enhanced audio-visual integration in periphery. Surprisingly, we did not observe a significant influence of the eccentricity of our targets on the RG derived from d' or IE scores (Figure 4A and Figure 4C). Rather, we even observed a stronger RG for central locations with RT measurements (Figure 4B and Supplementary Figure S4B). In support of these results, we observed that for the fastest latencies (percentiles) of the RT distributions, the RT probability in the bimodal condition exceeded the probabilistic sum of the RT observed in the auditory or visual unisensory conditions (Figure 5 and Supplementary Figure S7; Miller, 1982; but see also Otto & Mamassian, 2012). This violation of the race model prediction was however not observed for more peripheral locations (|45°| and |75°|), which indicates a reduced ability to integrate separate sensory information when presented in the periphery of the frontal space. 

Gingras and collaborators (2009) examined the performance of cats in localizing auditory, visual, and spatio-temporally congruent audio-visual stimuli at different eccentricities of the frontal space (±45°). They found that the enhanced performance obtained with audio-visual stimuli was inversely proportional to the best single stimulus accuracy at each location, a result which is consistent with the IE principle. In contrast with these results and reinforcing the hypothesis that the IE principle is not observed in our experiment, we found no correlation between the performance for the best unisensory modality and the RG (Figure 4D). 

As such, we show that IE does not strictly apply to the multisensory enhancements seen during audio-visual localization in humans. These results therefore contradict simple models predicting that the rules of IE governing MSI derived from unicellular studies in the brainstem (see for example Rowland, Stanford, & Stein, 2007; Stein, Stanford, Ramachandran, Perrault, & Rowland, 2009) directly relates to multisensory phenomena at the level of perception and behavior (Holmes & Spence, 2005). Actually, even at a neurophysiological level, it was already suggested that setting stimulus intensity such that both significant facilitation and suppression relative to the unisensory response could reasonably arise, and measuring absolute changes of multisensory gain as a function of unisensory responsiveness, are parameters that have been shown to reduce the likelihood that the rule of IE will be followed (Holmes, 2009). 

Although IE has been shown in numerous behavioral studies (Bolognini, Frassinetti, Serino, & Làdavas, 2005; Gondan, Niederhaus, Rösler, & Röder, 2005; Stein, Huneycutt, & Meredith, 1988), several reports also failed to show support for this principle (Diederich & Colonius, 2008; Kim & James, 2010; Ross et al., 2007; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Stevenson, Krueger Fister, Barnette, Nidiffer, & Wallace, 2012). For example, Ross and colleagues (2007) investigated whether seeing a speaker's articulatory movements would influence a listener's ability to recognize spoken words under noisy environmental conditions, and found that the redundancy RG is higher at intermediate auditory signal-to-noise ratios (and not at lowest signal-to-noise ratios as predicted by the IE principle). The authors interpreted their results by suggesting that there is a maximal integration window at intermediate signal-to-noise ratios, where the perceptual system can build on either the visual or the auditory modality. These findings appear to be consistent with those from our experiment, as RG tended to be greatest at intermediate eccentricities (|15°|, |45°|; see Figure 4C). A reason why these results were not observed from previous studies investigating audio-visual perception across the frontal space (Bolognini et al., 2005) could be that none of them presented the information in the far periphery (>55°). Therefore, these studies cut off approximately at the same eccentricity at which we observed a maximal gain. 

Another related question is whether the same mechanisms are implicated when comparing multisensory enhancement in the same brain receptive field (RF) or across different RFs. The vast majority of experiments explored the multisensory enhancement at a unique position in space while changing the level of stimulus intensity, therefore activating a unique RF with different level of intensity. However, in the current study, we varied the strength of the sensory signal by presenting it through different RFs having different spatial resolution. It might therefore be interesting to investigate if different level of stimulus intensity (i.e., high and low intensity) at each of the eccentricities may induce performances compatible with the principle of IE. 

The second main interest of the current study was to explore the impact of eccentricity-related reliability of auditory and visual information on the ventriloquist effect. The ventriloquist effect is a perceptual condition wherein a visual stimulus biases (captures) the localization of a spatially incongruent sound. It was first thought to reflect a winner-take-all advantage that favors visual over non-visual spatial information because vision typically provides the most reliable information for space perception (Welch & Warren, 1980). However, more recent studies demonstrated that information from independent sensory channels is integrated with a near-optimal or Bayesian strategy, such that the perceived location of the sound rather depends on the weighted sum of the sensory cues based on their respective reliability (Alais & Burr, 2004; Ernst & Bülthoff, 2004). In the spatial domain, Hairston et al. (2003b) examined the ability of a visual signal to influence the localization of an auditory target according to their eccentricity (±40° in fontal space). They found that the visual bias was strongly dependent on the location of the visual stimulus, with greatest bias in the center of visual space where the reliability of the visual targets is the greatest. Moreover, Teramoto and collaborators (2012) showed that the perceived direction of visual motion could be modulated by the presence of a moving sound and that the more the visual stimuli originated from the periphery, the stronger the effect. The current study supports and extends these results by demonstrating that the ventriloquist effect steadily decreases with eccentricity over the whole frontal space (Figure 6A). Moreover, we show that this reduction of visual capture is associated with more similar unimodal localization performance between vision and audition in periphery (Figure 2D-F; Supplementary Figure S4D through S4F). It is however important to note here that regarding the relation between the relative unisensory performances for visual over auditory targets, we observed a speed/accuracy tradeoff effect, as the performance based on d' increased with eccentricity and the opposite effect was found for RTs (Figure 2D through E). The combination of both scores in the IE scores still indicated a reduction in visual dominance as a function of the eccentricity (Figure 2F). 

We also demonstrate for the first time that the ventriloquist effect strongly correlates with the relative unisensory reliability between visual over auditory information for each eccentricity across the frontal space (Figure 6B): The better the unisensory localization abilities in vision compared to audition, the stronger the ventriloquist effect. These results further bolster the idea that, based on the relative spatial reliability of auditory and visual unisensory information, the ventriloquist effect results from a weighted combination of the information gathered from the two modalities (Alais & Burr, 2004). Audio-visual integration in space is therefore not an absolute process but relative to the location of the multisensory event from the observer's body position, which location differently impact on the reliability of the unisensory components. 

It is hypothesized that the topography of the auditory space is constantly calibrated by retino-visuomotor feedback in order to maintain stable spatial alignment with the visual space (Lewald, 2013; 2002a, b). This was notably evidenced by studies demonstrating that following an exposure period to spatially disparate audio-visual stimuli, the perceived location of a sound is being translocated in the direction of the visual stimuli to which observers had previously been exposed, known as the ventriloquist after-effect (Canon, 1970; Frissen, Vroomen, de Gelder, & Bertelson, 2003; Lewald, 2002a, b; Radeau & Bertelson, 1974; Recanzone, 1998). Based on the results of the current study, we hypothesize that the magnitude of this after-effect may vary according to the eccentricity where the audio-visual stimuli are presented during the exposure period, with stronger ventriloquist after-effect in the central visual field where the reliability of the visual target is the highest. 

Conclusions

The current study provides new insights into how the eccentricity of separate sensory estimates influences the merging of these inputs into a unified percept. First, we found that the multisensory gain was not affected by stimuli reliability, contrasting with the prediction made from the principle of IE. This finding suggests that multisensory principles derived from unicellular experiment carried out in the superior colliculus (Stein and Meredith, 1993) are not straightforward in their explanation of multisensory phenomena at the level of perception and behavior (Spence, 2013). Second, we demonstrated that the visual capture of sounds observed with spatially incongruent audio-visual targets steadily decreased with eccentricity. Finally, we showed that at every level of eccentricity, the ventriloquist effect directly relates to a weighted combination of the spatial resolution obtained in unisensory conditions. These results evidence that the external spatial coordinates of multisensory events relative to an observer's body (e.g., eyes' or head's position) influence how this information is merged, and therefore determine the perceptual outcome. These results might have implications for the design of optimal ergonomic environment where the attention of an observer has to be captured by auditory, visual or audio-visual cues delivered at various spatial locations. 

Supplementary Materials

rtf S1 

Acknowledgments

This research was supported in part by the Canada Research Chair Program (FL), the Canadian Institutes of Health Research (FL, GC), the Natural Sciences and Engineering Research Council of Canada (FL), and the Research Center of the Ste-Justine University Hospital (OC). 

Commercial relationships: none. 

Corresponding author: Olivier Collignon. 

Email: [email protected]. 

Address: Center for Mind/Brain Sciences, University of Trento, Italy.