Abstract
Like most animals, humans are endowed with multiple sensory modalities. While this attribute tremendously enriches our perception of the environment, it also poses the nervous system, at any instant, with the eminent problem of whether and how to integrate the signals from the different modalities. Temporally or spatially coincident signals in different sensory modalities may or may not originate from the same source in the world. The nervous system is constantly faced, therefore, with the problem of estimating which sensory signals have been caused by the same source and should be integrated, and which have been caused by different sources and should be segregated. We developed an ideal observer model which assumes one source for the signal in each modality; however, the sources are not taken to be statistically independent. Thus, the model allows inference about both cases in which separate entities have caused the sensory signals, and cases in which sensory signals are caused by one source. The model uses Bayesś rule to make inference about the causes. We examined auditory-visual perception in humans empirically, in order to compare it with the ideal observer. Observers were presented with variable number of flashes paired with variable number of beeps, and their task was to report the number of flashes they saw as well as the number of beeps they heard in each trial. We found that human auditory-visual perception is surprisingly consistent with the ideal observer, indicating that the rule used by the nervous system for combining and segregating auditory and visual signals is statistically optimal. These results provide the first unifying account for the entire spectrum of cue combination, ranging from no integration, to partial interactions, to complete fusion. Our findings also show that the sound-induced flash illusion (Nature 2000) is indeed consistent with a general, statistically optimal strategy used by the brain for combining auditory and visual signals.
L.S. is supported by UCLA R&R Setup Funds. W. M. is supported by the Swartz Foundation