Abstract
Human perception is highly multisensory. In nature several sources can be the cause of the stimuli and the nervous system has to decide in which cases the inputs received are from a single or multiple sources. This presents the nervous system with the constant problem of deciding which signals should be integrated and how, and which signals should be segregated. We investigated this question in the context of the so-called ventriloquist illusion, by which a ventriloquist can induce the perception of the sound being projected by a puppet by moving the puppet in synchrony with the sound. Depending on the angular separation of the visual and auditory sources, the percept can either be of two different causes (puppet moving, sound from ventriloquist) or a single one (puppet talking). It has been previously shown that an ideal observer model of multisensory processing derived from Bayesian inference can account for the sound-induced flash illusion and the general task of reporting number of pulsations which is primarily a temporal task. We examined whether this model can explain the ventriloquist illusion which involves a spatial task. We presented subjects with flashes of gabor wavelets on a noisy background accompanied by brief bursts of auditory noise. The visual stimulus and auditory stimulus were presented at the same or different locations, each at one of 5 locations along a horizontal line. The task was to indicate the location of the visual stimulus as well as that of the auditory stimulus in each trial. The subjects' auditory and visual responses were surprisingly consistent with the ideal observer in all conditions (R2=0.91). The finding that the Bayesian inference model can account for auditory-visual integration/segregation in two complementary tasks and paradigms suggests that Bayesian inference is a general principle governing multisensory information processing in humans, spanning situations of segregation to partial interaction to complete fusion.