Abstract
Unisensory information is of course processed before integration with signals from other senses. Sensory input in the visual and auditory systems are segregated into ON and OFF pathways that respond differentially to changes in input intensity. Specifically, an increase in intensity leads to an increment in neural response in the ON pathway, and decrement in the OFF pathway. The opposite occurs with an intensity decrease. We investigated how humans combine auditory-visual stimuli that consist of step changes in intensity. Subjects made temporal-order judgments. Stimuli consisted either of increments (ON stimuli) or decrements (OFF stimuli) in intensity over time. ON and OFF stimuli were paired in all possible combinations. In some pairings, the stimuli both increased or decreased in intensity. In other pairings, one increased while the other decreased. Subjects reported the modality that seemed to change first. Recently, we developed a model in which multisensory integration relies on elementary units analogous to the motion-energy units observed in insect visual systems. In its original form, the model consisted only of ON channels, so an increment in intensity in one modality paired with a decrement in the other would produce an inversion of perceived temporal order reminiscent of the "reverse phi" effect in visual motion perception. The results from the temporal-order judgment task showed that subjects could accurately identify the temporal order of the two signals even when the intensity in the two modalities changed in opposite directions. This suggests that unisensory information is separately processed in ON and OFF channels, yielding an estimate of the time of change without regard to sign, before the information is integrated. A simple model in which unisensory information is first segregated into ON and OFF channels, low-pass filtered, and then recombined before feeding into an energy unit yields behavior consistent with our experimental observations.
Meeting abstract presented at VSS 2016