Abstract
Recent neurophysiological results suggest that crossmodal interactions are commonplace even in primary sensory areas of the brain. Spatial frequency is a fundamental visual feature coded in V1, and is relevant for perceiving scenes, hierarchical structure, and objects. Temporal amplitude-modulation (AM) frequency is a fundamental auditory feature coded in the primary auditory cortex, and is relevant for perceiving auditory objects and speech. Spatial frequency and temporal AM frequency are thus fundamental building blocks of visual and auditory perception. Here we demonstrate that the processing of auditory AM frequency and the processing of visual spatial frequency are closely associated. Observers linearly matched a given visual spatial frequency to a specific auditory AM frequency, suggesting a consistent perceptual mapping between visual spatial frequency and auditory AM frequency. This crossmodal association is not merely subjective; here we show that AM sounds modulate visual attention and awareness in a frequency specific manner. When a pair of Gabors with different spatial frequencies were simultaneously presented with an AM sound (perceptually matched to one of the Gabors), observers detected a phase-shift more rapidly when it occurred on the sound-congruent Gabor, suggesting that AM sound guided attention to the matched spatial frequency. When a pair of Gabors with different spatial frequencies were presented to separate eyes to generate binocular rivalry, presenting an AM sound (perceptually matched to one of the Gabors) increased the proportion of perceptual dominance of the congruent Gabor, suggesting that an AM sound boosts signal strength for the matched spatial frequency in processes that select stimuli for awareness. The naïve observers who participated in the binocular rivalry experiment were unaware of any association between AM sounds and Gabor spatial frequencies. Additional results suggest that the association between auditory AM frequency and visual spatial frequency develops through multisensory experience of manually exploring surfaces.
NIH R01 EY018197, NSF BCS 0643191.