Abstract
Although the sight and sound of objects were thought to be encoded initially via separate visual and auditory pathways, then combined later in high-level multisensory areas, there is growing evidence that multisensory processing also occurs in early modality-specific sensory cortices. However, it remains unclear how uni-modal sensory information in the early sensory cortex is influenced by information from other senses when the association between information from different modalities is abstract in nature, without natural spatiotemporal correspondence. In order to address this question, using fMRI and multi-voxel pattern analysis, we examine whether motion direction information in V1 is modulated when a moving stimulus is presented with a changing pitch that is congruent or incongruent with respect to the direction of visual motion. Random dots moving either upward or downward in a circular annulus were presented with an ascending or descending pitch. While fixating at the center, subjects monitored random dots for occasional changes in their motion direction. The motion direction of random dots was successfully decoded in V1 when the direction of visual motion and changing pitch were congruent (e.g., ascending pitch and upward motion or descending pitch and downward motion), but decoding accuracy significantly decreased when they were incongruent. These findings suggest that visual motion information in the early visual cortex is modulated by concurrently present auditory information, even when their association is only "metaphoric."
Meeting abstract presented at VSS 2013