Purchase this article with an account.
David Brang, Satoru Suzuki, Marcia Grabowecky; The Stolen Voice Illusion. Journal of Vision 2016;16(12):461. doi: 10.1167/16.12.461.
Download citation file:
© 2017 Association for Research in Vision and Ophthalmology.
Auditory speech is typically accompanied by related visual cues that enhance speech perception and compensate for degraded auditory processing due to environmental noise or auditory deficits. This crossmodal enhancement is partly due to lip articulations crossmodally providing redundant contextual information to facilitate phoneme identification. However, past research has also demonstrated effects of face identity information, such that speech perception is impaired if lip articulations from one individual are presented simultaneously with the voice of another individual. We present a novel multisensory illusion (The Stolen Voice Illusion) that demonstrates that visual identity information can override the strong temporal cues that would normally indicate which voice is associated with which face. A female face and a male face articulating the same phoneme (e.g., /ba/) are presented side-by-side on the screen along with their voices. Critically, each voice is synchronized with the face of the incorrect gender: a female voice synchronized with male lip movements and male voice synchronized with female lip movements. One might expect that when the male-face/female-voice pair and the female-face/male-voice pair are presented asynchronously, temporal binding would make each face appear to speak with a voice of the opposite gender. Surprisingly, when the male-face/female-voice pair is presented gradually earlier than the female-face/male-voice pair, each voice is (incorrectly) perceived to originate from the matched-gender face up to about 500 ms of temporal asynchrony, as if the female voice migrated forward in time to bind with the later female face while the male voice migrated backward in time to bind with the earlier male face. When the interval is increased beyond the critical duration, the face-voice discrepancy abruptly becomes apparent. This novel illusion demonstrates the strong impact of visual identity on auditory speech perception, capable of overriding strong temporal cues that would otherwise indicate which voice was associated with which face.
Meeting abstract presented at VSS 2016
This PDF is available to Subscribers Only