Protective effects of combined audiovisual stimulation on temporal expectations in noisy environments

Abstract

In real life, we are exposed to a rich environment, a complex and continuous stream of multisensory information. This information needs to be integrated to generate a reliable mental model of our world. There is converging evidence that there are at least two optimization mechanisms to integrate incoming information: multisensory interactions (MSI) and temporal expectations (TE). However, how these mechanisms interact is currently unknown. In a series of 4 psychophysical experiments we tested whether MSI-induced behavioral benefits interact with TE-induced benefits, and whether these effects are affected by distinct experimental contexts. In particular, auditory (A) and/or visual (V) stimulus sequences were presented either alone or simultaneously in all experiments. Participants discriminated visual and/or auditory frequencies of deviant target stimuli (high/low) within each sequence. Moreover, temporal expectation about time-of-target-occurrence was manipulated block-wise: targets preferentially occurred either early ('early block') or late ('late block') within the stimulus sequence within each block. Task difficulty was further altered by using speakers ('same location', Exp. 1 & 3) or headphones ('different location', Exp. 2 & 4), and by changing the predictability of target modality (predictable: Exp.1 & 2, unpredictable: Exp. 3 & 4). Multisensory interplay was always quantified by comparing subject-specific performance during multisensory stimulation with performance in the best unisensory condition (max-criterion). We observed distinct effects for MSI: multisensory enhancement was dependent on task difficulty, increased with increasing noise and was dominant when participants reported having problems with the task. Remarkably, TE effects were also enhanced for multisensory relative to unisensory stimulation and TE effects for unisensory stimuli even vanished under high spatial uncertainty. Together, the pattern of results indicate that multisensory stimulation has a protective and enhancing effect on the generation and usage of temporal expectations, highlighting the need for multisensory paradigms in future studies investigating temporal expectations.

Meeting abstract presented at VSS 2017