Cross modal correlation search in the presence of visual distractors

Abstract

Previous studies on cross-modal event detection have focused on coincidence detection between single visual and auditory events. In the natural environment we are usually confronted with multiple visual and/or auditory events that occur simultaneously. In order to correctly match these auditory and visual events it is necessary to determine the temporal correlation between them. One question that arises is whether cross-modal correlation detection is a pre-attentive low-level process or an attention driven process. If cross-modal correlation detection is an attention driven process then each visual cue that could potentially be correlated with the sound needs to be analyzed in series. If it is a pre-attentive process then parallel processing may occur. To test if the cross-modal correlation detection is pre- or post-attentive we measured the time needed to find the visual target correlated with the sound, embedded among uncorrelated distractors, as a function of the number of visual elements in the stimulus. Our results suggest that the time required for 75% target detection increases linearly with the number of visual elements. This suggests an attention mediated search. Additionally, when the exposure duration was fixed, the accuracy to identify the correlated visual target steeply decreased with the number of visual elements. This cannot be ascribed to the visual crowding effect, since when the target location was indicated by a prior cue, correlation detection was as easy as when the target was presented alone. Both experiments indicate that these cross-modal temporal comparisons are done by serial attention demanding, rather than parallel pre-attentive, mechanisms.