Abstract
We previously demonstrated a linear perceptual relationship between auditory amplitude-modulation (AM) rate and visual spatial frequency using Gabors (Guzman et al., VSS2011). We further demonstrated that this frequency-based auditory-visual association generalizes to the perception of natural scenes (Sherman et al., VSS2011). Participants consistently matched specific auditory AM rates to visual scenes from diverse categories (nature, urban, and indoor). A correlation analysis indicated that this crossmodal association is mediated by the subjective impression of scene density (measured after the auditory-matching task), with a higher/lower density rating associated with a faster/slower AM-rate match. Our new result shows that both the density ratings and AM-rate matches are scale invariant, suggesting that the underlying crossmodal association is between visual coding of object-based density and auditory coding of AM rate. Based on these results, we hypothesized that concurrently presenting a fast (7Hz) AM rate (associated with dense scenes) or a slow (2Hz) AM rate (associated with sparse scenes) might influence the way in which visual attention is allocated to dense or sparse aspects within a scene. We tested this hypothesis by monitoring eye movements while participants examined each scene for a subsequent memory task. The initial five saccades had significantly smaller amplitudes and shorter inter-saccade fixation durations when the faster AM sound was played than when the slower AM sound was played. This suggests that a faster AM sound may operate like a high-pass filter, emphasizing scene details and promoting a local scanning strategy in which objects within a dense region are individuated. In contrast, a slower sound may operate like a low-pass filter, emphasizing larger structures and promoting a global scanning strategy, potentially facilitating perception of gist. In summary, our results suggest that auditory AM rate and object-based visual density are crossmodally associated, and that AM sounds can influence scene inspection through this association.
Meeting abstract presented at VSS 2012