Abstract
Cross-modal correspondences help us to determine which signals belong together when we receive information from multiple sensory systems. Many studies have shown people exhibit cross-modal correspondences involving features from various, different, sensory modalities. Moreover, some correspondences seem to reflect natural statistical mappings, thus implying that humans may acquire knowledge of cross-modal correspondences through daily perceptual experiences. We have demonstrated that participants can guess relationships between visual space and auditory pitch merely from perceptual experience, and predict visual stimuli according to audio stimuli (Hayashi & Yokosawa, 2018). However, such results can also be explained by considering evidence that they simply applied linear relationships to visual and auditory stimuli instead of assuming continuous relationship. Therefore, the present study aimed to clarify whether people can acquire complicated combinational relationship between visual and auditory feature. In this experiment, 5 or 15 stimulus pairs, presented consecutively, involved a pure tone and a small black disc (on a screen). Disc positions and tone pitch were determined by one of 8 kinds of curvilinear correlations in each trial. Also, the pairs were unrelated in some trials. After exposure to 5 or 15 pairs of stimuli, participants heard a pure tone of a particular (varied) frequency. Next, they guessed the spatial location of the disc previously related to frequency. Results show that participants extracted most of the complicated curvilinear relationships between visual and auditory feature, instead of just relying upon a linear relationship. Furthermore, participants could predict disc spatial positions more precisely after they were presented with 15 pairs of visual-audio stimuli than when they presented with 5 pairs of stimuli. In sum, participants learn perceptual relationships, even curvilinear ones. They can reconstruct non-linear continuous mapping of visual and auditory features from limited experiences.