Abstract
Previous research shows that music-to-color associations are mediated by emotion (Palmer et al., 2013), even for low-level musical stimuli, including two-note intervals and instrumental timbres (Griscom & Palmer, VSS-2012). To test the generality of such emotional effects beyond music and color, we first examined crossmodal relationships between natural sounds (e.g. zippers, flames, radio static) and line-based visual textures. For each sound, participants chose the three most-consistent and three least-consistent among 28 black-and-white textures. Subjects later rated each sound and texture individually on 5 emotional (e.g. happy/sad, angry/not-angry) and 8 non-emotional dimensions (e.g. sharp/smooth, simple/complex). For each dimension, we computed an index of sound-to-texture associations (STAs) as a weighted average of the ratings of the 6 textures chosen as going best/worst with each sound. We found medium correlations between ratings of the sounds and the STAs of the textures chosen to go best/worst with them for some non-emotional dimensions (e.g. simple/complex=+.57, slanted/not-slanted=+.59), but no emotional dimensions. Because previous emotion mediation effects involved highly emotional stimuli, such as music, we hypothesized that an emotional visual stimulus, such as abstract art, might result in emotion mediation with the same natural sounds. Using the same design, a sound-to-artwork association task produced medium correlations between sound ratings and the sound-to-artwork association indices of the artwork chosen to go best/worst with them for both emotional (e.g. angry/not-angry=+.62) and non-emotional dimensions (e.g. simple/complex=+.60). We also tested for synergistic effects using two modalities that both contained highly emotional content: the same 28 abstract-art pieces and 34 widely-varying musical selections from heavy-metal to Hindustani-sitar. Strong correlations emerged between music ratings and the music-to-artwork association indices of the artwork chosen as going best/worst with them for both emotional (e.g. calm/agitated=+.88) and non-emotional dimensions (e.g. active/passive=+.87). The results suggest that emotion-mediated crossmodal matches are context dependent, depending on the modalities involved.
Meeting abstract presented at VSS 2015