September 2015
Volume 15, Issue 12
Vision Sciences Society Annual Meeting Abstract  |   September 2015
Emotion mediation in audio-visual correspondences among natural sounds, texture, and art
Author Affiliations
  • Joshua Peterson
    University of California, Berkeley
  • Stephen Palmer
    University of California, Berkeley
Journal of Vision September 2015, Vol.15, 855. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Joshua Peterson, Stephen Palmer; Emotion mediation in audio-visual correspondences among natural sounds, texture, and art. Journal of Vision 2015;15(12):855.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Previous research shows that music-to-color associations are mediated by emotion (Palmer et al., 2013), even for low-level musical stimuli, including two-note intervals and instrumental timbres (Griscom & Palmer, VSS-2012). To test the generality of such emotional effects beyond music and color, we first examined crossmodal relationships between natural sounds (e.g. zippers, flames, radio static) and line-based visual textures. For each sound, participants chose the three most-consistent and three least-consistent among 28 black-and-white textures. Subjects later rated each sound and texture individually on 5 emotional (e.g. happy/sad, angry/not-angry) and 8 non-emotional dimensions (e.g. sharp/smooth, simple/complex). For each dimension, we computed an index of sound-to-texture associations (STAs) as a weighted average of the ratings of the 6 textures chosen as going best/worst with each sound. We found medium correlations between ratings of the sounds and the STAs of the textures chosen to go best/worst with them for some non-emotional dimensions (e.g. simple/complex=+.57, slanted/not-slanted=+.59), but no emotional dimensions. Because previous emotion mediation effects involved highly emotional stimuli, such as music, we hypothesized that an emotional visual stimulus, such as abstract art, might result in emotion mediation with the same natural sounds. Using the same design, a sound-to-artwork association task produced medium correlations between sound ratings and the sound-to-artwork association indices of the artwork chosen to go best/worst with them for both emotional (e.g. angry/not-angry=+.62) and non-emotional dimensions (e.g. simple/complex=+.60). We also tested for synergistic effects using two modalities that both contained highly emotional content: the same 28 abstract-art pieces and 34 widely-varying musical selections from heavy-metal to Hindustani-sitar. Strong correlations emerged between music ratings and the music-to-artwork association indices of the artwork chosen as going best/worst with them for both emotional (e.g. calm/agitated=+.88) and non-emotional dimensions (e.g. active/passive=+.87). The results suggest that emotion-mediated crossmodal matches are context dependent, depending on the modalities involved.

Meeting abstract presented at VSS 2015


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.