September 2024
Volume 24, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   September 2024
Classification of sound-symbolic speech sounds in visual cortex
Author Affiliations & Notes
  • stefano ioannucci
    Visual and Cognitive Neuroscience Lab, University of Fribourg, Switzerland
  • Carole Jordan
    Visual and Cognitive Neuroscience Lab, University of Fribourg, Switzerland
  • Anne-Sophie Carnet
  • Carolyn McGettigan
  • Petra Vetter
  • Footnotes
    Acknowledgements  This work was supported by a research grant from the BIAL Foundation (No. 238/20) to P.V. and C.M and by a PRIMA grant (PR00P1_185918/1) from the Swiss National Science Foundation to P.V.
Journal of Vision September 2024, Vol.24, 602. doi:https://doi.org/10.1167/jov.24.10.602
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      stefano ioannucci, Carole Jordan, Anne-Sophie Carnet, Carolyn McGettigan, Petra Vetter; Classification of sound-symbolic speech sounds in visual cortex. Journal of Vision 2024;24(10):602. https://doi.org/10.1167/jov.24.10.602.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Sound symbolism, also known as the "bouba-kiki effect", is a psychological phenomenon revealing a non-arbitrary link between meaningless speech sounds and visual features. For instance, individuals consistently match rounded, curvy shapes with the word "bouba" and jagged, angular shapes with "kiki" (Lockwood & Dingemanse, 2015; Ramachandran & Hubbard, 2001), suggesting a cross-modal association between speech sounds and visual shape. Despite extensive behavioral evidence demonstrating sound symbolism, neural evidence in the human brain remains limited (Peiffer-Smadja & Cohen, 2019; Revill et al., 2014). Building on research indicating early visual cortex's response to auditory stimuli (Vetter et al., 2014, 2020), our study investigated whether visual cortices, especially those sensitive to shape, show distinguishable neural activity patterns to the implicit shape conveyed by sound symbolic speech sounds. We collected 3T fMRI BOLD signals from 8 blindfolded participants listening to sound symbolic words (rounded, spiky, and mixed) and matching them with visual shapes (on a 4-point scale from more spiky to more round). Beta weights from visual regions (V1, V2, V3, and LOC) were extracted, and multi-variate pattern analysis assessed if sound-symbolic word categories yielded differential neural patterns. Behavioural results replicated the "bouba-kiki" effect – "rounded" words were associated with rounded shapes and vice versa for “spiky” words. Preliminary fMRI analyses indicated successful decoding of "round" and "spiky" sounds in visual cortical regions for 5 participants. These findings provide evidence that fMRI activity in early visual and shape-selective cortical regions can contain information about sound symbolic word l associations, in the absence of visual stimulation. This supports sound symbolism as a genuine cross-modal effect in the human brain and suggests that visual brain regions represent also high-level auditory information, emphasizing the depth of audiovisual interactions in visual cortex.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×