June 2007
Volume 7, Issue 9
Vision Sciences Society Annual Meeting Abstract  |   June 2007
Musical use of visual gestures: the importance of contextual information in sensory integration
Author Affiliations
  • Michael Schutz
    University of Virginia
  • Michael Kubovy
    University of Virginia
Journal of Vision June 2007, Vol.7, 306. doi:https://doi.org/10.1167/7.9.306
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Michael Schutz, Michael Kubovy; Musical use of visual gestures: the importance of contextual information in sensory integration. Journal of Vision 2007;7(9):306. https://doi.org/10.1167/7.9.306.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

At VSS 2005 we reported that percussionists use visual gestures to alter audience perception of musical note length (Schutz & Kubovy). This conflicts with previous research suggesting vision does not alter auditory judgments of tone duration (Walker & Scott, 1981). In order to determine why our results differ from prior research, we built a point light simulation of a percussionist's movements tracking the long and short gestures used to strike percussion instruments. After pilot studies demonstrated the simulation to be successful, a series of four experiments revealed:

  1. When using alternate sounds, the visual influence:

    1. Persists with percussive sounds (piano).

    2. Does not persist with non-percussive sounds (clarinet, trumpet, voice).

  2. Visual influence is lost when the moving dot pauses at the moment of impact before rebounding.

  3. Influence is lost when the motion path is rotated 180 degrees after impact to simulate moving “through the bar” rather than rebounding off of it.

  4. A static dot presented simultaneously with the percussive sound produces effects similar to those observed with gestures.

We conclude our initial results were driven by the perceived causal relationship between modalities. Only sounds caused by impacts (e.g. percussive sounds) are influenced by visual impact gestures (Experiment 1). Manipulations which obscure the perception of a visual impact have no effect on auditory judgments (Experiments 2). The most important aspect of the gesture is the sudden change of direction at the moment of impact (Experiment 3). However, the motion itself is important only in that it depicts a change. A spatially static visual change (e.g. the appearance of a dot on a previously blank screen) alters perception as reliably as the original gestures (Experiment 4). Together these studies show the importance of the relationship between the auditory and visual information in understanding the nature of sensory integration.

Schutz, M. Kubovy, M. (2007). Musical use of visual gestures: the importance of contextual information in sensory integration [Abstract]. Journal of Vision, 7(9):306, 306a, http://journalofvision.org/7/9/306/, doi:10.1167/7.9.306. [CrossRef]
 This research was supported in part by a grant from NIDCD R01 DC 005636 (Michael Kubovy, PI)

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.