Audiovisual congruence and the processing of synchrony in swing groove drumming

Abstract

Single cell data from macaque suggest special processing of the sights and sounds of biological actions (Kohler, Keysers, et al, Science 2002). Recently Arrighi, Alais & Burr (JOV, 2006) have examined this hypothesis using judgments of perceptual synchrony of audio and visual streams of conga drumming as well as with synthetic audio and visual streams. The perception of audiovisual temporal synchrony provides a window on how these two different sensory modalities are integrated. To further investigate the perception of audiovisual synchrony from human movements we first obtained 3D motion capture data (240 Hz) of a drummer performing swing groove drumming with an accent on the first beat at 60, 90 and 120 BPM. This movement data was converted into visual point light displays with points representing the shoulder, elbow, wrist, hand and 2 points on the drumstick and output as 60 Hz quicktime movies. Matching synthetic sounds were obtained by a simulation of the first 25 modes of a circular membrane that took as input the time and impact velocity of a strike and output the resulting audio signal. In one experimental condition the estimated impact velocity was provided to the algorithm and in the other condition the average impact velocity of all strikes was used. This allowed production of one audiovisual stream with a natural covariation of drummer motion and sound present, and another audiovisual stream with this covariation absent. For both the covariation present and absent conditions we presented observers with audiovisual displays of varying levels of positive and negative asynchrony between audio and visual streams and collected forced-choice judgments of synchrony. Our preliminary results indicate that the novel manipulation of eliminating the natural covariation between physical movement and audio signal influences observers' integration of sight and sound.

Footnotes

 Wellcome Trust, British Academy