October 2020
Volume 20, Issue 11
Open Access
Vision Sciences Society Annual Meeting Abstract  |   October 2020
How sound influences gaze when we watch movies
Author Affiliations & Notes
  • Lauren Y. L. Cao
    University of Toronto
  • Stephanie Yung
    University of Toronto
    Toronto Rehabilitation Institute
  • Winnie Wang
    University of Toronto
  • Dirk B. Walther
    University of Toronto
    Samsung Artificial Intelligence Center Toronto
  • Footnotes
    Acknowledgements  This work was supported by an NSERC Discovery Grant (#498390), the Canadian Foundation for Innovation (#32896) to DBW, and a NSERC USRA
Journal of Vision October 2020, Vol.20, 850. doi:https://doi.org/10.1167/jov.20.11.850
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Lauren Y. L. Cao, Stephanie Yung, Winnie Wang, Dirk B. Walther; How sound influences gaze when we watch movies. Journal of Vision 2020;20(11):850. https://doi.org/10.1167/jov.20.11.850.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Movies tightly control their audience’s attention, eliciting higher levels of eye movement synchrony than unedited footage of natural scenes or amateur videos (Dorr et al. 2010; Hasson et al., 2008). Existing research on the influence of sound on visual attention during movie viewing has shown that viewers follow speech turn-taking more closely when hearing related sound (Coutrot & Guyader, 2014). Here, we seek to examine the degree to which sound contributes to the strong attentional synchrony elicited by Hollywood films. We recorded eye movements for subjects watching commercial films paired with the movie’s original soundtrack (original sound condition), the soundtrack from another movie (alternate sound condition), or white noise (ambient sound condition). Ninety-two participants each viewed six movie clips, two in each sound condition. Using inter-subject correlation (ISC) analysis, we examined how sound influences participants’ eye movements. We calculated ISC between participants who watched the same movie paired with different soundtracks: original vs. original (ceiling), original vs. alternate, and original vs. ambient. We also computed the ISC between participants who watched different movies paired with the same soundtrack and the ISC of participants watching different movies paired with different soundtracks (floor). We calculated ISC using all eye movements made within an experimental block as well as time-resolved ISC using a sliding time-window. Additionally, we used multiple regression to identify the visual and auditory characteristics that elicit strong attentional synchrony. We found that pairing movies with a non-original soundtrack lowers similarity in eye movements between viewers for all movies analyzed. Auditory characteristics also influence time resolved ISC. Overall, these findings show both sound and image factor in the control that movies exert over our gaze with visual information playing the chief role in guiding attention. Our results offer insight into how sound contributes to the attentional synchrony elicited by Hollywood films.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.