August 2010
Volume 10, Issue 7
Vision Sciences Society Annual Meeting Abstract  |   August 2010
Eye movements while viewing captioned and narrated videos
Author Affiliations
  • Nicholas M. Ross
    Department of Psychology, Rutgers University
  • Eileen Kowler
    Department of Psychology, Rutgers University
Journal of Vision August 2010, Vol.10, 141. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Nicholas M. Ross, Eileen Kowler; Eye movements while viewing captioned and narrated videos. Journal of Vision 2010;10(7):141.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Research on visual attention has been mainly limited to static images, but in everyday life we often rely on a narrative to guide us through dynamically changing scenes. Sometimes the narrative is presented via audio and in special cases as a caption. Narratives can help guide attention, but may require additional processing that increases task demands, or distracts attention from relevant locations when presented as a caption. This study examined how narrative and video interact to drive attention while viewing documentary clips. Video clips (∼120 s each) were cut from 4 documentaries such that no talking heads were present in any frame. Videos were accompanied by narration in the form of either audio or captions. Videos with both audio and captions, or neither, were also tested. In order to motivate viewing, multiple choice tests on content were given after each clip. Captions were strong attractors of gaze. 56% of saccades were devoted to reading captions when no audio was present, and 41% when audio was present (surprisingly large, given that audio made reading of captions unnecessary). Durations of fixations were shorter for reading captions (∼260 ms) than inspecting the video (∼420 ms), regardless of the presence of audio. Fixations made to inspect the video clustered near the scene center when narration was present. In the absence of narration, eye movements were more exploratory; the 2D scatter of saccadic endpoint locations increased by up to 70%. These changes to the spatial distribution of saccades, as well as the adoption of the time-consuming strategy of reading captions even when redundant with audio narration, show that eye movements while inspecting videos are motivated mainly by a cognitive strategy of searching for clues that facilitate the interpretation of viewed events.

Ross, N. M. Kowler, E. (2010). Eye movements while viewing captioned and narrated videos [Abstract]. Journal of Vision, 10(7):141, 141a,, doi:10.1167/10.7.141. [CrossRef]
 NSF 0549115.

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.