September 2017
Volume 17, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   August 2017
Perceptual straightening of natural video trajectories
Author Affiliations
  • Olivier Henaff
    Howard Hughes Medical Institute
    Center for Neural Science, New York University
  • Robbe Goris
    Center for Perceptual Systems, University of Texas at Austin
  • Eero Simoncelli
    Howard Hughes Medical Institute
    Center for Neural Science, New York University
Journal of Vision August 2017, Vol.17, 402. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Olivier Henaff, Robbe Goris, Eero Simoncelli; Perceptual straightening of natural video trajectories. Journal of Vision 2017;17(10):402. doi:

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

In a video of an object moving or deforming over time, the vector of pixels follows a complex high-dimensional trajectory. Yet human observers understand such trajectories and immediately recognize the persistence of the object. Inspired by the "untangling hypotheses" (DiCarlo & Cox, 2007), we propose that the visual system builds a representation of temporally contiguous images that is less curved, such that linear operations are sufficient to capture object persistence. To test this hypothesis, we estimated the curvature of a set of natural image sequences in the pixel and perceptual domains. Pixel-domain curvature is computed as the angle between high-dimensional vectors corresponding to differences between consecutive frames. Perceptual curvature is computed by first estimating an internal trajectory that best accounts for the discriminability of pairs of frames under brief, peripheral presentation in a sequential ABX paradigm. Specifically, we formulated an observer model that measures distances in a fixed-dimensional perceptual space, then maximize the likelihood (over the entire data set for an observer) of the location of each frame within that space. The curvature of this perceptual trajectory is then computed as for the pixel-domain trajectory. Consistent with our hypothesis, we found that perceptual curvature of natural videos was systematically reduced relative to pixel-domain curvature. This suggests that the visual system non-linearly distorts incoming information so as to linearize natural image sequences. It follows that image sequences that are straight in the pixel domain should generally be curved perceptually. We tested this prediction by estimating the perceptual curvature of synthetic image sequences that fade linearly from an initial to a final frame. In this unnatural case, perceptual curvature was systematically larger than pixel-domain curvature. Together, these results demonstrate the existence of non-linear operations used by the visual system to build simple representations of naturally occurring temporal image transformations.

Meeting abstract presented at VSS 2017


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.