December 2022
Volume 22, Issue 14
Open Access
Vision Sciences Society Annual Meeting Abstract  |   December 2022
The role of gaze position in training visual brain encoders on free-viewing data
Author Affiliations
  • Marie St-Laurent
    Centre de recherche de l’Institut universitaire de gériatrie de Montréal
    Max Planck Institute for Human Cognitive and Brain Sciences
  • Katja Seeliger
    Max Planck Institute for Human Cognitive and Brain Sciences
  • Martin Hebart
    Max Planck Institute for Human Cognitive and Brain Sciences
Journal of Vision December 2022, Vol.22, 4091. doi:https://doi.org/10.1167/jov.22.14.4091
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Marie St-Laurent, Katja Seeliger, Martin Hebart; The role of gaze position in training visual brain encoders on free-viewing data. Journal of Vision 2022;22(14):4091. https://doi.org/10.1167/jov.22.14.4091.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Brain-encoding models can be trained to learn the correspondence between visual stimuli and the brain’s response to those stimuli. To learn meaningful visual features, input images are commonly aligned with the participant’s fovea, which complicates the use of training data acquired with free-viewing paradigms. Here, we tested whether an end-to-end brain encoder could be trained on movie-viewing fMRI data without requiring gaze fixation or image recentering. We trained a Neural Information Flow (NIF) model to predict responses in brain areas V1, V2, V3, hV4, V3a and V3b using data from a subject who watched 3 seasons of the sitcom Friends, from the Courtois-Neuromod project. With video stimuli as input, NIF couples brain areas with tensors that encode spatiotemporal features represented in its activity. As no eye-tracking data were acquired during viewing time, gaze position over movie frames was estimated with DeepGaze MR. Input images were either recentered around this estimated gaze position, or not recentered. Preliminary results indicate that NIF acquired biologically plausible features predictive of visual cortical activity without gaze fixation or input image realignment. The variance explained for predicted voxels was similarly distributed independent of whether gaze alignment was applied, indicating no improvement in performance for models trained on recentered movie frames. Eye-tracking data acquired post-hoc for individual Friends episodes revealed similar gaze prediction performance for DeepGaze MR relative to the assumption of central fixation. Moreover, 67% of the measured eye positions were in the central 3.5 degrees of visual angle. These results suggest that recentering may not be required, and that brain encoding models may learn visual representations from free-viewing data even without correcting for eye movements, albeit possibly with lower predictive performance than with eye-tracking. Our findings have implications for future work training models on free-viewing data without eye-tracking.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×