September 2024
Volume 24, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   September 2024
Sparse components distinguish visual pathways and their alignment to neural networks
Author Affiliations & Notes
  • Ammar Marvi
    MIT
  • Nancy Kanwisher
    MIT
  • Meenakshi Khosla
    UCSD
  • Footnotes
    Acknowledgements  This work was funded by NIH grant R01-EY033843
Journal of Vision September 2024, Vol.24, 759. doi:https://doi.org/10.1167/jov.24.10.759
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Ammar Marvi, Nancy Kanwisher, Meenakshi Khosla; Sparse components distinguish visual pathways and their alignment to neural networks. Journal of Vision 2024;24(10):759. https://doi.org/10.1167/jov.24.10.759.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

What distinguishes the representations and computations of the ventral, dorsal, and lateral visual streams, and why do current computational models often fail to reflect these differences? Prevailing hypotheses suggest specialized functions for each stream: the ventral stream in object recognition, the dorsal stream in visually guided action, and the lateral stream in motion and social information processing. However, linear encoding models of deep neural networks (DNN) optimized for object categorization predict responses across the three visual streams similarly well. Such findings may indicate a failure to capture neural tuning in model-brain comparison tools, especially those using linear mappings. To address this question we first employed data-driven factorization to identify dominant sparse components within each stream. This method revealed face, place, body, text, and food-selective components in the ventral stream; social interaction, implied motion, and hand-selective components in the lateral stream; and some less interpretable components in the dorsal stream. To systematically assess this effect and its relation to models we propose a new technique – Sparse Components Alignment (SCA) – to measure model-brain alignment while remaining sensitive to neural tuning. Using the same methodological framework as RSA, we assessed stimulus-level representational dissimilarities. However, instead of relying on population geometry, SCA computes pairwise distances between stimuli based on the likelihood that they are processed by the same sparse component. We report three findings: (1) sparse representations differ strikingly across streams, (2) DNNs optimized for object categorization are more similar to the ventral visual stream in these sparse representations, and (3) the clarity of these differences is markedly enhanced with SCA compared to linear encoding or RSA methods. Thus, SCA reveals a notably stronger fit between DNNs and the ventral visual pathway than between DNNs and other pathways, underscoring the importance of characterizing neural tuning—above and beyond representational geometry—in assessing model-brain alignment.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×