August 2023
Volume 23, Issue 9
Open Access
Vision Sciences Society Annual Meeting Abstract  |   August 2023
Net2Brain: A Toolbox to compare artificial vision models with human brain responses
Author Affiliations & Notes
  • Domenic Bersch
    Johann Wolfgang Goethe-Universität Frankfurt
  • Kshitij Dwivedi
    Johann Wolfgang Goethe-Universität Frankfurt
  • Martina Vilas
    Johann Wolfgang Goethe-Universität Frankfurt
    Ernst Struengmann Institute for Neuroscience
  • Radoslaw Martin Cichy
    Department of Education and Psychology, Freie Universität Berlin
    Berlin School of Mind and Brain, Faculty of Philosophy
    Bernstein Center for Computational Neuroscience Berlin
  • Gemma Roig
    Johann Wolfgang Goethe-Universität Frankfurt
  • Footnotes
    Acknowledgements  This work was funded with the support from the Alfons and Gertrud Kassel Foundation (G.R.), by the Hessian Center for AI Germany (, by the German Research Foundation (DFG, CI241/1-1, CI241/3-1 to R.M.C.) and by the European Research Council (ERC, 803370 to R.M.C.).
Journal of Vision August 2023, Vol.23, 4935. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Domenic Bersch, Kshitij Dwivedi, Martina Vilas, Radoslaw Martin Cichy, Gemma Roig; Net2Brain: A Toolbox to compare artificial vision models with human brain responses. Journal of Vision 2023;23(9):4935.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Several studies have demonstrated the potential of deep neural networks (DNNs) to serve as state-of-the-art computational models of the primate visual cortex. In the last decade, different implementations of DNNs (varying, for example, their architecture, objective function, or training algorithm) have been compared to uncover the computational principles, algorithms, and neurobiological mechanisms behind visual processing (Cadieu et al. 2014; Khaligh-Razavi and Kriegeskorte 2014; Yamins et al. 2014; Guclu and Gerven 2015; Cichy, Khosla, et al. 2016). To promote this line of research, new benchmarks, datasets, and challenges relevant to cognitive neuroscience experiments have been developed (Cichy, Roig, Alex Andonian, et al. 2019; Cichy, Roig, and Oliva 2019; Cichy, Kshitij Dwivedi, et al. 2021; Schrimpf et al. 2018; Nili et al. 2014). There are some toolboxes, that already facilitate the extraction of model activations, but these mainly focus on supervised image classification models (Muttenthaler and Hebart 2021). However, studies have shown that DNNs trained for different tasks could provide new information about the visual cortex (Tang, LeBel, and Huth 2021; Dwivedi et al. 2021). We, therefore, introduce Net2Brain, a toolbox for mapping model representations to human brain data. Net2Brain allows the extraction of activations over image and video datasets of any inserted custom model or any of the 600+ included DNNs trained for various visual tasks (e.g., semantic segmentation, depth estimation, action recognition), including multimodal models. In contrast to other toolboxes, Net2Brain handles all steps from feature extraction to analysis through a simple pipeline. It computes the representational dissimilarity matrices (RDMs) over the activations and compares them to brain recordings using representational similarity analysis (RSA), and weighted RSA, both using ROI-based and searchlight analysis. Net2Brain is open source and comes with brain data for immediate testing, and it is also straightforward to use your own recorded data.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.