Abstract
Several studies have demonstrated the potential of deep neural networks (DNNs) to serve as state-of-the-art computational models of the primate visual cortex. In the last decade, different implementations of DNNs (varying, for example, their architecture, objective function, or training algorithm) have been compared to uncover the computational principles, algorithms, and neurobiological mechanisms behind visual processing (Cadieu et al. 2014; Khaligh-Razavi and Kriegeskorte 2014; Yamins et al. 2014; Guclu and Gerven 2015; Cichy, Khosla, et al. 2016). To promote this line of research, new benchmarks, datasets, and challenges relevant to cognitive neuroscience experiments have been developed (Cichy, Roig, Alex Andonian, et al. 2019; Cichy, Roig, and Oliva 2019; Cichy, Kshitij Dwivedi, et al. 2021; Schrimpf et al. 2018; Nili et al. 2014). There are some toolboxes, that already facilitate the extraction of model activations, but these mainly focus on supervised image classification models (Muttenthaler and Hebart 2021). However, studies have shown that DNNs trained for different tasks could provide new information about the visual cortex (Tang, LeBel, and Huth 2021; Dwivedi et al. 2021). We, therefore, introduce Net2Brain, a toolbox for mapping model representations to human brain data. Net2Brain allows the extraction of activations over image and video datasets of any inserted custom model or any of the 600+ included DNNs trained for various visual tasks (e.g., semantic segmentation, depth estimation, action recognition), including multimodal models. In contrast to other toolboxes, Net2Brain handles all steps from feature extraction to analysis through a simple pipeline. It computes the representational dissimilarity matrices (RDMs) over the activations and compares them to brain recordings using representational similarity analysis (RSA), and weighted RSA, both using ROI-based and searchlight analysis. Net2Brain is open source and comes with brain data for immediate testing, and it is also straightforward to use your own recorded data.