Abstract
Over the past decade, deep neural networks (DNNs) have been the standard paradigm for modeling biological brains and behavior. While initial reports suggested that the ability of DNNs to model biology correlated with their object classification accuracy (Yamins et al., 2014), this no longer appears to be the case: image-evoked activity in a self-supervised ResNet50 — an architecture introduced seven years ago — has the highest correlation with IT recordings on Brain-Score.com. We recently discovered that DNNs are also becoming progressively less aligned with human perception as their object classification accuracy has increased. This problem however can be resolved through “neural harmonization”: a drop-in training module for DNNs that forces their learned visual strategies to be consistent with those of humans (Fel et al., 2022). DNNs that are trained for object classification and harmonized with behavioral data describing human visual strategies for the same task are more interpretable, performant, and accurate at predicting human behavior. Here, we investigated if harmonizing DNNs with human behavioral data could also yield better models of the primate visual system. To test this, we turned to recordings of primate IT while animals viewed complex natural images (Arcaro et al., 2020). These experiments produced spatially resolved activity maps, which illustrate how neurons respond to every part of an image, thus revealing which features drove neural responses. After fitting a variety of state-of-the-art DNNs trained for object classification to this data, ranging from convolutional neural networks to vision transformers, we discovered that harmonizing these models with human visual strategies significantly improved their predictions of IT neural activity and reproduced qualitative features of neurons’ spatial activity maps that unharmonized models did not. Our findings demonstrate the importance of large-scale human behavioral and psychophysics data for generating more accurate and interpretable models of brain and behavior.