Abstract
Modeling neural responses to naturalistic stimuli has been instrumental in advancing our understanding of the visual system. Dominant computational modeling efforts have been deeply rooted in preconceived hypotheses. Here, we develop a hypothesis-neutral computational methodology which brings neuroscience data directly to bear on the model development process. We demonstrate the effectiveness of this technique in modeling as well as systematically characterizing voxel tuning properties. We leverage the unprecedented scale of the Natural Scenes Dataset to constrain parametrized neural models of higher-order visual systems with brain response measurements and achieve novel predictive precision, outperforming the predictive success of state-of-the-art models. Next, we ask what kinds of functional properties emerge spontaneously in these response-optimized models? We examine trained networks through structural and functional analysis by running `virtual' fMRI experiments on large-scale probe datasets. Strikingly, despite no category-level supervision, since the models are optimized for brain response prediction from scratch, the units in the networks after optimization act strongly as detectors for semantic concepts like `faces' or `words', thereby providing one of the strongest evidences for categorical selectivity in these areas. Importantly, this selectivity is maintained when training the networks without images that contain the preferred category, strongly suggesting that selectivity is not domain-specific machinery, but sensitivity to generic patterns that characterize preferred categories. Beyond characterizing tuning properties, we study the transferability of representations in response-optimized networks on different perceptual tasks. We find that the sole objective of reproducing neural targets, without any task-specific supervision, grants different networks intriguing functionalities. Finally, our models show selectivity only for a limited number of categories, all previously identified, suggesting that the essential categories are already known. Together, this new class of response-optimized models combined with novel interpretability techniques reveal themselves as a powerful framework for probing the nature of representations and computations in the brain.