Purchase this article with an account.
Mark Lescroart, Pulkit Agrawal, Jack Gallant; Both convolutional neural networks and voxel-wise encoding models of brain activity derived from ConvNets represent boundary-and surface-related features. Journal of Vision 2016;16(12):756. doi: https://doi.org/10.1167/16.12.756.
Download citation file:
© ARVO (1962-2015); The Authors (2016-present)
Convolutional neural networks (ConvNets) have achieved almost human-level performance on object recognition tasks, and voxel-wise encoding models based on ConvNet features yield accurate predictions of human brain activity. This suggests that ConvNets might provide important insights into brain function. However, the features derived from ConvNets are difficult to describe or relate to other models, so it is difficult to make inferences about human vision directly from ConvNet models fit to brain data. Here we present a new method for interpreting voxel-wise ConvNet models. We used features derived from a popular ConvNet (AlexNet) to model fMRI responses elicited by a large set of naturalistic movies rendered using computer graphics software. As expected, the AlexNet model accurately predicted brain activity in a separate data set. To interpret the fit models, we determined which AlexNet features were associated with the presence of object boundary contours and large surfaces in the stimulus movies. (Boundaries and surfaces were computed based meta-data in the rendering software.) We find that different subsets of AlexNet features represent boundary contours and surfaces, particularly in intermediate layers of AlexNet. We then selectively deleted AlexNet channels representing either boundary contours or surfaces, and used these reduced models to predict brain activity. We find that deletion of boundary-related features impairs predictions in early and intermediate visual areas, while deletion of surface-related features impairs predictions in scene-selective areas. Thus, our results show that a ConvNet trained to classify images represents boundary- and surface-related features that are also represented explicitly in the human brain. Furthermore, our results suggest that specific ConvNet features could be used to quantify object boundaries and large surfaces in arbitrary stimuli. Our approach provides a general, objective method for leveraging the predictive power of ConvNets to make new discoveries about how features in natural images are represented in the brain.
Meeting abstract presented at VSS 2016
This PDF is available to Subscribers Only