August 2016
Volume 16, Issue 12
Open Access
Vision Sciences Society Annual Meeting Abstract  |   September 2016
Both convolutional neural networks and voxel-wise encoding models of brain activity derived from ConvNets represent boundary-and surface-related features
Author Affiliations
  • Mark Lescroart
    Helen Wills Neuroscience Institute
  • Pulkit Agrawal
    Electrical Engineering and Computer Science
  • Jack Gallant
    Helen Wills Neuroscience Institute
Journal of Vision September 2016, Vol.16, 756. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Mark Lescroart, Pulkit Agrawal, Jack Gallant; Both convolutional neural networks and voxel-wise encoding models of brain activity derived from ConvNets represent boundary-and surface-related features. Journal of Vision 2016;16(12):756.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Convolutional neural networks (ConvNets) have achieved almost human-level performance on object recognition tasks, and voxel-wise encoding models based on ConvNet features yield accurate predictions of human brain activity. This suggests that ConvNets might provide important insights into brain function. However, the features derived from ConvNets are difficult to describe or relate to other models, so it is difficult to make inferences about human vision directly from ConvNet models fit to brain data. Here we present a new method for interpreting voxel-wise ConvNet models. We used features derived from a popular ConvNet (AlexNet) to model fMRI responses elicited by a large set of naturalistic movies rendered using computer graphics software. As expected, the AlexNet model accurately predicted brain activity in a separate data set. To interpret the fit models, we determined which AlexNet features were associated with the presence of object boundary contours and large surfaces in the stimulus movies. (Boundaries and surfaces were computed based meta-data in the rendering software.) We find that different subsets of AlexNet features represent boundary contours and surfaces, particularly in intermediate layers of AlexNet. We then selectively deleted AlexNet channels representing either boundary contours or surfaces, and used these reduced models to predict brain activity. We find that deletion of boundary-related features impairs predictions in early and intermediate visual areas, while deletion of surface-related features impairs predictions in scene-selective areas. Thus, our results show that a ConvNet trained to classify images represents boundary- and surface-related features that are also represented explicitly in the human brain. Furthermore, our results suggest that specific ConvNet features could be used to quantify object boundaries and large surfaces in arbitrary stimuli. Our approach provides a general, objective method for leveraging the predictive power of ConvNets to make new discoveries about how features in natural images are represented in the brain.

Meeting abstract presented at VSS 2016


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.