December 2022
Volume 22, Issue 14
Open Access
Vision Sciences Society Annual Meeting Abstract  |   December 2022
Understanding the invariances of visual features with separable subnetworks
Author Affiliations
  • Christopher Hamblin
    Harvard University
  • Talia Konkle
    Harvard University
  • George Alvarez
    Harvard University
Journal of Vision December 2022, Vol.22, 4237. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Christopher Hamblin, Talia Konkle, George Alvarez; Understanding the invariances of visual features with separable subnetworks. Journal of Vision 2022;22(14):4237.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Visual feature detectors that are useful for high-level semantic tasks must often be invariant to differences in the input space, but how such invariant feature detectors are constructed through image-computable operations is a fundamental and poorly understood challenge. Deep convolutional neural networks have the potential to provide insight into this puzzle, as invariant feature tuning often emerges in the latent spaces of such networks — but, how? Here we present a novel pruning method we call 'feature splitting', which can split a single CNN feature into multiple, sparse subnetworks, each of which only preserves its tuning response to a selection of inputs. We focus on polysemantic units, which respond strongly and selectively to seemingly unrelated semantic categories (e.g., monkey faces and written text) as a case study for splitting a feature across its invariance structure. While a few examples of polysemantic units have been characterized in DNNs, here we develop a data-driven method for identifying polysemantic units in the network. Then, we extract multiple sparse subnetworks, each of which only preserves the feature’s response to a targeted subset of image patches (e.g., to monkey faces, or to written text). In such instances, we find our feature-splitting algorithm returns highly separable subnetworks, with few shared weights between them. These findings indicate that the tuning of polysemantic units draws largely on highly distinct image filtering processes, acting as an ‘or’ gate by summing the outputs of these processes. Broadly, these feature-splitting methods introduce a principled approach for dissecting a wide range of invariance structures necessary for high-level feature detection (e.g. over units that respond to both profile and frontal views of faces, or to objects presented at different scales in the image), isolating the separable and shared computations underlying invariance.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.