September 2019
Volume 19, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   September 2019
Enhancement of Representational Sparsity in Deep Neural Networks Can Improve Generalization
Author Affiliations & Notes
  • Hongjing Lu
    Department of Psychology, UCLA
    Department of Statistics, UCLA
  • Gennady Erlikhman
    Department of Psychology, UCLA
Journal of Vision September 2019, Vol.19, 209b. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Hongjing Lu, Gennady Erlikhman; Enhancement of Representational Sparsity in Deep Neural Networks Can Improve Generalization. Journal of Vision 2019;19(10):209b.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Sparse coding in the human visual system has been viewed as a fundamental mechanism to provide increased representational capacity and efficiency with discriminative features. For artificial networks, sparsity has been introduced in numerous ways, e.g., in the form of dropout during training, rectification of activation, or wholesale removal of network nodes or individual connections. However, the main goal of sparsifying artificial networks is to reduce overfitting and network size and the methods for enforcing sparsity are determined more by computational convenience than generalization improvement. We compared standard methods of introducing sparsity in deep learning models, such as dropout, with more human-like schemes like permanent, targeted removal of weights or nodes as might happen during synaptic pruning during normal human development. A series of simulations were systematically conducted using the handwritten digit dataset (MNIST) with a simple, fully-connected three-layer network. We show that introducing sparsity with human-like schemes can significantly improve generalizability in the form of far transfer to untrained datasets, such as digit images with added noise (either random Gaussian noise or extra digit parts in the background). These generalization tests are distinct from the typical ways of testing a network (i.e. with unseen exemplars that are similar to the training set). However, such far transfer tests more closely resemble the kind of generalization performed by the human visual system. Selective pruning for sparsity significantly increased recognition accuracy in the far transfer tasks by maximum 38%. A principal component analysis of the features encoded by the network showed that increased sparsity makes the digit representations more distinct. However, we also find that representational sparsity is bounded within a range: while insufficient sparsity reduces coding efficiency, over-sparsity could lead to reduction of generalization to untrained stimuli.

Acknowledgement: NSF grant BCS-1655300 

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.