Abstract
Sparse coding in the human visual system has been viewed as a fundamental mechanism to provide increased representational capacity and efficiency with discriminative features. For artificial networks, sparsity has been introduced in numerous ways, e.g., in the form of dropout during training, rectification of activation, or wholesale removal of network nodes or individual connections. However, the main goal of sparsifying artificial networks is to reduce overfitting and network size and the methods for enforcing sparsity are determined more by computational convenience than generalization improvement. We compared standard methods of introducing sparsity in deep learning models, such as dropout, with more human-like schemes like permanent, targeted removal of weights or nodes as might happen during synaptic pruning during normal human development. A series of simulations were systematically conducted using the handwritten digit dataset (MNIST) with a simple, fully-connected three-layer network. We show that introducing sparsity with human-like schemes can significantly improve generalizability in the form of far transfer to untrained datasets, such as digit images with added noise (either random Gaussian noise or extra digit parts in the background). These generalization tests are distinct from the typical ways of testing a network (i.e. with unseen exemplars that are similar to the training set). However, such far transfer tests more closely resemble the kind of generalization performed by the human visual system. Selective pruning for sparsity significantly increased recognition accuracy in the far transfer tasks by maximum 38%. A principal component analysis of the features encoded by the network showed that increased sparsity makes the digit representations more distinct. However, we also find that representational sparsity is bounded within a range: while insufficient sparsity reduces coding efficiency, over-sparsity could lead to reduction of generalization to untrained stimuli.
Acknowledgement: NSF grant BCS-1655300