Abstract
When people compare two objects that belong to the same category, they rely on visual dimensions that may be irrelevant to objects in another category. For instance, texture may be more important for distinguishing between different fruits than between different cars. Deep convolutional neural networks (DNN) learn visual features that can be used to approximate human similarity judgments (HSJ). Current understanding of the relation between DNN features and HSJ is based on models that consider all DNN features for computing pairwise similarity. Recent reweighting approaches use HSJ to supervise the learning of weights assigned to each feature, but similarly assign weights to all features. These may be computational and theoretical limitations: we hypothesized that when modeling the representational geometry of any given domain, many DNN features will not encode relevant information at all, and their removal may even improve the predictions of HSJ. We used unsupervised methods to identify those DNN features which do not contribute to approximating HSJ. We used 6 datasets, each containing 120 natural images. For each set of images separately, we considered less-relevant features as ones with: large percentage of zero activations, low standard deviation, low entropy, or high collinearity with other features. Our results show that: 1) For all datasets, up to 90% of features could be pruned without a meaningful drop in the predictions of HSJ, and for some datasets we observed slight but statistically significant improvements; 2) Pruning introduced slight and gradual increasing changes in the representational geometry of the embedding space; 3) The dominant latent dimension of the pruned embeddings was sufficient for ordering categories on a similarity continuum. In summary, while irrelevant features are common in images of natural categories, they are weakly informative when modeling HSJ, and in some cases their removal can produce better approximations of human representational geometry.