Abstract
Deep neural networks (DNNs) are appealing as models of human vision because they categorize complex images with human-like performance — i.e. they “do the categorization task” with unprecedented accuracy. However, there is a considerable gap between doing the task with similar performance and doing it like the human brain, using similar stimulus representations. Thus, before evaluating the potential contribution of DNNs as models of the brain, we first need to ensure that they perform the same task with the same features (e.g. to categorize a happy face with its diagnostic mouth and wrinkled eyes).
Here, we exploited the tight experimental control over such stimulus features afforded by an interpretable and realistic generative model of face information (GMF, SuppMatA), which generated 3.5M images with varying categorical factors (2,004 identities, 2 genders, 3 ages, 2 ethnicities and 7 expressions of emotion, and 3 vertical and horizontal angles of pose and illumination). We trained (> 99% accuracy) two ResNet10s to classify 2,004 identities across image variations (ResNetId) plus the multiple other factors of the GMF causing these variations (ResNetMulti).
Following training, we reverse correlated the internal representations of 4 familiar faces in humans and the ResNet models. All faithfully (and unfaithfully) represented the 4 identities using similar shape features (e.g. a forehead or chin, SuppMatC&D). However, face noise testing (i.e. GMF shape vs. texture noise, SuppMatB) revealed that ResNetId generalization behavior was hyposensitive to shape but hypersensitive to texture, whereas ResNetMulti was less extreme. These generalization biases were reflected in the hidden layers, as shown in a comprehensive analysis of activation variance in each layer up to decision (SuppMatE).
In sum, we enhanced the understanding of how DNNs “do the task” by first establishing a similarity of decision features with humans, before tracing how the DNNs’ layers represent the factors of the GMF.