Abstract
Neural networks trained on large image datasets have been shown to successfully model the ventral visual stream. However, there is a lack of understanding of the features captured by these models. Here, we investigate an interpretable alternative to deep learning models called the scattering transform. Similar to convolutional neural networks, scattering transforms have a hierarchical structure with multiple layers implementing convolutions, non-linear activations, and pooling. However, instead of using learned convolutional kernels, these models use pre-defined Morlet wavelets at different orientations and spatial scales. In a forward propagation, an image is passed through a convolutional layer, followed by a modulus activation function, and this process is repeated across multiple layers in a hierarchical manner. During read-out, scattering coefficients are obtained by average pooling of the activations for each wavelet in a layer. Each scattering coefficient reflects a specific combination of orientations and spatial scales across layers, and the scope of the pooling operation determines how much local spatial information is preserved. In our analysis, we focus on global scattering coefficients from the first two layers (S1 and S2), which reflect summary statistics of contours and contour co-occurrences. We fit voxelwise encoding models of category-selective areas using these scattering coefficients. First, we find that the S2 encoding model cross-validation performance is on par with AlexNet and outperforms S1 in both scene- and object-selective areas, suggesting that S2 encodes important image statistics captured by high-level visual areas. Second, we find that perpendicular contour co-occurrences in S2 outperform parallel contour co-occurrences for explaining both the scene- and object-selective areas. This suggests that these areas are sensitive to local changes in orientation that define basic elements of shape. Our findings suggest that the scattering transform may be a powerful and interpretable alternative to deep learning models of feature representation in high-level visual cortex.