Exploring visual object representations with similarity-matrix analysis

Abstract

We propose a novel approach to studying visual object representations on the basis of multi-channel brain-activity data as provided by high-resolution functional magnetic resonance imaging (hi-res fMRI). The core idea is to describe a region's representation of a set of object images in terms of the pairwise similarities of the multivariate response patterns elicited by the stimuli. Analogously, the predictions of theoretical models of the representation (including simple category models as well as computational models processing the actual stimuli) can be expressed in terms of the pairwise similarities between the response patterns associated with the stimuli. The similarity matrix, as an abstract description of the representational space, can then be used to quantitatively relate the data from each brain region to each of the models.

The similarity matrix is a square matrix containing a similarity estimate for each pair of single-image response patterns within a given region. For each pair of stimuli, the spatial response patterns are compared by linear correlation. We then assess how well each model's similarity matrix fits each functional region's similarity matrix. In addition, we continuously map the model fits by moving a multivariate searchlight (in the form of a spherical cluster of voxels) throughout the imaged volume.

Results indicate that the human inferotemporal representation of visually presented objects is well described as a categorical distributed code: a simple model of basic-level object categories (as has been widely assumed but never tested) predicts the similarity structure more accurately than other models. The human inferotemporal similarity matrix also appears to match the similarity matrix obtained by analogous analysis of monkey-IT single-cell responses to the same images.