Abstract
Representational similarity analysis has become a popular tool to study the nature of visual object representations (Kriegeskorte et al, 2008). Such analysis aims at characterizing patterns of brain activity via a so-called similarity matrix, which contains a similarity measure between all pairs of activity patterns elicited by a given stimulus set. This approach is particularly suited for comparing patterns of neural activity measured with different experimental techniques. Here we describe a computational approach based on distance-learning techniques and a statistical framework to uncover which stimulus dimensions are emphasized in the underlying representation. Our test is based on a restricted permutation test (Good, 2000) that we generalized to create a new “two-layer” test. The new test is specifically designed for hierarchically structured data such as natural object categories, to assess what level of a hierarchy a similarity matrix shares with the visual representation.
As a validation of the approach, we applied our method to the representational similarity analysis from Kriegeskorte et al. (2008) based on patterns of fMRI activity response in human and monkey inferior temporal (IT) cortex to natural images of objects. We show that an image representation based on relatively low-level color and shape features is able to account for the underlying similarity matrix. These results cast doubts on the original interpretation of the data suggesting a high-level semantic visual representation at the level of IT. Using the proposed method we are able to relate different aspects of the similarity matrix to different types of low level features. Our findings suggest that careful consideration should be taken when conducting experiments with natural object categories.
This study was supported by the European Union under DIRAC integrated project IST-027787, and by a DARPA grant to TS (DARPA-BAA-09-31).