Abstract
Previous research reported high-level object representations in the visual ventral stream (VS) at the interface between perception and semantics, reflecting behavioural similarity judgements. Here we used representational similarity analysis (RSA) to investigate object recognition (36 objects including bodies, faces, objects and scenes) comparing three behavioural tasks with functional magnetic resonance imaging (fMRI, 3T, n=20) and electroencephalography (EEG, 128 channels, n=20). In an attempt to capture implicit similarity representations, we used a novel discriminability task. As a measure of explicit similarities, we collected behavioural judgments using the multiple-arrangements (MA) task. Finally, to capture semantic representations, we obtained human-derived caption labels. We predicted that implicit similarity judgements would better capture perceptual representations, while explicit and semantic judgments would extend to higher-level, semantic or conceptual brain processes. Representational dissimilarity matrices (RDMs) were estimated using drift-rates for the discriminability task, using pixel euclidean distances for the multi-arrangements task, and using sentence embeddings of the human captions for the semantic labelling task (Figure 1A). We relate these behavioural RDMs to time-resolved RDMs (EEG, Figure 1B) and searchlight-defined RDMs (fMRI, all statistics are reported with p<0.001 corr., Figure 1C). The EEG-RDMs showed large and early peak correlations with the discriminability task, and weaker correlations for the semantic and multi-arrangement tasks (later in time). In fMRI, we see all three behavioural tasks strongly mapping (t values reaching 48.9) on the visual ventral stream (strongest for the discriminability task). The multi-arrangement and semantic tasks additionally involved higher-order regions of the medial temporal lobes and prefrontal cortices. Our results suggest that implicit similarity judgements better capture representational geometries in visual cortex but fail to predict higher-order representations. In contrast, multi-arrangements and semantic judgements account for higher-level representations. Therefore, we propose that accounting for behaviourally relevant information processes in the brain should rely on a battery of behavioural metrics.