Abstract
When we view a picture or read a word, we evoke its meaning (semantics) based on prior knowledge of the concept. We previously showed that pictures and words describing scene categories evoke similar representations in the inferior frontal gyrus, precuneus, angular gyrus and ventral visual cortex. Although we know the neural representations of pictures of tools and animals differ, are there similar differences when these concepts are evoked through words? To examine the nature of differences and similarities between these concepts, we examine the neural activity for pictures and words using a classifier to decode the BOLD signal within modalities and across modalities via multivariate pattern analysis (MVPA). We used picture stimuli of animate and inanimate exemplars from twenty-four categories derived from six superordinate classes (big cats, insects, birds, vehicles, tools and fruits). Word stimuli were approximate "captions" of the types of pictures used for the same category (e.g. 'a panther hunting at night'). In the fMRI experiment, subjects passively viewed first all the word stimuli for all twenty-four categories and then the picture stimuli for the same categories. A whole brain MVPA searchlight with six-way classification and a fine-grained four-way sub-class classification was performed. The six-way results from decoding using only the word stimuli revealed a distributed set of brain regions in the left hemisphere, including, the angular gyrus, precuneus, inferior frontal gyrus and putative visual areas in the lateral and ventral occipito-temporal cortex. Picture decoding was more extensive but revealed these same regions. Like natural scenes, we found that we could cross-decode between words and pictures of objects in inferior frontal gyrus, precuneus, angular gyrus, consistent with these regions being category general semantic hubs whereas the ventral visual regions show a differential specificity between the concepts for animals, tools and natural scenes.
Meeting abstract presented at VSS 2016