Abstract
Many objects have a natural place in the world—a context where they and other objects are typically found. For example, tea kettles and stoves are often found together in kitchens, while fire hydrants and traffic lights are found on city sidewalks. This type of contextual knowledge about object co-occurrence can help people identify where they are in the world and what other objects they might encounter. Investigations of object perception have long sought to understand how contextual knowledge is represented in the brain (Bar, 2004), but conclusive evidence for such representations has remained elusive. Here we used fMRI and machine learning to test the hypothesis that object co-occurrence statistics are encoded in the human visual system and automatically elicited by the perception of individual objects. Using a statistical-learning technique from computational linguistics and a database of ~22,000 densely labeled scenes (ADE20K), we learned a set of 8-dimensional representations (object2vec) that captured the latent statistical structure of object co-occurrence in natural images. We mapped these statistical representations onto cortical responses through voxel-wise encoding models of fMRI data collected while four participants viewed images of isolated objects (81 categories, 10 exemplars per category) and performed a perceptual task by responding whenever the stimulus was a visually warped object. We found that an anterior portion of the scene-selective parahippocampal place area links objects with representations of the statistical ensembles in which they typically occur. In contrast, the semantic properties of objects that could be learned through language-based co-occurrence statistics (word2vec) engaged representations in the neighboring object-selective posterior fusiform gyrus. Together these findings reveal a mechanism for encoding statistical regularities of object co-occurrence, providing probabilistic information about the natural contexts in which objects are typically encountered.
Acknowledgement: NIH R01 EY022350