The effect of object-scene associations upon representational similarity dissociates structured from image-based representations

Abstract

We live in a structured world. Some objects are most likely to appear in specific contexts: penguins in ice landscapes and lions in the savannah. How are such statistics of the world represented in the biological and artificial brain? Two main views can be considered: one assuming a structured representation (e.g., separating foreground from background), another centred on image-based computations (e.g., HMAX). Previous experiments that compared biological and artificial neural representations did not distinguish between structured and image-based representations. The effect of this distinction becomes apparent when we consider the problem of statistical regularities. In a structured representation, statistical regularities between identified components can be dissociated from the representation of the components themselves. In an image-based framework, there are no identified components and as such the coding of statistical regularities is more entangled with the representation of the components. Here we test these alternative perspectives with a stimulus set that includes (1) animals (on neutral backgrounds) and (2) their associated scenes, such as ladybugs and leaves. An fMRI event-related experiment (n = 20) confirmed that representations in visual cortex separate objects from their background; no clear representation similarity was observed for associated animals and scenes. In pre-trained DNNs, we found a much stronger entanglement of animals and their associated scenes (e.g., a gorilla and the jungle). This representational entanglement increased towards the last DNN layer. In sum, we show that the nature of representations in an information processing system, being structured or more image-like, has strong consequences for how statistical regularities are coded in such a system. Experiments that probe this nature of representations provide a powerful illustration of the uniqueness of human information processing compared to artificial neural networks.