Abstract
Revealing the features of mental representations is a longstanding goal of cognitive psychology. Although progress has been made in uncovering some low-level representations, there is currently no general framework for investigating representations of high-level visual concepts. We developed a computational method to parametrically map points in the semantic space of category labels to points in the space of visual features, allowing us to reconstruct the representations of many common visual concepts. Specifically, we synthesized “CNN-noise” images from random features in an intermediate layer of a convolutional neural network (CNN) and asked 100 observers to indicate what they saw in each image. We translated their written responses to vectors in a continuous space using a semantic embedding. We then used regressions to uncover how each CNN feature correlated to each dimension in this semantic space. Using this semantic-visual mapping, we could extract the CNN features associated with any concept. From these features, we could then synthesize an image to visualize the concept’s prototypical representation. We assessed the quality of these reconstructions (e.g., “grass”, “dog”, “night”) in a separate behavioral validation experiment with 35 observers: 252 of 350 reconstructions were recognized significantly better than chance, suggesting that our method succeeded in visualizing mental representations. We then assessed whether we could predict the semantic content perceived by observers in held-out CNN-noise images: we were able to generate labels closer to the true labels than labels generated by the CNN. Our model also explained similarity judgments of written visual concepts better than the semantic embedding. Finally, it explained unique variance in object representations from high-level visual cortex in fMRI, further suggesting that we captured the structure of mental representations. In conclusion, we developed a computational framework to integrate visual features and semantic dimensions, allowing us to reveal the features and structure of visual representations.