Abstract
How visual concepts are represented in the brain varies across tasks, contexts, and time. We developed a model-based behavioral method to investigate the level of abstraction of visual representations in these different situations. First, we synthesize stimuli that correspond to representations of specific natural images in different deep neural network (DNN) layers, with random DNN features added in different proportions (noise levels). We then ask observers to perform a task on these stimuli, quantify accuracy for each layer and noise level, and compute a discrimination threshold per layer. The layer that leads to the lowest threshold is deemed closest to the representational format. We demonstrated the validity of this approach in a behavioral experiment with 50 participants. Synthesized stimuli were created from features of a cat or dog image in an intermediate or high DNN layer, and observers categorized them as depicting either a cat or a dog. Half of participants were shown the two base images used to create the stimuli before the experiment, allowing them to search for specific image features (image group). The other half were not shown the images and thus could only rely on categorical features (category group). We hypothesized that the image group would have a lower threshold for the intermediate DNN layer and the category group for the high DNN layer. This pattern was largely reflected in a significant group x layer interaction of thresholds. The category group had a lower threshold for the high DNN layer, whereas the image group did not. This indicates that observers who were unaware of the original images relied on more abstract features. We further compared visual reconstructions of the observers’ mental representations. In conclusion, this method opens up new possibilities for investigating visual attention, prediction, and learning.