Saliency and the population receptive field model to identify images from brain activity

Abstract

One of the goals of visual neuroscience is to develop predictive models of brain activity to better understand the underlying mechanisms of the visual cortex. One example is the identification of presented images from fMRI recordings. Recently, Zuiderbaan et al. (2017) used the population receptive field (pRF) model and image contrast information to identify the presented stimulus from a set of natural images. Here, we extend the study by analyzing the predictive power of saliency-related information. Ultimately, we seek answers to these questions: What fraction of the responses is driven by saliency? Where in the visual cortex is saliency most represented? We test the Deep Gaze II and ICF (Kümmerer et al., 2017) saliency models. Deep Gaze II predicts the fixation density by using high-level features computed from a deep artificial neural network, while ICF uses only intensity and contrast features. We calculate the prediction response profile of every image as the summed overlap of its saliency map with the pRF at each cortical location. Then, we compute the correlation between the fMRI recordings and the prediction profiles of all images. An image is correctly identified if its predicted response elicits the highest correlation of the set. We find that ICF achieves the highest performance with a median correlation of .44 and 46.7 % accuracy (baseline 2.2 %) on V1, while Deep Gaze and the contrast model achieve .34 and .26 median correlation and 34.4 and 33.3 accuracy, respectively. On higher visual areas the performance gradually decreases. Both saliency models seem to be therefore better predictors of the presented stimuli than contrast information. The better performance of ICF might be explained because, while the pRF model only captures contrast energy information, the ICF maps may contain other information present in the measured responses, hence increasing the correlation of the predictions.

Acknowledgement: This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 641805