October 2020
Volume 20, Issue 11
Open Access
Vision Sciences Society Annual Meeting Abstract  |   October 2020
Precise identification of semantic representations in the human brain
Author Affiliations & Notes
  • Ian Charest
    School of Psychology, University of Birmingham, UK
    Centre for Human Brain Health, University of Birmingham, UK
  • Emily Allen
    Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, USA
  • Yihan Wu
    Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, USA
  • Thomas Naselaris
    Neurosciences Department, Medical University of South Carolina, USA
  • Kendrick Kay
    Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, USA
  • Footnotes
    Acknowledgements  ERC-StG 759432, NSF IIS-1822683, NIH P41 EB027061, NIH P30 NS076408, NIH S10 RR026783, W. M. Keck Foundation.
Journal of Vision October 2020, Vol.20, 539. doi:https://doi.org/10.1167/jov.20.11.539
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Ian Charest, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay; Precise identification of semantic representations in the human brain. Journal of Vision 2020;20(11):539. doi: https://doi.org/10.1167/jov.20.11.539.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Previous studies investigating semantic representations in the human brain present conflicting results regarding the localisation of such representations and whether they are distributed or modular. This is possibly due to the limited sensitivity and/or limited stimulus samplings of past experiments. We conducted a large-scale, high-field fMRI experiment (7T, whole-brain, gradient-echo EPI, 1.8 mm, 1.6 s) in which 8 subjects each viewed 9,000–10,000 colour natural scenes (taken from Microsoft COCO) while fixating and performing a continuous recognition task. We exploited this unprecedented dataset, termed the Natural Scenes Dataset (NSD), to investigate semantic representations in the brain using representational similarity analysis (RSA). Capitalising on the large array of naturalistic scenes, we obtained human-derived semantic labelings (sentence descriptions) from COCO and applied smooth inverse frequency sentence embeddings to construct semantic representational dissimilarity matrices (RDMs). We then used searchlight-based RSA in NSD to identify brain regions whose representation closely matches the semantic RDM. Significance was assessed using permutation testing. In each subject, we measured highly significant and reliable (t-values reaching 88) semantic representations in a highly distributed network including the ventral visual stream, the angular gyrus and temporoparietal junction, the medial and anterior temporal lobes (anterior hippocampus, perirhinal cortex), and the inferior frontal gyrus. In a separate analysis, we constructed RDMs based on a recurrent convolutional neural network (RCNN) model of object recognition and found that these RDMs mapped to a strikingly similar network of brain regions, albeit with weaker correlations. The immense scale of NSD allows precise quantification of semantic representations in individual subjects across the entire human brain. The promising performance levels achieved by the RCNN indicates that the NSD dataset can support further detailed investigations of the types of semantic brain representations that can and cannot be captured by current computational models of visual scene interpretation.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.