September 2017
Volume 17, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   August 2017
Semantic knowledge and action-based visual prediction
Author Affiliations
  • Nicholas Hindy
    Princeton Neuroscience Institute, Princeton University
  • Emily Avery
    Princeton Neuroscience Institute, Princeton University
  • Nicholas Turk-Browne
    Princeton Neuroscience Institute, Princeton University
Journal of Vision August 2017, Vol.17, 1169. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Nicholas Hindy, Emily Avery, Nicholas Turk-Browne; Semantic knowledge and action-based visual prediction. Journal of Vision 2017;17(10):1169.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Goal-directed behavior requires predicting the consequences of our actions. Predictions based on specific episodes in our past may arise from pattern completion in the hippocampus, which in turn reinstates the retrieved consequences in visual cortex (Hindy et al., 2016, Nature Neurosci). However, action-based predictions are not always tied to specific episodes, but instead to general semantic knowledge learned pedagogically or aggregated from many experiences over a long timescale. For example, when cracking open an egg we expect to see a yoke. It is unknown how such visual predictions based on long-term semantic memory are generated. We hypothesized that they may also be supported by the hippocampus. To test this hypothesis, we collected high-resolution fMRI data while participants selected among familiar actions that changed the state of one object into another. For the Novel condition, participants learned actions that changed objects in arbitrary ways (e.g., "point" or "wave" to transform a dollar bill into a carrot or a newspaper), and thus predictions needed to be based on recently experienced episodes. For the Known condition, the outcomes of actions were obvious (e.g., when you "roll" or "fold" a dollar bill) because predictions could rely on pre-existing semantic knowledge. In both conditions, responses in early visual cortex were attenuated when an outcome could be predicted from an action, and this predictive attenuation was related to background connectivity with the hippocampus. Examining the hippocampus more closely, there was a dissociation over the long axis, with Novel prediction in visual cortex related to the anterior hippocampus and Known prediction related to posterior hippocampus. These findings suggest that the hippocampus may mediate visual prediction even based on long-term semantic knowledge, but possibly through different mechanisms than observed for visual prediction based on recent episodic learning.

Meeting abstract presented at VSS 2017


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.