September 2017
Volume 17, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   August 2017
Top-down neural processing that supplements missing image features revealed by brain decoding with deep neural network representation
Author Affiliations
  • Mohamed Abdelhack
    Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University
    Department of Neuroinformatics, ATR Computational Neuroscience Laboratories
  • Yukiyasu Kamitani
    Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University
    Department of Neuroinformatics, ATR Computational Neuroscience Laboratories
Journal of Vision August 2017, Vol.17, 179. doi:10.1167/17.10.179
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to Subscribers Only
      Sign In or Create an Account ×
    • Get Citation

      Mohamed Abdelhack, Yukiyasu Kamitani; Top-down neural processing that supplements missing image features revealed by brain decoding with deep neural network representation. Journal of Vision 2017;17(10):179. doi: 10.1167/17.10.179.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

The problem of visual recognition entails matching the sensory input with the stored knowledge of semantic information. This process involves a feed-forward component where visual input is processed to extract features characterizing different objects. It is also presumed to involve an opposite processing pathway where high-level features propagate back providing prediction on the kind of visual stimulus presented. This top-down pathway appears to be useful particularly for tasks like processing degraded stimuli. The process by which the forward and backward pathways integrate leading to visual perception is still largely unknown. Here, using a deep neural network (DNN) trained to recognize objects as a proxy for hierarchical neural representations (Horikawa & Kamitani, 2015), we demonstrate that top-down processing pathway attempts to supplement missing visual features. We first trained multivoxel fMRI decoders to predict DNN features of multiple layers for stimulus images. The trained decoders were then used to analyze independent fMRI data collected while viewing pairs of normal and degraded images. Degraded images were created by blurring original images using averaging filters, and by binarizing slightly blurred images using thresholding. We found that decoded features from fMRI responses to degraded images were more correlated to DNN features calculated from the original images than from the degraded (presented) ones. This was especially salient in the lower level DNN representations. We also found that the task of categorizing the visual stimuli increased the correlation difference especially in higher visual areas. These results suggest the operation of the top-down pathway as it attempts to supplement the missing information in degraded images. The effect of the categorizing task may indicate how giving a prior guides the exploration efforts towards successful perception. This DNN-based brain decoding approach may reveal the interactions between the bottom-up and the top-down pathways, providing empirical evidence for existing and novel theoretical models.

Meeting abstract presented at VSS 2017

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×