September 2019
Volume 19, Issue 10
Open Access
Vision Sciences Society Annual Meeting Abstract  |   September 2019
Neural Encoding and Decoding with Convolutional Autoencoder for Predicting Emotional Judgment of Facial Expressions
Author Affiliations & Notes
  • Gary C.W. Shyi
    Department of Psychology, National Chung Cheng University, Taiwan
    Center for Research in Cognitive Science, National Chung Cheng University, Taiwan
  • Wan-Ting Hsieh
    Department of Electrical Engineering, National Tsing Hua University, Taiwa
  • Felix F.-S. Tsai
    Department of Electrical Engineering, National Tsing Hua University, Taiwa
  • Jeremy C.-C. Lee
    Department of Electrical Engineering, National Tsing Hua University, Taiwa
  • Shih-Tseng Tina Huang
    Department of Psychology, National Chung Cheng University, Taiwan
    Center for Research in Cognitive Science, National Chung Cheng University, Taiwan
  • Joshua O. S. Goh
    Graduate Institute of Brain and Mind Sciences, National Taiwan University, Taiwa
  • Ya-Yun Chen
    Center for Research in Cognitive Science, National Chung Cheng University, Taiwan
  • Chi-Chuan Chen
    Graduate Institute of Brain and Mind Sciences, National Taiwan University, Taiwa
  • Yu Song Haw
    Center for Research in Cognitive Science, National Chung Cheng University, Taiwan
Journal of Vision September 2019, Vol.19, 260c. doi:https://doi.org/10.1167/19.10.260c
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Gary C.W. Shyi, Wan-Ting Hsieh, Felix F.-S. Tsai, Jeremy C.-C. Lee, Shih-Tseng Tina Huang, Joshua O. S. Goh, Ya-Yun Chen, Chi-Chuan Chen, Yu Song Haw; Neural Encoding and Decoding with Convolutional Autoencoder for Predicting Emotional Judgment of Facial Expressions. Journal of Vision 2019;19(10):260c. doi: https://doi.org/10.1167/19.10.260c.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Convolutional Autoencoder (CAE) has become a popular approach to building artificial systems endowed with the capability of automatic emotion recognition under unsupervised learning. Here, we explored the viability of using CAE to unravel the neural encoding and decoding for predicting emotional judgment of facial expressions among human observers. Specifically, 25 young adults were asked to discriminate faces of varying degree of emotional expression with the neutral faces. The mean difference threshold for discriminating emotional expressions was then derived from averaging across the difference threshold for each of the six basic emotions (i.e., happiness, anger, sadness, surprise, fear, and disgust), and participants were then divided into high- versus low-thresholders based on the median split. They were then asked to explicitly judge the emotional intensity of displayed faces while their brains undertook fMRI scanning. The BOLD signals of 47,636 voxels based on the 90 ROIs specified according to the Automated Anatomical Labelling (AAL) were fed to a one-dimensional CAE model with three encoding layers and three decoding layers. The results of reconstructed brain activities based on the learned voxel weights were then remapped onto the AAL-defined ROIs to locate brain regions that may contribute differentially to the emotional judgments of facial expression among high- and low-thresholders. For low-thresholders who actually required smaller image differences to discriminate facial expressions, the results of the CAE suggests that brain regions that are involved in basic visual processing appear to play a greater role. In contrast, for high-thresholders who required greater image differences to discriminate facial expressions, brain regions responsible for emotion processing and those that are involved in top-down emotional regulations appear to play a greater role. Taken together, these findings highlight the utility of using the CAE, in conjunction with neuroimaging data, to unravel brain mechanisms underpinning emotional processing of facial expressions.

Acknowledgement: The Ministry of Science and Technology, Taiwan, ROC 
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×