Neural Encoding and Decoding with Convolutional Autoencoder for Predicting Emotional Judgment of Facial Expressions

Abstract

Convolutional Autoencoder (CAE) has become a popular approach to building artificial systems endowed with the capability of automatic emotion recognition under unsupervised learning. Here, we explored the viability of using CAE to unravel the neural encoding and decoding for predicting emotional judgment of facial expressions among human observers. Specifically, 25 young adults were asked to discriminate faces of varying degree of emotional expression with the neutral faces. The mean difference threshold for discriminating emotional expressions was then derived from averaging across the difference threshold for each of the six basic emotions (i.e., happiness, anger, sadness, surprise, fear, and disgust), and participants were then divided into high- versus low-thresholders based on the median split. They were then asked to explicitly judge the emotional intensity of displayed faces while their brains undertook fMRI scanning. The BOLD signals of 47,636 voxels based on the 90 ROIs specified according to the Automated Anatomical Labelling (AAL) were fed to a one-dimensional CAE model with three encoding layers and three decoding layers. The results of reconstructed brain activities based on the learned voxel weights were then remapped onto the AAL-defined ROIs to locate brain regions that may contribute differentially to the emotional judgments of facial expression among high- and low-thresholders. For low-thresholders who actually required smaller image differences to discriminate facial expressions, the results of the CAE suggests that brain regions that are involved in basic visual processing appear to play a greater role. In contrast, for high-thresholders who required greater image differences to discriminate facial expressions, brain regions responsible for emotion processing and those that are involved in top-down emotional regulations appear to play a greater role. Taken together, these findings highlight the utility of using the CAE, in conjunction with neuroimaging data, to unravel brain mechanisms underpinning emotional processing of facial expressions.

Acknowledgement: The Ministry of Science and Technology, Taiwan, ROC