Abstract
Extensive work has documented the perception of facial expressions, with particular focus on a handful of basic emotional expressions. However, the larger scope of facial expressions and their representational basis are less understood. Further, the neural processing of dynamic facial expression awaits clarification.
Here, we address these challenges through EEG-based decoding and image/movie reconstruction. Specifically, EEG data were collected from healthy adults who viewed 1-second videos displaying 24 dynamic facial expressions while performing an oddball task (i.e., angry expression recognition). Each stimulus started with a neutral expression and transitioned to displaying a peak expression over the course of ten 100ms frames. In addition, participants performed a behavioral pairwise rating task with the 24 expressions. EEG-based pattern analyses were carried out for expression decoding and, then, movie reconstruction was conducted based on the resulting decoding patterns.
Our investigation reveals, first, that the representational space of facial expression derived from both behavioral and neural data is broadly, though not exclusively, accounted for by the classical dimensions of valence and arousal. Second, the temporal profile of EEG-based decoding shows a gradual increase after 350ms reaching a classification plateau between roughly 600-1200ms following stimulus onset. Further, a frame-by-frame analysis indicates that peak accuracy is reached after the midway point of stimulus presentation (i.e., at the 6-7th frame of a video), prior to peak expression display (i.e., the 10th frame). Third, movie reconstruction is successfully achieved from the EEG signal and, consistent with the decoding results, reconstruction accuracy is maximized after the midway point of stimulus presentation.
Thus, our results shed light on the representational structure and the neural processing of dynamic facial expressions. Of note, they elucidate with an unprecedented level of detail the pictorial content of expression representations while also providing proof of concept for the possibility of EEG-based dynamic stimulus reconstruction.