Modeling fMRI responses to complex dynamic stimuli with two-stream convolutional networks

Abstract

Previous works reported correspondences between the early stages of visual processing and early layers of convolutional neural networks (CNNs) and between later stages of visual processing and later layers of CNNs. However, most previous studies comparing CNNs to neural responses focused on static stimulus features. Dynamic stimuli have been shown to elicit stronger responses in several visual regions, but little is known about how well CNN accounts for responses to dynamic stimuli. Here, we compare fMRI responses to quasi-naturalistic videos (Forrest Gump) to representations in a two-stream CNN: one stream encodes static features, and the other encodes dynamic features. Analysis of the static features of the CNN replicates the finding that features in early layers capture early visual responses (V1) better than later responses (IT) and reveal additional effects in the frontal cortex. However, the correlation between static CNN features and neural responses to complex dynamic stimuli is lower than that reported for static images. We then analyze the correlation for the dynamic features of the CNN, using searchlight analysis to investigate the relative contribution of the two types of features (static vs. dynamic) across different brain regions. We conclude that static CNN features explain only a limited amount of variance in neural responses to complex dynamic stimuli and that richer models are needed to account for how the visual system processes these types of stimuli.