Abstract
In many fMRI studies, viewing behavior is a major variable of interest, or one of confound. Concurrent eye tracking is expensive, often time consuming to set up and imposes experimental constraints (e.g. the eyes need to be open). Here, we developed DeepMReye; a deep-learning-based framework to decode viewing behavior from the MR-signal of the eye balls (see Frey et al., VSS2020). We trained and tested the model on data of more than 250 participants acquired on six 3T-MRI scanners with a variety of scanning protocols. Participants performed diverse viewing tasks including fixation, guided saccades and smooth pursuit, visual search, free movie- and picture viewing as well as eye movements when the eyes were closed. Our model successfully recovers gaze position and associated variables such as direction and amplitude at sub-TR resolution during these tasks, without the need for eye tracking equipment. A confidence score obtained for each decoded sample further indicates the intrinsic model certainty. Critically, our model generalizes across participants, tasks and MR-scanners, suggesting that viewing behavior could be reconstructed post-hoc even in existing fMRI data sets. To test this, we explore the boundary conditions and generalizability across fMRI-scanning protocols by systematically varying voxel size and repetition time (TR) in a subset of participants with concurrent eye tracking. In sum, DeepMReye allows to decode viewing behavior post-hoc from fMRI data, which can be integrated into existing fMRI pipelines to study or account for gaze related brain activity.