Abstract
Humans can extract visual information from complex scene images in as little as 17ms, and use it to interact within their environments, for instance by navigating through a scene (e.g. “I can go left”), or categorizing it (e.g. “this is a kitchen”) within 220 ms. fMRI studies have demonstrated that the cognitive processes underlying these tasks are supported by distinct neural systems: navigation is supported by the occipital place area; categorization is supported further downstream in the ventral visual pathway, by the parahippocampal place area. Taken together, these results imply that a complete account of task-based influences in information extraction from visual scenes requires considering both spatial (where in the brain) and temporal (when during processing) aspects of the interaction between the task being performed, and the task-relevant information in the visual scene. To that end, we recorded 20 participants’ EEGs while they completed 3 behaviorally matched tasks (navigation, categorization, and one-back repetition) on the same set of scene stimuli that were counterbalanced across categories (bedrooms, kitchens, living rooms) and navigable paths (left, center, right). In order to analyze the task-by-category-by-path interaction over time without losing spatial information, we developed a novel “with-or-without-you” representational similarity analysis (WOWY-RSA), and applied it to the ERP data. This allowed us to predict not only the time-course of the interaction, but also which electrodes contributed significant amounts of task-, category-, and path-relevant information to the overall RSA, thus preserving the EEG’s coarse spatial information. WOWY RSA revealed significant early decoding of these three sources of information at lateral central and central posterior electrode sites around 100ms after stimulus onset, and later decoding at fronto-central and lateral central electrode sites from 370-550ms. Overall, our results agree with the hypothesis that information extraction from visual scenes is task dependent, implicating spatiotemporally distinct neural mechanisms.