Verb-driven shifts of attention during sentence comprehension and dynamic scene processing

Abstract

We investigated how dynamic scene perception and sentence comprehension processes interact. Eye movements were monitored while participants watched motion pictures of events and listened to related sentences. Each scene contained an agent (corresponding to the grammatical subject of sentence) and different types of objects (with one corresponding to the grammatical object of the main verb in the sentence; the target). Three types of agent motion conditions were created: (a) towards the target object, (b) away from the target object, and (c) neutral — in which the agent did not move towards or away the target object. These three types of scenes were combined with sentence pairs differing only by verb type (e.g., Before preparing the dessert, the cook will crack/inspect the egg that is on the counter). We contrasted highly constraining causatives (e.g., crack, bend, fold) with weakly constraining psychological or perceptual verbs (e.g., see, notice, inspect). Participants were instructed to watch the movies and to listen to their respective sentences. Verb onsets were synchronized with frames in the movies that signaled the beginning of agent motion. We measured saccade onset times to target objects from verb onsets. Results showed a significant effect of motion type (neutral slower than “away” and “towards”). There was also a significant effect of verb type (causative faster than perception/psychological) only in the “away” condition. These results will be discussed in the context of a model of the interaction between independently processed visual and linguistic inputs.

Footnotes

 Research funded by grants from FQRSC and SSHRC (to RGdA) and from NSERC (to MWvG)