Introduction to special issue on “Prediction in Perception and Action”

Abstract

The wide diversity of articles in this issue reveals an explosion of evidence for the mechanisms of prediction in the visual system. When thought of as visual priors, predictive mechanisms can be seen as tightly interwoven with incoming sensory data. Prediction is thus a fundamental and essential aspect not only of visual perception but of the actions that are guided by perception.

Introduction

Prediction has long been recognized as an important feature of human behavior, and predictive mechanisms are found at many different levels of processing. Significant sensory-motor delays present a problem in a dynamically changing environment, and it is therefore necessary to predict the future state of the environment based on past experience of how it is likely to change over time. Predictive mechanisms help us to anticipate how our environment will change and to adjust our behavior accordingly. They also allow us to take account of the time required both to process sensory information and to move the body when interacting with our environment. Besides anticipating changes in the outside world, predictive mechanisms allow us to anticipate the future consequences of our own actions, making it easier to distinguish between external and self-generated sensory events. In general, prediction is important from basic levels of sensory-motor control, such as making an eye movement toward a moving object, to the most abstract levels of processing, such as predicting social behavior. 

Some of the clearest examples of prediction come from motor control. Babies learn to predict a moving object's future position within the first year of life (von Hofsten, 2004; Kubicek, Jovanovic, & Schwarzer, 2017). This is more advanced in adult observers, who even appear to be able to predict the trajectory of bouncing balls based on inferences about a ball's physical properties (e.g., Land & McLeod, 2000; Hayhoe & Ballard, 2005; Nusseck et al., 2007; Diaz et al., 2013). In the somatosensory system, it is commonly accepted that the proprioceptive consequences of a planned movement are predicted ahead of time using stored internal models of the body's dynamics (Mulliken & Andersen, 2009; Shadmehr et al., 2010; Wolpert, Miall, & Kawato, 1998), and the comparison of actual and predicted somatosensory feedback is a critical component of the control of movement. Indeed, when somatosensory feedback is severely compromised by somatosensory loss, the consequences for movement can be devastating (Cole & Paillard, 1995). 

Prediction plays a role at many different levels in motor control. For example, in eye movements, efference-copy mechanisms serve as a very basic form of prediction for differentiating self-induced from external visual motion patterns. Accurate motor control for more complex actions is based on processes ranging from simple sensory predictions to internal simulations of complete action sequences (Jeannerod, 1997). Despite the importance of somatosensory predictions, evidence for predictive visual representations has been less clear. The current issue of Journal of Vision provides a wealth of evidence for the importance of visual prediction at all levels of visual representation. 

A number of aspects to prediction are considered in this special issue and need to be distinguished. The most obvious is predicting future visual states of the world, such as the future location of a moving target. Another issue is the need to predict the visual consequences of self-motion. This is essential for separating externally from internally generated retinal motion. This is important for analysis of sensory information at all levels of abstraction, from using retinal motion signals to control pursuit eye movements to using cognitive information to make strategic decisions. Because prediction is based on past experience, the encoding of scene statistics (as visual priors or “internal models” of the world; Kersten et al., 2004; Fiser et al., 2010) is also important, both in terms of how such encoding is achieved and maintained and in terms of how they are encoded within the nervous system. Articles in this special issue cover all these aspects of prediction. This Introduction to the Special Issue provides an overview of the articles published in this issue in the context of the different ways in which visual predictions are important. It starts with predictions that allow one to anticipate changes in the world on the basis of visual information. Next are predictions that allow one to understand the sensory consequences of one's own motion, including eye and arm movements. It ends with studies that combine predictive mechanisms of self-motion and motion in the environment. 

Predicting sensory changes in the world

In its purest manifestation, the visual system should be able to compute perceptual representations that correspond to a predicted future state. This is perhaps easiest to understand in the case of object motion. Assad and Maunsell (1995) showed that neurons in the parietal cortex responded to a moving target throughout a period of stimulus occlusion. That is, the neurons responded as if they were able to extrapolate the response to the currently invisible target from previous exposure. A variety of perceptual phenomena might be related to a predictive representation of motion. Such mechanisms could lead to static stimuli being mislocalized relative to moving stimuli. One such phenomenon is called the “flash-grab” effect. Two articles in this issue by Hogendoorn and colleagues (van Heusden et al., 2019; Blom et al., 2019) provide support for extrapolation using motion signals. They explore the mechanisms involved, suggesting that a predictive signal is active in monocular parts of the human visual pathway. 

When moving to more complex predictions, one must consider how such predictions arise. The statistics of the visual environment shape the visual system, and this in turn shapes visual perception to allow one to make perceptual inferences about the state of the world. Vullings and Madelain (2019) test the idea that predictive mechanisms control learning of temporal and spatial properties of the environment. They use a visual search task in which targets are presented contingent on saccadic reaction times, essentially reinforcing either short- or long-latency saccades by presenting the visual target at a specific time. Their results show that saccade latencies are finely tuned to prediction-driven reinforcement contingencies. Notaro et al. (2019) also provide a demonstration of the process of learning environmental statistics. They monitor such learning by examining small anticipatory drifts and saccades in the direction of the most likely upcoming target. Zoeller et al. (2019) show that such learning can be very specific. They show that somatosensory experience with hard or soft objects modifies the force observers use when interacting with the object. Interestingly, visual or semantic information does not, indicating that the predictions are purely somatosensory. 

Several articles in this Special Issue are concerned with the neural mechanisms underlying prediction. In a now-classic article, Rao and Ballard (1999) introduced the idea of predictive coding. In their model of object recognition, high-level object representations are propagated to early visual areas, where they are subtracted from the incoming visual signals. The mismatch, or residual, reflects the unexplained sensory input that may need a revised model. At all stages of processing, sensory information is compared against predictions of expected sensory events made by higher-level perceptual areas, and the residuals, or prediction errors, are propagated upward to update perceptual models of the environment. This model is the percept. This idea naturally allows prediction in time, although Rao and Ballard (1999) did not explicitly address that issue, and the stored memory representations can be thought of as Bayesian priors. 

A number of recent articles have provided compelling evidence for low-level visual activity prior to stimulus presentation, consistent with the predictive coding hypothesis (Kok et al., 2017; de Lange et al., 2018). Two articles in the Special Issue provide evidence for high-level predictive representations using EEG. Oxner et al. (2019) show that prediction errors in surface segmentation are associated with visual mismatch negativity and with the P2 wave in the event-related potential. Based on amplitude differences in the error-related negativity, Maurer et al. (2019) suggest that the brain generates error predictions that can dissociate relevant from irrelevant errors and that relevant errors lead to larger behavioral adjustments than irrelevant ones. Krala et al. (2019) show the involvement of high-level cortical regions in prediction, irrespective of the sensory modality involved. These articles highlight the ubiquitous comparison of sensory information with predicted outcome across low-level and high-level cortical areas. 

Predicting the consequences of one's own actions

Perhaps the most fundamental aspect of prediction is the need to take account of the visual consequences of self-motion—in particular, the image displacement on the retina that accompanies an eye movement. There has been a substantial body of work demonstrating remapping of visual receptive fields before a saccade (Duhamel et al., 1997; Melcher & Colby, 2008). Predictive remapping occurs not only in lateral intraparietal cortex but also in superior colliculus, frontal eye fields, and area V3. Evidence from neurophysiological studies indicates that predictive remapping is mediated by a corollary discharge signal originating in the superior colliculus and mediodorsal nucleus of the thalamus (Sommer & Wurtz, 2004, 2008). This predictive remapping might be part of a mechanism for visual stability that relates the pre- and postsaccadic images of a stimulus (Melcher & Colby, 2008; Cicchini et al., 2012; but see Maij et al., 2009). In the current issue, Murdison et al. (2019) demonstrate that predictive remapping not only takes account of vertical and horizontal displacements caused by saccadic eye movements but also extends to the torsional change that occurs during an oblique saccade to a new location. This suggests that observers have finely calibrated expectations resulting from their own movements and are able to learn the complex image remappings that result from self-movements. 

Recent studies have shown that predictive remapping shifts the focus of attention prior to saccade onset (Rolfs et al., 2011) and leads to lingering attention after the saccade (Golomb et al., 2008). A novel computational model accounts for both types of attentional updating and shows that these phenomena rely on the same neural circuit (Bergelt & Hamker, 2019). Predictive remapping allows constancy of visual direction, but other aspects of integrating information across saccades need to be considered, such as relating the appearance of visual stimuli in peripheral and central vision, given that the spatial filtering in the retina provides such disparate signals. We do not typically perceive an object as entirely different when we look at it. Valsecchi et al. (2018) used a novel method of manipulating images to show that peripheral stimuli appear to have sharper edges than would be expected on the basis of peripheral acuity losses. That is, they appear as they would when viewed foveally. Thus, humans appear to learn the relation between peripheral and central images in order to maintain constancy of appearance as well as constancy of direction. 

One interesting development revealed in this issue is the multisensory nature of predictive mechanisms that take account of self-motion. It has been demonstrated that when participants reach to grasp an object, somatosensory sensitivity is suppressed at movement-relevant locations shortly before and during the movement (Buckingham et al., 2010; Colino, Buckingham, Cheng, van Donkelaar, & Binsted, 2014). Voudouris, Broda, and Fiehler (2019) had participants reach to grasp an object whose distribution of mass was either predictable from its visual appearance or unpredictable. They found that somatosensory sensitivity was suppressed more when participants could predict the mass distribution from visual features. Thus, visual information can be used to generate somatosensory predictions for the control of reaching and grasping. 

Arikan et al. (2019) also demonstrate the multisensory nature of suppression during self-generated movements. They found reduced blood oxygenation level dependent (BOLD) activity in somatosensory, visual, and auditory regions during self-generated movements (vs. passive, externally generated movements). Moreover, they found stronger suppression for multisensory than unisensory movements and confirmed the role of the cerebellum in detecting delays between the action and its visual consequences. Such predictive mechanisms seem to be enhanced in older age when sensory input becomes increasingly noisy. Klever et al. (2019) found stronger suppression of somatosensory sensitivity in a group of older compared to younger participants in a visually guided reaching task. Interestingly, the strength of suppression correlated negatively with individual executive capacities, highlighting the interaction between sensory, motor, and predictive processes modulated by cognitive resources. 

Krugwasser et al. (2019) investigated how predictions of sensory action consequences are processed. Participants saw a virtual hand moving either in the same manner as their own or with a temporal, spatial, or anatomical alteration. They had to attribute an action to the self or an external source. There were similarities in the sense of agency across temporal, spatial, and anatomical manipulations, indicating joint processing of the sense of agency across different sensorimotor aspects. The review by Fiehler, Brenner, and Spering (2019) further discusses the implications of sensory attenuations of predicted movement consequences, how they are linked to task demands, and the processing of such signals. 

Combining predictions of eye, arm, and object movements

Prediction plays an important role in the oculomotor system. Both smooth pursuit and saccadic eye movements reveal predictions of the future visual stimulus in both laboratory and real-world contexts (see review in Kowler, 2014; Diaz et al., 2013). In this issue, Fiehler et al. (2019) review the role of prediction in goal-directed movements. This review covers classic paradigms and novel approaches investigating predictions in the planning of eye and hand movements and touches on many of the other aspects of visual prediction introduced here. 

Combining predictions of eye, arm, and object movements is challenging, as they differ in latencies and dynamics. For example, a predictive component is necessary for the smooth coordination of pursuit and saccades. Goettker et al. (2019) show that the pursuit and saccadic system share a common internal representation of the target movement and interact closely to improve tracking responses rapidly. Congruently, Kwon et al. (2019) show that the integration of position and motion information extends to the ocular following response. Rothwell et al. (in press) compared the role of different cues in driving anticipatory smooth pursuit and anticipatory ocular torsion and found that the predictive drive for these two types of eye movements might be partly decoupled. Delle Monache et al. (2019) demonstrate a role of an internal model of gravity for oculomotor control, tailored to the requirements of the visual context, for both predictive saccades and pursuit. Eye movements scaled with gravity accelerations, but only when observers tracked a ball in the context of a pictorial scene, not when faced with a uniform background. These results emphasize that predictive eye movements are tuned to realistic scene properties. 

Another important aspect of the predictive component of pursuit is that it allows better performance in intercepting moving targets (Spering et al., 2011; Fooken et al., 2016). Binaee and Diaz (2019) show that predictive saccades and hand movements share a representation that is presumably important for interceptions. In their experiment, participants intercepted virtual balls with a racquet while the ball was occluded for varying periods. When occlusion increased the spatiotemporal demands of the task, some participants demonstrated a strong correlation between saccade prediction accuracy and the accuracy of hand placement. Fooken and Spering (2019) used a go/no-go manual interception paradigm to show that predictive eye and hand movements jointly signal the upcoming decision about a future target. Similar predictive behavioral responses are shown in a manual tracking task, where different cursor speeds were related to different perceptual decisions (Zeljko et al., 2019). Mann et al. (in press) used a virtual tennis environment to show how a combination of tracking and predictive eye and head movements helps keep gaze close to the ball despite the ball bouncing as it approaches. These studies focus on tracking and intercepting moving objects, but prediction is similarly important in other real-world tasks such as driving (Macuga et al., 2019; Smith & Loschky, 2019; Wolfe et al., 2019). 

It should be noted that the mechanisms underlying predictive eye movements or body movements are not entirely clear. Zhao and Warren (2015) argue that interception depends on some visual parameter that is monitored continuously and controlled by the interceptive action, such as maintaining a constant bearing angle, and that the role of spatial memory or prediction is very modest. In cases where the action precedes the moving object in time, it can be argued that the prediction is based purely on recent sensory data and that a visual representation mediating prediction is unnecessary. The most likely resolution of this issue is that the domains where these different mechanisms operate depend on factors such as internal and external noise and the time available for the response (see de la Malla & Lopez-Moliner, 2015; Hayhoe, 2017). Aguilar-Lleyda et al. (2018) used a timing coincidence task and presented a Kalman filter model that samples and updates the position of a moving target by optimally combining spatial and temporal information. The authors suggest that a single mechanism based on position tracking can account for both spatial and temporal relations between physical target and response. Finally, insights into the relation between manual actions and prediction abilities can be gained from developmental studies. In this issue, Gehb and colleagues (2019) show that object experiences gathered by specific manual exploratory actions might facilitate infants’ predictive abilities when reaching and grasping. 

Conclusions

The wide diversity of articles in this issue provides a broad overview of the role and mechanisms of prediction in the visual system. This includes predicting ongoing motion as well as building visual priors about the world with which to anticipate future events. In all cases, predictive mechanisms are tightly interwoven with incoming sensory data. The articles in this issue show that prediction is a fundamental and essential aspect of visual perception, as well as of the actions that are guided by perception. 

Acknowledgments

Commercial relationships: None. 

Corresponding author: Mary Hayhoe. 

E-mail: [e-mail]. 

Address: Center for Perceptual Systems, University of Texas at Austin, Austin, TX, USA.