Transsaccadic integration and perceptual continuity

Abstract

Our perception of the visual world seems continuous despite the fact that visual information is sampled in discrete fixations interrupted by saccadic eye movements. Two new behavioral studies show that perceptual continuity may be partly achieved by combining feature information of saccade target objects across saccades in a close-to-statistically optimal fashion.

In contrast to modern digital image sensors that sample the visual environment in full detail, spatial resolution of the human visual system is unevenly distributed. On the retina, for instance, spatial resolution drops steeply toward the periphery and objects can be processed in detail only within an astonishingly small region of about 2° around the center of gaze, corresponding to foveal vision (Anton-Erxleben & Carrasco, 2013). To see in detail what we are observing, we thus need to scan the visual scene with fast ballistic movements of the eyes (i.e., saccades) followed by intervals in which gaze is held almost stationary (i.e., fixations). This saccade and fixate strategy enables successively high resolution information about different objects in the scene by changing their retinal position. More precisely, objects that are targeted by an upcoming saccade are only coarsely represented in the periphery, whereas they are represented with high acuity when foveated (Figure 1a). The benefit of inhomogeneous mobile visual sensors, like the human eyes, is that they save a lot of neural and cognitive resources that would otherwise be needed to process the whole visual field in high resolution detail in a single glance. However, at the same time this strategy poses a number of challenges to the visual system. Among these challenges is the need to keep track of object locations and object identities across saccadic eye movements (Cavanagh, Hunt, Afraz, & Rolfs, 2010; Rolfs, 2015). More broadly, one could ask how visual information gathered at discrete sampling episodes or snapshots is combined to assure perceptual continuity. 

Figure 1

 

The problem of transsaccadic perception. (a) With each saccade, targeted objects change their retinal position (periphery to fovea) as well as their spatial resolution (low-resolution to high-resolution). (b) To test how the visual system deals with these two qualitatively different sources of information about one object, Ganmor et al. (2015) and Wolf and Schütz (2015) had subjects make saccades to a tilted target stimulus and measured the perception of orientation separately for information presented to the periphery only (depicted in blue), to the fovea only (depicted in orange), or to the periphery and fovea (depicted in green). They found that peripheral and foveal information was integrated in a close to statistically optimal fashion, that is, as a weighted sum, with weights depending on the relative reliability of the different information sources.

Several theoretical positions have emerged to address this classical problem. Early accounts, for example, proposed that detailed information from consecutive fixations is fused in a transsaccadic buffer into a single percept comparable to superimposing different images of one scene in a world-centered coordinate system (McConkie & Rayner, 1976). These accounts were, however, challenged by observers' inability to explicitly combine simple patterns across saccades (e.g., O'Regan & Lévy-Schoen, 1983). Together with the finding that large transsaccadic changes of the visual scene often go unnoticed by observers (Grimes, 1996), it was later suggested that only relatively coarse abstracted information is retained across saccades (if any; Irwin, 1996). However, over the past decade there are increasing signs that this latter evaluation of a rather poor and abstract transsaccadic memory was probably too pessimistic. For example, several recent studies have shown that, at least for the saccade target object or objects in its vicinity, relatively precise information about the object's location (Deubel, Schneider, & Bridgeman, 1996), spatial frequency (Herwig & Schneider, 2014), color (Wittenberg, Bremmer, & Wachtler, 2008) and visual form (Demeyer, De Graef, Wagemans, & Verfaillie, 2009; Herwig, Weiß, & Schneider, 2015) is retained across the saccade. But what happens to this presaccadic peripheral information after the saccade when more accurate foveal information is available? Is it replaced by or combined with the new foveal information? Two studies in this issue of Journal of Vision (Ganmor, Landy, & Simoncelli, 2015; Wolf & Schütz, 2015) provide new insights into this classical problem in visual neuroscience. 

Ganmor and colleagues as well as Wolf and Schütz independently report a series of behavioral experiments to test the perceptual consequences of retaining information across saccades. Both groups instruct their participants to foveate a peripheral grating target stimulus by executing a saccade toward it. After their saccade participants are asked to judge whether the target stimulus is tilted to the left or right. Typically, foveal information dominates peripheral information for most visual features including orientation (Strasburger, Rentschler, & Jüttner, 2011). Thus, a key feature of the experimental approach is that the stimulus changes its contrast during the saccade to counteract foveal superiority. It is well established that contrast directly influences the reliability of orientation information. In keeping with previous findings, discrimination performance in control trials, where orientation information is presented either to the periphery or to the fovea alone, is less reliable the lower the contrast (Wolf & Schütz, 2015) or can be directly kept comparable by adjusting the contrast adaptively (Ganmor et al., 2015). However, the new finding of both studies is that discrimination performance is enhanced and more reliable in experimental trials where orientation information is presented both to the periphery and to the fovea (i.e., reliability effect, hereafter). In addition to improvements in reliability, orientation perception can be also biased in the direction of the presaccadic information (i.e., biasing effect, hereafter). This is nicely demonstrated in further experiments of both groups, in which there is a transsaccadic orientation change in addition to the contrast change. For example, when the target stimulus is slightly tilted to the left before the saccade but slightly tilted to the right after the saccade, the orientation of the target stimulus is perceived to fall between the orientation of the presaccadic peripheral and postsaccadic foveal stimulus. Together these findings suggest that peripheral orientation information about the saccade target is not simply replaced by foveal information after the saccade. Instead, peripheral and foveal information is combined to improve visual perception (Figure 1b). 

To investigate this information combination quantitatively, both groups measure the reliabilities (defined as reciprocal variance) associated with orientation judgements separately for information presented either to the periphery or to the fovea alone. These measurements are then used to construct a maximum likelihood integrator to get a benchmark of statistically optimal integration (Ernst & Bülthoff, 2004). According to statistically optimal integration, peripheral and foveal information should be combined as a weighted sum, with weights depending on the relative reliability of the different information sources. Importantly, the maximum likelihood integrator almost perfectly predicts the observed transsaccadic discrimination performance, both for the observed amount of benefit in terms of reliability, as well as the extent in which orientation perception is biased in the direction of the presaccadic information. Please note that these additional analyses also show some indications for an overweighting of foveal information, which is more pronounced in the data of Ganmor and colleagues (2015) than in the data of Wolf and Schütz (2015). 

Both studies are complemented by additional experiments. For example, Ganmor and colleagues show that peripheral and foveal integration takes place only when subjects perform saccades, whereas there is no evidence for integration when the target is artificially moved during fixation. Moreover, Wolf and Schütz use a reverse correlation analysis to show that effects of peripheral information on the transsaccadic percept start to decline between 100 and 50 ms before saccade onset, whereas foveal information affects the transsaccadic percept immediately after saccade offset. Summing up, the results thus suggest that peripheral and foveal information is integrated by weighting the information according to the relative reliability of the fovea and periphery with a slight bias to overweighting foveal information. 

As stated previously, the question as to how information is integrated across saccadic eye movements is a classical problem in visual neuroscience and has been addressed several times before. A large part of these studies put emphasis on the integration of location information across saccadic eye movements (e.g., Niemeier, Crawford, & Tweed, 2003; Vaziri, Diedrichsen, & Shadmehr, 2006). However, there is now convincing evidence for visual features other than location being retained across saccades including spatial frequency, color, and shape (for the controversy on motion, see Melcher & Morrone, 2003; Morris et al., 2010). Most of these previous studies on visual features have investigated biasing effects as a marker of transsaccadic integration and did not directly test for underlying weighting functions (see Table 1).¹ By additionally investigating reliability effects and by directly comparing integration to the benchmark of maximum-likelihood, the studies by Ganmor and colleagues (2015) and Wolf and Schütz (2015) thus extend our knowledge on transsaccadic integration. Yet, these new and exciting results also leave open some questions, two of which will be briefly discussed in the following paragraphs. 

Table 1

 

Overview of recent studies indicating transsaccadic integration. Notes: DT = detection target.

 

One of these open questions is, under what conditions does transsaccadic integration occur and when does it not? While this is certainly a question for future research, there are already studies indicating that transsaccadic integration might depend on some kind of object continuity test. For example, changing the location of a saccade target across the saccade is typically very hard to detect (Bridgeman, Hendry, & Stark, 1975), which is in line with the idea that integration hampers the access to separate pre- and postsaccadic location information. However, change detection can be greatly improved by inserting a temporal postsaccadic blank (Deubel, Schneider, & Bridgeman, 1996), or by changing the polarity of the target across the saccade (Tas, Moore, & Hollingworth, 2012). In addition to benefits in location change detection, a postsaccadic blank as well as a polarity change strongly impairs postsaccadic letter identification (Poth, Herwig, & Schneider, 2015). The latter two findings suggest that, if there are indications for object discontinuity, pre- and postsaccadic information about the object's location can be kept separate resulting in better change detection performance but also in attentional competition between pre- and postsaccadic information (Schneider, 2013). Importantly, at the same time, indications for object discontinuity might prevent the integration of pre- and postsaccadic information (for a reduction of transsaccadic integration by postsaccadic blanking, see Demeyer, De Graef, Wagemans, & Verfaillie, 2010). A crucial next step will be to determine in detail how the visual system detects object discontinuity across saccades. Current theoretical ideas addressing this question range from a comparison of position only (e.g., Flombaum, Scholl, & Santos, 2009), or position and surface features (Hollingworth, Richard, & Luck, 2008) to a comparison of the predicted and actual activation in priority map regions coding top-down and bottom-up factors of attentional control (Schneider, 2013). 

The second open question is what exactly is integrated across saccades? As stated previously, transsaccadic integration is often discussed as a mechanism for maintaining the impression of perceptual continuity. The underlying reasoning is that integrating postsaccadic foveal information with retained presaccadic information conceals the discontinuous nature of visual sensory input after a saccade has been executed (see Figure 2, lower part). In essence, this idea can be best captured as a retrospective processing mechanism taking effect only after a particular event has occurred. However, this is probably only one part of the explanation. Another important factor to create smooth and continuous perception probably already starts before a saccade will be executed and can thus be best captured under the term predictive processing (Figure 2, upper part). In particular, this second mechanism requires predicting the saccades perceptual consequences (i.e., the foveation of the saccade target object), which is assumed to be based on transsaccadic associations of peripheral and foveal input (Herwig & Schneider, 2014, Herwig et al., 2015; Weiß, Schneider, & Herwig, 2014; and see Herwig, 2015, for a broader theoretical framing). Such a predictive processing mechanism is supported by studies showing that peripheral perception is biased toward the predicted foveal input. More precisely, participants in these studies first underwent a 30-min acquisition phase where, unnoticed by participants, one object systematically changed its spatial frequency (Herwig & Schneider, 2014; or shape, Herwig et al., 2015) during the saccade. In the following test phase, the perception of frequency (or shape, respectively) of peripheral saccade targets was biased toward the previously associated new foveal input. These findings suggest another transsaccadic integration mechanism taking place before a saccade is executed—an integration of the actual peripheral input with the predicted foveal input. 

Figure 2

 

The jigsaw of transsaccadic integration. Transsaccadic integration of peripheral and foveal information happens not only after a saccade (lower part) as investigated by Ganmor et al. (2015) and Wolf and Schütz (2015) but also before a saccade (upper part). In particular, presaccadic peripheral input is used to predict foveal input based on previously acquired transsaccadic associations of peripheral and foveal input (Herwig & Schneider, 2014). Solving the jigsaw of transsaccadic integration requires considering how all the different pieces are weighted and put together to assure perceptual continuity.

Thus, solving the jigsaw of transsaccadic integration probably requires considering all the different pieces contributing to perceptual continuity (Figure 2). A full explanation of transsaccadic integration will need to outline under what conditions and in what way the contributing pieces of actual peripheral and predicted foveal as well as memorized peripheral and actual foveal information are weighted and put together. Nevertheless, the experimental paradigms reported by Ganmor et al. (2015) and Wolf and Schütz (2015) in this issue of Journal of Vision offer a way to study transsaccadic integration in detail, opening up many new avenues of investigation. 

Acknowledgments

This work was supported by a grant of the German Research Council (Deutsche Forschungsgemeinschaft; DFG) Grant He6388/1-1 to Arvid Herwig and Werner X. Schneider. I thank Werner X. Schneider for comments on an earlier version of the manuscript. 

Commercial relationships: none. 

Corresponding author: Arvid Herwig. 

Email: [email protected]. 

Address: Department of Psychology, Bielefeld University, Bielefeld, Germany. 

Footnotes

¹  A direct test of statistically optimal integration requires measurement of reliabilities for the pre-, post-, and transsaccadic display separately and to compare transsaccadic perception against a benchmark performance.