Trans-saccadic integration of peripheral and foveal feature information is close to optimal

Abstract

Due to the inhomogenous visual representation across the visual field, humans use peripheral vision to select objects of interest and foveate them by saccadic eye movements for further scrutiny. Thus, there is usually peripheral information available before and foveal information after a saccade. In this study we investigated the integration of information across saccades. We measured reliabilities—i.e., the inverse of variance—separately in a presaccadic peripheral and a postsaccadic foveal orientation-discrimination task. From this, we predicted trans-saccadic performance and compared it to observed values. We show that the integration of incongruent peripheral and foveal information is biased according to their relative reliabilities and that the reliability of the trans-saccadic information equals the sum of the peripheral and foveal reliabilities. Both results are consistent with and indistinguishable from statistically optimal integration according to the maximum-likelihood principle. Additionally, we tracked the gathering of information around the time of the saccade with high temporal precision by using a reverse correlation method. Information gathering starts to decline between 100 and 50 ms before saccade onset and recovers immediately after saccade offset. Altogether, these findings show that the human visual system can effectively use peripheral and foveal information about object features and that visual perception does not simply correspond to disconnected snapshots during each fixation.

Introduction

The human visual system comprises a large field of view and high spatial resolution with an inhomogeneous representation across the visual field: high-acuity vision in the fovea and low-resolution vision in the periphery. To exploit this architecture, objects are selected based on peripheral information and then foveated with saccadic eye movements for further inspection in natural tasks (for reviews, see Schütz, Braun, & Gegenfurtner, 2011; Tatler, Hayhoe, Land, & Ballard, 2011). As a result, there are usually two sources of visual information about an object: peripheral information before the saccade and foveal information after the saccade. The question now is, what happens to peripheral object information acquired before foveation? Theoretically, in the extreme case peripheral information is only processed in order to select future fixation positions, then neglected for object perception. In this case, vision would start anew with each fixation. In the other extreme, peripheral information is maintained across the saccade and integrated with foveal information in a statistically optimal manner. This is the long-standing question of trans-saccadic perception and integration (for reviews, see Irwin, 1996; De Graef & Verfaillie, 2002; Melcher & Colby, 2008; Mathôt & Theeuwes, 2011; Higgins & Rayner, 2015). 

Trans-saccadic integration is far from trivial because of three reasons: First, peripheral information arrives earlier than foveal information and would have to be stored in memory. Depending on the level and type of memory, low-level features or an abstract representation (Irwin, 1991) might be maintained. Second, since early visual areas are retinotopically organized (Engel, Glover, & Wandell, 1997), saccades shift the receptive fields across the scene, and information would have to be transferred from peripheral neurons to foveal neurons. A potential mechanism could be predictive remapping, which leads to presaccadic activation from a stimulus that will be in the receptive field only after the saccade (Duhamel, Colby, & Goldberg, 1992). Thus, neurons with receptive fields at the fovea could already gather information from the saccade target before the saccade is actually executed. Third, visual processing differs in the peripheral and foveal visual field, which leads to differences not only in visual sensitivity (Rovamo, Virsu, & Näsänen, 1978; Makela, Whitaker, & Rovamo, 1993) but also in the appearance of visual stimuli between the periphery and the fovea (Davis, Yager, & Jones, 1987; McKeefry, Murray, & Parry, 2007; Valsecchi, Toscani, & Gegenfurtner, 2013). As a result, peripheral and foveal information need to be calibrated before they can be integrated. 

At present there is contradictory evidence whether peripheral and foveal information are integrated across saccades. On one hand, several studies have found little evidence for integration (Bridgeman & Mayer, 1983; Rayner & Pollatsek, 1983). For instance, when different spatial patterns are shown at the same location before and after a saccade, subjects have problems fusing the two stimuli (Irwin, 1991). Several studies have also shown poor performance in the detection of trans-saccadic changes (Bridgeman, Hendry, & Stark, 1975; Henderson & Hollingworth, 1999), which often has been interpreted as evidence that peripheral information is not preserved across saccades and thus not integrated with foveal information. However, poor change detection could also be interpreted as a sign of integration, since a strong fusion of peripheral and foveal signals might restrict access to the separate signals (Hillis, Ernst, Banks, & Landy, 2002) and thereby raise the threshold to detect changes during the saccade. Along these lines, the locations of briefly flashed objects are integrated at the time of saccades (Cicchini, Binda, Burr, & Morrone, 2013). 

On the other hand, some studies have shown evidence for integration. Peripheral preview can facilitate the recognition of objects (Pollatsek, Rayner, & Collins, 1984). When the contrast of peripheral and foveal stimuli is varied independently, this preview effect turns out to be larger for strong peripheral and weak foveal information (Demeyer, De Graef, Wagemans, & Verfaillie, 2009). Similarly, the color appearance of postsaccadic stimuli was biased in the direction of presaccadic information (Wittenberg, Bremmer, & Wachtler, 2008), and adding color noise to a presaccadic display biased trans-saccadic color perception towards postsaccadic information (Oostwoud Wijdenes, Marshall, & Bays, 2015). When the orientations of the peripheral and foveal object were varied independently, the distribution of perceived orientations was unimodal and located in between the peripheral and foveal stimulus orientations (Demeyer et al., 2010). While these studies suggest the involvement of peripheral information, they did not measure reliabilities and were not able to quantify the amount of peripheral information used. Thus, they leave open the question how the two sources of information are exactly integrated. 

The integration of several perceptual signals is often studied within a Bayesian framework (for reviews, see Ernst & Bülthoff, 2004; Kersten, Mamassian, & Yuille, 2004). According to maximum-likelihood estimation, independent signals should be weighted by their relative reliabilities to achieve a statistically optimal integration. Previous research has found evidence for optimal integration within modalities (Jacobs, 1999; Hillis, Watt, Landy, & Banks, 2004) and across modalities (Ernst & Banks, 2002; Alais & Burr, 2004), as well as for dynamic changes over time (Wolpert, Ghahramani, & Jordan, 1995). Optimal integration of pre- and postsaccadic information has been reported for the perceived location of objects (Niemeier, Crawford, & Tweed, 2003) as well as for the location of reach targets (Vaziri, Diedrichsen, & Shadmehr, 2006). Object location, however is a special case, because to compute object location in external space, the retinal position of the object and an extraretinal signal about gaze direction have to be combined (Henriques, Klier, Smith, Lowy, & Crawford, 1998), even in the absence of eye movements. In addition, the execution of a saccade changes both of these components. This is different for other features—for example, orientation or color—because they can be computed from retinal signals alone. In this case, extraretinal signals would be necessary only for trans-saccadic integration, not for the computation of the features per se. As a result, it might be that these features are treated differently in trans-saccadic perception. 

In this study we investigated whether peripheral and foveal feature information are integrated across saccades in a statistically optimal way. Therefore, we measured reliabilities of the presaccadic peripheral, the postsaccadic foveal, and the trans-saccadic percepts. Reliabilities are given by the inverse of the variance. The maximum-likelihood estimation makes two predictions with respect to the integrated signal. First, if peripheral and foveal information are incongruent, the trans-saccadic percept should be biased towards the more reliable information. Second, the reliability of the trans-saccadic percept should be higher than the reliability of peripheral and foveal information alone. We tested these predictions in Experiments 1 and 2. In Experiment 3 we tracked the weighting of peripheral and foveal information with high temporal resolution around the time of the saccade. In all of these experiments, we tested the natural scenario in which a peripheral object is foveated by a saccade; hence, presaccadic information was always peripheral, and postsaccadic information always foveal. 

Experiment 1: Biasing perception through presaccadic information

In this experiment we tested whether presaccadic peripheral information is maintained across the saccade and biases object perception in a way predicted by maximum-likelihood integration when there is a conflict between peripheral and foveal information. In order to do this, we measured peripheral and foveal reliabilities separately using two-alternative forced-choice orientation judgments. Based on these measurements, we predicted optimal peripheral weights. If both cues are integrated optimally, then peripheral and foveal information should be weighted according to their relative reliability (Equation 1). As contrast information is not maintained across the saccade (Melcher, 2005; Demeyer et al., 2009), we manipulated the foveal target contrast so that its reliability was either superior to the peripheral target, approximately equal, or inferior. We additionally measured perceptual responses with both peripheral and foveal information present. In this case, we introduced a conflict in order to measure the relative weight of peripheral and foveal information at different foveal reliabilities and then compared observed against predicted weights. 

Methods

Participants

Fourteen observers (12 women, two men, mean age = 24 years, range = 20–29) who were unaware of our goals participated in this experiment. Observers were students of the Justus Liebig University Giessen and were paid for participation. Experiments were in accordance with the principles of the Declaration of Helsinki and approved by the local ethics committee LEK FB06 at the university (proposal number 2013-0020). All observers gave informed consent and had normal or corrected-to-normal vision. 

Stimuli and setup

Stimuli were plaid stimuli—i.e., a vertical and horizontal grating, both with a spatial frequency of 2 c/° overlaid by a Gaussian window with a standard deviation of 0.4° of visual angle. The overall contrast remained constant, as both individual contrasts added up to a fixed value of 0.4. Thus, a vertical contrast of 0.1 means that the horizontal contrast was 0.3. Here, we report the contrast of the vertical component only. Whereas the horizontal component was perfectly aligned to the cardinal axis, the orientation of the vertical component varied throughout the experiment. To measure peripheral and foveal performance separately in peripheral and foveal trials, stimuli were replaced by a Gaussian blob either after or before the saccade. The blob had the same size as the plaid and a peak contrast of 0.4. 

Stimuli were displayed using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) in MATLAB on a VIEWPixx monitor at a 1920 × 1080 pixel resolution and a 120-Hz refresh rate. The monitor had a size of 51.5 × 29 cm and was viewed at a distance of 48.5 cm. This resulted in 32 pixels/°. The luminance of black, gray, and white pixels was 0.23, 121, and 230 cd/m², respectively, and the output of the monitor was linearized. Eye movements of the right eye were recorded using the Eyelink Toolbox (Cornelissen, Peters, & Palmer, 2002) and an EyeLink 1000 (SR Research Ltd., Ontario, Canada) with a sampling rate of 1000 Hz. Participant responses were recorded via a standard keyboard. 

Procedure

In all trials, participants had to indicate whether the vertical component of a plaid stimulus was tilted clockwise (cw) or counterclockwise (ccw). A fixation cross in the screen center prompted participants to start the trial by pushing the space bar. After a random time between 0.75 and 1.5 s, the target appeared 15° left or right from screen center. The fixation cross was removed after an additional 200 ms (overlap paradigm). Targets switched as soon as the EyeLink detected that the eye exceeded a distance of 1.5° with respect to the screen center. This guaranteed that the target would be switched during the saccade, when vision is suppressed (for a review, see Ibbotson & Krekelberg, 2011). At the end of each trial, a bar appeared at the target location in one of two random orientations (cw or ccw). Participants could alternate between these two orientations pressing the plus key and selected their desired response with the Enter key. Afterwards, participants received visual feedback about their response. 

In integration trials (Figure 1, left column), the foveal target was displayed for as long as the peripheral target in each trial. For instance, if it took 200 ms from target onset to target switch, the foveal target was presented for 200 ms as well. Thus the presentation duration depended on the eye-movement latency within each trial. This guaranteed that observers were provided with roughly the same amount of peripheral and foveal information within each trial. To guarantee that observers also had approximately the same viewing time across trials, they received feedback when the saccadic reaction time was too fast or too slow (target switch below 157.5 ms or above 257.5 ms). In these cases, a high or a low beeping sound was played and no visual feedback about the performance was shown on the screen. Observers were told to keep their eye-movement latency within the given time window, as trials outside this time window were not considered for analysis. 

Figure 1

 

Trial procedure for peripheral, foveal, and integration trials. In all trials, a saccade target appeared at 15° eccentricity and had to be foveated. Afterward, observers had to decide whether the vertical component of the plaid was tilted cw or ccw. In peripheral trials (right column), plaid stimuli were visible only until saccade onset and were then replaced by a Gaussian blob during saccade execution. In foveal trials (middle column), the Gaussian blob was visible until saccade onset and was then replaced by the plaid stimulus. Foveal targets (single and integration trials) were displayed in one of three possible contrasts (fovea high, mid, or low), whereas the peripheral target was always displayed at high contrast. In integration trials (left column), the tilt of the peripheral target was shifted by 2.5° either cw or ccw (here the shift is exaggerated for demonstration purposes). Stimuli are not drawn to scale.

To assure the same viewing time in single trials, peripheral and foveal targets were displayed for an individually fixed presentation duration. This duration was derived from the median individual latency in integration trials. Peripherally appearing plaid stimuli switched to a Gaussian blob after this time or if the eye exceeded the critical distance of 1.5°. In these cases, observers never had foveal information about the target (Figure 1, right column). Foveal targets switched from a Gaussian blob to a plaid stimulus during the saccade and were then displayed for the given presentation duration (Figure 1, middle column). 

Design

All participants started with the integration trials. In integration trials, the foveal contrast of the vertical component was either high (0.3), mid (0.1), or low (0.05), while the peripheral contrast was always high (0.3). Foveal orientations varied from −4° to +4° in eight steps (±0.5°, ±1°, ±2°, ±4°), with positive values indicating clockwise orientations. The orientation of the peripheral target was shifted either −2.5° or +2.5° with respect to the foveal target. We measured a psychometric function for every combination of foveal contrast and peripheral orientation shift. This resulted in 480 trials (3 foveal contrasts × 2 peripheral shift directions × 8 foveal orientations × 10 repetitions). 

Peripheral and foveal trials were measured interleaved in the same session. For every observer, we measured four psychometric functions identical with the contrast levels in integration trials (three foveal contrasts, one peripheral contrast). Psychometric functions were sampled with 10 data points based on 16 observations each (±0.5°, ±1°, ±2°, ±4°, ±8°). This resulted in 640 trials (4 psychometric functions × 10 orientations × 16 repetitions). 

Data and eye-movement analysis

Saccade onsets were detected offline using the EyeLink 1000 algorithm. Saccade latencies were defined as the first saccadic frame with respect to target onset. To keep peripheral and foveal viewing time constant, integration trials in which the saccade was not detected in the given time window (157.5 to 257.5 ms) were excluded from further analysis. In total, we rejected 20% of trials (on average: 97 out of 480 trials). 

Perceptual choices were converted into proportion cw responses for every stimulus orientation value, and a cumulative Gaussian was fitted to the data using psignifit 4.0 (Schütt, Harmeling, Macke, & Wichmann, 2015). Just-noticeable differences (JNDs) were defined as the standard deviation of the underlying Gaussian. The optimal peripheral weight (for a review, see Ernst & Bülthoff, 2004) is then given by  

 

with  

 

 

Results were compared using one-way repeated-measures ANOVAs and post hoc t-tests. If not noted otherwise, all t-tests were two-tailed and p-values were compared against a Bonferroni-corrected alpha level of 0.05. 

Results

To determine the extent to which the brain maintains peripheral information across saccades, we measured reliabilities for foveal and peripheral vision separately, as well as the bias of the trans-saccadic percept when peripheral and foveal information were set in conflict. The bias which presaccadic peripheral vision optimally exerts on the percept can be modeled using maximum-likelihood integration (Ernst & Banks, 2002). 

For each observer in each condition, we varied the vertical orientation and fitted a cumulative Gaussian to the proportion of cw responses over the different orientations. Figure 2A shows psychometric functions for one observer for the periphery and the three different foveal contrast conditions. Our manipulation of foveal reliability was successful, as JNDs increased with decreasing foveal contrast from 1.1° (fovea high, SD = 0.28°) to 2.8° (fovea mid, SD = 0.85°) to 7.4° (fovea low, SD = 2.14°), F(2, 26) = 105.50, p < 0.001. JNDs in the fovea mid condition were significantly elevated compared to the fovea high condition, t(13) = 9.89, p < 0.001, and those in the fovea low condition were higher than in the fovea mid condition, t(13) = 8.75, p < 0.001. For peripheral vision, JNDs were 2.1° on average (SD = 0.87°). Based on these separately measured JNDs, we predicted optimal peripheral weights as in Equation 1. 

Figure 2

 

Manipulation of foveal reliability and gaze-contingent paradigm. (A) Proportion cw responses for different orientations with fitted psychometric functions in peripheral and foveal trials for one observer. Positive orientations denote a cw tilt. Foveal discrimination performance increases with increasing contrast (light gray to black). In peripheral trials (red), contrast was identical to the high-foveal-contrast condition. (B) Horizontal eye position over time for one trial. The target is switched during saccade execution, allowing comparable peripheral and foveal viewing time.

To test these model predictions, we measured peripheral weights for the trans-saccadic percept. In integration trials, targets were switched during saccades (Figure 2B). Foveal contrast either remained high or was switched to mid or low. Additionally, peripheral information was misaligned by 2.5° in either direction. Consider that peripheral and foveal information are integrated: If peripheral information is tilted ccw, it requires a cw foveal signal so that the trans-saccadic percept is perceived as being aligned with the cardinal vertical axis. As we computed functions over foveal contrasts, this corresponds to a shift of psychometric functions towards more clockwise values (Figure 3A, orange curve). The influence which is exerted by peripheral information can be measured by the difference between curves (Figure 3A, orange and blue curve) at the orientations which were judged equally often as cw and ccw (the point of subjective equality, PSE). If peripheral information has no influence on the percept (no integration), PSE values for the different peripheral shifts should be zero. If, however, perception is solely driven by peripheral information, PSE values should be −2.5° (cw shift) and +2.5° (ccw shift). Thus the difference of these two PSE values relative to the peripheral shift corresponds to the observed peripheral weight (Figure 3B). 

Figure 3

 

Experiment 1, calculation of peripheral bias. (A) Proportion cw responses and psychometric functions in integration trials (fovea high condition). One psychometric function is fitted for each peripheral orientation shift. The observed peripheral bias is calculated as the difference in PSE values between cw (blue curve) and ccw shift (orange curve) relative to manipulation. (B) PSE values for the two peripheral shifts and the three foveal contrast conditions relative to the peripheral-shift manipulation. Lines indicate regressions for each foveal contrast. Slopes (× −1) correspond to the observed peripheral weight.

As predicted by the model, the influence of peripheral information on the trans-saccadic percept increased with decreasing foveal reliability from 0.24 (fovea high, SD = 0.13) to 0.55 (fovea mid, SD = 0.23) to 0.83 (fovea low, SD = 0.20), F(2, 26) = 48.95, p < 0.001—low versus mid: t(13) = 3.98, p = 0.002; mid versus high: t(13) = 5.4, p < 0.001. A peripheral weight of 0 would indicate that peripheral information was ignored, whereas a weight of 1 would indicate that foveal information was ignored. All peripheral weights were statistically different from 0—fovea high: t(13) = 7.08, p < 0.001; mid: t(13) = 8.97, p < 0.001; low: t(13) = 15.75, p < 0.001—and from 1—fovea high: t(13) = 22.45, p < 0.001; mid: t(13) = 7.3, p < 0.001; low: t(13) = 3.16, p = 0.008—indicating that in all conditions, peripheral and foveal information were both considered and neither of them was ignored. 

We predicted peripheral weights based on individual reliabilities from foveal and peripheral trials and compared observed against predicted weights (Figure 4). Observed weights coincided with predicted weights and did not differ significantly in any of the conditions (all p values > 0.1). For 11 out of 14 observers, peripheral weights increased consistently with decreasing foveal contrast (Supplementary Figure S1). A possibility to quantify the agreement between data and model predictions over all foveal contrasts is to compute the best fitting line through the origin, based on the aggregated data. If model predictions and data perfectly match, this would result in a slope of 1. Here, this line had a slope of b = 0.89. Such a slope below 1 suggests that the foveal reliability was slightly overestimated relative to the peripheral reliability for the low- and the mid-foveal contrast condition. 

Figure 4

 

Experiment 1, observed against predicted peripheral weights for the three foveal contrast conditions. All error bars denote 95% confidence intervals. Diagonal error bars mark the error of the differences between observed and predicted values and have to be compared to the identity line. Observed weights on the identity line are statistically optimal. Performance below the line indicates an overweighting of foveal information, whereas performance above the identity line indicates an overweighting of the periphery. The slope of the best fitting line through the data and the origin is b = 0.89.

Discussion

Here we investigated the extent to which peripheral object information acquired before the saccade is maintained after foveation and biases the trans-saccadic percept. Peripheral information clearly influenced the trans-saccadic percept. We found that the weight that was assigned to peripheral information increased with a higher relative reliability of peripheral vision. Under normal viewing conditions, when no contrast manipulation was introduced between peripheral and foveal vision, peripheral weights were optimal. When foveal contrast and thus the reliability of foveal vision was reduced, peripheral weights were also not different from optimal but showed a slight overweighting of the fovea. In everyday life, foveal vision has a far better resolution than peripheral vision. This tendency towards suboptimality for lowered foveal contrasts thus might reflect an internal prior of the visual system that foveal vision is superior to peripheral vision. 

Showing that presaccadic information biases trans-saccadic perception of object features and that the strength of this bias is modulated by the relative reliability of the pre- and postsaccadic signal (Demeyer et al., 2009; Oostwoud Wijdenes et al., 2015) can only be seen as a first hint for (optimal) integration. Neither of the aforementioned studies measured the reliabilities for the pre- and postsaccadic displays separately, and they were thus not able to compare the bias on the trans-saccadic percept to the benchmark of maximum-likelihood integration and draw conclusions regarding the efficiency of the peripheral preview effect. But even after showing that the bias of presaccadic peripheral vision on the trans-saccadic percept behaves in a manner that is statistically close to optimal, we cannot yet conclude that information is actually integrated across saccades. Optimal weights only indicate that the information is used in an optimal manner. Any bias, optimal or not, can also be explained with probabilistic cue switching (e.g., Nardini, Jones, Bedford, & Braddick, 2008; Serwe, Drewing, & Trommershäuser, 2009), where people stochastically alternate between the two cues without integrating them. To overcome this limitation and rule out the possibility of cue switching, we performed a second experiment, where the precision of a congruent trans-saccadic signal was measured and compared to the optimal prediction. 

Experiment 2: Reliability of the trans-saccadic percept

To show that peripheral and foveal information are actually integrated across saccades and that the bias shown in Experiment 1 is not due to cue switching, we measured the foveal and peripheral reliabilities separately and predicted the JND of the trans-saccadic percept that would result from statistically optimal cue integration. As the congruency between two cues has been shown to be essential for integration (Atkins, Fiser, & Jacobs, 2001; Parise, Spence, & Ernst, 2012), conflicts between cues could disrupt optimal integration. We therefore decided to assess the predictions of the JND of the trans-saccadic percept in a separate experiment, where peripheral and foveal orientations were always identical. 

Methods

For the reliability experiment, we measured 14 observers (11 women, three men, mean age = 23 years, range = 19–29) who were unaware of the aims of the experiment, three of whom had already taken part in Experiment 1. They all gave informed consent, were paid for participation, and had normal vision. 

Stimuli, setup, and trial procedure were identical with the first experiment, with the exception that the order of single and integration trials was now counterbalanced across participants. Therefore, participants were given auditory feedback in all trials whenever their eye movement in reaction to the target was not in a given time window (target switch below 137.5 ms or above 297.5 ms). Beforehand, all observers completed a training session that consisted of 180 integration trials in a separate session. 

In integration trials, the orientation of the vertical component varied in 10 steps from −4° to +4° (±0.25°, ±0.5°, ±1°, ±2°, ±4°). No peripheral shift was introduced this time, and the peripheral and foveal targets were thus always in line with respect to their orientation. Again, we measured performance for the same three foveal target contrasts (0.3, 0.1, 0.05) while the peripheral contrast was constant (0.3). This resulted in 480 trials (3 foveal contrasts × 10 orientations × 16 repetitions). In peripheral and foveal trials, we included orientation values of −8° and +8°. The single-trial session thus consisted of 768 trials (4 psychometric functions × 12 orientations × 16 repetitions). 

One observer was not susceptible to our contrast manipulation in single trials. JNDs for the three different foveal contrast conditions were all above 6°, with the worst performance in the high-foveal-contrast condition (7.4°). We did not include data from this participant for further analysis. Also, we excluded trials in which the saccade did not take place in the given time window. This applied to 7.8% of integration trials (on average: 37 out of 480 trials removed) and 8.9% of single trials (on average: 69 out of 768 trials removed). 

If two independent cues are integrated, then the reliability of the integrated percept (rel_int) is the sum of the two individual reliabilities (for a review, see Ernst & Bülthoff, 2004):  

 

 

The peripheral (rel_per) and foveal (rel_fov) reliabilities are given by Equation 2. The predicted JND of the integrated percept can then be derived by adjusting Equation 2:  

 

 

Results

Identical to Experiment 1, we measured the peripheral reliability as well as the foveal reliability for three different contrasts. Again, our manipulation of foveal reliability was successful, as JNDs increased with decreasing foveal contrast from 2.07° (fovea high, SD = 1.25°) to 3.6° (fovea mid, SD = 1.5°) to 6.6° (fovea low, SD = 1.1°), F(2,24) = 104.07, p < 0.001. JNDs in the fovea mid condition were significantly higher than in the fovea high condition, t(12) = 6.72, p < 0.001, and those in the fovea low condition were significantly higher than in the fovea mid condition, t(12) = 7.99, p < 0.001. JNDs derived from peripheral trials were 3.3° (SD = 1.83°) on average. Based on these JNDs, we predicted performance in integration trials (Equations 3 and 4) and compared predicted against observed performance. 

JNDs in integration trials also increased with decreasing foveal contrast from 1.41° (fovea high, SD = 0.91°) to 2.13° (fovea mid, SD = 1.31°) to 2.85° (fovea low, SD = 1.26°), F(2, 24) = 14.39, p < 0.001—mid versus high: t(12) = 2.96, p = 0.012; low versus mid: t(12) = 2.88, p = 0.014. Predicted and observed values highly coincided and did not differ significantly from each other (Figure 5; all p values > 0.3). To quantify the agreement between observed JNDs and model predictions, we fitted a line through the origin based on the aggregated data points. If predicted and observed values across all conditions perfectly matched, this line would fall on the identity line (slope of b = 1). Here, the best fitting line had a slope of b = 0.96. 

Figure 5

 

Experiment 2, observed against predicted JNDs averaged across observers, with error bars denoting 95% confidence intervals. Diagonal error bars mark the error of the differences between observed and predicted values and have to be compared to the identity line. Lines indicate the average JND in single trials. JNDs on the identity line are statistically optimal. Values above the identity line indicate suboptimal behavior. The slope of the best fitting line through the data and the origin is b = 0.96.

A strong model prediction to test integration is that the integrated percept should be more reliable than each of the single cues. This can be tested by comparing whether JNDs from integration trials are lower than the smaller of the two single JNDs. For each of the three contrast conditions, we tested separately whether JNDs in the trans-saccadic percept were lower than in the better of the two single conditions (one-sided t tests). This was true for the fovea high condition, where no contrast or any other manipulation between peripheral and foveal target was introduced, t(12) = 1.78, p = 0.05. In this condition, 11 out of 13 observers showed better performance than with either foveal or peripheral vision alone (Supplementary Figure S2). The same effect was observed in the fovea mid condition, t(12) = 1.82, p = 0.047. Here, nine out of 13 observers showed a benefit. In the fovea low condition, eight out of 13 observers had a lower JND than with either foveal or peripheral vision alone. This condition failed to show significance, t(12) = 1.32, p = 0.105. 

Discussion

In this experiment, we tested the second prediction of maximum-likelihood estimation to show that peripheral and foveal information is integrated across saccades. If information is integrated, then the reliability of the trans-saccadic percept should be the sum of the two single reliabilities and thus greater than the reliability of either peripheral or foveal vision on their own. For the high and mid foveal contrasts, JNDs from integration trials were significantly lower than the better of either the peripheral or foveal JNDs. For the low foveal contrast, no significant benefit of integration was observed. For all foveal contrasts, observed performance could be well predicted by optimal integration. This is strong evidence that information is integrated optimally across saccades. 

In terms of reliability, the benefit of integrating two signals is highest when they are both equally reliable. If they differ strongly, the integrated percept will be very similar to the more reliable of the two signals. Consequently, the effect size for the conditions in which peripheral vision is either inferior (high foveal contrast) or superior (low foveal contrast) should be comparatively low. Although the failure to show a significant decrease in JNDs for the low foveal contrast is in line with the tendency of suboptimal weights for decreased foveal contrasts in Experiment 1, it can also be explained by a lack of statistical power and does not necessarily speak against integration—especially because in this contrast condition, observed values perfectly matched predictions from statistical optimal behavior. Most importantly, despite any issues of statistical power, we could find evidence for integration even in the high-foveal-contrast condition. In this condition, the target was continuously displayed without any manipulation in either contrast or orientation and thus corresponds to normal vision, where objects do not change during the saccade. 

In a previous study, the aspect ratio of ellipses was varied independently before and after the saccade (Demeyer et al., 2010). That study also reported a bias in perception according to the peripheral information but did not observe a reduction in the variability of responses. However, in those experiments, trials with congruent and incongruent peripheral and foveal information were interleaved. It might be that integration is impaired with increasing probability and magnitude of incongruence between peripheral and foveal information, since a high correlation between cues seems to be a requirement for integration (Atkins et al., 2001; Parise et al., 2012). Here we provide evidence that information is integrated very close to optimally when no conflict between peripheral and foveal vision is introduced. 

Experiment 3: The time course of trans-saccadic integration

So far, we have shown that information about objects acquired before and after saccades is integrated according to its relative reliability. However, vision is a very dynamic process, and splitting it up into a presaccadic peripheral and postsaccadic foveal component does not yet tell us much about the exact time course of trans-saccadic information accrual. Specifically for presaccadically obtained information, opposite predictions can be derived from the literature: On the one hand, peripheral information acquired shortly before saccade onset might contribute most to the trans-saccadic percept because attention is enhanced (Kowler, Anderson, Dosher, & Blaser, 1995; Deubel & Schneider, 1996; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011) and crowding is reduced (Harrison, Mattingley, & Remington, 2013) at the saccade target prior to the onset of the movement. These results predict a continuously rising weight towards saccade onsets. On the other hand, there are findings and mechanisms which would suggest a reduced contribution of information immediately prior to saccade onset. First, due to delays in neural transmission, information which is available up to 80 ms prior to saccade onset does not modify the upcoming saccade (Becker & Jürgens, 1979; Caspi, Beutter, & Eckstein, 2004; Ludwig, Gilchrist, McSorley, & Baddeley, 2005; Ludwig, Davies, & Eckstein, 2014). This interval is called the “saccadic dead time.” Second, visual information starts to be suppressed up to 100 ms before an eye movement, and suppression is strongest around saccade onset (Diamond, Ross, & Morrone, 2000; Schütz et al., 2007; Bremmer, Kubischik, Hoffmann, & Krekelberg, 2009; Dorr & Bex, 2013). Both saccadic dead time and saccadic suppression predict a decrease in information gathering before saccade onset. To resolve this issue, we measured trans-saccadic perception with a reverse noise-correlation experiment. 

Reverse correlation (for a review, see Murray, 2011) is a method that allows for a very fine-grained temporal resolution of this time course. The logic is as follows: Throughout one trial, the target's orientation signal is not constant, but corrupted by dynamically changing noise. If any particular time point is contributing to the final percept, then perceptual responses and the orientation noise at this time point should be related. 

Methods

For the reverse-correlation experiment, we recorded data from four participants (all women, mean age = 25 years, range = 24–26). All four observers also participated in Experiment 1. They all gave informed consent, were paid for participation, and had normal vision. 

Trial procedure was identical to the integration trials in the first two experiments, except that no overlap paradigm was used and the true target orientation (signal) was corrupted by noise which changed every 25 ms. Again, the contrast of the vertical component was high (0.3) for peripheral targets and either remained constant (fovea high) or was reduced to 0.05 (fovea low) during the saccade. Importantly, the true underlying signal was negligibly small, with the noise determining most of the actual orientation. Although only the noise is relevant for the analysis, we included an underlying target signal so that performance would be slightly above chance. This was meant to keep observers involved with the task while maintaining the balance between hits and misses. The orientation signal was ±0.5°, ±1° for the fovea low condition and ±0.25°, ±0.5° for the fovea high, corrupted by noise with zero mean and a standard deviation of σ = 6°. Observers performed 1,600 trials in each of the two foveal contrast conditions. We had to discard 320 trials in the fovea high condition due to technical issues. Performance was 0.56 and 0.58, respectively, for the high and low foveal contrast conditions. For all observers in all conditions, the proportion of a correct response (hit) was between 0.5 and 0.6. Different contrast conditions were recorded in different sessions, and the order of the conditions was balanced across observers. 

Noise values from every trial were time logged to saccade onset for peripheral vision and to saccade offset for foveal vision. As saccades have different durations, it is not possible to determine a continuous graph throughout the saccade. However, we completed the analysis for the first 20 ms after onset and the last 20 ms prior to saccade offset. Noise values were recoded so that positive values coincided with the true underlying signal, then split up into hits and misses (Figure 6A). Then noise distributions for every millisecond were compared using receiver operating characteristic (ROC) analysis (Ludwig et al., 2014). If a given time point exerts no influence on the trans-saccadic percept, then the noise value at this time point does not influence the perceptual decision, and thus the two distributions of noise values for hits and misses are not distinguishable. If, however, a given time point exerts a high influence on the trans-saccadic percept, then a strong positive noise value should increase the likelihood of a correct response, whereas a noise value in the opposite direction would bias the response in favor of a false answer. The noise distribution for hits and misses should thus be easy to distinguish. The discriminability of two distributions can be measured using an ROC analysis. If two distributions are identical, the area under the ROC curve (AUC) is 0.5. If the two distributions are perfectly apart (no overlap), the area under the ROC curve is 1. 

Figure 6

 

Experiment 3, ROC analysis. (A) Histogram of noise distributions for foveal vision (fovea high) 15 ms after saccade offset. Trials were split up into hits and misses and then compared using an ROC analysis. Evaluation criteria for the histograms are slightly offset for better visibility. (B) ROC curve for the displayed distributions. For 20 linearly spaced criteria along the distributions, we compared the proportion of hits against the proportion of misses exceeding these criteria. Here, the AUC is 0.6.

To determine ROC curves, we computed the proportion of hits against the proportion of misses which exceeded given criteria (Figure 6B). Criteria were linearly spaced in 20 steps between the maximum and the minimum noise values. Then we determined the area under this ROC curve. The AUC corresponds to the influence this time point exerts on the trans-saccadic percept. This procedure was repeated for every millisecond. Alternatively, instead of using an ROC analysis, the two noise distributions can be averaged and their difference plotted as a function of time. As both procedures produced comparable results, we chose to use the ROC analysis, because it considers the whole noise distributions and not just their mean value. 

To determine saccade offsets offline, we fitted a linear regression to the eye position at every time point, with a time window of 11 ms. Offsets were defined as the first frame when the slope of the linear regression fell below 150°/s. 

Results

Using a reverse-correlation technique, we measured for each point in time how much that point contributes to the trans-saccadic percept. Here the contribution of each time point is expressed as AUC, the area under the ROC curve (Figure 6). Whenever the AUC is above 0.5 (chance level), information at this time point biases the trans-saccadic percept, with higher AUC values denoting a higher contribution. Please note that the AUC values are only informative within their specific time course, and should not be compared across different conditions. 

In both conditions, the peripheral AUC is above 0.5 (Figure 7), showing that peripheral vision in general contributes to the trans-saccadic percept. The relative contribution of peripheral vision is higher in the low- than in the high-foveal-contrast condition. Common to both conditions is the decrease starting between 100 and 50 ms prior to the saccade until a value of 0.5 is reached at saccade onset. 

Figure 7

 

Experiment 3, AUC time course for the fovea high (A) and the fovea low (B) condition. Black, gray, and red lines are the AUCs derived from an ROC analysis based on trials aggregated over all four observers. Thin lines correspond to the 95% confidence interval of between-subjects variability.

After saccade onset, the AUC remains constant around chance performance. This indicates that right after saccade onset, no (or only little) information is considered for perception. Prior to saccade offset, it rapidly increases and is instantly elevated as the eye lands at its target location. This increase is more pronounced in the fovea high condition (Figure 7A), where the AUC remains relatively constant throughout foveal vision. In the fovea low condition, performance is very dissimilar across participants, as can be judged from the confidence interval. Across participants, we can observe a postsaccadic enhancement after saccade offset, followed by a decrease starting 25 ms after the saccade and an oscillation between accuracy values of 0.51 and 0.54. However, a strong postsaccadic enhancement is observed in only one of the observers (Supplementary Figure S3). 

Discussion

Here we outlined the time course of trans-saccadic information gathering using a reverse noise-correlation approach. Consistent with Experiments 1 and 2, the individual contributions of peripheral and foveal vision to the integrated percept are modulated by their relative reliability. We also show that information gathering starts to decrease between 100 and 50 ms prior to the saccade and is lowest around saccade onset. Then, already before saccade offset, it increases, resulting in elevated foveal processing right at the end of the saccade. This can be observed especially under normal viewing conditions, when peripheral and foveal contrast were identical. 

Reverse-correlation allows for fine-grained temporal resolution, but the method has its limitations: AUC values are only informative within their specific time course, and absolute values should not be compared across conditions. There are also large interindividual differences in the time courses, which makes it difficult to make precise judgments about the timing of effects. This is why we want to highlight properties which are common in the pooled data (Figure 7) and across participants (Supplementary Figure S3). These are that performance is above chance for peripheral vision, it dips around the time of the saccade, and AUC values are instantly elevated after saccade offset. 

Decrease of information gathering before saccade onset

The gathering of information declined well before saccade onset. With respect to the underlying cause of this decline, it is important to note that AUC values indicate the weighting of information; hence we cannot distinguish whether no sensory information is taken in or whether it is taken in but not used. In the following we will discuss the relationship of this decline to other phenomena of peri-saccadic perception: saccadic suppression, presaccadic attention shifts, and saccadic dead time. 

The time course outlined here shares some similarities with peri-saccadic contrast sensitivity. Consistent with our results, saccadic suppression precedes a saccade by 50–100 ms and has its maximum around saccade onset (Diamond et al., 2000; Schütz et al., 2007; Dorr & Bex, 2013). But whereas suppression usually outlives the saccade by up to 50 ms, information gathering was immediately restored after saccade offset. Thus, the trans-saccadic time course outlined here cannot be fully explained by saccadic suppression. 

Previous research on the coupling of attention and eye movements has shown that attention is shifted towards the saccade target prior to the saccade, leading to a continuous rise of identification performance at the saccade target until saccade onset (Deubel & Schneider, 1996; Rolfs et al., 2011). This attentional modulation is exactly opposite to our decline of information gathering before saccade onset. There are three major methodological differences between our paradigm and the attention paradigms that might explain this discrepancy. First, while we used a target with low visual contrast, which might be very susceptible to saccadic suppression of contrast sensitivity, attention studies typically use high-contrast stimuli, which might be less susceptible to suppression (Rolfs et al., 2011). Second, several target locations have to be monitored at the same time in attention paradigms, while there was only one target location in our paradigm. Possibly the attentional benefit at the saccade target is only evident if attention has to be split among several locations. Third, studies on attention typically mask the target stimulus. If saccadic suppression reduces the efficacy of target and mask at the same time, these effects might cancel each other out, such that no effort of saccadic suppression is observable. 

Alternatively, the decay of peripheral information gathering prior to saccade onset could be related to the saccadic dead time (Becker & Jürgens, 1979; Caspi et al., 2004; Ludwig et al., 2005; Ludwig et al., 2014), during which incoming information does not influence saccade target selection anymore. Of course, target selection cannot be altered during the time of saccade motor programming, but it is not completely evident why the processing of object features should be interrupted by the programming of the saccade. 

The observed time course could also be interpreted as a sacrifice of peripheral information for foveal information. Peripheral information gathering could decline before the saccade such that foveal information can be used immediately when the eyes land on the target. In fact, the weighting of foveal information was already above chance at saccade offset, and it might be that this instantly elevated weighting is only possible if peripheral information briefly before saccade onset is neglected. This effect might not be specific for stimulus transitions from periphery to fovea, because instantly elevated processing after a saccade has also been observed for targets which were in the periphery after the saccade but had been foveated during the previous fixation (Ludwig et al., 2014). 

Increase of information gathering before saccade offset

Information gathering increased already before saccade offset and was instantly elevated at saccade offset. We think that this finding does not necessarily mean that information was actually gathered during the saccade, but that the rise in AUC before saccade offset could instead be caused by our experimental methods. Although we analyzed the data on a millisecond basis, the temporal resolution is inevitably limited by the frequency of noise changes in the experiment. Since the noise changed only every 25 ms, the data of each millisecond analysis interval are not independent of each other. Thus, despite the high temporal resolution in the analysis, we cannot reliably map sudden changes in performance like they might be observed around saccade offset (Figure 7A). Let us imagine a binomial process where information processing is completely turned off during the saccade and immediately switched on as the eye lands: Because the ROC analysis is computed for each millisecond but noise values are not independent on a millisecond basis, orientations right before saccade offset can be identical to orientations after offset. This overlap depends on the timing of the saccade offset relative to the updating of the noise value. As a consequence, the ROC analysis would yield values above chance even during the saccade. Hence the time window of 25 ms is too long to perfectly map this sudden switch, and would have resulted in a continuous increase even before the switch takes place. In the case of our results, this means that our time course is a low-pass-filtered version of the true underlying process and that the steep increase in information gathering at the end of the saccade could be even steeper and starting at a later time point. Regardless of the exact starting point of this increase, information gathering was immediately enhanced after saccade offset. To rule out the possibility that this instantly elevated performance is only due to the way the saccade offset is determined, we chose a criterion which provided us with a very early offset (Methods section; see also Figure 2B). 

General discussion

In this study we asked whether presaccadic peripheral information is maintained across saccades and integrated with postsaccadic foveal information in a statistically optimal manner. We measured peripheral and foveal orientation discrimination separately and then tested two predictions of maximum-likelihood estimation: When peripheral and foveal information was incongruent, peripheral information biased the trans-saccadic percept to an extent predicted by maximum-likelihood estimation (Experiment 1). By integrating congruent peripheral and foveal information, discrimination performance benefited in terms of reliability (Experiment 2). Moreover, we tracked the trans-saccadic time course and showed that information gathering starts to decrease well before the saccade, reaches a minimum at saccade onset, and is instantly elevated at saccade offset (Experiment 3). 

Across Experiments 1 and 2, optimal integration almost perfectly predicted the values observed. Neither any of the peripheral weights (Experiment 1) nor any of the JNDs (Experiment 2) were different from optimality. Thus the visual system is able to integrate information across saccades in a (nearly) optimal manner. In our experiments, conditions were chosen to facilitate integration by several factors. First, we varied foveal contrast to achieve comparable peripheral and foveal reliabilities. Since foveal vision is highly superior to peripheral vision for most tasks and features (for a review, see Strasburger, Rentschler, & Jüttner, 2011), the benefit of integration might be too small to be measured reliably under normal conditions. Second, there was only one object on the screen; and third, this object was the target for both the eye movement and the perceptual task. It might be that integration falls short of optimality for other conditions. Under the present conditions, however, peripheral and foveal information are integrated in a nearly optimal manner. Such an optimal integration requires three capacities: the maintenance of peripheral information from the pre- to the postsaccadic interval, the transfer to foveal neurons, and the calibration of peripheral and foveal information to overcome differences in processing across the visual field. 

A recent approach that could account for the calibration process highlights the importance of visual feature prediction and sensorimotor learning. Several studies have shown that trans-saccadic changes in object features (Cox, Meier, Oertelt, & DiCarlo, 2005; Li & DiCarlo, 2008) and associations between saccade direction and postsaccadic foveal displays (Bompas & O'Regan, 2006) can be learned. Moreover, the postsaccadic foveal appearance of a target is predicted based on presaccadic peripheral information (Herwig & Schneider, 2014; Weiß, Schneider, & Herwig, 2014). Such a prediction mechanism would be helpful in overcoming differences in resolution as well as biases across the visual field and creating visual stability. Of course, a comparison of presaccadic peripheral and postsaccadic foveal information is a necessary requirement for learning new associations between periphery and fovea. 

If integration relies on some sort of capacity-limited memory (Prime, Tsotsos, Keith, & Crawford, 2007; Bays & Husain, 2008), then integration should be impaired by the presence of other objects, as is the case in cluttered scenes. Previous research has shown that visuospatial attention is shifted towards the saccade target prior to the execution of the saccade (Kowler et al., 1995; Deubel & Schneider, 1996; Rolfs et al., 2011). It is possible that this shift of attention is necessary to preserve and transfer presaccadic information. In this case, integration of information about other objects in the periphery should be reduced and fall short of optimality. In line with this hypothesis, it has been shown that changes in natural scenes are more likely to be detected if they are close to the saccade landing position (Henderson & Hollingworth, 1999), and that saccades suppress processing at nontarget locations (Khan, Blohm, Pisella, & Munoz, 2015). These circumstances might also explain why previous studies in which the saccade and the perceptual target were not identical failed to detect integration of pre- and postsaccadic information (O'Regan & Lévy-Schoen, 1983; Irwin, 1991). 

One potential candidate for the transfer across space is spatial predictive remapping (Duhamel et al., 1992; Kusunoki & Goldberg, 2003), which describes the finding that neurons respond before a saccade to stimuli that will be in their receptive field only after the saccade. These findings have been interpreted in two ways: It could be that receptive fields are actually shifted (Duhamel et al., 1992), or it could be that only information is transferred from the future location of the receptive field (Cavanagh, Hunt, Afraz, & Rolfs, 2010). The former would limit integration to a brief period of time before the saccade, because predictive remapping occurs only 50 ms prior to a saccade (Kusunoki & Goldberg, 2003) and because information could only be taken into account once the receptive field was remapped. The finding of Experiment 3 that peripheral information is accounted from −200 ms until shortly before saccade onset cannot solely be explained by the shift of receptive fields, because peripheral information is used already before remapping takes place. Hence, it supports the transfer-of-information view, which suggests that information is accumulated continuously in peripheral neurons and then transferred to foveal neurons. This view is also supported by the presence of remapped responses prior to a saccade to stimuli which were briefly flashed and already disappeared at the time of remapping (Umeno & Goldberg, 2001). 

Our study provides the first evidence that information about object features can be integrated optimally across saccades. Using computational modeling, it has been shown that several perceptual phenomena reflect optimal integration of pre-, peri- and postsaccadic information about object location (Niemeier et al., 2003). As mentioned in the Introduction, location is a special feature because it is essential to eye movements and because extraretinal signals are necessary to compute location. This is not the case for other object features like orientation, color, and so on. Here we could show that peripheral information is used not only for the calculation of location but also for identification of purely perceptual features like orientation. 

The fact that foveal information is processed immediately after the offset of the saccade is a new and quite surprising finding. The predictive remapping (Duhamel et al., 1992) and the presaccadic shift of attention (Kowler et al., 1995; Deubel & Schneider, 1996) to the saccade target might allow this early processing of foveal information. Ecologically, it is also a sign of optimal behavior because it allows minimization of fixation durations and sampling of more locations in the visual surround. Such optimal processing of peripheral and foveal information is important for many tasks. Eye movements in visual search have been shown to be similar to an ideal searcher that maximizes information gain with each fixation (Najemnik & Geisler, 2005). This requires a representation of the visibility of the search target at different eccentricities and a memory of how much information has been gathered at different locations in the search area. An optimal integration of peripheral and foveal information across saccades would maximize the extraction of information and could be an important aspect in optimizing visual search performance. In addition, uncertainty about visual objects is available not only for optimal gaze guidance—which is a rather automatic process most of the time—but also for decision making when subjects have to choose the stimulus with lower uncertainty (Barthelmé & Mamassian, 2010). Understanding the mechanisms and limitations of trans-saccadic perception is a key factor towards a better insight into more complex processes like visual search and perceptual decision making. 

Conclusions

The present work shows that the visual system can store and integrate purely perceptual information across saccades with an efficiency that is indistinguishable from optimality. It helps to understand how our brains are able to create a stable perceptual world despite the drastically varying sensory input. Visual perception thus does not correspond to disconnected snapshots during each fixation but rather to an assembled, integrated stream of information. 

Acknowledgments

This work was supported by DFG grant SFB/TRR 135. We thank Karl R. Gegenfurtner and Matteo Valsecchi for comments on an earlier version of the manuscript, and Nina S. Krieger for help with data collection. Data from this study were presented at the 18th European Conference on Eye Movements. Data are publicly available at the doi:10.5281/zenodo.33878. 

Commercial relationships: none. 

Corresponding author: Alexander C. Schütz. 

Email: [e-mail]. 

Address: Allgemeine und Biologische Psychologie, Philipps-Universität Marburg, Gutenbergstr. 18, 35032 Marburg, Germany.