Transsaccadic integration is dominated by early, independent noise

Abstract

Humans are able to integrate pre- and postsaccadic percepts of an object across saccades to maintain perceptual stability. Previous studies have used Maximum Likelihood Estimation (MLE) to determine that integration occurs in a near-optimal manner. Here, we compared three different models to investigate the mechanism of integration in more detail: an early noise model, where noise is added to the pre- and postsaccadic signals before integration occurs; a late-noise model, where noise is added to the integrated signal after integration occurs; and a temporal summation model, where integration benefits arise from the longer transsaccadic presentation duration compared to pre- and postsaccadic presentation only. We also measured spatiotemporal aspects of integration to determine whether integration can occur for very brief stimulus durations, across two hemifields, and in spatiotopic and retinotopic coordinates. Pre-, post-, and transsaccadic performance was measured at different stimulus presentation durations, both at the saccade target and a location where the pre- and postsaccadic stimuli were presented in different hemifields across the saccade. Results showed that for both within- and between-hemifields conditions, integration could occur when pre- and postsaccadic stimuli were presented only briefly, and that the pattern of integration followed an early noise model. Whereas integration occurred when the pre- and post-saccadic stimuli were presented in the same spatiotopic coordinates, there was no integration when they were presented in the same retinotopic coordinates. This contrast suggests that transsaccadic integration is limited by early, independent, sensory noise acting separately on pre- and postsaccadic signals.

Introduction

In a visually rich environment, humans must constantly move their eyes to sample the surrounding world, making saccadic eye movements two to three times per second: areas of interest are selected with low-resolution peripheral vision, and then brought into higher-resolution foveal focus by the saccade. With each saccade the visual system must then reconcile the low-resolution presaccadic information with the high-resolution postsaccadic information. However, despite this constant shift of visual information across the retina, we retain a continuous and stable percept of the world. Transsaccadic integration provides one account for this perceptual stability, with several studies showing that humans can use properties from both the pre- and postsaccadic percepts of a stimulus, for example: color (Wittenberg, Bremmer, & Wachtler, 2008; Oostwoud Wijdenes, Marshall, & Bays, 2015; Schut, Van der Stoep, Fabius, & Van der Stigchel, 2018), orientation (Wolf & Schütz, 2015), shape and form (Demeyer, De Graef, Wagemans, & Verfaillie, 2009), numerosity (Hübner & Schütz, 2017), and spatial information (Prime, Niemeier, & Crawford, 2005; Cicchini, Binda, Burr, & Morrone, 2013). Recent studies have also used Maximum Likelihood Estimation (MLE) to demonstrate that integration occurs in a near-optimal manner (Ganmor, Landy, & Simoncelli, 2015; Wolf & Schütz, 2015; Hübner & Schütz, 2017), and that depleting potential cognitive resources such as attention (Stewart & Schütz, 2018a) and memory (Stewart & Schütz, 2018b) results in a departure from optimal integration. Although previous work has shown that integration can occur for many types of stimuli of differing levels of complexity, there are still a number of open questions as to how integration occurs. This study aimed to investigate four aspects of transsaccadic integration: The first, and predominant question is whether transsaccadic performance follows an early noise model, a late noise model, or a model of temporal summation. Second, we tested whether integration can occur even when stimuli are presented very briefly before and after the saccade. The third aspect of the study investiated whether integration is limited to the saccade target, or if it can occur even when the pre- and postsaccadic stimuli are presented in different hemifields. Fourth and finally, we compared transsaccadic integration in spatiotopic and retinotopic coordinates. 

Modelling noise in transsaccadic integration

Recent transsaccadic integration studies have taken a cue from the realm of multi-sensory integration, and have used models of cue combination to measure the optimality of transsaccadic integration: Wolf and Schütz (2015), Ganmor et al. (2015), and Hübner and Schütz (2017) have all used Maximum Likelihood Estimation (MLE) to predict optimal transsaccadic reliability from the measured reliabilities of pre- and postsaccadic performance alone. MLE predicts that if integration occurs, the reliability of the transsaccadic percept should equal the summed reliabilities of the pre- and postsaccadic percepts (Ernst & Bülthoff, 2004). These studies found that transsaccadic performance was near-optimal based on measurements of the single percepts, for both simple orientation stimuli (Ganmor et al., 2015; Wolf & Schütz, 2015), and more complex numerosity estimates (Hübner & Schütz, 2017). This MLE model assumes only that sources of noise from the separate signals will be independent and Gaussian (Ernst & Bülthoff, 2004); however, here we want to determine where the predominant source of noise occurs in the integration process. Jones (2016) outlines two models, early or late noise, each of which has different assumptions about where this noise is added. The early noise model assumes that in the integration of two sensory signals, each signal is subject to its own independent source of sensory noise, and that any subsequent postintegration decision noise will be negligible (Jones, 2016). In the case of transsaccadic integration, this means that the pre- and postsaccadic signals are each subject to independent sensory noise before integration, and transsaccadic performance will be limited exclusively by the amount of sensory noise introduced by these independent signals. In the late noise model, noise is assumed to be added after integration occurs, so that all variability in the response is added after the signals have been combined. This late noise could be neuronal noise added to the postintegration neural circuits, or even later decision or response noise (Jones, 2016). 

Determining whether integration follows the early or late noise model may provide insight into how pre- and postsaccadic information is represented: For example, the early noise model may suggest that pre- and postsaccadic signals are encoded retinotopically, with each signal being subject to independent sensory noise from the relevant neural populations or that saccades trigger a reset in the visual system (Bartlett, Ovaysikia, Logothetis, & Hoffman, 2011; Wutz, Muschter, van Koningsbruggen, Weisz, & Melcher, 2016; Benedetto, & Morrone, 2017), leading to independent noise before and after the saccade. Conversely, the late noise model may suggest that integration occurs very early, in a single, spatiotopic channel. By measuring the time-course of integration, we are also able to test a third alternative: that the increase in performance in the transsaccadic condition is merely due to the increased exposure time of the stimulus in the transsaccadic condition compared to the pre- or postsaccadic conditions alone. By comparing transsaccadic performance at a certain duration with pre- or postsaccadic performance for double that duration, we can estimate whether transsaccadic benefits reflect increased temporal summation of the stimulus. 

Temporal and spatial properties of integration

Most studies that have found optimal integration of pre- and postsaccadic stimuli have presented both pre- and postsaccadic stimuli for the approximate time of the saccade latency in that task (Wolf & Schütz, 2015; Hübner & Schütz, 2017; Stewart & Schütz, 2018a; Stewart & Schütz, 2018b), or in some cases even longer for the presaccadic stimulus (Ganmor et al., 2015). Fabius, Fracasso, and Van der Stigchel (2016) showed, however, that presaccadic information can be transferred rapidly across saccades for immediate comparison with a postsaccadic stimulus: An illusory jump could be perceived after a saccade when a presaccadic inducer was paired with a postsaccadic inducer shown for just 16.7 ms after the saccade. Additional ERP evidence has also shown that a presaccadic stimulus can affect postsaccadic stimulus processing shortly after saccade offset (Edwards, VanRullen, & Cavanagh, 2017), suggesting that presaccadic information may be immediately available after the saccade, and that integration of pre- and postsaccadic information may occur on a very short time-scale. These studies primarily examined information transfer where pre- and postsaccadic information was in the same spatiotopic location, and Fabius et al. (2016) in particular showed specifically strong effects when the pre- and postsaccadic stimuli were presented in the same spatiotopic coordinates. However, this rapid transfer of information across the saccade has been observed even when the pre- and postsaccadic stimuli are presented in different hemifields, with hemifield-related visual updating being observed from about 50 ms after saccade offset (Bellebaum & Daum, 2006), and evidence that visual information can be transferred and updated as fast as 50–100 ms after saccade offset (Muckli, 2012). These studies suggest that information can be updated quickly at different locations in the visual field; however, it is unclear whether optimal integration of pre- and postsaccadic information can also occur on a similar time-scale, and if optimal integration across hemifields is possible. A recent study showed that transsaccadic integration does not vary as a result of saccade landing distance from the integration target (Schut et al., 2018). Prime et al. (2005) also found that spatial information at various locations in the visual field could be integrated across saccades, suggesting that integration mechanisms may act in a broader manner across the visual field. 

Overview of experiments

In Experiment 1 we tested presaccadic, postsaccadic, and transsaccadic performance at two locations: either at the saccade target, or at a location halfway between fixation and target, such that the pre- and postsaccadic stimuli were presented in different hemifields. Stimuli were presented for differing durations before and/or after saccade onset, to determine the effect of stimulus duration on transsaccadic performance. This allowed us to compare three models of integration across the measured time-course: early noise, late noise, or temporal summation. In Experiment 2, we directly tested the reference frame of integration, by testing pre-, post-, and transsaccadic performance when the pre- and postsaccadic stimuli appeared in the same spatiotopic location (as in Experiment 1), and an additional condition when pre- and postsaccadic stimuli appeared in the same retinotopic location; that is, the postsaccadic stimulus was shifted on the screen by the same amount as the saccade amplitude. 

Experiment 1

Method

Participants

Twelve participants (one male, 11 female) aged between 19 and 24 years participated for money or course credit. All participants were naïve as to the purposes of the experiment and had normal or corrected-to-normal vision. Ethics approval was obtained from the local ethics commission of the Department of Psychology of Marburg University (proposal number 2015-35k), and experiments were conducted in accordance with the Declaration of Helsinki (1964). 

Equipment

Stimuli were presented on a 91 × 51 cm back projection setup with a PROPixx projector from VPixx Technologies and Stewart Filmscreen screen. This had a resolution of 1920 × 1080 and a refresh rate of 120 Hz, with a viewing distance of 106 cm. Background luminance was 70 cd/m², and the screen was calibrated to ensure a linear gamma correction. A hotspot correction was applied to ensure equal luminance across the screen. Eye movements were recorded with an Eyelink 1000 (SR Research Ltd., Ontario, Canada) with a sampling rate of 1000 Hz. Experimental software was written in MATLAB (MathWorks, Natick, MA) using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997). Participants responded using a standard keyboard and mouse. 

Stimuli

The central fixation target was a combination of a bulls-eye and a cross-hair shape (Thaler, Schütz, Goodale, & Gegenfurtner, 2013), which on each trial was randomly presented in a color generated in DKL color-space (Derrington, Krauskopf, & Lennie, 1984) with a set Cartesian value of 0.4 in the L + M axis, 0.6 on the L − M axis, and 0 on the S axis, and randomized polarity to avoid the build-up of afterimages. Placeholders were gray rings with diameter 1.3°, and luminance 55 cd/m². The saccade target was a black dot with diameter 0.2°, and luminance 2.08 cd/m². Perceptual stimuli were oriented Gabors with a standard deviation of 0.4° and a spatial frequency of 1 c/°, oriented either 5° clockwise or counter-clockwise (Figure 1B). Stimulus contrast was determined prior to the experiment for each participant. Contrast thresholds were measured for postsaccadic stimuli for each location in an identical procedure to postsaccadic trials in the main experiment, with the postsaccadic stimulus being presented for 40 ms. This duration was chosen to optimize performance levels for shorter stimulus durations. A QUEST procedure was used to determine a postsaccadic threshold equivalent to a d′ of 1.5. Thresholds were measured in three blocks of 40 trials each, with the final threshold being the mean of the three blocks. To equate performance across conditions, for the within-hemifields location: postsaccadic stimuli were presented at threshold contrast, and presaccadic stimuli were presented at 2 × the postsaccadic contrast. For the between-hemifields location, presaccadic stimuli were presented at 1.2 × threshold, and postsaccadic stimuli were presented at 2 × the presaccadic contrast. The mask consisted of band-pass filtered noise with a central frequency of 1 c/° and Gaussian SD of 0.4°. 

Figure 1

 

Task procedure and stimuli. (A) Events in a trial. (B) Examples of perceptual stimulus and mask. (C) Stimulus presentation location for within-hemifields condition and between-hemifields condition. (D) Timeline of events in a trial. (E) Density distribution of actual stimulus durations across all trials and participants.

Procedure

We tested performance at two locations, with three eye-movement conditions at each location. In the within-hemifields condition, the discrimination stimulus appeared in the same location as the eye movement target (15° eccentricity). We chose this location, because of two reasons: First, it resembles natural exploration behavior where peripheral stimuli are brought onto the fovea by saccades and second, previous studies already reported near-optimal integration for that location (Ganmor et al., 2015; Wolf & Schütz, 2015; Hübner & Schütz, 2017). Since the foveal representation is split between the two hemispheres (Lavidor & Walsh, 2004), some portion of the postsaccadic stimulus will be mapped into the other hemisphere than the presacccadic stimulus. However, this effect is alleviated by the typical undershoot of saccadic eye movements: For the within-hemispheres location, the average undershoot across all participants and conditions was 0.35°. In the between-hemifields condition, the discrimination stimulus appeared halfway between fixation and saccade target (7.5° eccentricity), and 4° above the horizontal meridian (Figure 1C). Stimulus locations were blocked, and participants were informed where the stimulus would appear at the start of each block. 

Stimulus duration

For all trials, the perceptual stimulus appeared at an estimated time of 50 or 80 ms before saccade onset; this then corresponded to an estimated stimulus duration of 50 or 80 ms. To estimate presentation time, median saccade latency was calculated across the preceding 20 trials and the stimulus was presented 50 or 80 ms before this time: For example, if the median saccade latency was 200 ms, the stimulus would be presented 150 or 120 ms after the saccade target appeared. Due to variability in the distribution of saccade latencies, the actual presentation time was distributed across a broad range of durations (Figure 1E). The estimated presaccadic duration (50 or 80 ms) was used for presaccadic trials, postsaccadic trials, and transsaccadic trials to ensure an even distribution of presentation durations across conditions. 

Presaccadic trials

In presaccadic trials, participants fixated the center of the screen and pressed the space bar to start. Four placeholders appeared on the screen. These placeholders indicated all possible stimulus locations across conditions to reduce spatial uncertainty about where the stimulus would appear. After a random delay between 1.5 s and 2.5 s, the saccade target (small black dot) appeared either 15° left or right of the fixation cross (Figure 1A). The perceptual stimulus (oriented Gabor) then appeared at the estimated time of 50 or 80 ms before the median saccade latency. After saccade onset, the stimulus disappeared and the saccade target alone was shown for the same duration as the presaccadic perceptual stimulus; then the mask was presented for the same presentation duration. Participants then gave a response as to whether the Gabor was tilted clockwise or counter-clockwise by using the “+” key on the keyboard to toggle the orientation of an on-screen bar, and pressing the “enter” key to make a selection. 

Postsaccadic trials

Postsaccadic trials had the same procedure as presaccadic trials, except that the saccade target (black dot) was presented until saccade onset, and after saccade onset the perceptual stimulus appeared for the same estimated presentation duration as a presaccadic stimulus would have been presented in that trial. The mask appeared directly after presentation of the perceptual stimulus. 

Transsaccadic trials

In transsaccadic trials, the saccade target (black dot) appeared; then the presaccadic perceptual stimulus appeared for the estimated presentation duration. After saccade onset, and once the presaccadic stimulus duration had elapsed, the postsaccadic perceptual stimulus appeared for the same duration as the presaccadic stimulus. Thus for a 50 ms transsaccadic trial, the pre- and postsaccadic stimuli were both presented for 50 ms. This switch from pre- to postsaccadic stimulus therefore happened at a variable point during the saccade. After the postsaccadic stimulus, the mask was presented for the same duration again. Pre- and postsaccadic stimuli had the same orientation. We also tested an additional condition where the postsaccadic stimulus was presented counter-clockwise to the presaccadic stimulus (at an orientation of 2° clockwise or counterclockwise). Switch trials were excluded for further analysis as the task difficulty made it impossible to distinguish between a switch in pre-/postsaccadic weighting over time or a general increase in performance from chance level as stimulus duration increased (see Supplementary File S1). 

Exclusions

Trials were excluded if the saccade endpoint was not within 2 SD of radial distance from the mean endpoint for each participant; if the saccade latency was below 50 ms, or the saccade latency was above 2 SD from the median saccade latency for each participant. One participant was excluded due to lack of data after exclusions were applied (mean number of data-points per bin less than 10 in one condition). In total, 28% of trials were excluded, with a further 4% excluded for technical reasons: Thus, 14,688 trials across 11 subjects were included for analysis. 

Analyses

Performance was calculated as the d′ of response to the orientation of the perceptual stimulus. A log-linear transformation was applied to account for cases where the performance was either 0% or 100% correct (Hautus, 1995). Performance at each time-bin was calculated using a 40 ms sliding window, and the resulting performance was smoothed for each participant using a robust local regression smoother with a moving window of 10 ms (robust Lowess smoothing) to exclude the effect of local outliers on the overall performance curve and on model comparisons (note that this smoothing did not affect overall model comparison results). Where d′ values were negative, values were adjusted to 0. 

Temporal alignment of stimuli

The presaccadic stimulus switched to the postsaccadic stimulus when the eye had exceeded 2° and when the presaccadic stimulus presentation duration had elapsed; however, the exact time of the switch relative to saccade onset was variable due to variability in estimated presaccadic stimulus presentation duration relative to the saccade. The presaccadic stimulus switched to the postsaccadic stimulus only after the presaccadic stimulus had been presented for the entire estimated stimulus duration for that trial; hence the point during the saccade where the stimulus switched was variable across trials. To ensure that the time of the stimulus switch relative to the saccade was consistent, we temporally aligned the switch to the middle of the saccade. Saccade duration was calculated for each participant (to calculate saccade offset, a linear regression was fitted to the velocity of the eye at every time point after peak velocity, with a moving window of 40 ms, and the offset was determined to be the first frame where the intercept of the fitted regression was equal to 0. For each trial, the stimulus switch time was adjusted to be aligned with the middle of the median saccade duration for that participant. Median saccade duration across participants was 46 ms; given the switch from the pre- to postsaccadic stimulus would happen in the center of the saccade, analyses for the GAMM are conducted from 23 ms stimulus presentation duration to account for the probe at shorter durations being presented entirely during the saccade. This period is marked with a shaded gray area in Figures 3 through 7. 

Models

We compared two models of integration using Maximum Likelihood Estimation. The early noise model assumes that pre- and postsaccadic stimuli are subject to separate sources of sensory noise before integration occurs (Figure 2; Jones, 2016):  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}d{^{\prime} _{int:early}} = \sqrt {d{^{\prime} _{pre}}^2 + d{^{\prime} _{post}}^2} \end{equation}

 

Figure 2

 

Outline of models used for comparison. Early and late noise models of Maximum Likelihood Estimation integration (adapted from Jones, 2016), and an alternative explanation for integration, whereby the integrated percept is equal to prolonged exposure of the single pre- or postsaccadic stimulus.

Figure 3

 

Model predictions for transsaccadic condition. Pre- and postsaccadic performance are assumed to be equal (dark blue, dark green, solid lines). Predictions are shown for the early noise model (yellow, dotted), late noise model (black, dotted), and temporal summation (light blue, dotted).

Figure 4

 

Performance for pre-, post- and transsaccadic conditions between and within hemifields. (A) Performance and 95% CI for each stimulus duration for presaccadic condition (green), postsaccadic condition (blue) and transsaccadic condition (red) when the perceptual stimulus is at the saccade target. The gray shaded area denotes time when the stimulus would have been presented during the saccade. (B) Performance as in (A) when the perceptual stimulus switched hemifields during the saccade. (C) Difference between transsaccadic performance or best single performance (pre- or postsaccadic) for each stimulus duration in the between-hemifields condition. Difference curve is obtained from the GAMM used to determine significance between conditions, including random effects of individual participants. Negative values (red shaded area) indicate transsaccadic performance was better than the best-single performance. Error is 95% CI. (D) Difference curve as in (C) for the between-hemifields condition.

Figure 5

 

Model comparisons at the saccade target. (A) Transsaccadic performance (red), early noise model predictions (yellow) and late noise model predictions (black). (B) Information weights for model comparisons for early and late noise. (C) Transsaccadic performance (red), early noise model predictions (yellow), temporal summation predictions for 2 × presaccadic performance (light green) and 2 × postsaccadic performance (light blue). (D) Information weights for model comparisons for early noise vs temporal summation models. All error bars are 95% CI.

Figure 6

 

Model comparisons for the between-hemifields condition. (A) Transsaccadic performance (red), early noise model predictions (yellow), and late noise model predictions (black). (B) Information weights for model comparisons for early and late noise. (C) Transsaccadic performance (red), early noise model predictions (yellow), temporal summation predictions for 2 × presaccadic performance (light green) and 2 × postsaccadic performance (light blue). (D) Information weights for model comparisons for early noise versus temporal summation models. All error bars are 95% CI.

Figure 7

 

Stimulus and placeholder layout in the spatiotopic and retinotopic conditions.

The late noise model assumes rather that the pre- and postsaccadic stimuli are integrated first, and any noise is added after integration occurs (Jones, 2016):  

\begin{equation}\tag{2}d{^{\prime} _{int:late}} = d{^{\prime} _{pre}} + d{^{\prime} _{post}}\end{equation}

 

We compared these models with a model of temporal summation, which assumes that any benefit from integration occurs because the stimulus is presented for double the amount of time in the transsaccadic condition (stimulus is presented both pre- and postsaccadically) compared to the pre- or postsaccadic conditions alone. If this is the case, performance for a transsaccadic stimulus presented for a certain duration (t) should be equal to performance on either single stimulus (pre- or postsaccadic) presented for double that duration (2t; Figure 2). For example, transsaccadic performance at 20 ms stimulus duration would be compared with pre- or postsaccadic performance at 40 ms duration:  

\begin{equation}\tag{3}{d^{\prime} _{int:sum}}\left( t \right) = {d^{\prime} _{pre}}\left( {2t} \right)\end{equation}

or  

\begin{equation}\tag{4}{d^{\prime} _{int:sum}}\left( t \right) = {d^{\prime} _{post}}\left( {2t} \right)\end{equation}

 

Model predictions

Figure 3 illustrates the behavior of each model, with the assumption of equated performance in pre- and postsaccadic conditions. Integration benefit from the MLE models will be maximal when the individual percepts are equated, and decrease as performance on the individual percepts draws further apart. To illustrate the models, we assume a logistic growth function for the pre- and postsaccadic performance, to reflect the experimental data which shows a plateau in performance (Equation 5).  

\begin{equation}\tag{5}f\left( x \right) = {L \over {1 + {e^{ - k\left( {x - {x_0}} \right)}}}},\!\end{equation}

where L = 3, k = 0.2, x₀ = 0, and x ranged from −20 to 60. Both early and late noise models predict an increase in the overall performance level with integration, whereas the temporal summation model predicts a shift in performance relative to stimulus duration, but no overall increase in the level of performance.  

Model comparisons

For each participant, model predictions were calculated for each of the three models: early noise, late noise, and temporal summation. The Akaike Information Criterion (AIC) was calculated as  

\begin{equation}\tag{6}AIC = n\log \left( {{{{\sigma }}^2}} \right) + 2K,\!\end{equation}

where n = the number of time-points included for the comparison (for example, 1–60 ms), Display Formula\({{{\sigma }}^2}\) = the residual sum of squares between the predicted performance for each model and the observed performance in the transsaccadic condition, and K = the number of free parameters in the model (0 for all models).  

Akaike information weights (Display Formula\({w_j}\)) were then calculated as (Equation 7):  

\begin{equation}\tag{7}{w_j} = {{\exp \left( { - {1 \over 2}{\Delta _j}} \right)} \over {\mathop \sum \nolimits_{r = 1}^R \exp \left( { - {1 \over 2}{\Delta _r}} \right)}}\end{equation}

Where Display Formula\({\Delta _j}\) is the difference between the AIC for any given model and the minimum AIC for all models being compared.  

Saccade landing error

To ensure that differences between pre-, post-, and transsaccadic stimuli were not due to differences in landing position between the conditions, we compared saccade landing error (radial error from saccade target) using a 1 × 3 ANOVA. To provide further evidence for a null effect of stimulus condition we calculated Bayes Factors using the BayesFactor package in R, using default priors (g-prior of variance and Jeffrey's prior on effects). Mean landing error for the within-hemifields condition was presaccadic: 1.00°; post-saccadic: 1.00°; transsaccadic: 1.01°; and for the between-hemifields condition, pre-saccadic: 1.12°; post-saccadic: 1.10°; transsaccadic: 1.15°. There was no effect of stimulus presentation condition for the within-hemispheres condition: F(2, 30) = 0.039, p = 0.96, BF₁₀ = 0.21, or the between-hemispheres condition: F(2, 30) = 0.82, p = 0.45, BF₁₀ = 0.34. 

Results

Temporal profile of integration

Observed vs best single performance

To determine whether integration had occurred, we compared the observed transsaccadic performance with the best single performance (pre- or postsaccadic) at every timepoint. We fitted a generalized additive mixed model (using R package mgcv), with a thin plate regression spline smoother applied to across the time-course, for each eye-movement condition (transsaccadic or best-single) and participant (random smooths with the same factor smoothing parameters across subjects), and a correction for auto-correlation at successive time-points. All factors included in the model are outlined in Table 1. We used the best-single condition as the baseline for comparison to compare the overall level of performance between eye movement conditions, and the change in performance between conditions over time (difference smooths). Figure 3C and D shows the difference smooth for the GAMM, indicating the difference between best single and transsaccadic performance. The time-points at which the 95% CI do not contain 0 are considered significant: in Figure 3C and D when the difference smooth is less than 0 (shaded pink area), transsaccadic performance is better than best single performance. Model estimates are shown in Table 1 for both within- and between-hemifields conditions. 

Table 1

 

Estimated effects from the GAMM for parametric coefficients, and smooths and random effects, including estimated degrees of freedom (E.df), reference degrees of freedom (Ref.df), F and p values.

 

For the within-hemifields condition, the difference smooth for duration × eye-movement condition indicated that the difference between best single and transsaccadic conditions across time was significant, and transsaccadic performance was significantly better than best single performance from 42–79 ms. For the between-hemifields condition, the difference smooth for duration × eye-movement condition similarly showed that transsaccadic performance was significantly different from best single performance from 55–89 ms. As stimulus durations below 23 ms would have been presented entirely within the saccade (Figure 3, gray shaded area), the actual duration where integration occurs would be 19–56 ms for the within-hemifields condition and 32–66 ms in the between-hemifield conditions. This indicates that an integration benefit occurs from 42 ms presentation duration in the within-hemifields condition, and 55 ms presentation duration in the between-hemifields condition, suggesting that integration benefits can be observed even when the pre- and postsaccadic stimuli are only presented briefly. However, this integration benefit did not continue for the entire measured time-course. This “end” of integration is likely due to the plateau in performance as the stimulus presentation duration was longer: If the best-single percept reaches a performance level close to maximum possible performance, then we cannot measure any integration benefits beyond this maximum. As stimuli were presented at a contrast threshold measured to obtain a d′ of 1.5 at 40 ms duration, this contrast level would equate to near-ceiling performance at longer durations. As the best-single performance reaches the maximum possible performance level, there is less potential to measure any better performance from integration (see Supplementary File S1 for full analysis). 

Model predictions, within-hemifields

We compared the early noise integration model and late noise integration model (Figure 2A; Jones, 2016) from the start of the measured time-course to 79 ms (end of integration period according to the GAMM). We chose this cut-off as any improvements in performance after the integration period could not be measured due to the ceiling effect in performance (see Supplementary File S1); thus, the model predictions could not accurately reflect the data. Figure 5A shows the measured transsaccadic performance with the predicted performance from the early and late models, and the information weights for each model (Figure 5B). The early noise model fitted the observed performance better than the late noise model: mean w_i early = 0.76, w_i late = 0.24, SD = 0.43 for both models. The early noise model fitted better for 8/11 participants. Given the early noise model more accurately fitted the data, we then compared this model with the temporal summation model for the time period of 1–60 ms: As the measured time-course ended at 120 ms, 60 ms was the maximum point at which transsaccadic performance at duration t (60 ms) could be compared with pre- or postsaccadic performance at time 2t (120 ms). Figure 5C shows the measured transsaccadic versus predicted performance from the temporal summation and early noise models. The early noise model fitted the observed data better than the temporal summation predictions from presaccadic performance: w_i early = 0.74, w_i 2 × presaccadic = 0.26, SD = 0.45, 8 of 11 participants, or postsaccadic performance: w_i early = 0.73, w_i 2 × postsaccadic = 0.27, SD = 0.46, 8 of 11 participants. 

Model predictions, between-hemifields

As above, transsaccadic performance was first compared with the early and late noise models (Figure 6A). The early noise model fitted the observed performance better than the late noise model: w_i early = 0.84, w_i late = 0.16, SD = 0.302, 10 of 11 participants. Figure 6C shows the measured transsaccadic versus predicted performance from the temporal summation and early noise models. The early noise model fitted the observed data better than the temporal summation model using presaccadic performance: w_i early = 0.91, w_i 2 × presaccadic = 0.09, SD = 0.301, 10 of 11 participants, and postsaccadic performance: w_i early = 0.83, w_i 2 × post-saccadic = 0.17, SD = 0.39, 9 of 11 participants. 

These results indicate that for both within- and between-hemifields conditions, the early noise model predicted transsaccadic performance better that the late noise model for the majority of participants. This suggests that the pre- and postsaccadic signals would each be subject to independent sensory noise, which is introduced before integration occurs. Furthermore, the early noise model fits the data better than the temporal summation model, indicating that the increased performance in the transsaccadic condition is not due to the increased exposure time of either pre- or postsaccadic stimuli. 

Discussion

The results of Experiment 1 showed that transsaccadic integration occurred when stimuli were presented only briefly, irrespective of whether pre- and postsaccadic stimuli were processed in the same or the opposite hemifield. Additionally, the early-noise model explained transsaccadic performance better than the late noise, or temporal summation models. This suggests that noise is added to pre- and postsaccadic stimuli independently, before integration occurs. This in turn suggests that pre- and postsaccadic stimuli are either encoded by separate neural populations, where each neural population is subject to its own independent source of sensory noise or that saccades trigger a reset of the visual system (Bartlett et al., 2011; Wutz et al., 2016; Benedetto & Morrone, 2017), such that the noise in pre- and postsaccadic processing is decorrelated. 

Experiment 2

In previous studies about the optimality of transsaccadic integration, pre- and postsaccadic stimuli were aligned in external space, i.e., were matched in spatiotopic coordinates (Ganmor et al., 2015; Wolf & Schütz, 2015; Hübner & Schütz, 2017; Stewart & Schütz, 2018a, 2018b). We conducted a second experiment where the postsaccadic stimulus was shifted by the same amount as the saccade amplitude, to determine whether the same pattern of integration occurs when pre- and postsaccadic stimuli are encoded by the same retinotopic neural population. 

Method

Participants

Eleven participants (aged between 20 and 30 years) participated in Experiment 2. Three of these had previously completed Experiment 1. 

Procedure

We tested pre-, post-, and transsaccadic performance in two conditions: spatiotopic, where the pre- and postsaccadic stimuli appeared in the same location on the screen (similar to the between-hemifields condition in Experiment 1); and retinotopic, where the postsaccadic stimulus location was shifted by the same amount as the saccade amplitude (Figure 7). The eye-movement target was 10° eccentricity. Presaccadic stimuli were presented at half this eccentricity (5°), at 4° above the horizontal meridian. In the spatiotopic condition, postsaccadic stimuli appeared at the same screen location as presaccadic stimuli. In the retinotopic condition, postsaccadic stimuli were shifted 10° (saccade amplitude) relative to the presaccadic stimulus in the direction of the saccade, thus were presented at 15° eccentricity. Spatiotopic and retinotopic conditions were blocked, and participants were informed where the stimulus would appear at the start of each block. All other methods were identical to those in Experiment 1. 

Exclusions

Three participants were excluded as their mean d′ performance was negative for either the spatiotopic or retinotopic condition, indicating that they could not do the task. The same exclusions as in Experiment 1 were applied to the remaining eight participants. In total 26% of trials were excluded, resulting in 9025 trials across eight subjects being included for analysis. 

Analysis

All analyses were identical to those of Experiment 1. 

Results

Temporal profile of integration

Observed versus best single performance

The median saccade duration across participants was 46 ms. As in Experiment 1, we used this value to determine that the period of time where the stimulus would have been presented entirely during the saccade was 23 ms (marked in gray in Figure 8). To determine whether integration occurred, we fitted a GAMM, as in Experiment 1 (Table 2). For the spatiotopic condition, the parametric coefficient for eye-movement condition indicates that transsaccadic performance was overall better than best-single performance. The difference smooth for duration × eye-movement condition indicated that the difference between best single and transsaccadic conditions across time was significant, and transsaccadic performance was significantly better than best-single performance from 30–45 ms and 58–119 ms. For the retinotopic condition, the parametric coefficient for eye-movement condition indicates that transsaccadic performance was overall worse than best-single performance. The difference smooth for duration × eye-movement condition indicated that the difference between best single and transsaccadic conditions across time was significant; however, transsaccadic performance was significantly worse than best-single performance from 55–98 ms. As stimulus durations below 25 ms would have been presented entirely within the saccade (Figure 8, gray shaded area), the actual duration where integration occurs would be 7–22 ms and 35–85 ms for the spatiotopic condition. 

Table 2

 

Estimated effects from the GAMM for parametric coefficients, and smooths and random effects, including estimated degrees of freedom (E.df), reference degrees of freedom (Ref.df), F and p values.

 

Figure 8

 

Performance for pre-, post-, and transsaccadic conditions for spatiotopic and retinotopic conditions. (A) Performance and 95% CI for each stimulus duration for presaccadic condition (green), postsaccadic condition (blue), and transsaccadic condition (red) in the spatiotopic condition. The gray shaded area denotes time when the stimulus would have been presented during the saccade. (B) Performance as in (A) for the retinotopic condition. (C) GAMM difference curve as in Figure 4 for the spatiotopic condition. (D) Difference curve as in (C) for the retinotopic condition.

These results show that we were able to replicate the transsaccadic integration benefit in the spatiotopic condition (as in Experiment 1), but there was no integration benefit in the retinotopic condition, and transsaccadic performance was in fact worse than best-single performance. 

Model predictions, spatiotopic condition

As in Experiment 1, we compared transsaccadic performance in the spatiotopic condition to the early and late noise models (Figure 9) from the start of the measured time-course until 119 ms (end of integration period). The early noise model fitted the observed performance better than the late noise model: w_i early = 0.75, w_i late = 0.25, SD = 0.46, 6 of 8 participants. This again replicates the results of Experiment 1, suggesting that integration performance is limited by early, independent noise. We did not perform the model comparison for the retinotopic condition, because there was no integration in that condition, which renders a comparison of integration models futile. 

Figure 9

 

Early versus late noise model comparisons for the spatiotopic condition.

Discussion

Experiment 2 replicated the results of Experiment 1, showing that integration occurred in spatiotopic coordinates from around 30–119 ms presentation duration. However, the same participants did not show any benefit of integration in the retinotopic condition. This suggests that integration occurs in spatiotopic but not retinotopic coordinates. 

General discussion

In Experiment 1 we tested pre-, post-, and transsaccadic performance for varying stimulus presentation durations before and/or after the saccade, when the pre- and postsaccadic stimuli were presented at the saccade target, or when the stimuli switched hemifields across the saccade. The results clearly demonstrated that a transsaccadic benefit can be observed even when the stimulus is presented very briefly, and this is true both within and between hemifields. This transsaccadic performance also followed an early noise model of integration rather than a late noise or temporal summation model, showing that transsaccadic performance is limited by the addition of early, independent sensory noise before integration occurs. In Experiment 2, we replicated the experimental and modelling results from Experiment 1 when pre- and postsaccadic stimuli were matched in spatiotopic coordinates, but we found no integration when pre- and postsaccadic stimuli were matched in retinotopic coordinates. This suggests that the primary reference frame of transsaccadic integration is spatiotopic rather than retinotopic. 

Early versus late noise

The early noise model of integration fitted transsaccadic performance better than the late noise or temporal summation models, both in the within- and between-hemifields conditions. As mentioned before, this finding could be explained by two alternatives: first, pre- and postsaccadic stimuli might be encoded by separate, retinotopic neural populations, each subject to their own source of independent sensory noise. Second, saccades might trigger a reset of the visual system (Bartlett et al., 2011; Wutz, Muschter, van Koningsbruggen, Weisz, & Melcher, 2016; Benedetto & Morrone, 2017), which leads to a decorrelation of noise between the processing of the pre- and postsaccadic stimulus. The finding of Experiment 2, that integration occurred only in spatiotopic coordinates but not in retinotopic coordinates, does not distinguish between these alternatives because even spatiotopic integration might rely on independent retinotopic channels. 

Indeed, there is much evidence to suggest that low-level feature information such as orientation is encoded retinotopically in V1 (Engel, Glover, & Wandell, 1997; Ress & Heeger, 2003); however, it is unclear whether the subsequent integration of this low-level information may also depend on similar retinotopic processes. Had the late noise model fitted better, this would unanimously suggest that pre- and postsaccadic information may be processed by a single, spatiotopic channel, and that the integration of this information would happen very early such that the predominant source of noise would occur after integration. This distinction between whether integration occurs in spatiotopic or retinotopic coordinates is interesting in the broader context of coordinates of perceptual stability in general: Many accounts of perceptual stability argue that perceptual updating occurs in a spatiotopic reference frame: for example, spatiotopic updating of information (Fabius et al., 2016) and memory (Zerr et al., 2017) across saccades, and adaptation after-effects (Melcher, 2005). However, other studies suggest that processes underlying perceptual stability may occur in a more retinotopic manner: For example, predictive remapping occurs in retinotopic coordinates (Duhamel, Colby, & Goldberg, 1992), visual features are preserved and remapped in retinotopic coordinates (Harrison, Retell, Remington, & Mattingley, 2013; Melcher, 2008) and transsaccadic memory and attention show retinotopic effects (Golomb, Nguyen-Phuc, Mazer, McCarthy, & Chun, 2010). These divergent results suggest that both retinotopic and spatiotopic mechanisms contribute to the perceived unity and stability of the world, and it is likely that there is a complex interplay between the two (Burr & Morrone, 2012). It is also important to consider that integration may also occur at different stages of processing, and that this may differ depending on the complexity and features of the stimuli, and the inherent physiological differences in encoding the stimuli. For instance Burr and Morrone (2011) suggest that the integration that may occur at earlier stages of processing such as V1 should not be spatiotopic, as V1 encodes stimuli in retinotopic coordinates. At later stages of processing, however, for example for features such as form and motion which may show additional spatiotopic encoding in MT and MST (d'Avossa et al., 2006; Crespi et al., 2011), integration could also occur on a spatiotopic level; nevertheless, note that the issue of whether MT does in fact encode information in spatiotopic coordinates is contentious (Gardner, Merriam, Movshon, & Heeger, 2008; Knapen, Rolfs, Wexler, & Cavanagh, 2010), and it is likely that, similar to other transsaccadic processes such as the motion after-effect, there are both spatiotopic and retinotopic effects present (Ezzati, Golzar, & Afraz, 2008). The implications of this are twofold: First, different stimulus features may give rise to different forms of integration, and second, low-level integration of basic features may follow this retinotopic, early noise model, whereas higher-level integration of more complex form or motion may follow a more spatiotopically defined late noise model. As the late noise model also predicts greater integration benefits than the early noise model (Figure 4), it would perhaps be easier to test integration in these cases, and to observe greater integration effects. The current study clearly showed that integration occurred in spatiotopic but not retinotopic coordinates and also showed that integration performance, and thus potentially perceptual stability may not rely solely on higher level perceptual effects that seem to arise spatiotopically, potentially after integration, but that the low-level retinotopic representations of the stimuli that occur before integration are also crucial. Interestingly, Fabius et al. (2016) showed that integration of information can occur in spatiotopic coordinates, but also in retinotopic coordinates (albeit to a lesser extent). This difference from our results could reflect the different stimuli used: Whereas our experiment was an orientation detection task, Fabius et al. (2016) used motion to induce an illusory jump in the direction of a noise patch. This difference in integration coordinates could reflect the coordinates in which the different stimuli (orientation vs. motion) are encoded. Experiment 2 showed not only a lack of retinotopic integration, but an additional cost in transsaccadic compared to best-single performance. This could be due to the stimulus shift in the retinotopic condition: When the pre- and postsaccadic stimuli appear in different spatiotopic coordinates, they may no longer be grouped as a single object that would benefit from integration. This may point to the system having a general assumption of spatiotopic constancy, as observed for instance in saccadic suppression of displacement (Bridgeman et al., 1975; Deubel, Schneider, & Bridgeman, 1996). Preserving separate representations of the pre- and postsaccadic stimulus might require additional resources in attention and working memory (Schneider, 2013; Poth, Herwig, & Schneider, 2015). 

Both models of integration predict that the neuronal responses to each cue are linearly summed: This is consistent with the physiological evidence showing that the integration of sensory signals occurs by the summation of neuronal signals (Stein & Stanford, 2008), for example in the superior colliculus (Meredith & Stein, 1983; Stanford, Quessy, & Stein, 2005). An increase in combined neuronal sensitivity in MSTd neurons has also been correlated to near-optimal integration according to the MLE cue-combination model (Gu, Angelaki, & DeAngelis, 2008): A difference in reliability between the two cues results in a change in their neural weighting (Morgan, DeAngelis, & Angelaki, 2008), analogous to psychophysical studies showing that in cue-combination the less reliable cue is given less perceptual weighting (Ernst & Banks, 2002; Alais & Burr, 2004). This shows a consistency between the model predictions for both behavioral and physiological data. 

It must also be noted that while this study examined the two extreme cases of either only early noise or only late noise. In reality it is likely that performance may be subject to both early sensory and late decision noise, which could result in integration performance falling between the predictions of the early and late noise models (Saarela & Landy, 2012). However, in the current study, the predictions for early versus late noise models were both fairly close to the observed transsaccadic performance; hence, it would be impossible to disentangle the contributions of early versus late noise. As Jones (2016) notes, this separation of early vs late noise may not in itself be a trivial task, given factors such as perceptual learning and evolution of cue combination strategies, and correlation of noise between the cues. Indeed, although the early noise model assumes that the noise associated with each cue is independent, it is likely that there may be some degree of correlation between the firing in these neural populations (Salinas & Sejnowski, 2001). Given, however, that the data fit the early noise model predictions better, they suggest that the contribution of late noise, if any, is smaller than that of early noise, so we can conclude that performance in this integration task is limited by the addition of early noise before integration occurs. 

Temporal integration of brief signals

This study showed that integration can occur when the pre- and postsaccadic stimuli are presented for very short durations. This supports previous evidence that information can be rapidly updated across saccades (Fabius et al., 2016), and suggests that optimal integration can occur on a very short timescale. Thus far, studies of transsaccadic integration, and especially optimal transsaccadic integration, have tested stimuli that are presented for the duration of the saccade latency or longer (Wolf & Schütz, 2015; Ganmor et al., 2015; Hübner & Schütz, 2017), but our results show that integration can also occur when stimuli are only viewed briefly. This suggests that presaccadic information is immediately available for comparison with postsaccadic information: This idea is consistent with findings showing that presaccadic information influences postsaccadic processing shortly after the saccade (Edwards et al., 2017), and also that presaccadic information can be used to predict postsaccadic information (Herwig & Schneider, 2014; Herwig, Weiß, & Schneider, 2015). These studies again all used longer pre- and postsaccadic stimulus durations, but the results of the current study suggest that this presaccadic information can be utilized postsaccadically even when stimuli are presented for as brief a duration as 42 ms before the saccade, so it may be the case that information presented even briefly before saccade onset could bias postsaccadic perception. Interestingly, however, although integration can occur at brief stimulus durations, this may not be the case for all measures of transsaccadic association: Zimmermann, Morrone, and Burr (2013) found that the transsaccadic discrimination of spatial target displacements improves with longer presaccadic durations, albeit only for large targets (Collins, 2016; Zimmermann, 2016), and interpreted this as evidence for a slow build-up of a spatiotopic representation. This finding is interesting, as the current study suggests that briefly presented presaccadic feature information can be used for integration to achieve a perceptual benefit; however, brief stimulus presentation may not be useful for producing a spatially precise stimulus correspondence. A key difference, however, between these studies is that in our study, pre- and postsaccadic stimuli always appeared with the same orientation and at the same location, whereas in Zimmermann et al. (2013) the location of the pre- and postsaccadic stimuli had to be compared and judged: This meant that the presaccadic stimulus had to be encoded into memory in order to survive being overwritten by the postsaccadic stimulus, which may require longer time. 

Spatial integration across hemifields

The further finding that we can observe transsaccadic integration benefits across hemifields concurs with previous studies showing that information can rapidly update across hemifields (Bellebaum & Daum, 2006; Muckli, 2012). More generally, it supports findings of integration of location information across the visual field (Prime et al., 2005) as well as a recent study that found that transsaccadic integration is not specific to the saccade target (Schut et al., 2018). This is also in line with evidence suggesting that orientation adaptation information can be transferred across hemispheres during a saccade (Zimmermann, Weidner, Abdollahi, & Fink, 2016). 

The pattern of transsaccadic integration following the early noise model was consistent in the within- and between-hemifields conditions, and suggests that the mechanisms underlying integration may be the same whether pre- and postsaccadic information is presented in the same or different hemifields. Anatomical evidence suggests that at the level of early visual processing, for example V1–V4, targets in different hemifields are processed independently and retinotopically (Wandell, Dumoulin, & Brewer, 2007). This theory fits the early noise hypothesis that pre- and postsaccadic information are subject to independent sources of sensory noise for the between-hemifields condition, as each stimulus would be processed by separate, retinotopically defined neurons in early visual cortex. By this logic, given the within-hemifields condition shows the same pattern of early-noise integration, these ideas lend weight to the hypothesis that the pre- and postsaccadic stimuli at the saccade target are similarly selected via separate, retinotopic neural populations. 

This integration across hemifields may also give us an insight into the higher level mechanisms driving the integration process. For example, hemifield independence in attentional processing has been found in attentive tracking and search tasks (Alvarez & Cavanagh, 2005), but only when the task required location rather than feature-specific attention. Indeed, feature-based attention was also found to be coordinated across hemifields whereas location-based attention was independent (Cohen & Maunsell, 2011). Transsaccadic integration has been found to rely on attentional processes (Stewart & Schütz, 2018a): Taken together with the results of the current study that suggest that integration can occur across hemifields, this finding may point to integration occurring via a feature-specific attentional resource that is not hemifield-independent. This account would agree with studies suggesting that integration may occur via correspondence of object representations across a saccade (Irwin, 1996), or that information is integrated via a feature integration mechanism (Kahneman, Treisman, & Gibbs, 1992). It is, however, unclear whether this process may occur by an automatic integration of the entire visual field, or whether only areas that are flagged as being attentionally relevant due to task demand are integrated (Cavanagh, Hunt, Afraz, & Rolfs, 2010; Stewart & Schütz, 2018a). 

Conclusion

This study showed that transsaccadic integration can occur even when stimuli are only presented briefly before and after the saccade, and that this can occur even when the pre- and postsaccadic stimuli are presented in different hemifields. Integration occurs only when pre- and postsaccadic stimuli are presented in the same spatiotopic but not retinotopic coordinates. This time-course of integration fits an early noise model of integration better than late noise or temporal summation models, suggesting that transsaccadic integration performance is limited by early, independent sources of noise that are added to pre- and postsaccadic sensory signals before integration occurs. 

Acknowledgments

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No 676786). We thank Casimir Ludwig for helpful comments on an earlier version of the manuscript and Hannah Walter, Lena Weinert, Julia Schnitter and Almut Gitter for helping with data collection. Data are available at https://doi.org/10.5281/zenodo.2656263. 

Commercial relationships: none. 

Corresponding author: Emma E. M. Stewart. 

Email: . 

Address: Experimental and Biological Psychology, University of Marburg, Marburg, Germany.