Intra-saccadic motion streaks as cues to linking object locations across saccades

Abstract

When visual objects shift rapidly across the retina, they produce motion blur. Intra-saccadic visual signals, caused incessantly by our own saccades, are thought to be eliminated at early stages of visual processing. Here we investigate whether they are still available to the visual system and could—in principle—be used as cues for localizing objects as they change locations on the retina. Using a high-speed projection system, we developed a trans-saccadic identification task in which brief but continuous intra-saccadic object motion was key to successful performance. Observers made a saccade to a target stimulus that moved rapidly either up or down, strictly during the eye movement. Just as the target reached its final position, an identical distractor stimulus appeared on the opposite side, resulting in a display of two identical stimuli upon saccade landing. Observers had to identify the original target using the only available clue: the target's intra-saccadic movement. In an additional replay condition, we presented the observers’ own intra-saccadic retinal stimulus trajectories during fixation. Compared to the replay condition, task performance was impaired during saccades but recovered fully when a post-saccadic blank was introduced. Reverse regression analyses and confirmatory experiments showed that performance increased markedly when targets had long movement durations, low spatial frequencies, and orientations parallel to their retinal trajectory—features that promote intra-saccadic motion streaks. Although the potential functional role of intra-saccadic visual signals is still unclear, our results suggest that they could provide cues to tracking objects that rapidly change locations across saccades.

Introduction

The dystopian science-fiction television series “Black Mirror” featured an episode in which people could record memories through their pupils (Armstrong & Welsh, 2011). Replays of these memories revealed a common but wrong intuition of how our eyes capture the world around us. They showed skillfully crafted videos, with smooth camera movements from one location to the next. In reality, we make several saccadic eye movements every second that rapidly shift the entire image of the visual scene across the retina. Upon each new fixation, each object in the scene is projected onto a new part of the retina and processed by new populations of neurons throughout retinotopic visual cortex. Yet, these jerky displacements are not part of our perceptual experience—the visual world is stable. Whereas this phenomenon has received attention for centuries and has inspired research and theory to this date (Binda & Morrone, 2018; Burr & Morrone, 2011; Cavanagh, Hunt, Afraz, & Rolfs, 2010; Hall & Colby, 2011; Higgins & Rayner, 2015; Marino & Mazer, 2016; Wurtz, 2018; Ziesche & Hamker, 2014), a fundamental question remains unanswered: How does the visual system keep track of an object that is changing locations on the retina as the eyes move (Rolfs, 2015; Wurtz, 2008)? That is, how do we determine a correspondence between two successive views of an object across a saccade? 

Here we hypothesized that intra-saccadic information could contribute to object identification across saccades. For example, motion streaks, which occur due to the slow integration of visual signals at early stages of visual processing (Geisler, 1999), are generated each time eye movements cause object movement across the retina (Bedell & Yang, 2001; Brooks, Impelman, & Lum, 1981; Duyck, Collins, & Wexler, 2016; Matin, Clymer, & Matin, 1972). Motion streaks—and intra-saccadic smear in general—imposed by our own eye movements have been widely considered a hindrance to perceptual stability that are counteracted by specialized mechanisms and thus eliminated from perception (for a collection of examples, see Castet, 2010). These mechanisms range from passive accounts, such as shearing forces in the retina that reduce visual sensitivity during saccades (Richards, 1969) and pre- and post-saccadic masking (Castet, 2010; Matin et al., 1972), to active suppression of information in the magnocellular pathway (Burr, Morrone, & Ross, 1994; Ross, Morrone, Goldberg, & Burr, 2001), as well as combinations of these (Volkmann, Riggs, White, & Moore, 1978; Wurtz, 2018). We know, however, that intra-saccadic motion perception is possible if the stimulus has a velocity similar to that of the saccade (Castet & Masson, 2000; Castet, Jeanjean, & Masson, 2002). Similarly, if the visual scene is briefly illuminated during a saccade, observers have a clear impression of a smeared and blurry visual scene (Campbell & Wurtz, 1978). In addition, stimuli undergoing saccadic suppression can still influence post-saccadic judgments, even if the observer is unaware of them (Watson & Krekelberg, 2009). Recently, the hypothesis has been proposed that effectively modulating the spatiotemporal power distribution in the retinal image saccades enhances low spatial frequencies (SFs) and thus facilitates a coarse-to-fine strategy of post-saccadic visual processing (Boi, Poletti, Victor, & Rucci, 2017; Rucci, Ahissar, & Burr, 2018). Indeed, Boi et al. (2017) showed that contrast sensitivity to post-saccadic low SF (but not high SF) information is greater if the stimulus has its onset during the saccade than if the same stimulus is presented with a contrast ramp during fixation. A functional role of visual processing during the saccade might thus be facilitating the processing of coarse information during early fixation. In particular, a potential function role of intra-saccadic visual information, such as motion streaks, remains elusive. 

Perception of intra-saccadic motion streaks induced by saccades across a static stimulus (often presented in a dimly lit, uniform background or in complete darkness) has been investigated in past studies. For example, Matin et al. (1972) investigated the perceived length of motion streaks induced by varying on-durations of a single light source, and Brooks et al. (1981) studied the threshold elevation in detecting them independent of pre- and post-saccadic masks during both real and simulated saccades. Bedell and Yang (2001) used a similar paradigm and found that, given similar prolonged post-movement durations of the light source being on, perceived streaks were still significantly longer during fixation than during saccades, suggesting an additional attenuation of smear around saccades. More recently, Duyck et al. (2016) used an objective technique to quantify streak efficiency by tasking their participants with localizing a gap (realized by briefly dimming a light-emitting diode during the saccade) in an intra-saccadic motion streak. 

All of the studies showed that the perceived streak was directly related to presentation duration, revealed the location of the inducing light source, and could be attenuated by presenting prolonged post-movement endpoints. Although it is again impossible to conclude from these results whether or not intra-saccadic motion streaks are relevant to the visual system, the hypothesis that they could be is informed by three insights. First, motion and contrast detection during saccades have been shown to be possible, provided that the presented stimulus has been optimized for the direction and velocity of the saccade (Castet & Masson, 2000; Castet et al., 2002; García-Pérez & Peli, 2011; Mathôt, Melmi, & Castet, 2015; Schweitzer & Rolfs, 2019). Second, information undergoing perceptual omission is not eliminated from visual processing (Watson & Krekelberg, 2009). Third, in a static visual environment, self-induced retinal input is related to any ongoing eye movement and could thus provide information about the direction, amplitude, and velocity of eye movements (Matin et al., 1972). 

To test the hypothesis whether it is, in principle, possible to use strictly intra-saccadic continuous object motion to link the identities of objects across saccades as they change locations on the retina, we developed a trans-saccadic identification paradigm. Observers (N = 15) made a saccade toward a target stimulus in the visual periphery. Upon saccade landing, the display contained two stimuli—the target stimulus and an identical distractor—one above and one below the target's previous location. Observers had to identify the original target stimulus, which had moved rapidly but continuously to its new location as the eyes were in flight (see Methods section). The distractor stimulus, in turn, merely appeared on the opposite side as soon as the target stimulus had reached its final location. To identify the target stimulus in a two-alternative forced choice task (Figure 1a), therefore, observers could not simply rely on detecting a displacement (Brooks, Yates, & Coleman, 1980; Wexler & Collins, 2014) or use efficient encoding of the pre-saccadic object location in memory (Zimmermann, Morrone, & Burr, 2013), as they would in a classic saccadic suppression of displacement task (Bridgeman, Hendry, & Stark, 1975), because the size of the displacements was always the same for both post-saccadic stimuli and thus entirely predictable across all trials. Instead, in our task intra-saccadic continuous object motion was the key to linking the target's pre- and post-saccadic locations. 

Figure 1.

 

Experimental procedure. (a) Saccade condition, where human observers made horizontal saccades of 14.6 dva to a target stimulus. Upon saccade detection, the target moved either up or down by 3.6 dva with short movement durations (4–17 ms). In Experiments 1, 3, and 4, a second, identical distractor stimulus was presented immediately upon completion of the stimulus movement, at the alternative location, opposite the target stimulus. In Experiment 2, the target stimulus disappeared as soon as it reached its final vertical position and reappeared along with a distractor after a 250-ms blanking period. (b) Replay condition, where, in a fixation condition, we simulated intra-saccadic retinal input by replaying the trajectory of the stimulus according to the observer's own eye movement as recorded in the saccade session. (c) In Experiments 1 and 2, stimuli were 50% contrast noise patches in a Gaussian aperture (SD = 0.45 dva), bandpass-filtered to either low SF (0.28–1.12 cpd) or high SF (1.12–4.5 cpd). In Experiments 3 and 4, stimuli were Gabor patches of varying orientations (sigma = 0.45 dva) with SF of 0.56, 1.12, or 2.25 cpd. (d) Illustration of retinal stimulus trajectories in an example trial (rightward saccade and upward stimulus movement). Stimulus movements in both saccade and replay conditions would produce the same retinal trajectories, but only the target stimulus could produce a continuous streak. The shape of that streak depends on both saccade and stimulus speed. (e) Average retinal trajectories for the range of movement durations used (shown again for rightward saccades and upward stimulus movements).

Presenting precise, continuous object motion during the brief saccadic interval required visual presentation at speeds an order of magnitude faster than standard laboratory screens can display. Using a digital light processing (DLP) projector tailored to this purpose, we updated stimulus positions at submillisecond resolution strictly during the saccade and achieved high velocities of vertical motion ranging from 213 to 853 degrees of visual angle (dva) per second due to extremely short movement durations (4 to 17 ms). To ensure that participants could perform the task in principle and to have a comparison for performance during saccades, we also assessed performance during fixation. To this end, we upsampled participants’ eye movement data (recorded at 500 Hz) to the frequency of the projector (updating the display at 1440 Hz) and replayed the trajectory of the stimulus in a second session (Figure 1b). This procedure simulated the retinal consequences of the saccade task with high temporal fidelity. 

Across four experiments, we investigated to what extent the rapid movement of objects across the retina induced by combined stimulus and eye movements informs the identification of the original pre-saccadic object from the two identical post-saccadic objects. We found that performance in this trans-saccadic identification task was influenced by the distinctness of the motion streak as induced by intra-saccadic retinal object movement, as well as by its temporal and spatial extent throughout the saccade. Subtle motion streaks, available for a short period during the saccade, yield only low, yet above-chance, performance (Experiments 1 and 3), unless post-movement blanking interval is introduced (Experiment 2). In contrast, performance increased to a large extent when high-contrast objects with an orientation optimized for the retinal trajectory of the stimulus moved over large portions of the saccade (Experiment 4). Given that static, salient objects in the visual scene generate intra-saccadic visual input throughout the entire duration of the saccade, these results invite the intriguing hypothesis that they might provide a parsimonious visual cue that helps the visual system to keep track of retinal locations of objects across saccades. 

Methods

Participants

Fifteen participants per experiment were recruited through word of mouth and campus mailing lists. They had normal or corrected-to-normal vision and gave written informed consent prior to beginning the experiment. Monetary reimbursement was offered for their time. The study was conducted in agreement with the latest version of the Declaration of Helsinki and approved by the Ethics Committee of the German Society for Psychology, and participants provided written informed consent before participation. All experiments reported here were preregistered on the Open Science Framework (see below). 

Experiment 1

Two sessions (saccade and replay, each a maximum of 800 trials) were run on separate days. Participants (10 female, 5 male) had a mean age of 25.5 years (range, 19–40) and received 18 Euros for both sessions. Ten of 15 participants had right ocular dominance, and 14 participants were right-handed. Ocular dominance was determined using a variant of the Porta test (Della Porta, 1593). Observers were asked to align both hands relative to a salient vertical line in their environment. By closing one eye or the other in alternation, they reported which eye caused a larger horizontal shift of the world. Pre-registration, data, and participant data can be found on the Center for Open Science website (https://osf.io/zszd9). 

Experiment 2

Participants (9 female, 6 male) with a mean age of 22.7 years (range, 18–28) completed two sessions (maximum of 800 trials) on separate days and received 18 Euros for both sessions. Nine participants had right ocular dominance, and 13 participants were right-handed. Pre-registration and data can be found at https://osf.io/c95g6/. 

Experiment 3

Thirteen female and three male participants with a mean age of 23.7 years (range, 19–39) completed two sessions (maximum of 768 trials) on separate days and received 16 Euros for both sessions. As in the second experiment, nine participants had right ocular dominance, and 13 participants were right-handed. Pre-registration and data can be found at https://osf.io/7apa8/. 

Experiment 4

Nine female and six male participants with a mean age of 25 years (range, 18–31) completed one session (maximum of 768 trials) and received 10 Euros as remuneration. Out of a total of 15 participants, nine had right ocular dominance, and 14 participants were right-handed. Pre-registration and data can be found at https://osf.io/7q9pr/. 

Apparatus

Stimuli were projected onto a 200 × 113 cm video-projection screen (Celexon HomeCinema, Tharston, Norwich, UK), using a Propixx DLP Projector (Vpixx Technologies, Saint-Bruno, QC, Canada) running at a temporal resolution of 1440 frames per second and a spatial resolution of 960 × 540 pixels². Experiments took place in a dimly lit, sound-attenuated room. The gray background used in all experiments had an average luminance of 30 cd/m². Observers sat 270 cm from the projector with their head supported by a chin rest. Eye movements were measured using an Eyelink 2 head-mounted system (SR Research, Osgoode, ON, Canada) with a sampling rate of 500 Hz. Stimulus display was implemented in MATLAB 2015a (MathWorks, Natick, MA, USA), using Psychtoolbox (Brainard, 1997; Kleiner, Brainard, Pelli, Ingling, Murray, & Broussard, 2007) and Eyelink Toolbox (Cornelisen, Peters, & Palmer, 2002) extensions, running on a Dell Precision T7810 Workstation with a Debian 8 operating system (Round Rock, TX, USA). Responses were collected via a standard US-English keyboard. 

Stimuli

All stimuli, both noise and Gabor patches, were enveloped in a Gaussian window with a standard deviation of 0.45 dva. The fixation dot used in all experiments and sessions was a white circle of 0.3-dva radius. When fixated, the area within the circle was filled by another white circle of 0.1-dva radius. 

Experiments 1 and 2

Stimuli were random noise patches (Gaussian pixel noise) of low or high SFs, bandpass-filtered either from 0.28 to 1.125 cycles per degree of visual angle (cpd) or from 1.125 to 4.5 cpd (see Figure A6 in the Appendix). All noise patches were scaled to an amplitude of 0.5, thus reducing their contrast to 50%. 

Experiments 3 and 4

Stimuli were Gabor patches of varying SF (0.56, 1.125, or 2.25 cpd) and orientation (0°, 45°, 90°, or 135°). In Experiment 3 their contrast was 50%, whereas in Experiment 4 a contrast of 100% was used. Phases were 0° or 180°, so that patches were mid-gray at their center. 

Procedure

Saccade and replay trials were performed in separate sessions on separate days. All trials within a session were presented in a random and interleaved fashion. There was an equal proportion of rightward and leftward saccades and of upward and downward stimulus movement. 

Saccade session

Participants fixated a dot in the left or right half of the screen at 7.4 dva horizontal eccentricity from the screen center. Fixation control was passed after 500 ms of fixation within a 1.3-dva radius around the fixation dot. After 5 seconds without fixation or 25 re-fixations, the trial was aborted and a new calibration requested. The extinction of the fixation dot after successful fixation was the cue to make a saccade to the target stimulus, either to the right or to the left, which was located on the other half of the screen at 14.6-dva horizontal eccentricity. As soon as the saccade was detected (see Online saccade detection section), the target moved vertically upward or downward at high velocities (see below). By pressing either the arrow-up or arrow-down key on the keyboard, participants indicated whether the target stimulus moved up or down. To be able to respond, a participant’s gaze position had to first reach the initial target location (i.e., a circular boundary of a 1.8-dva radius around the initial target area). 

Experiment 1

Noise patch stimuli traveled a distance of 3.6 dva. The duration and speed of the movement were manipulated via the number of frames displayed between the start and end locations of the target: six to 24 frames in steps of two (0.7-ms frame duration), translating to movement durations of 4 to 17 ms and stimulus velocities of 213 to 853 dva/s. As soon as the target stimulus reached its final position, a similar distractor stimulus was displayed at the mirror location above or below the initial target location, so that after the saccade both identical stimuli were located at a 3.6-dva vertical eccentricity from the initial target location. From each participant (N = 15), we collected 10 trials in each experimental condition: session (2) × saccade direction (2) × stimulus movement direction (2) × stimulus movement duration (10) × stimulus SF (2). In subsequent analyses, we collapsed across saccade and stimulus movement directions, so that each experimental cell (session × stimulus movement duration × stimulus SF) contained up to 40 trials per participant. 

Experiment 2

All stimulus and procedure parameters were similar to Experiment 1; however, as soon as the target stimulus reached its final position, a blank screen was introduced for 250 ms before both the target and distractor stimulus reappeared. Similar to Experiment 1, there were 10 trials per experimental condition. 

Experiment 3

Gabor patch stimuli traveled a distance of 3.6 dva for a duration of 11 ms (16 frames) at a corresponding velocity of 320 dva/s. There were eight trials per experimental condition: session (2) × saccade direction (2) × stimulus movement direction (2) × stimulus phase (2) × stimulus SF (3) × stimulus orientation (4). As in the two previous experiments, data were collapsed across saccade and stimulus movement directions. 

Experiment 4

Gabor patch stimuli traveled a distance of 7.1 dva for a duration of 22 ms (32 frames), thus at a velocity equal to that for Experiment 3 (320 dva/s). Similar to Experiment 3, eight trials were run in each experimental condition. 

Replay session

In the replay condition, the fixation dot was at screen center. Just as in the saccade session, fixation control was passed after 500 ms of fixation. Participants were required to remain in the initial fixation area while the target stimulus used in the saccade session moved from the periphery of the screen toward the central fixation point, depending on the direction of the saccade (from the right for leftward saccades or from the left for rightward saccades). To imitate the movement of the stimulus across the retina during a saccade, eye movement data recorded in each trial of the saccade condition was saved, smoothed using a local regression algorithm (LOWESS) with a span of 13, and resampled to 1440 Hz to be replayed at a screen refresh rate during the replay condition. Thus, the saccadic velocity profile, the stimulus characteristics, and the stimulus movement in a specific trial remained largely the same in both saccade and replay conditions. To reduce the temporal uncertainty in the replay condition, the saccade onset was set to a fixed duration after cue onset which was determined by the observer's median saccade latency minus 100 milliseconds, as recorded in the saccade session. Only trials in which fixation control was passed, no wrong key was pressed, and no early responses occurred were included in the replay sessions. 

Online saccade detection

Saccades were detected online using a custom-made velocity-based detection algorithm inspired by the Engbert-Kliegl detection algorithm for microsaccades (Engbert & Mergenthaler, 2006). Eye position was sampled online in all trials. With the onset of the saccade cue, all valid samples collected since the beginning of that fixation period served as input for the algorithm, which was repeatedly executed after every retrieval of a gaze position sample until the saccade was successfully detected. As a first step, eye position data, smoothed by a moving window of a span of five samples, were transformed into a two-dimensional velocity space for x and y coordinates separately. The velocity detection threshold was determined by the median velocity plus the median-based standard deviation multiplied by a factor of 15. To detect a saccade, at least the two most recent samples with a velocity above this threshold had to be registered. As an additional criterion, the direction of eye movement above threshold was computed. A horizontal rightward saccade was only detected when its direction was in the range of 360° ± 25°, whereas a leftward saccade had to be in the range of 180° ± 25°. A more thorough description of the algorithm can be found in Schweitzer and Rolfs (2019). 

With a mean online saccade detection latency (online detection relative to saccade onset detected offline) of 12.5 ms (Experiment 1), 12.5 ms (Experiment 2), 12.3 ms (Experiment 3), and 11.4 ms (Experiment 4), we achieved physical stimulus onsets of ∼27 ms after the onset of the saccade (Figures A1 and A2 in the Appendix). This latency already includes a deterministic video latency of the ProPixx projection system used which occurs because the presentation of the ProPixx projector (unlike a CRT monitor) updates only once the entire signal because the refresh of the graphics card is transferred (the duration of one refresh cycle at 120 frames per second; personal communication, Peter April, June 2018; see also Schweitzer & Rolfs, 2019). 

Data analysis

Preprocessing

Data preprocessing and all subsequent analyses were implemented in R (R Core Team, 2015). Trials were excluded in which participants did not pass fixation control, pressed a response key before reaching the initial target area, or pressed the wrong response key. Subsequently, saccades were detected offline using the velocity-based saccade detection algorithm by Engbert and Mergenthaler (Engbert & Mergenthaler, 2006), using a minimum duration of eight samples (i.e., 16 ms at a sampling rate of 500 Hz) and a threshold parameter value of 5. We excluded trials in which saccades could not be detected or participants made more than two saccades (between saccade cue onset and reaching the saccade target area). Trials with one or two saccades (e.g., in case of a single corrective saccade) were included for analysis if the amplitude of the primary saccade was within ±3.1 dva around the instructed saccade amplitude of 14.6 dva. This decision was based on the sum of the defined experimental target areas (i.e., a 1.3-dva radius around the fixation location and a 1.8-dva radius around the saccade target). As a next step, trials were excluded in which display frames during stimulus movement were dropped and therefore prolonged stimulus durations. Furthermore, we removed all trials in which online saccade detection did not succeed during the relevant saccade or stimulus movement was not finished before saccade offset. 

To compute the physical onset and offset of the stimulus movement, we added the deterministic video latency of the projector of 8.3 ms (mentioned above) to the respective time stamps of the synchronization with the vertical blank (median latency for online saccade detection, 12 ms; median latency for physical stimulus onset relative to online saccade detection, 15 ms; median saccade duration, 62 ms). As it is crucial to determine the exact time of presentation during the saccade, we validated our procedure in a separate experiment using photodiode measurements (Schweitzer & Rolfs, 2019). Averaged across movement durations and participants, stimulus movements finished 24 ms (Experiment 1, SD = 6.7 ms), 25 ms (Experiment 2, SD = 6.6 ms), 29 ms (Experiment 3, SD = 6.7 ms), and 14 ms (Experiment 4, SD = 5.8 ms) before saccade offset. Crucially, these timings depended on both the presented movement duration and saccade duration. Because both saccade duration (M = 62 ms, SD = 10 ms) and physical stimulus onset relative to saccade onset (M = 27 ms, SD = 2.2 ms) were largely similar across conditions (Figures A1 and A2), movement duration reliably predicted the time left until saccade offset (β = –0.97; t = –9.6; 95% CI, –1.17 to –0.78). Moreover, we excluded trials in which offline saccade detection produced unreasonable results, such as when saccade durations were longer or saccadic peak velocities were higher than the participant's individual 97.5% quantile (average cutoff at 90-ms saccade duration and 517 dva/s saccadic peak velocity). In addition, in the replay condition, trials were also excluded if a participant’s gaze did not stay within 2 dva around the initial fixation zone. As a result, around three-quarters of the initial number of trials remained for analysis (see below and Open Methods at OSF, https://osf.io/7q9pr/). Summary statistics for saccadic peak velocity, saccade amplitude, saccade duration, saccade latency, and stimulus onset relative to the onset of the saccade are shown in Figures A1 and A2 in the Appendix. 

Experiment 1

We excluded 26% of all trials in the saccade session and 29% of all trials in the replay session during preprocessing. 

Experiment 2

We excluded 24% of trials in the saccade session and 19% of trials in the replay session during preprocessing. 

Experiment 3

We excluded 29% of saccade trials and 21% of replay trials. 

Experiment 4

In the saccade condition, 28% of trials were excluded. 

Task performance

The sensitivity index (d′) was computed for every observer and condition. Within-subject SEM values were computed based on Cousineau's method (Cousineau, 2005), applying Morey's correction (Morey, 2008). For hypothesis testing, task performance was modeled using linear mixed-effects models (Bates, Mächler, Bolker, & Walker, 2015) in which observers were added as the random factor (intercept only). Additional analyses on performance were conducted using logistic mixed-effects regression on the dichotomous response variable “correct response”. To test the significance of fixed factors, we calculated 95% confidence intervals of estimates via bootstrapping (30,000 repetitions) and conducted hierarchical model comparisons using likelihood ratio tests. 

Experiments 1 and 2

Target matching performance (expressed as d′) was computed for each observer and within-subject conditions session, SF, and movement duration. In the models, movement duration (ms) was included as a continuous variable centered around the median value of the levels (i.e., 10.4 ms), and the factors of session and SF were contrast coded (session, –0.5 = replay, 0.5 = saccade; SF: –0.5 = high, 0.5 = low). 

Experiments 3 and 4

Performance (expressed as d′) was again computed for each observer and the within-subject conditions of session, SF, and relative orientation. In the models, relative orientation was treated as a dummy-coded ordered factor (reference condition was orthogonal) and SF (in cpd) as a continuous variable centered at its median level (i.e., 1.125 cpd), whereas sessions were again contrast coded (–0.5 = replay, 0.5 = saccade). 

Estimating relative orientation

Retinal stimulus positions across time (i.e., stimulus position relative to current gaze position) were computed by subtracting the gaze position data of the participant's dominant eye (resampled to 1440 Hz for the replay session) during stimulus movement from the screen position vector of the stimulus. Subsequently, for each frame of stimulus display, we computed the direction of the moving stimulus across the retina in radians (0, vertical; –π/4, 45° tilt counter-clockwise; π/4, 45° tilt clockwise; ±π/2, horizontal). The angle of retinal trajectory, as displayed in Figure A3 in the Appendix, was the median of all directions for each trial. Relative orientation was computed by subtracting the orientation of the stimulus from the angle of its retinal movement trajectory (see also Figures 3a–3e). As relative orientation would depend on saccade direction and stimulus movement direction, we normalized the angles in a way that values larger than zero represent retinal trajectories steeper than stimulus orientation. 

Reverse regression

In Experiments 1 and 2, a reverse regression analysis was performed to determine which spatial frequencies and orientations drive correct responses. As a first step, similar to previous studies (Li, Barbot, & Carrasco, 2016; Wyart, Nobre, & Summerfield, 2012), we convolved noise patch stimuli with two Gabor filters (in sine phase and cosine phase) of varying orientations (from −π/2 to π/2 in 11 equal steps) and SFs (steps 0.28, 0.38, 0.52, 0.71, 0.96, and 1.31 cpd for low-SF noise patches and 1.31, 1.79, 2.43, 3.31, and 4.5 cpd for high-SF noise patches) and extracted the energy of each SF-orientation component (Figure 3a). In doing so, we obtained a two-dimensional energy map for each individual noise patch, which was normalized for each observer and SF condition (Figure 3b). Importantly, to compute correct filter responses to each orientation, Gabor filters were applied not only to orientations from −π/2 to π/2, but also to their counterparts from −π/2 + π to π/2 + π. This was done because the real and imaginary parts of the filters are computed by sine and cosine, respectively; thus, filter responses for one orientation may be different from that orientation + π, although the orientation of the Gabor is the same (Movellan, 2002). As a consequence, the energy of an orientation in a noise patch was the mean of both of these filter responses. To allow a comparison across trials, orientations were subsequently transformed to relative orientations by subtracting orientation steps from the angle of retinal trajectory in the respective trial (Figures 3c–3e; see also the Estimating relative orientation section). Finally, relative orientation steps were organized in 10 bins with a width of π/10 and a center of 0. As a second step, logistic regressions were applied to predict correct responses from the filter responses (for an example, see Figure A4 in the Appendix). Movement duration was included as an additive continuous predictor to control for its effect on performance. Regressions were run in every experimental condition and component (experiment × session × relative orientation × SF). The log odds estimate of each model served as an indicator of how strongly correct responses were driven by a given component in a given experimental condition. 

Results

In Experiment 1, the target stimulus was a noise patch (Figure 1c), bandpass-filtered to either low or high SF ranges. The target moved either up or down briefly after the onset of the saccade. As soon as the target had reached its final vertical position, we presented a distractor stimulus at the vertical location opposite of the target. Thus, whereas the target stimulus moved continuously, the distractor stimulus appeared when the target stimulus reached its final vertical position, forcing the participant to use intra-saccadic motion streaks (see Figure 1d for a schematic) to pair the post-saccadic target stimulus with the pre-saccadic one. 

To quantify an observer’s ability to link pre- and post-saccadic object locations in this task, we computed the sensitivity index (d′) for each condition and observer. Overall, performance in the saccade condition was low but clearly above chance for low SF, with average d′ = 0.24, t(14) = 4.4, and P = 0.0006; for high SF, d′ = –0.04, t(14) = –1.1, and P = 0.29. To compare conditions, we fitted linear mixed-effects models (Bates et al., 2015) to these indices and computed confidence intervals for the given estimates using parametric bootstrapping (Table A1 in the Appendix for full model specification and results). Performance between saccade and replay conditions differed significantly when collapsed across SF and movement durations (β = –0.38; t = –7.67; 95% CI, –0.48 to –0.28) (Figure 2, upper row), and performance at longer movement durations was significantly lower in the saccade condition compared to the replay condition (β = –0.064; t = –5.1; 95% CI, –0.09 to –0.04), in particular at low SFs (β = –0.05; t = –2.0; 95% CI, –0.098 to –0.001). Although this finding is compatible with the general notion of saccadic suppression (i.e., the reduction of visual sensitivity before or during a saccade), the origin of poor localization performance remains unclear: Were intra-saccadic motion streaks actively suppressed from visual processing, or did post-saccadic masking limit the brief intra-saccadic input's access to conscious awareness? 

Figure 2.

 

Target identification performance in Experiment 1 (top row) and Experiment 2 (bottom row). Mean performance (expressed as d′) across participants (N = 15 in both experiments) as a function of stimulus movement duration in saccade (left column) and replay condition (right column). All error bars represent ±2 within-subject SEM. Movement durations translate directly to movement velocity, as the traveled distance was kept constant at 3.6 dva.

We addressed this question in Experiment 2 (Figure 2, bottom row), in which we introduced a blanking period of 250 ms right after the target movement offset to alleviate post-saccadic masking (Campbell & Wurtz, 1978; Deubel, Schneider, & Bridgeman, 1996; Duyck et al., 2016). In the absence of a static post-saccadic retinal image and an immediate distractor stimulus, overall performance in the saccade condition improved drastically as compared to Experiment 1 (d′_{Exp=1,saccade} = 0.1; d′_{Exp=2,saccade} = 1.2; t(28) = 7.15; P < 0.0001), and was even higher than performance in the corresponding replay condition (β = 0.16; t = 3.1; 95% CI, 0.06–0.27). Although compared to the replay condition performance was again lower for low SF stimuli (β = –0.57; t = –5.4; 95% CI, –0.78 to –0.37), high SF stimuli were more accurately localized during the saccade than during fixation (d′_{SF=high,replay} = 0.1; d′_{SF=high,saccade} = 0.55; t(14) = 3.3; P = 0.005). 

We designed the trans-saccadic identification task specifically such that the target and distractor stimulus could only be distinguished based on the continuous motion streak elicited by the target. More specifically, it is the vertical component of the streak, induced by the movement of the stimulus, that is crucial to correct identification of the target (Figure 1d). Note that, although such intra-saccadic stimulus movements rarely (if ever) occur in natural environments, we chose to use stimuli moving orthogonally to the direction of the saccade to separate the respective contributions of eye and stimulus movement to the generation of the streak. If, indeed, participants used intra-saccadic motion streaks to link object locations across saccades, then their performance in our task must be related to the distinctiveness of that vertical component. This distinctiveness should depend on at least three factors. 

First, as the distance the target traveled was kept constant, shorter movement duration should yield less distinct streaks. Our results support this assumption, as longer movement durations (and, thus, lower retinal speeds) led to higher identification performance in both Experiment 1 (β = 0.03; t = 5.3; 95% CI, 0.02–0.045) and Experiment 2 (β = 0.1; t = 15.0; 95% CI, 0.09–0.11). 

Second, at high retinal velocities (median saccadic peak velocity of 392 dva/s; see Figures A1 and A2 in the Appendix), low SFs yielded higher performance than high SFs (Burr & Ross, 1982). This, too, was borne out by Experiment 1 (β = 0.59; t = 11.9; 95% CI, 0.49–0.69) and Experiment 2 (β = 1.62; t = 30.7; 95% CI, 1.52–1.72). In addition, performance increased with movement duration, in particular for low SF as compared to high SF stimuli in both Experiment 1 (β = 0.056; t = 4.5; 95% CI, 0.03–0.08) and Experiment 2 (β = 0.08; t = 6.3; 95% CI, 0.06–0.11), suggesting that low SFs also produced more distinct motion streaks when high retinal velocities were imposed by saccades. 

Finally, stimuli with an orientation parallel to their movement trajectory result in lower temporal alternations between high and low luminance on the retina; thus, they should produce more pronounced streaks than stimuli with the orthogonal orientation. To explore the contribution of orientation relative to the retinal movement trajectory (henceforth referred to as relative orientation; see Figure 3 and Figures A3 and A5 in the Appendix), we used a reverse regression approach, implemented as a two-step procedure. First, we convolved the noise patch from every trial (Figure 3a) with Gabor filters of varying orientations and SFs (Li et al., 2016; Wyart et al., 2012). Each stimulus could thus be represented by an array of filter responses quantifying the energy of certain SF-orientation components in a given noise patch (Figure 3b). Stimulus orientations were subsequently converted to levels of relative orientation based on the retinal trajectory of each stimulus in each trial (Figures 3c–3e). In a second step, we used logistic regression to predict correct responses from the filter responses in a given component. Specifically, for each combination of experiment, session, relative orientation, and SF, we computed a beta weight describing to what extent that combination drove target identification performance (Figure A4 in the Appendix). 

Figure 3.

 

Impact of SF and relative orientation of a target stimulus on performance in Experiments 1 and 2, established using reverse regression. (a) The individual noise patch of each trial was convolved with a bank of Gabor filters of various orientations (–π/2 to π/2) and SFs (0.28–4.5 cpd), resulting in filter energy maps. (b) Each position in the array represents the energy of a certain oriented SF component in a stimulus. For example, the presented noise patch responds most strongly to a Gabor filter of 0.71 cpd and a tilt of 0.31 rad; ω_stim denotes the orientation component of the stimulus as presented on the screen (0 = vertical). (c) We derived the angle of the retinal trajectory of the target from the eye movement and stimulus movement data for each trial. Here, ω_traj represents the angle of the retinal trajectory. (d, e) Relative orientation (ω_rel) is computed by subtracting the orientations (ω_stim) of the stimulus from the angle of the retinal trajectory (ω_traj). Hence, the orientation space of each stimulus is normalized to its retinal movement trajectory (0 = stimulus orientation is parallel to its retinal movement trajectory). (f, g) Results from the reverse regression analysis of Experiment 1 and 2, respectively. Colors indicate the log odds estimate of logistic regressions fitted to predict correct responses from filter responses. High beta values for an oriented SF component indicate that these components drive correct identifications of post- and pre-saccadic stimuli. The marginal normalized means illustrate average relative orientation and SF tuning for low (red) and high (blue) SF subspaces.

In the replay condition, this reverse regression analysis revealed a clear tuning around a relative orientation of zero. An orientation parallel to the movement trajectory of the stimulus was associated with higher performance, in particular at low SF (Figures 3f and 3g, right panels). In the saccade condition, tuning could also be observed, but it was less pronounced and shifted slightly toward negative relative orientations. The latter suggests that orientations more vertical than the angle of retinal trajectory were discriminated more accurately (Figures 3f and 3g, left panels). In addition, SFs below 0.5 cpd seemed to relate to correct identification less strongly in the saccade condition than in the replay condition. This is compatible with a reduction of contrast sensitivity around the onset of saccades, which is specific to very low SFs (Burr et al., 1982; Volkmann et al., 1978). 

Although the reverse-regression analyses were planned (see pre-registrations), the random nature of noise patches renders this approach quasi-experimental; therefore, we conducted Experiment 3 to test whether the reverse regression results can be confirmed. Experiment 3 was identical to Experiment 1 in all respects, except that we replaced bandpass-filtered noise-patch stimuli with Gabor patches of a fixed number of SFs (0.56, 1.12, and 2.25 cpd) and orientations (horizontal, vertical, 45° clockwise, 45° counterclockwise). Importantly, depending on the horizontal saccade direction and the vertical stimulus movement direction—and given a specified movement duration (11.1 ms) that was optimal for the stimulus orientations used (Figure A3)—these orientations translated directly to a critical set of relative orientations. Stimuli were parallel, oblique, or orthogonal to the retinal trajectory of the stimulus (for an illustration, see Figure A5 in the Appendix). For example, given a rightward saccade, a Gabor with a counterclockwise tilt of 45° would have a parallel relative orientation if it moved upward and an orthogonal relative orientation if it moved downward. 

Note that we devised Experiment 3 as a proof of concept (which we built upon in Experiment 4). As in the previous experiments, observers had to rely on short motion streaks generated during very brief periods of vertical motion (about 17% of the total duration of a saccade, in order to achieve strictly intra-saccadic presentations). Based on Experiment 1 and the fact that the stimulus was much less rich in orientation and SF content, we expected performance to be very low in the saccade condition. Indeed, as in Experiment 1, performance was much lower during saccades than during fixation (Figure 4a). Crucially, however, average target identification performance was significantly higher than chance for all SF levels, provided the Gabor orientation was parallel to its retinal motion trajectory, where d′_parallel = 0.24, t(14) = 3.47, and P = 0.004, but it was at chance for other orientations, such that d′_{not parallel} = 0.05, t(14) = 0.99, and P = 0.34 (Figure 4b). We next fitted linear mixed-effects models across both saccade and replay conditions (see Table A2 in the Appendix for the full model), as well as separately for each condition in a hierarchical model comparison. In the saccade condition, only relative orientation significantly increased log likelihood: ΔLL = +4.62, χ²(2) = 9.24, and P = 0.01; neither SF, where ΔLL = +0.06, χ²(1) = 0.01, and P = 0.91, nor an interaction of both factors, where ΔLL = +1.4, χ²(2) = 2.7, and P = .24, did. In the replay condition, log likelihood increased with relative orientation: ΔLL = +30.7, χ²(2) = 61.4, and P < 0.0001; SF: ΔLL = +8.2, χ²(1) = 16.4, and P < 0.0001; and their interaction: ΔLL = +26.34, χ²(2) = 52.7, and P < 0.0001. This shows that relative orientation mattered in both saccade and replay conditions and confirmed that targets with orientations parallel to the trajectory of the stimulus were identified more accurately. More accurate identification of low SF stimuli was evident only in the replay condition. This could have two explanations. Either performance is indeed independent of SFs during saccades or overall performance was just too low in the saccade condition to uncover any effects of SF. 

Figure 4.

 

Target identification performance in Experiment 3 and 4. (a) Mean performance across participants (N = 15) as a function of relative orientation in the saccade condition (left column) and in the replay condition (right column) of Experiment 3. (b) Comparison of performance for parallel vs. non-parallel stimulus orientations in Experiment 3. Each dot represents one observer in a SF condition. For all SFs in the saccade condition, targets with orientations parallel to the motion trajectory were identified above chance. (c) Mean performance in Experiment 4 (N = 15) with stimuli at full contrast and twice the movement duration. (d) Comparison of performance for parallel vs. non-parallel stimulus orientations in Experiment 4. Performance was significantly higher for parallel stimulus orientations, in particular at low and medium SFs. All error bars represent ±2 within-subject SEM.

In Experiment 4, therefore, we investigated whether performance in the saccade task would improve compared to Experiment 3 if intra-saccadic streaks were more prominent, as they should be during natural vision. To this end, we doubled stimulus movement duration to 22 ms (about 35% of the total duration of a saccade) and increased the traveled distance to 7.1 dva, thus leaving stimulus velocity unaltered. We also increased stimulus contrast to 100%. As a consequence, intra-saccadic motion streaks extended further in time and space while keeping post-saccadic masking intact. Under these conditions, identification performance increased to a level previously found only during fixation or blanking. As in Experiment 3, it was by far the highest when Gabor patches were oriented parallel to their movement trajectory (Figure 4c). Indeed, compared to orthogonal orientations, performance was significantly higher in both oblique (β = 0.46; t = 3.6; 95% CI, 0.21–0.71) and parallel (β = 1.61; t = 12.6; 95% CI, 1.36–1.86) orientations (Table A2 in the Appendix). In addition, performance decreased with higher SFs (β = –0.33; t = –2.67; 95% CI, –0.58 to –0.09). Log-likelihood increased with relative orientation, where ΔLL = +40.4, χ²(2) = 80.8, and P < 0.0001; with SF, where ΔLL = +18.0, χ²(1) = 36.1, and P < 0.0001; and with their interaction, where ΔLL = +15.1, χ²(2) = 30.2, and P < 0.0001. Although stimulus velocity remained unaltered compared to Experiment 3, performance increased markedly with longer streaks and enhanced contrast, demonstrating that relative orientation of a stimulus indeed predicts performance in keeping track of the original stimulus across the saccade by means of intra-saccadic motion streaks. 

Discussion

Saccadic eye movements impose motion blur on the retina. Although this intra-saccadic smear has the potential to provide excellent visual feedback about the movement of the eyes across the visual scene (Watson & Krekelberg, 2009), the resulting intra-saccadic blur is assumed to be suppressed to maintain perceptual stability (Burr et al., 1982; Castet, 2010; Ross et al., 2001). Here we studied intra-saccadic motion streaks induced by single objects moving rapidly across the retina during saccades, as a model example for intra-saccadic smear. We investigated the novel hypothesis that these intra-saccadic signals might serve a purpose in the visual system, facilitating trans-saccadic object localization by linking pre- and post-saccadic retinal object locations. 

Visual processing of intra-saccadic motion smear

There is strong evidence that performance in our experiments relies on the detection of motion streaks rather than motion per se or simply luminance contrast. Geisler (1999) suggested that perception of motion streaks is likely grounded in motion detectors with both orientation and directional tuning (for neurophysiological evidence, see Geisler, Albrecht, Crane, & Stern, 2001) and therefore in motion and contrast mechanisms, provided that movement speeds are high enough to induce streaks and stimuli have sufficient contrast to allow the orientation-selective units to resolve the motion streak (Edwards & Crane, 2007). The key difference between previous studies on motion streaks and ours is the speed of the stimuli used. Whereas previous studies presented stimulus speeds of 1 to 24 dva/s (e.g., Apthorp & Alais, 2009; Edwards & Crane, 2007; Geisler, 1999), our stimuli induced retinal speeds (i.e., the vector sum of stimulus and eye movement speed) in the range of 300 to 1000 dva/s in both saccade and replay session. 

That these speeds inevitably produce motion streaks is compatible not only with the phenomenological appearance of the stimuli (i.e., a blurred trace that is readily detectable when blanking is applied) but also with previous literature showing that contrast and motion perception alone at these high velocities is difficult or even impossible. Indeed, Burr & Ross (1982) showed that contrast sensitivity to a grating drifting at only 100 dva/s decreased by almost one log unit when its spatial frequency was increased from 0.1 to 0.7 cpd, and that gratings drifting at 800 dva/s could only be detected if their SF was below 0.1 cpd. Moreover, Schweitzer and Rolfs (2019) found that Gabor patches (0.5 cpd and 0.5 dva SD Gaussian aperture) rapidly drifting within their aperture (thus not producing any motion streaks) became impossible to detect during fixation when drift velocities came close to or exceeded saccadic peak velocities (∼400 dva/s). Finally, Castet et al. (2002) presented striking evidence that intra-saccadic gratings (with orthogonal orientation to the saccade direction, saccadic peak velocities around 280 dva/s) induced motion percepts if their SFs were 0.04 and 0.18 cpd, but not if their SF was 1.81 cpd, because the latter induced temporal frequencies of ∼500 Hz, which cannot be resolved by motion detectors. 

Note that the high-SF stimuli applied in Experiments 1 and 2 (on average 2.25 cpd) could thus induce temporal frequencies from 675 to 2250 Hz on the retina when being moved at the speeds we used, which parsimoniously explains low performance in all high-SF conditions (including the replay conditions). Further support is provided by Experiments 3 and 4: If motion detection were the major predictor for performance in our paradigms, then we would expect that task performance would be best for Gabor stimuli (especially at low SF) if their orientation were orthogonal to their motion trajectory, as these orientations would optimally activate motion detectors (Adelson & Bergen, 1985; Reichardt, 1987). However, the contrary was the case. In both saccade and replay conditions, performance for orthogonal stimuli was rarely above chance level; instead, stimuli with orientations matching their retinal movement trajectory led to maximum task performance. These stimuli allowed for effective temporal summation; that is, they generated a streak. 

This effect is consistent with the motion streak literature. Masks oriented parallel to the trajectory of a stimulus masked motion direction more efficiently than orthogonal masks (Burr & Ross, 2002; Geisler, 1999), and neurons in monkey V1 showed greater activation parallel to the direction of motion when motion was sufficiently fast to produce a streak (Geisler et al., 2001). These striking similarities suggest a specific role of motion streaks in our study that cannot be accounted for by contrast and motion detection mechanisms alone. 

Our findings also demonstrate that intra-saccadic smear is not removed from visual processing but can lead to identification performance comparable to that found during fixation, as demonstrated by the blanking conditions used in Experiment 2. In fact, observers consistently reported a clear and vivid impression of a blurred trajectory, which was likely to drive localization performance in the blanking condition. Blanking is known to relieve saccadic suppression of displacement (Deubel et al., 1996), but given that stimulus motion was orthogonal (and not parallel) to the saccade direction and covered distances of more than 3 dva, observers had no trouble identifying the trans-saccadic stimulus displacements (Wexler & Collins, 2014). Instead, we assume that blanking alleviated post-saccadic masking. This assumption is compatible with results indicating that the prolonged presence of intra-saccadic stimuli beyond the end of the eye movement significantly reduces the amount and length of perceived smear (Balsdon, Schweitzer, Watson, & Rolfs, 2018; Bedell & Yang, 2001; Duyck et al., 2016; Matin et al., 1972). Although temporal masking is often attributed to stimuli covering the entire display (Castet et al., 2002) or even the entire visual field (Campbell & Wurtz, 1978), it is possible to achieve similar effects with just simple flashes of light without a necessary spatial or retinal overlap, an effect attributed to meta-contrast masking (Matin et al., 1972). Interestingly, this mechanism has been shown to be equally effective both during real and simulated saccades (Brooks et al., 1980; Brooks et al., 1981). The absence of post-saccadic masking, however, is not representative for the circumstances of natural vision, where every intra-saccadic input is followed (and thus masked) by a more reliable and stable retinal image (Castet, 2010). Blanking therefore provided an estimate for an upper bound of identification performance in the saccade condition. 

Remarkably, participants also achieved a similarly high level of performance during saccades even when post-saccadic masking was intact; provided the motion streaks extended over a large share of the saccade duration, the inducing stimuli had high contrast and orientations parallel to their retinal trajectory. In particular, using a motion streak duration of 22 ms in Experiment 4—covering just over a third of the saccade duration—performance drastically improved compared to Experiment 3, where stimuli moved across the retina at the same velocity and angle as in Experiment 4 but for only half the time (11 ms). As these experimental conditions are still quite specific and hardly comparable to vision under natural viewing conditions, one can at this point only speculate about whether motion streaks in natural scenes—where input is rich in SF, orientation, and color content, and motion streaks are present throughout the entire saccade—would enable reliable trans-saccadic identification. Further research will be needed to investigate how our results with motion streaks induced by single objects generalize to more complex large-field intra-saccadic smear: By moving entire visual scenes during saccades to increase, reduce, or eliminate induced smear, it would be possible to systematically investigate how intra-saccadic large-field smear impacts trans-saccadic object identification in more natural visual configurations. 

Task performance during saccades and during fixation

With medium contrast stimuli, brief movement durations and post-saccadic masking intact (Experiments 1 and 3), target identification performance in the saccade condition was relatively poor compared to the replay condition. This was expected based on the short motion streak distinguishing the target from the distractor stimulus and was consistent with a number of established phenomena. First, the pre- and post-saccadic images might have acted as forward and backward masks, respectively. During real (as opposed to simulated) saccades, the entire visual field moves across the retina, causing the brief, feeble intra-saccadic input to be temporally surrounded by two powerful, high-intensity masks (Castet et al., 2002; Castet, 2010). This view is compatible with the explanation that the prolonged presence of intra-saccadic stimuli after saccade offset might have acted as a meta-contrast mask reducing perceived smear (Breitmeyer & Ganz, 1976; Campbell & Wurtz, 1978; Duyck et al., 2016; Matin et al., 1972). 

Although the experiment was conducted in a dimly lit room, the screen border and immediate surroundings may have also contributed to the difference in saccade and replay conditions. For example, false screen borders (i.e., peripheral backgrounds surrounding a uniform presentation area) increased detection thresholds of peri-saccadic stimuli both during saccades and when moved at saccade-like speeds during fixation (Idrees, Baumann, Franke, Muench, & Hafed, 2019). These results suggest that peripheral large-field motion without overlap with the relevant stimulus can decrease visual performance during saccades. In our paradigm, the saccade condition caused such large-field motion, whereas the replay condition did not, even though the stimulus trajectories on the retina were similar. Second, contrast sensitivity decreases around the onset of the saccade, a phenomenon known as saccadic suppression (Burr et al., 1982). Although this effect is widely interpreted as a mechanism of active visual suppression to maintain perceptual stability (Burr et al., 1994; Ross et al., 2001), it was recently argued that this threshold elevation could be explained by signal-dependent noise introduced during saccade execution that subsequently leads to the down-weighting of peri-saccadic visual information (Crevecoeur & Körding, 2017). For example, using a blank screen, previous studies have observed an elevation of contrast threshold during real saccades (up to 50 ms after saccade offset), but not during simulated saccades (Diamond, Ross, & Morrone, 2000). 

Alternatively, this effect could also be interpreted in terms of strictly visual factors, as there is evidence that a threshold elevation can occur already on the retinal level, such as in isolated mouse and pig retinae presented with saccade-like image displacements (Idrees et al., 2019). Finally, the sudden onset of the distractor stimulus right after completed target motion might have caused an attentional distraction from the brief intra-saccadic streak (Balsdon et al., 2018). Each of these effects might have also contributed to the difference between saccade and replay conditions. Moreover, saccades lead to other changes, such as attention shifts (Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011) or changes in perceptual tuning (Li et al., 2016; Ohl, Kuper, & Rolfs, 2017), that cannot be controlled for in a simulated saccade condition. 

Distinct motion streaks are associated with high task performance

Across all experimental conditions, we found that low SFs were identified more accurately than high SFs. Due to the high retinal velocities resulting from combined rapid eye and stimulus movement, high SF stimuli are rendered invisible, whereas low SFs—which are predominant in smeared visual scenes—remain resolvable (Burr, 1981; Burr & Ross, 1982). One interesting exception was the result that high SFs were more reliably identified during saccades than during fixation when the blanking period was introduced in Experiment 2. We speculate that this result is a consequence of well-established effects: During the preparation of a saccade, visuospatial attention selected the target of the movement (Deubel & Schneider, 1996; Kowler, Anderson, Dosher, & Blaser, 1995; Ohl et al., 2017; Rolfs & Carrasco, 2012), increasing visual sensitivity for high SFs in an obligatory fashion (Li et al., 2016; Li, Pan, & Carrasco, 2019). In the context of our task, this increase in sensitivity at the saccade target could have led to two complementary phenomena. On the one hand, masking of high-SF stimuli may have been more efficient when targets were present upon saccade landing, which was the case in Experiments 1, 3, and 4. On the other hand, performance for high-SF stimuli may have been enhanced when post-saccadic masking was alleviated by the blanking condition. These effects could well explain the enhanced performance for high-SF intra-saccadic stimuli in Experiment 2. 

In most conditions, however, correct target identifications were strikingly more prominent for low-SF stimuli. In classic studies of saccadic suppression, however, the strongest reductions in contrast sensitivity were found at low SFs (Burr et al., 1982; Volkmann et al., 1978). This result has been taken to suggest that the visual system selectively removes SF content that dominates intra-saccadic stimulation to preserve visual stability, possibly by raising thresholds specifically in the magnocellular pathway (Burr et al., 1994; Ross et al., 2001). Indeed, it has been suggested that a modulation of visual processing around saccades occurs as early as in the lateral geniculate nucleus (Sylvester, Haynes, & Rees, 2005; Thilo, Santoro, Walsh, & Blakemore, 2004). Although the lowest SFs used in our experiments (0.28 cpd in Experiments 1 and 2; 0.56 cpd in Experiments 3 and 4) were still a tenfold higher than the lowest SFs used in studies of saccadic suppression, (i.e., 0.02–0.04 cpd) (e.g., Burr et al., 1982; Burr et al., 1994; Diamond et al., 2000), we found that low-SF stimuli could be identified most accurately. In contrast to studies of saccadic suppression, however, we presented stimuli at medium to high contrasts rather than at threshold. Indeed, recent evidence shows that saccades reformat the visual input signal to emphasize post-saccadic low-SF information (Boi et al., 2017). Thus, intra-saccadic visual signals are not eliminated from visual processing and could constitute a valuable source for trans-saccadic object identification. 

Most previous studies on intra-saccadic motion streaks used oscilloscopes (e.g., Bedell & Yang, 2001; Brooks et al., 1980; Brooks et al., 1981) or light-emitting diodes (e.g., Duyck et al., 2016), allowing for limited control over the stimulus. The application of noise and Gabor patches made it possible to study observers’ performance as a function of streak distinctiveness while controlling for contrast, SF, and duration. Exploratory reverse regression suggested an effect of relative orientation; that is, orientations parallel to the retinal motion direction elicit stronger motion streaks, probably due to effective signal summation (Burr, 1981). Experiments 3 and 4 confirmed this hypothesis by showing that Gabor stimuli were identified more accurately when their orientation matched their retinal trajectory. Although this result provides strong evidence that the features of the target (and its resulting intra-saccadic motion streak) drove performance in our task, one might argue that the transient of the distractor onset contributed to observers’ decisions. Several points speak against this assumption: First, in the non-blanking experiments, both target and distractor were always displayed at their final locations at the same time, so that temporal offset could not be used as a cue. Second, even if the onset of the distractor did inform responses along with the target, the efficiency of the streak of the target must have nevertheless been processed for it to be used as a contrast in this two-alternative forced-choice task. Third, we have shown previously that transients of irrelevant distractors can impair direction judgments of intra-saccadic stimulus movement (Balsdon et al., 2018), suggesting that if any effect was present then the onset of the distractor in our paradigm would have had a negative effect on task performance. 

Ecological validity of the paradigm

Given our setup and the nature of the task, the effect of relative orientation had to be investigated in a sparse and artificial setting that does not necessarily represent visual processes under natural circumstances. We speculate, however, that even in natural vision, where the amount of visual information is undoubtedly denser and more cluttered, motion streaks may play a role when saccade directions match stimulus orientations available in the scene. Indeed, both saccades (Najemnik & Geisler, 2008; Otero-Millan, Macknik, Langston, & Martinez-Conde, 2013) and the distribution of orientations in natural scenes (Coppola, Purves, McCoy, & Purves, 1998; Torralba & Oliva, 2003) have strong biases for the cardinal directions, and thus might promote intra-saccadic motion streaks during active visual behavior. In the case of cardinal directions, a large proportion of signals would be parallel and orthogonal to the saccade direction. In fact, a possible neuronal sensor for inferring direction from motion streaks was proposed as a combination of an oriented, but not direction-selective, cell that is sensitive to orientation parallel to the saccade direction and a perpendicularly oriented direction-selective cell that is sensitive to motion signals orthogonal to the saccade direction (Geisler, 1999). Future research will have to determine whether our findings from a sparse laboratory environment translate to natural vision. 

Crucially, we devised our paradigm as a proof of concept to test whether human observers are, in principle, capable of using high-speed stimulus movement, which induces motion streaks during saccades, to identify the original, pre-saccadic target stimulus. As observers judged motion streaks elicited by combined eye and stimulus movement in a sparse visual display, the paradigm's ecological validity may be limited. For example, rapid object movement strictly during saccades will rarely occur in the natural world, so that motion streaks will almost exclusively be induced by eye movements over stable visual scenes. In fact, unlike the experiments reported here, most previous studies on intra-saccadic motion streaks displayed static sources of light and studied the perceived saccade-dependent smear (e.g., Bedell & Yang, 2001; Brooks et al., 1981; Duyck et al., 2016; Matin et al., 1972) and were therefore unable to critically test the hypothesis whether induced smear could be used as a cue to matching pre- and post-saccadic targets. 

Moreover, natural scenes may not always contain demarcated objects like the target stimuli applied here, so intra-saccadic motion streaks in cluttered scenes are likely to be less salient in these cases. Indeed, we showed recently that even onsets and offsets of spatially distant distractor stimuli around the offset of saccades can impair perceptual performance in detecting intra-saccadic target movement (Balsdon et al., 2018). Based on the results presented here, we cannot claim that the role of motion streaks in tracking objects is specific to trans-saccadic vision. Indeed, it would be fascinating if the same visual cues would be used to link object locations both during fixation and across saccades. 

Implications for understanding visual stability

When we make saccades to scan a visual scene, objects constantly change locations on the retina and thus produce smear in the process. To achieve visual stability, the visual system must not only keep track of these retinal locations but also deal with the retinal consequences of the eye movement. Processing of intra-saccadic motion streaks could potentially serve both tasks. First, there would be no need for active removal of intra-saccadic smear from visual processing, as long as an elevation of threshold would prevent these signals from reaching conscious awareness (Watson & Krekelberg, 2009), such as, for example, via pre- and post-saccadic masking (Castet, 2010). In the paradigm described here, we did not assess whether observers were aware of a motion streak but instructed them to indicate the original pre-saccadic stimulus. In all experiments, except for experiment 2, observers generally reported low confidence and almost never “seeing” a motion streak, suggesting that conscious awareness might not have been a necessary condition for correct responses in this task. Second, motion streaks might facilitate the establishment of object correspondence in terms of smooth spatiotemporal continuity (Kahneman, Treisman, & Gibbs, 1992; Mitroff & Alvarez, 2007), as well as surface features, such as color (Hollingworth et al., 2008; Richard, Luck, & Hollingworth, 2008), which is thought to be less affected (Burr et al., 1994; Diamond et al., 2000; Knöll, Binda, Morrone, & Bremmer, 2011), albeit not unimpaired by mechanisms of saccadic suppression (Braun, Schütz, & Gegenfurtner, 2017). In addition, it might be an interesting perspective that intra-saccadic motion streaks could contribute to object localization across saccades, when the eyes fail to hit their target (Collins, Rolfs, Deubel, & Cavanagh, 2009). To test whether motion streaks really aid the establishment of object correspondence across saccades in a normal visual environment, future experiments may use implicit measures (e.g., corrective saccades to displaced stimuli) and study objects in more complex visual scenes (e.g., McConkie & Currie, 1996). 

We conclude that intra-saccadic motion streaks routinely produced by objects in the visual field are not removed from processing. Although their function (if there is any) in natural vision is still unclear, our results provide a proof of concept that (1) observers can identify objects across saccades based only on image smear and (2) identification performance increased with the distinctness and duration of the resulting motion streaks. Identification of objects based on motion streaks could potentially constitute a previously undiscovered contribution to visual stability, complementing pre- and post-saccadic landmarks (Deubel, 2004), efference copies (Collins et al., 2009), or attentional remapping (Rolfs et al., 2011). High-speed projection systems for accurate stimulus display provide opportunities for future research to investigate the potential contribution of visual feedback during saccades to the perception of stability across these rapid eye movements. 

Conclusions

Saccades are the fastest and most frequent human movements. By relocating the fovea (the retina's receptor-packed center), they provide rapid access to high-acuity vision across the entire visual scene. Each saccade, however, shifts objects in the scene to new parts of the retina. How does the brain keep track of these objects to maintain perceptual continuity? Using a novel trans-saccadic identification task, we show that intra-saccadic motion streaks—arising from slow integration of retinal signals—can in principle be used as cues to linking object locations across saccades. These results challenge the long-standing assumption that intra-saccadic motion signals are eliminated from visual processing to reduce saccade-induced blurring of sensory information. We have yet a long way to go in understanding the potential function of intra-saccadic vision. The current results suggest that perception of intra-saccadic object motion could potentially constitute a parsimonious contribution to perceptual continuity in active vision. 

Acknowledgments

We thank Jan-Niklas Klanke, Clara Kuper, Polina Arbuzova, Luke Pendergrass, Julius Krumbiegel, and Olga Shurygina for their help with running the experiments, Sven Ohl for his valuable comments on reverse regression and linear mixed-effects modeling, and Patrick Cavanagh for valuable feedback on a draft of the manuscript. 

RS was supported by the Studienstiftung des deutschen Volkes and the Berlin School of Mind and Brain. MR was supported by the Deutsche Forschungsgemeinschaft (DFG, grants RO3579/2-1, RO3579/8-1 and RO3579/10-1). The DFG and the Open Access Publication Fund of Humboldt-Universität zu Berlin supported the publication of this work. 

Commercial relationships: none. 

Corresponding author: Richard Schweitzer. 

Email: [e-mail]. 

Address: Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany. 

Appendix

  

Table A1.

 

Model summary for linear mixed-effects regressions in Experiments 1 and 2.

 

Table A2.

 

Model summary for linear mixed-effects regressions in Experiments 3 and 4.

 

  

Figure A1.

 

Summary statistics for saccade parameters in Experiments 1 and 2. Each data point represents the average across participants for the experimental conditions of movement duration and SF. All error bars represent ±2 within-subject SEM. To check for differences in saccade parameters across conditions, repeated-measures ANOVAs were run for each experiment dependent variable. Significance levels were Bonferroni-corrected for the number of ANOVAs (5 parameters × 4 experiments), resulting in a significance level of 0.0025. Saccade amplitudes were larger for high SF stimuli in both Experiment 1—F(1,14) = 16.8, η² = 0.019, and P = 0.001—and Experiment 2—F(1,14) = 21.8, η² = 0.012, and P = 0.0004. Furthermore, stimulus onsets were earlier for the shortest movement duration in Experiment 2: F(9,126) = 4.55, η² = 0.075, and P < 0.0001. No other effects were significant.

Figure A2.

 

Summary statistics for saccade parameters in Experiments 3 and 4. Each data point represents the average across participants for the experimental conditions of SF and relative orientation. All error bars represent ±2 within-subject SEM. Repeated-measures ANOVAs (corrected α = 0.0025) revealed that saccade amplitudes were again larger in high SF conditions: F(2,28) = 16.6, η² = .033, and P < 0.0001. No other effects were significant.

Figure A3.

 

Distributions of absolute angle of retinal trajectory across both sessions of each experiment. Movement durations are coded by color. In Experiments 1 and 2 (left columns), short movement durations and high stimulus velocities led to steeper retinal trajectories (0, vertical trajectory), whereas longer movement durations and lower stimulus speeds resulted in more flat trajectories (±π/2, horizontal trajectory). In Experiment 3 and 4 (right columns), we aimed at producing absolute retinal trajectories around 45° to achieve optimal relative orientations.

Figure A4.

 

Exemplary results from the reverse regression across relative orientation components in the replay condition of Experiment 2 for one stimulus condition (SF = 0.71).

Figure A5.

 

Illustration of distinctiveness of motion streaks for vertical, tilted by 45°, and horizontal Gabor patches (top to bottom, left column). Assuming a rightward horizontal movement trajectory, the relative orientation of the Gabor patches would be orthogonal, oblique, or parallel, respectively. To illustrate retinal smear, an image motion filter with the length of the displayed arrow was used (right column).

Figure A6.

 

Spatial frequency power spectra of the low-SF (orange) and high-SF (blue) noise patches used in Experiments 1 and 2. A two-dimensional fast Fourier transform was performed on the bandpass-filtered noise, and the radial average of the result was computed. Power dropped steeply around the specified high-pass and low-pass cut-off values of 0.28 and 1.12 cpd for low-SF stimuli and 1.12 and 4.5 cpd for high-SF stimuli (dotted vertical lines).