Transsaccadic integration of visual information is predictive, attention-based, and spatially precise

Abstract

Eye movements produce shifts in the positions of objects in the retinal image, but observers are able to integrate these shifting retinal images into a coherent representation of visual space. This ability is thought to be mediated by attention-dependent saccade-related neural activity that is used by the visual system to anticipate the retinal consequences of impending eye movements. Previous investigations of the perceptual consequences of this predictive activity typically infer attentional allocation using indirect measures such as accuracy or reaction time. Here, we investigated the perceptual consequences of saccades using an objective measure of attentional allocation, reverse correlation. Human observers executed a saccade while monitoring a flickering target object flanked by flickering distractors and reported whether the average luminance of the target was lighter or darker than the background. Successful task performance required subjects to integrate visual information across the saccade. A reverse correlation analysis yielded a spatiotemporal “psychophysical kernel” characterizing how different parts of the stimulus contributed to the luminance decision throughout each trial. Just before the saccade, observers integrated luminance information from a distractor located at the post-saccadic retinal position of the target, indicating a predictive perceptual updating of the target. Observers did not integrate information from distractors placed in alternative locations, even when they were nearer to the target object. We also observed simultaneous predictive perceptual updating for two spatially distinct targets. These findings suggest both that shifting neural representations mediate the coherent representation of visual space, and that these shifts have significant consequences for transsaccadic perception.

Introduction

Our eyes are constantly moving. We observe the world via a continuous sequence of brief glances punctuated by rapid eye movements called saccades. While the positions of objects generally remain stable in world-based (spatiotopic) coordinates, every saccade produces a shift in each object’s retinal position. Consider the example time course of a planned saccade in Figure 1. Figure 1A depicts a desk that houses a number of objects, namely a mug, stapler, and pencil. An observer fixates the desk at the position of the cross and plans a saccade to the circle, indicated by the dashed arrow. We denote the spatiotopic positions of the three objects according to their presaccadic positions with respect to the direction of the impending saccade. In this example, the mug to the immediate left of the fixation point is the target of our visual attention and, accordingly, we refer to its spatiotopic position as the target location (denoted by the green circle). The stapler is located in the direction congruent with the impending saccade (the saccade-congruent location, red circle), while the pencil is located in the direction opposite the impending saccade (the saccade-incongruent location, blue circle). While the observer maintains fixation, the objects in the scene fall within the receptive fields of unique populations of neurons. As depicted in Figure 1B, the spatiotopic positions of the objects do not change after the observer completes the saccade (note the unchanged color scheme). However, the retinal coordinates of each object have changed to reflect their new positions on the retina. The mug now falls within the receptive fields of neurons that previously corresponded to the saccade-incongruent location, while the stapler falls within the receptive fields of neurons that corresponded to the target location. 

Figure 1.

 

Remapping and the hypothesized consequences of saccade-related neural activity for an observer’s perception of an attended target. (A) A portion of the visual field preceding a planned saccade. The black cross represents the current fixation location and the black dot represents the saccade endpoint, while the dashed arrow indicates the direction and magnitude of the planned saccade. The colored circles represent fixed locations in world coordinates (green: target location, red: saccade-congruent location, blue: saccade-incongruent location). In this example, the observer maintains attention at the mug located at the target location across the saccade. (B) The same portion of the visual field after completion of the saccade. The attended target now falls in the part of the visual field previously associated with the saccade-incongruent location, while the portion of the visual field previously associated with the target now contains the contents of the saccade-congruent location. (C-F) Perceptual consequences of several proposed forms of saccade-related neural activity. Plotted curves (left column) represent the contribution of information from different spatial locations over time, while the image patches (right column) represent the observer’s perception of the target immediately before and after the saccade. The shaded gray regions represent the duration of the saccade itself, during which little visual information is available (Krekelberg, 2010). (C) If the visual system somehow compensates for all perisaccadic changes in neural position tuning (i.e., due to predictive remapping, attentional updating, or converging receptive fields), then only information from the target location will contribute to its appearance and the perceived target will be undistorted. (D) Under predictive attentional updating (with or without predictive remapping), visual features from the future retinotopic location of the target (the saccade-incongruent location; blue) may contribute to the target’s perceptual appearance. (E) If predictive attentional updating is not completed prior to the completion of the saccade, visual features from the presaccadic retinotopic location of the target will contribute to the observer’s perception of the target once the saccade has been completed (the saccade-congruent location; red). (F) Under the converging receptive fields hypothesis, or under predictive remapping without predictive attentional updating, visual features from locations nearest the saccade endpoint (saccade-congruent; red) may contribute to the target’s perceptual appearance.

How we perceive the world as stable despite the visual system receiving different retinal images before and after a saccade is a fundamental problem in vision that has been the subject of behavioral and physiological investigations for more than a century. A longstanding hypothesis has been that the visual system makes use of a corollary discharge of the motor signals driving an eye movement to predict its perceptual consequences and anticipate the new retinal positions of visual objects following the eye movement (von Helmholtz, 1924; Sperry, 1950). 

Based on physiological investigations over the last 30 years, two distinct neural mechanisms have been proposed that support this hypothesis. The first, called predictive remapping, proposes that the visual system compensates for an impending saccade by “remapping” or shifting the retinotopic position tuning of visual neurons before execution of the saccade. Early recordings of visually responsive neurons in the lateral intraparietal cortex (LIP) found that roughly 80 to 100 ms before the onset of a saccade, the spatiotopic location of a probe that caused an increase in firing rate for a given neuron shifted to the location that corresponded to the neuron’s post-saccadic retinal position, as if the neuron’s receptive field had “remapped”, updating the position tuning of its receptive field to anticipate the predicted consequences of the impending saccade (Duhamel et al., 1992; Gottlieb et al., 1998; Heiser & Colby, 2006; Kusunoki & Goldberg, 2003). Evidence of this phenomenon has since been observed in retinotopic areas throughout the occipital and parietal cortices (Merriam et al., 2007; Nakamura & Colby, 2002; Neupane et al., 2016). Predictive remapping is thought to be triggered by activity in a visuo-motor circuit that specifies a corollary discharge of the impending saccade, encompassing neurons in the superior colliculus (SC), frontal eye fields (FEF), and LIP (Sommer & Wurtz, 2002, 2004, 2006; Umeno & Goldberg, 1997). 

The second mechanism, called predictive attentional updating, pertains to scenarios in which covert attention is directed to a region of visual space in the peripheral visual field. Covert attention modulates the response of neurons that possess receptive fields overlapping the attended location, which is thought to cause improved performance on behavioral tasks such as faster reaction times and higher accuracy; for a review, see Carrasco (2011). In order for an observer to correctly maintain attention at a specific location across a saccade, the visual system must update the neuronal population that receives attentional modulation (the “attentional state”) between those neurons with receptive fields overlapping the attended location before and after the saccade. Predictive attentional updating proposes that the visual system solves this problem by updating the relevant neurons’ attentional state before the saccade occurs (Bergelt & Hamker, 2019; Cavanagh et al., 2010; Golomb, 2019; Marino & Mazer, 2016; Rolfs et al., 2011). It has been proposed that physiological data that have previously been taken as evidence of predictive remapping should instead be re-interpreted through the lens of predictive attentional updating, as many recordings showing predictive neural activity come from brain areas that are known to be modulated by attentional priority, such as LIP (Mirpour & Bisley, 2012, 2016). More recent studies have found empirical support for predictive attentional updating (independent of position tuning) in area V4 (Marino & Mazer, 2018). 

It is important to delineate the differences between predictive attentional updating and predictive remapping. While it is possible that these two mechanisms occur simultaneously during natural viewing, the differences between the two are best highlighted by considering the effects of each mechanism on its own. Predictive attentional updating proposes that the retinotopy of neurons stays fixed before a saccade, while the attentional state of neurons is shifted to reflect the post-saccadic location of the attentional focus. Conversely, during predictive remapping the attentional state of neurons remains fixed before the saccade, but the spatial selectivity of individual neurons shift to reflect their post-saccadic receptive field location. 

More recently, studies have proposed a third neural mechanism that calls into question the predictive nature of perisaccadic remapping. Recordings of the receptive field structure of neurons in the frontal eye fields indicate that instead of predictively remapping their receptive fields toward the corresponding postsaccadic retinotopic locations, neurons become highly sensitive to visual information around the endpoint of the impending saccade (Zirnsak et al., 2014). This has lead to an alternative hypothesis that in the face of an impending saccade, visual neurons do not adjust to anticipate for the predicted shift of the retinal image. Instead, the apparent shifts result from an attentional modulation that privileges responses from the neuronal population surrounding the target of the eye movement, with the result that visual receptive fields effectively collapse toward the retinotopic location of the saccadic endpoint (Neupane et al., 2016; Tolias et al., 2001; Zirnsak & Moore, 2014). Theoretical and computational modelling efforts have outlined how an attentional gain signal centered on the saccadic target can cause this convergence of receptive fields toward the saccadic endpoint (Hamker et al., 2008; Hartmann et al., 2017; Zirnsak et al., 2010). 

It remains an outstanding area of research to characterize whether and how these three neural mechanisms may influence perception. Consider the case where an observer prepares a saccade as in Figure 1A-B while covertly attending a peripheral stimulus, in this case the mug (denoted as the target by the green circle). Depending on which mechanisms are active, there are at least four distinct perceptual outcomes that the observer may experience: 

• (1) If the visual system somehow compensates for predictive neural activity, then perception of the attended stimulus should not be affected by the planned saccade, regardless of which mechanism mediates the update. We depict this scenario in Figure 1C. On the left hand side is the time course of perceptual integration across the example saccade sequence in Figure 1A-B. The horizontal axis represents the time relative to the saccade onset, and the vertical axis represents the contribution of each stimulus location to the observer’s perception of the target location (the color scheme is the same as in Figure 1A). Before and after the saccade, only the target location contributes to the observer’s perception of the target, which leads to a veridical perception of the mug as depicted in the right side of Figure 1C. Psychophysical evidence consistent with this possibility has been reported by Yao et al. (2016).
• (2) If the visual system does not completely compensate for predictive neural activity, then perisaccadic perception of the target may be influenced by visual stimuli present at the post-saccadic retinal location of the target. We term this possible outcome predictive perceptual updating. Prior to the execution of a saccade, stimuli located at the target location as well as the saccade-incongruent location can contribute to the appearance of the attended target. This outcome may arise from predictive attentional updating, where neurons with receptive fields that will overlap the target after the saccade has been completed receive attentional modulation prior to execution of the saccade. The time course of this scenario is depicted in the left side of Figure 1D. Only the target location contributes to the observer’s perception of the target both well before and after the saccade. However, roughly 100 ms before the execution of the saccade, predictive neural activity causes the object at the saccade-incongruent location to contribute to the observer’s perception of the target (compare the green and blue curves). This leads the observer to perceive the stimulus at the saccade-incongruent location to be located at the target location (right side of Figure 1D). Numerous psychophysical investigations have observed patterns of data consistent with this scenario (Arkesteijn et al., 2019; Hunt & Cavanagh, 2011; Jonikaitis et al., 2013; Rolfs et al., 2011; Szinte et al., 2016, 2018).
• (3) If the visual system predictively updates the locus of attention but does not complete the updating before the completion of the saccade, then those neurons with receptive fields that overlapped the target stimulus before the saccade may remain “attended” after completion of the saccade. In this case, the saccade-congruent location will contribute to the observer’s perception of the target after the saccade has been completed (note the red curve on the left side of Figure 1E), leading to an inaccurate perception of the target (Figure 1E, right side). Recent studies have observed patterns of data consistent with this phenomenon, which has been dubbed the “retinotopic attentional trace” (Golomb et al., 2008, 2010A, 2010B; Golomb 2019).
• (4) Finally, transsaccadic perception may be an integration of the percept of a currently attended target with visual information that is in the direction of the impending saccade; we will refer to this phenomenon as forward perceptual updating. This instance is depicted in Figure 1F. In this case, perception of the target will be influenced by a stimulus located at the saccade-congruent location. Two different mechanisms could give rise to this pattern. If perception of a target is determined exclusively by which neurons receive attentional modulation, then predictive remapping (in the absence of predictive attentional updating) would lead to this result because the (attended) neurons with receptive fields that originally correspond to the target location perisaccadically shift to correspond to the saccade-congruent distractor. This pattern is also predicted if visual receptive fields do not predictively remap but instead collapse towards the planned endpoint of the impending saccade because the saccade-congruent stimulus is the closest visual object to the saccadic goal.

The current study

Numerous empirical studies have found evidence that saccade-related neural activity has perisaccadic perceptual consequences. Most of these studies have used a variant of a standard psychophysical attentional capture paradigm (e.g., Golomb et al., 2008, 2010A, 2010B; Hunt & Cavanagh, 2011; Jonikaitis et al., 2013; Rolfs et al., 2011; Szinte et al., 2016, 2018). Observers are tasked with monitoring stimulus displays in anticipation of a briefly flashed stimulus, or probe, while making a saccade. The observer makes a judgment about the probe (e.g., report the stimulus orientation) and the probe is moved around the display to test different locations. Psychophysical thresholds are measured based on the accuracy of judgments made about this probe at each tested position, which are used to infer the locus of attention. 

The use of this standard methodological approach leaves some important questions unanswered about the spatiotemporal dynamics of perisaccadic perception. Psychophysical paradigms that employ a single stimulus on each trial are generally not sensitive enough to measure whether and how an observer’s perception of a target may be influenced by multiple stimuli. For example, it is not possible to obtain direct measurements of information integration from across-trial differences in accuracy that can be used to determine whether and how predictive neural activity causes the visual system to integrate visual information from multiple locations into a single percept of an attended target (although see Szinte et al. (2016) for indirect evidence of perisaccadic integration inferred from across-trial differences in accuracy). Furthermore, the use of rapidly flashed probe stimuli may act as an (inadvertent) cue to orient attention to the location of the flashed stimulus. It is well known that a rapid stimulus onset will capture attention in an involuntary manner (Carrasco, 2011). Stimuli do not often rapidly appear and disappear in natural scenes, leaving it unclear whether the conclusions drawn from paradigms using such stimuli are generalizable to natural viewing conditions. 

The current study was designed to determine the spatiotemporal dynamics of perisaccadic perception by employing an experimental paradigm sensitive enough to measure the contribution of multiple locations in the visual field to an observer’s perception of a target. In addition, we aimed to test a prediction that follows from empirical and theoretical accounts that predictive perceptual updating is closely linked to attention (Berman & Colby, 2009; Gottlieb et al., 1998; Melcher, 2009; Wurtz, 2008). Wurtz (2008) pointed out that it may be very computationally expensive to predictively remap every object in the visual field. He proposed that because visual attention and eye movements are typically linked in natural tasks, the predictive update only needs to be made for objects near the saccadic target and that the limited capacity of attention may help the visual system to constrain the problem. This account is consistent with a study by Gottlieb et al. (1998) demonstrating that the receptive field remapping reported in LIP only occurs for objects that are attended (either because they are task relevant or because they attract attention via sudden stimulus onsets). If the neural mechanisms that underlie predictive perceptual updating are intimately linked with attention, it raises the possibility that multiple attended targets may be predictively updated simultaneously when attention is split between two locations. We explore this possibility in Experiment 3. 

We used a psychophysical reverse-correlation analysis (Eckstein & Ahumada, 2002; Neri et al., 1999), which is an objective method that can use noise added to a stimulus to characterize how observers integrate visual information across temporal intervals and spatial locations to make a perceptual judgment (Caspi et al., 2004; Mareschal et al., 2006). While the use of psychophysical thresholds or accuracy measures provide merely an indirect measurement of attentional allocation, reverse correlation provides a direct estimate of the locations from which visual information is gathered. This approach also allows us to present stimuli at static and permanent locations that mitigate the possibility of attention being inadvertently cued to a location. We asked participants to judge the appearance of an attended object that always remained at the same location, whose luminance changed dynamically across an interval that included the planning and execution of a saccade. We then used a reverse correlation analysis to compute the contributions, as a function of time, of features from various spatial locations to the reported appearance of the target. 

Experiment 1

We first set out to characterize the transsaccadic spatiotemporal profile of visual information integration that contributes to an observer’s perception of an attended target. Experiment 1 consisted of two separate experiments designed to achieve this goal. In Experiment 1A, participants were instructed to perform horizontal saccades while discriminating the average luminance polarity (bright or dark) of a flickering Gaussian-shaped target that appeared above the initial fixation point. To assess the potential effects of saccade-related neural activity, the display also included two flickering distractors that horizontally flanked the target (Figure 2A). We positioned the distractors at the saccade-congruent and saccade-incongruent locations relative to the target as described in the Introduction, consistent with previous studies (Jonikaitis et al., 2013). The intensity of the target and of the distractors was varied over time by randomly sampling luminance noise values from independent normal distributions every 10 ms (Figure 2B). The participant’s gaze position and the luminance noise added to the target and distractors were recorded as a function of time. 

Figure 2.

 

Experiment 1. (A) Stimulus sequence for a trial of the luminance polarity discrimination task. Saccades (indicated schematically by arrows) could be instructed either to the right or to the left of fixation. Colored circles indicate the location of luminance blobs in terms of the saccade direction (green: target location, red: saccade-congruent location, blue: saccade-incongruent location). The colored circles and arrows did not appear in the actual display. (B) Temporal representation of luminance across the full 500 ms stimulus duration. Colors correspond to spatial locations in (A). The shaded gray region indicates the interval during which the target signal appeared (when present). (C) Results of the reverse correlation aggregated across five human observers in Experiment 1A (left), and for an ideal observer (right) viewing the same stimulus sequences (see Figure A.1 and Table A.1 in the Appendix for individual results). For both plots, the colored curves represent the average luminance noise values at the corresponding location in (A) as a function of the time relative to saccade onset (vertical black line; the shaded region adjoining this line represents the median saccade duration). The dashed black curves and translucent area represent 95% prediction intervals for the reverse correlation value expected when observers completely ignore the information at a particular location. Portions of the colored curves outside of this interval indicate spatiotemporal points that contribute significantly to the perceptual judgment. (D) Results of the reverse correlation aggregated across two human observers in Experiment 1B where the stimulus was the same as in Experiment 1A but observers maintained fixation. Since this control experiment involves no saccades, the luminance noise values were aligned to the stimulus onset when computing the reverse correlation. The vertical black line represents the average saccade latency observed in Experiment 1A, included for reference.

To reveal how human observers accumulate visual information around the time of a saccade, we computed psychophysical kernels describing the average contribution of each location (i.e., the target, the saccade-congruent distractor, and the saccade-incongruent distractor) to the bright/dark discrimination as a function of time. The resulting psychophysical kernel yields the average spatiotemporal window of integration, specifying how the brain acquires information across the target and distractor locations and integrates it over time to guide the luminance polarity judgment about the target. From this analysis we can differentiate between potential mechanisms that may underlie transsaccadic perception by comparing our findings with the predictions laid out in Figure 1C–F. 

Experiment 1B was a control experiment designed to assess whether the results observed in Experiment 1A were due to an impending saccade or an artifact within the stimulus display. Participants viewed the same stimulus sequence as in Experiment 1A but maintained fixation. We reasoned that if the results of Experiment 1A were the result of saccade-related neural activity, we should observe a qualitatively different pattern of results in Experiment 1B. 

Methods

Participants

A total of seven participants with normal or corrected-to-normal vision took part in this experiment, five in Experiment 1A and two in Experiment 1B. All participants were undergraduate students at Rutgers University and all but one (JPW) were naïve to the aims of the study. Author JPW ran in Experiment 1B and also participated in Experiments 2 and 3. All participants provided written informed consent and the experimental procedures were reviewed and approved by the Rutgers University Institutional Review Board for the Protection of Human Subjects. 

Stimuli and apparatus

Stimuli were presented on a 22 inch color CRT monitor (Philips 202P4, Amsterdam, the Netherlands) set to a screen resolution of \(1280\times 1024\) pixels and a vertical refresh rate of 100 Hz. The display was viewed binocularly at a viewing distance of 70 cm from the observer, so that the display subtended a visual angle of approximately 21.1\(^\circ\)\(\times\)15.8\(^\circ\). Stimuli were generated and presented using MATLAB software (Mathworks, Natick, MA) and the Psychophysics Toolbox extensions (Brainard, 1996). The monitor luminance was linearized using a hardware lookup table, so that the full bit depth (8 bits per color channel) remained available following linearization. 

Figure 2A depicts the stimulus display. The target and flanker stimuli were flickering isotropic luminance “blobs” whose spatial luminance profiles were defined by two-dimensional Gaussian functions with a spatial standard deviation of 0.5\(^\circ\). The temporal flickering was created by randomly selecting the luminance amplitudes of the Gaussian blobs in each 10 ms frame. On “signal absent” trials, each luminance amplitude was selected independently from a normal distribution centered on the mean background luminance of 40\(\ \text{Cd/m}^2\), with a standard deviation of 4\(\ \text{Cd/m}^2\)(10% RMS contrast; depicted in Figure 2B), and the stimulus lasted a total of 500 ms, or 50 frames. Reverse correlation analysis was conducted on “signal absent” trials. For “signal present” trials, a mean luminance increment or decrement was added to the middle 40 frames (i.e., frames 6–45) of the target, which was equal to either the median of the thresholds calculated in each completed session of training trials up to that point in the study, or twice this value. This manipulation means that the magnitude of the target increment or decrement varied across sessions, but remained constant within a session. Moreover, note that because “signal present” trials were excluded from these analyses, the statistical properties of the stimuli used to compute the reverse correlation were consistent throughout the course of the experiment (i.e., both across and within sessions). 

Eye movement recording and analysis

Gaze position was monitored using an infrared video-based eye tracker sampling at 1000 Hz (Eyelink 1000, SR Research, Kanata, Ontario, Canada). Head position was fixed using a forehead and chin rest and measurements were made using only the right eye. A calibration procedure (consisting of nine points sampled along a 20\(^\circ\)\(\times\) 15\(^\circ\)grid) was performed at the start of each block of trials and the block proceeded only when the average calibration error was less than 0.25\(^\circ\). Saccades were detected online when the eye velocity exceeded a threshold of 200\(^\circ\)/sec. Saccade latencies were computed offline by fitting a sigmoidal function (a scaled cumulative cosine distribution function) to gaze position as a function of time and selecting the 1% point of this function as the start of the saccade. Since the required saccade amplitude was always 10\(^\circ\), this means that the saccadic latencies reported in our analyses indicate the time from instruction required for gaze to move approximately 0.1\(^\circ\) toward the saccadic target. To determine landing error, we further defined the landing position of the saccade as the 99% point of this function. Because participants always made horizontal saccades, the landing error was calculated as the signed horizontal difference between the landing position and the location of the fixation point after the observer was instructed to make the saccade (i.e., the saccadic goal). Negative values reflect an “undershooting” of the saccade, while positive values reflect “overshooting.” 

Procedure

Each trial began with a small white fixation marker appearing in the center of the display and flanked at 10\(^\circ\) to the left and right by peripheral gray markers indicating potential saccadic targets (Figure 2A). Participants started the trial by pressing a button while fixating the center of the display. 

After a randomly chosen stimulus onset asyncrony between 250–500 ms, the central fixation marker disappeared and one of the peripheral markers underwent a rapid luminance reversal (white, then black) indicating its identity as the new fixation point. The flashing of the new fixation point, along with a simultaneous auditory beep, served as a saccadic “go” signal. The direction of the new fixation point (left or right) was selected randomly on each trial. Coincident with the appearance of the new fixation point, the discrimination target, a flickering luminance blob, appeared 5° above the original fixation marker. Participants were told to shift their gaze to the new fixation point as quickly as possible while monitoring the luminance of the target blob, and to continue fixating the new fixation point until the blob stopped flickering. At the end of the trial, participants used a keypress to indicate the target’s luminance polarity (i.e., whether the target blob was brighter or darker on average than the gray background). Trials were discarded and repeated if participants did not maintain gaze at the central fixation for the full stimulus onset asynchrony, did not begin an eye movement during the allotted time (between 130 and 350 ms after the saccadic goal appeared), made more than one saccade, or the eye movement did not land within 1.5\(^\circ\) of the indicated saccadic goal. 

Participants performed both training trials and test trials, each arranged in separate blocks of 100. In training trials, the central target blob appeared alone, without any flankers. The purpose of these trials was two-fold: 1) to refamiliarize participants with the appearance and time course of the target signal and 2) to measure and adjust the luminance contrast threshold used to set the mean luminance of the target signal in “signal present” test trials. The mean luminance contrast of the target in each trial was selected using an adaptive psychophysical procedure (Kontsevich & Tyler, 1999), and the luminance polarity was selected randomly. At the end of each training trial, participants received auditory feedback indicating whether their responses were correct or incorrect. 

In test trials, the target blob appeared simultaneously with two flickering flankers displaced 10\(^\circ\) to its left and right. Participants were instructed to ignore the flankers and report only the luminance polarity of the target. Test trials included both a “signal present” condition in which the magnitude of the mean luminance increment or decrement added to the target was set to equal the threshold contrast corresponding to 81% accuracy during training (25% of test trials) or twice the threshold (5% of test trials) measured in the training trials, and a “signal absent” condition (70% of test trials) in which the luminance values of the blob for each frame were sampled from a distribution with a mean equal to that of the gray background (40\(\ \text{Cd/m}^2\)). Reverse correlation was computed using only “signal absent” trials. The “signal present” trials were included as a reminder of the relevant stimulus time course (i.e., using a suprathreshold target helped to remind participants of which time points were relevant to their perceptual judgments), and to determine the rate at which correct and incorrect audio feedback should be provided on “signal absent” trials. As in the training trials, participants received auditory feedback after each trial. For “signal absent” trials (in which there was no correct response), the feedback was generated randomly according to the participant’s response accuracy computed across all “signal present” trials completed in the ongoing block. 

Participants completed a total of 5,000 test trials over seven or eight 1 hour sessions run on separate days. Each experimental session consisted of a 100 trial block of training trials followed by five to seven blocks of test trials. Progress through these blocks was self-paced (typically requiring 5–7 minutes to complete) and participants were encouraged to take breaks between blocks. 

Data analysis

We computed psychophysical kernels describing the average contribution of each location to the bright/dark discrimination as a function of time. We restricted this analysis to “signal absent” trials (70% of all trials). For each trial, we aligned the time series of the target and distractor luminance noise values to the initiation of the saccade. We then computed a space–time psychophysical kernel by averaging the time series of these luminance noise values across trials in which participants responded “bright” with the negative of luminance noise values across trials in which participants responded “dark,” thereby treating every response as if it were bright. We do this so that we can compute the reverse correlation for all trials together (i.e., this is mathematically equivalent to computing separate psychophysical kernels for each response class, and then subtracting the dark kernel from the bright kernel, except that the resulting average is automatically adusted for any bias in response class). The resulting psychophysical kernels indicate the average noise luminance value presented when the participant reported the target to be bright as a function of time throughout the stimulus duration. If the visual information presented at a particular location and time contributed to the participant’s luminance discrimination of the target, then the average luminance noise should be positive. However, if a particular location and time did not contribute to the judgment, the average noise luminance value reflected in the psychophysical kernel should be near zero at that point. 

To assess the statistical significance of our results, we conducted two analyses. We first computed 95% prediction intervals for the reverse correlation value expected when observers completely ignore the information at a particular location. Any kernel value that is outside of this interval indicates spatiotemporal points that contribute significantly to the perceptual judgment. We also ran a more powerful test by averaging the reverse correlation value at each of the target and distractor locations over the 100 ms interval preceding the saccade and using a t test to determine whether this value was significantly greater than zero. 

Finally, to quantitatively characterize the deviation of the human observers from ideal performance and to identify potential sources of inefficiency, we computed two different measures of efficiency: sampling efficiency and absolute efficiency. Sampling efficiency (Burgess et al., 1981; Legge et al., 1987) characterizes the extent to which observers make use of the available signal information, discounting the effects of any internal noise. For a linear observer performing a discrimination task like the one used in this study, this measure characterizes the fit between the ideal classification/discrimination template and that used by the observer. In particular, sampling efficiency \(F_{\text{samp}}\) can be computed as the square of the normalized dot-product between these two templates (Murray et al., 2005),  

\begin{eqnarray} F_{\text{samp}} = (\mathbf {t}_{\text{ideal}} \,{\boldsymbol\cdot }\,\mathbf {t})^2, \quad \end{eqnarray}

(1)

where \(\mathbf {t}_{\text{ideal}}\) and \(\mathbf {t}\) are vectors representing normalized (i.e., unit length) versions of the templates used by the ideal and human observers, respectively. In our analyses, \(\mathbf {t}_{\text{ideal}}\) was represented exactly, consisting of a boxcar function at the target location (i.e., all samples occuring during the middle 40 frames had equal non-zero weights) and zeros elsewhere; while the human templates were estimated using the psychophysical kernels obtained via reverse correlation. 

Absolute efficiency (Burgess et al., 1981; Legge et al., 1987; Tanner & Birdsall, 1958), on the other hand, characterizes how well observers perform relative to the ideal overall, including any effects of internal noise. Murray et al. (2005) showed that a linear observer’s absolute efficiency \(F_{\text{abs}}\) can be estimated as  

\begin{equation} F_{\text{abs}} = \left[\frac{(\mathbf {t}_{\text{ideal}}\,{\boldsymbol\cdot }\,\mathbf {c})^2}{\sigma ^2_\mathbf {c}}-1\right] \frac{\Phi (d^\prime /2)\Phi (-d^\prime /2)}{n\phi (d^\prime /2)}, \end{equation}

(2)

where \(\mathbf {c}\) represents the estimate of the observer’s unnormalized template; \(\sigma _\mathbf {c}\) represents the standard error of the individual spatiotemporal elements in that estimate; \(n\) represents the number of trials used to compute the estimated template; \(\phi\) and \(\Phi\) represent the probability density and cumulative distribution functions, respectively, for a standard normal variable; and \(d^\prime\) is a measure of the observer’s sensitivity computed from their performance (i.e., \(d^\prime = 2\Phi ^{-1}[\text{prop. correct}]\)).¹ 

Results

In Experiment 1A, participants made horizontal saccades while discriminating the average luminance of the target position. Recall that at the start of each session, participants completed a block of training trials to estimate the luminance threshold contrasts used to set the mean luminance values of the “signal present” condition in test trials. Participants completed between 7 and 9 training blocks over the course of their participation in the study. Threshold contrasts for individual participants ranged between 2.5% and 4.2%. 

On training trials, the target was flanked by two distractors located 10\(^\circ\) to the left and right. We first analyzed saccade kinematics for these trials. The average saccadic latency was 177 ms (standard error of the mean: \(\pm\) 7) and the average saccade duration was 34 ms (\(\pm\) 2). Participants’ saccades tended to be slightly hypometric, with an average landing error of −0.47\(^\circ\)\(\pm\) 0.07. 

We next sought to determine participants’ accuracy in reporting the target luminance polarity during test trials. We did not analyze accuracy on “signal absent” trials because the average luminance of the target is expected to be the same as the background luminance on each of these trials. For “signal present” trials, in which a luminance increment or decrement was added to the target, the average accuracy across all completed trials was 75.2% (\(\pm\) 2.1%). It is worth noting that the luminance increment/decrement included on “signal present” trials was chosen to be each observer’s median luminance threshold contrast from all completed blocks of training trials, which corresponded to 81% accuracy. One might then expect that performance in “signal present” trials should be close to 81%; however, given that test trials included flanking distractors and training trials did not, we interpret this lesser accuracy to reflect the greater difficulty of the test trials. 

Figure 2C (left) shows the results of the reverse correlation analysis conducted across all five participants. Each spatial position in the psychophysical kernel is depicted as a colored curve representing the average luminance (as a function of time from the saccade onset) for responses corresponding to a “bright” judgment. Positive values outside the bounds of the 95% prediction intervals surrounding 0 indicate that the corresponding location was used to make the luminance discrimination of the target position. Individual kernels computed for each participant qualitatively match the aggregated results and are depicted in Figure A.1. 

As expected, participants used information from the actual target location. Both before and after the onset of the saccade, positive luminance noise values at the target location (green curve) contributed to bright polarity responses, indicating that participants did integrate information across the eye movement. However, starting about 80 ms before the onset of the saccade, positive luminance noise values at the future retinal location of the target (blue curve; saccade-incongruent location) also contributed to bright polarity responses. This behavior is consistent with what we would expect under a predictive perceptual updating where participants start to integrate from the predicted post-saccadic retinal location of the target in advance of the saccade. To test the robustness of this result, and to allow testing for this presaccadic mislocalization in the results of individual observers, we ran a second, more powerful, test by averaging the reverse correlation value at each of the target and distractor locations over the 100 ms interval preceding the saccade to determine whether this value was significantly greater than 0. This test confirmed the result. The average presaccadic contributions of the target [t(16241) = 23.652, \(p = 1.3\times 10^{-121}\)] and saccade-incongruent distractor [t(16241) = 9.732, p = \(2.5\times 10^{-22}\)] positions were significantly greater than 0, while the contribution of the saccade-congruent location was not [t(16241) = 1.246, p = 0.21]. This result was also significant in 4 of the 5 individual participants (see Figure A.1 and Table A.1 in the Appendix). 

We completed an additional analysis designed to determine what the results of the reverse correlation would be if an observer used only information from the target location to make the luminance discrimination, equally weighting each frame of the target’s luminance during the target presentation interval. Such a strategy represents the ideal decision process. For each trial, we computed the temporal average of the luminances presented at the target location and used the resulting value to determine the ideal observer’s polarity response (i.e., classified “bright” if this value was greater than 40\(\ \text{Cd/m}^2\) and “dark” otherwise). We then computed the psychophysical kernel for the ideal observer using the same reverse correlation method used for the human observers. The resulting psychophysical kernel is depicted in Figure 2C (right). The target location provides a sustained contribution to the ideal observer’s judgment throughout the target presentation interval. Note that the ideal observer’s kernel differs from a boxcar function only because the psychophysical kernel is computed as a function of the human saccade onset times (which were variable) rather than as a function of stimulus onset time. 

To quantitatively characterize human observers’ use of visual information we also computed two measures of efficiency: absolute efficiency and sampling efficiency (see definitions in the Data analysis section). The absolute efficiency computed for the human observer (using aggregated human data) was 12.8% (bootstrapped 95% confidence intervals = [11.7%, 14.3%]), which is within the range of absolute efficiencies computed for luminance contrast discrimination performance in similar tasks involving static stimuli (Murray et al., 2005). The sampling efficiency computed for the human observers was considerably greater at 69.1% ([66.6%, 73.9%]), which suggests that much of the human inefficiency relative to the ideal observer can be attributed to internal noise sources, rather than to a poor template (i.e., poor use of the available information) per se. 

While we attribute the influence of the saccade-incongruent distractor on the luminance judgment of the target to be a result of the impending saccade in Experiment 1A, it is possible this result occurred due to an artifact in the stimulus. For example, it is possible that the rapidly flickering luminance changes elicit an exogenous cueing effect that directs an observer’s attention to the saccade-incongruent distractor. To assess this possibility, we conducted Experiment 1B, which was a control experiment where two new participants viewed identical stimulus sequences as in Experiment 1A but maintained fixation. Participants again completed one block of test trials at the beginning of each experimental session (JPW: 7 blocks; DJA: 9 blocks). The threshold contrast was 1.0% for both participants. 

The psychophysical kernel for Experiment 1B is depicted in Figure 2D. This time course reveals that the target position influenced the luminance judgment throughout the presentation of the stimulus, and that the contribution of the target stimulus to the final judgment decreased with time. Importantly, there is no evidence of any contribution from either of the distractors. The absolute efficiency of 24.9% ([22.6%, 27.7%]) computed for this experiment was greater than that of Experiment 1A. This finding indicates that the contribution of the saccade-incongruent location to the luminance discrimination observed in Experiment 1A was indeed the result of the impending saccade and not of exogenous cueing nor some artifact of the stimulus. 

Discussion

The results of Experiment 1 show that an observer’s perception of an attended target is influenced by an impending saccade. Roughly 80 ms before the execution of the saccade, perception of the target is influenced by both the target and the future retinotopic location of the target. While this pattern of data is generally consistent with previous descriptions of predictive perceptual updating, our reverse correlation analysis reveals that the visual system integrates visual information from multiple locations into a unified percept of an attended object. We discuss the implications of this finding in greater detail in the General discussion. 

We can further determine how the saccade influences the observer’s ability to make the luminance discrimination by comparing participant performance to the results of our ideal observer analysis. The ideal observer makes the luminance discrimination by integrating the luminances at only the target location, which yields a strong and sustained influence of the target on the resulting judgment. The human participant psychophysical kernels differ from the ideal observer results in four distinct ways. First, observers used luminance information from both the target and saccade-incongruent location just before the saccade to make the luminance discrimination. Second, the target location influences the human judgments to a lesser degree than the ideal observer (compare the relative values of the green curves for the human and ideal observers in Figure 2C). This suboptimality in the human data may reflect the relative influence of the saccade-incongruent location in the discrimination process. Alternatively, it may be due to decreased sensitivity for the luminance presented at the target location as a result of the saccade, consistent with reports that little visual information is available when the eyes are moving (Krekelberg, 2010). There is some evidence for this because the values of the psychophysical kernel for the target location when fixation was maintained in Experiment 1B are larger than those in Experiment 1A. Likewise, the human observers’ absolute efficiency in Experiment 1B (24.9%) was greater than in Experiment 1A (12.8%). Third, contrary to the ideal observer, the human kernels reveal that the luminance discrimination was not influenced by visual information present at any of the locations for about 40 ms after the onset of the saccade. This is likely to be due to decreased sensitivity of the human observers to the luminance at the target location during the saccade. Fourth, the influence of the target decayed with time after completion of the saccade, whereas the ideal observer bases its judgment on the target luminance throughout the trial. A similar finding has been reported in a study measuring the temporal dynamics of attentional selection during fixation, suggesting that the pattern observed here is not due to saccades (Shimozaki et al., 2007). One possible explanation is a lapse in attention owing to an inefficiency in the mechanisms responsible for maintaining attention, e.g., temporal capacity limitations. Another possibility is that the decrease is related to response selection processes that occurred before the conclusion of the trial. 

Taken together, these results indicate that participants integrate visual information in a manner consistent with a predictive perceptual updating that is based on the corollary discharge signal of the saccade. The specific pattern of results we observe is consistent with a predictive attentional updating. As depicted in Figure 1D, predictive attentional updating would lead to the integration of visual information from the target and saccade-incongruent distractor, which is what we observed. Our results are not consistent with the “collapsing receptive fields” hypothesis or predictive remapping in the absence of predictive attentional updating, which both predict an integration of visual information at the target and saccade-congruent distractor locations (Figure 1F). 

However, the stimulus configuration in Experiment 1 is too spatially coarse to definitively rule out other mechanisms that don't necessarily involve prediction. For example, in tasks requiring explicit localization of flashed targets around the time of a saccade, perceptual reports often show evidence of graded perceptual distortions, i.e., distortions whose magnitude varies as a function of the appearance of the target relative to saccade onset (Honda, 1989, 1991; Matin & Pearce, 1965; Schlag & Schlag-Rey, 1995), of spatial compression (Kaiser & Lappe, 2004; Lappe et al., 2000; Ross et al., 1997), or of generalized visual uncertainty regarding target location (Binda et al., 2007). 

If the integration observed in Experiment 1 were simply the result of position uncertainty or of spatial compression around the time of the saccade, then we would expect information from locations that are nearer to the target (and thus more spatially confusable with the target location) to contribute more to the target luminance discrimination than information from locations that are farther from the target. However, if presaccadic integration is the result of a predictive perceptual updating based on the corollary discharge signal, then the integration should be specific to the predicted post-saccadic retinal location of the target. In other words, a distractor placed at the post-saccadic retinal location of the target should contribute more to the target luminance discrimination than a distractor placed at some intermediate spatial location that is physically nearer to the target position. We assess this question in Experiment 2. 

Experiment 2

Experiment 2 tested whether the integration we observed in Experiment 1 is a result of a predictive updating based on a corollary discharge signal or another mechanism such as position uncertainty or spatial compression. To do so, we measured the spatial specificity of integration by introducing additional distractors near the attentional target. As in Experiment 1, participants were instructed to perform horizontal saccades while discriminating the average luminance polarity (bright or dark) of a flickering Gaussian-shaped target that appeared above the initial fixation point. However, in Experiment 2, the display included four (rather than two) flickering distractors that horizontally flanked the target (Figure 3A). These distractors included two placed at positions identical to those used in Experiment 1, to the left and right of the target (corresponding to the post-saccadic retinal location of the target following rightward or leftward saccades, respectively). They also included two distractors placed at intermediate locations, centered at the midpoints between the target and the original distractors. To reveal how human observers accumulate visual information around the time of a saccade, we again computed psychophysical kernels describing the average contribution of each location to the bright/dark discrimination as a function of time. 

Figure 3.

 

Experiment 2. (A) Stimulus sequence for a trial of the four-distractor version of the luminance polarity discrimination task. As in Figure 2, colored circles indicate the location of luminance blobs in terms of the saccade direction. In addition to the saccade-congruent (red) and saccade-incongruent (blue) distractor locations used in Experiment 1, Experiment 2 added distractors at intermediate locations (yellow and cyan, respectively). To keep participants from confusing the target location with nearby distractor locations we added a vertical white marker that surrounded the target location. (B) The results of the reverse correlation aggregated across four human observers. Individual participant results are provided in Figure A.2 and Table A.2 in the Appendix.

Methods

Participants

Four participants took part in this experiment. All observers had normal or corrected-to-normal vision, and all but one (JPW) were naïve to the aims of the study. JPW was an author and also participated in Experiment 1B and Experiment 3. All participants provided written informed consent and the experimental procedures were reviewed and approved by the Rutgers University Institutional Review Board for the Protection of Human Subjects. 

Procedure

The stimulus display in Experiment 2 was identical to that of Experiment 1 except with the addition of two distractors that were presented at intermediate locations between the target position and far distractors. Figure 3A depicts an example display. Test trials in Experiment 2 included four flickering flankers (two on the left and two on the right of the target blob) instead of the two used in Experiments 1A and B. The two additional flankers were each centered between the original flankers and the target, so that the five blobs (four flankers plus one target) were spaced 5\(^\circ\) apart and centered on an imaginary horizontal line 5\(^\circ\) above the fixation point. Furthermore, because the target and flankers were now closer together, a broken vertical line was added to the display to disambiguate the location of the target blob. 

Training trials were identical to those of Experiment 1. The session structure and procedure were identical to those of Experiment 1. Participants completed a total of 7,000 test trials in 10 or 11 sessions run on separate days. 

Results

Participants completed between 6 and 8 training blocks over the course of their participation in the study. Threshold contrast was 11% for all four participants. The average saccadic latency was 158 \(\pm\) 5 ms and the average saccade duration was 34 \(\pm\) 1 ms. The average landing error was −0.26\(^\circ\) \(\pm\) 0.05°. Participant accuracy on “signal present” trials was 64% \(\pm\) 1.7 %. 

Figure 3B depicts the results of the reverse correlation analysis conducted across all four participants. The results for the target and distractor locations corresponding to those used in Experiment 1 were unchanged. Participants used information from the actual target location (green curve) before and after the onset of the saccade [t(17797) = 9.866, p = \(6.7\times 10.0^{-23}\)], and positive luminance noise values at the future retinal location of the target (blue curve; saccade-incongruent location) also contributed to “bright” polarity responses [t(17797) = 10.488, p = \(1.2\times 10.0^{-25}\)]. However, luminance noise values at the intermediate location positioned between the target and the future retinal location of the target (cyan curve) contributed only weakly to the polarity response [t(17797) = 4.345, p = \(1.4\times 1.00^{-05}\)]. This weak, but significant, contribution was driven by a single participant; three of the four participants did not reveal a contribution of this intermediate location to the luminance discrimination (for details, see Figure A.2 and Table A.2 in the Appendix.) No other distractor contributed to the luminance discrimination (all p values \(\gt \) 0.2). While the influence of the saccade-incongruent location seems to be greater than the intermediate location based on the psychophysical kernel in Figure 3B, we conducted a t test to statistically compare the averaged reverse correlation value at each location over the 100 ms before the saccade. The reverse correlation value for the saccade-incongruent location (0.0943\(\ \text{Cd/m}^2\)) was significantly larger than at the intermediate location (0.0393\(\ \text{Cd/m}^2\); [t(17797) = 4.304, p = \(8.4\times 10.0^{-06}\)]). 

The absolute efficiency (17.7% [16.6%, 19.3%]) and sampling efficiency (63.9% [60.7%, 66.7%]) were comparable with those obtained in Experiment 1. 

Discussion

The results of Experiment 2 indicate that perisaccadic integration of visual information is spatially specific to the post-saccadic retinal location of the target. Although the reverse correlation of one observer did yield a small but significant contribution of the intermediate distractor between the target and the saccade-incongruent location, the majority of participants showed no such pattern. This behavior is consistent with what we would expect under a predictive perceptual updating in which participants started to integrate from the predicted post-saccadic retinal location of the target in advance of the saccade. 

It is also worth noting that the relative presaccadic contributions of the target and saccade-incongruent locations differed compared to Experiment 1 (compare Figure 2C and Figure 3B). In Experiment 1, the contribution of the target location was larger than the saccade-incongruent distractor and both locations contributed simultaneously to the judgment before the saccade. However, in Experiment 2 both locations made roughly equal contributions and they seem to occur sequentially in time. This pattern of data seems to be caused by inter-individual differences in the time course of integration, which can be seen by examining the individual participant results in Figure A.2. For three of four participants, the only location that contributes to the observer’s luminance judgment before the saccade is the saccade-incongruent location. However, one participant (VS) used the luminances presented at the target location before the saccade in addition to the saccade-incongruent location and the intermediate distractor between them. The contribution of the intermediate distractor in VS's brightness judgments may be due to a few possibilities. First, it may be that this participant exhibited consistent undershooting in the landing position of their impending saccade, which could have lead to a predictive attentional updating of the less eccentric intermediate distractor. To assess this possibility, we evaluated VS’s landing error compared with the landing error of the other participants. VS undershot saccades on average by −0.36\(^\circ\), which was larger than any of the other participants (JPW: −0.14\(^\circ\), LK: −0.28\(^\circ\), NDK: −0.27\(^\circ\)). However, the magnitude of VS’s undershoots were much smaller than the roughly 4\(^\circ\) that separated the outer boundaries of the saccade-incongruent and intermediate distractors, indicating that a biased predictive attentional updating cannot explain VS’s data. Another possibility is that the visual field was spatially compressed perisaccadically in the direction of the impending saccade. Although we cannot rule out this possibility, the fact that the target location remained heavily weighted at the same time as the intermediate and saccade-incongruent conditions (compare each line in the interval just before the saccade for subject VS in Figure A.2) suggests that the perceived location of the target did not vary, seeming to rule out a global compression of the full visual field. Finally, it may be that uncertainty in the position of the target or saccade-incongruent distractor lead to VS weighting the intermediate distractor in their judgments. Although we attempted to mitigate the possibility of location uncertainty in this experiment by including a thin line that indicated the position of the target, it is possible that this manipulation did not totally work for subject VS, at least for some trials. 

Despite these individual differences, our results are clearly consistent with a predictive perceptual updating that is driven by predictive attentional updating, and incompatible with an account of perisaccadic perception resulting from collapsing receptive fields at the goal of the impending saccade. 

Experiment 3

Experiments 1 and 2 show that the attended target of a perceptual discrimination judgment is influenced by visual information at nontarget locations in a manner consistent with predictive perceptual updating based on the corollary discharge signal. This result agrees with what we would predict based on a predictive attentional updating account. 

In Experiment 3, we aimed to determine the flexibility of the neural mechanisms underlying perisaccadic perceptual updating. Several authors (Berman & Colby, 2009; Wurtz, 2008) have pointed out that the brain mechanisms that would be required for predictive perceptual updating make it unlikely that we update every detail of the visual image in anticipation of a saccade and have suggested instead that only attended objects or locations are updated. In fact, experiments on change blindness have shown that we are generally oblivious to changes made to unattended locations or objects (O’Regan & Noë, 2001; Rensink et al., 1997; Simons & Rensink, 2005), and evidence from physiological (Gottlieb et al., 1998) and perceptual (Melcher, 2009; Szinte et al., 2018) experiments have demonstrated that predictive perceptual updating in particular occurs only for attended objects or locations. It is not clear from previous studies whether these mechanisms concurrently update locations corresponding to several spatially distinct attended targets, or if they are limited to updating only a single attended target at a time. 

To answer this question, Experiment 3 had participants simultaneously attend to and track the average luminances of two peripheral targets across a saccadic eye movement. As in Experiments 1 and 2, participants were instructed to perform horizontal saccades while discriminating the average luminance polarity of a flickering Gaussian-shaped target that appeared above the initial fixation point. The top half of the display was identical to that used in Experiment 1, with a discrimination target located at the central location and two distractors horizontally flanking the target. However, in Experiment 3, the bottom of the display also contained a central target and two flanking distractors, so that the resulting configuration of stimuli was vertically symmetric (Figure 4A). The participants' task in this experiment was to simultaneously attend to both the upper and lower central targets and to report the average luminance polarities of both targets following a saccade. To facilitate this report, we presented a response screen at the end of each trial displaying icons representing each of the possible combinations of top/bottom luminance polarities (see Figure 4B). Participants responded by fixating the icon corresponding to their perception and pressing a response key. A critical feature of this experiment is that the luminance values of the two attentional targets were chosen independently. Thus, neither of the targets contained information about the other and participants had to independently integrate luminance information at each of the two target locations to accurately compute the luminance polarities. 

Figure 4.

 

Experiment 3. (A) Stimulus schematic for the two target version of the luminance polarity discrimination task. As in Figure 2, colors indicate the horizontal location of luminance blobs in terms of the saccade direction (green: target location; red: saccade-congruent location; blue: saccade-incongruent location), while the line style indicates their vertical positions (solid: top; dashed: bottom). (B) The response screen for Experiment 3. Observers simultaneously indicated the perceived polarity of the top and bottom targets by fixating the icon corresponding to the perceived configuration and pressing a button. (C, D) Results of the reverse correlation aggregated across five human observers, where the reverse correlation is computed based on the response to the top (C) or bottom (D) target. Individual participant results are provided in Figure A.3 and Tables A.3 and A.4 in the Appendix.

As in Experiments 1 and 2, we computed psychophysical kernels to characterize how human observers accumulated visual information to make the bright/dark discrimination. However, because observers had to make two independent discriminations we computed two separate psychophysical kernels, based on the observers' polarity responses to each of the two targets. We reasoned that if the mechanisms that underlie predictive perceptual updating are flexible and linked to attention, then we should observe a pattern of data consistent with predictive perceptual updating for both attentional targets. 

Methods

Participants

Five participants took part in this experiment. All observers had normal or corrected-to-normal vision, and all but one (JPW) were naïve to the aims of the study. All participants provided written informed consent and the experimental procedures were reviewed and approved by the Rutgers University Institutional Review Board for the Protection of Human Subjects. 

Procedure

Experiment 3 was similar to Experiment 1A, but contained two attentional targets, one above and one below the central fixation point (Figure 4A). Training trials were structured similarly to those in Experiment 1, but participants had to indicate independently whether each of the targets was brighter or darker than the background luminance. In the adaptive psychophysical procedure, the magnitude of the mean luminance increment or decrement added to the target was adjusted simultaneously for the upper and lower targets. The stimulus configuration of test trials in Experiment 3 resembled a vertically reflected version of that used in Experiment 1A. Each trial included two attentional targets, centered horizontally on the original fixation point at the center of the screen and displaced vertically \(5^\circ\) above and below the fixation point, and four flankers displaced \(10^\circ\) to the left and right of the two targets. In each trial, there were four possible target configurations and responses: 1) top dark and bottom dark, 2) top dark and bottom bright, 3) top bright and bottom dark, and 4) top bright and bottom bright. To make the response mapping intuitive, we introduced a response screen at the end of the trial that showed cartoons of each possible target configuration (Figure 4B). Participants responded by fixating the configuration that best represented their perception and pressing a button. As in Experiment 2, participants completed a total of 7,000 test trials in 10 to 11 sessions run on separate days. 

Results

Participants completed between 9 and 13 training blocks. Threshold contrasts ranged between 21% and 26%. The average saccadic latency was 197 \(\pm\) 4 ms, while average saccade duration was 35 \(\pm\) 3 ms. Landing error was on average −0.45\(^\circ\) \(\pm\) 0.08. Accuracy on “signal present” trials was 70.1% ± 3.1 %. 

The psychophysical kernels obtained from the reverse correlation analysis across all five participants in Experiment 3 were consistent with those found in Experiment 1 (individual results are depicted in Figure A.3 and Tables A.3 and A.4). Regardless of whether the reverse correlation was conditioned on the response to the top (Figure 4C) or bottom (Figure 4D) target, participants (appropriately) used information from the corresponding target location (top: solid green curve; bottom: dashed green curve) before and after the onset of the saccade. In addition, positive luminance noise values at the future retinal location of the target (top: solid blue curve; bottom: dashed blue curve) also contributed to “bright” polarity responses. With one minor exception, luminance noise values at other locations did not contribute significantly to polarity responses. The one exception was that luminance polarity judgments regarding the bottom target (Figure 4D) were influenced weakly by the luminance of the top target (solid green line) via a sort of contrast effect; participants were slightly more likely to judge the bottom target as “bright” when the top target was dark, and conversely, were more likely to judge the bottom target as “dark” when the top target was bright. However, this contrast effect was only evident in the post-saccade interval. Importantly, the anticipatory influence of the stimulus at the future retinal location of the target was independent across target locations. 

We computed human efficiencies in this task separately for discrimination of the top and bottom targets. For the top target, absolute efficiency was 24.0% ([22.5%, 25.5%]) and sampling efficiency was 68.0% ([66.4%, 71.0%]). For the bottom target, absolute efficiency was 21.0% ([19.8%, 22.4%]) and sampling efficiency was 65.2% ([60.7%, 65.9%]). The slightly lower efficiencies for the bottom than for the top discrimination likely reflects the (weak) influence of the top target’s luminance in those judgments described above. 

Discussion

Experiment 3 reveals that predictive perceptual updating occurs independently for multiple attended targets. For both attentional targets, luminances presented before the saccade at the target and the post-saccadic retinal location of the target contributed to the observer’s judgment. The time course of these kernels reveal that both locations simultaneously influenced the luminance discrimination in the perisaccadic window, indicating independent integration of visual information for separate judgments at the two attentional foci. 

However, it is important to highlight an alternative account that could potentially explain qualitatively similar results. If observers simply made one judgment at a time, alternating between decisions regarding the top target and the bottom one and ignoring the other on any given trial, then the qualitative result would be similar to the one we obtained. Because the psychophysical kernels are computed as an average across trials, then our kernels might reflect the effects of both judgments, even if they were made in separate trials. However, the efficiencies computed for the human observers rule out this possibility. 

In particular, note that we computed human efficiencies separately for the top and bottom judgments, and that for each of these judgments the computed absolute and sampling efficiences were greater than those that we obtained in Experiment 1, when observers made only a single judgment. This point is important because, if observers ignored the stimuli roughly one-half of the judgments (i.e., ignored the bottom stimulus when making a judgment about the top target and vice versa), then these ignored stimuli would contribute additional noise to the observers’ performance, resulting in a lower computed efficiency. That is, the effect of observers ignoring one of the targets half of the time in Experiment 3 would be equivalent to the effect of substituting half of observers’ judgments in Experiment 1 with random guesses. Thus, the fact that human efficiencies did not decrease (and in fact increased) in Experiment 3 relative to those in Experiments 1 and 2 rules out this alternative account. 

That the psychophysical kernels show the post-saccadic target locations for both judgments simultaneously influencing the luminance discrimination in the perisaccadic interval, combined with the fact that the computed efficiencies are incompatible with an “alternation” strategy, indicates that human observers can independently integrate visual information for separate simultaneous judgments across at least two attentional foci. 

To our knowledge, we are the first to show this pattern of data. These results are consistent with the notion of a flexible neural predictive updating mechanism that is intimately tied to attention, as has previously been proposed by others (Gottlieb et al., 1998; Melcher, 2009). 

General discussion

Taken together, the current study suggests that the human visual system makes predictions about changes in the retinotopic positions of attentional targets based on the corollary discharge signal; that these predictions can be made concurrently and independently for multiple, spatially separated attentional targets; and that these retinotopic predictions have perceptual consequences, resulting in the transient presaccadic integration of visual information from both the presaccadic and postsaccadic retinal locations of the target. 

Overall, our results are consistent with physiological measurements of the time course of predictive remapping (Duhamel et al., 1992; Gottlieb et al., 1998; Nakamura & Colby, 2002; Umeno & Goldberg, 1997) and predictive attentional updating (Marino & Mazer, 2018). They are also consistent with functional accounts that characterize the predictive perceptual updating as linked to attention (Cavanagh et al., 2010; Hunt & Cavanagh, 2009; Marino & Mazer, 2016, 2018; Rolfs et al., 2011), and with related psychophysical demonstrations of saccadic “unmasking” (De Pisapia et al., 2010; Hunt & Cavanagh, 2011). Moreover, the results of Experiment 2 in particular are consistent with the finding that the retinotopic location associated with a target “jumps” from the presaccadic to the postsaccadic receptive field discretely, without sliding or expanding to encompass both sites (Sommer & Wurtz, 2006), and the results of Experiment 3 are consistent with theoretical suggestions that predictive perceptual updating involves the concurrent updating of a small set of attentional targets (Berman & Colby, 2009; Wurtz, 2008). 

However, our results disagree with those of experiments using explicit visual localization of flashed stimuli that indicate perceptual distortions around the time of a saccade are a result of a functional compression of visual space rather than a predictive update (Honda, 1989; Kaiser & Lappe, 2004; Lappe et al., 2000; Matin & Pearce, 1965; Panichi et al., 2012; Ross et al., 1997; Schlag & Schlag-Rey, 1995). In particular, whereas our psychophysical kernels suggest a discrete and precise integration of visual information consistent with predictive neural mechanisms, reports from the visual localization tasks often show evidence of graded mislocalization (Honda, 1989, 1991; Matin & Pearce, 1965; Schlag & Schlag-Rey, 1995), of spatial compression (Kaiser & Lappe, 2004; Lappe et al., 2000; Ross et al., 1997), or of generalized visual uncertainty regarding target location (Binda et al., 2007). We suspect that these differences stem from the fact that localization reports are subjective and involve relative position judgments that may not directly reflect the effects of the corollary discharge signal (Lappe et al., 2000). For example, evidence from localization tasks that have used pointing or striking responses (Burr et al., 2001; Hansen & Skavenski, 1985; Morrone et al., 2005) or manipulated visual references (Lappe et al., 2000; Morrone et al., 2005), have found that subjective localization judgments can change dramatically as a function of various task details, including the response modality, and the presence, form and timing of visual references. 

A critical feature of the current study is that participants were required to attend to and make a judgment about an attentional target that was visible across the eye movement, reflecting a way in which visual information is typically obtained and used in natural visual tasks. Therefore, the patterns of visual information integration revealed in our psychophysical kernels likely represent distortions that subtly impact our perception of peripherally attended targets in everyday tasks. 

One manner in which information integration is critical during natural viewing is when an observer must use information about stimuli located in the periphery to select a saccadic target, for example during visual search. In such instances, the observer’s perception of a stimulus is determined by the integration of low quality visual information in the periphery before the saccade with high quality information obtained from the fovea after the saccade. Empirical studies have observed that observers optimally combine the information from periphery and fovea when making perceptual judgments across a saccade in an manner consistent with maximum likelihood estimation, weighting each source according to the relative precision of information available at each location (Wolf & Schütz, 2015), or doing so in a near optimal way (Ganmor et al., 2015). Such optimal integration has also been found to occur for stimuli that remain in the periphery throughout a saccade (Stewart & Schütz, 2019). Although our study did not explicitly test instances where peripheral and foveal information are integrated or where an observers selects the location of their saccade, the findings reported here suggest that predictive perceptual updating may limit the visual system’s ability to successfully integrate information across saccades during naturalistic viewing behavior. 

A major advantage of the reverse correlation technique we used in the current study is that, because participants are not asked to provide any explicit reports of position or distance, it provides an objective measure of attentional allocation. It is worth noting that a previous study that also attempted to characterize the perceptual effects of saccades using reverse correlation failed to discover the effect of predictive perceptual updating (Panichi et al., 2012). This result is likely because its stimulus configuration did not allow the authors to probe the postsaccadic retinal target location (i.e., the saccade-incongruent location) during the presaccadic interval. Specifically, Panichi et al. (2012) tasked observers with monitoring two horizontal noise arrays of randomly selected luminance squares while making a saccade. The noise arrays were presented above and below the locations of the fixation point and saccadic goal. Crucially, observers began each trial fixating the left side of the screen and were instructed to make rightward saccades. The fixation point was positioned vertically between the leftmost edge of the noise arrays, meaning that there was never any presaccadic visual stimulation at the postsaccadic retinal location of the attended target (which spanned the length of the noise arrays). Thus, the stimulus configuration was not able to assess whether predictive perceptual updating influenced observer’s judgments. Moreover, the observer’s task (report whether a bright square was presented at any location in either the top or bottom luminance arrays) very likely differed in the required spatial distribution and allocation dynamics of attention than in the task reported here, which necessitated focused attention at a single target location throughout the trial. 

The reverse correlation method used in the current study revealed that perceptual decisions about an attended target are simultaneously influenced by presaccadic visual information presented at multiple locations in the visual field, namely the presaccadic and postsaccadic retinal location of the target. This result adds to existing accounts of predictive perceptual updating by showing that the focus of attention does not jump sequentially from one location to the next, which has previously only been inferred from accuracy measurements of psychophysical performance from an attentional cueing task (Szinte et al., 2016). For this to occur, the visual system must integrate the perceptual information from multiple populations of neurons simultaneously. This integration could be accomplished in a number of different ways, for example, by modulating the attentional state of both neuronal populations simultaneously. 

Although the focus of this study is on presaccadic perception, it should be noted that we did not observe a postsaccadic influence of any distractors on the luminance discrimination. This finding is in contrast with reports of a postsaccadic retinotopic attentional trace that modulates the attentional state of neurons with receptive fields that overlapped the target prior to the saccade (Golomb et al., 2008, 2010A, 2010B; Golomb, 2019). The studies reporting this phenomenon have primarily used attentional cueing paradigms and inferred attentional allocation via accuracy and reaction time measures. In our experimental design, this phenomenon would have been revealed through a postsaccadic influence of the saccade-congruent location. That we do not observe this result suggests that something specific to the method used by previous studies may have contributed to their findings. For example, it may be that attention was exogenously cued in previous paradigms, leading to a different spatiotemporal attentional profile. 

Finally, this study makes clear that predictive perceptual updating is closely linked with attentional allocation. Although it remains possible (even likely) that predictive remapping occurs during tasks in which covert attention is allocated to a target, as in the tasks performed here, the pattern of results we observed requires a predictive updating of the attentional state of neurons. Moreover, our findings are inconsistent with a recent account suggesting that neither the perceptual distortions nor the physiological shifts in attentional and position tuning measured near saccade onset reflect predictive updating, but that instead they represent the actions of an attentional gain signal centered on the saccade target that causes visual receptive fields to collapse toward the retinotopic location of the saccadic endpoint (Hamker et al., 2008; Zirnsak et al., 2010; Zirnsak & Moore, 2014). 

Acknowledgments

The authors thank Umang Parikh for his help in collecting data for Experiment 1A, and Eileen Kowler for her extensive comments and advice on early drafts of this article. 

Supported by NSF Award BCS-1456822 (to M.M.). 

Commercial relationships: none. 

Corresponding author: James P. Wilmott. 

Email: [email protected]. 

Address: Department of Cognitive, Linguistic, & Psychological Sciences, Brown University, Providence, RI, USA. 

Footnotes

1  Readers should be careful in interpreting the measure of “absolute efficiency” used here. Because the trials included in the analysis were all noise-only (i.e., “signal absent”) trials, proportion correct was computed according to the deterministic trial-by-trial classifications of the ideal observer (i.e., based on whether the average luminance at the location was actually lighter or darker than the background on that particular trial), and not based on the category labels, which were meaningless for the noise-only conditions. As a result, the \(d^\prime\) values used to compute the absolute efficiency measure here differ somewhat in meaning from those used in Murray et al. (2005). Moreover, the resulting absolute efficiency measure cannot be mapped straightforwardly onto the classic Tanner and Birdsall (1958) definition of efficiency (in which \(\eta = d^{\prime 2}_{\text{obs}}/d^{\prime 2}_{\text{ideal}}\)). For example, if we consider performance in terms of the (arbitrary) category labels, then \(d^\prime _{\text{ideal}} = 0\) for the noise-only conditions, where there is no signal contrast. Conversely, if we consider performance in terms of the (deterministic) category classifications of the ideal template, then \(d^\prime _{\text{ideal}} = \infty\). Neither case allows us to compute a meaningful value of human efficiency in terms of \(d^\prime _{\text{ideal}}\).

Appendix

Individual participant results

Experiment 1A

In addition to the psychophysical kernel we computed on aggregated data in Experiment 1A discussed above (depicted in Figure 2C), we also computed kernels for each individual participant to ensure that our results were not merely an artifact of pooling data across different participants. Figure A.1 depicts the result of these analyses. With the exception of subject BK, each participant reveals a qualitative pattern consistent with the results of the aggregated analysis; namely, that both the target and the saccade-incongruent distractor influence the brightness judgment before the execution of the saccade. 

Figure A.1.

 

Individual participant results of Experiment 1A. Color scheme as in Figure 2.

We also conducted individual tests to determine the statistical significance of the contributions of each stimulus location to the judgment. For each participant, we averaged the reverse correlation value at each of the target and distractor locations over the 100 ms interval preceding the saccade to determine whether this value was significantly greater than zero. Table A.1 summarizes the results of this analysis for each participant. The average presaccadic contributions of the target and saccade-incongruent distractor positions were significantly greater than zero in four of the five individual participants, consistent with the analysis conducted on the aggregated data. 

Experiment 2

Figure A.2 depicts the results of the reverse correlations computed for each participant in Experiment 2. With the exception of subject VS, each participant reveals a qualitative pattern consistent with predictive attentional updating, where the distractor at the saccade-incongruent location impacts the brightness judgment before the execution of the saccade. For subject VS, the target as well the saccade-incongruent distractor and the intermediate distractor influenced the brightness judgements. VS’s results could be due to either an incomplete spatial compression of portions of the visual field or position uncertainty about the target and/or the saccade-incongruent distractor (see Experiment 2 section for discussion). Individual t tests on the presaccadic contributions of each location are summarized in Table A.2. 

Figure A.2.

 

Individual participant results in Experiment 2. Color scheme as in Figure 3.

Experiment 3

Figure A.3 depicts the reverse correlations for the top target (Figure A.3A) and bottom target (Figure A.3B) for each participant in Experiment 3. Overall, the individual kernels reveal a pattern consistent with simultaneous predictive attentional updating of each target. Individual t tests on the pre-saccadic contributions of each location are summarized in Tables A.3 and A.4. 

Table A.1.

 

Statistical tests of presaccadic contributions to judgment in Experiment 1. The Bonferroni-corrected type I error rate for the individual t tests is \(\alpha =0.0167\). Asterisks indicate significant presaccadic contributions to the perceptual judgment.

 

Figure A.3.

 

Individual participant results in Experiment 3. Psychohpysical kernels for individual participants computed separately for judgments made about the (A) top and (B) bottom targets. Color scheme and line styles as in Figure 4.

Table A.2.

 

Statistical tests of presaccadic contributions to judgment in Experiment 2. The Bonferroni-corrected type I error rate for the individual t tests is \(\alpha =0.01\).

 

Table A.3.

 

Statistical tests of presaccadic contributions to judgment of top target in Experiment 3. The Bonferroni-corrected type I error rate for the individual t tests is \(\alpha =0.0083\).

 

Table A.4.

 

Statistical tests of presaccadic contributions to judgment of bottom target in Experiment 3. The Bonferroni-corrected type I error rate for the individual t tests is \(\alpha =0.0083\).