Serial dependence in orientation judgments at the time of saccades

Abstract

We actively seek information from the environment through saccadic eye movements, necessitating continual integration of presaccadic and postsaccadic signals, which are displaced on the retina by each saccade. We tested whether trans-saccadic integration may be related to serial dependence (a measure of how perceptual history influences current perception) by measuring how viewing a presaccadic stimulus affects the perceived orientation of a subsequent test stimulus presented around the time of a saccade. Participants reproduced the position, and orientation of a test stimulus presented around a 16° saccade. The reproduced position was mislocalized toward the saccadic target, agreeing with previous work. The reproduced orientation was attracted toward the prior stimulus and regressed to the mean orientation. These results suggest that both short- and long-term past information affects trans-saccadic perception, most strongly when the test stimulus is presented perisaccadically. This study unites the fields of serial dependence and trans-saccadic perception, leading to potential new insights of how information is transferred and accumulated across saccades.

Introduction

Visual input is constantly changing through saccadic eye movements, which poses a great challenge to perception. Many studies have found that perception is strongly suppressed and distorted during saccades (Binda, Cicchini, Burr, & Morrone, 2009; Burr, Holt, Johnstone, & Ross, 1982; Burr, Morrone, & Ross, 1994; Diamond, Ross, & Morrone, 2000; Morrone, Ross, & Burr, 2005; Ross, Morrone, & Burr, 1997). In spite of this, trans-saccadic perception remains stable: but how the visual system integrates presaccadic and postsaccadic information is largely unknown. 

Many studies over the past decade or so have investigated how perceptual systems incorporate past information into current perception, a phenomenon termed “serial dependence”. These studies have shown that perception of attributes such as orientation (Cicchini, Mikellidou, & Burr, 2017; Fischer & Whitney, 2014; Fritsche, Mostert, & de Lange, 2017), numerosity (Cicchini, Anobile, & Burr, 2014), spatial position (Bliss, Sun, & D'Esposito, 2017; Manassi, Liberman, Kosovicheva, Zhang, & Whitney, 2018), motion direction (Alais, Leung, & Van der Burg, 2017; Bae & Luck, 2020), facial identity and expression (Liberman, Fischer, & Whitney, 2014; Taubert, Alais, & Burr, 2016), or eye gaze (Alais, Kong, Palmer, & Clifford, 2018) are biased toward the previous stimulus. Besides simple features, the effects have also been confirmed with some complex features such as ensemble representations (Manassi, Liberman, Chaney, & Whitney, 2017) and variance (Suárez-Pinilla, Seth, & Roseboom, 2018), as well as in oculomotor behaviors (Darlington, Beck, & Lisberger, 2018; Goettker & Stewart, 2022). 

The effects have been modelled by an ideal observer model (Cicchini et al., 2014; Cicchini, Mikellidou, & Burr, 2018) (see also Equation 1 in modeling section), which resembles a Kalman filter, where the expected response is the weighted sum of current and previous stimuli. The model predicts that the weight given to the past stimulus should depend on several factors. One is the reliability (inverse variance) of the current stimulus, with the estimates of unreliable stimuli weighted more towards previous stimuli. Another important factor is the similarity between previous and current stimuli, with the weight of previous stimuli decreasing as the squared difference increases, reflecting a lower probability that the previously viewed stimulus was the same as the current one. Several studies have shown that the perceptual bias is not a defect of the system, but leads to more efficient perception (Cicchini et al., 2014; Cicchini et al., 2018). 

The experimental paradigm of these studies almost always involves the stimulus being changed, somewhat artificially, between trials within a static environment. However, spatiotemporal integration is especially important when we need to actively search for information from moving sensors, such as the ever-moving eyes. Evidence suggests that each impending eye movement is accompanied by a corollary discharge signal, causing many interesting physiological effects and perceptual phenomena. For example, “remapping neurons” predictively shift their receptive fields transiently before eye movements (Duhamel, Colby, & Goldberg, 1992; Wurtz, 2008); there is a strong spatiotemporal distortion of perisaccadically perception (Binda, Bruno, Burr, & Morrone, 2007; Binda et al., 2009; Morrone et al., 2005; Ross et al., 1997); saccadic eye movements also produce strong suppression to the information presented at the time of saccades (Benedetto & Morrone, 2017; Burr et al., 1982; Burr et al., 1994; Diamond et al., 2000). 

These phenomena suggest that perisaccadic stimuli are perceived unreliably, confirmed by direct measurements of position judgements at the time of saccades (Binda et al., 2007). Maintaining stability in the face of this uncertainty may rely on past experience shaping current perception. Besides very recent past information such as that driving “serial dependence,” long-term information may also play an important role in trans-saccadic perception. One clear example of the effects of long-term perceptual history is the “central tendency,” or “regression to the mean,” first reported by Hollingworth (1910): “Judgments of time, weight, force, brightness, extent of movement, length, area, size of angles all show the same tendency to gravitate toward a mean magnitude” (pp 461–462). More recently this idea has been expressed in Bayesian terms, which refers to the use of past knowledge as a prior and combining it with current sensory input (likelihood) to form the perception, termed the posterior (Cicchini, Anobile, Chelli, Arrighi, & Burr, 2022; Jazayeri & Shadlen, 2010). Within the Bayesian framework, the mean distribution of the stimulus derived from long-term past experience can serve as a prior, providing a possible distribution of the stimulus, and the short-term past experience updates the prior, both of which jointly influence the current perception. 

Given the Bayesian principles underlying integration of both short-term and long-term perceptual experiences with current perception, we expect that at the time of saccades the influence of both will be stronger than during fixation. To test this prediction, and to understand more fully trans-saccadic perception, we measured perceived position and orientation for stimuli presented around the time of saccades. To help understand the roles of contextual and current information during trans-saccadic perception, we modeled the data with an ideal observer model whose weights were measured directly in the participants. The study aims to connect trans-saccadic perception and serial dependencies, providing new insights into understanding patterns of information integration across saccades. 

Methods

Participants

Twelve observers participated in the experiment (six female). All participants had normal or corrected-to-normal vision and provided written informed consent before taking part in the study. The sample size was consistent with estimates from a power analysis, based on the design in which each participant was measured under three correlated conditions (three types of prior or test). Calculations performed with G*Power suggest that at least 11 participants were needed to achieve a moderate effect size with α = 0.05 and power = 0.80. With the exception of one author (X.I.E.), all were naive to the purpose of the experiment. Experimental procedures were approved by the regional ethics committee (Comitato Etico Pediatrico Regionale, Azienda Ospedaliero-Universitaria Meyer, Firenze, Florence, Italy) and were in line with the declaration of Helsinki. 

Apparatus

Experimental measures were performed in a quiet and dimly lit room. Stimuli were generated with Psychotoolbox in MATLAB r2020b (The MathWorks) and presented with PROPixx projector (VPixx Technologies Inc., Canada) by back projection onto a matte white PVC screen (Epson ELP-SC21B, 180 × 100 cm) with a resolution of 1920 × 1080 pixels and a refresh rate of 120 Hz. Participants sat 1 m from the screen, which subtended 90 × 50 cm. Head position was stabilized by a chin and headrest. 

The position of one eye was monitored at 1000 Hz with the EyeLink 1000 system (SR Research, Kanata, Canada). A nine-point calibration and validation sequence was made at the beginning of each session, and eye drift corrected at the beginning of each block. 

Stimuli and procedures

The test and prior stimuli were Gabor patches (Gaussian-windowed sinusoidal grating) presented at the center of a mean luminance screen background. The Gabor stimulus was 3 c/deg spatial frequency, 80% contrast, 1.36° standard deviation, and random phase on every presentation (Figure 1A). 

Figure 1.

 

Stimulus and Task. (A) Stimulus arrangement. Participants first fixated F0, and the prior stimulus was presented in the center. Then F0 disappeared, and F1 appeared. Participants were saccaded from F0 to F1 with a single saccade, and the test stimulus was presented in the center of the screen before or after the saccade onset. Participants moved the response bar (RB) to reproduce the position and orientation of the test stimulus. (B) Time-course of presentations, with zero corresponding to saccade onset. (C) Experimental conditions. The test stimulus comprised six orientations, the prior stimulus orientations relative to the test stimulus (±15° or 90°).

In the reproduction task, both perceived position and orientation of the Gabor stimulus were reproduced by the participants. Trials started with participants viewing a fixation dot (F0), 8° left of screen center. The presaccadic prior stimulus was briefly presented for 17 ms (two monitor frames). After 800 ∼ 1200 ms random interval, F0 disappeared and a saccadic target (F1) appeared immediately 8° right of screen center. Participants saccaded directly to F1. After a random delay (10 ∼ 400 ms) from the saccadic target, the test stimulus, a brief 17-ms Gabor patch was displayed in the center of the screen (Figures 1A, B). Participants first moved a response dot to their perceived position of the test and then reproduced the test orientation by rotating the response bar by mouse. If participants did not see the test at all, they could skip the trial without responding. To ensure that participants did not completely ignore the prior stimulus, they were required to report the approximate orientation of the prior stimulus by pressing the left or right arrow at the end of 10% trials. 

For each block, there were 60 trials in total. Six test orientations (±35°, ±45°, ±55°) were randomly assigned in each trial (10 trials for each orientation), and the prior orientation was randomly chosen from the positive (Test + 15°), negative (Test − 15°), and orthogonal (Test + 90°) conditions (Figure 1C). Each participant performed 15-20 blocks. 

After the reproduction task, participants made a further two-alternative forced-choice (2-AFC) orientation judgment task to measure orientation thresholds of prior and test stimuli. Stimulus parameters and procedures were similar to the reproduction task. When measuring the orientation threshold of the prior stimulus, participants kept fixation at F0, and a 17-ms Gabor stimulus was presented in the center of the screen. When measuring orientation thresholds of the test, participants were required to saccade from F0 to F1, and a 17-ms Gabor stimulus presented after the appearance of F1 at time T (details see below). After the stimulus disappeared, participants were required to press the left or right arrow to judge whether the response bar was more clockwise or counter-clockwise relative to the stimulus. 

There were three different conditions for time T: T_lat − 80 ms, T_lat ms, and T_lat + 80 ms, corresponding to the three situations where the test stimulus presented presaccadically, perisaccadically, and postsaccadically, respectively (latencies chosen individually for each participant, depending on their saccadic latencies). Each block took one of these conditions. The experiment used the constant stimuli method, and the orientation difference between the stimulus and the response bar for each trial was randomly selected among 7 preset orientations. Each orientation difference was repeated 10 times, with a total of 70 trials in a block. For each experimental condition (prior, test-pre, test-peri, test-post), a total of four blocks were completed. 

Data analysis

To analyze the saccade data, we used EyeLink parsing algorithms. Saccades were distinguished from fixations based on specific thresholds: the saccadic velocity threshold was set at 35°/s, the saccadic acceleration threshold was set at 9500°/s², and the saccadic motion threshold was set at 0.15°. We excluded saccades with an amplitude smaller than 1° to eliminate the effects of micro-saccades. 

Before analyzing reproduced position and orientation errors, we selected valid trials based on the saccade and response. Three types of trials were discarded from data analysis. 

No-response trials

Participants did not see the test stimulus, so they pressed a key to skip the trial. 85.0% no-response trials occurred when the stimulus appeared between −20 to 30 ms from saccade onset (Figure 2A), indicating that the stimulus was highly suppressed around saccade onset. 

Figure 2.

 

Saccade and response information of each trial. (A) The number of no-response trials when the test stimulus was presented at each 8.33 ms time bin from saccade onset. (B) Distribution of saccadic landing error of all (blue bar) and valid (yellow bar) trials. (C) Mean number of valid trials across participants. Error bars represent the standard error of the mean across participants. (D) Distribution of saccade latencies. (E) Distribution of saccade durations. (F) The number of trials where the test stimulus was presented during the first saccade (blue line) or out of the first saccade (yellow line) at each time bin.

Unreasonable-response trials

Orientation reproduction error exceeded two standard deviations of the mean (i.e., reproduction error larger than 36°). 

Bad-saccade trials

The first saccade landing error more than two standard deviations of the skew-normal fitted mean (i.e., landing error >6.6° or <−4.6°) (Figure 2B). In addition, the trials where the fixation before the first saccade was 3° away from F0 would also be labeled as a bad-saccade trial. 

A total of 30.3% of trials were discarded, leaving 777 (SD = 103) valid trials of all participants, comprising 10.3% no-response trials, 3.1% unreasonable-response trials, and 16.8% bad-saccades trials. There was no significant difference in the number of valid trials across different test orientations or prior orientations (repeated-measures analysis of variance [ANOVA], for test orientation, F(2, 22) = 0.166, p = 0.848 \(\eta _p^2\) = 0.015; for prior orientation, F(2, 22) = 1.955, p = 0.165, \(\eta _p^2\) = 0.151, Figure 2C). 

We also analyzed saccadic information to ensure that the results in different time bins were relevant to eye movement. The average saccade latency of valid trials was 156.6 ms (Figure 2D), and the average saccade duration was 72.8 ms (Figure 2E). Thus 16.8% of the test stimulus were presented during the first saccade (Figure 2F, blue line). When we split the data into five time bins to analyze perceived position and orientation errors (Figures 3 and 4), for the perisaccade period (0 ∼ 25 ms), all trials occurred during the saccade, for the postsaccade period (25 ∼ 65 ms), 86.3% of trials, and for the far-postsaccade period (65 ∼ 200 ms), only 4.5% of trials. This shows that the five time-bins do reflect the trans-saccade perception in different stage of saccade. 

Figure 3.

 

Errors of perceived position at the time of saccades. (A) Perceived position of the test Gabor stimulus presented briefly around the time of a 16°-saccade with different prior stimulus orientations. The data of all participants are merged and averaged in each 8.33-ms time bin. The light-colored areas represent the standard error of all trials in each time bin. (B) Same data as in A with coarser time bins. The data are split into far-presaccade (< −65 ms), presaccade (−65 ∼ −15 ms), perisaccade (−15 ∼ 25 ms), postsaccade (25 ∼ 65 ms), far-postsaccade (> 65 ms). (C) Position errors with different test stimulus orientations. Error bars represent the standard error of the mean across participants.

Figure 4.

 

Biases in orientation perception at the time of saccades. (A, B) Reproduced orientation errors for the three different priors. (C, D) Reproduced orientation errors for the three different test orientations. (E) Reproduced orientation errors for different test and prior stimulus orientations. In A and C, the data of all participants are merged and averaged within each 8.33 ms time bin. The light-colored areas represent the standard error of all trials in each time bin. In B, D, and E, the orientation errors are first calculated for each participant at far-presaccade, presaccade, perisaccade, postsaccade, and far-postsaccade, and then average the error of each participant. Error bars represent the standard error of the mean across participants.

The perceived position and orientation errors of each trial were calculated as the difference between reproduced position/orientation and the test stimulus position/orientation. To monitor changes in reproduced position and orientation, data were averaged in 8.33 ms time bins from −200 ms to 200 ms. As patterns of errors for positive and negative angles were similar, to simplify the results, when the test orientation was in the fourth quadrant, we flipped all the orientations (prior orientation, test orientation, reproduced orientation) along the x-axis in that trial. In effect this means that a positive error was actually in the vertical direction, and that a positive prior stimulus was more vertical than the test. 

The orientation thresholds of prior and test stimuli were calculated from the 2AFC task. Specifically, we pooled responses from all participants to obtain the “aggregate participant.” Proportion of “more clockwise” responses were plotted as a function of orientation difference to yield psychometric functions, which were fitted with the logistic function: \(p = \frac{1}{{1 + {{\rm{e}}^{( { - {\rm{k*}}( {{\rm{\theta }} - {{\rm{\theta }}_{50}}} )} )}}}}\), where p is proportion of more clockwise choice, θ is the relative orientation and k is a constant determining curve width. Threshold, or just noticeable difference (JND) was estimated as: JND = (θ₆₈ − θ₃₂)/2, where θ₆₈ and θ₃₂ were the relative orientation when making the more clockwise choice for 32% and 68% of trials. When calculating the threshold for the test depending on when the stimulus appeared relative to saccade onset, the data were divided into presaccade (<−15 ms), perisaccade (−15 ∼ 25 ms), and postsaccade (> 25 ms). 

Statistics were performed in JASP (12.1.0). Repeated-measures ANOVA was used to compare the effects of prior stimulus orientation, test stimulus orientation, and time from saccade onset on position and orientation errors. 

Results

Mislocalization effect

Participants first reproduced the position of a trans-saccadic test stimulus. The test was always at screen center, but the perceived position was mislocalized toward the saccadic target when the test was presented around the saccade onset (Figure 3A). The data were split into five time bins based on when the stimulus presented at different times from saccade onset (Figure 3B). The strongest mislocalization effect was 2.95° ± 0.67° (perceived position in far-presaccade minus perceived position in perisaccade), which was significantly higher than 0 (t-test, t(11) = 4.421, p = 0.001, Cohen's d = 1.276). This shows that the stimulus is strongly distorted and suppressed during saccades, so any results in orientation perception in this experiment should reflect trans-saccadic perception. The mislocalization effect is smaller than the effects reported in the previous research (Binda et al., 2009), possibly because of longer duration than usually used (17 ms vs. 4 ms). In addition, the position task was orthogonal to the orientation task, as evidence shows that neither prior orientations nor test orientations affect the perception of test positions (repeated-measures ANOVA, for prior orientation, F(2, 22) = 0.133, p = 0.876, \(\eta _p^2\) < 0.001; for test orientation, F(2, 22) = 3.040, p = 0.068, \(\eta _p^2\) = 0.216, Figures 3B and 3C). 

Serial dependence in orientation perception

To understand how prior and average stimulus orientation affect trans-saccadic perceived orientation, we plotted the orientation errors around saccade onset for different test and prior stimuli (Figure 4). The results show a clear priming effect in which the reproduced orientation was attracted toward the prior stimulus, so the error was positive (1.22° ± 0.91°) when the prior was 15° and negative (−2.04° ± 0.98°) when the prior was −15° from the test; when the prior was orthogonal to the test, the orientation error fluctuated around 0° (−0.52° ± 0.89°) (Figures 4A and 4B). The attraction effect was strongest when the test was perisaccadic: the mean of bias with positive prior and flipped bias with negative priors was 3.14° ± 0.63° at perisaccade, whereas it was only 1.79° ± 0.35° at far-presaccade and 1.09° ± 0.31° at far-postsaccade. Apart from recent past information, long-term mean information can also affect current perception, which was clearly confirmed in our results. We found that the mean perceived orientation was 37.4° ± 1.26° and 51.6° ± 1.15° when the test was 35° and 55°, respectively, which means that the reproduced orientation regressed to the mean orientation (±45°) (Figures 4C and 4D). This central tendency effect was also strongest at perisaccade, which was 4.43° ± 0.93°, compared to 2.80° ± 0.83° and 2.57° ± 0.81° at far-presaccade and far-postsaccade, respectively. 

To test the significance of the attraction towards the prior and test, we first performed a three-way repeated-measures ANOVA (prior × test × saccade), which showed a significant main effect of prior stimulus (F(2, 22) = 33.973, p < 0.001, \(\eta _p^2\) = 0.755) and test stimulus (F(2, 22) = 16.091, p < 0.001, \(\eta _p^2\) = 0.594), but no main effect of time from saccade onset (F(4, 44) = 1.039, p = 0.398, \(\eta _p^2\) = 0.086). However, there was a significant interaction between prior and saccade (F(8, 88) = 3.079, p = 0.004, \(\eta _p^2\) = 0.219), and among the three factors as well (F(16, 176) = 3.285, p < 0.001, \(\eta _p^2\) = 0.230) (Figure 4E). The insignificant main effect of time from saccade onset was as expected because the error was bidirectional. We therefore repeated the analysis defining positive error as “toward the prior” (in practice flipping the results for priors of −15°). This ANOVA revealed a highly significant main effect of time from saccade onset (F(4, 44) = 8.521, p < 0.001, \(\eta _p^2\) = 0.436). To interpret the significant three-way interaction effect, we conducted two two-way ANOVAs for the test-neutral condition (Test = 45°) and prior-neutral condition (Prior = 90°) to analyze the priming and central tendency effects separately. In the test-neutral condition, we observed a significant priming effect (F(2, 22) = 27.183, p < 0.001, \(\eta _p^2\) = 0.712), which also interacted with saccade (F(8, 88) = 4.813, p < 0.001, \(\eta _p^2\) = 0.304). The simple main effect analysis further showed that the saccade effect was significant when the prior was 15° (F(4, 44) = 3.109, p = 0.024) or −15° (F(4, 44) = 4.771, p = 0.003), indicating that the priming effect was modulated by the time from saccade onset. Similarly, in the prior-neutral condition, we observed a significant central tendency effect (F(2, 22) = 15.191, p < 0.001, \(\eta _p^2\) = 0.580), which also interacted with saccade (F(8, 88) = 3.656, p = 0.001, \(\eta _p^2\) = 0.249). Further analysis revealed a significant simple main effect of saccade when the test was 35° (F(4, 44) = 6.447, p < 0.001), indicating that the central tendency effect was modulated by the time from saccade onset as well. 

From the perspective of memory, the response of participants combined the perceived orientation with a randomly chosen orientation. Mixture models (Zhang & Luck, 2008) are a good way to separate guess rate from response accuracy without discarding large orientation errors, and the difference between actual and modeled simulated orientation can reveal serial dependence effects. We applied the mixture model to our data, yielding results consistent with our findings that perisaccade perception has higher uncertainty, and also a stronger serial dependence effect (see Appendix for details). 

Modeling

To understand better how prior and historical average information jointly affect trans-saccadic perception at different stages of saccade onset, we modeled the time-dependent changes in priming and central tendency effects. We simulated the response of an ideal observer, whose response is the weighted sum of current and past stimuli, all weighted by their reliability (inverse variance) (Cicchini et al., 2018). In this experiment, the information about the past was both short- and long-term, and their effect on perception interacted (Figure 4E). Therefore we modified the ideal observer model to consider the influence of long-term average information on current perception. The weights for an ideal response were selected to minimize the overall mean squared error of the perceived orientation. 

Specifically, the response of an ideal observer to the current stimulus is the weighted sum of the current stimulus, of the prior stimulus, and of the average orientation for the entire experiment (45°):  

\begin{eqnarray}&& {{\rm{R}}_{{\rm{curr}}}} \nonumber \\ && = {{\rm{w}}_{{\rm{prior}}}}{{\rm{S}}_{{\rm{prior}}}} + {{\rm{w}}_{{\rm{av}}}}{{\rm{S}}_{{\rm{av}}}} + \left( {1 - {{\rm{w}}_{{\rm{prior}}}} - {{\rm{w}}_{{\rm{av}}}}} \right){{\rm{S}}_{{\rm{curr}}}}, \quad \end{eqnarray}

(1)

where R_curr is the response to the current, S_prior, S_av, and S_curr are the prior, average, and current stimulus, respectively, and the weights given to the prior and average are w_prior and w_av. The weight of the current stimulus is derived from the other two, because the weights are constrained to sum to unity. 

The overall variance of the responses is:  

\begin{eqnarray} && {\rm{VAR}} \nonumber \\ && = {{\rm{w}}_{{\rm{prior}}}^2{\rm{\sigma }}_{{\rm{prior}}}^2{\rm{\;}} + {\rm{w}}_{{\rm{av}}}^2{\rm{\sigma }}_{{\rm{av}}}^2{\rm{\;}} + {\left( {1 - {{\rm{w}}_{{\rm{prior}}}} - {{\rm{w}}_{{\rm{av}}}}} \right)^2}{\rm{\sigma }}_{{\rm{curr}}}^2.} \quad \end{eqnarray}

(2)

 

The bias is given by both the prior and average:  

\begin{eqnarray}{\rm{BIAS}} = {{\rm{w}}_{{\rm{prior}}}}{{\rm{d}}_{{\rm{prior}}}} + {{\rm{w}}_{{\rm{av}}}}{{\rm{d}}_{{\rm{av}}}} \quad \end{eqnarray}

(3)

where d_prior and d_av represent the difference between the current and the prior or average stimulus. The overall mean squared error (ERR) is:  

\begin{eqnarray}{\rm{ERR}} = {\rm{\;BIA}}{{\rm{S}}^2} + {\rm{VAR}}. \quad \end{eqnarray}

(4)

 

Thus the optimal response is to find appropriate w_prior and w_av that minimize ERR. 

To this end, we first measured thresholds to estimate the variance of prior and current stimuli. The thresholds (JNDs) of prior and test stimuli (including test at presaccade, perisaccade, and postsaccade) represent the variance of prior and current stimuli, which are σ_prior = 4.57, \({{\rm{\sigma }}_{{\rm{test}}\_{\rm{pre}}}} = 8.8\), \({{\rm{\sigma }}_{{\rm{test}}\_{\rm{peri}}}} = 20.68\), \({{\rm{\sigma }}_{{\rm{test}}\_{\rm{post}}}} = 5.83\) (Figure 5). Because the variance of average information cannot be measured directly, it is set as a free parameter. 

Figure 5.

 

The psychometric functions and thresholds (JNDs) of prior (A) and test (B) stimuli for 2AFC orientation discrimination tasks. The psychometric functions show the proportion of “more clockwise” responses from the “aggregate participant” with each relative orientation. The thresholds are the mean of the relative orientation when making more clockwise choice for 32% and 68% of trials.

To estimate the variance to attribute to central tendency (\({\rm{\sigma }}_{{\rm{av}}}^2\)), we calculated the predicted orientation errors with this model separately for six conditions: 3 test conditions (35°, 45°, 55°) × 2 prior conditions (15°, −15°). The results with prior of 90° were not used because the orthogonal prior should not affect the response (see Figure 4B: the mean orientation bias was only −0.17° ± 0.12° when the prior was orthogonal to the test. Previous serial dependence studies also showed that the serial dependence almost disappears when the relative orientation is more than 30° (Cicchini et al., 2018). For each estimate, \({\rm{\sigma }}_{{\rm{av}}}^2\) varied between 0 and 225, and the weights of prior and average stimulus calculated to minimize ERR. The resulting best fit to yield the largest goodness of fit was given by \({\rm{\sigma }}_{{\rm{av}}}^2 = 62\) (σ_av = 7.9). This estimate, together with the measured variances of the prior and current stimuli, were used to predict the results of Figure 6A (dashed lines). The ideal response captures the general trend of the data. 

Figure 6.

 

The modeling results. (A) Model-estimated orientation errors (dashed lines) and experimentally measured orientation errors (solid lines) with different test and prior orientations. (B) Weights of priming effect, central tendency effect, and current stimulus are obtained by the model.

Figure 6B plots the weights of the ideal observer model as a function of time from saccade. The blue curve shows the weighting of the current stimulus, around 0.5 for presaccadic and postsaccadic stimuli, and dropping to 0.1 for perisaccadic stimuli. This means that even in normal viewing, the weight of the current stimulus is about 50%, with the other two contextual cues contributing to the other 50% of the response. But for perisaccadic stimuli, the orientation of the current stimulus contributed only 10% of the response, with 90% coming from contextual cues. Central tendency has higher weighting than the prior, mainly because the weighting varies inversely with the difference in orientation (d in Equation 3). For central tendency (toward 45°), d = ±10° (test stimuli either 35 or 55°), whereas the prior was always ±15° from the test. Moreover, the difference in weighting for central tendency effect between presaccade/postsaccade and perisaccade is higher than that of priming effect. This result probably causes overestimation of the error in test of 35° and 55° for the perisaccade period, and underestimation of the error in test of 45° for the perisaccade period, which will be further discussed. 

Discussion

The present study investigated trans-saccadic position and orientation perception. We found that perisaccadic perception is strongly suppressed and also mislocalized, as many previous studies have found (Binda et al., 2007; Binda et al., 2009; Ross et al., 1997). However, we also showed that the perceived orientation of perisaccadic stimuli is integrated with information derived from previous stimuli, including the stimulus presented just before the saccade and the average orientation accumulated across many past stimuli. Consistent with serial dependence effects, stimuli with greater uncertainty depended more strongly on past experience, confirming that serial dependence effects exist not only in static, sequentially presented stimuli, but also in more natural dynamic perceptual processes. 

Serial dependence highlights the generative nature of perceptual processing: the idea that the brain generates a model of the world based on past experiences and updates the model using current sensory data. The generative process is particularly important when we actively explore the environment through mobile sensors. For example, the oculomotor system needs to be informed about each impending saccade and disentangle the self-generated signals (such as retinal motion and displacement) from external sensory signals, as neurons in early visual cortex shift their receptive field, which called predictive remapping (Burr & Morrone, 2011; Duhamel et al., 1992). 

In addition to the involvement of early visual cortex, our results are consistent with the idea that the effects of past experience involve a post-perceptual, top-down process. For example, the priming effects do not differ if the stimulus is presented presaccadically or postsaccadically (Figure 4A), even though the presaccadic and postsaccadic signals come from different parts of the retina (but the same part of external space). They are equally integrated with the current stimulus, suggesting the action of higher, spatiotopic mechanisms. Although some previous studies suggested that serial dependence occurs only within a retinotopic spatial window (Collins, 2019), other studies showed that postsaccadic visual processing can be modulated by presaccadic stimulus (Buonocore, Dimigen, & Melcher, 2020; Huber-Huber, Buonocore, Dimigen, Hickey, & Melcher, 2019), even when they are not retinotopic. Serial dependence of orientation perception has also been shown to be spatiotopic (Mikellidou, Turi, & Burr, 2017), depending on the orientation of the stimulus in space when the head tilts between trials, changing the orientation of stimuli on the retina. Other evidence for the involvement of high-level processes comes from the central tendency effect. The average orientation is dynamically updated from past perceptual experiences over a period of time: the accumulation must involve higher-level cognitive processes as well as its influence on the current percept. Although the priors and average percept are probably calculated at a high level, most evidence suggests that they act early in visual processing (Cicchini, Benedetto, & Burr, 2021; Cicchini et al., 2017). The results of this study are not inconsistent with that claim. 

The modeling results support the idea that priming and central tendency effects work together and that they both exert a stronger influence at the time of saccades. The fits of the modeling were not perfect, but they capture the general trend. In particularly, the best-fitting weights of both the central tendency and priming effects were maximal for perisaccadic stimuli, showing that the contextual effects are strongest there. However, there is a systematic prediction error: the model underestimates error in Test_45 and overestimates error in Test_35, and Test_55 for the perisaccadic stimuli. This is probably because of the higher weight of central tendency for the perisaccade period. The weight calculation is based on the ideal observer's response, in which the weight depends on the reliability and similarity of contextual and current stimulus. But for a human observer, one may give a higher weight for recent past information when the current sensory input is unreliable, because the world should remain stable over the short period for human observers. However, our current model did not take this temporal contextual information into consideration. This systematic prediction error may reveal the importance of recent information in perisaccadic perception. In general, the modeling reinforces the utility of the ideal observer model for serial dependence effects and confirms that serial effects scale with stimulus uncertainty and similarity of current and previous stimuli. 

This study reveals how past experience affects trans-saccadic perception, but we still lack the understanding of the neural mechanism underlying how past experiences are stored and transmitted in the brain. One possibility is that propagation and updating of the prior is reflected in neural oscillations at low frequencies (Friston, Bastos, Pinotsis, & Litvak, 2015). Some evidence has shown alpha oscillations of responses in face recognition (Huang, Chen, & Luo, 2015) and gender classification (Bell, Burr, Crookes, & Morrone, 2020), and auditory detection and discrimination (Ho, Burr, Alais, & Morrone, 2019), when the prior and current stimulus are congruent. Thus it seems likely that during saccades previous information is transmitted through alpha oscillations and that oscillations are synchronized with saccadic onset. Indeed, studies have found that saccadic eye movements synchronize perceptual oscillations in sensitivity (Benedetto & Morrone, 2017; Hogendoorn, 2016; Neupane, Guitton, & Pack, 2017; Wutz, Muschter, van Koningsbruggen, Weisz, & Melcher, 2016). Preliminary evidence from our laboratory suggests the presence of oscillatory behavior of orientation synchronized with saccades. However, how the oscillation changes with the prior still needs to be determined. 

Interestingly, although orientation perception changed with prior orientation, we did not observe any serial dependence on localization performance. This may seem to contrast with psychophysical (Cicchini, Binda, Burr, & Morrone, 2013) and neurophysiological results (Duhamel et al., 1992) showing that congruency of information between presaccadic and postsaccadic stimulation influences localization and induces a compression toward the most reliable stimulation than in our condition in the prior stimulus. However, in previous studies the presaccadic stimulation was temporally closer to saccadic onset, whereas here the lag could be as large as two seconds. This suggests that the memory trace of serial dependence change the decaying time depending on the specific analysis. Orientation is associated with a stable property of the object and, hence, could have a stronger accumulation time, whereas localization is a volatile property that is altered dynamically with all our movements, and, hence, it would be detrimental to be modulated by experience over a large temporal window (Taubert et al., 2016). 

Conclusion

In conclusion, we combine the ideal observe model to the concept of oriented spatiotemporal receptive fields to study trans-saccadic perception. To preserve visual continuity, the visual system combines presaccadic with postsaccadic signals from different parts of the retina, as a form of serial dependence in space and time. We unite the fields of serial dependence and trans-saccadic perception, leading to important new insights of how information is accumulated across saccades. 

Acknowledgments

Supported by the European Union (EU) and Horizon 2020—grant agreement No. 832813—ERC Advanced “Spatio-temporal mechanisms of generative perception—GenPercept” and Natural Science Foundation of China Grant 31900799. 

Data availability: All data that support our findings are publicly available at https://drive.google.com/drive/folders/17kMmJ-tt8LlZi1M5o-NiOpd3FBKohayc?usp=drive_link. 

Commercial relationships: none. 

Corresponding author: David C. Burr. 

Email: [e-mail]. 

Address: Department of Neuroscience, Psychology, Pharmacology, and Child Health, University of Florence, Florence, Italy. 

Appendix

Our task requires participants to report the perceived orientation of a stimulus presented during saccade, a task that is also limited by the capacity of working memory storage. Mixture models in working memory studies (e.g., Zhang & Luck, 2008) assume that reported stimuli include perceived stimuli stored in memory, as well as randomly guessed stimuli. MemToolbox (Suchow, Brady, Fougnie, & Alvarez, 2013) was created to fit working memory data via Bayesian and maximum likelihood estimation procedures (see Suchow et al. (2013) for detailed methods). 

We fitted our reproduced orientation data in presaccades, perisaccades and postsaccades separately by MemToolbox. The mixture model estimates two main parameters, one being “guess rate” and the other being “standard deviation,” which reflects the precision of perception. The results in Figure A1A show that perisaccade perception has higher guess rates and higher standard deviations, indicating greater uncertainty and lower precision in perisaccadic perception. Additionally, the residuals between the model simulated data and the actual data also reflect serial dependence effects. Specifically, when the prior is 15°, there are more positive residuals when the relative orientation is around 15°, and more negative residuals around −15°, indicating that the model simulation underestimated the orientation error because the model did not take priors into consideration. When the prior is −15°, the residual patterns are opposite (Figure A1B). Moreover, perisaccadic trials had more pronounced residuals. Therefore we believe that the results of the mixture model are consistent with the current study's findings and support our conclusion from another perspective. 

Figure A1.

 

Mixture model results. (A) Guess rates and standard deviations of the perceived orientation in presaccadic, perisaccadic, and postsaccadic trials calculated by mixture model. (B) Differences between actual and model simulated orientations in presaccadic, perisaccadic, and postsaccadic trials.