Serial dependencies act directly on perception

Abstract

There is good evidence that biological perceptual systems exploit the temporal continuity in the world: When asked to reproduce or rate sequentially presented stimuli (varying in almost any dimension), subjects typically err toward the previous stimulus, exhibiting so-called “serial dependence.” At this stage it is unclear whether the serial dependence results from averaging within the perceptual system, or at later stages. Here we demonstrate that strong serial dependencies occur within both perceptual and decision processes, with very little contribution from the response. Using a technique to isolate pure perceptual effects (Fritsche, Mostert, & de Lange, 2017), we show strong serial dependence in orientation judgements, over the range of orientations where theoretical considerations predict the effects to be maximal. In a second experiment we dissociate responses from stimuli to show that serial dependence occurs only between stimuli, not responses. The results show that serial dependence is important for perception, exploiting temporal redundancies to enhance perceptual efficiency.

Introduction

The visual world tends to be stable over time, at least in the short term. This stability, or redundancy, is routinely exploited by video compression algorithms to achieve hundred-fold compression ratios. Biological perceptual systems also take advantage of the continuity, revealed by several experimental paradigms. For example, when asked to reproduce or classify stimuli, subjects typically err towards the previous stimulus, exhibiting so-called “serial dependence,” a form of compulsory averaging of features of successive stimuli, exploiting temporal redundancies to increase signal-to-noise ratios (Burr & Cicchini, 2014; Fischer & Whitney, 2014; Kiyonaga, Scimeca, Bliss, & Whitney, 2017). These effects have been observed for orientation (Fischer & Whitney, 2014; Liberman, Zhang, & Whitney, 2016; Manassi, Liberman, Chaney, & Whitney, 2017), numerosity (Cicchini, Anobile, & Burr, 2014; Corbett, Fischer, & Whitney, 2011), facial gender and expression (Kondo, Takahashi, & Watanabe, 2012; Liberman, Fischer, & Whitney, 2014; Xia, Leib, & Whitney, 2016) and even beauty (Kondo et al., 2012). Serial dependence supports generative theories of perception such as “predictive coding,” which propose that perception is based on stable internal models of the world that are continually updated by sensory information, so both past and present experiences shape our immediate perception (Friston, 2009). 

While the idea that serial dependence results from integration of sensory information is theoretically compelling, it is not clear as to whether it occurs in sensory systems or in later decision stages (or both). Fischer et al.'s (2014) original report provided evidence that serial dependence occurs even when observers are not asked to perform a behavioral response, showing that these mechanisms are in part automatic. Consistently BOLD activity in primary visual cortex (V1) in response to an orientation is influenced by the orientation of previous stimuli presented in the same position, even when no response was requested (St John-Saaltink, Kok, Lau, & de Lange, 2016). 

On the other hand, Fritsche, Mostert, and de Lange (2017) have recently claimed that positive serial dependence, which is typically measured in reproduction tasks, occurs at a postperceptual decisional stage rather than at the level of perception. They devised a clever technique to separate perception from decision, where subjects simply stated whether two stimuli (one of which had been preceded by an oriented inducer) had the same or different orientation. The effects of prior stimuli were then measured by shifts in the response distributions (see Figure 1A). They measured perceptual biases only for inducers oriented 20° away from the current stimulus (as this matched the peak effect in the reproduction trial), and found a repulsive adaptation effect of 1.4° (about 3.5%) and concluded that the positive serial effects found with reproduction tasks result from decision rather than perceptual processes. 

Figure 1

 

Serial effects vary with interstimulus orientation difference and sensory reliability. (A) Illustration of the technique, with sample data for inducer-biased stimulus difference of ±10°, clockwise (blue symbols) or counterclockwise (red) from the test. Data have more than 1,500 trials per curve, error bars show binomial SEM. (B) Main effect of inducer as a function of inducer-biased stimulus orientation difference. Error bars are SEM calculated by bootstrap. The star shows results for conditions similar to those used by Fritsche et al. (2017; with white response bar); the square reproduces their results. (C) Weight of inducer as function of orientation difference. Trials separated by orientation (and hence precision) of the comparison stimulus. Stimuli close to the diagonals are plotted in orange; stimuli close to the cardinal axes are in green. The thick color-coded lines show the predictions of an ideal observer model, which optimizes performance as function of distance and sensory reliability (Cicchini et al., 2014), assuming either high judgement noise (4.5°) or low (1.5°). Isolated asterisks indicate statistical significance *p < 0.05, **p < 0.01. (D) Comparison between reproduction and comparison biases. Black symbols are data from the comparison task, green and orange symbols from reproduction. Dashed green and orange lines show the fit of the Bayesian model with additional postperceptual noise of 7°. Shaded region span ±1 SEM.

One of the key features of serial dependence is that it is strongest when two successive stimuli are not too dissimilar, a feature that has been likened to a Kalman filter, which reduces noise by adaptively averaging only stimuli that do not differ by very much (Burr & Cicchini, 2014). We therefore repeated the experiment of Fritsche et al. (2017) over a wide range of stimulus differences. Crucially, we found that when successive stimuli are quite similar, perceptual serial effects were positive. We also devised a technique to separate serial dependence in stimuli and response, which showed that the bulk of the dependence depended on the stimuli, and that motor decisions play little role. Taken together our results indicate that serial dependence is a common trait of both perceptual and postperceptual decision processes. 

Methods

Experiment 1

Sixteen participants (10 female, age 26 ± 5 years) took part in the experiment and gave informed consent according to the Declaration of Helsinki. All participants except the two authors were naive to the purpose of the experiment. Stimuli were presented on the face of a calibrated 23-inches LCD monitor of 26° horizontal eccentricity and 14.5° vertical eccentricity. 

The experimental paradigm was a slight variation of experiment 3 of Fritsche et al. (2017). Each trial began with the presentation of an eccentric inducer stimulus (Gabor stimulus, spatial frequency 0.3 cycles per degree [cpd], contrast 25%, 500 ms, 3.2° full-width half-height) followed by a mask (random noise filtered at 0.3 cpd, contrast 50%, 1000 ms), at a random location either rightward or leftward of fixation (8° horizontal, 4° vertical eccentricity). Participants were first asked to reproduce the inducing Gabor orientation by moving the mouse and setting the orientation of a virtual line marked at its ends by two small circles (diameter 0.2°) to match the inducer orientation. Participants confirmed the orientation with the space bar and the reproduction cursor disappeared. Two seconds later, two Gabors followed by masks were presented and subjects asked to indicate by keypress whether they were identical. Orientation of the Gabors was chosen so that the Gabor appearing in the same position of the inducer had a fixed orientation difference from the inducer (from 5° to 20°, clockwise or counterclockwise). The other Gabor stimulus was oriented ±9° from the comparison Gabor (in steps of 3°), and presented on the opposite side of the screen. Other stimulus and mask parameters were identical to those of the inducer presentation. One second after the comparison had been made a new trial began. 

The paradigm is a close replication of that of Fritsche et al. (2017). The main difference was replacing the white response bar (which could in principle perturb serial dependencies). Instead, we employed the two circles of similar mean luminance to the background, not presented in the same region of the Gabor. Other minor changes include the eccentricity of the stimuli (8° vs. 10°) to keep stimuli far from the monitor border and the absence of trial cuing. Each participant completed at least 1,500 trials spread over the five experimental conditions (four orientation differences with two small circles and one condition with 20° orientation difference and white response bar). Participants were typically tested in two sessions of 4 hours, spaced no more than 10 days apart. For each participant, at the beginning of every five sessions the program calculated an ordering of conditions for the upcoming sessions by randomizing the five experimental conditions. This ensured that conditions were evenly distributed through the course of data collection, minimizing systematic training or fatigue effects. 

Data were analysed by plotting probability “perceived same” as a function of orientation difference between the two comparison stimuli for the two inducer stimuli (see Figure 1a for an example). We estimated the inducer effect in two alternative ways. In one method, we performed the cross-correlation between the two curves, with linear interpolation between data points (as in Figure 1a). This is a nonparametrical way to assess the shift between two curves. In the second method, we followed Fritsche et al. (2017) and fitted the curves with Gaussian functions taking the difference between peaks as the main effect. The two methods yielded similar results, but we preferred the cross-correlation approach, as it makes no assumption on the normality of the responses. Similar to Fritsche et al., we removed trials in which the response in the reproduction task was more than 3 SDs (i.e. 27°) from the average response, indicating a potential attentional lapse during the presentation of the inducer. We also excluded reproduction responses that were less than 500 ms or more than 3500 ms. In Figure 1C and D we classified trials as near-cardinal if the orientation of the comparison stimulus at the inducer location was within 15° of 0° or 90°, and near-oblique if within 15° of ±45°. 

Statistical significance was assessed with the same permutation test used by Fritsche et al. (2017). Briefly we ran permutations of the original dataset resampling with replacement of the subjects and shuffling the labels clockwise/counterclockwise to create a distribution of effects under the null hypothesis. The test was performed 10,000 times and provided one-tail alpha values for each condition (which we then doubled to obtain two-tailed p values). Error bars were calculated by bootstrap for raw data points (such as in Figure 1a). For the main effects (such as in Figure 1B and C) we simulated 10,000 bootstrap experiments, creating at each iteration an aggregate observer resampling participants with replacement. For each condition we then resampled responses from the aggregate observer with replacement. Standard deviation of the simulated effects yielded standard errors. 

Experiment 2

Seven participants (three female, age 29 ± 6 years) carried out the experiment and gave informed consent according to the Declaration of Helsinki. All participants except the two authors were naive to the purpose of the experiment. Stimuli were presented on the face of a calibrated 23-inches LCD monitor of 26° horizontal eccentricity and 14.5° vertical eccentricity. 

The experimental paradigm was a close replication of the adjustment paradigms of Fischer et al. (2014) and Fritsche et al. (2017). Each trial began with the presentation of an eccentric inducer stimulus (Gabor stimulus, spatial frequency 0.3 cpd, contrast 25%, 500 ms, 3.2° full-width half-height) followed by a mask (random noise filtered at 0.3 cpd, contrast 50%, 1000 ms), either rightward or leftward of fixation (8° horizontal, 4° vertical eccentricity). Subjects were instructed to reproduce either the orientation of the Gabor patch or its mirror (flipped about the vertical axis) by moving the mouse and setting the orientation of a small bar (width 0.2°, length 1°). Subjects confirmed the orientation with the space bar and the reproduction bar disappeared. 

Stimulus orientation and task were manipulated to create three conditions, run in different blocks. In the both-response-and-stimulus condition, stimuli were all extracted around the oblique axis (from 30° to 60° in steps of 3°) and every trial required a direct reproduction (Figure 2, top row). In the similar-stimulus condition (Figure 2, bottom row), stimuli were extracted from the same distribution as before (from 30° to 60°), but on every second trial subjects were asked to make a “mirror reproduction.” In order to ensure minimal leakage from one reproduction to the other, the response bar was presented in different hemifields on odd and even trials. In the similar-response condition (Figure 2, middle row), on even trials the stimuli were around the diagonal orientation and response was direct; on odd trials the stimuli were extracted around the antidiagonal (from −60° to −30° in steps of 3°) and subjects were asked to make a mirror reproduction. In order to maximize response history effects, the reproduction bar was presented always in the same position, 6° below fixation. Color of the fixation dot (yellow or magenta) instructed the subjects whether they had to perform a direct or a mirror reproduction. Seven subjects took part to the experiment leading to about 2,000 trials per condition. 

Figure 2

 

Isolating sensory and response components in serial dependence. In the traditional paradigm (top row), both successive stimuli and successive responses are similar so it is hard to disentangle the locus of the serial effects. In two new conditions (middle and bottom rows) we requested a mirror reproduction every second trial. In this way we could design sequences where the stimulus changed by roughly 90° on every trial and thus the subject was led to make very similar reproductions across trials (similar responses, middle row); alternatively, stimuli could be similar across trials (similar stimuli, bottom row) and responses were markedly different on every trial. Left columns indicate sample stimuli, middle column percepts, and right columns responses. Dashed lines indicate accurate behavior; colored lines indicate the orientation of the previous stimulus (blue) or response (red) when similar; filled lines responses illustrate a case of attractive serial effect. Rightmost plots for each row illustrate how the serial effect was computed. Briefly we subdivided trials according to the current stimulus (in this example 45° degrees) and plotted the responses as function of the previous trial orientation. For each test orientation we obtained a slope that we then averaged in order to obtain the average effect.

In order to measure the relative contribution of the previous trial, for all trials of a specific orientation we regressed the orientation of the previous trial from them. This yielded slope values for each stimulus that we averaged to obtain the average contribution of the preceding trial (see Cicchini et al., 2014). Trials where the orientation difference between current and previous trial exceeded 18° were excluded. In the standard paradigm (both stimulus and response), in all trials stimuli and responses lie in the same quadrant (the first quadrant in Cartesian convention), and it is straightforward to regress the previous orientation from the current response. In the similar-response condition (Figure 2, middle row), serial effects arose because of the similarity between the responses requested; in order to retrieve this sequence, one needs to mirror-flip the stimulus sequence every second trial. In the case of similar stimuli, serial effects arose because of stimulus similarity (which is trivial as all stimuli belong to the first quadrant); however, every second trial subjects were required to make a mirror response that mirror-flips their percept (along with any bias). In order to retrieve the sequence of perceived orientations, every second trial the response needs to be mirror-flipped (see example in last row). 

Results

Experiment 1

We first repeated Fritsche et al.'s (2017) experiment over a wider range of stimuli. Briefly, on each trial we first presented an “inducer” Gabor stimulus whose orientation was reproduced by subjects by rotating a virtual bar (with only the ends visible) to match the Gabor orientation. This was followed by two Gabor patches (one in the same position as the inducer), which observers judged to be of the same or different orientation. Although observers only give a same–different response, any effects of prior stimuli will be revealed by shifts in the response distributions under different adaptation conditions (see Figure 1A). 

Figure 1B shows the magnitude of serial dependence as a function of the difference of inducer and test orientation, revealing a different picture from that reported by Fritsche et al. (2017). We replicate their result of slightly negative effects for inducers differing from the test by 20°, although these failed to reach significance in our hands (−0.8°, p = 0.27 under identical conditions to theirs, −0.4°, p = 0.36 with our procedure). However, inducer orientations differing from the test by less than 10° yielded strong and highly significant positive serial dependencies. When we calculate the weights of the serial dependence, using the technique of Cicchini et al. (2014), we see they rise to 0.2 (black symbols, Figure 1C), implying that 20% of the response can be attributed to the previous inducer orientation. These weights are similar to those observed in numerosity judgments requiring a reproduction technique (Cicchini et al., 2014). 

As many contextual effects scale with sensory uncertainty (Cicchini, Arrighi, Cecchetti, Giusti, & Burr, 2012; Jazayeri & Shadlen, 2010; Kording & Wolpert, 2004) we examined the effect of uncertainty on perceptual serial dependence, exploiting the fact that visual sensitivity to vertical and horizontal stimuli is higher than to oblique stimuli (Gibson, 1937; Mikellidou, Cicchini, Thompson, & Burr, 2015). We therefore separated the results for cardinal (near-horizontal or near-vertical) and oblique stimuli, and calculated the weights of serial dependence separately for each. Oblique stimuli (orange symbols of Figure 1C) showed very strong serial dependence, with weights as high as 0.3, while stimuli close to the cardinal orientations, which have less intrinsic uncertainty, showed a nonsignificant weight of only 0.04. These data also suggest that serial dependence varies across all orientations, and measuring serial dependencies around cardinal axes (like in experiment 2 of Fritsche et al., 2017, or experiment 3 of Fischer & Whitney, 2014) may not be optimal. 

We modeled our data with an ideal-observer model designed for stimulus reproduction (Cicchini et al., 2014), in which the weight of past information increases with the sensory noise of the average stimuli (σ), and with the inverse of the difference in orientations between past and current stimulus (d):  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}w = {{{\sigma ^2}} \over {2{\sigma ^2} + {d^2}}} = {1 \over {2 + {{\left( {{\raise0.7ex\hbox{$d$} \!\mathord{\left/ {\vphantom {d \sigma }}\right.\kern-1.2pt}\!\lower0.7ex\hbox{$\sigma $}}} \right)}^2}}}{\rm }\end{equation}

 

On inspection, the maximum weight that can be given to the previous stimulus is 0.5, equal to that of the current stimulus, then decreases with increasing orientation difference. The rate of decrease depends on the sensory noise, falling to 1/3 when σ = d, then decreasing hyperbolically with the squared ratio of d/σ. We simulated optimal behaviour for a realistic observer with 4.5° resolution for oblique stimuli and 1.5° for cardinal stimuli (Mikellidou et al., 2015). Although our model was developed to be optimal in reproduction tasks, it predicts much of the trend of the same/different classification-task data. These data show clearly that perceptual serial dependencies scale with sensory uncertainty, in the same way as serial dependencies measured with reproduction, although the tasks are completely different (Cicchini et al., 2014; Cicchini & Burr, 2016). 

As the paradigm developed to isolate perceptual effects also involves reproducing stimulus angle, we examined our data to see if the reproduction affects successive reproductions (rather than test-probe comparison). The results (Figure 1D) reveal strong serial dependence, even though the reproduction trials were separated by the comparison stimuli. When successive reproduction stimuli were similar, the serial dependencies were similar to the perceptual comparison. However, the range over which interactions occurred was much larger. This is broadly consistent with our model, if we assume that serial dependence does not only act on sensory representations, which determine the percept, but also continues to bias increasingly noisier postperceptual representations, which are used to generate an adjustment response. The clear prediction from Equation 1 is that the effect should be stronger and extend to larger differences in orientation. We assumed that the extra noise due to these processes is about 7°, yielding an acceptable fit to both sets of data. Note that in this case we neither predict nor find a large difference in serial dependence for oblique and cardinal orientations, as the predominant noise is postsensory, swamping the sensory noise. The results and simulations highlight the fact that positive serial dependencies probably operate on both perceptual and postperceptual processes, an efficient strategy to reduce noise arising from both. 

Experiment 2

The results so far show that both perceptual and postperceptual mechanisms contribute to serial dependencies. Do the postperceptual mechanisms include the reproduction response? 

In the traditional paradigm, stimulus orientation and response orientation are coupled so this is not possible (Figure 2, top row). We introduced two new conditions in which subjects responded differently to the stimulus orientation on alternate trials: On odd trials the color of the fixation dot instructed them to respond “directly,” aligning the response with the stimulus, while on even trials they were instructed to make a “mirror” response, aligning the bar to the angle vertically symmetric to the stimulus. In this way, we could create sequences where stimuli are near orthogonal but successive responses are quite similar (provoking serial dependence of the response; see Figure 2 middle row). Alternatively, we could create sequences where the successive stimuli were similar (and hence should have strong serial dependencies) but the responses were near-orthogonal (and should have little serial effect: Figure 2, middle row). For each condition, we plot the error in the current trial as function of the orientation difference and obtain the weight of the previous trial upon current reproduction (see methods). 

Figure 3 shows average serial dependence for the three crucial conditions. The results are clear: The condition where successive stimuli were similar but responses orthogonal (blue) yielded consistent and significant serial effects (average 12% ± 6%, t = 2.1, p = 0.04). However, when the successive responses were similar (but stimuli orthogonal) there was no serial dependence (average 0.01% ± 5%, n.s.). The condition when both responses and stimuli were similar (gray) yielded the strongest serial effects (17% ± 7%, t = 2.5, p = 0.006). 

Figure 3

 

Sensory integration occurs despite changes in response. Average contribution of previous trial orientation for the three conditions of Experiment 2 (both stimulus and response, similar response, and similar stimulus). Error bars are SE. (*p < 0.05, ** p < 0.01)

When the stimulus range is relatively narrow, generic perceptual distortions such as regression to the mean (Anobile, Cicchini, & Burr, 2012; Cicchini et al., 2012; Jazayeri & Shadlen, 2010) or repulsion from cardinals (Girshick, Landy, & Simoncelli, 2011; Tomassini, Morgan, & Solomon, 2010) could come into play. In order to control for these possible artefacts, we reran our analysis (for an “aggregate observer,” pooling over subjects), after shuffling the trial order. There were no dependency between neighbouring trials, after trial order had been removed (0.001% ± 0.03), indicating that the slopes we report reflect a genuine contribution of intertrial serial dependencies. 

Discussion

Our experiments show that positive serial dependence can occur at the perceptual level. Furthermore, when response orientation is dissociated from the stimulus, there is very little evidence for serial dependence in the motor response. Several researchers have pointed out that both positive and negative aftereffects occur routinely in perception, and each serves an important role (Burr & Cicchini, 2014; Chopin & Mamassian, 2012; Fritsche et al., 2017; Taubert, Alais, & Burr, 2016). However, our experiment suggests that repulsive history effects in perception occur only under some experimental conditions, which are not optimal for revealing serial dependencies. At smaller stimulus distances and for more noisy stimuli, where we expect serial dependencies to be maximal, we indeed found positive aftereffects, as large as 35%, implying that the previous stimulus was nearly half as important for perception as the actual physical stimulus. Perceptual serial effects are consistent with recent fMRI results, suggesting that neural responses as early as in V1 are biased positively by previous stimuli in a spatially selective fashion (St John-Saaltink et al., 2016). 

Our second experiment showed that observers spontaneously bias responses toward the previous stimuli, even when the response strategy alternated on every trial. Previous findings (Fischer & Whitney, 2014) have shown that serial dependence occurs even when no response is required between stimuli. We further show that even when subjects do respond to stimuli, the responses themselves do not directly affect serial dependencies: Trials that required similar response but were based on different stimuli yielded much smaller effects. This is surprising as motor responses often reflect optimizing strategies (Cicchini et al., 2012; Jazayeri & Shadlen, 2010; Kording & Wolpert, 2004; Trommershauser, Maloney, & Landy, 2008) and incorporate stimulus statistics (Roach, McGraw, Whitaker, & Heron, 2017). However, these paradigms typically measure optimal strategies as the assimilation of a prior that represents the whole stimulus history, whereas here we are measuring the contribution of the motor adjustment of each single trial upon the subsequent one. 

Fritsche et al. (2017) showed that the magnitude of serial effects depends strongly on the task. We agree with this, and show that in the same tasks designed to isolate perceptual from reproduction effects, there are also very strong serial effects in reproduction. These effects are stronger than those for the perceptually isolated task, and extend over a wider range, suggesting that serial dependence is not confined to purely perceptual processes, but also affects postperceptual processes. As it would seem that this is not driven by the response itself, the most probable source of these additional biases is the decision process leading up to the reproduction. 

Temporal context is clearly important for perception, and the effects can be either positive or negative. Empirically, positive and negative aftereffects can be dissociated relatively easily: Negative aftereffects occur after relatively prolonged exposure to very salient adaptor stimuli, are well localized spatially, usually in retinotopic coordinates, and are largely independent of attention; serial dependencies, as we have seen, are strongest for a sequence of weak stimuli, have poor and largely spatiotopic spatial localization, and are highly dependent on attention (Fischer & Whitney, 2014; Kanai & Verstraten, 2005; Pantle, Gallogly, & Piehler, 2000; Yoshimoto & Takeuchi, 2013; Yoshimoto, Uchida-Ota, & Takeuchi, 2014). What are the functional roles of these opposing forms of serial dependence? Most likely they trade off competing perceptual requirements. In space perception, there exists a well-known trade-off between the need to integrate to reduce noise, and to segregate to perceive small objects. Similar requirements occur over time, and are dealt with at many levels. For example, all sensory systems tend to have two classes of receptors, slow adapting or tonic receptors that integrate over time, and fast adapting or phasic receptors that respond to change. Serial dependencies and negative aftereffects may continue this dichotomy at higher levels of analysis: Positive dependencies cause an adaptive form of integration, reducing noise by exploiting temporal consistencies; negative aftereffects aid segregation, amplifying small but potentially important changes overtime. Negative adaptation is clearly also important for calibrating perceptual systems to the prevailing conditions (Chopin & Mamassian, 2012). However, it remains very much an open question exactly how these two opposing effects interact. The studies discussed here provide further evidence for how neural perceptual systems maximize efficiency by exploiting temporal properties of natural scene statistics. 

As objects do not normally morph spontaneously into completely different entities, averaging recent perceptual history with the current percept reduces noise and helps stabilize what we see over time, leading to a more robust, predictable, and stable sensation of the world. This is clearly a fundamental perceptive mechanism. On the other hand, it makes less sense to integrate transient perceptual attributes, such as facial expressions, which can change from moment to moment, and there is good evidence that there is no positive serial dependence of expression. Indeed there seems to be negative adaptation instead, which could serve to enhance change of expression (Taubert et al., 2016). 

To sum up, our work reveals strong experimental and theoretical similarities between perceptual and postperceptual serial dependencies and suggests that positive dependencies are a common strategy to optimize both perception and postperceptual decision making. These results are evidence for generative models of perception, such as predictive coding (Friston, 2009; Rao & Ballard, 1999) and suggest a common computational solution for improving sensory uncertainty across several systems. 

Acknowledgments

This work was supported by the European Research Council (FP7; Early Sensory Cortex Plasticity and Adaptability in Human Adults, Grant Number 338866) and the Italian Ministry of Education, University and Research-Futuro in Ricerca Grant 2013 (FIRB RBFR1332DJ). 

Commercial relationships: none 

Corresponding Author: David Burr. 

Email: . 

Address: Department of Neuroscience, Psychology, Pharmacology and Child Health, University of Florence, Firenze, Italy.