Purchase this article with an account.
David Pascucci, Ömer Dağlar Tanrikulu, Ayberk Ozkirli, Christian Houborg, Gizay Ceylan, Paul Zerr, Mohsen Rafiei, Árni Kristjánsson; Serial dependence in visual perception: A review. Journal of Vision 2023;23(1):9. doi: https://doi.org/10.1167/jov.23.1.9.
Download citation file:
© ARVO (1962-2015); The Authors (2016-present)
How does the visual system represent continuity in the constantly changing visual input? A recent proposal is that vision is serially dependent: Stimuli seen a moment ago influence what we perceive in the present. In line with this, recent frameworks suggest that the visual system anticipates whether an object seen at one moment is the same as the one seen a moment ago, binding visual representations across consecutive perceptual episodes. A growing body of work supports this view, revealing signatures of serial dependence in many diverse visual tasks. Yet, the variety of disparate findings and interpretations calls for a more general picture. Here, we survey the main paradigms and results over the past decade. We also focus on the challenge of finding a relationship between serial dependence and the concept of “object identity,” taking centuries-long history of research into account. Among the seemingly contrasting findings on serial dependence, we highlight common patterns that may elucidate the nature of this phenomenon and attempt to identify questions that are unanswered.
Serial dependence has a long history in psychology and psychophysics, starting from the adoption of scaling methods and the finding of systematic effects in psychophysical reports that extend over trials. Two such effects were termed assimilation—when perceptual reports were positively correlated with previous stimuli and responses—and contrastive effects—when the sign of this serial correlation was negative (see “A brief history of research on history biases”).
More recently, the term serial dependence has been used to describe, typically, a positive bias to judge present stimuli as more similar to recent stimuli than they are (i.e., a form of assimilation). In particular, serial dependence involves a systematic bias in perceptual decisions during psychophysical tasks, which are attracted toward aspects of the stimulus presented one or a few trials before. For instance, the decision about the orientation of a visual stimulus is biased toward the orientation of the stimulus seen before (Figures 2 and 3).
Originally, however, the term comes from time-series analysis and refers to statistical dependencies between events that are close in time (Huitema, 1986). Compared to serial and autocorrelation, serial dependence implies any linear or nonlinear relationship that can be used to predict future events from the past. It describes a unidirectional effect (the past affects the present and not the other way around) but no causality—a third unknown variable may be the true source mediating the effect. From this perspective, serial dependence is neither a mechanism nor a phenomenon exclusive to vision and cognition but a property of a time series (e.g., of behavioral and neural data).
Compared to the vast amount of behavioral work, only a few studies have been performed on the neural underpinnings of serial dependence. In one of the first, Schwiedrzik and colleagues (2014) used functional magnetic resonance imaging (fMRI) in a task with multistable stimuli to disentangle the effects of perceptual hysteresis (e.g., positive serial dependence) from those of adaptation. They found that these two opposite biases, revealed in many other following works, as we discuss in this review, reflect distinct neural circuits: Hysteresis involves a distributed network of higher visual areas and frontoparietal areas—particularly the right dorsomedial prefrontal cortex, which is involved in predictions and memory—whereas adaptation is restricted to activity changes in early sensory areas.
In a later study, John-Saaltink and colleagues (2016) demonstrated that prior stimuli modulate activity in early sensory cortex, with an effect highly specific to V1. Notably, the effects were not due to the prior stimulus itself but to the reported stimulus even on error trials, where the reported stimulus did not match the physical one.
Within the Bayesian framework, van Bergen and Jehee (2019) used fMRI decoding techniques to decode representations of sensory uncertainty in probabilistic distributions derived from population-level activity. They found that the uncertainty decoded from the activity in the early visual cortex can be related to the strength of serial dependence, showing that, at least when comparing the relative uncertainty between two trials (e.g., high to low and low to high), the behavioral bias reflected optimal cue combination (larger serial dependence when the sensory uncertainty decoded on the previous trial was lower).
Leveraging the high temporal resolution of electroencephalography (EEG), other works have investigated the electrophysiological correlates and temporal dynamics of serial dependence. Several studies have shown that prior stimulus information can be decoded from evoked EEG scalp activity in the current trial. These decoding results were obtained in tasks where the previous stimulus was either relevant or irrelevant and induced attractive or repulsive biases (Bae & Luck, 2019; Fornaciai & Park, 2020a). An important finding comes from the work of Barbosa and colleagues, which demonstrated that latent traces of prior stimuli can be reactivated by the onset of a new event (e.g., the start of a new trial), likely causing interference with the representation of a new stimulus and serial dependence effects (Barbosa et al., 2020; Stein et al., 2020). Using transcranial magnetic stimulation (TMS), these authors also provided evidence for a causal link between the reactivation of memory traces in the prefrontal cortex and serial dependence in behavior (Barbosa et al., 2020; see also de Azevedo Neto and Bartels, 2021).
The analysis of serial dependence involves quantifying systematic relationships between the decision on the current trial and the history of events on preceding trials. The exact pipeline depends on the type of task and data (e.g., adjustment responses or forced choices). We here provide a brief description of standard approaches used in different contexts.
For simplicity, we will use orientation adjustment tasks as an example, but the analyses described here equally apply to color, direction, or any other continuous stimulus.
Data preparation. Adjustment errors are computed as the angular (or linear) difference between the reported and the actual orientation in each trial. Typically, adjustment errors are widely scattered. To correct for outlier errors and unlikely reports, a correction is often performed by removing trials according to an arbitrary cutoff (e.g., 30°; Cicchini et al., 2018), standard deviation–based thresholds (e.g., 3 SD; Fritsche et al., 2017), or other parametric approaches (e.g., Grubb's test; Pascucci et al., 2019). Adjustment times are also used to clean the data, for instance, by removing trials with responses faster than 500 ms or slower than 10 s (Ceylan et al., 2021). In some cases, the trial following an outlier (e.g., a guess) is also removed, since no meaningful serial effects can be expected following a trial in which the observer had a lapse of attention or guessed the response. An additional procedure is to demean the cleaned errors to remove systematic tendencies to reproduce the stimulus in one direction or another (e.g., CW or CCW). Because adjustment errors are prone to other sources of biases, including, for instance, anisotropies and category boundary effects in orientation estimation, some authors have usually corrected for nonlinearities in the stimulus space (Manassi et al., 2018; Pascucci et al., 2019; van Bergen & Jehee, 2019). This has been done with different methods, from fitting sinusoidal to polynomial functions on the errors over stimulus space, to cleaning the data of biases, such as the cardinal or orientation bias and category biases in color space. Standard and validated approaches for this step have yet to be reached.
A largely unaddressed issue concerns the influence of swap errors on serial dependence. Swap errors refer to an observer reporting the wrong item in a working memory task. In serial dependence, an observer might report the previous instead of the current target, simply because of a “swap” rather than a combination of the prior and current percept. Almeida, Barbosa, and Compte (2015) reported that in a typical working memory task, swaps and attractive bias co-occur. As swap errors appear as a 100% bias, even a few swaps could create the appearance of spurious serial dependence.
Model fitting. The magnitude of the serial dependence in adjustment errors is computed by fitting the first derivative of a Gaussian function (abbreviated as DoG) to the errors as a function of Δ (the difference between the previous stimulus feature and the current one). The typical form of this function is
The most used parameter to determine the magnitude of serial dependence is the amplitude or half-amplitude of the DoG function (α). While this is usually estimated on the aggregate data of many subjects, to avoid the pitfalls of aggregated data, the DoG fitting procedure can be expanded to allow for variability between individual participants by fitting a mixed-effects model (Pascucci et al., 2019). Another modeling approach is to fit a hierarchical Bayesian model to the data (Sadil et al., 2021). In this model, the authors accounted for individual differences and rotational biases as well.
As mentioned throughout the article, serial dependence patterns also contain a combination of attractive and repulsive components, an aspect that simple DoG curves fail to capture. Alternative functions have been proposed to overcome this limitation (Bliss et al., 2017). Because of the putative nature of these opposite biases, a plausible approach would be to use the difference between two Gaussian functions, one accounting for the positive and one for the negative component (see Figures 3A,B)—that is, the classic “Mexican-hat” profile. This, however, would come at the expense of model complexity.
Model-free approaches. An alternative approach to model fitting is to compute the average error in a range of Δ values, typically close to zero. For example, subtracting the average error for Δ in the 1° to 25° range from the average error in the corresponding negative range (Samaha et al., 2019) quantifies the amount of systematic deviation of errors from zero—this deviation or “bias” is positive for attractive serial dependence and negative for repulsion. This approach is a straightforward way of quantifying serial dependence with few assumptions, particularly useful when limited data points are available, which is often the case in analyses for a single observer. Restricting the analysis to values of Δ close to zero is reasonable when serial dependence effects are expected in this range. A disadvantage of this approach, which also applies to the DoG fitting procedure described above, is that it does not allow capturing more complex patterns across the entire Δ range. While attractive biases are typically present for small Δ, repulsive biases appear for larger Δ. One way to address this problem would be to use more than a single bin, discretizing Δ as a function of several distances between the previous and present stimulus (e.g., three to four bins from “close” to “far”), then analyzing the data with standard repeated-measures approaches.
Statistical testing is commonly performed using a permutation approach on the parameter of interest, typically the half-amplitude α of the DoG function (Fischer & Whitney, 2014; Fritsche et al., 2017). A null distribution of α is obtained by fitting the DoG function to a large number of randomized data sets generated by randomly flipping the sign of errors in each trial (or shuffling the correspondence between single-trial Δ and errors). The proportion of such data sets that is more extreme than the parameters from the DoG function fitted to actual data denotes p, which can be interpreted in the same way as the p-value in standard frequentist statistics. Comparisons between conditions can be performed using a similar permutation approach by shuffling the labels of the conditions for each randomized data set. This approach has been often used for aggregated data from multiple subjects. In the framework of nonlinear mixed-model analysis, the final estimate of the parameters and their uncertainty can be used to quantify statistical significance (Pascucci et al., 2019).
When the task involves a choice between two alternatives, results are typically analyzed using a psychometric function, in which stimulus value predicts response. Serial dependence can be quantified as a shift in the threshold or point of subjective equality (PSE) of the psychometric function. For example, in comparing which one of two stimuli is tilted more clockwise, the PSE corresponds to the point of “no difference.” A shift in the PSE can be informative of whether the perception of the current stimulus has been affected by a preceding inducer, with attractive and repulsive biases that depend on the direction of the shift (Fritsche et al., 2017). Beyond PSE, more sophisticated statistical models have been also proposed to quantify or account for serial biases in psychophysical forced-choice tasks (Gekas et al., 2019).
This PDF is available to Subscribers Only