Effects of satisfying and violating expectations on serial dependence

Abstract

Serial dependence refers to the phenomenon that observers tend to report stimuli as being more similar to previous stimuli than they really are (attractive dependence) or, in some cases, as more different than they really are (repulsive dependence). Numerous experiments have demonstrated serial dependence for a range of modalities and stimulus features, highlighting the role of bottom-up sensory interactions. However, comparatively less research has focused on how higher-level cognitive factors, such as expectations, might influence serial dependence. Here, we manipulated expectations by having observers respond to target luminance gratings that occurred at the end of a sequence of non-target gratings. The sequence either rotated predictably (inducing an expectation), varied randomly (inducing no expectation), or rotated predictably but had a random target orientation (violating expectations). We found that observers produced less errors and indicated less uncertainty in their estimations of expected stimuli but their responses were biased away from the penultimate stimulus in the sequence (repulsive dependence). In contrast, following random sequences, responses showed an attractive bias to the penultimate stimulus in the sequence. Unexpected targets showed a mixture of both biases, such that when targets happened (by chance) to appear as expected, responses were repulsed, but responses to target orientations that more clearly violated expectations were attracted. These results indicate that, whereas attraction to previous stimuli may be a default strategy employed in response to random and unexpected events, certain expectations can reverse the default bias into a repulsive one.

Introduction

Although we tend to think of our perceptual decisions as being made on the basis of evidence available in the moment, recent work on the phenomenon of serial dependence suggests that what we report experiencing can be biased by brief stimuli that occurred up to tens of seconds into the recent past. A large body of literature has established serial dependence across different sensory modalities, such as vision (Cicchini, Anobile, Burr, 2014; Fischer & Whitney, 2014; Fründ, Wichmann, & Macke, 2014) and audition (Motala, Zhang, & Alais, 2020) and across a range of low-level (Alais, Leung & der Burg, 2017; Bliss, Sun, & D'Esposito, 2017; Cicchini, Mikellidou, & Burr, 2017; Cicchini, Mikellidou, & Burr, 2018; Fornaciai & Park, 2018; Samaha, Switzky, & Postle, 2019) and high-level visual features (Liberman, Fischer, & Whitney, 2014; Manassi & Whitney, 2022; Suárez-Pinilla, Seth, & Roseboom, 2018), suggesting that the incorporation of recent history may be a general mechanism for stabilizing perception over time (Cicchini et al., 2018; Fritsche, Spaak, & de Lange, 2020; Manassi & Whitney, 2022). Moreover, serial dependence has been shown across a range of perceptual tasks, working memory tasks, and longer-term memory tasks (Kiyonaga, Scimeca, Bliss, & Whitney, 2017). 

Whereas much work has focused on the stimulus features and task parameters for which serial dependence manifests, comparatively less research has looked at the role of higher-level cognitive factors in modulating serial dependence. It is known from prior work that spatial attention (Fischer & Whitney, 2014) as well as cross-modal attention can (Lau & Maus, 2019) mediate serial dependence in perception, although a stimulus need not be task relevant itself to bias subsequent reports (Fischer & Whitney, 2014; Fornaciai & Park, 2018). In memory, serial dependence is sensitive to the task-irrelevant context that can be shared across memory episodes (Fischer, Czoschke, Peters, Rahm, Kaiser, & Bledowski, 2020), suggesting that more than just the specific content one is remembering is carried from one moment to the next. 

The present work examines the role of expectation, and violations of expectation, on serial dependence for orientation. We modeled our paradigm after a recent experiment that used rotating or random sequences of gratings to study how expectations impact orientation tuning in the mouse visual cortex (Tang et al., 2021). In our version of the paradigm, human observers viewed sequences of grating stimuli on each trial but responded only to the final stimulus in the sequence (hereafter the “target”) via continuous estimation. Some sequences rotated in predictable steps, establishing an expectation about what the target stimulus would be, and some varied randomly, providing no expectation. On rotating trials, the target stimulus could be expected if its orientation matched what was expected based on prior rotation, or unexpected if the target stimulus was random (despite the preceding sequence having rotated). Thus, we created conditions where a target was expected, unexpected, or random. 

We examined how responses to target stimuli were biased by the immediately preceding, task-irrelevant stimulus in the sequence (the penultimate stimulus). We were principally interested in how serial dependence was impacted by the status of the target as expected, unexpected, or random. Because most prior work on serial dependence randomized stimuli from trial to trial, we expected to replicate the attractive bias on random trials. Because both expected and unexpected stimuli have been suggested to lead to a sharpening of bottom-up sensory responses (Kok, Jehee, & de Lange, 2012; Kok, Mostert, & de Lange, 2017; Summerfield & de Lange, 2014; Tang et al., 2021) – which could help override biases from prior inputs – we could potentially see less serial dependence on those trials. On the other hand, serial dependence is thought to be an adaptive mechanism that exploits the temporal dependencies inherent in most naturalistic contexts, which makes stimuli predictable based on their recent past (Cicchini et al., 2018). Thus, one could argue that an expected stimulus would trigger greater attractive serial dependence because there is a stronger temporal dependency, whereas an unexpected stimulus explicitly violates temporal dependencies which could lead to a reduction of serial dependence. Last, we were also interested in how expectations may modulate behavioral accuracy and subjective uncertainty in the primary estimation task, given findings that expectations can alter neuronal encoding of orientation information (Kok et al., 2012; Kok et al., 2017; Tang et al., 2021). 

Methods

Participants and data availability

Thirty-six observers were recruited from the University of California, Santa Cruz (mean age = 21.75 years old, SD = 6.01, 6 men, 29 women, and 1 non-binary). All observers reported normal or corrected visual acuity, provided informed consent, and were compensated monetarily and via course credit when appropriate. Sample size was chosen to be approximately one third greater than our recent serial dependence work (Samaha et al., 2019) that focused on group-level statistical inferences given that the present work included three conditions of interest, as opposed to two. This experiment was conducted in accordance with the University of California, Santa Cruz Institutional Review Board. In accordance with the practices of open science and reproducibility, all raw data and code used in the present analyses are freely available via the SamahaLabUCSC webpage within the Open Science Framework (https://osf.io/kpjtb/). 

Stimuli and apparatus

Our task and stimuli were modeled after a recent paper that used this paradigm to study the effects of expectation (and violation of expectation) on neuronal responses in mouse visual cortex (Tang et al., 2021). Visual stimuli were sinusoidal luminance gratings (2 cycles per degree of visual angle [DVA], zero phase), presented centrally within a circular aperture subtending 5 DVA and at 80% Michelson contrast. A small light gray fixation point (0.17 DVA) was superimposed on top of each grating and was present throughout the task. The fixation turned from gray to light red for the duration of the target stimulus to help observers avoid confusing the target with other stimuli in the sequence. Immediately following the target, a 100% contrast mask was presented for 250 ms to erase any afterimage from the target grating. The mask was a black-and-white radial checkerboard pattern the same size as the target (5 DVA) which had four radial arms and 12 angular arms. 

A sequence of gratings was presented in each trial at a rate of 4 Hz with no gap between stimuli (i.e. each grating was displayed for 250 ms). The number of gratings in each sequence (including the target) varied randomly at uniform between five and eight, so that observers could not predict exactly when the target would appear. When the sequence rotated the grating turned in steps of 30 degrees, sufficient to produce apparent motion, and was pseudorandomly selected to rotate clockwise (CW) or counter-clockwise (CCW) equally. On trials with random sequences, the sequences (including the target) were selected randomly with uniform probability between 0 and 179 degrees. 

Stimuli were presented on a middle-gray background of 50 cd/m² luminance on a gamma-corrected VIEWPixx electroencephalogram (EEG) monitor (1920w × 1080h pixels, 120 Hz refresh rate). Observers were seated and positioned in a chin and headrest approximately 74 cm from the screen. The experiment was controlled with Psychtoolbox-3 (Kleiner et al., 2007; Pelli, 1997) running in the MATLAB environment (version 9.8) under an Ubuntu (version 18.04) operating system. 

Procedure

Each trial began with a central gray fixation dot for a time between 500 and 700 ms (randomly sampled at uniform) and then a sequence of gratings appeared at fixation. The gratings were presented at 4 Hz (250 ms each) and varied between five and eight gratings in each sequence. On 60% of the trials, the gratings changed orientation upon each presentation in 30-degree steps either CW or CCW. On half of these rotation trials, the final stimulus in the sequence (which was the target stimulus that required a response) was presented at the expected orientation given the previous rotation (i.e. 30 CW or CCW of the preceding stimulus, depending on whether the sequence rotated CW or CCW, respectively). Thus, 30% of the total trials had “expected” targets. On the remaining half of the rotation trials, the orientation of the target stimulus was chosen randomly at uniform from between 0 and 179 degrees, making 30% of the total trials “unexpected.” On the remaining 40% of trials, all stimuli in the sequence (including targets) were chosen at random (uniformly sampled from between 0 and 179 degrees), making these trials not expected or “random.” The masking stimulus always immediately followed the target and was also displayed for 250 ms. Then, after a 250 ms blank screen (fixation only), a thin line appeared (same diameter as the gratings) at a random orientation. The observer's task was to rotate the line using lateral movements of the computer mouse to match the orientation of the target grating as closely as possible. A mouse click locked in the orientation response and froze the line, which then prompted a symmetrical blue wedge to appear on either side of the frozen line (see Figure 1A). The size of the wedge (initialized randomly on each trial) could also be adjusted via lateral mouse movement and was meant to be used by the observer to express their subjective uncertainty in their estimation response. Specifically, we instructed observers to: 

 

“Adjust the blue wedge based on the confidence of your answer. The narrower the wedge, the more points you will get as long as the wedge contains the true orientation of the target. If the wedge does not contain the true orientation, you will get zero points on that trial. You will get the most points by making the wedge narrower when you are more certain about the orientation and wider when you are less certain about the orientation. You will be compensated according to your score at the end of the experiment.”

 

Figure 1.

 

Task design and metacognitive performance. (A) Example stimulus sequences on random, expected, and unexpected trials. Grating stimuli were presented at 4 Hz (250 ms each) and contained between 5 and 8 stimuli per sequence (5 stimuli-long sequence are shown here for simplicity). Observers responded to the last grating in the sequence (the “target” stimulus), which was denoted with a red fixation cross to help avoid confusion. A 250 ms mask was presented after target onset to erase any afterimage. Following an orientation estimation response, observers provided a confidence judgment by adjusting a blue wedge. Larger wedge sizes indicated greater uncertainty in the true orientation. In random sequences (top) each grating took on a random orientation (sampled from the full orientation space). In expected sequences (middle), each new grating, including the target, was rotated 30 degrees from the previous grating. A given sequence was either CW or CCW with equal probability. In unexpected sequences (bottom), all stimuli preceding the target rotated in 30 degree steps but the target stimulus was a random orientation. For serial dependence analysis, we considered how error on the target stimulus was related to the orientation of the immediately preceding stimulus in the sequence (the penultimate stimulus). Onscreen text was not presented in the actual experiment. (B) For each observer we correlated their absolute response error on each trial with the corresponding wedge size provided on that trial. Nearly all observers showed a positive correlation, indicating that trials with greater error were also rated with greater uncertainty (larger wedge size). Observers with significant correlations are marked with solid black triangles.

Thus, we incentivize observers to adjust the wedge size according to their felt uncertainty using a simple rule whereby a wedge size that was so small that it did not capture the target orientation would result in zero points, but fewer points were awarded the larger the wedge became to the point where a wedge that covered the full orientation space was also worth zero points. Regardless of the actual score, observers were all compensated the same amount at the end of the study. No time pressure was imposed on either the estimation or the uncertainty response. Observers first completed 30 trials of practice where their response error was printed to the screen after each trial. They then completed a total of 480 trials of the main task, split into six blocks, without any feedback. The total task time was around 1 hour. 

Analysis of error, bias, and uncertainty

We first analyzed response errors (the circular distance between the target and response; positive coded as CW error) as well as subjective uncertainty (wedge size) in the random, expected, and unexpected condition. To assess whether our novel way of measuring confidence was actually sensitive to variation in task performance (i.e. high error trials were associated with larger wedge sizes), we computed, for each observer, the Spearman correlation coefficient (Rho) between single-trial absolute error (in degrees) and uncertainty (i.e. wedge size in degrees). If uncertainty responses track actual task performance, we would anticipate a positive correlation for the majority of observers. 

We then analyzed how the expectation manipulation influenced response error and response bias. Overall response error was computed as the standard deviation (SD) of single-trial response for each observer and condition. A larger SD indicates a greater spread of response errors and hence more responses further from zero error (i.e. greater overall error). Bias was computed as the mean signed error across trials for each observer and condition. We coded a positive mean error as indicating a bias toward CW and a negative mean as indicating a bias towards CCW (see Figure 2). Last, we analyzed mean uncertainty by computing the mean wedge size across trials for each condition and observer. A smaller wedge size indicates lower uncertainty or higher confidence in the choice. 

Figure 2.

 

Expectation effects on error, response bias, and subjective uncertainty. (A) Density estimates of response error distributions averaged across observers on random (black), expected (blue), and unexpected (red) trials. Response errors were quantified in terms of their spread (standard deviation [SD], and their mean [bias]). The right panel of A shows that error (filled squares) on expected trials was lower as compared to both random and unexpected trials and there was no significant response bias (empty squares) toward CW or CCW, as the means of the distributions were all centered on zero. (B) Distribution of uncertainty responses (wedge size) in each condition. Observers responded with more certainty (smaller wedge size) on expected compared to random trials but also on unexpected compared to random trials, even though there was no corresponding boost in accuracy for unexpected trials. Note that the wedge size in degrees corresponds to one side of the mirrored wedge, so it should be doubled to correspond to the full orientation space (i.e. 90 degrees in the plot equals a completely uncertain response and is the largest possible value). (C) The left two plots display response errors on expected trials broken down by the rotation direction (CW or CCW) of the sequence. The rightmost two plots show the same but for unexpected trials. Error (SD) was not impacted by rotation direction, however, on expected trials, observers showed a CW bias following CW rotation when compared to CCW rotation (magenta distribution is shifted to the right of the blue distribution). This was not the case on unexpected trials, suggesting a selective repulsive effect of expectations on estimation responses. Shaded bands and error bars denote ± 1 across-subjects SEM. * Denotes p < 0.05, ** denotes p < 0.01, and *** denotes p < 0.001.

Two-tailed, repeated-measures t-tests were used to compare error, bias, and uncertainty among random, expected, and unexpected conditions. For visualization purposes, means and standard error of the means (SEM) are shown in addition to full response error and uncertainty distributions. The distributions were visualized by computing probability density estimates for each observer and condition using the MATLAB function ksdensity.m which uses a normal kernel function evaluated in 2-degree steps within the range of the data using a sliding window size of about 3 degrees. The resulting density functions were then averaged over observers and conditions (as seen in Figure 2) 

Serial dependence analysis

Serial dependence curves (see Figure 3) were computed by recording the target error on the current trial and the relative difference in orientation between the target and the penultimate stimulus from the sequence on the same trial. Note that here we are not looking at serial dependencies across trials but rather between the target and most-recently viewed stimulus prior to the target. These data were then smoothed using a 25 degree-wide (Samaha et al., 2019) moving average filter that moved in steps of 1 degree. When mean target error is plotted as a function of the relative difference between the target and penultimate stimulus in the sequence, attractive serial dependence manifests as a CCW (or negative) mean error when the penultimate stimulus was CCW with respect to the target and a CW mean error when the penultimate stimulus was also CW with respect to the target (see Figure 3). Note that only the random and unexpected conditions had target stimuli which varied randomly across the full range of possible orientations relative to the penultimate stimulus. In the expected condition, the target was always +30 or 30 degrees away from the penultimate stimulus, depending on the rotation direction (CW or CCW, respectively) of the sequence. As a result, there are only two data points in the expected condition, rather than a full curve (see Figure 3B). 

Figure 3.

 

Effects of expectation on serial dependence. (A) The upper panel shows group-averaged error as a function of the orientation difference between the target and penultimate stimulus on random trials. A DoG fit to the data (thick black line) revealed a positive amplitude parameter (bootstrap distribution shown in the lower panel) which was significantly different from zero, indicating that, on trials with random sequences, responses were biased toward the penultimate stimulus (attractive dependence). (B) On expected trials, responses showed a repulsive bias, being further along the trajectory of rotation rather than closer to the penultimate stimulus. (C) Responses on unexpected trials showed a mixture of attractive and repulsive biases. When the target orientation happened, by chance, to resemble the expected orientation (i.e. approximately ±30 degrees) a repulsive bias was evident, like on expected trials. However, when the target clearly violated expectations (relative differences beyond approximately ± 40 degrees) responses became attractive. This pattern was captured well by a sine wave fit to the data (thick black line), the amplitude parameter of which was significantly non-zero. Shaded bands and error bars denote ± 1 bootstrapped SEM. * Denotes p < 0.05 and *** denotes p < 0.001.

We used a bootstrap approach to quantify serial dependence in random, expected, and unexpected conditions. For the random condition, data were well fit by a derivative of Gaussian (DoG) function as has been widely used in most serial dependence research (Bliss et al., 2017; Fischer & Whitney, 2014; Gallagher & Benton, 2022; Samaha et al., 2019; Suárez-Pinilla et al., 2018). The DoG has the form:  

\begin{eqnarray*}y = xawc{e^{ - {{\left( {wx} \right)}^2}}}\end{eqnarray*}

where x is the orientation difference between the penultimate and target stimulus, a is the amplitude of the curve peaks, w is the width of the curve, and c is the constant \(\frac{{\sqrt 2 }}{{{e^{ - 0.5}}}}\) which scales the amplitude parameter of interest to numerically match the height of the curve in degrees. Following others (Bliss et al., 2017; Fritsche, Mostert, & de Lange, 2017; Samaha et al., 2019), we fit the DoG to group-averaged data using a random subsample (with replacement) of observers on each of 20,000 bootstrap permutations. For each permutation, the amplitude parameter of the fit was saved, generating a distribution of amplitude parameters which was converted to a p value by dividing the number of bootstrap samples below zero by the number of permutations (20,000) and multiplying by two (two-tailed, nonparametric test of the amplitude parameter against zero). If no bootstrap samples fell below zero, the p value was set to (1/20000)*2. 

This same general approach was used to test for serial dependence in the expected condition except on each bootstrap permutation the difference in mean (signed) error between CW and CCW rotation directions was saved. For the unexpected condition, data did not clearly follow a DoG function, but had bi-phasic attractive and repulsive peaks (see Figure 3C). Thus, we fit a sine wave to the data, with the amplitude parameter capturing serial effects. The sine function had the form:  

\begin{eqnarray*}y = asin\left( {bx + c} \right)\end{eqnarray*}

where x is the orientation difference between the penultimate and target stimulus, a is the amplitude of the sine wave, b is the frequency of the wave, and c is the phase. We statistically tested the amplitude parameter against zero by dividing the number of bootstrap samples below zero by the number of permutations (20,000) and multiplying by two. As before, if no bootstrap samples fell below zero, the p value was set to (1/20000)*2. Last, because serial dependence has been shown to extend backward in time beyond just the most recent stimulus (Fischer & Whitney, 2014), we re-ran the bootstrap analysis described above but with serial dependence computed relative to the circular average of the entire preceding sequence of gratings, rather than relative to just the penultimate orientation. 

Results

Effects of expectation on error, uncertainty, and bias

To first gauge whether observers used the uncertainty response as expected, we computed the correlation between error and uncertainty across single trials for each observer. As shown in Figure 1B, 33 out of 36 observers had a positive correlation, indicating that they increased the wedge size on trials with larger errors (group-level t-test of the correlations against zero: t(35) = 10.49, p = 2.3*10⁻¹²). Moreover, 29 out of the 36 observers had a significant (p < 0.05) correlation at the single-trial level. This indicates the vast majority of observers understood task instructions and used the uncertainty response in way that meaningfully tracked performance. 

We next turned to the effect of expectations on overall error, uncertainty, and response bias. As shown in Figure 2A, observers systematically made fewer errors (smaller SD) when the target orientation was expected as compared to when it was random (t(35) = 3.63, p = 8.7*10⁻⁴) or unexpected (t(35) = 2.63, p = 0.012). Responses on unexpected trials, however, were not discernibly more accurate than on random trials (t(35) = 1.04, p = 0.304). 

Subjective uncertainty responses (Figure 2B) indicated that observers also felt more certain (smaller wedge size) when estimating expected targets as compared to random ones (t(35) = 3.52, p = 0.001). Interestingly, uncertainty was also lower on unexpected trials compared to random trials (t(35) = 3.17, p = 0.003), suggesting that observers felt relatively more confidence in their responses to unexpected targets without a concomitant increase in accuracy. Uncertainty on expected and unexpected trials did not differ significantly (t(35) = 1.61, p = 0.115) suggesting that the boost in confidence relative to random trials was comparable when expectations were met and when they were violated. This pattern of statistical effects also held when uncertainty response distributions were log₁₀ transformed prior to averaging (not shown), which made the responses more normally distributed. 

Bias (mean signed response error) did not significantly differ between any condition (Figure 2A, right; all p values ≥ 0.4), indicating that observers were not systematically biased to respond CW or CCW as a function of target expectation. However, when we divided expected and unexpected trials into those with CW or CCW rotating sequences, we found that response errors were systematically shifted CW following CW rotation (mean error = +1.6 degrees; SEM = 0.76) and shifted CCW following CCW sequences (mean error = −1.18 degrees; SEM = 0.79), leading to a significant difference between bias on expected CW and CCW rotation trials (Figure 2C, left; t(35) = 2.04, p = 0.048). This indicates that observers tended to report the orientation of expected targets as being further along the trajectory of rotation that it really was (i.e. a repulsive bias). This effect was only observed on expected trials – when a sequence rotated CW or CCW but the target orientation was random (i.e. the target was unexpected), responses were not systematically biased by the preceding rotation direction (Figure 2C, right; t(35) = 0.072, p = 0.942). The CW-CCW difference on expected trials was also significantly greater than that on unexpected trials (t(35) = 2.06, p = 0.046), demonstrating an interaction effect such that the repulsive bias was selectively present on expected trials. Although this result implies some repulsive serial dependence in the expected condition only, this analysis only considers mean bias conditioned on the previous rotation direction rather than examining potentially more subtle effects by conditioning mean bias on the relative difference between the target and penultimate stimulus. Therefore, we tested for effects of the penultimate stimulus orientation on mean errors across all three expectation conditions in our analysis of serial dependence. 

Effects of expectation on serial dependence

Mean serial dependence curves along with model fits and corresponding bootstrapped parameter distributions are shown in Figure 3. On random trials (see Figure 3A), observers demonstrated a clear attractive bias such that errors to the target stimuli were systematically biased toward the penultimate stimulus in the sequence (mean, 95% bootstrap confidence interval [CI], DoG amplitude = 4.01 degrees, 95% CI = 3.13, 4.88, p = 1.0*10⁻⁴). There was a small zone of repulsion when the penultimate stimulus was within approximately ±10 degrees of the target, which was likely due to a tilt after-effect given that there was no mask and no delay between the penultimate and target stimulus, however, the overwhelming direction of the bias is attractive, reflected in a positive amplitude parameter, as in most prior work using random stimulus sequences (Bliss et al., 2017; Cicchini et al., 2017; Cicchini et al., 2018; Fischer & Whitney, 2014; Fritsche et al., 2017; Samaha et al., 2019). 

In contrast to the random condition, response errors in the expected condition (see Figure 3B) were systematically repulsed away from the penultimate stimulus (mean CW-CCW difference = 2.8 degrees, 95% CI = 0.28, 5.55, p = 0.028), indicating a bias toward responding further along the trajectory of rotation. Note that in the expected condition the penultimate stimulus was always either 30 degrees CW or CCW relative to the target, so there are only two data points in this analysis. Comparing the difference in mean error between ±30 degrees in the expected condition to the difference in mean error between ±30 degrees in the random condition verified that the bias was significantly reversed at ±30 degrees in the random compared to expected conditions (mean difference = −7.96 degrees, 95% CI = −10.34 to −5.69, p = 1.0*10⁻⁴). This indicates that attractive serial dependence in orientation estimation can be reversed in the face of expectations induced via rotation. 

Serial dependence curves in the unexpected condition followed a more complex, bi-phasic attractive and repulsive profile better fit by a sinusoid. As shown in Figure 3C, the amplitude of the sine wave fit was significantly greater than zero (mean sine amplitude = 3.48, 95% CI = 2.59, 4.40, p = 1.0*10⁻⁴). The bi-phasic profile in the unexpected condition could be reasonably approximated by summing the mean curve in the random condition with the mean data points from the expected condition (Pearson's r = 0.78, p = 1.7*10⁻³⁹). This is because the unexpected condition contained trials that happened, by chance, to appear very similar to the expected condition from the observer's point of view. For instance, because in the unexpected condition the target orientation was random, it could have appeared very close (or even exactly at) the orientation value expected from the preceding rotational sequence. Thus, this result, in essence, replicates the repulsive bias induced when the target orientation meets expectations but further shows that when the target violated expectations (e.g. target-penultimate orientation differences beyond approximately ±40 degrees) observers revert to an attractive bias, making responses more similar to prior inputs. 

Last, the attractive dependence found in the random condition was also present with a similar magnitude when referencing serial dependence to the mean orientation of the entire sequence, rather than just the penultimate orientation (mean DoG amplitude = 3.93 degrees, 95% CI = 2.88, 4.98, p = 1.0*10⁻⁴), suggesting an influence of many of the most recent orientations. The sinusoidal effect found in the unexpected condition was still somewhat evident (mean sine amplitude = 1.29 degrees, 95% CI = −0.63, 2.3, p = 0.066), although the magnitude was about half of that found when referencing to the penultimate stimulus, perhaps suggestive of a stronger effect of expectation violation regarding the most recent stimulus than on the average of the recent past. Finally, in the expected condition, no serial effect was observed with respect to the average of the entire preceding sequence (p = 0.92), which is reasonable given that the difference between the target and the mean of the entire sequence on expected trials is always around 90 to 70 degrees (depending on sequence length), which are not orientation differences that typically show serial dependence even in the random condition (e.g. the extrema of the x-axis in Figure 3A). 

Discussion

We studied how meeting and violating perceptual expectations impacted task performance, confidence, and serial dependence. Our principle findings concerning serial dependence are that, whereas a target with a random orientation shows an expected attractive bias to the most recent stimulus, responses were repulsed from prior stimuli on expected trials and were attracted to them on unexpected trials. In most real-word contexts, stimuli are relatively stable over time. Attractive serial dependence, which assimilate the past into current estimates, is therefore theorized to be an adaptive mechanism that exploits natural temporal dependencies across stimuli in order to discount momentary noise in perceptual estimates. It may seem surprising then, that when a stimulus in our task was maximally expected based on preceding inputs it showed an opposite bias – a repulsive effect whereby targets were judged as being further from the penultimate stimulus. In addition, when a sequence conferred no degree of expectation or even when targets explicitly violated expectations, observers still showed an attractive bias toward the task-irrelevant penultimate stimulus. 

One possibility is that the repulsive effect observed on expected trials is an instance of representational momentum (RM). RM refers to a bias whereby dynamic stimuli for which there is some implied trajectory or consistency over time are reported as further along their implied trajectory than they really are. For instance, a moving triangle might be reported and remembered as further along the trajectory of motion than it actually was (Freyd & Finke, 1984). In this sense, RM refers to a bias toward future stimulus states, rather than past ones (as in serial dependence). Our results show that strong expectations can bias perceptual reports toward future states, whereas a lack or even violation of expectations can bias perceptual reports toward past states. Although the large and sudden steps in degrees (±30) used in our rotating sequences does not produce a perception of smooth motion, there is an implied rotational movement, which is commonly found to induce RM (Brouwer, Franz, & Thornton, 2004; Merz, Meyerhoff, Frings, & Spence, 2020; Pascucci & Plomp, 2021). Interestingly, RM does not always manifest for other continuous changes that are not motion-based, such as gradual luminance changes (Brehaut & Tipper, 1996) or gradual morphing manipulations (Thornton & Hubbard, 2002). Thus, it is possible that the repulsive bias we see in the expected condition is specific to expectations induced via implied movement. Future work manipulating expectations via other types of cues can test the generality of this finding. Regardless of the mechanisms underlying the repulsive effect in the expected condition, though, our work shows, for the first time, that violating a rotation-based expectation breaks RM and reverts the bias back to an attractive one. 

Our findings stand somewhat in contrast to a recent paper that studied serial effects using similar rotating grating sequences wherein targets were either expected or random (Pascucci & Plomp, 2021; see also Pascucci et al., 2019; experiment 5). Although that study found a repulsive effect for expected sequences, like we did, they did not observe an attractive bias in the random condition. They reasoned that attractive serial dependence is typically strongest for attended, task-relevant stimuli and because the penultimate stimulus in the sequences requires little-to-no attention and is not task-relevant, there was no attractive bias. However, our results clearly show an attractive bias to the penultimate stimulus, both in the random condition and in the unexpected condition when the targets clearly violated expectations. Some possible stimulus and task differences between the studies could be relevant. First, Pascucci and Plomp (2021) used a block-wise manipulation, such that trials of random or expected sequences were occurring in different blocks and cues were provided before each block. In our case, observers did not know if the sequence would be random, expected, or unexpected before the trial began. Whether such higher-level expectations can influence serial effects remains an open question. The stimuli in Pascucci and Plomp (2021) were also somewhat shorter (200 ms each) and presented with short (400 ms) gaps between, perhaps weakening any attractive effect. Last, an important difference is that we used a strong masking stimulus after the target so that observers could not base their responses on an after image of the target. This was not the case in both prior papers (Pascucci et al., 2019; Pascucci & Plomp, 2021) 

Beyond effects on serial dependence, we also observed an interesting pattern of results regarding error and confidence. Given prior work suggesting that expectations sharpen perceptual representations (Kok et al., 2012; Kok, Brouwer, van Gerven, & de Lange, 2013), we anticipated a reduction of response errors on expected trials because observers could predict in advance the precise target orientation. This reduction was evident in both response errors and subjective uncertainty, although we do not interpret these effects as necessarily perceptual in nature, because observers could have responded based on just their expectation. We also anticipated a reduction of error on unexpected trials (when compared to the random condition) given recent findings, using this very same paradigm, that expectation violations can sharpen sensory responses (Tang et al., 2021). However, we did not see a reduction of response error on unexpected trials. Interestingly, though, observers did report an increase in certainty on unexpected trials, suggesting that this particular form of expectation violation may partly dissociate confidence from objective performance. 

To summarize, we find that expectations induce a serial effect akin to representational momentum, pushing perceptual reports towards future stimulus states, rather than past ones. This could be seen as a violation of the oft-cited principle that serial dependence emerges as an adaptive strategy used to improve perceptual estimates in the face of temporally predictable stimulus features. Even more striking, target stimuli that explicitly violated expectations were still judged as more similar to past stimuli than they really were, indicating that temporal stability or predictability of stimulus features is indeed not a prerequisite for attractive serial dependence to occur. 

Acknowledgments

Commercial relationships: none. 

Corresponding author: Jason Samaha. 

Email: [e-mail]. 

Address: Psychology Department, University of California Santa Cruz, 1156 High Street, Santa Cruz, CA 95064, USA.