Flexible top-down control in the interaction between working memory and perception

Abstract

Successful goal-directed behavior often requires continuous sensory processing while simultaneously maintaining task-related information in working memory (WM). Although WM and perception are known to interact, little is known about how their interactions are controlled. Here, we tested the hypothesis that WM perception interactions engage two distinct modes of control – proactive and reactive – in a manner similar to classic conflict-adaptation tasks (e.g. Stroop, flanker, and Simon). Participants performed a delayed recall-of-orientation WM task, plus a standalone visual discrimination-of-orientation task the occurred during the delay period, and with the congruity in orientation between the tasks manipulated. Proactive control was seen in the sensitivity of task performance to the previous trial's congruity (i.e. a Gratton effect). Reactive control was observed in a repulsive serial-dependence produced by incongruent discriminanda. Quantitatively, these effects were explained by parameters from a reinforcement learning-based model that tracks trial-to-trial fluctuations in control demand: reactive control by a phasic control prediction error (control PE), and proactive control by a tonic level of predicted conflict updated each trial by the control PE. Thus, WM-perception interactions may be controlled by the same mechanisms that govern conflict in other domains of cognition, such as response selection.

Introduction

Working memory (WM) and perception are highly intertwined processes. Influential models have suggested that they may rely on overlapping cognitive and neural resources (e.g. Desimone & Duncan, 1995; Kastner & Ungerleider, 2000; Awh & Jonides, 2001; D'Esposito & Postle, 2015). Neurally, visual WM and visual perception have been found to share common codes in visual processing areas (e.g. Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009; Riggall & Postle, 2012; Rademaker, Chunharas, & Serences, 2019). Behaviorally, information maintained in WM impacts perceptual processing and is, in turn, biased by sensory information (e.g. Olivers, Meijer, & Theeuwes, 2006; Gayet, Paffen, & van der Stigchel, 2013; Kiyonaga & Egner, 2016; Teng & Kravitz, 2019; Teng & Postle, 2021). By one account, these interactions are automatic (Soto, Heinke, Humphreys, & Blanco, 2005; Soto, Hodsoll, Rotshtein, & Humphreys, 2008) because attention can be involuntarily captured by distractors matching the content in WM during visual search, even when this influence impairs task performance (Soto et al., 2005; Olivers et al., 2006). By contrast, there are several lines of evidence that the influence of WM on perceptual processing is at least partly subject to voluntary top-down control. In particular, the interaction between WM and perception varies according to the task demands. Increased probability of color-word incongruity is associated with a decrease in the cost of incongruity on a WM version of the Stroop task (Kiyonaga & Egner, 2014) and a decrease in attentional capture by the contents of WM during concurrent visual search (Woodman & Luck, 2007; Carlisle & Woodman, 2011; Kiyonaga, Egner, & Soto, 2012). Such top-down cognitive control is characterized by its flexibility in adapting to changing environmental demand of cognitive control on the fly (Botvinick, Braver, Barch, Carter, & Cohen, 2001; Shenhav et al., 2013; Egner, 2017). However, little is known about the temporal dynamics of the flexible cognitive control of WM and perception. Here, we addressed this by assessing inter-trial adjustment of control of WM-perception interactions and formalizing these control-related processes, with the goal of providing a general framework to account for these observations. 

Typically, cognitive control is studied in the context of individual tasks in which the level of conflict can vary from trial to trial (for example, word and color congruity in the Stroop task [Stroop, 1935]), or the potency of flanking stimuli in the Ericksen Flanker task (Eriksen & Eriksen, 1974), or the location congruity between stimulus and response in the Simon task, (Simon, 1990). Here, we addressed whether similar modes of control might govern the interaction between representations from two different sources – WM and visual perception – when the two are processed concurrently. We did this by adopting the dual-task procedure from Teng and Postle (2021), in which subjects performed a perceptual orientation discrimination task while maintaining a separate oriented stimulus in WM (Figure 1). The critical manipulation was the congruity between the discriminandum and the memorandum. We have previously reported that within a trial, incongruity between the two representations worsens sensitivity in the discrimination task and reduces precision in the WM task (Teng & Postle, 2021). In the current study, we focused on how memorandum-discriminandum incongruity might influence performance on the subsequent trial (c.f., Gratton, Coles, & Donchin, 1992). Although the WM and discrimination tasks were behaviorally independent from each other, we reasoned that because these processes share resources (e.g., Olivers et al., 2006; Gayet et al., 2013; Kiyonaga & Egner, 2016; Teng & Kravitz, 2019; Teng & Postle, 2021), incongruity between the two representations would be detected by the cognitive system to engage cognitive control. 

Figure 1.

 

Design of Experiment 1 and hypothesized temporal dynamics of proactive and reactive cognitive control with respect to stimulus representation. (A) Two consecutive trials. On each trial, memorandum and discriminandum were presented centrally and sequentially. The discriminandum required a speeded button press report of tilt (L/R, with left hand). Then, 0.5 seconds later, participants were prompted to report the orientation of the memorandum by clicking on the correct orientation on the circle with the mouse using their right hand. On trial n, the orientation of the discriminandum is congruent with that of the memorandum; on trial n+1 the discriminandum is incongruent. (B) Hypothesized level of proactive control (darker shading = higher level of control), which represents the predicted probability of incongruity between the memorandum and discriminandum and is updated on each trial by the control prediction error (control PE). (Note that the effects of proactive control on stimulus processing are not illustrated in this figure.) (C) Control PE, which reflects the discrepancy between the level of proactive control and the actual congruity of the discriminandum. (D) Reactive control is not triggered on congruent trial n but is triggered by the large control PE on incongruent trial n+1. (E) Illustration of the proposed influence of reactive control on the strength of the neural representation of the discriminandum, from the time of its onset until 0.5 seconds after the report of its tilt (discrimination response). On both trials, the visual presentation of the discriminandum triggers an increase in orientation channels encoding its orientation. On trial n, after stimulus offset, the absence of reactive control allows it to decay, whereas on trial n+1, the effect of reactive control is to suppress it below baseline. (Note that, for simplicity, this illustration does not include the concurrent representation of the memorandum, which is identical to that of the discriminandum on congruent trials.)

Cognitive control can operate in two modes: proactive and reactive (Braver, Gray, & Burgess, 2007; Braver, 2012). Proactive control involves the anticipatory adjustment of cognition and performance to counteract expected interference. A classic example is the “Gratton effect,” in which color-word conflict on a trial n in the Stroop task produces a distinctive change in performance on trial n+1: a reduction in the response-time (RT) cost if color and word are again incongruent, or a reduction in the RT benefit if color and word are congruent (Gratton et al., 1992). The level of proactive control recruited to guide behavior can be updated flexibly based on the integrated history of past events (Botvinick et al., 2001; Egner, 2007; Gratton et al., 1992; Jiang et al., 2014; Braem, Bugg, Schmidt, Crump, Weissman, Notebaert, & Egner, 2019). Reactive control, in contrast, refers to processes that are engaged as needed only after conflict is detected. For example, when the proportional congruity of an item was manipulated (e.g. the word “red” frequently colored in red versus the word “blue” frequently colored in an incongruent color), the likelihood of congruity was unknown until stimulus onset, and would require reactive engagement of control at the moment. RT was found to slow down in the incongruent condition but to a greater extent for the frequently congruent item than the frequently incongruent item (Bugg, 2012; Bugg & Hutchinson, 2013), suggesting reactive control processes associated with specific items. 

The fact that our procedure used two independent tasks allowed us to explore two novel questions. First, with regard to proactive control, would control configurations for each trial be adjusted flexibly based on the experienced (in)congruity of the preceding trial? We predicted an increase in the level of proactive control following an incongruent trial, leading to a reduction in the interactions between WM and visual perception. That is, because incongruity between memorandum and discriminandum has deleterious effects on performance of both tasks (Teng & Postle, 2021), incongruity experienced on trial n would induce subjects to approach trial n+1 in an elevated state of preparedness and/or caution (subjective states that correspond to the construct of proactive control). As a result, the costs to performance of incongruity on trial n+1 would be less than they had been on trial n, but so too would be the benefits of congruity trial n+1. Note that incongruity can happen at different levels of stimulus and response representations and drive cognitive control (Egner, 2008), so although the predicted pattern of decreased cost coupled with decreased benefit would echo that of the Gratton effect (Gratton et al., 1992), incongruity in our task would only exist at the level of stimulus processing, not as conflict in stimulus-response (S-R) mappings, because our two tasks were independent, with different response modes (c.f., Hazeltine, Teague, & Ivry, 2002). 

The second novel question that we planned to explore pertained to within-trial reactive control. Because discriminandum-memorandum incongruity degrades recall of the latter (Teng & Postle, 2021), we asked whether detection of incongruity also triggers a (reactively recruited) process to minimize interference that a trace of the incongruent discriminandum can exert on recall by suppressing the representation of the discriminandum. We explored this question by making predictions about two distinct dependent variables: WM recall precision; and the serial dependence of WM recall on trial n+1 on the discriminandum from trial n. With regard to WM recall precision, we drew on the fact that active suppression of interfering perceptual input can manifest as degraded subsequent performance with the suppressed feature (e.g. Sawaki & Luck, 2010; Gaspelin, Leonard, & Luck, 2015). For this reason, we predicted that suppression of an incongruent discriminandum on trial n would lead to worse WM recall on trial n+1 when the memorandum on trial n+1 was similar in orientation to the discriminandum from trial n. (Conversely, such n-to-n+1 similarity would improve recall precision on trial n+1 when trial n had featured a congruent discriminandum.) With regard to serial dependence, we drew on recent findings suggesting that the need to actively remove a stimulus held in WM has a distinctive downstream consequence: whereas the serial bias exerted by a critical item is attractive under conditions that do not encourage its active removal from WM, the serial bias exerted by that same item is repulsive under conditions that do encourage its active removal (Shan & Postle, 2022). Thus, we predicted that although WM recall would be attracted toward the previous trial's discriminandum when it had been congruent, it would be repelled away from the previous trial's discriminandum when it had been incongruent. (Note that, because our WM task did not entail the retention of more than one item, the sometimes attractive and sometimes repulsive effects between items simultaneously held in WM [e.g. Chunharas, Rademaker, Brady, & Serences, 2022] were not of primary theoretical interest, and are not considered in the main body of this report. Some results relating to within-trial stimulus interactions are presented as Supplementary Material). 

To foreshadow our results, in Experiment 1, we confirmed our predictions about both types of cognitive control: a Gratton effect indicated that congruity between stimuli in the dual-task prompted trial-by-trial adjustments of the level of proactive control; and two effects – deteriorated precision, and a repulsive serial bias exerted by incongruent discriminanda on recall on the subsequent trial – supported the idea that incongruent discriminanda elicited reactive suppression. To confirm our interpretation that these latter effects were a consequence of top-down reactive control and not a byproduct of a passive adaptation process, Experiment 2 used discriminanda that were equally difficult to process but for which congruity with the memorandum was not a relevant factor. Experiment 2 replicated the proactive control effects from Experiment 1 but did not show evidence of a repulsive serial bias produced by incongruent discriminanda, strengthening our confidence that incongruent discriminanda trigger suppression via reactive control. Finally, to ground these observations in a formal framework, we fit the data from Experiment 1 to a reinforcement learning-based model that has successfully accounted for cognitive control in a variant of the Stroop task (Jiang, Heller, & Egner, 2014; Jiang, Beck, Heller, & Egner, 2015). This suggests that interactions between WM and perception may be controlled by the same processes that flexibly adjust cognitive control based on environmental demand, thereby providing a parsimonious framework for the top-down control of stimulus-guided action and of interactions between stimulus representations needed for different tasks. 

Experiment 1

To examine the temporal dynamics of the flexible cognitive control of WM and perception, we assessed inter-trial adjustment of control in the WM-perception interaction in a dual-task paradigm involving discrimination on a perceptual stimulus while simultaneously maintaining another item of unpredictable congruity in WM. On each trial, subjects first encoded the orientation of a Gabor patch into WM, then, during the delay, performed a “left”/“right” tilt discrimination task on a Gabor embedded in noise and finally recalled the orientation of the memorandum by adjusting a response dial (see Figure 1A). Figure 1A illustrates two consecutive trials, and the rows below it depict the hypothesized dynamics of control and stimulus representation. In general, the level of proactive control increases after an incongruent trial and decreases after a congruent trial. Figure 1B illustrates a relatively low level of proactive control at the beginning of trial n (indicated by light shading) that persists until after the processing of the trial's discriminandum. Because trial n being congruent matches the low proactive control state, a relatively weak control prediction error (control PE; the extent of mismatch between the proactive control level and the observed congruity; see Model-based analysis with Flexible Control Model) is generated (see Figure 1C), which has two consequences: first, reactive control is not engaged (see Figure 1D); second, the level of proactive control decreases (see Figure 1B; indicated by transition to lighter shading). Finally, a consequence of the absence of reactive control is that the representation of the discriminandum is allowed to gradually decay (see Figure 1E). The decreased level of proactive control persists until the onset of trial n+1’s discriminandum. At this point, because the discriminandum on trial n+1 does not match the memorandum (see Figure 1A), a strong control PE is generated (see Figure 1C), and this has two consequences: first, reactive control is engaged (see Figure 1D); second, the level of proactive control increases (see Figure 1B; indicated by transition to darker shading). The consequence of engaging reactive control is that the representation of this discriminandum is suppressed (see Figure 1E). It should be noted that there are other scenarios of control dynamics that are not visualized in Figure 1. For example, control PE would also be high on a congruent trial in a high proactive control state, but the hypothesized reactive control process would not be triggered on those trials because of the absence of the memorandum-discriminandum interference. 

Methods

Subjects

Forty-six individuals were recruited from Amazon Mechanical Turk (14 women; mean age = 38.5 ± 9.4). All subjects reported normal or corrected-to-normal vision and were provided with informed consent approved by the University of Wisconsin–Madison Health Sciences Institutional Review Board. All were compensated monetarily for their participation. 

Procedure and stimuli

Stimuli were created with customized JavaScript code and presented remotely to subjects using PsiTurk (Gureckis, Martin, McDonnell, Rich, Markant, Coenen, Halpern, Hamrick, & Chan, 2016). 

The experiment comprised five experimental blocks, followed by one control block. In the experimental blocks, each trial began with the onset of a red dot at central fixation. After 0.5 seconds, the dot turned white and, concurrently, a Gabor patch (“memorandum”; radius = 100 pixels; contrast = 0.6; spatial frequency = 0.02 cycles/pixel; phase angle randomized between 0 degrees and 180 degrees) appeared for 0.5 seconds at the center of the screen. Subjects were instructed to remember this orientation as precisely as possible throughout the trial (the white dot remained at center of the screen for the duration of the trial). Following a 1 second inter-stimulus interval (ISI), a second Gabor patch (“discriminandum”) appeared centrally for 0.2 seconds, and subjects were instructed to report whether the discriminandum was tilted clockwise or counterclockwise to the vertical direction as quickly as possible with key presses (“E” and “F” keys). Then, 0.5 seconds after this response, the memory probe (orientation dial with a random starting orientation) appeared, and subjects were instructed to use a mouse click on the dial to align it with the orientation of the memorandum. The inter-trial interval (ITI) was 1 second (see Figure 1A). 

The procedure for the control task was the same as the experimental task, but the discriminandum was a hexagon (radius = 200 pixels), and subjects made a judgment based on whether one side was concave (rather than straight; Supplementary Figure S1). Thus, this control task featured an equally difficult discrimination task but one for which congruity relative to the memorandum was not a factor. 

The orientation of each memorandum was selected randomly, with replacement, from a fixed set of 12 values spaced by 15 degrees, and with jitter ranging from 0 degrees to ±3 degrees added for each trial. On experimental blocks, the discriminandum was a Gabor patch with similar properties to the memorandum, except its contrast varied unpredictably (random selection with replacement from 6 values: 0.04, 0.12, 0.20, 0.28, 0.36, 0.44, 0.52, or 0.60), and noise was added by replacing 20% of the pixels with random brightness. Congruity in orientation between memorandum and discriminandum was manipulated, such that, unpredictably, on half of trials it matched, and on half of trials the orientation of the discriminandum differed from that of the memorandum by 35 degrees or 75 degrees, clockwise or counterclockwise, randomly determined. (Note that this means that, for analyses that compare the effects of an incongruent discriminandum versus a congruent discriminandum, for the latter we cannot disentangle the influence of a congruent discriminandum, per se, versus the effect that this congruent discriminandum might have on boosting the representation of that trial's memorandum.) Each subject completed five 60-trial blocks of the experimental task with orientation congruity and the six levels of discriminandum contrast counterbalanced across blocks. Then they completed one 50-trial block of the control task. 

Analysis of discrimination task data

We summarized performance in the discrimination task according to accuracy and RT. Accuracy was measured in terms of percentage of trials with a correct clockwise or counterclockwise response. For RT analyses, RTs on trials with incorrect responses were first discarded. Further exclusion occurred for trials with RTs beyond 2.5 standard deviations of the individual subject's mean RT. This resulted in an average exclusion of 1% of trials per subject. The within-subject mean RTs were used for the statistical analyses reported below. The between-subjects mean RTs were used for visualization purposes. 

Serial dependence analysis

We quantified the influence of the discriminandum from a given trial on WM recall on the subsequent trial using a standard model-based analysis of serial dependence (Bliss, Sun, & D'Esposito, 2017; Fritsche, Mostert, & de Lange, 2017; Samaha, Switzky, & Postle, 2019). First, trials with errors greater than 45 degrees were discarded, in order to exclude trials that were likely to be guesses. Second, for each subject, we subtracted the average error (signed) across trials from each trial's error. This demeaning step removed systematic response bias (either clockwise or counterclockwise). 

Trials were divided based on trial n congruity. Errors on trial n+1 were sorted by the relative difference in orientation between the discriminandum of trial n and the memorandum of trial n+1. Then, separately for congruent and incongruent conditions on trial n, a derivative-of-Gaussian function (DoG; Fischer & Whitney, 2014; Bliss et al., 2017; Fritsche et al., 2017) was fitted to the data (see below). The DoG function has the following form:  

\begin{equation*}y = xawc{e^{ - {{\left( {wx} \right)}^2}}}\end{equation*}

where x is the relative orientation of the discriminandum of the previous trial, a is the amplitude of the peaks of the curve, w is the width of the curve, and c is the constant of \(\sqrt 2 / {e^{ - 0.5}}\), which scales the amplitude parameter to numerically match the height of the curve in degrees. 

Before fitting the data, each subject's data were smoothed by a 15-trial moving median filter, following the convention of previous studies (Bliss et al., 2017; Fritsche et al., 2017; Samaha et al., 2019). The DoG was then fit to the group-averaged data. Fitting involved two free parameters, amplitude a and width w, which were allowed to vary between −15 degrees to 15 degrees and 0.02 to 0.2, respectively. An amplitude greater than zero indicates an attractive serial dependence bias toward previous trial discriminandum; an amplitude less than zero indicates a repulsive bias away from previous trial discriminandum. The width w parameter scales the width of the curve, and a greater value means a broader tuning width. 

Statistical significance was determined by a bootstrapping procedure (80,000 iterations) in which we sampled 46 subjects with replacement and fit the DoG function to the average of the resampled group data with the same procedure described above. For each iteration of the bootstrap, we recorded the amplitude of the fit, which resulted in a final distribution of 80,000 amplitude values. A two-tailed p value was calculated as the smaller of the two: proportion of amplitude values above zero, and proportion of amplitude values below zero, multiplied by two. 

We also computed a model-free measure of serial bias by computing the median error (signed) for trials for which the relative distance between the trial n discriminandum and trial n+1 memorandum fell within the 0 degree to 45 degrees range (both directions) and subtracting that from the median error on all trials within that range (e.g. Samaha et al., 2019). 

Additional analyses

In addition to what we have reported here, our design allows for the examination of several additional questions that, although potentially interesting, are not germane to the present theoretical questions. We carried out analyses related to two of these questions – the effect of discriminandum contrast on serial dependence effects, and within-trial biasing interactions between memoranda and incongruent discriminanda – and the results are reported as Supplementary Material. 

Results

Proactive control

Does an instance of incongruity cause an increase in the level of proactive control? To answer this, we examined whether the congruity effect – the difference in performance between congruent and incongruent conditions – on the discrimination task on trial n+1 was modulated by congruity on trial n. Specifically, if an incongruent discriminandum on trial n prompts an increase in the level of proactive control, the RT cost of an incongruent discriminandum on trial n+1 would be reduced, as would the benefit of a congruent discriminandum on trial n+1. Figure 2 indicates that this was, indeed, the pattern of the results. For performance on the discrimination task, a repeated-measures analysis of variance (ANOVA) was conducted on RT and accuracy, with factors of trial n (previous trial) congruity and trial n+1 (current trial) congruity. For RT, there was a significant main effect of trial n+1 congruity (F(1,45) = 17.90, p < 0.001, partial η² = 0.29). The main effect of trial n congruity was not significant (F(1,45) = 0.06, p = 0.80, partial η² = 0.001). Importantly, there was a significant interaction (F(1,45) = 12.35, p = 0.001, partial η² = 0.22), with the difference between the congruent and incongruent condition on trial n+1 being smaller after an incongruent trial n, as predicted by our proactive control hypothesis (see Figure 2A). Accuracy on the discrimination task was higher for congruent (M = 90%; 95% confidence interval [CI] = 86% to 94%) than incongruent (M = 79%; 95% CI = 73% to 85%) discriminanda, (F(1,45) = 17.98, p < 0.001, partial η² = 0.29), replicating the findings in Teng and Postle (2021). There was no main effect of trial n congruity, and no interaction (p values > 0.23). Finally, a repeated-measures ANOVA on the precision of WM recall revealed a main effect of trial n congruity (F(1,45) = 39.13, p < 0.001, partial η² = 0.47), with the main effect of trial n+1 congruity approaching significance (F(1,45) = 3.94, p = 0.053, partial η² = 0.08), and no interaction (F(1,45) = 1.17, p = 0.29, partial η² = 0.03; see Figure 2B). 

Figure 2.

 

The influence of a trial's congruity on the congruity effect on the subsequent trial. (A) Response time (RT) in the discrimination task plotted by previous (trial n) and current (trial n+1) trial congruity. When trial n was incongruent, the congruity effect on trial n+1 was reduced, a hallmark of increased proactive control. (B) Precision of the memory report was influenced by trial n congruity with a trend-level effect of trial n+1 congruity and no interaction, a pattern revealing no influence of proactive control. Con: congruent; Inc: incongruent. Error bars correspond to normalized ±1 SEM (Morey, 2008).

Reactive control

Our working model posits that the detection of an incongruent discriminandum triggers the (reactive) suppression of the representation of that discriminandum, in order to minimize the influence of its residual trace on WM recall on that same trial (i.e. on trial n.). We assessed evidence for this reactive control with two dependent measures: WM recall precision on trial n+1 and serial bias on trial n+1. 

WM recall

Active suppression of interfering perceptual input can manifest as degraded subsequent performance with the suppressed feature (e.g. Sawaki & Luck, 2010; Gaspelin et al., 2015). Based on this, we examined the precision of WM recall on trial n+1 as a function of the relative distance between the memorandum from trial n+1 and the discriminandum from trial n. The logic was that suppression of the incongruent discriminandum on trial n should disrupt the processing of the memorandum on trial n+1 in an orientation-dependent way (i.e. more disruption for memoranda more similar to the previous trial's discriminandum). To test this, we first split trials by previous-trial congruity, then, for each condition, sorted them into four bins of relative orientation distance: 0 to 22.5 degrees, 22.5 degrees to 45 degrees, 45 degrees to 67.5 degrees, and 67.5 degrees to 90 degrees. Inspection of Figure 3A reveals a pattern that is consistent with the reactive suppression hypothesis: when the previous trial's discriminandum had been congruent, recall precision was best for small relative distances; but when the previous trial's discriminandum had been incongruent, recall precision was worst for small relative distances (even lower than baseline for the 0-22.5 degrees bin). A repeated-measures ANOVA with factors of previous trial congruity and relative orientation difference revealed no main effects of congruity or distance (F(1,45) = 3.76, p = 0.06, partial η² = 0.08; F(3,135) = 0.58, p = 0.63, partial η² = 0.01, respectively), but a significant interaction between the two factors (F(3,135) = 12.06, p < 0.001, partial η² = 0.21). Furthermore, compared with the baseline level of precision as measured in the control block, recall precision was enhanced for close-distance conditions after a congruent trial (0-22.5 degrees: t(45) = 3.69, p < 0.001, Cohen's d = 0.54; 22.5-45 degrees: t(45) = 2.26, p = 0.028, Cohen's d = 0.34; uncorrected), but impaired for close-distance conditions after an incongruent trial (0-22.5 degrees: t(45) = 2.16, p = 0.036, Cohen's d = 0.3; uncorrected), effects that differed from each other (t(45) = 4.06, p < 0.001, Cohen's d = 0.69). 

Figure 3.

 

Evidence for reactive suppression of the incongruent discriminandum. (A) Recall precision on trial n+1 as a function of relative distance from discriminandum on trial n. When the discriminandum from trial n was congruent, precision on trial n+1 was highest for the smallest relative distance, and declined with increasing distance to an asymptotic level. In contrast, when the discriminandum from trial n was incongruent, precision on trial n+1 was lowest for the smallest relative distance and increased with increasing distance to an asymptotic level. Dashed line is precision from the control task. (B) Serial dependence of WM recall on the discriminandum from the previous trial: when the discriminandum from trial n was congruent, its influence on WM recall on trial n+1 was attractive; when the discriminandum from trial n was incongruent, its influence on WM recall on trial n+1 was repulsive. Shaded bands represent ±1 SEM; Solid curves correspond to DoG fits; *corresponds to bootstrapped p values < 0.05; Error bars correspond to normalized ±1 SEM (Morey, 2008).

To rule out the possibility that the poorer recall precision on trials with small relative distances from the previous trial's incongruent discriminandum was due to the influence of that trial's memorandum, we also assessed recall precision on trial n+1 as a function of the relative distance in orientation between the memorandum from trial n and the memorandum from trial n+1. This revealed a trend toward numerically higher precision for the close-distance conditions, although this effect did not achieve statistical significance (F(3,135) = 2.28, p = 0.09, partial η² = 0.05; Supplementary Figure S2B). 

Serial dependence

In a previous study of WM, using similar stimuli, a critical item from trial n exerted a repulsive bias on trial n+1 when subjects were encouraged to actively remove it from WM, but the same item exerted an attractive bias when subjects could just “passively drop” it from WM (Shan & Postle, 2022). Based on this, we reasoned that if incongruent memoranda were subject to reactive suppression, they might also exert a repulsive serial bias on recall on trial n+1. The results were consistent with this reasoning: whereas WM recall on trial n+1 exhibited a significantly attractive bias toward congruent discriminanda from trial n (a = 1.51 degrees, 95% CI = 0.85 degrees to 2.37 degrees, p = 0.006, bootstrapped), it exhibited a significantly repulsive bias away from incongruent discriminanda from trial n (a = −1.1 degrees, 95% CI = −1.92 degrees to −0.4 degrees, p = 0.028, bootstrapped; Figure 3B). The difference in absolute amplitude of these two serial dependence effects was not significant, p = 0.28 (bootstrapped). The width parameters w did not differ between the attractive (w = 0.02, 95% CI = 0.02 to 0.09) and the repulsive (w = 0.047, 95% CI = 0.02 to 0.07) serial biases, p = 0.31 (bootstrapped). (It is worth noting that memorandum and discriminandum had exact the same orientation on congruent trials, and thus the enhanced recall precision and attractive bias may reflect influences from both or either of the two representations.) 

To verify that our results were not due to the choice of fitting procedure, we computed a model-free estimate of serial bias. One-sample t-tests against zero showed a significant attractive bias following congruent trials, 2.12 degrees, t(45) = 2.99, p = 0.005; and a significant repulsive bias following incongruent trials, −1.98 degrees, t(45) = −3.84, p < 0.001, supporting the model-based results above. 

Because two orientations were presented on incongruent trials, and both were likely to impact subsequent trial processing, we also analyzed serial dependence of WM recall on trial n+1 with respect to the orientation of the memorandum on trial n in this subset of trials. The purpose of this analysis was to rule out the concern that the repulsive bias was in fact reflecting the influence of the memorandum, because the orientations of the memorandum and the discriminandum were correlated with fixed distances of ±35 degrees/75 degrees apart (see Methods). Serial dependence on the previous trial's memorandum was numerically attractive for incongruent trials (a = 0.42 degrees, 95% CI = −0.55 degrees to 1.42 degrees), with a w of 0.02, 95% CI = 0.02 to 0.08; see Supplementary Figure S2A), but this attractive bias did not approach statistical significance (p = 0.33, bootstrapped). Thus, the lack of a significant repulsive influence from trial n’s memorandum effectively rules out this concern. For completeness, we also combined all the trials across congruity and examined the serial dependence on the previous trial's memorandum, and found it to be attractive (a = 1.09 degrees, 95% CI = −0.50 degrees to 1.79 degrees, p = 0.03, bootstrapped), with a w of 0.022, 95% CI = 0.02 to 0.12. 

Discussion

Using a task design that allowed us to assess the influence of one trial on the next, we found that the interaction between WM and perception is shaped by recent trial history. In particular, the effects of congruity from trial n on several aspects of performance on trial n+1 suggest recruitment of both proactive control and reactive control to govern the interaction between stimulus representations associated with the two tasks. 

A hallmark of proactive control was seen in the fact that incongruity on trial n led to a decrease in the cost of incongruity, as well as to a decrease in the benefit of congruity, of discrimination performance on trial n+1 (see Figure 2A). This Gratton effect suggests that, just as is the case with the Stroop task, an instance of incongruity leads to an increase in the level of proactive control that manifests as “more cautious” performance on the subsequent trial. This implies that similar proactive control mechanisms may be recruited to control perception-WM interactions as are recruited to control conflict in S-R mappings (Gratton et al., 1992). We note that this influence of the previous trial's congruity was not observed in WM recall performance (see Figure 2B). This could be a consequence of the nested design of our dual-task procedure. Model-based analysis of these data, to be reported after Experiment 2, suggests that it is a signal generated by discriminandum congruity that prompts the adjustment of proactive control (see Figure 1). This would have the effect of obscuring the influence of the level of proactive control that had been set during the previous trial. 

Whereas proactive control uses congruity on trial n to optimize performance on trial n+1, we assume that reactive control is recruited to optimize performance on trial n (because memorandum-discriminandum incongruity impairs WM recall on that same trial). More specifically, our working model is that the detection of an incongruent discriminandum triggers the (reactive) suppression of the representation of that discriminandum, in order to minimize the influence of its residual trace on WM recall. Consistent with this account is the fact that the influence of the discriminandum from trial n on WM recall on trial n+1 is reversed depending on congruity: a congruent discriminandum on trial n exerts an attractive serial bias on WM recall on trial n+1 and boosts the precision of recall of similarly-oriented memoranda; an incongruent discriminandum on trial n, in contrast, exerts a repulsive serial bias on WM recall on trial n+1 and worsens the precision of recall of similarly oriented memoranda. 

It is important to note that our interpretation of the repulsive serial bias exerted by incongruent memoranda differs from at least one other prominent account of serial dependence in WM. Whereas we interpret it as a result of reactive suppression, others have proposed that repulsive serial bias is a consequence of the reactivation of stimulus representations from the previous trial. Specifically, Bae and Luck (2019) have provided evidence for this “trace reactivation” mechanism by recording the electroencephalogram (EEG) while subjects performed delayed recall of single orientations, and found that although decodability of the memorandum from trial n disappeared after the end of the trial, it returned during portions of trial n+1. Importantly, in these experiments, they also observe a repulsive effect of the memorandum from trial n on recall on trial n+1. The interpretation, then, is that the interaction of the reactivated trace from trial n and the current item from trial n+1 results in a repulsive bias, for the same reasons that two items held on the same trial repel each other (e.g. Chunharas et al., 2022). Indeed, we also found evidence for within-trial repulsive interactions between memoranda and incongruent discriminanda. We think it unlikely, however, that trace reactivation can account for the patterns of serial dependence in our data for the simple reason that on congruent trials we observed an attractive serial bias. Trace reactivation can only explain repulsive serial biases, and so cannot apply to results like ours, and several others (e.g. Fischer & Whitney, 2014; Fritsche et al., 2017; Czoschke, Fischer, Beitner, Kaiser, & Bledowski, 2019; Samaha et al., 2019), that find attractive serial biases. Although a detailed consideration of what factors might predict attractive versus repulsive serial biases in WM is outside the scope of this paper, we will return, in the General Discussion, to an account that may accommodate the results from this report. Nevertheless, another prevailing account for repulsive bias is that it reflects the tendency of adaptation in perceptual circuits to repel the features of the previously processed stimulus (Pascucci, Mancuso, Santandrea, Della Libera, Plomp, & Chelazzi, 2019; Fritsch et al., 2020; Trapp, Pascucci, & Chelazzi, 2021). Considering this possibility, it was important that we confirm that the repulsive serial bias observed in Experiment 1 was truly a consequence of goal-directed reactive control, and not a byproduct of a passive adaptation process. 

Experiment 2

In order to confirm that the repulsive serial bias observed in Experiment 1 (see Figure 3A) was a consequence of top-down reactive control and not a byproduct of a passive adaptation process, we repeated the procedure from Experiment 1, but with a modification intended to minimize the active processing of the oriented Gabor while making a perceptual discrimination. Specifically, in Experiment 2, the perceptual discrimination stimulus was a small shape overlayed on the Gabor (see Supplementary Figure S1B), and subjects were instructed to make a “circle”/ “square” judgment about the shape. We reasoned that if the repulsive bias from Experiment 1 was a product of reactive control, the irrelevance of the orientation of the Gabor in the discrimination stimulus would reduce the likelihood that incongruity with the orientation of the memorandum would trigger reactive suppression. This would predict an absence of the repulsive serial bias and the relative-distance effects observed in Experiment 1 (see Figure 3). Alternatively, if these effects were caused by passive adaptation, this would predict a replication of the repulsive serial bias and the relative-distance effects observed in Experiment 1. 

Method

Subjects

An a priori power analysis was carried out based on the repulsive serial bias reported in Experiment 1 to determine the minimal sample size required to achieve a bootstrapped two-tailed p value of 0.05. We first randomly selected with replacement a subset of subjects (starting with a sample of 1 subject and iterating up to a sample of 46 subjects). For each selected sample, a DoG curve was fit. This process was repeated 10,000 times, and the p value corresponding to the amplitude of the repulsive bias reported in Experiment 1 (i.e. the negative DoG fit plotted in Figure 3A) was determined by finding its percentile within the bootstrapped distribution and taking the smaller value of percentile 100-percentile. We repeated this process 500 times for each sample size, resulting in a distribution of 500 p values each. Last, we identified the smallest sample size for which 95% of the generated p values were smaller than 0.05, which was 43 subjects. Thus, to be certain that our sample size was sufficient to detect the effect if it exists, 47 individuals were recruited via Amazon Mechanical Turk (22 women; mean age =36.4 ± 7.3 years). All subjects reported normal or corrected-to-normal vision and were provided with informed consent approved by the University of Wisconsin–Madison Health Sciences Institutional Review Board. All were compensated monetarily for their participation. 

Procedure and stimuli

The stimuli and procedure were the same as those in Experiment 1 with the following exceptions. The discrimination task presented a small shape (a circle [6 pixels in radius] or a square [10 pixels in width]) at the center of the screen and overlaying a noisy Gabor (same properties as Experiment 1) for 0.2 seconds. Subjects reported their speeded “circle”/“square” judgment with key presses (“E” and “F” keys). Although irrelevant for the discrimination task, the orientation of the Gabor was congruent with the orientation of the memorandum on 50% of the trials, and incongruent (35 or 70 degrees away) on the other 50% trials. 

Subjects first completed five blocks of 60 trials of the experimental task, 300 in total, and then they completed a 50-trial block of control trials (identical to Experiment 1). 

Results

Proactive control

RTs on the discrimination task were markedly longer than in Experiment 1. Additionally, despite the irrelevance of orientation for the discrimination task, congruity of the discrimination stimulus on trial n did influence discrimination performance on trial n+1 in a manner suggesting the recruitment of proactive control. An incongruent discrimination stimulus on trial n resulted in a loss of the congruity effect on trial n+1 and reduced RT in both conditions of trial n+1. This was confirmed by a repeated-measures ANOVA, which revealed no main effect of trial n, or of trial n+1, trial congruity (F(1,46) = 2.23, p = 0.14, partial η² = 0.05; F(1,46) = 3.23, p = 0.08, partial η² = 0.07, respectively) but did reveal a significant interaction between the two factors (F(1,46) = 4.66, p = 0.036, partial η² = 0.09). A follow-up pairwise comparison confirmed that when trial n was incongruent, RT on trial n+1 did not differ between congruent and incongruent discrimination trial types (t(46) = 0.98, p = 0.33, Cohen's d = 0.14; Figure 4A). No effect was found for accuracy in this task. Subjects’ accuracy was at ceiling level (average of 94% correct). Finally, a repeated-measures ANOVA on the precision of WM recall revealed no main effect of trial n congruity (F(1,46) = 0.12, p = 0.73, partial η² = 0.003), a main effect of trial n+1 congruity (F(1,46) = 4.43, p = 0.041, partial η² = 0.09), and a significant interaction between the two (F (1,46) = 8.27, p = 0.006, partial η² = 0.15). Consistent with the RT results, incongruity on trial n abolished the difference between congruent and incongruent discrimination accuracy on trial n+1 (t(46) = 0.22, p = 0.83, Cohen's d = 0.07; Figure 4B). 

Figure 4.

 

Performance in the discrimination task and memory recall in Experiment 2. (A) The congruity effect in RT on the discrimination task on trial n+1 was not present (compare with Figure 2A). (B) The congruity effect in the precision of WM recall on trial n+1 was also not present (compare with Figure 2B). Error bars correspond to normalized ±1 SEM (Morey, 2008).

Reactive control

WM recall

An influence of the relative distance between the discrimination stimulus from trial n and the memorandum from trial n+1 was only evident when the discrimination stimulus from trial n was congruent. Critically, and in contrast with Experiment 1, when the discrimination stimulus from trial n was incongruent, there was no influence of relative distance on the precision of WM recall on trial n+1 (Figure 5A). A repeated-measures ANOVA yielded a main effect of distance (F(3,138) = 6.37, p < 0.001, partial η² = 0.12) and a significant interaction between previous trial congruity and distance (F(3,138) = 6.00, p < 0.001, partial η² = 0.12). 

Figure 5.

 

Task-irrelevant incongruity did not elicit reactive suppression. (A) Following a congruent trial, precision of the memory report on the current trial was modulated by the similarity between previous trial distractor and current trial memorandum. No relationship was observed following an incongruent trial. Dashed line is precision from the control task. (B) Orientation of the discrimination stimulus on trial n exerted an attractive bias on WM recall on trial n+1 when it was congruent but no influence on WM recall on trial n+1 when it was incongruent. Shaded bands represent ±1 SEM; Solid curves correspond to DoG fits; *corresponds to bootstrapped p values < 0.05; n.s. = non-significant; Error bars correspond to normalized ±1 SEM (Morey, 2008).

For completeness, because the orientation of the memorandum was different from that of the discriminandum on an incongruent trial, we repeated the same analysis of recall precision on trial n+1 but in relation to the relative distance between the memorandum on trial n and the discriminandum on trial n+1. Precision increased with increasing similarity (F(3,138) = 8.36, p < 0.001, partial η² = 0.15; Supplementary Figure S4). 

Serial dependence

As was the case in Experiment 1, congruent discrimination stimuli exerted a significant attractive bias on WM recall on trial n+1 (a = 1.21 degrees, 95% CI = 0.74 degrees to 1.9 degrees, p = 0.035, bootstrapped), with a width parameter w of 0.026, 95% CI = 0.02 to 0.05. Additionally, and of primary theoretical importance, incongruent discrimination stimuli on trial n exerted a weakly, but not significant, attractive bias on WM recall on trial n+1 (a = 0.39 degrees, 95% CI = −0.28 degrees to 1.17 degrees, p = 0.14, bootstrapped), with a w of 0.023, 95% CI = 0.02 to 0.14 (Figure 5B). Model-free analysis of serial bias showed consistent results. One-sample t-test against zero revealed significant attractive bias following congruent trials, 1.02 degrees, t(46) = 2.02, p = 0.04; and insignificant repulsive bias following incongruent trials, −0.30 degrees, t(46) = −0.75, p = 0.45. Additionally, the attractive serial bias exerted by the memorandum from trial n when trial n was incongruent was significant (a = 1.53 degrees, 95% CI = 0.89 degrees to 2.28 degrees, p < 0.001, bootstrapped), with a w of 0.036, 95% CI = 0.02 to 0.21 (Supplementary Figure S5). 

Discussion

The goal of this experiment was to assess whether the finding of a repulsive serial bias exerted by the incongruent discriminandum from trial n on WM recall from trial n+1 in Experiment 1 may have been a consequence of passive perceptual adaptation. We therefore modified the discrimination stimulus by making the oriented Gabor irrelevant for the interpolated discrimination task, reasoning that this configuration would also exert a repulsive serial bias if this procedure could produce such a passive effect. The failure to find evidence for either a repulsive serial bias or an effect of relative orientation difference on WM recall on trial n+1 suggests that these effects from Experiment 1 were due to the active processing of distractor orientation, which cannot be explained as a consequence of passive perceptual adaptation. These findings are consistent with results from Bae and Luck (2020) showing that serial bias in motion direction was only observed when subjects had to report the motion direction on the previous trial, but not when the direction was merely encoded. Here, the modulation of repulsive bias by task demand suggests that reactive control is engaged only when the incongruent discriminandum requires post-perceptual processing. 

Interestingly, in Experiment 2, we still observed a modulation of the congruity effect by previous trial congruity, in both the discrimination task and memory recall, suggesting a reduction in the extent of interactions between WM and perception. Because cognitive control is effortful (Cools, 2016), it is likely that, depending on the task context, proactive and reactive control may not always both engaged, a pattern observed in a variety of conflict tasks (Braver, 2012; Gonthier et al., 2016; Braver et al., 2021).This suggests that a congruency-related signal (perhaps the control PE – see Model-based analysis with Flexible Control Model) is generated regardless of task relevance, but the interpretation of this signal for the recruitment of proactive versus reactive control can differ, depending on context. It is likely that the congruity between the two processes is always monitored regardless of the task-relevance of the perceptual stimuli and evokes a general, tonic change in top-down control, consistent with previous findings that control demand contingency associated with distractors could also influence behavior (Bugg, Jocoby, & Chanani, 2011; Chiu, Jiang, & Egner, 2017). 

In addition, we note that recall bias and precision covaried in almost all conditions across both experiments. Indeed, such covariation has been observed in previous studies (e.g. Cicchini, Mikellidou, & Burr, 2018) and may reflect the representational integration process to sharpen the precision of the memory representation (e.g. Fukuda, Pereira, Saito, Tang, Tsubomi & Bae, 2022). However, given the structure of our paradigm, we are unable to adjudicate whether the influence was from the memorandum or the congruent discriminandum. Alternatively, the covariation may be accounted for by a Bayesian ideal observer model which weights present and previous stimuli according to current stimulus reliability, thus supporting serial dependence as a beneficial outcome of an optimal response strategy in the face of sensory uncertainty (Cicchini et al., 2018). 

Model-based analysis with flexible control model

We have demonstrated that the interaction between WM and perception is influenced by the congruity of the previous trial and that this influence manifests in two qualitatively distinct ways: (1) a Gratton effect in the processing of the discriminandum on the subsequent trial; and (2) influences on the bias and precision of WM recall on the subsequent trial. We have reasoned, based on previous literature, that these effects may reflect the recruitment of proactive (e.g. Braver, 2012; Jiang et al., 2014) and reactive (e.g. Shan & Postle, 2022) control processes, respectively (see Figure 1). Importantly, these dual modes of cognitive control have typically been studied with conflict monitoring paradigms like the Stroop, Flanker, and Simon tasks (Stroop, 1935; Eriksen & Eriksen, 1974; Simon, 1990). In this final section, we evaluated whether the results from Experiment 1 could be accounted for by the formal, quantitative framework of a reinforcement learning-based model of cognitive control. Success would suggest that WM-perception interactions may be governed by the same mechanisms that govern conflict during speeded S-R tasks. 

The flexible control model (FCM; Jiang et al., 2014; Jiang et al., 2015) uses a reinforcement learning algorithm with a flexible learning rate to account for the flexible adjustment of control in a changing environment. This model has successfully simulated classic proportional congruency effects and sequential modulation effects in the Stroop paradigm (Jiang et al., 2014), has identified an anterior insula-basal ganglia-PFC (AI-BG-PFC) control network involved in conflict task performance (Jiang et al., 2015), and predicts effects of PFC-targeted transcranial magnetic stimulation (TMS) on Stroop behavior (Muhle-Karbe, Jiang, & Egner, 2018). An important advantage of this model over the analyses carried out for Experiment 1 is that it incorporates the complete trial history to compute trial-by-trial levels of proactive and reactive control, not just the previous trial. For example, an incongruent trial after five consecutive congruent trials might be expected to trigger larger changes in the levels of both reactive and proactive control than would an incongruent trial after five trials with alternating congruity. 

At the conceptual level, for each forthcoming trial, the FCM makes a prediction of the anticipated conflict (the probability of next trial being incongruent) based on congruity history; this is quantified in a predicted conflict parameter that can be viewed as a proxy for proactive control. On that trial, the actual observed congruity elicits a continuous and unsigned control prediction error (control PE) that has two effects: it triggers reactive control (and its value can be taken as a proxy for the magnitude of reactive control) and it is integrated with trial history to update the value of predicted conflict. (Note that the relation between the control PE and reactive control will vary with task context – in Experiment 2, one can assume that the control PE was only used to update predicted conflict because reactive control may not have been needed for task irrelevant distractors.) 

Fitting the FCM model to our data was important for several reasons. First, it would allow us to formally assess the possibility that the Gratton effect in WM-perception interaction was a result of the updating of the level of proactive control (predicted conflict) by observed congruity from the preceding trial. Second, it would allow us to determine whether the repulsive bias exerted by the incongruent discriminandum on trial n is related to the magnitude of the control PE that it generates. This would support the proposition that reactive suppression of an incongruent discriminandum accounts for this novel observation of a repulsive serial bias exerted after a lag of just one trial. 

Method

Detailed descriptions of the FCM can be found in Jiang et al. (2014; Jiang et al., 2015). The FCM takes the observed congruity (o) as input (discrete: 1 for incongruent and 0 for congruent) and outputs parameter estimates for trial-wise predicted conflict (f; proxy for the level of proactive control) and a flexible learning rate (α). f is the model's prediction for the probability of the forthcoming trial being incongruent and thus corresponds to the level of proactive control required (continuous, f ∈ [0,  1]). The flexible learning rate (α) represents the model's belief of the volatility of the environment. Because volatility was not experimentally manipulated in Experiment 1, α will not be reported in these analyses. The model tracks a joint probability distribution of the learning rate and the level of predicted conflict, which is updated on each trial based on a reinforcement learning rule: f_{n + 1} ← f_n + α × (o_n − f_n). Thus, f is estimated based on a weighted integration of short-term and long-term congruity distribution, with the weights determined by the flexible learning rate (α). For each upcoming trial, before observing the actual congruity, the model first predicts the level of conflict (likelihood of incongruity) based on the joint distribution. Upon observing the actual congruity of the current trial, the model uses the deviation between predicted conflict and observed congruity (o − f) to generate a “control prediction error” (control PE), which it uses to update the joint distribution with Bayes’ rule and for simulation of the next trial. The control PE is calculated as follows: if trial n is congruent, control PE_n = f_n; if trial n is incongruent, control PE_n = 1 − f_n. 

Figure 6 provides a visualization of the model estimates of f and PE, together with the observed congruity for 20 consecutive trials (trial 101 to 120) from one subject from Experiment 1. The model's behavior across trials 103 to 111 are instructive. Values from trials 103 to 109 illustrate that as the model experienced seven congruent trials in a row (observed incongruity values of 0), the values of predicted conflict and of the control PE declined steadily. On trial 110, the instance of incongruity created a large discrepancy between the relatively low value of predicted conflict and the value of observed congruity, which elicited a large control PE. This control PE from trial 110, in turn, had the effect of increasing the value of predicted conflict for trial 111. Note that although the FCM does not explicitly represent reactive control, the control PE is a “reaction” to the congruity of the discriminandum, and so in these analyses, we assume that the level of reactive control that is elicited by the discriminandum will be proportional to the value of the control PE. Thus, we assume that the control PE has two functions: it is “read into” one part of the system to generate the strength of reactive suppression, and it is “read into” a different part of the system to update the level of proactive control. 

Figure 6.

 

Flexible Control Model fits for 20 consecutive trials (trials 101-120) from a representative subject from Experiment 1. Purple: Actual observed incongruity, with 1 corresponding to incongruent and 0 to congruent. Green: Predicted conflict (scaled to range from 0 to 1, with larger values indicating higher predicted conflict, 0.5 equally likely to be congruent/incongruent). Yellow: Control prediction error (control PE), calculated as the difference between predicted conflict and observed congruity.

For RT analyses of the discrimination task, trials were median split by level of predicted conflict and then averaged by trial congruity (Figure 7). For serial dependence analysis of memory recall, we selected the one third of the trials with the lowest control PE from trial n and the one third of the trials with the highest control PE from trial n. For each group of trials, we then performed DoG fitting for WM recall on trial n+1 (Figure 8). 

Figure 7.

 

Effects of predicted conflict on discrimination RT. The congruity effect on trial n+1 was modulated by the level of predicted conflict: while the congruity effect was strong on trials that the FCM predicted to have low conflict, it was diminished on trials predicted to have high conflict. Error bars correspond to normalized ±1 SEM (Morey, 2008).

Figure 8.

 

The magnitude of repulsive bias was further modulated by control PE. (A) Serial bias exerted by discriminanda from high control PE trials. Memory recall on trial n+1 was significantly attracted toward trial n discriminandum when trial n was congruent and repulsed when trial n was incongruent. (B) Serial bias on the trial following a low PE trial. Neither the attractive bias nor the repulsive bias was significant. Shaded bands represent ±1 SEM; Solid curves correspond to DoG fits.

Results

Discrimination RT tracked by FCM

Results from the model-based analysis (see Figure 7A) also yielded evidence for a Gratton effect (c.f. see Figure 2A). A repeated-measures ANOVA with factors of predicted conflict (two levels: low and high) and current trial congruity yielded a main effect of congruity (F(1,45) = 7.79, p = 0.008, partial η² = 0.15), no main effect of predicted conflict (F(1,45) = 0.31, p = 0.58, partial η² = 0.007), and a significant interaction between the two (F(1,45) = 4.56, p = 0.038, partial η² = 0.092). Follow-up pairwise comparisons indicated that the congruity effect was only significant on trials with low predicted conflict (t(45) = 3.34, p = 0.001, Cohen's d = 0.51; for high predicted conflict t(45) = 0.67, p = 0.51, Cohen's d = 0.1). 

Control prediction error predicts repulsive bias

To assess our interpretation that the repulsive serial bias exerted by the incongruent discriminandum is a consequence of reactive suppression – itself a process recruited in proportion to the magnitude of the control PE – we selected the congruent trials with control PEs in the highest and lowest tertiles, and the incongruent trials with control PEs in the highest and lowest tertiles, and analyzed serial dependence on the next trial. For trials with high control PEs, the pattern resembled that resulting from simply sorting trials by congruity (see Figure 3): congruent discriminanda exerted an attractive bias on the next trial (a = 1.81 degrees, 95% CI = 1.20 degrees to 2.47 degrees, p < 0.001, bootstrapped; w = 0.025, 95% CI = 0.02 to 0.15) and incongruent discriminanda exerted a repulsive bias (a = −1.19 degrees, 95% CI = −1.96 degrees to −0.32 degrees, p = 0.005, bootstrapped; w = 0.023, 95% CI = 0.02 to 0.09). For trials with low control PEs, in contrast, the attractive bias exerted by congruent discriminanda no longer achieved significance (a = 1.16 degrees, 95% CI = −0.11 degrees to 1.88 degrees, p = 0.11, bootstrapped; w = 0.021, 95% CI = 0.02 to 0.05), and incongruent discriminanda exerted no discernable bias (a = −0.14 degrees, 95% CI = −0.91 degrees to 0.68 degrees, p = 0.63, bootstrapped; w = 0.02, 95% CI = 0.02 to 0.03). For congruent discriminanda, these effects for high and low control PEs did not differ (∆ = 0.65 degrees, 95% CI = −0.13 degrees to 1.21 degrees, p = 0.34, bootstrapped). For incongruent discriminanda, the differences between the amplitudes for high and low control PEs approached significance (∆ = 1.05 degrees, 95% CI = −0.02 degrees to 1.93 degrees, p = 0.054, bootstrapped). 

Discussion

We used the FCM to derive quantitative trial-by-trial estimates of the levels of proactive and reactive control and found that these estimates predicted the congruity effects on both discrimination RT and the serial dependence of WM recall. Specifically, the repulsive serial dependence reported in Experiment 1 was further modulated by the magnitude of control PE such that the repulsive effect only existed when the previous trial was incongruent and had a high control PE. These results allow us to explicitly interpret the empirical findings from Experiment 1 in relation to formally defined mechanisms of cognitive control that are specified in the FCM, as well as to articulate predictions about the neural bases of these effects. Consistent with the previous notion that proactive and reactive control are two distinct cognitive operations that may operate independently (e.g. Gonthier et al., 2016; Mäki-Marttunen et al., 2019), these results offered further evidence that the two mechanisms had dissociable effects and may be engaged simultaneously. 

General discussion

With a combination of experimental and modeling approaches, we demonstrated that the interaction between working memory and perception can be modulated by top-down control on a trial-to-trial basis. Specifically, incongruity in representations between the two processes evokes two distinct forms of control: proactive and reactive. First, we found the classic Gratton effect in the discrimination task, and this effect was explained by the parameter estimate of the FCM that is associated with proactive control (Jiang et al., 2014; Jiang et al., 2015). This indicates that subjects were updating the level of proactive control for each trial, based on the perceived conflict of the most recent trial. It is noteworthy that although the Gratton effect is typically observed when conflict is varied in the S-R mappings of a task, here the conflict was at the level of stimulus representation on two tasks that were independent and that each required a response with a different hand. Second, incongruity between the memorandum and the discriminandum also triggered reactive control, as evidenced by a repulsive serial bias of WM recall on the previous trial's discriminandum. Results from Experiment 1 indicated that this repulsive bias was only exerted by incongruent discriminanda, and the FCM fit to the data indicated that it was generated by trials with the largest values of control PE, the parameter estimate generated in response to an item's observed conflict. Together, these results demonstrate the flexible recruitment of proactive and reactive control in handling incongruity arising from working memory and perceptual representations. 

Previous studies have provided evidence that working memory and perceptual attention share resources and have overlapping representations (e.g. Gayet et al., 2013; Kiyonaga & Egner, 2016; Olivers et al., 2006; Teng & Kravitz, 2019; Teng & Postle, 2021) and are subject to cognitive control (Woodman & Luck, 2007; Carlisle & Woodman, 2011; Kiyonaga et al., 2012). For example, the proportion congruity effect could be explained by increased proactive control in high proportional congruity blocks (Kiyonaga & Egner, 2014). Here, we leveraged a dual task paradigm comprising two independent tasks with potentially interfering information to uncover how the interaction between the two processes is controlled. First, we show that despite the fact that the memoranda and discriminanda were relevant only for their respective task, incongruity between them led to an interaction that modulated performance on subsequent trials. Next, through fitting of the FCM to our data, we showed that the interaction is governed by trial-by-trial fluctuations in the strength of both proactive and reactive control. That the FCM generalized to the results of the current study supports the notion that WM is subject to domain-general cognitive control. 

The divergence in the results between Experiments 1 and 2 highlights the strategic nature of both proactive and reactive control and the conditions under which they are recruited. Evidence for the engagement of proactive control was observed in both experiments, including when the orientation of the discrimination stimulus was irrelevant for either the WM or the perceptual discrimination tasks in Experiment 2. This is consistent with findings using conflict adaptation paradigms (Stroop, 1935; Eriksen & Eriksen, 1974), in which incongruity typically occurs with task-irrelevant information (e.g. the colors of the Stroop stimuli, the flanker items in the flanker task). Because proactive control is an anticipatory, goal-driven process that serves to mitigate interference before it occurs, it appears to be engaged whenever there is the possibility of distraction from new sensory inputs, insensitive to the task relevance of the stimuli. In contrast, evidence for engagement of reactive control was markedly reduced in Experiment 2. This result suggests that merely perceiving the distractor was not sufficient to trigger the orientation-specific suppression, but rather active processing of the orientation of the discrimination stimulus was necessary for its engagement. Thus, suppression may be more context-dependent and modulated by the level of processing of the perceptual stimuli consistent with the “need-based” nature of reactive control being engaged only after conflict has been detected. 

As reviewed above, both attractive and repulsive serial biases have been observed in perceptual and working memory task behavior (e.g. Fischer & Whitney, 2014; Bae & Luck, 2017; Bliss et al., 2017; Fritsche et al., 2017; Bae & Luck, 2019; Czoschke et al., 2019; Samaha et al., 2019; Bae & Luck, 2020). To establish a unified account for the cooccurrence of those biases, Pascucci et al. (2019) proposed a hierarchical two-process model in which repulsion from previous stimuli arises from low-level sensory adaptation, whereas attraction arises from decisional inertia across trials. A separate but not mutually exclusive account put forth by Fritsche, Spaak, and De Lange (2020) describes an ideal observer that utilizes efficient encoding, which gives rise to repulsive biases, in conjunction with Bayes-optimal decoding of sensory information, which underlies attractive biases. Because the influence of recent history on decision-making processes is stronger, but also shorter-lived than are adaptation effects, the following pattern is typically observed: The serial dependence of trial n - 1 on trial n is attractive, and this attractive bias then declines across trials until the effect flips, such that the serial dependence of trial n - 5 on the same trial n will now be repulsive (Fritsch et al., 2020). Building on this, Shan & Postle (2022) proposed that repulsive bias observed in recall on trial n+1 arises from the active removal of an item from WM on trial n, which may be implemented by an appropriation of the mechanisms of perceptual adaptation by reactive control. Here, in Experiment 1, we also observed a repulsive serial bias with a lag of just one trial, raising the intriguing possibility that the reactive suppression observed here reflects the same mechanism as the “hijacked adaptation” that implements the active removal of information from WM (Shan & Postle, 2022), through down-modulating the gain of the recently engaged feature-specific circuits (Clifford, Wyatt, Arnold, Smith, & Wenderoth, 2001; Clifford, Wenderoth, & Spehar, 2000). In our dual task paradigm, it is similarly the case that it would be advantageous to “remove” any residual trace of the incongruent discriminandum to decrease the likelihood that it would interfere with the impending recall of that trial's memorandum. However, those control processes may not always be without consequence: on incongruent trials with a high control PE, reactive suppression of the discriminandum resulted in “collateral damage”: a strong repulsive serial bias and reduced precision on the subsequent trial. 

Studying cognitive control in the context of WM-perception interactions provides important insights for understanding the tight relationship among different cognitive functions that are usually studied in isolation. Using a behavioral procedure that is different from the standard conflict adaptation paradigms, and for which incongruity exists purely at the level of stimulus processing, we nonetheless observed cognitive control with many of the hallmarks first discovered with these paradigms. Furthermore, a reinforcement learning-based model originally developed to formalize how cognitive control is deployed in conflict adaptation tasks successfully captured the dynamics of control observed in our dual-task procedure. We conclude that common control processes may be recruited to support adaptive behavior for many tasks that can be summarized as visual cognition. 

Acknowledgments

Supported by National Institute of Health grants R01-MH064498 and R01-095984 to B.R.P. 

Open Science Statement: The data and material for the study are publicly accessible at Open Science Framework (https://osf.io/78aek/). 

Commercial relationships: none. 

Corresponding author: Chunyue Teng. 

Email: [e-mail]. 

Address: Department of Psychiatry, University of Wisconsin–Madison, 6001 Research Park Boulevard, Madison, WI 53719, USA.