Reward interacts with modality shift to reduce cross-modal conflict

Abstract

Previous studies have shown that reward can enhance cognitive control and reduce conflict in visual processing. Here we investigate (a) whether and how reward influences cross-modal conflict control and (b) how the shift of attention across modalities modulates the effect of reward on cross-modal conflict control. In four experiments, a cue indicating the reward availability of a given trial (reward vs. no reward) was presented prior to a target. The target was either a visual or an auditory letter, which was accompanied by a distracting letter from the other modality. The identity of the distracting letter was either the same as or different from the identity of the target letter (congruent vs. incongruent). When the cue modality was constant (Experiment 1) or changed across different experimental blocks (Experiment 3), the interference effect (i.e., the response time difference between incongruent and congruent trials) was smaller following a reward cue than a no-reward cue, suggesting that reward can reduce cross-modal conflict. In contrast, when the cue modality was changed trial-by-trial in an unpredictable way (Experiments 2 and 4), reward reduced cross-modal conflict only when the cue and the target were from different modalities and had a long stimulus onset asynchrony (SOA) between them but not when they shared the same modality or had a short SOA between them. These results suggest that reward can facilitate cross-modal conflict resolution, and this effect may critically depend on both the preparatory state between the cue and the target and timing to initiate cognitive control.

Introduction

In daily life, we often receive signals simultaneously from different sensory modalities, such as vision and audition. Cross-modal conflicts arise when information from different modalities are incompatible with each other. To resolve conflict, cognitive control is needed to enhance the processing of task-relevant information and suppress distraction from task-irrelevant information (van Veen & Carter, 2006). For example, when reading a textbook, we may need to recruit most of our cognitive resources to concentrate on the book and ignore any sound nearby. 

One of the important factors that modulates cognitive control is reward. Evidence from behavioral, event-related potentials and neuroimaging measures suggests that reward can enhance cognitive control (Botvinick & Braver, 2015; Padmala & Pessoa, 2011; Pessoa, 2015; Soutschek, Stelzel, Paschke, Walter, & Schubert, 2015; Vuillier, Whitebread, & Szucs, 2015; Wang, Yu, & Zhou, 2013). For example, Padmala and Pessoa (2011) presented a picture of a house or building together with a letter string on the picture and asked participants to indicate whether the picture was a house or a building. The identity of the letter string could be neutral (“XXXXX”), congruent (“HOUSE”), or incongruent (“BUILDING”) with the picture (e.g., a house picture). A cue was presented prior to the target, indicating whether participants could earn monetary reward after they made a fast and accurate response. They found that the interference effect (i.e., response times [RTs] in the incongruent condition minus RTs in the neutral condition) was reduced when the cue predicted monetary reward as compared with a no-reward cue. This reduced interference effect by reward was accompanied by decreased activity in the left fusiform gyrus, a region for representing words (i.e., distractor), and with decreased activity in the medial prefrontal cortex (MPFC), a region frequently observed in conflict control (Botvinick, Braver, Barch, Carter, & Cohen, 2001; Padrão, Rodriguez-Herreros, Zapata, & Rodriguez-Fornells, 2015; Ridderinkhof, Ullsperger, Crone, & Nieuwenhuis, 2004). The reduced activity in the MPFC during conflict resolution was predicted by the activity in the frontoparietal attentional regions that were activated by the reward-predictive cue. Based on these results, the authors argued that the onset of the reward-predictive cue enhanced top-down control, which increased the subsequent attentional filtering of the task-irrelevant word and reduced the response conflict (Padmala & Pessoa, 2011). With similar cue–target paradigms, previous studies also demonstrated that reward can enhance cognitive control by improving task-relevant processing (Etzel, Cole, Zacks, Kay, & Braver, 2015; Hughes, Mathan, & Yeung, 2013; van den Berg, Krebs, Lorist, & Woldorff, 2014; Vuillier et al., 2015). 

Research on the interaction between reward and cognitive control was almost exclusively conducted in the visual domain (Padmala & Pessoa, 2011; Soutschek et al., 2015; van den Berg et al., 2014; Veling & Aarts, 2010). In the current study, we sought to examine whether and how reward would modulate cognitive control in a cross-modal context. This is important because what we encounter most in everyday life are multisensory situations, and cross-modal distraction appears when information from different modalities is incompatible (Chen & Zhou, 2013; Donohue, Appelbaum, Park, Roberts, & Woldorff, 2013; Elliott, Cowan, & Valle-Inclan, 1998; Weissman, Warner, & Woldorff, 2004, 2009). We used a cue–target paradigm in which a cue was presented prior to the target to indicate whether a fast and correct response would result in a reward. The target was either a visual or an auditory letter, which was accompanied by an auditory or a visual distracting letter. The identities of these two items could be either congruent or incongruent (Chen & Zhou, 2013; Weissman et al., 2009). Participants were required to discriminate the identity of the letter in one modality (i.e., the target modality) and ignore the letter in the other modality (i.e., the distractor modality). We predicted that, if reward enhances cognitive control in the cross-modal context, the cross-modal conflict should be reduced after a cue that predicts reward as compared with a cue that predicts no reward. 

According to previous studies, the reward-predictive cue in the cue–target paradigm triggers a preparatory state, which is critical to reducing the subsequent conflict. In a multisensory context, one of the most important factors that influence the preparatory state between the cue and the following target is whether there is attentional shift between modalities. When hearing a tiger roaring in the wild, individuals may need to shift attention from audition to vision to check whether there are any signs of tigers nearby. To avoid a potential threat in this case, attention shift from the auditory modality to the visual modality is crucial. In laboratory settings, the processing of the target stimulus is influenced by attentional shift from the modality of the cue to the modality of the subsequent target (Turatto, Benso, Galfano, & Umiltà, 2002; Turatto, Galfano, Bridgeman, & Umiltà, 2004; Wang, Yue, & Chen, 2012). For example, Turatto et al. (2002) presented two stimuli (S1, S2) sequentially and asked participants to detect or discriminate S2. The two stimuli could be either from the same modality or from different modalities. Results showed that responses to S2 were delayed when S1 and S2 were from different modalities as compared with when they were from the same modality, suggesting that the modality shift from S1 to S2 elicited a switch cost and impaired the processing of S2. 

Although both reward and modality shift may play important roles in cross-modal processing, it is unclear whether reward interacts with modality shift in modulating the processing of the target. A recent study (Marien, Aarts, & Custers, 2014) using the abovementioned modality shift paradigm (Turatto et al., 2002) provided evidence for this interaction. In this study, fast and correct responses in half of the trials resulted in reward, and responses in the other half did not. The authors found that responses to the second stimulus were facilitated by reward only when there was a modality shift between the two stimuli but not when there was no modality shift. They argued that cognitive control was adaptively initiated to cope with a context in which the current control demand was unpredictable (i.e., unpredictable modality shift), improving task performance only when the control demand was high (e.g., when modality shift was required) but not when the control demand was low (Marien et al., 2014). Similarly, in a digit recall task, Bijleveld, Custers, and Aarts (2009) manipulated the control demand in a trial-by-trial manner. They found that the cognitive effort, as indexed by pupil dilation measured online, increased following a high-reward cue relative to a low-reward cue but only for the high control demand condition (i.e., when participants were required to recall five digits to gain the reward), not for the low control demand condition (i.e., when participants were required to recall three digits). Taken together, these results suggest that reward interacts with the adaptive control system to cope with an unpredictable environment, recruiting more cognitive resources only when effortful processing is required (Bijleveld et al., 2009; Marien et al., 2014). 

In the current study, we varied the modality congruency between the cue and the subsequent target and also the stimulus onset asynchrony (SOA) between the cue and the target. We predicted that reward would interact with modality shift during the preparatory phase to modulate the subsequent control of cross-modal conflict, and the reward-induced conflict reduction should be observed mostly when there is a modality shift between the cue and the target. Moreover, given that sufficient time is necessary for the preparatory state (Chiew & Braver, 2016), we predicted that the interaction between reward and modality shift would be influenced by the SOA between the cue and the target. 

Experiment 1 served as a baseline experiment, in which the target modality was blocked (visual target block and auditory target block), and the modality of the cue was always the same as the modality of the target. In Experiment 2, the modality of the cue changed unpredictably on a trial-by-trial basis, i.e., the cue could be presented in the same modality as the target (ipsimodal condition) or in a different modality (cross-modal condition) such that attention had to shift from the cue modality (audition/vision) to the target modality (vision/audition). Experiment 3 also included the ipisimodal and cross-modal conditions, but these conditions were separated in different sessions such that modality shift was predictable. Experiment 4 used the same design as Experiment 2 but included two cue–target SOAs: short (310–350 ms) vs. long (910–950 ms) in contrast to only a long SOA (1050–1450 ms) in Experiment 2. 

Experiment 1

Experiment 1 was conducted to examine whether reward could reduce cross-modal conflict. The modality (visual vs. auditory) of the target was blocked in two different sessions: a visual target session and an auditory target session. In both sessions, visual and auditory letters were presented simultaneously, and the identities of the two letters could be the same or different. In the visual target session, participants were asked to discriminate the identity of the visual letter and ignore the auditory letter; in the auditory target session, they were asked to discriminate the identity of the auditory letter and ignore the visual letter. Importantly, a cue indicating the availability of a performance-contingent reward was always presented in the target modality; that is, the cue predicted the modality of the upcoming target. 

Method

Participants

A group of 19 healthy participants (four males, age range 18–26 years) from colleges in Beijing took part in Experiment 1. All the participants were right-handed, had normal or corrected-to-normal vision, and self-reported normal hearing. This study was carried out in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of the School of Psychological and Cognitive Sciences, Peking University. 

Apparatus and materials

Four visual and four auditory stimuli were used in all the experiments. Visual stimuli included two symbols (&, #) as cue items and two letters (A, O) as target items, which were rendered in white color and presented at the center of a black background. The size of the cue and target items was 1.5° (horizontal) × 2° (vertical) in visual angle. Participants were seated 57 cm away from a CRT monitor. 

The auditory stimuli were two cue tones—sine wave tone of a frequency of 200 Hz (low tone) and 600 Hz (high tone)—and two letter sounds (A, O) in a male voice, recorded at a sampling rate of 44,100 Hz (16 bit, mono) as target sounds. All sounds were normalized and presented at approximately 55 dB, binaurally, through headphones. The duration of each sound was 450 ms. 

Design and procedure

The experiment consisted of two task sessions: a visual target session and an auditory target session (Figure 1). At the beginning of each trial, a white fixation cross (+), measured 0.2° × 0.2° in visual angle, was presented for 500 ms. A unimodal cue that shared the same modality as the target was presented for 450 ms. For the visual target session, one of the two visual cue items (e.g., &) indicated that a reward would be given after a response (i.e., reward condition), and the other visual cue (e.g., #) indicated that no reward would be given (i.e., no-reward condition). For the auditory target session, one of the two auditory cue items (e.g., high tone) indicated the reward condition, and the other auditory cue (e.g., low tone) indicated the no-reward condition. The association between reward availability and the identity of the cue items was counterbalanced across participants. After a variable interval of 600–1000 ms, a visual letter together with an auditory letter were presented for 450 ms. Participants were asked to discriminate the identity of the target letter (A vs. O), using the index finger of the left and right hands, respectively. The mapping between the two buttons on the keyboard and two target identities were counterbalanced across participants. The intertrial interval was 600–1000 ms. 

Figure 1

 

Trial structure in the visual target session (left) and the auditory target session (right) of Experiment 1. A unimodal cue with the same modality as the target indicated reward availability in the current trial. Participants were instructed to discriminate the target letter (A or O). Feedback indicating the total number of coins won until that moment was presented at the end of each block.

Thus, the experiment had a 2 (target modality: visual target vs. auditory target) × 2 (reward: reward vs. no reward) × 2 (target congruency: congruent vs. incongruent) within-participant factorial design. There were 48 trials in each of the eight experimental conditions. The 384 experimental trials were divided into eight blocks of equal length with four blocks for each task session. The order of the two sessions was counterbalanced across participants. Trials from the four experimental conditions (reward-congruent, reward-incongruent, no reward–congruent, no reward–incongruent) were equally distributed in each block and were presented in a pseudorandomized order. 

Prior to the formal experiment, participants received 24 practice trials for the visual target session and 24 practice trials for the auditory target session. The procedure for practice trials was the same as the main experiment except that participants were instructed that the stimuli (i.e., the cue) that appeared before the letters were task-irrelevant should be ignored. Response feedback (correct vs. incorrect) was presented after a button press. Participants were asked to respond as quickly and accurately as possible. For each participant, the mean RT in each of the two sessions (visual target session and auditory target session) during the practice was calculated and used as the baseline RT for the formal experiment. 

Participants were informed of the reward availability of the cue stimuli at the beginning of the formal experiment. For the reward condition, participants won one coin per trial if the response met the criterion (both accurate and faster than the baseline RT) and won zero coins if the response was incorrect or slower than the baseline RT. For the no-reward condition, participants won zero coins regardless of their performance. Feedback denoting the total number of coins earned during that block was presented at the end of each block. At the end of the experiment, the coins were exchanged for cash according to a proportion of 1:0.06, (for each one coin, they earned 0.06 yuan, 1 yuan ≈ $0.16). Participants could earn up to 12 yuan of reward based on their performance, which would be added to their basic payment (20 yuan) for taking part in the experiment. 

Data analysis

Incorrect trials were first excluded from the RT analysis. For each participant, trials with RTs more than three standard deviations above or below the mean RT in each experimental condition were discarded as outliers. In the current experiment, 1.0% of all data points were removed. A repeated-measures ANOVA was conducted on the mean RT of the remaining trials in each experimental condition with target modality (visual target vs. auditory target), reward (reward vs. no reward), and target congruency (congruent vs. incongruent) as within-subject variables. Similar analysis was conducted on the mean error rates, which were calculated as the proportions of incorrect and missing trials in each condition. 

Results

ANOVA on RTs (Figure 2) showed a main effect of target modality, F(1, 18) = 6.32, p = 0.022, η_p² = 0.26, with shorter RTs for the visual target session than for the auditory target session (400 vs. 424 ms). The main effect of reward was significant, F(1, 18) = 12.21, p = 0.003, η_p² = 0.40, as participants responded faster for reward trials than for no-reward trials (401 vs. 424 ms). The main effect of target congruency was also significant, F(1, 18) = 52.35, p < 0.001, η_p² = 0.74, with shorter RTs for the congruent conditions than for the incongruent conditions (402 vs. 423 ms). Moreover, the interaction between target modality and reward was significant, F(1, 18) = 5.80, p = 0.027, η_p² = 0.24, and so was the interaction between reward and congruency, F(1, 18) = 19.58, p < 0.001, η_p² = 0.52. The three-way interaction was not significant, F(1, 18) = 0.45, p = 0.511, η_p² = 0.02. 

Figure 2

 

Experiment 1. Left: Mean RTs with standard errors as a function of the experimental condition. Right: the interference effects (i.e., RTs in the incongruent condition minus RTs in the congruent condition) with standard errors as a function of the experimental condition.

A separate 2 (reward vs. no reward) × 2 (congruent vs. incongruent) ANOVA was then conducted for the visual target session and the auditory target session, respectively. For the visual target session, the main effect of congruency was significant, F(1, 18) = 37.84, p < 0.001, η_p² = 0.68, with longer RTs for the incongruent condition than for the congruent condition (410 vs. 390 ms). However, the main effect of reward did not reach significance, F(1, 18) = 2.83, p = 0.110, η_p² = 0.14. The interaction between reward and congruency was significant, F(1, 18) = 7.21, p = 0.015, η_p² = 0.29. A planned t test on simple effects showed that the interference effect (i.e., RTs in the incongruent condition minus RTs in the congruent condition) was smaller in the reward condition than in the no-reward condition (14 vs. 26 ms), t(18) = 2.67, p = 0.015. For the other direction of the interaction, a planned t test showed faster RTs for the reward trials than for the no-reward trials only in the incongruent condition, t(18) = 2.17, p = 0.043, not in the congruent condition, t(18) = 0.93, p = 0.363. 

For the auditory target session, the main effect of reward was significant, F(1, 18) = 15.90, p = 0.001, η_p² = 0.47, and so was the main effect of target congruency, F(1, 18) = 17.75, p = 0.001, η_p² = 0.50. Participants responded faster to reward trials than to no-reward trials (407 vs. 441 ms) and faster to congruent trials than incongruent trials (413 vs. 435 ms). The interaction between reward and congruency was significant, F(1, 18) = 8.51, p = 0.009, η_p² = 0.32. A planned t test on simple effects showed that the interference effect was smaller in the reward condition than in the no-reward condition (13 vs. 31 ms), t(18) = 2.91, p = 0.009. For the other direction of the interaction, a planned t test showed that the reward effect (i.e., RTs in the no-reward condition minus RTs in the reward condition) was smaller in the congruent condition than in the incongruent condition (25 vs. 43 ms), t(18) = 2.93, p = 0.009. 

A 2 × 2 × 2 ANOVA on error rates showed only a main effect of target congruency, F(1, 18) = 16.11, p = 0.001, η_p² = 0.47, with more errors for the incongruent trials than for the congruent trials (4.7% vs. 2.2%). 

Discussion

In Experiment 1, we found faster responses when reward was expected, replicating previous findings that reward facilitates behavioral performance (Kiss, Driver, & Eimer, 2009; Padmala & Pessoa, 2011, 2014; Wei & Kang, 2014). Moreover, we found that reward reduced the cross-modal interference effect in both the visual target session and the auditory target session, suggesting that reward enhances cognitive control irrespective of target modality (Botvinick & Braver, 2015; Padmala & Pessoa, 2011; Pessoa, 2009) when the target modality is predictable. 

Experiment 2

In Experiment 2, we aimed to investigate whether and how attentional modality shift influences the effect of reward on cross-modal conflict processing. We manipulated the modality of the cue such that the cue modality could be either the same as or different from the modality of the target, leading to a modality shift in the latter case (cross-modal conditions) as compared with the former case (ipsimodal conditions; Turatto et al., 2002; Wang et al., 2012). We expected that the shift from the cue modality to the target modality in the cross-modal conditions would result in a switch cost as compared with the ipsimodal conditions. To cope with the switch cost, an adaptive control system might be recruited, leading to a potential interaction between reward and modality shift in modulating the control of the cross-modal conflict (Marien et al., 2014). 

Method

Participants

A new group of 20 college students (11 males, 18∼25 years old) took part in Experiment 2. All the participants were right-handed, had normal or corrected-to-normal vision, and self-reported normal hearing. 

Apparatus and materials

The apparatus and materials were the same as in the previous experiment. 

Design and procedure

The procedure was the same as in Experiment 1 with the following exceptions: The cue modality was manipulated on a trial-by-trial basis: Either an auditory cue (high tone or low tone) or a visual cue (# or &) was presented in each trial. Specifically, for half of the trials in each block, a visual cue was presented to indicate reward availability of the current trial, and for the other half of trials, an auditory cue was presented. The visual cue and the auditory cue trials were randomly mixed and equally distributed in each block. 

The experiment had a 2 (cue–target modality congruency: ipsimodal vs. cross-modal) × 2 (target modality: visual target vs. auditory target) × 2 (reward: reward vs. no reward) × 2 (target congruency: congruent vs. incongruent) within-subject factorial design. There were 48 trials for each of the 16 experimental conditions. The 768 trials were divided into 16 blocks of equal length with eight blocks in each target session. The order of the visual and auditory sessions was counterbalanced across participants. Each participant received 32 practice trials for the visual target session and 32 practice trials for the auditory target session to become familiarized with the task. The baseline RTs were calculated based on the participant's responses in these practice trials. 

Results

Omissions and incorrect responses were excluded from analysis. For each participant, trials with RTs more than three standard deviations above or below the mean RT in each experimental condition were discarded as outliers (1.3%). A 2 × 2 × 2 × 2 ANOVA on RTs (Figure 3) showed a significant main effect of cue–target modality congruency, F(1, 19) = 8.22, p = 0.010, η_p² = 0.30, with shorter RTs for ipsimodal than for cross-modal trials (437 vs. 447 ms). The main effect of reward was significant, F(1, 19) = 15.73, p = 0.001, η_p² = 0.45, with shorter RTs for reward than for no-reward trials (427 vs. 457 ms). The main effect of target congruency was also significant, F(1, 19) = 143.89, p < 0.001, η_p² = 0.88, with shorter RTs for congruent than for incongruent trials (427 vs. 457 ms). The interaction between reward and target congruency was significant, F(1, 19) = 7.59, p = 0.013, η_p² = 0.29. There was also a significant three-way interaction between cue–target modality congruency, reward, and target congruency, F(1, 19) = 5.19, p = 0.034, η_p² = 0.22. No other effects reached significance. 

Figure 3

 

Experiment 2. Top: Mean RTs with standard errors as a function of the experimental condition. Bottom: the interference effects with standard errors as a function of the experimental condition.

RTs were collapsed across the visual target session and the auditory target session, and a separate 2 (reward vs. no reward) × 2 (congruent vs. incongruent) ANOVA was carried out for the ipsimodal and cross-modal conditions, respectively. For the ipsimodal conditions, ANOVA showed a main effect of reward, F(1, 19) = 19.69, p < 0.001, η_p² = 0.51, and a main effect of target congruency, F(1, 19) = 121.09, p < 0.001, η_p² = 0.86. Participants responded faster to reward trials than to no-reward trials (422 vs. 453 ms) and faster in the congruent condition than in the incongruent condition (422 vs. 453 ms). However, the interaction between reward and target congruency did not reach significance, F(1, 19) = 0.29, p = 0.595, η_p² = 0.02. 

For the cross-modal conditions, ANOVA showed a main effect of reward, F(1, 19) = 11.14, p = 0.003, η_p² = 0.37, as well as a main effect of target congruency, F(1, 19) = 83.94, p < 0.001, η_p² = 0.82. Participants responded faster in the reward condition than in the no-reward condition (433 vs. 461 ms) and faster in the congruent condition than in the incongruent condition (432 vs. 462 ms). Importantly, the interaction between reward and target congruency was significant, F(1, 19) = 10.84, p = 0.004, η_p² = 0.36. A planned t test showed that the interference effect was smaller in the reward condition than in the no-reward condition (21 vs. 40 ms), t(19) = 3.29, p = 0.004. For the other direction of the interaction, a planned t test indicated that the reward effect was smaller in the congruent condition than in the incongruent condition (19 vs. 38 ms), t(19) = 3.31, p = 0.004. 

The analysis on error rates showed a main effect of target congruency, F(1, 19) = 10.42, p = 0.004, η_p² = 0.35, with more errors on incongruent trials than on congruent trials (4.6% vs. 2.4%). No other effects reached significance. 

Discussion

In Experiment 2, we observed longer RTs in the cross-modal conditions than in the ipsimodal conditions, suggesting an impaired target processing when there was a shift from the cue modality (audition/vision) to the target modality (vision/audition). This RT cost was consistent with other studies (Turatto et al., 2002; Turatto et al., 2004), indicating an attentional switch cost when modality shift is required even though the target modality was kept constant throughout the visual or auditory session. Importantly, our results showed that reward reduced conflict only in the cross-modal conditions, not in the ipsimodal conditions. This finding was consistent with a recent study showing that reward facilitates target detection only when the modality of the target is different from the modality of the preceding cue (Marien et al., 2014). A possible explanation for this interaction between reward and modality shift is that processing resources are consumed and control demand is high when attention has to be shifted from the modality of the cue to the modality of the target; the reward system in the brain activated by the reward cue interacts with the control system to provide additional effort to resolve the conflict (Marien et al., 2014; Pessoa, 2009). This happens when the modality of the cue is randomly selected for a particular trial (as in this experiment) and when the modality shift for a given trial is unpredictable and the control demand for this trial is unknown beforehand. If modality shift is predictable such that the control demand is known in advance, cognitive control could be well prepared during the preparatory phase between the cue and the target, and reward could then reduce the following cross-modal conflict irrespective of whether modality shift was involved or not. This possibility was tested in the next experiment. 

Experiment 3

The aim of Experiment 3 was to test whether and how reward would reduce cross-modal conflict when modality shift from the cue to the target was predictable. The cross-modal and ipsimodal conditions were separated in different sessions such that modality shift was required in the cross-modal session but not required in the ipsimodal session. 

Method

Participants

A group of 16 healthy participants (11 males, age range 19–26 years old) took part in Experiment 3. All the participants were right-handed, had normal or corrected-to-normal vision, and self-reported normal hearing. They did not participate in the previous experiments. 

Apparatus and materials

The apparatus and materials were the same as in the previous experiments. 

Design and procedure

Given that the pattern of effects observed in Experiments 1 and 2 was independent of the target modality, only visual targets were used in Experiment 3. The experimental procedure was essentially the same as the procedure in Experiment 2 except that the modality of the cue was kept constant in a specific session in Experiment 3. There was a visual cue session (the ipsimodal conditions) and an auditory cue session (the cross-modal conditions). Specifically, for the auditory cue session, the reward availability was indicated by an auditory cue (high or low tone), and for the visual cue session, the reward availability was indicated by a visual cue (# or &). 

Results

Omissions and incorrect responses were excluded from analysis. For each participant, trials with RTs more than three standard deviations above or below the mean RT in each experimental condition for each participant were discarded as outliers (0.8%). A 2 (ipsimodal vs. cross-modal) × 2 (reward vs. no reward) × 2 (congruent vs. incongruent) repeated-measures ANOVA on RTs (Figure 4) showed a main effect of reward, F(1, 15) = 4.79, p = 0.045, η_p² = 0.24, with shorter RTs in the reward conditions than in the no-reward conditions (396 vs. 427 ms), as well as a main effect of target congruency, F(1, 15) = 16.26, p = 0.001, η_p² = 0.52, with shorter RTs for congruent trials than for incongruent trials (406 vs. 416 ms). The main effect of cue–target modality congruency was not significant (p > 0.1), indicating the absence of modality switch costs. The interaction between reward and target congruency was significant, F(1, 15) = 4.92, p = 0.042, η_p² = 0.25, but the three-way interaction was not, F(1, 15) = 0.78, p = 0.391, η_p² = 0.05. Collapsing over the ipsimodal (visual cue) and cross-modal (auditory cue) conditions, the interference effect was significantly smaller in the reward conditions than in the no-reward conditions (7 vs. 13 ms), t(15) = 2.22, p = 0.042; the reward effect was significantly smaller in the congruent conditions than in the incongruent conditions (28 vs. 34 ms), t(15) = 2.34, p = 0.034. The analysis of error rates showed no effects at all. 

Figure 4

 

Experiment 3. Left: Mean RTs with standard errors as a function of the experimental condition. Right: the interference effects with standard errors as a function of the experimental condition.

Discussion

In contrast to Experiment 2, the attentional shift from the cue modality to the target modality was made predictable in Experiment 3 such that cognitive control could be well prepared before the target was encountered. As a result, neither the RT nor the error rate in the cross-modal conditions differed from those in the ipsimodal conditions. This result, in contrast to the delayed responses caused by the unpredictable modality shift in Experiment 2 (see also Turatto et al., 2002; Turatto et al., 2004), indicated that cognitive control was well prepared in advance for the modality shift in Experiment 3. Reward expectation influences cross-modal conflict control in a flexible way, which depends to a certain extent on the preparatory state for modality shift in the current trial. 

Experiment 4

In Experiment 4, we sought to further clarify whether the interaction between reward and modality shift in Experiment 2 was influenced by the preparatory state for the subsequent target. A recent study suggested that the reward modulation on cognitive control depends on whether there is sufficient time between the reward cue and the following target (Chiew & Braver, 2016). Here we reasoned that, if the preparatory state is critical for the interaction between reward and modality shift, the occurrence of this interaction should be modulated by the time available for the preparation of cognitive control. To test this hypothesis, we manipulated the cue–target SOA in Experiment 4. 

Method

Participants

Twenty healthy participants (five males, 18∼26 years old) took part in Experiment 4. All the participants had normal or corrected-to-normal vision, self-reported normal hearing, and were right-handed. They did not participate in the previous experiments. 

Apparatus and materials

The apparatus and materials were the same as those in the previous experiments. 

Design and procedure

The procedure was the same as in Experiment 3 with the following exceptions: The cue–target modality congruency was manipulated on a trial-by-trial basis as in Experiment 2. Moreover, the cue stimuli were presented for 300 ms, and the cue–target SOAs were manipulated in two separate sessions. For the short SOA session, the cue–target SOA was 310–350 ms; for the long SOA session, the SOA was 910–950 ms. The order of the long and short SOA sessions was counterbalanced across participants. 

Thus the experiment had a 2 (SOA: short vs. long) × 2 (cue–target modality congruency: ipsimodal vs. cross-modal) × 2 (reward: reward vs. no reward) × 2 (target congruency: congruent vs. incongruent) within-participant design. There were 768 trials in total, which were divided into 12 blocks with 64 trials in each block. Participants first completed 32 trials of practice, which provided a measurement of the baseline performance. 

Results

Omissions and incorrect responses were removed from analysis. For each participant, trials with RTs more than three standard deviations above or below the mean RT in each experimental condition were discarded as outliers (1.2%). Repeated-measures ANOVA on RTs (Figure 5) with the four within-subject factors showed a main effect of cue–target modality congruency, F(1, 19) = 9.82, p = 0.005, η_p² = 0.34, with shorter RTs for the ipsimodal than for the cross-modal conditions (416 vs. 431 ms); a main effect of reward, F(1, 19) = 8.36, p = 0.009, η_p² = 0.31, with shorter RTs for the reward condition than for the no-reward condition (408 vs. 439 ms); and a main effect of target congruency, F(1, 19) = 45.78, p < 0.001, η_p² = 0.71, with shorter RTs for the congruent than for the incongruent conditions (417 vs. 430 ms). There was no main effect of SOA, F(1, 19) = 2.83, p = 0.109, η_p² = 0.13. However, SOA interacted marginally with target congruency, F(1, 19) = 4.30, p = 0.052, η_p² = 0.19, and significantly with reward, F(1, 19) = 4.65, p = 0.044, η_p² = 0.20. Importantly, there was a four-way interaction between SOA, cue–target modality congruency, reward, and target congruency, F(1, 19) = 4.60, p = 0.045, η_p² = 0.20. 

Figure 5

 

Experiment 4. Top: Mean RTs with standard errors as a function of the experimental condition. Bottom: the interference effects with standard errors as a function of the experimental condition.

To investigate the interactions, we calculated and analyzed the target congruency (interference) effect for the ipsimodal and cross-modal conditions, respectively. For the ipsimodal conditions, the 2 (SOA: long vs. short) × 2 (reward: reward vs. no reward) ANOVA showed no significant main effect of SOA, F(1, 19) = 1.41, p = 0.250, η_p² = 0.07, nor a main effect reward, F(1, 19) = 0.01, p = 0.939, η_p² < 0.001. The interaction between SOA and reward was not significant, F(1, 19) = 0.13, p = 0.727, η_p² = 0.01. For the cross-modal conditions, although there was no significant main effect of either SOA, F(1, 19) = 0.89, p = 0.358, η_p² = 0.05, or reward, F(1, 19) = 1.36, p = 0.259, η_p² = 0.07, there was a significant interaction between SOA and reward, F(1, 19) = 8.07, p = 0.010, η_p² = 0.30. Further paired t tests showed that, at the long SOA, the interference effect was smaller for the reward conditions than for the no-reward conditions (6 vs. 20 ms), t(19) = 2.44, p = 0.025, and at the short SOA, the effect did not differ between the two conditions (12 vs. 7 ms), t(19) = 1.12, p = 0.276. 

ANOVA on the error rates showed an interaction between reward and target congruency, F(1, 19) = 4.42, p = 0.049, η_p² = 0.19, and a marginally significant interaction between SOA and cue–target modality congruency, F(1, 19) = 4.24, p = 0.053, η_p² = 0.18. Collapsing the data over SOA and cue–target modality congruency, we found a higher error rate for the incongruent than for the congruent conditions, but only for the no-reward trials (2.8% vs. 2.2%), t(1, 19) = 2.45, p = 0.024, and not for the reward trials (2.6% vs. 2.7%), t(1, 19) = 0.36, p = 0.724. Similarly, collapsing the data over reward and target congruency, we observed a higher error rate for the ipsimodal than for the cross-modal conditions, but only for short SOA trials (3.0% vs. 2.1%), t(19) = 3.01, p = 0.007, and not for long SOA trials (2.4% vs. 2.7%), t(19) = 0.52, p = 0.612. 

Discussion

In Experiment 4, we observed more delayed RTs for the cross-modal conditions than for the ipsimodal conditions, replicating the finding of the modality shift cost when this shift was unpredictable (Experiment 2). Moreover, we found that the cross-modal interference effect was smaller for the reward than for the no-reward conditions at the long SOA but not at the short SOA, indicating that the modulatory effect of the interaction between reward and modality shift on cognitive control occurs only when there is enough time for modality shift. The absences of the modulatory effect for the cross-modal conditions at the short SOA may simply be because there was not enough time for the system to initiate proactive control (Chiew & Braver, 2016). 

General discussion

In four experiments, we investigated (a) whether reward could reduce cross-modal conflict and (b) whether reward interacts with modality shift to modulate cross-modal conflict. We found that reward can enhance cognitive control and reduce cross-modal conflict irrespective of the target modality (Experiments 1 and 3). However, this reward-driven conflict resolution depended crucially on the preparation of modality shift between the reward-predictive cue and the subsequent target. Specifically, when information concerning modality shift or no shift could be obtained before the cue and the target were presented and the system could be prepared in advance, reward reduced cross-modal conflict irrespective of whether there was a modality shift for the current trial (Experiment 3). In contrast, when information concerning modality shift or no shift could not be known before the presentation of the cue and the target, reward reduced cross-modal conflict only when modality shift was required. Furthermore, this conditional reward-driven conflict resolution occurred only when there was enough time for the system to initiate proactive control (Experiments 2 and 4). 

One of the important components of reward is motivation (“wanting”; Berridge & Robinson, 2003). A cue indicating reward delivery for successful performance would activate the motivational component of reward, which could elicit behavior changes (Notebaert & Braem, 2015). A number of studies suggest that reward-induced motivation promotes behavior performance and enhances cognitive control (Botvinick & Braver, 2015; Chiew & Braver, 2016; Kang, Zhou, & Wei, 2015; Padmala & Pessoa, 2011; Pessoa, 2009; Soutschek et al., 2015; Wei & Kang, 2014). Extending the evidence in the visual domain, our findings demonstrate that reward can enhance conflict resolution in the cross-modal context, reducing cross-modal conflict regardless of the target modality. Taken together, these results suggest a general role of reward in enhancing cognitive control. 

Previous studies have shown that the onset of the cue triggers the preparation for the following cognitive control, and this preparatory state can be modulated by reward (Etzel et al., 2015; Padmala & Pessoa, 2011). For example, Etzel et al. (2015) found that the frontoparietal network, which is responsible for attentional control, was activated during the cue phase. Using multivariate pattern analysis, they showed that this network was involved in decoding the task information, and the decoding accuracy was higher when the cue predicted reward than when the cue did not (Etzel et al., 2015). By recording EEG, previous studies also showed that the frontocentrally distributed contingent negative variation, a component reflecting task preparation, was more negative after a reward cue than after a no-reward cue (Hughes et al., 2013; van den Berg et al., 2014; Vuillier et al., 2015). These findings demonstrate that reward improves the preparatory state triggered by the cue, which in turn enhances cognitive control. 

Extending the abovementioned studies, we further demonstrate that reward does not simply enhance cognitive control but rather interacts with the preparatory state in modulating cognitive control. Specifically, whether reward enhances cognitive control and reduces cross-modal conflict is critically dependent on the preparation for the modality shift between the cue and the target. When the modality shift or no shift can be prepared in advance, i.e., when both the cue and target modalities are manipulated block-wise (Experiment 3), the well-prepared attentional shift does not impair the preparatory state during the cue–target interval in a specific trial. In this situation, reward reduces cross-modal conflict irrespective of whether modality shift is required or not. In contrast, when the modality shift or no shift cannot be prepared in advance (Experiments 2 and 4), the unpredictable attentional shift hinders the preparatory states during the cue–target interval, which leads to increased control demand for a specific trial. To deal with the unpredictable control demand, the reward boosts mental effort to resolve the conflict when facing the high demand (Bijleveld et al., 2009; Marien et al., 2014). 

Note that, the current study differed from Marien et al. (2014), which also showed an interaction between reward and modality shift, in two important aspects. First, a block manipulation of reward, which induced a sustained reward effect, was used in Marien et al. In the current study, however, we manipulated the reward expectation by recruiting a trial-by-trial reward-predictive cue and further revealed the interaction between transient reward and modality shift. Second, Marien et al. used a simple detection task to measure the cognitive control, which limited their ability to broaden their conclusion. In contrast, the current study adopted an interference paradigm that provides more direct evidence concerning the modulation of cognitive control. 

The contingent reward modulation on cognitive control is in agreement with the motivation intensity theory (Brehm & Self, 1989), which claims that the effort invested in a specific task is determined by the magnitude of potential motivation (e.g., monetary reward) and the difficulty of the task. From this perspective, to determine how much of one's cognitive resources are to be invested in the current task, one should estimate the resources that are required by the cognitive control system and evaluate whether the resource allocation is worthwhile for the potential reward. In a situation in which the control demand is kept constant (Experiments 1 and 3 in the present study), more resources are invested to gain a potential reward as long as the required resources are not beyond an individual's certain limit. However, when an unpredictable environment is encountered and the required control demand is not known in advance (Experiments 2 and 4 in the present study), it is crucial to conserve the limited resources optimally. To this end, the adaptive control system is initiated such that extra resources induced by reward are more likely recruited when there is a high control demand. Future studies are needed to investigate the neural mechanism of how reward interacts with the adaptive control system during cross-modal conflict resolution. Indeed, there are ongoing debates on similar issues in the visual domain. For example, Padmala and Pessoa (2011) proposed that reward enhances top-down control and selectively attenuates the processing of distractor stimuli, which in turn facilitates conflict resolution. In contrast, Soutschek et al. (2015) argued that reward reduces conflict by improving target processing rather than distractor inhibition (Soutschek et al., 2015). In the same vein, reward could reduce cross-modal conflict either by enhancing the identity representation from the target modality or by inhibiting the identity representation from the distracting modality or both. Further studies are needed to reveal the specific role of reward in modulating the perceptual/semantic representations of the target and the distractor. 

Another finding in the present study was that the reward effect on reducing cross-modal conflict was modulated by the available time between the cue and the target. Specifically, reward reduced cross-modal conflict only when there was sufficient time available for the preparation (i.e., with a long SOA between the cue and the target) and not when the time was insufficient for the preparation (i.e., in the short SOA conditions). This finding is consistent with a recent study that showed the reward effect on cognitive control was influenced by both the task-related expectation and time. In this study, Chiew and Braver (2016) used a cued flanker task to investigate the potential interaction between reward and task-informative cues (indicating the congruency of the target stimuli) in cognitive control. In their experiment 1, the reward and task information were presented simultaneously; they found that reward reduced the flanker interference when the cue indicated the task information. In their experiment 2, the reward cue and task-informative cue were presented sequentially, leading to an early reward condition (i.e., the reward cue presented before the task-informative cue) and a late reward condition. Results in the early reward condition, but not the results in the late reward condition, replicated findings in their experiment 1. The authors suggested that reward promoted strategic use of the informative cues to influence selective attention, and sufficient time was needed for initial proactive control. In line with this suggestion, in the present study, sufficient time during the preparation phase was also needed for the adaptive coping with the unpredictable modality shift and for the strategic distribution of cognitive resources. 

A third observation in the current study is shorter RTs for the ipsimodal conditions than for the cross-modal conditions (Experiments 2 and 4). The onset of the cue is likely to induce an automatic allocation of attention to the cue modality, facilitating the subsequent processing of the target if the target is from the same modality (Parmentier, Elford, Escera, Andrés, & San Miguel, 2008; Turatto et al., 2002). Similar results have been reported by Weissman et al. (2004), who showed faster RTs for visual targets when the preceded cue was presented in the visual modality as compared to the auditory modality. Nevertheless, although the modality of the target could be primed by the modality of the cue, this overall modality priming had no impact upon the interference effect, suggesting that the ipsimodal cue may simply increase attentional preparation to the target modality but cannot enhance cognitive control. 

To conclude, by adopting a cue–target paradigm, we demonstrate that reward can enhance cognitive control and reduce cross-modal conflict, but the reward-driven conflict resolution depends crucially on the preparatory state between the cue and the target. When the modality shift from the cue to the target can be prepared in advance, reward reduces cross-modal conflict regardless of whether there is modality shift or not. However, when the modality shift cannot be prepared in advance, reward reduces cross-modal conflict only when the modality shift is required for the current trial and when there is sufficient time for task preparation. These results suggest that whether and how reward modulates cognitive control critically depends on the preparatory state for cognitive control. 

Acknowledgments

This study was supported by the National Basic Research Program of China (973 Program: 2015CB856400). We thank Mr. Philip R. Blue for help in the preparation of the manuscript and two anonymous reviewers for constructive comments on an earlier version of the manuscript. 

Commercial relationships: none. 

Corresponding author: Xiaolin Zhou. 

Email: [e-mail]. 

Address: School of Psychological and Cognitive Sciences, Peking University, Beijing, China.