Separate and combined effects of action relevance and motivational value on visual working memory

Abstract

Visual working memory contents can be selectively weighted according to differences in their task-relevance. In the present study, we examined the influence of two more indirect selection biases established by a concurrent task or learned reward associations: action relevance and motivational value. In three experiments, memory performance was assessed with the same color change detection task. Potential action relevance and motivational value were each determined by a specific feature of the memory items (location or shape, respectively) and manipulated orthogonally. Investigated separately (Experiments 1A and 1B), both selection biases modulated visual working memory. In combination (Experiment 2), action relevance and motivational value still each exerted an influence, but not in a fully independent and additive manner. While action relevance impacted performance irrespective of the reward associated with the items, an effect of motivational value was only observed for action-relevant items. These results support the notion that visual working memory is automatically biased as an inherent part of action planning. More generally, these findings highlight the versatile nature of visual working memory: Contents can be flexibly weighted to reflect differences in their importance, taking into account several sources of information.

Introduction

When interacting with our visual environment, we extract information about the importance of certain objects or parts of our surroundings from various sources. For instance, an apple might draw our attention, because we need it for the apple pie we want to make, because it is rotten and sticks out from the other fresh green ones in the bowl, or because looking at apples always brings back fond childhood memories of picking them in our grandparents' garden. Such different sources bias the limited resources of the visual system in favor of the most relevant visual information, ensuring that it is selected for further processing from the large amount of information available. 

Which and how different sources of selection bias control the allocation of visual attention to perceptual input has been investigated for decades and is still an active topic of research (e.g., Awh, Belopolsky, & Theeuwes, 2012; Failing & Theeuwes, 2017). Somewhat more recently, research has also begun to examine how attention is biased towards relevant information during later stages of visual processing, selectively modulating the encoding and storage of visual information in working memory. The ability to maintain information over short periods of time is highly limited in capacity (e.g., Luck & Vogel, 2013; Ma, Husain, & Bays, 2014), and filtering the relevant from the irrelevant is thus no less important for visual working memory than it is for visual perception. 

Research on such selective processing in visual working memory has largely focused on differences in the direct task-relevance of upcoming or already maintained items: Presenting cues that provide information about which items will most likely be tested either prior to the memory items (precues) or during the retention interval (retrocues), has shown that attention within visual working memory (internal attention) operates in a similar manner as attention to perceptual input (external attention), biasing storage towards relevant visual locations (Astle, Summerfield, Griffin, & Nobre, 2012; Griffin & Nobre, 2003), features and feature dimensions (Heuer & Schubö, 2016a; Heuer, Schubö, & Crawford, 2016; Kalogeropoulou, Jagadeesh, Ohl, & Rolfs, 2017; Niklaus, Nobre, & van Ede, 2017), or object categories (Lepsien & Nobre, 2007; Lepsien, Thornton, & Nobre, 2011). Information can thus not only be filtered from gaining access to visual working memory (e.g., Jost & Mayr, 2016; Vogel, McCollough, & Machizawa, 2005), but even already maintained information can, depending on the validity of the cues, be removed (Williams, Hong, Kang, Carlisle, & Woodman, 2013; Williams & Woodman, 2012) or weighted to reflect more graded differences in task-relevance (Gunseli, van Moorselaar, Meeter, & Olivers, 2015; Heuer & Schubö, 2016b; Zokaei, Ning, Manohar, Feredoes, & Husain, 2014). 

While the direct manipulation of task-relevance with the retrocue paradigm has attracted considerable interest in recent years (for an overview see Souza & Oberauer, 2016), there are also other sources of a selection bias that influence visual working memory. In the present study, we examine two somewhat more indirect types of selection bias that are established by concurrent tasks or learned reward associations: action relevance and motivational value. 

Action relevance

In everyday life, we are almost continuously engaged in some sort of action. While you are reading this, your eyes are systematically moving across the page and your hand might be holding a pen (or controlling the cursor on the screen), ready for use. Thus, actions and action intentions profoundly affect which aspects of our visual environment are most important to us and consequently how we perceive the world around us (e.g., Schütz-Bosbach & Prinz, 2007). A few recent studies have shown that actions continue to modulate visual processing beyond the perceptual stage: Action relevance also imposes priorities on the information that is encoded and maintained in working memory, in that spatial locations (Heuer, Crawford, & Schubö, 2017; Ohl & Rolfs, 2017) as well as feature dimensions (Heuer & Schubö, 2017) are selectively weighted to reflect differences in their importance for an action. 

Of particular interest for the present study is the spatial bias that has been shown to be established in visual working memory when an action is planned and executed: Memory representations spatially corresponding to an action goal location can be recalled more accurately and faster than representations corresponding to action-irrelevant locations. Heuer et al. (2017) had participants perform a manual pointing movement during the retention interval of a visual working memory task. The pointing movement was performed towards the locations on the display that also served as memory item locations, with the movement goal for a given trial being indicated by a movement cue presented at fixation. Except for the temporal and spatial overlap, memory and movement task were unrelated: The memory item that had been presented at the movement goal location was tested just as often as any other item in the memory task. Thus, unlike a retrocue, the movement cue had no predictive value for the upcoming test item. Nevertheless, performance for memory items that had been presented at the movement goal was better than for items presented at other, action-irrelevant locations. Converging results have been obtained for saccadic eye movements. Using a similar design, Ohl and Rolfs (2017) found that memory for items presented at the goal of a saccade, planned and executed during the retention interval following a movement cue, was better than for items presented at other locations. Both studies included control conditions, in which no movement or a movement to a location that never corresponded to a memory item location (i.e., the fixation dot) had to be executed following the movement cue. These confirmed that the observed differences in performance for potentially action-relevant and action-irrelevant memory items were not the result of an automatic shift of attention induced by the nonpredictive movement cue (Heuer et al., 2017; Ohl & Rolfs, 2017). 

As already pointed out above, there was no strategic advantage in selectively weighting the memory items according to differences in their potential action-relevance, because the item presented at the movement goal location was never more task-relevant than the other memory items. Indeed, Ohl and Rolfs (2017) even observed better memory performance for the item presented at the saccade goal when this item was far less likely to be tested than any other item. These findings point to a plausible mechanism underlying the influence of action-relevance on visual working memory. The preparation and execution of manual movements and saccades are known to involve the allocation of spatial attention towards the action goal, enhancing visual sensitivity at that location (e.g., Baldauf & Deubel, 2008; Baldauf & Deubel, 2010; Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Kowler, Anderson, Dosher, & Blaser, 1995; Rolfs & Carrasco, 2012), and such action-related shifts of attention have been shown to occur automatically and involuntarily (e.g., Deubel & Schneider, 1996; Deubel, Schneider, & Paprotta, 1998; Hoffman & Subramaniam, 1995). Presumably, attention is also shifted internally, within working memory, towards representations spatially corresponding to action goal locations, yielding preferential maintenance of potentially action-relevant information and thereby ensuring that the information that may be required for action planning and online control is easily available. According to this reasoning, action relevance obligatorily biases maintenance in visual working memory as an inherent part of action planning. 

Motivational value

Objects and events in our visual world have different value for us, and we accordingly choose our actions to obtain rewarding outcomes. For instance, we have learned that choosing a fresh green apple instead of a rotten brown one will result in a more rewarding pie. Motivational value is thus an important incentive that affects the perceived importance of situations or specific objects and consequently also the processing of visual information (e.g., Anderson, 2016; Chelazzi, Perlato, Santandrea, & Della Libera, 2013; Failing & Theeuwes, 2017; Pessoa, 2015). 

The general prospect of receiving reward can boost visual working memory capacity, most likely via an increase in motivation. When information about the amount of reward that can be earned is provided at the beginning of each trial, memory performance has been found to be better when a high reward can be earned than when a low or no reward can be earned (Kawasaki & Yamaguchi, 2013; Krawczyk, Gazzaley, & D'Esposito, 2007; Sanada, Ikeda, Kimura, & Hasegawa, 2013). Motivational value can not only globally enhance memory performance, but also modulate working memory for specific stimuli or features, with better performance for items associated with a high reward (Gong & Li, 2014; Klink, Jeurissen, Theeuwes, Denys, & Roelfsema, 2017; Thomas, FitzGibbon, & Raymond, 2016; Wallis, Stokes, Arnold, & Nobre, 2015), and worse performance for value-neutral items that are presented along with highly rewarded items (Infanti, Hickey, & Turatto, 2015) or items that were associated with a loss (Thomas et al., 2016). In these studies, an association between a task-irrelevant feature and reward magnitude was established in training tasks, but reward was no longer provided during the subsequent visual working memory tasks (except for the study by Klink et al., 2017, which included both a training task and reward feedback during the memory task). It appears that such an association continues to influence visual working memory processing, at least for some time (see also Klink et al., 2017), even when there is no strategic advantage in prioritizing stimuli associated with a high reward for storage. 

Effects of motivational value on visual working memory, similar to effects of action relevance, can be assumed to be the result of an attentional bias. Reward has a profound impact on the allocation of selective attention, with highly-rewarded items automatically capturing attention irrespective of their task relevance (e.g., Anderson, 2016; Anderson, Laurent, & Yantis, 2011; Chelazzi et al., 2013; Della Libera & Chelazzi, 2009). It is tempting to assume that reward-related modulations of visual working memory are simply the consequence of an attentional bias at the perceptual stage, that is, that high-reward items are prioritized for access to visual working memory and encoded with a higher probability or higher precision. But whereas a recent study presenting reward cues at different stages of visual working memory processing indicated that a reward-related weighting of memory items is particularly pronounced when information about their motivational value is available at encoding (Klink et al., 2017), there is also evidence indicating that such a weighting is unlikely to be the result of an attentional bias at encoding alone, but that maintenance processes, too, can be modulated (Thomas et al., 2016). Such a modulation during maintenance might be achieved through the same mechanism as when a weighting within visual working memory is implemented to reflect differences in explicit task-relevance, namely through an internal allocation of attention. 

Rationale

Different sources of selection bias are mostly studied in isolation, but outside the laboratory, they are typically encountered in combination. In the present study, we set out to investigate how action relevance and motivational value, two presumably predominant types of selection bias in everyday life, combine to modulate visual working memory. In a change detection task, participants memorized the colors of four items, all of which were equally likely to be tested and therefore equally task-relevant. Action relevance and motivational value of different items were each determined by a specific feature of the memory items (location and shape) and manipulated in an orthogonal fashion. An item could be rendered action-relevant due to a spatial correspondence with the goal of a pointing movement during the retention interval, or action-irrelevant when it was presented at one of the locations that did not serve as movement goal on a given trial. An item could be assigned a high motivational value, when its shape (e.g., a square) was associated with a high reward, or a low motivational value, when its shape (e.g., a circle) was associated with a low reward. Unlike in previous studies, the reward assignment was not established in a separate training task, but participants received the reward associated with the shape of the tested item for correct responses in the memory task itself. Thus, there was a strategic advantage in prioritizing high-reward items, and we accordingly expected better performance than for low-reward items. Weighting items to reflect differences in action-relevance, by contrast, had no strategic value for either the memory or the movement task: The memorized information (i.e., color) was not required to perform the pointing movement, and the movement goal location was not predictive of the memory test item location. But if action-relevance does indeed automatically bias maintenance in visual working memory, such a weighting should nevertheless be observed. The effects of action relevance and motivational value were first examined in isolation (Experiments 1A and 1B) and then in combination (Experiment 2). 

Experiment 1A: Action relevance

Experiment 1A examined the effect of action-relevance on visual working memory maintenance using a slightly modified version of the paradigm employed by Heuer et al. (2017). During the retention interval of a color change detection task, participants performed a pointing movement following a movement cue that indicated one of the memory item locations, but that was not predictive for the memory task. In one half of the experiment, participants were to point to the peripheral location indicated by the cue, and in the other half they were to always point to the fixation dot. This fixation movement goal condition controlled for an effect of the cue itself. The test item in the memory task was subsequently presented either at the cued location, or at a noncued location. We expected that performance at the cued location would be better than at noncued locations, but only so when the movement goal was peripheral, that is, when the cued location was also the action goal location, whereas the noncued locations were action-irrelevant. When the movement goal was the fixation dot, all maintained items spatially corresponded to action-irrelevant locations, and no differences in performance at cued and at noncued locations should be observed. 

Methods

Participants

Fourteen students of Philipps-Universität Marburg participated in the experiment (12 female, two male; aged 18–31 years, mean age 23 years). Informed written consent was obtained from all participants prior to the experiment. The experiment was approved by the Ethics Committee of the Faculty of Psychology at Philipps-Universität Marburg and conducted in accordance with the ethical standards laid down in the Declaration of Helsinki. Participants were naive to the purpose of the experiment, right-handed, and had normal or corrected-to-normal visual acuity and color vision. Visual acuity and color vision were tested with the OCULUS Binoptometer 3 (OCULUS Optikgeräte GmbH, Wetzlar, Germany). 

Apparatus and stimuli

Participants were seated in a dimly lit room, facing a monitor (22 in., 1,680 × 1,050 pixels) placed at a distance of 104 cm from their eyes. At a distance of 55 cm from the participants' eyes, a framed glass plate was mounted on the table. The glass plate was adjusted to the eye height of each participant to ensure that it always covered the entire monitor. Pointing movements were performed towards this glass plate. Participants had a wooden board with a response box to the left and a movement pad to the right in front of them. To respond to the test item of the memory task, participants pressed the buttons on the response box using their left middle and index fingers. Between pointing movements, their right hand was positioned on the movement pad, on which a white cross marked the starting position for index finger and thumb. A Windows PC running E-Prime 2.0 (Psychology Software Tools, Inc) controlled stimulus presentation and response collection. To identify pointing errors and outliers, movements were recorded using a magnetic motion-tracking device (Polhemus Liberty 240/8, Polhemus Inc) with a sensor fixed on top of the tip of participants' right index finger. Movement data collection was controlled using MATLAB (MathWorks, Natick, MA). 

All stimuli were presented against a gray background. Memory and test items were squares or circles subtending an area with a diameter of 1.16° of visual angle, presented at four fixed positons at a distance of 4.74° of visual angle from fixation. Circle outlines with a diameter of 1.93° of visual angle marked these positions and were present throughout the entire experiment. The distance between adjacent memory items (center to center) was 6.6° of visual angle. In each trial, two memory items were circles and two were squares, equally distributed across the four positions and tested with equal probability. The colors of the memory items were randomly selected from a set of seven isoluminant colors (blue, green, ochre, orange, pink, red, and violet). On a given trial, all memory items had different colors. The color of the test item was either identical to that of the memory item previously presented at that location or randomly selected from the remaining six colors. The cue was a line (0.5° × 0.06° of visual angle) originating from the fixation dot and pointing towards one of the four memory/test item positions. The fixation dot subtended 0.17° of visual angle, and the go-signal (enlarged fixation dot) 0.23° of visual angle. 

Procedure and design

The task is illustrated in Figure 1A. Each trial started with the presentation of the memory array for 200 ms. It consisted of four memory items (two circles and two squares), and participants were instructed to memorize their colors. After an interval of 800 ms, a movement cue was presented for 200 ms. This cue was a line pointing from the fixation dot to one of the four positions. To provide ample time for action preparation and specification of the movement goal prior to the start of the movement, the cue was followed by a short interval with a duration varying randomly between 280 and 320 ms (in steps of 10 ms), after which a go-signal in the form of an enlargement of the fixation dot was presented until the movement was initiated. In one half of the experiment, participants were to execute a pointing movement towards the peripheral position indicated by the cue (the top right position in the example illustrated in Figure 1A), and in the other half they were to always point towards the fixation dot, irrespective of where the cue had pointed. Participants were instructed to maintain fixation throughout the experimental trials, even during the pointing movements. After completion of the movement and after another fixed interval of 600 ms, a test item was presented at one of the four positions, and participants were to indicate whether the color of this test item matched the color of the memory item previously presented at that position. The test item was present until response, but participants were encouraged to respond quickly. The next trial started 1,000 ms after the response. 

Figure 1

 

Experiment 1A. (A) Trial procedure and conditions. A trial started with presentation of the memory array for 200 ms. This array consisted of four items (two circle- and two square-shaped) and participants were instructed to memorize their colors. After 600 ms, a movement cue appeared for 200 ms, pointing to one of the four locations. Upon presentation of a go-signal (enlargement of the fixation dot), appearing after a variable delay of 280–320 ms, participants were to perform a pointing movement. In one half of the experiment, the goal of this movement was the peripheral location indicated by the cue. In the other half of the experiment, the goal was the fixation dot, irrespective of where the cue had pointed. After completion of the movement and after another interval of 600 ms, a test item was presented at one of the locations until response. Participants were to indicate whether the test item color matched the color of the item previously presented at that location. In 25% of all trials, the test item was presented at the cued location. In the other 75% of trials, it was presented at one of the other three locations. (B) Results. Accuracy shown separately for the movement goal conditions (peripheral in the left panel and fixation in the right panel) and for test items presented at the cued (left in each panel) or at a noncued location (right in each panel). Black lines show the mean values; gray lines show the values of individual participants. Error bars show standard errors of the means.

Testing took place in two sessions on separate days. On the first day, participants performed a short training task to get acquainted with the pointing task (e.g., to practice maintaining fixation even during the movement). This task was not related to the main task, which was performed on the second day. After the experiment, participants filled in a questionnaire to ensure that no inappropriate strategies (e.g., memorizing only a self-chosen subset of items; see Heuer et al., 2017) were used. 

The experiment consisted of 384 trials, organized in blocks of 48 trials each. The movement goal (peripheral vs. fixation) changed after the first half of the experiment: In one half of the experiment, the movement goal was the peripheral position indicated by the cue and spatially corresponded to a memory item location, and in the other half the movement goal was the fixation dot. The order of the two movement goal conditions was balanced across participants. Test item position (cued location vs. noncued location) was varied in each trial. All four memory item positions were equally likely to be tested (i.e., the cued location in 25% of trials and a noncued location in 75% of trials). 

Data analyses

Trials with excessively long reaction times (>2.5 SD from mean reaction time, calculated separately for each participant; on average, 2.9% of all trials) and trials with pointing errors or pointing outliers (on average, 2.5% of all trials) were removed from the data. Trials with pointing errors or outliers were defined as trials in which the movement endpoint deviated by more than 2.5 SD from the mean of all movement endpoints for a given movement goal on the horizontal or vertical axis. The primary measure of interest for memory performance was accuracy in percent correct. Additionally, reaction times were analyzed to ensure that there were no speed-accuracy trade-offs. For reaction times, only trials with correct responses were included. Individual mean accuracy and reaction time measures were submitted to two-way, repeated measures ANOVAs with the factors movement goal (peripheral vs. fixation) and test item position (cued vs. noncued). 

Results and discussion

Figure 1B shows accuracy in percent correct, separately for the two movement goal conditions (peripheral and fixation) and for test items presented at the cued and at noncued locations. Accuracy for items presented at the cued location (77.08% ± 2.56%) was higher than for items presented at noncued locations, 72.9% ± 2.26%; F(1, 13) = 8.25, p = 0.013, partial η² = 0.388. Importantly, an interaction, F(1, 13) = 5.86, p = 0.031, partial η² = 0.311, revealed that this difference in accuracy between cued and noncued locations differed across movement goal conditions. While accuracy at the cued location (77.76% ± 3.26%) was higher than at noncued locations (69.42% ± 2.79%) when the movement goal was peripheral, t(13) = 2.9, p = 0.006; one-tailed t test, accuracy at cued (76.39% ± 2.71%) and noncued locations (76.37% ± 2.29%) was at the same level when the movement goal was fixation, t(13) = 0.01, p = 0.495; one-tailed t test. There were no significant effects for reaction times. 

These results are a replication of the findings of Heuer et al. (2017) under somewhat modified conditions (i.e., in a nonlateralized task): When the cued location spatially corresponded to the action goal location while all noncued locations were action-irrelevant (i.e., in the peripheral condition), performance at the cued location was better than at the other locations. When, by contrast, all memory item locations were action-irrelevant (i.e., in the fixation condition), performance at cued and noncued locations did not differ. Thus, maintained items were selectively weighted to reflect differences in their potential action-relevance. 

Comparing performance across the two movement goal conditions appears to suggest that this weighting resulted from an inhibition of action-irrelevant information rather than from a facilitation of action-relevant information. When the movement goal was peripheral, accuracy at noncued locations was lower than when the movement goal was fixation, whereas performance at the cued location was at the same level in both movement goal conditions. However, it is important to note that while the fixation movement goal condition controls for an effect of cue presentation, it is not an appropriate baseline for memory performance in the peripheral condition. The peripheral condition, in which participants were to point to a different location in each trial, was more demanding than the fixation condition, which likely resulted in differences in overall accuracy. Thus, no conclusions can be drawn from the comparison of performance across movement goal conditions. 

Experiment 1B: Motivational value

Experiment 1B examined the isolated influence of motivational value on visual working memory encoding and storage. The same color change detection task as in Experiment 1A was used, but there was no additional movement task, and thus there were no differences in the potential action-relevance of maintained information. Motivational value was coupled to the shape of the memorized items: Either circle-shaped or square-shaped items were associated with a high reward, and the other with a low reward, which participants received for correct responses. Importantly, both types of items were equally likely to be tested and thus equally important for performance in the memory task itself. We expected better performance for items associated with a high reward than for items associated with a low reward. 

Methods

Unless stated otherwise, the methods of Experiment 1B were identical to those of Experiment 1A. 

Participants

Fourteen students of Philipps-Universität Marburg participated in the experiment (11 female, three male; aged 19–28 years, mean age 23 years). None of them had participated in Experiment 1A. 

Apparatus and stimuli

Participants had the same wooden board as in Experiment 1A in front of them, with a response box to the left and a movement pad to the right, and pressed the two buttons on the response box with their left middle and index fingers to respond to the test item in the memory task. As no movements were executed in this experiment, no movement data were recorded, the glass plate was not placed in front of the monitor, and participants' right hands rested on the board and did not have to be positioned on the start position. The reward feedback presented at the end of each trial (“+9,” “+1,” or “+0”) subtended 0.77° of visual angle. 

Procedure and design

The task is illustrated in Figure 2A. The presentation of the memory array was followed by a retention interval of 3,600 ms. This long interval was chosen to ensure that the overall duration of the retention interval would be similar to the overall duration of the retention interval in Experiment 1A. After response to the test item, reward feedback was presented for 800 ms. Correct responses were rewarded with either a high (+9 points) or a low reward (+1 point). If participants responded incorrectly, they received no reward (+0 points). Reward points were converted into monetary reward at the end of the experiment (1€ for 625 points). Reward magnitude was coupled to the shape of the tested item; that is, either squares or circles were associated with a high reward, and the other with a low reward. This reward assignment was constant for each participant, but balanced across participants. Participants were informed about this assignment prior to the experiment, but they were also informed that all items were equally likely to be tested and that consequently both types of items were equally important for adequate performance in the memory task. 

Figure 2

 

Experiment 1B. (A) Trial procedure and conditions. A trial started with presentation of the memory array for 200 ms. This array consisted of four items: two circle- and two square-shaped. Participants were instructed to memorize the colors of the items. After an interval of 3,600 ms (matched to the approximate overall duration of the retention interval in Experiment 1A), a test item was presented at one of the four locations, and participants were to indicate whether this test item was of the same color as the memory item previously presented at that location. After response to the test item, reward feedback was presented for 800 ms. For correct responses, participants received either a low (+1 point) or a high reward (+9 points). Reward magnitude depended on the shape of the test item: Either circles or squares were associated with a low, and the other with a high reward. This assignment of reward to shape was constant for each participant but balanced across participants. The two shapes were equally likely to be tested. Participants received no reward (+0 points) for incorrect responses. (B) Results. Accuracy shown separately for test items associated with a low reward (left) and for test items associated with a high reward (right). Black lines show the mean values; gray lines show the values of individual participants. Error bars show standard errors of the means.

Testing took place in one session. The experiment consisted of 384 trials. In half of all trials, the test item was an item associated with a low reward, and in the other half the test item was associated with a high reward. 

Data analyses

Trials with excessively long reaction times (>2.5 SD from mean reaction time, calculated separately for each participant; on average, 3% of all trials) were removed from the data. Accuracy and reaction times were computed separately for trials, in which the test item was associated with a low reward, and for trials, in which the test item was associated with a high reward, and compared using paired, one-tailed t tests. 

Results and discussion

Figure 2B shows accuracy in percent correct separately for test items associated with a low reward and for test items associated with a high reward. Accuracy for high-reward items (79.59% ± 3.67%) was higher than for low-reward items, 73.64% ± 3.85%; t(13) = 2.4, p = 0.016. Even though the response was not speeded, reaction times to high-reward items (1,126 ± 76 ms) were also shorter than for low-reward items, 1179 ± 88 ms; t(13) = 1.86, p = 0.043. 

These findings demonstrate that participants prioritized information with a higher value for visual working memory processing. Even though low- and high-reward items were equally likely to be tested, this reward-related bias conferred a strategic advantage. The task was rather difficult and mistakes were bound to be made (overall accuracy was at 77%). Biasing processing in favor of high-reward items ensured that such mistakes were likely to incur only a small cost (i.e., the loss of a low reward). The present task does not allow differentiating between prioritization at encoding and prioritization during maintenance, but as the assignment of reward magnitude to a specific shape was constant for each participant, it is likely that high-reward items were already preferentially encoded. 

Experiment 2: Action relevance and motivational value

Experiment 2 investigated the combined influence of action relevance and motivational value on visual working memory encoding and storage by merging the tasks of Experiments 1A and 1B. The color change detection task remained unchanged: Participants memorized the colors of four items, which were equally likely to be tested and thus equally task-relevant. But individual items differed in their action relevance and motivational value: They could be either action-relevant or action-irrelevant, and be associated with either a low reward or a high reward. These two sources of bias were each associated with a specific feature of the items (location and shape, as in Experiments 1A and 1B) and manipulated in an orthogonal fashion. Different outcomes of the combination of these selection biases were conceivable: The effects of action relevance and motivational could be additive and independent, or they could interact, with one selection bias “dominating” the other. 

Methods

Unless stated otherwise, the methods used in Experiment 2 were identical to those in Experiment 1A. 

Participants

Twenty-four students of Philipps-Universität Marburg participated in the experiment (17 female, seven male; aged 19–25 years, mean age 22 years). None of them had participated in either Experiment 1A or Experiment 1B. Sample size was increased for Experiment 2 to increase statistical power for the analysis of potential interactions between action relevance and motivational value. 

Procedure and design

The trial procedure of Experiment 2 (shown in Figure 2A) was essentially a combination of the procedures of Experiments 1A and 1B: Up to and including test item presentation, the procedure was identical to that of Experiment 1A. The test item was then followed by the reward feedback, as in Experiment 1B. A trial started with the presentation of the memory array for 200 ms, consisting of two square- and two circle-shaped items, and participants were to memorize the memory item colors. After an interval of 800 ms, the movement cue was presented for 200 ms. Following an interval of 280–320 ms (varied randomly in steps of 10 ms), the go-signal appeared and participants executed a pointing movement towards either the peripheral location indicated by the cue, or the fixation dot. After completion of the movement and after another fixed interval of 600 ms, the test item was presented at either the cued or a noncued location until response, and participants had to indicate whether the test item color matched the color of the memory item previously presented at that location. After response to the test item, reward feedback was provided for 800 ms. For correct responses, participants received either a high (+9 points) or a low reward (+1 point), depending on the shape of the tested item. Either squares or circles were associated with a high reward, and the other with a low reward. This assignment was constant for each participant, but balanced across participants. As in Experiment 1B, participants were informed about the reward assignment, but they were also informed that all items were tested equally often and were thus equally important for adequate task performance. For incorrect responses, participants received no reward (+0 points). Reward points were converted into monetary reward at the end of the experiment (1€ for 1,000 points). The number of reward points that corresponded to 1€ was adjusted to the larger trial number and longer duration (as compared to Experiment 1B). 

Testing took place in two sessions on separate days. On the first day, participants performed the same pointing movement training task as in Experiment 1A. On the second day, the main experiment was performed. This experiment consisted of 768 trials, equally distributed among reward conditions (low and high reward) and movement goal conditions (fixation and peripheral). All four memory item positions were equally likely to be tested (i.e., the test item was presented at the cued location in 25% of all trials, and at a noncued location in 75% of all trials). The order of the two movement goal conditions was balanced across participants. 

Data analyses

Trials with excessively long reaction times (>2.5 SD from mean reaction time, calculated separately for each participant; on average, 2.6% of all trials) and trials with pointing errors or pointing outliers (on average, 2.8% of all trials) were removed from the data. Individual mean accuracy and reaction time measures were submitted to two-way repeated measures ANOVAs with the factors test item position (cued vs. noncued) and reward (low vs. high), separately for each movement goal condition (peripheral vs. fixation). We will focus on these two separate ANOVAs for the sake of simplicity and clarity, but we will also briefly report the results of a three-way, repeated measures ANOVA including all factors. The primary purpose of this analysis is to confirm that differences in the results of the two separate ANOVAs go along with the corresponding interactions with the factor movement goal. 

Results and discussion

Figure 3B shows accuracy in percent correct separately for the two movement goal conditions (peripheral vs. fixation), test item position (cued vs. noncued), and reward (low vs. high). For the peripheral movement goal condition (left panel in Figure 3B), a main effect of test item position, F(1, 23) = 18.33, p < 0.001, partial η² = 0.444, revealed that accuracy was higher for items presented at the cued location (77.29% ± 1.65%) than for items presented at noncued locations (69.74% ± 1.47%). This difference in performance at cued and noncued locations was observed for both low-reward, t(23) = 3.15, p = 0.003; one-tailed t test, and high-reward test items, t(23) = 4.25, p < 0.001; one-tailed t test. Thus, memory for items presented at a location spatially corresponding to an action goal location was better than for items presented at action-irrelevant locations, irrespective of the reward value associated with the items. There was no main effect of reward, but a trend for higher accuracy for high-reward items (75.2% ± 1.7%) than for low-reward items (71.83% ± 1.72%). The interaction between reward and test item position fell just short of significance, F(1, 23) = 3.31, p = 0.082, partial η² = 0.126, but follow-up t tests revealed that while accuracy for high- and low-reward items was at the same level at the cued position, t(23) = 0.6, p = 0.276; one-tailed t test, accuracy for high-reward items was significantly higher than for low-reward items at noncued locations, t(23) = 2.23, p = 0.018; one-tailed t test. 

Figure 3

 

Experiment 2. (A) Trial procedure and conditions. A trial started with presentation of the memory array (200 ms), consisting of four items: two circles and two squares. Participants were instructed to memorize the item colors. After 600 ms, a movement cue pointing to one of the four locations appeared for 200 ms. Upon presentation of the go-signal (enlargement of the fixation dot) after a variable delay of 280–320 ms, participants were to perform a pointing movement towards the display. In one half of the experiment, the goal of this movement was the peripheral location indicated by the cue, and in the other half the goal was the fixation dot. After completion of the movement and after another interval of 600 ms, a test item was presented at one of the locations, and participants were to indicate whether this test item was of the same color as the memory item previously presented at that location. In 25% of trials, the test item was presented at the cued location, and in the other 75% of trials, it was presented at a noncued location. After response to the test item, reward feedback was presented for 800 ms. For correct responses, participants received either a low (+1 point) or a high reward (+9 points). Reward was coupled to the shape of the items: Either circles or squares were associated with a low, and the other with a high reward. This assignment was constant for each participant, but balanced across participants. For incorrect responses, participants received no reward (+0 points). (B) Results. Accuracy shown separately for the movement goal conditions (peripheral in the left panel and fixation in the right panel), for test items presented at the cued (to the left in each panel) and at noncued locations (to the right in each panel) and for test items associated with a low (blue line) and a high reward (orange line). Solid and bold lines show the mean values, dashed and semitransparent lines show the values of individual participants. Error bars show standard errors of the means.

For the fixation movement goal condition (right panel in Figure 3B), only a main effect of reward was observed, F(1, 23) = 7.48, p = 0.12, partial η² = 0.25, attributable to higher accuracy for high-reward items (76% ± 1.72%) than for low-reward items (69% ± 2.34%). The absence of an effect of test item position once again confirmed that the nonpredictive cue by itself had no effect, indicating that the effect of test item position with a peripheral movement goal was indeed due to the differences in potential action-relevance and not the result of movement cue presentation. 

The analysis of reaction times only revealed a significant effect of test item position, F(1, 23) = 27.32, p < 0.001, partial η² = 0.54, for the peripheral movement goal condition, with faster responses to items presented at the cued position (840 ± 47 ms) than at noncued locations (885 ± 49 ms). Thus, memory items previously presented at an action goal were recalled not only with higher accuracy but also faster than items presented at action-irrelevant locations, although the response was not speeded. 

Submitting individual measures to a three-way, repeated measures ANOVA including the additional factor of movement goal confirmed the effect of action relevance, indicated by an interaction of the factors movement goal and test item position for both accuracy, F(1, 23) = 10.79, p = 0.003, partial η² = 0.32 and reaction times, F(1, 23) = 7.34, p = 0.013, partial η² = 0.24. There was also a main effect of test item position in terms of accuracy, F(1, 23) = 13.79, p = 0.001, partial η² = 0.38, and reaction times, F(1, 23) = 14.47, p = 0.001, partial η² = 0.39, and a main effect of reward for accuracy, F(1, 23) = 5.99, p = 0.022, partial η² = 0.21. None of the other effects were significant. 

General discussion

This study examined how action relevance and motivational value, two predominant types of selection bias affecting the importance of specific parts of our visual surroundings, combine to modulate visual working memory. The effects of action relevance and motivational value were first examined in isolation and then in combination, using the same change detection task to measure visual working memory performance in all experiments. 

In isolation, both action relevance and motivational value were found to influence memory for visual information over short periods of time, corroborating previous findings (e.g., Gong & Li, 2014; Heuer et al., 2017; Klink et al., 2017; Ohl & Rolfs, 2017; Wallis et al., 2015). Information that was potentially action-relevant due to a spatial correspondence with an action goal could be better recalled than information spatially corresponding to action-irrelevant locations (Experiment 1A). A control condition, in which movements were performed towards the fixation dot, confirmed that the observed differences in performance did indeed reflect differences in action relevance and were not the result of an automatic shift of attention following the nonpredictive movement cue. Examined separately (Experiment 1B), visual working memory performance also reflected differences in the motivational value of specific items: Memory was better for items associated with a high reward than for items associated with a low reward. Notably, unlike previous studies (Gong & Li, 2014; Infanti et al., 2015; Thomas et al., 2016; Wallis et al., 2015), the assignment of reward to a specific feature (in this study, shape) was not established in a separate task performed prior to the memory task, but participants received the respective reward for correct responses in the memory task itself. Therefore, biasing visual working memory processing in a value-related manner conferred a strategic advantage, even though low- and high-reward items were tested equally often: It minimized the risk of losing a high reward by responding incorrectly to one of the respective items. 

Investigated in combination (Experiment 2), action relevance and motivational value still each exerted an influence on visual working memory, but not in a fully independent and additive manner. Whereas action relevance impacted on performance irrespective of the motivational value associated with the items, an effect of motivational value was only observed for action-irrelevant items. At the descriptive level, though, there was still a trend for better performance for action-relevant items associated with a high reward than for action-relevant items associated with a low reward. In fact, the interaction between action relevance and motivational value failed to reach significance. 

As outlined in the Introduction, the basic mechanism underlying the weighting of visual working memory contents according to differences in action relevance and motivational value can be presumed to be the same: More attentional resources are allocated towards specific items, improving memory for these items at the expense of others. One might indeed speculate that implementations of biases induced by action relevance and motivational value tap into the same limited attentional resource, resulting in the underadditive effects observed in Experiment 2. However, there are other possible explanations that could account for the underadditive effects of motivational value and action-relevance, and that do not require the assumption of a shared resource. 

For one, it is conceivable that the lack of a significant effect of motivational value on action-relevant information was due to a ceiling effect. This ceiling might simply reflect the maximal average performance level that could be achieved in this rather difficult task. Moreover, a high motivational value was always assigned to two different items on each trial, whereas only one item was rendered action-relevant. This might have resulted in a weaker priority signal for a given high reward item as compared to the single action-relevant item. Lastly, the timing of attentional orienting towards action-relevant and high-reward items might have been different in the present study. As in previous studies (Heuer et al., 2017; Ohl & Rolfs, 2017), the action-related weighting of representations could only have been implemented during maintenance, because the movement was cued and executed well after the initial encoding of the memory items. The reward assignment, by contrast, was not varied on a trial-by-trial basis, but kept constant for each participant. Thus, it seems reasonable to assume that high-reward items, unlike action-relevant items, were already prioritized upon initial presentation of the memory array, meaning that they were encoded with a higher probability and/or a higher precision. This would be in line with previous studies, which indicate that reward appears to primarily influence the encoding phase (e.g., Infanti et al., 2015; Klink et al., 2017). 

If the two types of selection bias operated at different points in time, that is, at different stages of visual working memory processing, this would mean that the effect of action relevance relied on an internal allocation of attention, whereas the effect of motivational value might have relied on external attention. Whereas it has been shown that there is a substantial overlap of internal and external attention in terms of behavioral consequences and underlying neural networks (e.g., Dell'Acqua, Sessa, Toffanin, Luria, & Jolicoeur, 2010; Griffin & Nobre, 2003; Nee & Jonides, 2009; Poch, Campo, & Barnes, 2014), there is also evidence that they might differ in some regards (e.g., Nee & Jonides, 2009; Nobre et al., 2004; Tanoue & Berryhill, 2012). Moreover, the attentional modulation of different stages of processing might have involved different mechanisms through which memory for important pieces of information was improved. More specifically, associated reward might have affected the precision, with which an item was initially encoded (see also Klink et al., 2017), whereas action relevance, only introduced during the maintenance stage, could perhaps no longer increase the representational quality, but shielded action-relevant information from decay or interference. The latter idea would dovetail with findings obtained with retrocues, which also act during the maintenance stage: Whereas only a few studies have observed benefits in precision, retrocues have consistently been reported to improve recall probability (for an overview, see Souza & Oberauer, 2016), which appears to be at least partly due to a protection from visual interference (Souza, Rerko, & Oberauer, 2016). Thus, while the basic process underlying the effects of action relevance and motivational can be assumed to be the same, namely an attentional prioritization, it seems unlikely that these different types of selection bias rely on exactly the same mechanism. 

Interestingly, the underadditivity of the effects came at the expense of the influence of motivational value but not action relevance. This is perhaps the most striking aspect of the findings of Experiment 2: The effect of action relevance was particularly pronounced and robust, with items spatially corresponding to the action goal being recalled with higher accuracy and faster than action-irrelevant items, regardless of their motivational value. This is quite remarkable for two reasons. First, there was no strategic advantage in biasing memory processing towards action-relevant information. It did not benefit performance in the memory task, because all items were equally likely to be tested, and it did not benefit performance in the movement task either, because the color information was not needed to perform a pointing movement towards a marked location. Second, in this task context, action relevance was pitted against a presumably very strong selection bias that conferred a direct advantage for participants. Biasing memory processing towards high-reward items would not have benefited performance in the memory (or movement) task either, but it would have directly affected the overall positive outcome associated with task performance, that is, the magnitude of the monetary reward, which participants received after completion of the experiment. Nevertheless, there was a robust influence of action relevance, whereas an effect of motivational value was only reliably observed for action-irrelevant information. These findings provide strong support for the idea that an attentional weighting of visual working memory representations reflecting differences in action relevance occurs involuntarily as an inherent, hard-wired part of action planning (see also Ohl & Rolfs, 2017). Similar to what has been shown for the deployment of spatial attention towards external events (e.g., Deubel & Schneider, 1996; Deubel, Schneider, & Paprotta, 1998a; Hoffman & Subramaniam, 1995), internal attention would accordingly be automatically drawn to representations spatially corresponding to an action goal, thereby improving memory for information that might be important for an upcoming action at the cost of action-irrelevant information. 

It should be noted that we did not use eye tracking to ensure that participants kept fixation at the center of the screen even during the execution of pointing movements. As saccadic eye movements have also been shown to bias memory performance in favor of information presented at the saccade goal (Ohl & Rolfs, 2017), the observed effects of action relevance could be argued not to be the result of the pointing movements, but of accompanying eye movements. However, we did specifically train participants to perform the pointing movements without looking to the movement goal location in a separate session prior to the main experiment, and the variance of movement endpoints (not reported for the present experiments, but as in our previous study; Heuer et al., 2017) indicated that they did not saccade to the goal location. In spite of this, we cannot entirely rule out that small eye movements towards the direction of the pointing goal location occurred. Importantly, even if the observed effects were due to eye movements rather than pointing movements, this would still be an effect of action relevance. Moreover, recent work suggests that effects of eye and hand movements on visual working memory are independent and rely on separate, effector-specific attentional mechanisms (Hanning & Deubel, 2018). 

In summary, the present experiments have corroborated and extended previous findings demonstrating that action relevance and motivational value modulate visual working memory processing, even when they are irrelevant for the task at hand. When these two selection biases co-occur, as is presumably often the case in everyday situations, they still each exert an influence, but not in a fully additive and independent manner. The effect of action relevance was found to be particularly robust and was observed irrespective of the motivational value of tested items, indicating that an action-related bias might be implemented automatically during action planning. At a broader level, these findings show that visual working memory representations can be flexibly weighted to reflect differences in their relevance, as determined by different sources of information. This ability to take into account several factors affecting the importance of specific pieces of visual information optimizes the efficient use of the capacity-limited working memory system. 

Acknowledgments

This research was supported by the Deutsche Forschungsgemeinschaft (German Research Foundation; SFB/TRR 135, TP B3). The authors would like to thank Aylin Hanne for assistance with data collection. 

Commercial relationships: none. 

Corresponding author: Anna Heuer. 

Email: . 

Address: Experimental and Biological Psychology, Philipps-Universität Marburg, Marburg, Germany.