A value-driven mechanism of attentional selection

Abstract

Abstract

Abstract:

Abstract  Attention selects stimuli for cognitive processing, and the mechanisms that underlie the process of attentional selection have been a major topic of psychological research for over 30 years. From this research, it has been well documented that attentional selection can proceed both voluntarily, driven by visual search goals, and involuntarily, driven by the physical salience of stimuli. In this review, I provide a conceptual framework for attentional control that emphasizes the need for stimulus selection to promote the survival and wellbeing of an organism. I argue that although goal-driven and salience-driven mechanisms of attentional selection fit within this framework, a central component that is missing is a mechanism of attentional selection that is uniquely driven by learned associations between stimuli and rewards. I go on to review recent evidence for such a value-driven mechanism of attentional selection, and describe how this mechanism functions independently of the well-documented salience-driven and goal-driven mechanisms. I conclude by arguing that reward learning modifies the attentional priority of stimuli, allowing them to compete more effectively for selection even when nonsalient and task-irrelevant.

Introduction

The external world presents a large amount of information to be processed by an organism. Visual scenes often contain many different objects, each of which is comprised of many component features. Representing these objects and features is a fundamental task of the visual system, as what is represented will then be available for higher-order cognitive processes, such as reasoning, decision making, and memory storage. 

Given the large amount of information contained in visual scenes, the brain's ability to represent all of this information is severely limited. Stimuli compete for representation in the brain, requiring a process of selection by which selected objects are represented at the expense of others (e.g., Desimone & Duncan, 1995; Reynolds, Chelazzi, & Desimone, 1999). This core mechanism of selection is referred to as selective attention, and it has been the topic of intense investigation for over 30 years. 

Attentional priority as a function of stimulus value

Which stimuli are selected by attention has important implications for the survival and wellbeing of an organism. In order for a stimulus to be acted upon, it must first be attended such that it becomes available to resource-limited cognitive systems such as working memory and decision making. Stimuli that are not attended often fail to reach awareness (Mack & Rock, 1998; Most et al., 2001; Rensink, O'Regan, & Clark, 1997). How quickly a stimulus is selected via attention is also of critical importance. By more rapidly attending to a stimulus, individuals maximize the amount of time they have to act upon the stimulus. Failing to rapidly attend to a stimulus may result in a missed opportunity to obtain a reward or to avert a negative outcome. 

I argue here that attentional priority reflects the overall value of selecting the different stimuli in an environment. Core mechanisms of computing attentional priority will evolve to the extent that they increase the likelihood that an organism will survive and thrive. More specifically, attentional priority will be computed in such a way that, if the same computations are repeated over many instances, these computations will maximize the rewards and minimize the losses experienced by the organism. Although the attentional priority of a particular stimulus in a particular context will not always be adaptive, the computations that underlie attentional priority will give rise to selection that is as adaptive as possible in the long run. 

Two mechanisms of attentional selection have been well defined in the literature: a goal-driven mechanism and a salience-driven mechanism (e.g., Connor, Egeth, & Yantis, 2004). I briefly review each one here and argue that each is consistent with a value-based framework for the computation of attentional priority. An attentional system optimized to maximize rewards and minimize losses will prioritize both goals and salience in determining which stimuli are selected. 

Goal-driven attentional selection

Our goals are often adaptive. When we are hungry, we search for food. When we know danger may be present, we look for it and try to avoid it. Our goals are also very flexible and can rapidly adapt to changes in expectations and task demands (e.g., Lien, Ruthruff, & Johnston, 2010). Thus, it would seem useful for an adaptive system of attentional control to be able to factor ongoing goals into the computation of attentional priority. 

A wealth of research demonstrates that currently active goals play a powerful role in computing attentional priority. Attention can be deployed to a particular location when individuals are cued in advance to attend to that location in preparation for an upcoming target (e.g., Posner, 1980). Such deliberate attention allocation can greatly reduce distraction by stimuli outside of the focus of attention (Yantis & Johnston, 1990). Goal-driven attentional control can also operate through the prioritization of stimulus features. Knowledge of the specific features of the upcoming target increases the efficiency of visual search, such that attentional selection can be largely restricted to stimuli that possess a target-defining feature (Wolfe, 1994; Wolfe, Cave, & Franzel, 1989). 

Goal-driven attentional selection is at times voluntary and deliberate, but may proceed rapidly and automatically as well. When the target of visual search is known in advance, stimuli that possess a target-defining feature capture attention. This is referred to as contingent attentional capture (Folk, Remington, & Johnston, 1992). Distractors that share the defining feature of the target selectively produce a spatial cuing effect consistent with attentional selection (Anderson & Folk, 2010, 2012; Folk & Anderson, 2010; Folk et al., 1992; Folk, Leber, & Egeth, 2002; Folk & Remington, 1998). This goal-related selectivity in attentional selection is further supported by eye movement measures (Ludwig & Gilchrist, 2002, 2003) and neurophysiological indices of stimulus processing (e.g., Eimer & Kiss, 2008; Serences et al., 2005; Serences & Yantis, 2007). Contingent attentional capture is adaptive in that it rapidly orients attention to likely targets, facilitating more rapid target localization, at the possible expense of selecting feature-similar nontargets that need to be rejected. It reflects a hybrid form of voluntary and involuntary attentional control—the instantiation of the goal state is voluntary, while the allocation of attention that follows is not. 

Salience-driven and novelty-driven attentional selection

Our goals will not always encapsulate what is relevant to our survival in a given context. Opportunities and dangers that we do not expect may suddenly present themselves, such as a predator leaping out from hiding. In addition, goal-directed attentional control is cognitively demanding to maintain, and individuals experience periodic lapses in the ability to efficiently carry out deliberate visual search (Leber, 2010; Leber, Turk-Browne, & Chun, 2008; Lechak & Leber, 2012). A mechanism of attentional selection that can function independently of ongoing goals would be adaptive provided that the benefits of having attention automatically captured outweigh the cost of potential for distraction from goal-directed processing. 

Attentional selection is influenced by physical stimulus salience. Visual search for a target is slowed by the presence of a physically salient nontarget (Theeuwes, 1991, 1992, 1994, 2010; Yantis & Jonides, 1984), which involuntarily draws eye movements (e.g., Theeuwes, de Vries, & Godijn, 2003; Van der Stigchel & Theeuwes, 2005). Computational models that can account for the search behavior of observers place a strong emphasis on the physical salience of stimuli (e.g., Itti & Koch, 2001). The capture of attention by physically salient stimuli is also supported by neurophysiological measures that show preferential processing of a salient distractor (e.g., Hickey, McDonald, & Theeuwes, 2006). 

The extent to which salience-driven attentional priority can be overridden by goal-driven attentional control is currently a matter of debate. Physically salient stimuli that do not match a currently active target template have consistently failed to produce evidence of attentional capture using both behavioral (e.g., Folk et al., 1992; Folk & Remington, 1998) and neurophysiological measures (Eimer & Kiss, 2008, 2010; Lien, Ruthruff, Goodin, & Remington, 2008). Salient but task-irrelevant stimuli most robustly capture attention when the features of the upcoming target cannot be anticipated (e.g., Bacon & Egeth, 1994; Folk & Anderson, 2010), suggesting that goal-driven attentional control may be capable of gating the influence of salience on attentional selection. However, this position is not without controversy (see Acta Psychologica, 135(2), for a review and commentary), and whether goals or salience play the more dominant role in determining which stimulus is selected via attention remains a contested issue (e.g., Anderson & Folk, 2012; Belopolsky, Schreij, & Theeuwes, 2010; Sawaki & Luck, 2010; Theeuwes, 2010). 

Related to, although distinct from, salience-driven attentional capture is attentional priority for novel stimuli. Stimuli that are encountered less frequently in a particular context capture attention over more familiar stimuli (Horstmann & Ansorge, 2006; Johnston, Hawley, Plewe, Elliott, & DeWitt, 1990; Johnston & Schwarting, 1997; Neo & Chua, 2006). In addition, stimuli that have been selected less frequently in a prior task can also have heightened attentional priority. When participants have prior experience searching for stimuli defined by a particular color in one task, stimuli possessing that color cause less interference than stimuli possessing a different color in a new and unrelated task (Anderson, Laurent, & Yantis, 2012). 

Salient visual events can carry important information concerning potential reward availability or danger. The abrupt appearance of a new object is a highly salient event (Yantis & Hillstrom, 1994), and rapidly assessing this event in order to plan a response, if appropriate, would be beneficial for an organism. Likewise, the potential reward or danger signaled by an unfamiliar stimulus will be less well known to the observer, who may benefit from further scrutiny of the stimulus. The temporary distraction from goal-directed processing caused by attentional capture is often very brief (e.g., Theeuwes, 1991, 1992) and can be contrasted with the potential cost of missing an opportunity to procure an available reward or escape danger, particularly when the opportunity is fleeting. Computational modeling suggests that a bias for attending to salient and novel stimuli can serve in the interest of maximizing overall reward procurement (Laurent, 2008). 

Beyond salience and goals

So far, I have argued that mechanisms for computing attentional priority serve in the interest of maximizing rewards and minimizing losses, and that salience-driven and goal-driven mechanisms of attentional selection are consistent with this framework. But are these two mechanisms sufficient to satisfy the requirements of a system built to secure the survival and wellbeing of an organism? I propose that an adaptive system of attentional selection must also be understood in terms of the influence of prior reward learning. Reward-related stimuli will not always be physically salient, nor will an organism always be searching for a particular reward-related stimulus when it is encountered. An attentional system that only factors goals and salience into the computation of priority for selection is likely to result in missed opportunities to obtain a reward or escape danger, as both goals and salience are only indirectly related to the value of a stimulus. If the computation of attentional priority is truly rooted in maximizing rewards and minimizing losses, this would predict a direct influence of the learned value of stimuli on selection, such that stimuli that have been learned to predict rewards compete for selection regardless of their salience and goal relevance. 

In the section that follows, I review recent evidence for a uniquely value-driven mechanism of attentional selection. I argue that reward learning modifies attentional priority such that previously reward-related stimuli can compete effectively for selection even when nonsalient and task irrelevant. Broadly speaking, attentional priority is both a function of dynamic influences that reflect changes in goals and local feature contrast, as well as more enduring representations of learned stimulus-outcome relationships that independently exert an influence on selection. 

Evidence for a value-driven mechanism of attentional selection

Recent research has identified three distinct ways in which reward influences attentional selection. The first is through the efficiency of selection in visual search, the second is through the priming of previously rewarded stimuli, and the third is through attentional capture driven by reward history—what my colleagues and I have referred to as value-driven attentional capture. I explore each of these value-driven attentional effects here, with particular emphasis on value-driven attentional capture. I also draw reference to related studies on attentional selection involving stimuli with intrinsic ecological value. 

Ecologically pertinent stimuli capture attention

Certain stimuli that are particularly important to promoting survival and wellbeing have been shown to have high attentional priority. For example, socially relevant stimuli capture attention (e.g., a happy face, Hodsoll, Vinding, & Lavie, 2011), as do sexually relevant stimuli (e.g., a naked body, Most, Smith, Cooter, Levy, & Zald, 2007). Although this provides evidence of involuntary attentional capture by valuable stimuli, the mechanisms that underlie such attentional selection are unclear. One potential explanation for the high attentional priority of ecologically pertinent stimuli is that they have acquired learned value through repeated pairings with reward, and it is this representation of learned value that is driving attention. At the same time, however, such effects can also be explained by mechanisms that do not implicate learning. For example, a bias to attend to socially relevant stimuli is evident from infancy (e.g., Farroni, Csibra, Simion, & Johnson, 2002; Farroni et al., 2005; Franz, 1961), consistent with an inherited attentional bias that precedes learning. Because of this ambiguity, the remainder of the studies I review in this section employ methodologies in which the stimulus–reward associations that underlie observed attentional biases are experimentally controlled. 

Reward modulates efficiency of selection in visual search

Selection processes in visual search are influenced by stimulus–reward associations. Selection of a target amongst nontargets is more efficient when the target is associated with the delivery of a reward (e.g., Kiss, Driver, & Eimer, 2009; Kristjansson, Sigurjonsdottir, & Driver, 2010). Even with complex reward structures, individuals are able to allocate attentional priority to different targets in order to maximize total reward procurement (Navalpakkam, Koch, & Perona, 2009; Navalpakkam, Koch, Rangel, & Perona, 2010). When stimulus selection is rewarded, individuals make more efficient use of cue information to restrict attentional processing (Pessoa & Engelmann, 2010). Such effects of reward on attention can be explained by reward operating through the enhancement of goal-directed control, reflecting motivated performance. 

Evidence from human functional magnetic resonance imaging (fMRI), psychophysics, and single unit recording in rats and monkeys further supports the notion that associations with reward modulates the attentional priority of stimuli. The response to a stimulus in early visual areas can be predicted from the recent reward history of that stimulus, and this response adapts to reflect changes in experienced reward (Serences, 2008). Orientation tuning becomes sharper for orientations that are associated with reward outcome (Serences & Saproo, 2010), a process that can proceed even without conscious awareness (Seitz, Kim, & Watanabe, 2009). When a stimulus is learned to predict high reward, it persistently evokes increased activity in the lateral intraparietal area (area LIP; Peck, Jangraw, Suzuki, Efem, & Gottlieb, 2009), and stimulus-evoked activity in area V1 of the visual cortex reflects the anticipated timing of the receipt of reward (Shuler & Bear, 2006). 

Learned stimulus–reward associations have been further shown to affect subsequent attentional selection. In one study, participants learned to associate different faces with different monetary reward outcomes in a training phase, and these faces later appeared as targets in a subsequent attentional blink task. Faces previously associated with high monetary gains or losses were more accurately reported than other faces when presented as the second of two targets, suggesting that they had higher attentional priority (Raymond & O'Brien, 2009). In a similar study reported by Della Libera and Chelazzi (2009), stimuli previously associated with high and low rewards could appear as both targets and nontargets in a subsequent object identification task. Stimuli previously associated with high reward were more rapidly identified as targets and more slowly rejected as distractors, while the opposite was true for stimuli previously associated with low reward (see also Della Libera, Perlato, & Chelazzi, 2011). 

Reward-motivated attentional priorities can also influence the selection of task-irrelevant stimuli in much the same way that task goals influence selection in contingent attentional capture. Participants engaged in a Stroop task in which quickly and accurately naming particular colors was met with a monetary reward. These rewarded colors were named more quickly than other, nonrewarded colors, indicating that attentional priorities for the colors reflected the reward structure (Krebs, Boehler, & Woldorff, 2010). Importantly, words that named a reward-associated color produced greater response time interference than other color-words, which was further supported by associated increases in neural activity in the presupplementary motor area as revealed through human fMRI (Krebs, Boehler, Egner, & Woldorff, 2011). 

Reward primes attentional selection

When a stimulus is selected on a given trial, the selection of that stimulus is facilitated on subsequent trials, a phenomenon referred to as intertrial priming or priming of pop-out (e.g., Belopolsky et al., 2010; Folk & Remington, 2008; Found & Müller, 1996; Kristjansson & Campana, 2010; Maljkovic & Nakayama 1994; Müller, Heller, & Ziegler, 1995; Müller, Reimann, & Krummenacher, 2003; Theeuwes & Van der Berg, 2011). In intertrial priming, recently selected stimuli are more rapidly identified as targets and are more slowly rejected as distractors. Recent evidence shows that intertrial priming is modulated by the extent to which the selection of a stimulus was recently rewarded. 

In the first study to assess reward-modulated priming, Della Libera and Chelazzi (2006, experiment 1) employed a global-local number identification task pioneered by Navon (1977). Participants were shown a large number (global feature) comprised of identical smaller numbers (local features) on a given trial, and were cued in advance on which feature to perform an identity judgment. Participants were randomly given a high or low monetary reward for correctly identifying the cued feature. These were referred as prime trials, each of which was followed by an unrewarded probe trial on which only one of the two feature judgments could be performed (e.g., a large 6 made up of small x's). Following a high reward, response time was faster on probe trials when the judgment from the prime trial was repeated and slower when the judgment switched, consistent with intertrial priming of the rewarded feature. This pattern was reversed, however, following the receipt of a low reward, suggesting that participants were biased against repeating the same judgment in this case. A follow-up experiment reported the same reward-dependent pattern of priming when the color of stimuli was primed in an object discrimination task (Della Libera & Chelazzi, 2006, experiment 2). 

In another important study on reward-modulated priming, Hickey, Chelazzi, and Theeuwes (2010a) employed a paradigm in which participants performed visual search for a shape singleton target (e.g., diamond among circles). On some of the trials, one of the nontarget shapes was presented in a unique color, which served as a physically salient singleton distractor (e.g., a red shape among green shapes). Participants were randomly given a high or low reward for correctly identifying the shape singleton target on a given trial. Critically, the color of the target and distractor could either be repeated or swapped on the subsequent trial, allowing for the assessment of intertrial priming of color and its relation to prior reward. The results were very similar to those reported by Della Libera and Chelazzi (2006) in that priming was again dependent on reward feedback—large priming effects were observed following high reward that were reversed or eliminated following low reward. The magnitude of the extent to which reward modulated priming in this way was well predicted by reward-evoked activity in anterior cingulate cortex (Hickey et al., 2010a). 

A follow-up study revealed that the magnitude of reward-modulated priming across individuals is positively correlated with individual reports of the extent to which reward motivates behavior (Hickey, Chelazzi, & Theeuwes, 2010b). Thus, individuals who are the most motivated by rewards are also the most influenced by recent reward history in visual search. Reward modulated priming was also shown to be specific to stimuli whose selection as a target was associated with reward (Hickey, Chelazzi, & Theeuwes, 2011) and is evident in eye movement trajectories in addition to covert attention allocation (Hickey & van Zoest, 2012). 

Accounts of attentional control have been forwarded that explain distractor suppression in terms of an individual's motivation to suppress visual features (e.g., Geyer, Müller, & Krummenacher, 2008; Moher, Abrams, Egeth, Yantis, & Stuphorn, 2011; Müller, Geyer, Zehetleitner, & Krummenacher, 2009). One possible account of reward-modulated priming is that reward feedback dynamically adjusts an individual's incentive to suppress the prior target feature on a trial-by-trial basis, leading to more or less attentional capture as a function of prior reward. There are several aspects of reward-modulated priming that suggest it is to some degree independent of such voluntary influences on attention. Reward-modulated priming occurs despite the fact that the rewards are completely unrelated to task performance, a design element that participants are informed of (Hickey et al., 2010a, 2010b, 2011). Furthermore, reward-modulated priming persists even when observers are provided with direct incentive to search for the nonprimed feature on the following trial (Hickey et al., 2010a), suggesting that reward-modulated priming is resistant to countermanding goal-driven attentional control. Brain circuits involved in reward processing and incentive motivation are believed to be interrelated (e.g., Berridge, 2012; Berridge & Robinson, 1998); this suggests that the experience of reward may modulate the incentive salience of a stimulus in an automatic fashion. 

Previously reward-associated stimuli involuntarily capture attention

Up to this point, the extent to which attentional priority is modulated specifically by learned stimulus–reward associations is unclear and difficult to assess. The receipt of reward clearly modulates stimulus priming, but such an effect cannot be attributed to a learned association. In reward-modulated priming, attentional priority is biased by the actual receipt of reward (Hickey et al., 2010a), such that the attentional priority of reward-related stimuli constantly fluctuates based on their most recent reward history (Della Libera & Chelazzi, 2006; Hickey et al., 2010a, 2010b, 2011). 

Although the effects of stimulus–reward associations on search efficiency are consistent with an automatic and involuntary effect of learned value on selection, they may also reflect the contribution of reward motivating the voluntary, goal-directed control of attention. Attending to reward-related stimuli and procuring rewards often represent explicit goals of the task in studies on reward and attention (Maunsell, 2004). Indeed, in all of the aforementioned studies on how reward modulates selection in visual search, the reward-associated stimuli (a) could appear as a sought target or otherwise provided information about the target and/or (b) could appear when performance was still motivated by currently available rewards. One possibility, then, is that the influence of learned stimulus—reward associations on attentional priority is subsumed entirely within the domain of voluntary, goal-driven attentional control. In this sense, reward merely acts to strengthen goal-directed attentional control through mechanisms of motivation. By such an account, learned stimulus–reward associations have no specific role in modulating attentional priority. 

In a series of studies, my colleagues and I set out to directly test whether stimuli that have been previously associated with reward through learning involuntarily capture attention (indicating that they have high attentional priority) even when entirely task-irrelevant and when rewards are no longer available. We used arbitrary stimuli and experimentally controlled the amount of reward learning, rather than using stimuli with intrinsic ecological value. In doing so, we compared the attentional priority of irrelevant stimuli that differed only in their prior history with reward. Attentional capture by such previously rewarded stimuli would be uniquely consistent with a value-driven mechanism of computing attentional priority that is independent of salience-driven and goal-driven attention mechanisms. 

The value-driven attentional capture paradigm

To study how reward learning modifies subsequent attentional priority, my colleagues and I developed a paradigm originally reported in Anderson, Laurent, and Yantis (2011b) that has been modified in several different studies to address interrelated questions concerning how reward learning modifies attentional priority. In this paradigm, all participants begin the experiment by completing a training phase in which they search for a target circle defined by one of two colors; each color is equally likely to serve as the target color on a given trial (Figure 1A). Each target is presented among five nontarget circles, each of a different color, thus requiring visual search. Participants report the orientation of a bar contained within the target circle and receive a monetary reward provided that they respond correctly within the allotted time. After each correct response, reward feedback is provided in which a small amount of money is added to a running total that participants are told they will be paid at the end of the experiment. Importantly, the amount of monetary reward received is related to the color of the target. One target color (counterbalanced across participants) is followed by a comparatively high reward on 80% of the trials on which it is correctly reported, and by a comparatively low reward on the remaining 20%; for the other target color, these mappings are reversed. Thus, participants learn two things in the training phase: (a) that each of the target colors is associated with the delivery of a monetary reward and (b) that one target color tends to be followed by a greater amount of reward than the other on average. 

Figure 1

 

Example displays from the training phase and test phase of paradigms used to assess value-driven attention capture. (A) Training phase with reward feedback. (B) Test phase with nonsalient distractors. (C) Test phase involving a different task. (D) Test phase with physically salient distractors. (E) Training phase without reward feedback. Each trial was followed by a blank intertrial interval. The number of trials, the magnitude of reward, the duration of each display, and the position of the colored stimuli varied across different experiments. ISI: interstimulus interval.

Following the training phase, participants subsequently perform a test phase consisting of unrewarded visual search for a physically salient shape singleton target (e.g., a circle among diamonds, as in Theeuwes, 1992). The search array again consists of six shape stimuli, and previously reward-associated color stimuli occasionally appear as irrelevant nontargets. Participants are instructed to ignore color and focus on identifying the unique shape; the previously reward-related color stimuli are entirely irrelevant to the task. Of interest is whether these valuable distractors show evidence of attentional capture that is attributable to prior reward learning. 

Valuable stimuli are distracting even when nonsalient and task-irrelevant

Following our training phase, we had participants complete a test phase in which each of the six shapes in the search array were presented in a different color (Anderson et al., 2011b; see Figure 1B). The results showed that the previously reward-associated distractors slowed visual search, mirroring the effects of a physically salient distractor (e.g., Theeuwes, 1991, 1992; see Figure 2A and B). Analysis of response time distributions showed that this slowing occurred on even the fastest response trials, suggesting that the valuable distractors consistently had a sufficiently high attentional priority to compete with the target for selection. 

Figure 2

 

Value-driven attentional capture, reflected in the slowing of response time by the presence of previously reward-associated distractors. (A) Results from experiment 1 of Anderson et al. (2011b), following 1,008 trials of training. (B) Results from experiment 3 of Anderson et al. (2011b), following 240 trials of training. (C) Results from experiment 1 of Anderson and Yantis (2012), following 300 trials of training. (D) Results from Anderson and Yantis (2013), following participation in A–C.

In this experiment, the valuable distractors are only identifiable on the basis of prior reward history. This is an important aspect of the design. The colors we selected for the distractors were not intrinsically more salient than the colors in which the other stimuli were rendered (Anderson et al., 2011b). And because the target is a shape singleton, it is the most physically salient item in the display. Therefore, both goal-driven and salience-driven attentional priority should be the highest for the target under these conditions, competing with the previously reward-associated distractors for attention. The fact that these distractors involuntarily captured attention in spite of this competition demonstrates that value-driven attentional capture does not depend on salience-driven or goal-driven mechanisms of selection, reflecting an independent mechanism of selection driven specifically by reward history. 

Evidence for the value-driven capture of spatial attention

The aforementioned results are consistent with the capture of spatial attention by previously reward-associated but currently irrelevant stimuli. Another possibility, however, is that the valuable distractors were creating a nonspatial form of distraction, such as a filtering cost (e.g., Treisman, Kahneman, & Burkell, 1983) or interference caused by an increase in arousal. We have provided two sources of evidence that definitively show that the previously observed distraction is indeed spatial in nature. 

The first source of evidence comes from an analysis reported in the same study (Anderson et al., 2011b). The test phase we employed in this paradigm used a rapid-trial design, allowing for the potential measurement of inhibition-of-return (IOR). When attention is withdrawn from a stimulus, subsequent inhibition of the formerly attended location can be observed, which is believed to serve in the interest of preventing repetitive selection patterns (Posner, Rafal, Choate, & Vaughan, 1985). We showed that following distractor present trials, participants were slower to report the target when it appeared at the position occupied by the distractor on the previous trial than when it appeared in a different position (Anderson et al., 2011b), consistent with lingering IOR indicative of prior attentional capture by the distractor. 

Compelling evidence that previously reward-associated stimuli capture spatial attention was reported in a study in which we measured eye movements in addition to manual responses (Anderson & Yantis, 2012). The experimental design was very similar to Anderson et al. (2011b), with both the training and test phases consisting of visual search amongst multicolored shapes (see Figure 1A and B). It is well understood that shifts of covert attention precede and direct eye movements (e.g., Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995), such that eye movements occur to the location of highest attentional priority (e.g., Thompson & Bichot, 2005). In addition to replicating the behavioral effect of response time slowing in this study (Figure 2C), we showed that the valuable distractors were substantially more likely to be fixated than other nontargets (Anderson & Yantis, 2012). This value-driven oculomotor capture occurred despite the fact that eye movements were neither required nor encouraged by the task, suggesting that eye movements to previously reward-associated stimuli occur naturally and are difficult to suppress. Similar results were subsequently reported by Theeuwes and Belopolsky (2012) using salient distractors, who additionally showed significantly greater oculomotor capture for previously high-value compared to previously low-value distractors. 

Relation to visual working memory capacity and impulsiveness

Visual working memory (VWM) capacity refers to the amount of visual information that can be maintained over brief periods of time with minimal or no contribution of verbal memory. VWM capacity is typically expressed in terms of the number of objects an individual can maintain in memory, which has an upper limit of approximately four (Awh, Barton, & Vogel, 2007; Luck & Vogel, 1997). However, considerable individual differences exist in the number of items that can be maintained in VWM. Seminal work by Fukuda and Vogel (2009, 2011) demonstrates that VWM capacity is negatively correlated with measures of top-down control over attentional selection—individuals with relatively low VWM capacities are less able to restrict attentional selection to only target-relevant items and are thus more distracted by irrelevant stimuli. 

In our own research, the magnitude of distraction caused by previously reward-associated but currently irrelevant stimuli similarly correlates negatively with VWM capacity. This is true for both the slowing in response time (Anderson et al., 2011b) as well as the likelihood of eye movements toward the valuable distractors (Anderson & Yantis, 2012); individuals with lower VWM capacities are especially distracted by reward-associated stimuli. These findings dovetail nicely with those of Fukuda and Vogel (2009, 2011) and provide further evidence that value-driven attentional capture reflects an involuntary effect of attentional selection rather than a voluntary reward-based strategy—individuals who are less able to efficiently exert voluntary control over attentional selection are less able to overcome distraction by irrelevant stimuli of learned value. 

Another factor that has been shown to co-vary with the magnitude of value-driven attentional capture is trait impulsiveness. Trait impulsiveness broadly refers to the extent to which individuals are able to successfully exercise control over behavior (e.g., Dickman & Meyer, 1988; Patton, Stanford, & Barratt, 1995). We measured trait impulsiveness using a standardized questionnaire (Barratt Impulsiveness Scale; Patton et al., 1995) and showed that individuals who report being more impulsive are particularly slowed by previously reward-associated distractors following extended training (Anderson et al., 2011b). This is further consistent with the idea that value-based attentional priority reflects an automatic process that competes with goal-directed processing, such that individuals who evidence greater capacity for exercising executive control are better able to overcome reward-related influences on attention. 

Generalizability of reward learning in driving attention

One important question concerning the learning that underlies value-driven attentional priority is which aspects of a stimulus become associated with reward. One possibility is that the reward learning that underlies value-driven attentional priority is narrow in scope, such that it is specific to a particular object. This level of specificity is typical of perceptual learning (e.g., Kyllingsbaek, Schneider, & Bundesen, 2001; Shiffrin & Schneider, 1977). Another possibility, however, is that the underlying reward learning is more flexible, such that novel stimuli sharing a previously reward-associated feature will have increased attentional priority. By such a mechanism of attentional control, prior reward learning can be broadly applied to new encounters with stimuli in order to inform selection, maximizing gains accordingly. 

Earlier evidence from our original value-driven capture experiments suggested that value-driven capture can occur for stimuli that are not an exact match to the formerly rewarded targets. In our test phase, the distractor could be either a diamond or a circle, whereas during training the reward-associated stimulus had always been a circle (Anderson et al., 2011a, 2011b; Anderson & Yantis, 2012; see Figure 1B). However, the training and test phases were very similar in many respects, involving visual search among differently colored outline shapes. To test the generalizability of reward learning in modifying attentional priority, we used a very similar training phase as in a prior experiment (Anderson et al., 2011b, experiment 3; see Figure 1A) and introduced a test phase involving a different task with different color stimuli (Figure 1C). We had participants perform a flankers task in which the irrelevant flanking letters could share the same color as the formerly reward-associated targets from training (Anderson et al., 2012). Our results showed larger flanker compatibility effects for flankers that shared the color of a formerly high-reward target, despite the fact that rewards had never been associated with letters and participants had only experienced rewards in the context of visual search (and not in a focused attention task like the one used in the test phase). 

The outcome from this experiment demonstrates that prior reward learning has a broad influence on subsequent computations of attentional priority that extends to newly encountered stimuli and contexts. Whether value-driven attentional priority is sensitive to object-level information has never been explicitly tested; however, even a purely feature-based account would predict some degree of enhanced attentional priority for a complete match to the previously rewarded stimulus if the representation of each reward-associated feature is independently biased. What the aforementioned findings make clear is that the presence of a previously reward-associated feature is sufficient to modify the attentional priority of even a novel stimulus, reflecting generalization of reward learning. 

Relation to physical stimulus salience

Our original demonstration of value-driven attentional capture used nonsalient distractors in the test phase (Anderson et al., 2011b; see Figure 1B). In that series of experiments, we showed that nonsalient stimuli are essentially processed as if they have increased physical salience when previously associated with reward. In a subsequent study, we investigated how this value-driven increase in attentional priority relates to the physical salience of stimuli. 

One possibility is that learned value influences attention through a purely selection-based mechanism, ensuring that reward-associated stimuli “stand out” and are selected first. By such a mechanism of attentional control, learned value only influences selection when the reward-associated stimulus does not already stand out on its own due to its physical salience. Another possibility is that learned value directly modifies attentional priority independently of selection-based processes, essentially increasing the experienced salience of reward-associated stimuli (see Thompson & Bichot, 2005). 

We adjudicated between these two possible accounts of the mechanisms underlying value-driven attentional priority by employing the same training phase as in Anderson et al. (2011b; see Figure 1A) and including a test phase in which the previously reward-associated distractors were the most physically salient stimulus in the display (Anderson et al., 2011a; see Figure 1D). Under these conditions, the previously reward-associated stimuli have priority for selection over the target purely on the basis of their physical salience (e.g., Theeuwes, 1991, 1992, 2010). If learned value acts only to ensure that previously reward-associated stimuli are preferentially selected via attention, attentional capture by the distractor should not be affected by whether it was previously associated with high or low reward. However, if learned value influences attentional priority by increasing the experienced salience of previously reward-associated stimuli regardless of priority for selection, learned value should influence attentional priority above and beyond physical salience such that attentional capture is more robust for previously high-value compared to previously low-value stimuli. Our results showed that physically salient stimuli previously associated with high reward slowed visual search to a greater extent than physically salient stimuli previously associated with low reward, despite the fact that both robustly captured attention independently of their reward history (Anderson et al., 2011a). 

Rate of reward learning in modifying attentional priority

It is interesting to note how quickly stimulus-reward associations can be learned such that they modify attentional priority. Early studies on reward and attention used long training phases spanning several days (Della Libera & Chelazzi, 2009), as is typical in studies of perceptual learning (e.g., Kyllingsbaek et al., 2001; Shiffrin & Schneider, 1977). In our first experiments on reward and attention, we used a training phase with only 1,008 trials (Anderson et al., 2011a, experiment 1; Anderson et al., 2011b, experiment 1). Later experiments demonstrated robust value-driven attentional capture with as little as 300 (Anderson & Yantis, 2012) and 240 (Anderson et al., 2011b, experiment 3; Anderson et al., 2012) trials. Unpublished data from our lab even shows significant value-driven attentional capture in only 144 trials. 

In all of our studies, the magnitude of distraction caused by the reward-associated stimuli has been very similar regardless of the length of training (see Figure 2). This suggests two things about value-driven attentional priority: (a) Stimulus-reward associations can be learned very rapidly compared to other forms of experimental learning such as perceptual learning and (b) once learned, these stimulus-reward associations have a powerful influence on attentional priority that quickly approaches asymptote such that further reinforcement does little if anything to strengthen their influence on attention. It is worth noting, however, that even 144 trials can be thought of as a large amount of learning experience when extrapolated to everyday encounters with stimuli—for example, we can learn whether we like a particular food after only a few instances of consuming it. It would be interesting for future research to track the influence of reward learning on involuntary attentional control as the reward learning unfolds; for example, it is unclear whether the effects of reward learning on attentional priority are gradual or reflect more of a categorical shift once the learning is sufficiently strong. 

Persistence of reward learning in driving attentional selection

Another interesting question concerns the persistence of reward learning in the computation of attentional priority. There are at least two possible mechanisms by which memory representations of learned stimulus—reward associations could influence attention. The first is a mechanism by which the link between stimulus—reward associations in memory and the attentional control system must be maintained through reinforcement. In essence, value-driven attentional priorities reflect only what has recently proven useful in maximizing rewards and eventually extinguish in the absence of reward feedback. Another possibility is that learned stimulus—reward associations have a direct influence on attentional priority that will remain intact as long as the memory is maintained, which can be indefinitely. 

In most of our prior studies, value-driven attentional capture has been assessed immediately following training (Anderson et al., 2011a, 2011b, 2012; Anderson & Yantis, 2012). In our original report of value-driven attentional capture, we showed that reward learning can influence attentional priority even several days following training (Anderson et al., 2011b), but such a finding could be consistent with either of the two aforementioned mechanisms. To definitely distinguish between these mechanisms, we recruited participants to repeat the test phase 7–9 months following initial participation in one of our reward learning studies. If the connection between the previously learned stimulus–reward associations and the attentional control system must be maintained through reinforcement in order for reward learning to have a continued influence on attention, the stimulus–reward associations experienced by our participants in prior studies should have no effect on attentional capture. Despite no explicit memory for the previously experienced stimulus–reward contingencies, the results showed that prior reward learning continued to exert a robust influence on attentional selection even over half a year later, without any intervening reinforcement (Anderson & Yantis, 2013; see Figure 2D). This provides strong evidence for a mechanism of attentional control by which memory representations of the associated reward value of stimuli directly influence attention, reflecting persistent changes in how the brain represents stimuli following reward learning. 

Distinction between reward history and search history

In our paradigm, participants spend the training phase searching for particular color stimuli. One possible explanation for our results is that this search history alone is sufficient to create persistent changes in attentional priority, without any contribution of reward learning. Persistent attentional biases for former targets have been reported following extended visual search experience, often spanning several thousand trials over multiple days (e.g., Kyllingsbaek et al., 2001; Shiffrin & Schneider, 1977). Such former-target biases occur without any explicit reward feedback. 

There are several sources of evidence that argue against a former-target bias as an explanation for our results. Our training phases have included as little as 240 trials—far less than is required to produce robust effects of perceptual learning (Kyllingsbaek et al., 2001; Shiffrin & Schneider, 1977). Such perceptual learning is also typically very stimulus specific—for example, for a particular letter among other letters, value-driven capture occurs for different shapes (e.g., red diamond when the red target was previously a circle, Anderson et al., 2011a, 2011b; Anderson & Yantis, 2012, 2013) and even for novel stimuli (e.g., red letter when the target was previously a geometric shape, Anderson et al., 2012). Furthermore, value-driven attentional capture can occur in the absence of explicit memory for the learned stimulus-reward associations (Anderson & Yantis, 2013). 

To definitively rule out a former-target bias as an explanation for value-driven attentional capture, we ran experiments using otherwise identical training phases that did not include the reward feedback component (Figure 1E). Participants spent just as many trials searching for two color targets, which later served as nontarget distractors in a subsequent test phase (Figure 1B and D). Without the reward feedback and consequent reward learning, the effect of the distractors was abolished (Anderson et al., 2011a, 2011b). This result demonstrates that reward learning is necessary to produce the observed attentional capture in our paradigm. 

A note about attention and punishment learning

In the present review, I focus on the contribution of reward learning to the computation of attentional priority. It is worth noting that the conceptual framework I present also predicts that stimuli that have been learned to predict negative outcomes should also capture attention according to the same value principle. Such punishment-driven attentional priority is beyond the scope of this review, although I will briefly draw reference to some early evidence suggesting that punishment history also biases attention. Fear-evoking stimuli have been shown to capture spatial attention (Kennedy & Most, 2012; Most, Chun, Widders, & Zald, 2005; Most & Wang, 2011) in much the same way that erotic images do (Most et al., 2007). Furthermore, stimuli that predict an aversive white noise burst capture spatial attention (Koster, Crombez, Van Damme, Verschuere, & De Houwer, 2004) and impair target report in an attentional blink task (Smith, Most, Newsome, & Zald, 2006), although the extent to which this effect persists when the stimulus–sound associations no longer hold is unknown. Further research will be needed to more firmly establish a unique role for punishment and aversive conditioning in driving attention allocation. 

Value-driven attentional priority and theories of selection history

Recent theories of attentional control emphasize the importance of selection history, broadly construed, in determining priority for selection. Awh, Belopolsky, and Theeuwes (2012) argue that prior selection of a stimulus can modify its attentional priority in future encounters, in ways that are not well accounted for by either salience-based or goal-based models of attention. Such selection history can include reward-based effects, in addition to phenomena such as priming (e.g., Found & Müller, 1996; Maljkovic & Nakayama, 1994; Müller et al., 1995; Müller et al., 2003), contextual cuing (e.g., Chun & Jiang, 1998; Jiang, Swallow, Rosenbaum, & Herzig, 2013), and perceptual learning (e.g., Kyllingsbaek et al., 2001; Shiffrin & Schneider, 1977). Similarly, Hutchinson and Turk-Browne (2012) argue that memory representations of stimuli and context play an important role in attentional selection. For example, attention is automatically biased to aspects of a scene that contain previously experienced statistical regularities (Zhao, Al-Aidroos, & Turk-Browne, in press), and the contents of working memory can automatically guide attention even in the presence of countermanding goals (e.g., Olivers, Meijer, & Theeuwes, 2006; Olivers, Peters, Houtkamp, & Roelfsema, 2011). 

Value-driven attentional capture fits broadly within these frameworks, and demonstrates a specific role of prior reward in defining selection history. When previously associated with reward, former targets take on an ability to capture attention that cannot be explained by reward-independent selection history, as such biases depend on the magnitude of prior reward and are not evident when selection of the targets goes unrewarded (Anderson et al., 2011a, 2011b, 2012; Anderson & Yantis, 2013). This contrasts with other memory-based and selection-history effects on attention described above, which are known to occur in the absence of explicit reward feedback. One possibility is that value-driven attentional capture is independent of these other influences of selection history, reflecting a uniquely reward-based mechanism of learning. Another possibility is that the receipt of reward amplifies the influence of other mechanisms of selection history in a more indirect sense, with value-driven attentional capture reflecting more rapid and powerful perceptual learning or memory-based distraction. A more detailed and integrated account of how reward feedback factors into selection history will likely become apparent as research on reward learning and attentional control progresses. 

Conclusions

I began this review by providing a conceptual framework for the computation of attentional priority that emphasizes the need for stimulus selection to promote the survival and wellbeing of an organism. Evidence in support of this conceptual framework was then provided, which centered on findings that stimuli previously associated with reward capture spatial attention. I argue that learned associations between stimuli and reward have a direct and automatic influence on the computation of attentional priority that is independent of salience-driven and goal-driven attention mechanisms. 

When stimuli are learned to predict reward, these stimuli gain a competitive advantage in perception that promotes selection even when they are nonsalient and task-irrelevant. This value-based attentional priority can be persistent, being robust to extinction in the absence of available rewards. Furthermore, value-based attentional priority can generalize to other stimuli and contexts, promoting the application of prior learning to new situations. Certain individuals appear to be more susceptible to value-driven attentional capture than others as a function of VWM capacity and trait impulsiveness. A better understanding of the nature of how reward learning creates more enduring changes in attentional priority may be important for furthering our understanding of clinical conditions in which persistently disordered patterns of attention allocation have been implicated, such as addiction (e.g., Berridge, 2012; Field & Cox, 2008; Lubman, Peters, Mogg, Bradley, & Deakin, 2000; Marissen et al., 2006), attention deficit/hyperactivity disorder (e.g., Castellanos & Proal, 2012), and autism (e.g., Sasson, Elison, Turner-Brown, Dichter, & Bodfish, 2011; Sasson, Turner-Brown, Holtzclaw, Lam, & Bodfish, 2008). 

Acknowledgments

This research was supported by NRSA F31-DA033754 to B. A. A. 

Commercial relationships: none. 

Corresponding author: Brian A. Anderson. 

Email: [e-mail]. 

Address: Department of Psychological & Brain Sciences, Johns Hopkins University, Baltimore, MD, USA.