Stimulus competition mediates the joint effects of spatial and feature-based attention

Abstract

Distinct attentional mechanisms enhance the sensory processing of visual stimuli that appear at task-relevant locations and have task-relevant features. We used a combination of psychophysics and computational modeling to investigate how these two types of attention—spatial and feature based—interact to modulate sensitivity when combined in one task. Observers monitored overlapping groups of dots for a target change in color saturation, which they had to localize as being in the upper or lower visual hemifield. Pre-cues indicated the target's most likely location (left/right), color (red/green), or both location and color. We measured sensitivity (d′) for every combination of the location cue and the color cue, each of which could be valid, neutral, or invalid. When three competing saturation changes occurred simultaneously with the target change, there was a clear interaction: The spatial cueing effect was strongest for the cued color, and the color cueing effect was strongest at the cued location. In a second experiment, only the target dot group changed saturation, such that stimulus competition was low. The resulting cueing effects were statistically independent and additive: The color cueing effect was equally strong at attended and unattended locations. We account for these data with a computational model in which spatial and feature-based attention independently modulate the gain of sensory responses, consistent with measurements of cortical activity. Multiple responses then compete via divisive normalization. Sufficient competition creates interactions between the two cueing effects, although the attentional systems are themselves independent. This model helps reconcile seemingly disparate behavioral and physiological findings.

Introduction

Our ability to see depends on what we are looking for and where we expect it to be. Even without changes to the retinal input, neural responses and behavioral sensitivity depend on beliefs about the task relevance of particular visual field locations and visual features. These effects are attributed to covert spatial attention and feature-based attention, respectively. They minimize the expenditure of metabolic resources on the profusion of inconsequential sensory input while strengthening the internal representations of what is relevant for the task at hand (reviewed by Carrasco, 2011, 2014; Maunsell & Treue, 2006; Scolari, Edward, & Serences, 2014). 

Endogenous spatial attention (SA) can be manipulated experimentally by pre-cues that indicate the peripheral location that is most likely to be relevant for a subsequent perceptual decision. In general, stimuli at cued locations evoke stronger responses in the cortex and are perceived more accurately than stimuli at uncued locations. These SA effects often reflect increases in contrast sensitivity or spatial resolution, which correspond to modulations in the response properties of cortical visual neurons (see reviews by Anton-Erxleben & Carrasco, 2013; Carrasco, 2011; Carrasco & Barbot, 2015). 

Feature-based attention (FBA) is the prioritization of items that have a particular feature value within a dimension, such as a color, an orientation, or a direction of motion. FBA can be manipulated by pre-cues that indicate the most likely feature value of a task-relevant stimulus. Some early studies concluded that cueing a feature does not directly improve processing of stimuli with that feature but only guides SA to their locations (e.g., Moore & Egeth, 1998; Shih & Sperling, 1996). However, we now know that FBA strengthens visual processing even when target stimuli are spatially superimposed with distractors (e.g., Alais & Blake, 1999; Lankheet & Verstraten, 1995; Liu, Larsson, & Carrasco, 2007; Liu, Stevens, & Carrasco, 2007; White & Carrasco, 2011). Moreover, the effects of FBA spread globally across the visual field, even to irrelevant or unattended locations (e.g., Andersen, Hillyard, & Müller, 2013; Liu & Mance, 2011; Rossi & Paradiso, 1995; Sàenz, Buraĉas, & Boynton, 2002; Serences & Boynton, 2007; Störmer & Alvarez, 2014; Treue & Martínez-Trujillo, 1999; White & Carrasco, 2011). 

FBA are SA are therefore clearly distinct. Our question is how they relate, what mechanisms they share, and how they jointly determine the quality of perception. These specific issues are related to the more general question of whether the top-down control of attention relies on a unified executive system or multiple parallel systems that prioritize different attributes of sensory inputs. A unified system would be more likely to give special priority to stimuli with conjunctions of relevant attributes. 

The interaction of SA and FBA

SA and FBA have usually been studied separately, but in natural vision, they often operate simultaneously. Are SA and FBA independent systems, or does the effect of each depend upon the other? Consider a basketball player who uses FBA to monitor opponents in red jerseys moving among her teammates in green. She could also use SA to track a particular opponent coming from the right side. Knowing both the color and the side, does she get the sum of both attentional benefits? If she is wrong about the location and the opponent appears off to the left, does she nonetheless get the full benefit of attending to the correct color? She would not if the two forms of attention interact superadditively or are deployed jointly. 

Physiological studies have measured sensory responses in visual cortex to directly tap the interaction of FBA and SA at particular stages of visual processing. Single-neuron responses in macaque areas MT and V4 increase in magnitude for stimuli at attended locations and with attended features (color, form, or motion direction). The two attentional effects on firing rates appear to be independent and additive (Hayden & Gallant, 2009; Patzwahl & Treue, 2009; Treue & Martínez Trujillo, 1999), with perhaps a small multiplicative component (Hayden & Gallant, 2009). Early electroencephalography (EEG) responses (specifically, steady-state visual evoked potentials) show a similar pattern of independence between the effects of cueing location and cueing color (Andersen, Fuchs, & Müller, 2011). These physiological results suggest that FBA and SA have independent top-down influences on the strength of visual encoding. However, later scalp event-related potentials (ERPs) starting 200–300 ms poststimulus show superadditive interactions, with stronger spatial enhancements for attended features (Andersen et al., 2011; Bengson, Lopez-Calderon & Mangun, 2012; Handy, Green, Klein, & Mangun, 2001; Hillyard & Münte, 1984). 

Like the relatively late ERPs, behavioral experiments with visual discrimination tasks have revealed superadditive interactions between cueing effects. These studies typically have a 2 × 2 design, crossing valid and invalid spatial cues with valid and invalid feature cues. Only one such study measured discrimination accuracy, albeit with an indirect measure of FBA that was mediated through a spatial attention capture effect (Leonard, Balestreri, & Luck, 2015). The remainder have used reaction times (RTs) as the primary measure. RTs are faster at the validly pre-cued location but more so for stimuli with expected (pre-cued) features than unexpected features (Bengson et al., 2012; Bengson & Mangun, 2011; Handy et al., 2001; Kingstone, 1992; Lambert & Hockey, 1986; but see Egner et al., 2008). The initial conclusion was that spatial and feature information interact, such that the “spotlight” of SA “fails” to illuminate unexpected features (Klein & Hansen, 1987). 

From these behavioral studies, we may conclude that SA and FBA attention are not independent systems but are deployed conjointly. However, modulations of RTs alone could reflect modulations of processing speed, signal strength, or response criterion (Carrasco & McElree, 2001; Carrasco, McElree, Denisova, & Giordano, 2003; McElree & Carrasco, 1999; McElree & Dosher, 1989; Ratcliffe, 1978; Wickelgren, 1977). False alarm rates in a detection task show a similar interaction (Andersen et al., 2011) but are also susceptible to criterion effects. Therefore, decision-related processes at a later stage could obscure the independent attentional effects that are visible in early visual responses (Bengson et al., 2012; Handy et al., 2001; Klein & Hansen, 1990). How or why this happens has not been made clear. 

The present study

In summary, there is confusion about the interaction of SA and FBA, with different measures contradicting each other. Behavioral measures suggest a superadditive interaction, whereas early cortical responses show independence. To establish a linking hypothesis between perception and neural responses, we developed a psychophysical paradigm to measure the joint effects of SA and FBA on visual sensitivity and fit the data with a novel computational model of cortical processing. Observers viewed four groups of dots, one red group superimposed with one green group on both sides of a central fixation point (Figure 1a). The task was to report the location (upper vs. lower half of the display) of a target saturation increment that occurred briefly in one quadrant of one of the four dot groups. We manipulated the observers' spatial and FBA with pre-cues (colored lines at trial's start; Figure 1c) that indicated the most likely hue and/or side of the target. We measured visual sensitivity (d′) for every combination of the location cue and the color cue, each of which could be valid (60%), neutral (20%), or invalid (20% of trials; Figure 1d). Therefore, we evaluated the effect of spatial cueing for attended colors, unattended (or less attended) colors, and with equal attention to both colors. Conversely, we evaluated the effect of color cueing at the attended location, at the unattended (or less attended) location, and with equal attention to both locations. The information in the spatial cue (left vs. right side of the display) and the color cue (red vs. green dots) was therefore orthogonal to what the observer had to report (target in the upper vs. lower half of the display). 

Figure 1

 

Stimuli and design. (a) Example dot stimuli for Experiment 1, with four simultaneous saturation increments and (b) for Experiment 2, with a single target saturation increment in the upper right red dots. (c) Trial sequence. Numbers below each segment indicate duration in milliseconds. (d) The cues presented at fixation. (e) Proportions of trials in each cue-validity condition.

In general, the internal mechanisms that account for attentional pre-cueing effects depend on the presence of competing distractor information (potential target stimuli) along with the target stimulus change. When stimulus competition is low, cueing effects may represent primarily signal enhancement during stimulus encoding. Signal enhancement is an increase in the signal-to-noise ratio of the target's representation, as is reflected in attentional effects on neural responses in visual cortex. When stimulus competition is high, however, cueing effects may additionally reflect distractor exclusion from the capacity-limited processes of encoding, short-term memory, or decision making (Cameron, Tai, & Carrasco, 2002; Carrasco, Penpeci-Talgar, & Eckstein, 2000; Carrasco, Williams, & Yeshurun, 2002; Eckstein, Peterson, Pham, & Droll, 2009; Foley & Schwartz, 1998; Lu & Dosher, 2000; Pestilli & Carrasco, 2005; Sperling & Dosher, 1986). Accordingly, we measured the cueing effects and their interaction with high (Experiment 1) and low (Experiment 2) stimulus competition. In Experiment 1, four saturation increments occurred simultaneously, one in each of the four dot groups. The other three were potential target changes that acted as competing distractors. In Experiment 2, the task was the same, but there were no distractor saturation changes: Only the target dot group had a saturation increment. In both experiments, a post-cue at the end of the trial indicated the side and color of the target saturation change to be localized. The post-cue's purpose was to equate—across all trials—uncertainty as to the target's side and color, so that the observer could make a decision based on the appropriate sensory evidence, even when the pre-cue was invalid. 

The interaction between cueing effects depended on the presence of distractor events simultaneous with the target change. To account for our data, we present a novel model of cortical stimulus processing that simulates independent, additive attentional modulations on visual neurons, combined with the canonical cortical computation of normalization (Carandini & Heeger, 2012). The model reveals that competition between neural representations can drive interactions of fundamentally independent attentional mechanisms. 

Experiment 1

Methods

Participants

Nine volunteers¹ participated in Experiment 1 (ages 19–44 years; five male). All were naive as to the purposes of the experiment, had normal or corrected-to-normal vision, and were paid for their participation and gave informed consent in accordance with the Declaration of Helsinki. 

Stimulus and task

Observers viewed the stimuli on a color-calibrated CRT monitor (1,280 × 960 resolution; 85-Hz refresh rate). An Apple iMac computer running MATLAB (MathWorks, Natick, MA), with the Psychophysics (Brainard, 1997; Pelli, 1997) and Eyelink (Brainard, 1997; Cornelissen, Peters, & Palmer, 2002) toolboxes, controlled stimulus presentation and response collection. 

The display (Figure 1a) consisted of two arc-shaped apertures filled with a mixture of red and green dots. The dots therefore formed four “fields”: one red field superimposed with one green in the aperture on the left and one red superimposed with one green in the aperture on the right. The diameter of each dot was 0.47 degrees of visual angle (dva), and their density within the apertures was 1.4 dots/dva². Each aperture was outlined in white lines (one pixel) and extended from 4.5 to 11 dva into the periphery and from 50° polar angle above to 50° below the horizontal meridian. A horizontal white line divided each aperture in two. Each arc-shaped aperture was divided into four radial segments but with no visible borders except the horizontal dividing line. We constrained dot placement such that there was always an equal number of red and green dots in each segment. Each dot stayed in any given location for 240 ms before jumping to another location within the same segment. The dots' displacements were all randomly out of phase to prevent a strategy of attending to any particular dot and to mask transients caused by the saturation increments. The monitor was color calibrated and the dot colors were controlled in C.I.E. L*a*b color space, with red and green hues 105° apart. Reds and greens were equal in luminance, and their baseline saturation levels were set to ∼50% of the maximum possible given the monitor's gamut. 

Each trial (Figure 1b) began with an intertrial interval lasting between 495 and 650 ms, during which only the central fixation spot (a 0.1 dva white dot enclosed by a 0.3-dva-diameter white ring), the arc outlines, and horizontal dividing lines were present. Then a pre-cue, formed of one or two line segments (0.75 × 0.25 dva) just to the right and/or left of the fixation dot, appeared for 500 ms (Figure 1c). After a delay of 200 ms, the presentation of the colored dots began. The dots jumped about within their apertures for 300 ms before the saturation increments, which lasted 200 ms. Four saturation increments occurred simultaneously, one in each of the four dot fields (red right, green right, red left, green left; see Figure 1a). Each saturation increment occurred in one quadrant of one arc-shaped group of dots of one color. We selected the particular segment of each dot group that underwent the saturation increment randomly on each trial, with the constraint that two saturation increments never overlapped spatially. 

The dots then returned to their original color for 100 ms before disappearing. Immediately thereafter, a post-cue appeared. The post-cue was a horizontal line segment (0.75 × 0.25 dva), colored either red or green and pointing either right or left, with its inner end just touching the fixation dot (Figure 1c). The post-cue indicated which of the four dot fields was the target. The post-cue's color matched that of the target dots before the saturation increment. 

The observer's task was to report whether the saturation increment in the target dot field was in the upper or lower half of the arc (above or below the dividing white line). A click sound 600 ms after the onset of the post-cue prompted the observer to respond by pressing the up or down arrow on the keyboard (responses before the click were not accepted). The response was not speeded and accuracy was emphasized. A high- or low-pitched feedback tone then indicated whether the response was correct or incorrect. 

The pre-cues indicated the most likely color and/or side of the target saturation change, with 75% validity. They were formed from the same line segments as the post-cues and can be divided into four classes (Figure 1c): 

1.  
   Neutral + neutral: two identical gray lines, one on each side of the fixation mark. These gave no information about color or location.
2.  
   Color only (location neutral): the same two lines but colored either red or green. The target was equally likely to be on the left or right but 75% likely to be in dots with the cue's color.
3.  
   Location only (color neutral): a single gray line pointing to either left or right. The target was equally likely to be red or green but 75% likely to be on the indicated side.
4.  
   Color + location: a single colored line, red or green and pointing left or right. These gave both pieces of information, because the target was most likely to be in the cue's color and, independently, most likely to be on the cue's side.

On 75% of trials in which the location was pre-cued (with or without color information), the target was in fact on the pre-cued side. Those we call “location valid” trials. The remaining trials were “location invalid” trials. Analogously, the color information was valid on 75% of trials and invalid in the rest. With combined color + location pre-cues, each attribute was independently likely to be valid. Therefore, even if the target was on the uncued side (location invalid), its color was 75% likely to match the pre-cue (and vice versa). 

Observers were explicitly informed of all the above information about the pre-cues and their probabilities of being valid, neutral and invalid. They were therefore encouraged to attend selectively to the dots indicated by the pre-cues, because that would help them see the saturation change in the target dots in the majority of trials. 

The experiment had a 3 × 3 design: Each pre-cue could be color valid, color neutral, or color invalid and location valid, location neutral, and location invalid. Therefore, our trials were divided into a total of nine cue-validity conditions, with the proportions indicated in Figure 1d. All nine conditions were randomly intermixed within blocks of trials (with the constraint that the first trial of any block was never invalid). The valid and invalid conditions would have been sufficient to measure the two-by-two FBA × SA interaction, but by including the neutral conditions, we were also able to evaluate each type of attention operating on its own and to differentiate benefits brought about by valid cues and costs brought about by invalid cues. 

Each observer completed between 3,000 and 3,225 trials of the main experiment over five or six 1-hr sessions on different days. The first day began with practice with only neutral pre-cues until the observer achieved at least 75% accuracy with saturation increments smaller than the maximum possible. Then the magnitudes of the saturation increments were adjusted separately for red and for green, with 160 trials of an adaptive staircase. Following correct trials, we reduced the intensity level by 0.033 log units, and following incorrect trials, we increased it by 0.1 log units (Kaernbach, 1991). We halved the step sizes after four reversals and estimated the 75% correct threshold by averaging the intensity levels of all but the first two reversal points. We then used those threshold intensity levels in the main experiment. As a relief from being at threshold most of the time, we set the intensity levels to triple the threshold on a random four trials in each main experimental block of 79 trials. Those trials were excluded from the analysis. 

The average incremented saturation levels were 142% of the baseline for red (range across observers: 125%–162%) and 157% for green (range: 130%–175%). Over the course of sessions in the experiment, these levels were individually adjusted to keep performance near 80% correct for both red and green targets. The final day‘s intensity levels were on average 79% of the thresholds used on the first day. 

Eye tracking

We recorded the right eye's gaze position with an EyeLink 1000 Desktop Mount (SR Research, Ontario, Canada). Trials began once the observer fixated the central dot for at least 200 ms. If participants broke fixation between the pre-cue onset and the dots' offset, the trial ended immediately and was appended to the end of the block. We defined fixation breaks as gaze position deviations that exceeded 2 dva horizontally or 3.5 dva vertically (greater vertical leniency to avoid artifactual fixation breaks caused by pupil size changes). 

Data analyses

We computed sensitivity as d′ by labeling correct responses to targets in the upper hemifield as “hits” and incorrect responses to targets in the lower hemifield as “false alarms” (the opposite assignment is mathematically equivalent) and then using the following formula:  

 

where z is the normal z-score, p_h is the hit rate, and p_f is the false alarm rate. This measure uses our 2AFC localization data to approximate what d′ would be in a yes/no detection task. RTs, our secondary dependent variable, were computed as the total time processing the saturation increment, the latency between increment onset and the manual response, which includes the stimulus duration (300 ms) and the forced delay between dots offset and the response tone (600 ms).  

We used repeated-measures analyses of variance (ANOVAs) to analyze d′ and the geometric means of correct RTs. As a measure of ANOVA effect sizes, we report generalized eta squared (η_G²; Bakeman, 2005; Olejnik & Algina, 2003; Fritz, Moritz, & Richler, 2012). The absolute magnitude of η_G² is difficult to interpret except in comparison with similar studies, but Bakeman (2005) endorses the following general guidelines for η²-type measures: 0.02 is a small effect, 0.13 is medium, and 0.26 is large. Note that—in contrast to partial eta squared (η_P²)—η_G² is calculated with a denominator that includes the variance across participants, even in repeated-measures designs, to facilitate comparisons across different types of experimental design. For within-subject designs, therefore, η_G² is systematically smaller than η_P² (Bakeman, 2005; Olejnik & Algina, 2003), and our effect sizes should be interpreted with this in mind. For paired t tests, we report effect sizes as Cohen's d (J. Cohen, 1988; small = 0.2, medium = 0.5, large = 0.8). 

To confirm the observed patterns, we also conducted a bootstrapping analysis, in which we simulated repeating the experiment 10,000 times. On each repetition, we generated new responses by drawing from binomial distributions with the true response rates of a set of observers drawn with replacement from our sample. An effect (e.g., color valid – color invalid) was deemed significant if the 95% confidence interval on the distribution of bootstrapped differences did not include 0. 

Results

Average d′ levels are depicted in Figure 2a. The validity of the color cue significantly modulated d′, F(2, 16) = 69.5, p < 10⁻⁷, η_G² = 0.82, which was highest in color valid trials, intermediate in neutral trials, and lowest on color invalid trials. Sensitivity also depended on the validity of the location cue, F(2, 16) = 22.1, p < 10⁻⁴, η_G² = 0.18. Critically, there was a significant interaction between the two cue types, F(4, 32) = 5.55, p = 0.002, η_G² = 0.11. 

Figure 2

 

Results from Experiment 1 (left column) and Experiment 2 (right column). (a, c) Average d′ levels. Yellow points are model fits. (b, d) Mean reaction times on correct trials (RTs). Error bars indicate ± one within-subject SEM (Morey, 2008).

To explore the interaction in the accuracy data, we examined the spatial cueing effect separately for each color cue condition. The spatial cueing effect (Δd′ = spatial valid – spatial invalid; light vs. dark bars in Figure 2a) was significant in color valid trials (Δd′ = 0.53), F(2, 8) = 27.5, p < 10⁻⁵, η_G² = 0.43; marginal in color neutral trials (Δd′ = 0.15), F(2, 8) = 3.23, p = 0.066, η_G² = 0.14; but not significant in color invalid trials (Δd′ = 0.10), F(2, 8) = 1.55, p = 0.24. Moreover, the spatial cueing effect was larger in color-valid trials than in both color-neutral, t(8) = 2.83, p = 0.02, Cohen's d = 0.46, and color-invalid, t(8) = 3.81, p = 0.005, d = 0.88, trials. These patterns held for all but one observer. 

The interaction goes the other way as well: Although the color cueing effect was significant in all spatial cueing conditions (all ps < 10⁻⁶, all η_G² > 0.70), the difference Δd′ between color valid and color invalid was greater on spatial valid than in spatial invalid trials (1.46 vs. 1.04 d′ units), t(8) = 3.81, p = 0.005, d = 0.88.² Bootstrapped confidence intervals (CIs) confirmed all the ANOVA and t-test statistics: 95% CIs on bootstrapped differences excluded zero if and only if p < 0.05. For the interaction of FBA and SA, the 95% CI on the difference of the SA effect across color valid and invalid trials (color valid Δd′ – color invalid Δd′) was [0.156 to 0.68] and across color valid and neutral trials was [0.09 to 0.70]. 

The RTs mirrored the accuracy data (Figure 2b): main effect of the color cue, F(2, 16) = 10.122, p = 0.001, η_G² = 0.05; main of the location cue, F(2, 16) = 7.34, p = 0.005, η_G² = 0.006; interaction: F(4, 32) = 3.5, p = 0.017, η_G² = 0.003. The modulations of RTs are relatively small, most likely because of our emphasis on accuracy, our primary dependent variable, and the enforced 600-ms delay after stimulus offset. Also, the η_G² values include in the denominator the variance due to individual differences in overall RTs, which was larger than the cue validity effects. However, our measure of primary interest was not the RTs but d′, and the effect sizes on d′ are much larger. Importantly, the fact that the same pattern held in RTs and in d′ rules out possible speed-accuracy tradeoffs and demonstrates the robustness of our attentional effects. The fact that RTs were longer on invalid trials despite the enforced 600-ms delay could indicate that observers took more time to process the post-cue and respond to the appropriate stimulus when it did not match the pre-cue. 

Some observers reported that the task seemed easier when the target color change was in the outer segments of the arc (very top or bottom) than when it was adjacent to the horizontal boundary (Figure 3, top row). Indeed, a 3 × 3 × 2 repeated-measures ANOVA, with factors color cue validity, location cue validity, and target segment, showed that sensitivity was overall higher when the target was in the outer segments (1.40 vs. 1.12 d′), F(1, 8) = 15.87, p = 0.004, η_G² = 0.21. Moreover, the color cueing effect was stronger in the outer segments (Δd′ = 1.48) than in the inner segments (Δd′ = 1.26), as revealed by an interaction between target segment and color cue validity, F(2, 16) = 4.72, p = 0.025, η_G² = 0.03. However, the spatial cueing effect did not depend on target segments, and there was no three-way interaction among color and location cueing and target segment (both Fs < 1). Finally, the two-way interaction between color and location cue validity was significant in both the inner and outer segments analyzed separately (p = 0.02 and 0.01, respectively; η_G² = 0.09 and 0.07, respectively). The RTs qualitatively mirrored the d′ data, but no effects involving target segments were significant (all ps > 0.20). 

Figure 3

 

Data from both experiments divided by whether the target occurred adjacent to the horizontal boundary (“inner segments”; left column) or in the upper- or lower-most segments of the dot field (“outer segments”; right column). Error bars indicate ± one within-subject SEM.

The color effect within the spatial neutral condition (Δd′ = color valid – color invalid) was significantly correlated with the spatial effect within the color neutral condition (Δd′ = spatial valid – spatial invalid): Spearman's rho = 0.77, p = 0.02. The same analysis on RTs yielded a marginal correlation (rho = 0.65, p = 0.07). The correlation between the FBA and SA effects could indicate that the two types of attention share mechanisms or simply that the observers who put more effort into following the spatial cues also put more effort into following the color cues. 

Discussion

In this first experiment, with competing distractor saturation changes simultaneous with the change in the target dot field, FBA and SA interacted superadditively to modulate d′. SA only helped observers see dots at the cued location that also had attended colors. And although FBA spread across the visual field, its effects were strongest within the focus of SA. 

The interaction between location cues and feature cues was not due to a floor effect. SA was not effective in the color invalid condition, but d′ was overall substantially greater than 0, allowing room for spatial costs or benefits. Moreover, the spatial effect was significantly smaller in the color neutral than in the color valid condition, but accuracy in both of those color conditions was in a range wherein spatial cueing effects in principle could have been equally strong (as in Experiment 2, see below). 

The interaction is superadditive: the total benefit when attending to both the color and the location of a stimulus is more than the sum of each individually. The interaction could also be described as multiplicative: SA multiplies d′ by a fixed factor after FBA has already modulated it. The multiplicative factor can be estimated in the ratios of d′ levels: (spatial valid)/(spatial invalid). That ratio was actually larger in the color invalid condition than in the color valid condition (1.61 vs. 1.38), but the difference was not statistically reliable, t(8) < 1, consistent with a constant multiplicative factor for SA. Similarly, the ratio (color valid)/(invalid) did not differ across spatial valid and spatial invalid conditions (6.37 vs. 6.30), t(8) < 1. The difficulty with the multiplicative interpretation is that one effect of attention is conceptualized as a multiplication after the other (as in Boynton, 2005, 2009). From these data alone, we cannot say which one comes first. 

To perform this task in the neutral condition, all four changes must be seen and stored in a visual short-term memory trace until the post-cue (which indicates the target) is processed. Pre-cueing effects could reflect signal enhancement and/or distractor exclusion: biased competition in the visual short-term memory trace or another form of excluding the distractor changes from the decision process (Cameron et al., 2002; Carrasco et al., 2000, 2002; Eckstein et al., 2009; Foley & Schwartz, 1998; Lu & Dosher, 2000; Pestilli & Carrasco, 2005; Sperling & Dosher, 1986). From these data alone, we cannot say whether the superadditive FBA × SA interaction arises in these “later” stages or whether it might apply to attentional modulation of early visual processing. 

Experiment 2

In the second experiment, we investigated the interaction of FBA and SA with low stimulus competition. The target saturation change occurred in isolation, without the three distractor changes, but everything else about the stimulus and task was the same. Under these conditions, signal enhancement contributes proportionally more to pre-cueing effects than distractor exclusion (e.g., Cameron et al., 2002; Carrasco et al., 2000, 2002). Therefore, the behavioral cueing effects with low stimulus competition may match the effects of FBA and SA on the magnitudes of sensory signals in visual cortex, which are independent of each other and roughly additive (e.g., Andersen et al., 2011; Hayden & Gallant, 2009; Patzwahl & Treue, 2009). 

Methods

Participants

Thirteen naive volunteers with normal or corrected-to-normal vision participated in Experiment 2 (ages 19–34 years; 7 male) in exchange for a fixed monetary payment. We had more observers in this experiment than the first because we expected the cueing effects to be smaller without distractor changes, and therefore we would need more power. A power analysis³ using the data set from Experiment 1 predicts that 13 observers would be sufficient to find a significant FBA × SA interaction of half the magnitude but with the same level of noise. We are therefore confident that in Experiment 2, with 40,793 trials analyzed, we could have found any interaction worth reporting. 

Stimuli and task

All methodological details were identical to Experiment 1, except that there was only one saturation increment, in one quarter segment of one of the four groups of dots (e.g., in one quarter of the red dots on the right, as in Figure 1b). The post-cue pointed to the same side and had the same hue as this target saturation increment. As in Experiment 1, the observer's task was to report whether the target saturation change was above or below the horizontal white dividing line. 

The post-cue was not strictly necessary to perform the task in this experiment, but it eliminated any uncertainty about the target's side and color equally for all conditions. Without a post-cue, any potential effects of the pre-cue could result from changes in the decision process, giving more weight to noisy pre-cued stimulus representations (Eckstein, Shimozaki, & Abbey, 2002). The use of a post-cue, therefore, allows us to minimize decision-related effects and better isolate pre-cueing effects on the strength of perceptual encoding. It also facilitates the comparison of the results with those of Experiment 1. 

The average incremented saturation levels were 130.1% of the baseline for red (range across observers: 123.5%–140.1%) and 135.7% for green (range: 129.4%–156.4%). The final day's intensity levels were on average 70% of those on the first day. 

Results

Figure 2c shows the average d′ levels for localizing the target saturation increment. The validity of the color cue significantly modulated d′, F(2, 24) = 40.7, p < 10⁻⁷, η_G² = 0.41, as did the validity of the location cue, F(2, 24) = 3.48, p = 0.047, η_G² = 0.04. There was no interaction between the two cue types, F(4, 48) = 1.60, p = 0.19. These patterns are especially clear when excluding the neutral conditions, which were somewhat variable across participants: main effect of the color cue, F(1, 12) = 28.0, p < 0.001; main effect of the spatial cue, F(1, 12) = 10.7, p = 0.007; interaction, F(1, 12) = 0.47, p = 0.51. 

The lack of interaction indicates that the two cue types functioned independently. The average color cueing effect (Δd′ = color valid – color invalid) was 0.41 d′ units and as large at uncued locations (light bars) as at cued locations (dark bars), t(12) = 0.68, p = 0.51. For 5 of the 13 observers, the color cueing effect was numerically larger at uncued than at cued locations; the opposite was true for the remaining eight. The magnitude of the total spatial cueing effect (Δd′ = location valid – location invalid; the difference between light and dark bars in Figure 2c) was on average 0.14 d′ units and indistinguishable in the color valid and color invalid conditions, t(12) = 0.68, p = 0.51. In terms of individual observers, 12 of 13 performed better in the spatial valid than invalid conditions. Seven of the 13 observers had spatial cueing effects that were numerically smaller for the cued than the uncued color; the opposite was true for the remaining six. The SA effect was reduced in the color neutral condition but not reliably so, t(12) = 1.27, p = 0.23. That unusual reduction was driven by one observer with a large inverted spatial cueing effect in the color neutral condition (z-score of −2.2). 

The bootstrapping procedure confirmed all statistics from the ANOVA: 95% CIs for the overall spatial and color cueing effects did not include 0 (respectively, [0.06 to 0.21] and [0.29 to 0.53]). The 95% CIs for the interactions did include 0 and were therefore not significant. Specifically, the CI for the difference of the spatial effect (Δd′ = valid – invalid) between color valid and color invalid was [−0.15 to 0.31]. The CI for the difference of spatial effect and between color valid and color neutral was [−0.10 to 0.40]. The differences in color cueing effects between spatial cue conditions were also not significant: The CI for comparing the color effect across spatial valid and invalid was [−0.15 to 0.31] and across spatial valid and neutral was [−0.29 0.08]. 

The RTs mirrored the accuracy data, with no indication of a speed-accuracy tradeoff (Figure 2d). A two-way ANOVA revealed main effects of the color cue, F(2, 24) = 6.07, p = 0.007, η_G² = 0.004, and the location cue, F(2, 24) = 3.49, p = 0.047, η_G² = 0.0004, but no interaction, F(4, 48) = 1.01, p = 0.41. Again, these RT effect sizes are small compared with those on sensitivity, which was expected given that we strongly emphasized accuracy over speed. 

As in Experiment 1, sensitivity depended on whether the target color change was in the inner or outer segments of the arc (Figure 3, lower row). A 3 × 3 × 2 repeated-measures ANOVA, with factors color cue validity, location cue validity, and target segment, confirmed that d′ was overall higher when the target was in the outer segments (1.08 vs. 1.25 d′), F(1, 12) = 6.98, p = 0.02, η_G² = 0.06. There was also an interaction between segment and color cue validity, F(2, 24) = 6.84, p = 0.005, η_G² = 0.02, and between segment and location cue validity, F(2, 24) = 5.82, p = 0.009, η_G² = 0.02. Both cueing effects (Δd′ = valid–invalid) were stronger when the target was in the outer segments. The color effect was significant in both (mean Δd′ = 0.48 in outer segments; 0.34 in inner segments; both ps < 0.001, η_G² > 0.20). The spatial cueing effect was strong in the outer segments (mean Δd′ = 0.21), F(2, 24) = 7.23, p = 0.003, η_G² = 0.09, but was absent in the inner segments (mean Δd′ = 0.07), F(2, 24) < 1, η_G² = 0.003. In neither the inner nor the outer segments did color and location cueing effects interact, nor was there a three-way interaction with target segment (all Fs < 1). Therefore, the independence of FBA and SA is also evident in the subset of the data with the strongest cueing effects. RTs were not affected by target segment nor by any interactions between target segment and cue validity (all ps > 0.1). 

Finally, we examined the degree of correlation between the cueing effects across observers, as another way to test whether the two types of attention are independent. The color effect (Δd′ = color valid – color invalid) within the spatial neutral condition was not significantly correlated with the spatial effect (Δd′ = spatial valid – spatial invalid) within the color neutral condition (Spearman's rho = 0.18; p = 0.54). The correlation was also absent for the cueing effects on RTs (rho = 0.21; p = 0.50). 

Discussion

In this second experiment, we found that without saturation changes in the distractor dot fields, SA and FBA have independent and additive effects on visual sensitivity (d′). These data confirm that FBA enhances sensory signals across the visual field (e.g., Andersen et al., 2013; Martínez-Trujillo & Treue, 2004; Serences & Boynton, 2007; White & Carrasco, 2011). They suggest furthermore that the “spotlight” of SA equally illuminates all features at that location. These data are consistent with reports that FBA and SA have independent effects on the magnitude of neural responses in visual cortex (Andersen et al., 2011; Hayden & Gallant, 2009; Patzwahl & Treue, 2009). In contrast, previous behavioral studies have always reported superadditive interactions between FBA and SA (e.g., Bengson & Mangun, 2011; Handy et al., 2001; Kingstone, 1992; Leonard et al., 2015). 

We propose that our psychophysical data from Experiment 2 match the physiological data because the benefits of valid pre-cues we observed reflect primarily an increase in the signal-to-noise ratio of the target representation itself. Signal enhancement is likely for two reasons. First, unlike in Experiment 1, the target saturation change occurred in isolation with no physical changes in the other dot groups. The internal responses to those other dots with constant saturation were probably noisy, such that there may have seemed to be other small changes along with the target change. An attentional mechanism of distractor exclusion could have improved performance on valid pre-cue trials by excluding this noise. However, that mechanism must have played a smaller role than in Experiment 1, in which there were three potentially relevant saturation changes simultaneous with the target and physically equally strong. We conclude that signal enhancement contributed proportionally more to the attentional effects in Experiment 2 (e.g., Cameron et al., 2002; Carrasco et al., 2000, 2002). Second, the immediate post-cue resolved any uncertainty about the target's color or location, allowing the observer to base his or her decision on sensory information encoded from the correct stimulus, even on invalid trials. Therefore, higher d′ on valid trials means that cued stimuli were encoded more precisely than uncued stimuli and not that they received more weight in short-term visual memory or in the decision process or had lower response criteria. 

Comparison of Experiments 1 and 2

Finally, we evaluated whether the magnitudes of the color and spatial cueing effects and the nature of their interaction varied across the two experiments. To this end, we conducted a three-way ANOVA on d′ with factors color cue validity, location cue validity, and experiment. In this analysis, we assessed the total cue validity effects, defined as the difference in performance between valid and invalid cue conditions. The main effect of experiment was not significant, F(1, 20) < 1, but there were main effects of the color cue, F(1, 20) = 110, p < 10⁻⁸, η_G² = 0.71, and of the spatial cue, F(1, 20) = 43.3, p < 10⁻⁵, η_G² = 0.17. Both of those individual cueing effects interacted with experiment, being significantly larger in Experiment 1, respectively, F(1, 20) = 39.9, p < 10⁻⁵, η_G² = 0.46; F(1, 20) = 7.14, p = 0.015, η_G² = 0.03. Importantly, there was a three-way interaction among color cue validity, spatial cue validity, and experiment, F(1, 20) = 5.40, p = 0.03, η_G² = 0.03. The interaction between the two cueing effects was significantly stronger (superadditive) in Experiment 1 than in Experiment 2. From this, we conclude that the joint effects of FBA and SA are mediated by stimulus competition. 

An analogous three-way ANOVA on RTs produced a similar pattern. There was no main effect of experiment, F(1, 20) < 1, η_G² = 0.04, but there were main effects of the color cue, F(1, 20) = 12.1, p < 0.01, η_G² = 0.01, and of the spatial cue, F(1, 20) = 11.1, p < 0.01, η_G² = 0.002, although neither of those significantly interacted with experiment (both p > 0.10). The three-way interaction was marginally significant, F(1, 20) = 4.03, p = 0.059, η_G² = 0.0005, and in the same direction as the three-way interaction on d′. There were no speed-accuracy tradeoffs. 

The Independent Systems With Competition Model (ISC)

We strove to account for both experiments with a computational model of visual processing based on physiological effects in visual cortex and classic signal detection theory. The most parsimonious model would be one in which the difference in stimulation alone accounts for the difference in cueing effects across experiments. Therefore, our model rests on two basic assumptions: 

1.  
   SA and FBA independently modulate the signal-to-noise ratio of early visual responses that encode stimulus saturation. FBA applies equally across the visual field, and SA modulates response strength equally for all features. This assumption is in agreement with measures of sensory responses in visual cortex (Andersen et al., 2011; Hayden & Gallant, 2009; Patzwahl & Treue, 2009).
2.  
   Multiple stimulus representations compete for limited resources in a winner-take-all fashion, via competitive normalization (Carandini & Heeger, 2012, Lee, Itti, Koch, & Braun, 1999). Divisive interactions between representations favor those that were already strong, such as those boosted by attention.

Below, we formalize the model to demonstrate that independent attentional modulations of sensory responses can produce both an independent pattern of cueing effects and a superadditive interaction. With four key parameters that are held constant across experiments, this simple model can fit our data very well. The model is illustrated in Figure 4. It has three main stages. 

Figure 4

 

Diagram of the putative visual processing stages formalized in the ISC model. Stimulus inputs to the model are illustrated on the left. In this example, the red dots on the right are cued. In the first panel, the lengths of the colored bars represent the encoded saturation levels r_ij for each of the top (j = 1) and bottom (j = 2) halves all four dot fields (i). Note that dots with physically incremented saturation have longer bars (larger r), red bars are longer than green bars, and, independently, bars on the right are longer than bars on the left. The middle panel represents the stimulus representations after they have been exponentiated and normalized by each other. The final panel illustrates how the model makes a decision about the target dot field (q), with additive noise (represented by the spread of the distribution of x_q) limiting accuracy. See text for details.

Stage 1: Encoding + attention

Many hue-selective neurons increase their responsiveness with stimulus saturation (e.g., Hanazawa, Komatsu, & Murakami, 2000; Li, Liu, Juusola, & Tang, 2014; Schein & Desimone, 1990). A saturation increment could be read out from these neurons by comparing the response to the baseline saturation and the response to the incremented saturation (De Valois & Marrocco, 1973). Attention could improve detection accuracy by increasing the mean difference between responses to the baseline and incremented stimuli, for instance, via multiplicative gain. Based on those findings, our model simulates neural responses to the top and bottom halves of each of our four groups of colored dots. The means of these noisy responses depend on a multiplicative attentional gain factor, which is adjusted by independent and additive SA and FBA parameters. 

Specifically, on each simulated trial, the model produces eight saturation responses, labeled r_ij: one for the top (j = 1) and bottom (j = 2) of each dot field i (with i ∈ {1,2,3,4} for green left, red left, green right, red right). These r values represent the sensory responses of visual neurons that encode the hue and saturation of dots at particular locations in the display. The mean μ_ij of responses r_ij in a given attentional condition is equal to a stimulus drive ρ multiplied by an attentional gain factor g:  

 

where  

•  
  ρ_ij = μ_base if dots i,j have the baseline saturation
•  
  = μ_base + δ_e if dots i,j have a saturation increment

The free parameter μ_base represents the mean response to baseline dots in the neutral condition. δ_e is a free parameter that represents the starting signal-to-noise ratio (SNR) in Experiment e (δ₁ should be larger than δ₂ because the physical saturation increments were larger in Experiment 1). In each pre-cue condition (e.g., red dots on the right are cued), there is one gain factor g_i for each dot field i:  

 

 

The FBA factor f_i is set to a value F for color cued dots (benefits), 0 in color neutral conditions, and −F for color uncued dots (costs). Similarly, the SA factor s_i is set either to S, 0, or −S. Values F and S are free parameters. For simplicity, we assume the attentional costs to be equal in magnitude to the attentional benefits. Although g is applied to the responses to both baseline and incremented dots, it can improve (or impair) encoding of the increment by increasing (or decreasing) the mean distance between them. Each r_ij is drawn from a unit Gaussian with mean μ_ij. 

Stage 2: Competition

After the saturation levels are encoded, the eight responses compete with each other via divisive normalization. Each r_ij gets raised to an exponent k, and then all eight are divided by their sum. We label the resulting representations v_ij:  

 

 

These values v represent the activity levels in a second stage of visual processing, either in the same population of neurons as the responses r in Stage 1 or a different population (immaterial for our present purposes). 

The exponent implements the competition: It can amplify the dominance of sensory responses that start out relatively high (e.g., for dot fields that are attended) and exaggerate small differences in the upper range. These effects are potentiated when more of the initial responses are large, making the denominator large, as occurs in Experiment 1 when there are four simultaneous saturation increments. 

Stage 3: Late noise + decision

To report whether the increment was in the top or bottom of the post-cued target dot field q (with q ∈ {1,2,3,4}), the model takes a noisy estimate x_q of the difference between the responses to the top and bottom halves:  

 

 

The difference estimate x_q is corrupted by a second stage of noise: An independent random Gaussian variable (μ = 0, σ = 1) is added to it. The model then reports that the saturation increment was in the top half if x_q > 0 and the bottom half otherwise. The late noise means that an initially correct saturation estimate that gets a boost from the normalization is more likely to remain correct in the final decision. Factor α simply scales all the values up and down together to match the final mean signal-to-noise ratio (d′) to the global mean of d′ measurements to which we fit the model. 

By simulating thousands of such trials, we computed the model's d′ levels in each condition of attention. To fit the model, we first varied six free parameters: μ_base, δ₁, δ₂, F, S, and k. For each parameter set, for each attentional condition, we simulated 100,000 trials and computed d′ levels from the model's perceptual decisions. We searched through regular grids of the six-dimensional parameter space to find good starting levels and then ran nonlinear optimization routines to minimize the squared residuals between the model and the mean d′ data. The pattern of cueing effects is primarily determined by F, S, and k. The initial sensory response levels controlled by μ_base, δ₁, and δ₂ can modulate the effects of noise at later stages (as the noise standard deviations were fixed at 1). 

Results

The best fit, plotted in yellow points in Figures 2a and 2c, accounted for 95% of the variance in sensitivity observed in the 18 conditions of Experiments 1 and 2. The best-fitting parameters were baseline sensory response μ_base = 1.01; Experiment 1 SNR δ₁ = 1.03; Experiment 2 SNR δ₂ = 6.9; spatial gain modulation S = 0.03; feature-based gain modulation F = 0.12; exponent k = 6.4. The scale factor α = 2.17. 

Discussion

In our Independent Systems with Competition (ISC) model of visual processing, both experiments are modeled with the same attentional gain changes, the same exponentiation and normalization, and the same simple decision process. Nonetheless, the model accurately replicates the main patterns of interest in our experimental data: First, the cueing effects were larger in Experiment 1 than in Experiment 2, even though the sensory gain changes were the same. Second, only in Experiment 1 was there a superadditive interaction between the spatial and feature cueing effects, whereas the two effects were additive in the simulation of Experiment 2. 

Those differences in the model's outputs are due only to the fact that there were more stimuli with physical saturation changes in Experiment 1, which potentiates the effect of the normalization. The normalization could reflect biased competition during encoding or during the maintenance of multiple stimuli in short-term memory until the post-cue prompts a decision about just one of them. Attentional gain changes during encoding give advantages to cued stimuli, and stimulus representations that “win” the subsequent competitive normalization are less likely to be corrupted by the second stage of noise. In Experiment 2, the same normalization process occurs, but only one item has an increment; that one strong response is likely to dominate the normalization regardless of the attentional condition. In that case, FBA and SA can reduce errors primarily by reducing the impact of noise in the initial encoding stage, and they do that independently. 

Our method of implementing stimulus competition via exponentiation and normalization is similar to the “winner-take-all” competition described by Lee et al. (1999). Note that in contrast to that model, however, our exponent k does not vary across attention conditions. The exponent in our competitive normalization is also related to the “max-pooling” operation that accounts for the effects of attention on cortical BOLD responses in the presence of salient distractors (Pestilli, Carrasco, Heeger, & Gardner, 2011). 

The ISC model differs in important ways from previous models that simulated both SA and FBA with “normalization” as a key computational component. Reynolds and Heeger (2009) and Boynton (2005, 2009) simulated populations of individual neurons, whereas we simulate responses to entire stimuli, which could be conceptualized as the mean firing rates of many neurons selective to each stimulus. Those previous models used normalization to implement suppression of each neuron by a pool of others with a range of tuning preferences (reviewed by Carandini & Heeger, 2012). Importantly, they did not specify a prenormalization exponent. The relatively large exponent k is critical to the competition between responses in our model (and in Lee et al., 1999; Pestilli et al., 2011) and in creating the interaction in Experiment 1. Furthermore, we include responses to all stimuli in the denominator of the normalization (i.e., the “suppressive field”) rather than responses of a more local pool of neurons. This is important to our model, and it may indicate that the type of competition responsible for the interaction in our data is more global and perhaps—but not necessarily—at a later stage of processing than the normalization in early visual cortex (as suggested by Lee et al., 1999). Such global competitive normalization between all stimuli has been used to account for working memory performance (Bays, 2014, 2015). 

Our method of applying attentional gain changes before normalization is in agreement with the normalization model of attention of Reynolds and Heeger (2009). Their model, however, is underspecified with regard to how SA and FBA combine to shape the “attention field.” Our results as well as previous physiological studies (e.g., Hayden & Gallant, 2009; Patzwahl & Treue, 2009) suggest that the spatial and feature gain profiles should add linearly. This differs from Boynton's (2005, 2009) model in which a multiplicative gain factor for FBA is applied after the normalization and after SA has modulated the response. Such a model would not predict the independent cueing effects that we found in Experiment 2, because even if the FBA and SA parameters vary independently, they would always interact in a superadditive fashion (the FBA parameter would have a larger effect for spatially attended stimuli, being multiplied against larger responses). 

Note that we cannot rule out these alternative models, as we have not quantitatively tested them against our data. At the moment, the purpose of our model is to demonstrate that independent and additive attentional modulations can produce cueing effects that are either independent or that interact superadditively. Therefore, an interaction in behavioral data does not imply that the underlying attentional systems interact. Moreover, we need not assume that one attentional modulation occurs before the other. Future work could design experiments specifically to compare the quantitative predictions of competing models for SA and FBA. Finally, more elaborate models in the future should take into account that the profile of FBA modulation within a feature dimension can have nonmonotonic enhancement and suppression as a function of feature value (Ho, Brown, Abuyo, Ku, & Serences, 2012; Störmer & Alvarez, 2014). 

General discussion

Summary

We manipulated SA and FBA with pre-cues that informed observers of the most likely location and/or color of a target saturation increment to be detected. Visual sensitivity was highest when the target's location and color were both validly pre-cued and lowest when both were wrongly pre-cued. In Experiment 1, with four simultaneous saturation changes, the cueing effects interacted strongly. Spatial cueing effects (valid – invalid) were largest for the attended color, and color cueing effects were largest at the attended location. In Experiment 2, when the target saturation change occurred in isolation, the spatial and color pre-cues had independent and roughly additive effects. Thus, deploying one type of attention potentiates the effect of the other only in the presence of stimulus competition: when the target stimulus change is accompanied by distractor changes in other potential targets. 

Different degrees of uncertainty reduction cannot account for the distinct attentional patterns across experiments (see Cameron et al., 2002; Carrasco et al., 2000, 2002; Eckstein et al., 2002, 2009). The post-cue, which indicated the side and color to respond to, resolved uncertainty only 100 ms after the target increment in both experiments. Also, although both cue types provided the same amount of information, the color cueing effects were larger than the location cueing effects in both experiments. The reason for this differential effect may be that observers were asked to detect an increase in saturation, which is more related to hue than to location. The difference in effect size did not skew the measurements of how the effects combine, however, because we observed clear additivity in one experiment and a clear interaction in another. 

In both experiments, attention may also have been influenced by the automatic formation of “surfaces” or “objects”: groups of dots that cohere into structures that guide attention (Ciaramitaro, Mitchell, Stoner, Reynolds, & Boynton, 2011; Driver, Davis, Russell, Turatto, & Freeman, 2001; Ernst, Boynton, & Jazayeri, 2013; He & Nakayama, 1995; Valdés-Sosa, Cobo, & Pinilla, 2000; Wannig, Rodríguez, & Friewald, 2007). Such structure could either have facilitated or impaired the voluntary attentional selection as instructed by our spatial and color cues. For instance, FBA would have to counteract the formation of one surface by all dots within an aperture, and SA would have to counteract the formation of surfaces by dots unified by color or location above/below the horizontal meridian. Nonetheless, to the extent that those factors may have played a role, they were constant across the experimental manipulations of our study, and they did not prevent reliably measurable cueing effects. 

In both experiments, we found that discriminating the target's location was easier when it was far from the horizontal dividing line, which is not surprising as performance suffers when the target appears close to the boundary between two regions (see Carrasco & Chang, 1995). It is less clear why the cueing effects also tended to be stronger in the outer segments. However, we emphasize that the interaction between types of attention (Experiment 1) or lack thereof (Experiment 2) did not depend on the segment in which the target appeared. 

In contrast to previous investigations, we measured how SA and FBA together determine visual sensitivity—that is, how accurately observers can detect near-threshold stimuli. RTs (e.g., Bengson & Mangun, 2011; Kingstone, 1992; Lambert & Hockey, 1986) and detection false alarm rates (Andersen et al., 2011) have shown superadditive interactions. Both of those measures, however, may be affected by cognitive factors in the decision process (e.g., Carrasco & McElree, 2001; Carrasco et al., 2003; Ratcliff, 1978; Wickelgren, 1977). In a previous study measuring visual sensitivity, we found that the automatic shift of SA to the peripheral target of an impending saccade is not accompanied by attentional enhancement of the target object's features (White, Rolfs, & Carrasco, 2013). This finding also suggests independence between the two attentional control systems, but only in the data presented here did we cross valid, neutral, and invalid cues to fully examine the potential interaction. 

One other recent study measured perceptual accuracy and reported that FBA and SA interact (Leonard et al., 2015). Specifically, they measured SA capture effects: decrements in accuracy at an attended location caused by irrelevant distractors elsewhere. Those decrements were stronger if the distractor matched the target in color; this is the putative FBA effect. Importantly, the modulatory effect of color was weaker for distractors farther from the attended target location, suggesting an interaction of FBA and SA. This finding is at odds with studies showing that FBA is spatially global (e.g., Liu & Mance, 2011; Sàenz et al., 2002; Serences & Boynton, 2007; Treue & Martínez Trujillo, 1999; White & Carrasco, 2011) and with the fact that early visual responses in cortex show independent effects of SA and FBA (Andersen et al., 2011; Hayden & Gallant, 2009; Patzwahl & Treue, 2009). However, Leonard et al.'s (2015) measure of FBA was indirect: It depended on an irrelevant colored item exogenously diverting SA from the target stimulus. As the authors acknowledge, this complexity could be responsible for the dependence of the FBA effect on spatial distance. In contrast, both of our attentional effects were purely endogenous, and the FBA measure was direct. Moreover, according to our model, the stimulus competition in their display—stimuli of many different colors in rapid succession and three differently colored items at once in the critical frame—could have produced the interaction in the behavioral output. 

In our Experiment 2, sensitivity is the first behavioral measure to show independence between the two cueing effects, consistent with modulations of neural responses in visual cortex (Andersen et al., 2011; Hayden & Gallant, 2009; Patzwahl & Treue, 2009; Treue & Martínez Trujillo, 1999). In Experiment 2, with no distractor saturation changes, cueing effects on d′ represent mainly signal enhancement, as there was low external noise. In Experiment 1, pre-cues may have also improved performance by “excluding” the distractor changes during stimulus encoding or from a visual short-term memory buffer (e.g., Cameron et al., 2002; Carrasco et al., 2000, 2002; Eckstein et al., 2009; Foley & Schwartz, 1998; Lu & Dosher, 2000; Pestilli & Carrasco, 2005; Reeves & Sperling, 1986; Smith & Ratcliff, 2009). Put another way, cueing biases the competition between stimulus representations to greater effect when there are more simultaneous changes in the input, as we simulated in a model fit to the data. 

The ISC model

The key ingredients of our ISC model are (a) independent modulations of sensory response gain by SA and FBA and (b) a normalization process that implements winner-take-all competition between multiple stimulus representations. We fit this model to both experiments with identical parameters for the independent attentional modulations, competitive normalization, internal noise, and decision processes. Nonetheless, it reproduced the superadditive interaction in Experiment 1 and the additive cueing effects in Experiment 2. The superadditive interaction was due only to the greater number of physical saturation increments, which exaggerated the effect of the exponent in the normalization stage. 

Previous models that include similar elements of attentional gain before normalization (Reynolds & Heeger, 2009) or similar exponentiation before normalization (Lee et al., 1999; Pestilli et al., 2011) would not fully account for our data. Our data require a novel combination of those elements: independent and FBA and SA parameters that add up to determine the multiplicative gain on sensory responses and a constant large exponent applied to each response before the divisive normalization. 

We do not prove that our model is better than alternative models that could also explain our data, but we note that they are likely to be more complicated. For instance, one could postulate that the nature of the interaction between the two forms of top-down attention is itself flexible and somehow switched by the task demands imposed by distractors. That could be modeled by adding an additional “interaction” parameter to adjust either F or S depending on the value of the other. However, we can already account for 95% of the variance in both experiments' data without that additional parameter. 

Alternatively, SA and FBA could each operate via multiple mechanisms, some independent and some not. Experiment 2 suggested that signal enhancement occurs independently for attended colors and for attended locations. The interaction in Experiment 1 could indicate that additional attentional mechanisms operate jointly. One such mechanism is external noise exclusion (e.g., Lu & Dosher, 2000), which is distinct from signal enhancement and would be more effective in the presence of distracting saturation changes (as in Experiment 1). A related distinction between attentional mechanisms is blocking versus attenuation (Yigit-Elliot, Palmer, & Moore, 2011). Blocking refers to the filtering out of distractor stimuli at irrelevant locations, and attenuation is the reduction in signal strength at locations less likely to contain the target. These mechanisms could theoretically have distinct patterns of joint SA and FBA control, and blocking could be more active in our Experiment 1. 

But in the ISC model, the difference in the stimuli is enough to explain the different cueing patterns we found across experiments, without the need to postulate different mechanisms. In all conditions, SA and FBA independently modulate the signal-to-noise ratio of sensory measurements of the dots' colors, and nothing else. We favor this model for its simplicity and for its agreement with the neurophysiological evidence for (nearly) independent attentional effects on sensory responses in visual cortex. 

The fact that cueing effects increased with stimulus competition is largely consistent with the “biased competition” framework of attention (Desimone & Duncan, 1995; Kastner & Ungerleider, 2001). However, in our model, the mechanisms of attention do not themselves affect the mechanisms of competition (normalization). Rather, it is the difference in the input when there are competing stimuli that causes the normalization to exaggerate cueing effects. A recent EEG study questioned the assumptions of biased competition by showing that the magnitude of covert SA effects are unaffected by the presence of competing stimuli (Keitel, Andersen, Quigley, & Müller, 2013). Our data are at face value inconsistent with this finding. However, a range of methodological differences could account for the discrepancy, and it is unknown how an effect of competition on occipital steady-state visual evoked potentials (SSVEPs) relate to an effect of competition on perceptual sensitivity. 

Our model is consistent with several previous physiological reports in that the apparent interaction of FBA and SA depends on the stage of processing measured. Regardless of the complexity of the stimulus, the attentional effects are independent at the first stage of encoding. This is true in our model and in physiological measures of early cortical responses (Andersen et al., 2011; Hayden & Gallant, 2009; Patzwahl & Treue, 2009). However, later neural responses, specifically ERPs measured from the scalp, show interactions beginning 200 to 300 ms after stimulus onset (Andersen et al., 2011; Bengson et al., 2012; Eimer, 1995; Handy et al., 2001; Hillyard & Münte, 1984). There is no consensus on the cognitive processes involved at those later stages. Some claim that the interaction reflects target identification (Andersen et al., 2011) or the updating of working memory (Bengson et al., 2012; Bengson & Mangun, 2011). According to our model, they could represent a stage of global competition between potentially relevant stimulus representations. 

However, the interactions in RTs and in late ERPs sometimes occur even with low stimulus competition, when only a single stimulus is presented at a time (Bengson et al., 2012; Handy et al., 2001; Kingstone, 1992), which we did not find in our Experiment 2 (with d′ and RTs). The proposed ISC model is not strictly consistent with those studies, but we note that they failed to observe any main effects of feature cueing in early responses, perhaps because the tasks used did not fully demand selective FBA. The interaction in RTs as opposed to d′ may be specific to higher-level processes such as the speed of decision making (Handy et al., 2001; van Ede, de Lange, & Maris, 2012). We eliminated such factors by presenting an immediate post-cue to resolve any decision uncertainty, by emphasizing accuracy rather than speed, and by forcing observers to wait 600 ms before responding. 

Attentional control

The present results are consistent with a growing body of evidence that SA and FBA are controlled independently but modify visual processing in similar ways. Psychophysical data indicate that SA enhances sensory gain across feature values within a dimension, and FBA selectively “tunes” the population of feature detectors (Baldassi & Verghese, 2005; Ling, Liu, & Carrasco, 2009). However, the two types of attention seem to have similar effects on population activity in the visual cortex (e.g., Cohen & Maunsell, 2011; Ling, Jehee & Pestilli, 2014; Patzwahl & Treue, 2009; Treue & Martínez Trujillo, 1999). The “feature-similarity gain” model (Martínez-Trujillo & Treue, 2004) accounts for a wide range of such data: The primary mechanism of attention is to modulate the gain of individual visual neurons as a function of the similarity between their tuning preferences and the task-relevant feature values, and location is just another dimension along which neurons are tuned. A similar notion is central to the normalization model of attention (Reynolds & Heeger, 2009), which has received support from studies of both SA (Herrmann, Montaser-Kouhsari, Carrasco, & Heeger, 2010) and FBA (Herrmann, Heeger, & Carrasco, 2012). 

Regarding top-down control systems, functional magnetic resonance imaging studies indicate that a common frontal-parietal network is activated by shifts of attention to locations (e.g., left vs. right), feature dimensions (e.g., motion vs. color), and feature values (e.g., red vs. green; Egner et al., 2008; Liu, Slotnick, Serences, & Yantis, 2003; Giesbrecht, Woldorff, Song, & Mangun, 2003; Serences & Boynton, 2007). However, within these common regions are intermingled but independent neural networks that control selection by different dimensions (Greenberg, Esterman, Wilson, Serences, & Yantis, 2010; Liu, Hospardaruk, Zhu, & Gardner, 2011). Furthermore, some areas are specialized for SA (Schenkluhn, Ruff, Heinen, & Chambers, 2008). The existence of independent attentional control systems could explain why the temporal dynamics of FBA and SA differ, as measured psychophysically (Liu et al., 2007) and in single V4 neurons (Hayden & Gallant, 2005). Our ability to model the attentional effects as independent is also consistent with this notion. 

SA and FBA are two critical processes that enable the selective prioritization of elements of sensory input. There are several other types of attention, and in some cases, their interactions also depend on the stage of processing measured. For instance, spatial and temporal expectations, when combined, have a complex pattern of interaction on RTs and various ERP components (Doherty, Rao, Mesulam, & Nobre, 2005). Endogenous spatial orienting and exogenous or stimulus-driven capture effects interact differently in measures of manual RTs versus oculomotor responses (Schreij, Los, Theeuwes, Enns, & Olivers, 2014). Finally, the interaction of endogenous and exogenous SA on perceptual accuracy unfolds through time. For perceptual decisions forced to be early, the two types of attention independently modulate d′, but with enough processing time, focused endogenous attention can reduce the effect of exogenous distraction (Grubb, White, Heeger, & Carrasco, 2014). 

Conclusion

Spatial and feature-based attention (FBA) additively enhance visual signals when combined. Interactions emerge when there are distracting events simultaneous with the target. We link these patterns of sensitivity to neural processes with a novel computational model, the ISC model, in which both types of attention independently modulate the gain of visual neurons. Responses to multiple stimuli then compete via normalization, a canonical cortical computation. The competition introduces a nonlinearity that can create superadditive cueing patterns, without any interactions between the top-down modulations themselves. Therefore, contingent on the nature of the visual scene, relatively simple and independent attentional effects early in processing can develop into more complicated patterns in the perceptual output. 

Author contributions

A. L. W. and M. C. designed the experiments, A. L. W. collected the data, A. L. W. and M. R. analyzed the data, and all three authors designed the model and wrote the article. 

Acknowledgments

We thank Michael Landy for invaluable advice on the model, Gijs Brouwer for help controlling the colors of the stimuli, Wanghaoming Fang for assistance collecting data, and members of the Carrasco Lab for helpful discussions. We are also thankful for the following grant support: National Institutes of Health (NIH) grants RO1-EY-16200 and RO1-EY-019693 to M. C., NIH training grant T32 EY007136 to NYU, and the Emmy Noether program of the Deutsche Forschungsgemeinschaft (RO 3579/2-1) to M. R. 

Commercial relationships: none. 

Corresponding author: Alex L. White. 

Email: [email protected]. 

Address: Department of Psychology, University of Washington, Seattle WA, USA. 

Footnotes

¹  A set of nine observers is within the typical range for visual psychophysics: In fact, the average number of subjects per experiment in all of the behavioral studies cited in this article is 9.17. With more than 3,000 trials analyzed per observer (28,190 total), our data set was comparably large.

Footnotes

²  Note that the statistics for the comparison of the spatial effect across color valid and invalid conditions are the same as for the comparison of the color effect across spatial conditions. They are based on the four conditions, subtracted in a different order.

Footnotes

³  In this power analysis, we conducted a bootstrapping procedure for a variable number m of observers, drawing m observers with replacement from the original set and generating from them new binomial data. For each level of m, we then had a distribution of the differences between the spatial cueing effect in the color valid condition and the spatial cueing effect in the color invalid condition. To test our power for detecting a much weaker interaction, we subtracted from this distribution half of its mean and then estimated its 95% confidence interval. That CI excluded zero for m > 12.