The effect of background color on asymmetries in color search

Abstract

Many previous studies have shown that background color affects the discriminability and appearance of color stimuli. However, research on visual search has not typically considered the role that the background may play. Rosenholtz (2001a) has suggested that color search asymmetries result from the relationship between the stimuli and the background. Here we test the hypothesis that background color should have an effect on asymmetries in visual search based on color, using searches for color stimuli on different colored backgrounds. Observers searched for a single known target stimulus among homogeneous distractor stimuli. The target stimulus differed from the distractors only in chromaticity, but targets and distractors both differed from the backgrounds in luminance so that they were easily visible regardless of chromaticity. Target/distractor pairs differed primarily in saturation (Experiments 1, 2, & 3) or in hue (Experiment 4). Each member of each pair of colors served as target and distractor color on both achromatic and red backgrounds. When the stimuli were presented on an achromatic background, response times were shorter when the more saturated member of each pair of colors served as the target color. When the same stimuli were presented on a red background, the asymmetry was either reversed or abolished. When target and distractors differed in hue, there was little asymmetry on the achromatic background but a sizable asymmetry for some color pairs on the red background. On both backgrounds, the magnitude of the asymmetry varied with the difference between the stimulus colors and the background color. Results confirm that asymmetries in color search are dependent on the relationship between the stimulus colors and the background color. Two candidate models are suggested that show promise in predicting these experimental results: Rosenholtz’ saliency model (1999, 2001a) and a modification to signal detection theory models in which the observation noise is proportional to the difference between target/distractor color and background color.

Introduction

Asymmetries in visual search tasks have been an important phenomenon in the study of visual attention for the last couple of decades. Asymmetries are often found in search tasks in which the observer searches for a target stimulus presented among a set of uniform distractor stimuli (Wolfe, 2001). An asymmetry occurs when switching the role of target and distractor stimuli causes a difference in performance. For example, when an observer searches for a more saturated red target stimulus among less saturated red distractor stimuli, on a dark, achromatic background, the search may be very fast. But when the colors are reversed so that the target stimulus is less saturated than the distractors, the search is typically somewhat slower (Nagy & Cone, 1996). 

Theoretical explanations of the asymmetries have often been related to properties of feature coding mechanisms within the visual system (e.g., Treisman & Souther, 1985). One of us (Rosenholtz, 2001a) has suggested instead that many of the experiments that have yielded asymmetric search results were asymmetrically designed. In the example given above, Rosenholtz argues effectively that the design was asymmetric because the less saturated stimulus is more similar to the neutral background than the more saturated stimulus; one might imagine that this would make search for the less saturated target more difficult than search for the more saturated target. Most studies of asymmetries in visual search have not explicitly considered the background on which the stimuli were presented. 

Rosenholtz (1999, 2001a) has proposed a saliency model that can qualitatively predict many of the asymmetries found in visual search on the basis of a simple measure of target-distractor similarity, without resorting to asymmetries in the underlying feature coding mechanisms. An important feature of this model is that it considers the background on which target and distractor stimuli are presented. The saliency model assumes that the background color can also act as a “distractor.” Because the less saturated stimulus is more similar to the neutral background than the more saturated stimulus, the saliency model predicts that search should be faster when the more saturated red stimulus is the target than when the less saturated stimulus is the target. The model further predicts that by changing the background color, one should be able to make a color search asymmetry reverse, disappear entirely, appear, or change magnitude. 

Many previous studies have shown that background color affects the discriminability and appearance of color stimuli. Chromatic discrimination thresholds are smallest for stimuli that are approximately achromatic when the background is dark or achromatic, but on chromatic backgrounds the smallest discrimination thresholds occur for stimulus chromaticities that are similar to the background chromaticity (e.g., Krauskopf & Gegenfurtner, 1992; Miyahara, Smith, & Pokorny, 1993). The color appearance of stimuli of fixed chromaticity is also altered when background color is changed. Contrast effects and assimilation effects both occur depending on the spatial configuration of the stimuli. When contrast effects occur, the color appearance of the stimulus is shifted away from the background color (e.g., Shevell & Wei, 2000) but when assimilation effects occur, the appearance of the stimuli is shifted toward the background color (e.g., Monnier & Shevell, 2001). 

In what follows, we present the first direct test of the hypothesis that the background color on which stimuli are presented can have a direct influence on visual search asymmetries. We chose to compare searches for color targets on a couple of different backgrounds and used a variety of target and distractor colors that were chosen on the basis of previous experiments. 

In Experiments 1 and 2, we focus on an asymmetry reported by Nagy and Cone (1996), who found that on an achromatic background, search for a more saturated reddish target among less saturated reddish distractors was easier than vice versa. We demonstrate that by changing the background color to saturated red, one can reverse this asymmetry. In Experiment 3, we show in a similar experiment that when searching for a more saturated blue target among less saturated blue distractors, and vice versa, switching from an achromatic background to a red background can eliminate a search asymmetry. In Experiment 4, we examine search for a target that differs from distractors primarily in hue, because Nagy and Cone (1996) reported little or no asymmetry in search for targets defined by a hue difference on a dark achromatic background. We show that by changing the background color from achromatic to a saturated red, one can produce a search asymmetry for stimuli that differ in hue where previously there was no asymmetry. 

Methods

Apparatus

Stimuli were generated on a Nanao Flexcan T2–17 color monitor. The monitor was driven by a Radius Thunder 30 color-graphics card installed in a MacIntosh 8500 computer. The monitor was calibrated with a Minolta CS 100 chromameter. Calibrations were completed separately for each background field to be used in the experiments, because the presence of the background fields altered the gamma functions of the phosphors slightly. The calibration data were used to produce color look up tables containing luminance levels for each of the phosphor levels. The lookup tables were used in conjunction with a program written in the lab to generate desired colors using a least squared error criterion. The phosphor levels for the desired colors were then stored in text files, which were used by another program that produced the displays and conducted the experiments. The program also collected response times for each trial and provided feedback to the observer in the form of a tone. 

The stimuli were circular disks with 0.14 deg diameter. The stimulus disks were presented in random locations within a circular area with 4.25 deg diameter and centered on the monitor screen. The stimuli were positioned so that no two stimuli contacted or overlapped each other. On each trial,54 disks were presented. We arbitrarily focused on one large set size in order to limit the experiments and because previous work has explored search asymmetries for color stimuli presented on dark backgrounds with similar set size conditions (Nagy & Cone, 1996). The disks were identical except for the target disk, which differed in chromaticity from the other stimuli. Stimuli were always equated in luminance, and the stimulus luminance always differed from the background luminance so that the stimuli were always easily visible regardless of their chromaticity. In most of the experiments, the luminance was fixed at 7 cd/m², but in Experiment 2, the luminance of the stimuli was fixed at 4 cd/m². The stimuli were presented against different background colors in different conditions, but background color was not changed within an experimental session. The background color filled the monitor screen. Achromatic and red backgrounds were fixed at a luminance of 5 cd/m² in all experiments. The display was viewed from a distance of 125 cm in a darkened room. 

Procedure

Procedures were similar for all of the experiments. Each trial was initiated by the presentation of a fixation cross in the center of the display area. After an interval of 1 s, the fixation cross was turned off and the stimuli, which were drawn within one frame, were displayed on the screen. When the stimuli appeared, the observer’s task was to determine whether a target stimulus was present in the display. Each block of trials began with five practice trials, which indicated to the observer the colors of the target and distractor stimuli for that block of trials. The observer was asked to determine as rapidly as possible if a target was present in the display and depress the mouse button as soon as a decision was made. Observers were free to make eye movements in order to detect the target if necessary. The computer recorded the time interval between the onset of the stimulus display and this button push. Two thirds of the trials contained a target, whereas the remaining third contained only distractors. When the mouse button was pressed, the stimuli were turned off and a vertical line that divided the screen in half appeared. To indicate that the target was present, the observer placed the cursor in the left half of the screen and depressed the mouse button again. The cursor was placed in the right half of the screen and the mouse button depressed to indicate that the target was absent. A tone was sounded immediately after the button push if the observer made an error. To avoid speed accuracy tradeoffs, the observer was instructed to maintain accuracy at 90% or better while responding as rapidly as possible. If the errors exceeded the number allowable, the block of trials was interrupted and restarted. A short time after the response indicating presence or absence of the target, the fixation cross appeared on the screen initiating the next trial. Each block of trials contained 40 target present trials and 20 target absent trials. From 6 to 14 blocks of trials were completed within an experimental session that lasted from slightly less than 1 to about 2 hr. Within each block of trials, target and distractor colors were fixed. Each observer ran each condition twice, and the mean log response time was calculated for the 80 target present trials in these two runs. 

Observers

Seven females and two males, including two of the authors, NB and ALN, served as observers in the various different experiments described below. They ranged in age from 21 to 54 years, and all had normal color vision as indicated by Rayleigh matches and scores on the Ishihara Psuedoisochromatic Plate test. 

Experiment 1: Changing the background can reverse an asymmetry

As mentioned above, Nagy and Cone (1996) have shown that against a dark gray background, search for a more saturated red target among less saturated red distractors is faster than vice versa. Here we explore what happens to that asymmetry when the background color is changed to a dark saturated red. 

Observers searched for a more saturated reddish target among homogeneous less saturated distractor stimuli and also searched for the less saturated target among homogeneous more saturated red distractor stimuli. Each pair of searches was conducted with stimuli presented against an achromatic background and also with the same stimuli presented against a red background. The chromaticities of the stimuli and the backgrounds are illustrated in Figure 1. The axes in this plot represent the chromaticity coordinates in the MacLeod and Boynton (1979) chromaticity diagram. These axes are often referred to as cardinal color directions because they are thought to represent the two independent color-opponent mechanisms in the peripheral portions of the human visual system. Many experiments, both physiological and psychophysical, support this idea (see Boynton, 1986; Lennie & D’Zmura, 1988). 

Figure 1

 

Chromaticities used in the first two experiments are shown in the cone excitation diagram. Stimulus colors are represented by Xs, and background colors are represented by open circles. Red, green, and blue disks connected by solid lines indicate the phosphor chromaticities and color gamut for the monitor.

White is located at coordinates of approximately 0.666, 0.016, and the spectrum locus is indicated by the curved gray line. Moving away from the white locus in any direction represents more saturated colors. Moving around a circle centered on the white locus represents varying the hue of the stimulus. The stimuli in the first experiment were chosen so that they varied in L chromaticity and had a fixed S chromaticity and luminance. 

Thus the stimuli vary in chromatic saturation and in the way they excite one of the peripheral color opponent mechanisms (L) while producing approximately the same excitation in the other color opponent mechanism. 

Each of the five Xs in Figure 1 indicates a stimulus color. Though the chromaticity differences between neighboring colors are the same on the L axis (an L chromaticity difference of .0265), this does not necessarily represent equal perceptual differences. In fact, the perceptual differences between neighboring colors is probably not the same, but decreases for pairs of colors that are further away from the background color (Miyahara, Smith, & Pokorny, 1993). The two open circles in the plot indicate the chromaticities of the achromatic and the red background colors. One of the stimuli has the same chromaticity as the achromatic background, and one has the same chromaticity as the red background. All of the target and distractor stimuli were the same luminance (7 cd/m²), but were much brighter than the background (5 cd/m²). Therefore, all of the stimuli were easily visible against the background regardless of their chromaticities. Seven pairs of colors were constructed from the five stimulus colors labeled 1–5 in the figure. 

Four pairs consisted of each color paired with its nearest neighbor (i.e., 1 & 2, 2 & 3, 3 & 4, and 4 & 5). Three additional pairs with a larger chromatic difference between the members of the pair were made by pairing colors 1 & 3, 2 & 4, and 3 & 5. These seven pairs of colors were used as stimulus colors against both achromatic and red background colors. The two colors in each pair served alternately as target and distractor colors on each background. Figures 2 and 3 illustrate the appearance of one pair of colors on the achromatic and red backgrounds. 

Figure 2

 

Illustration of the appearance of a more saturated target among less saturated distractors from Experiment 1 on the achromatic and red backgrounds. Not drawn to scale.

Figure 3

 

Illustration of less saturated target among more saturated distractors from Experiment 1 on the achromatic and red backgrounds. Not drawn to scale.

Three observers, NB, MC, and ALN, completed two blocks of 60 trials for each target color on each background. The mean log response time for the 80 target-present trials from these two blocks was calculated for each observer. Data from the 40 target-absent trials are not shown, though in general they follow the same trends as the target-present trials. Because results were similar for the three observers, mean results across observers are shown. Figure 4 shows results for color pairs with the smaller chromaticity differences, and Figure 5 shows results for the larger chromaticity differences. Only three color pairs are shown in Figure 4, because some observers found the fourth color pair (colors 4 & 5) too difficult to complete while maintaining 90% accuracy. Error bars indicate 95% confidence intervals for the mean of the three observers. The red squares indicate that the more saturated stimulus in each pair of colors served as the target among less saturated distractors, and white circles indicate that the less saturated stimulus served as the target among more saturated distractors. In each figure, the upper panel shows results obtained on the achromatic background field and the lower panel shows results on the red background. 

Figure 4

 

Experiment 1, smaller chromaticity differences [target/distractor pairs (1,2), (2,3), and (3,4) in Figure 1]. Mean log search times plotted against the lower L chromaticity in each pair of stimuli. Results on the achromatic background are shown in the upper panel, and results on the red background are shown in the lower panel.

Figure 5

 

Experiment 1, larger chromaticity differences [target/distractor pairs (1,3), (2,4), and (3,5) in Figure 1]. Mean log search times plotted against the lower L chromaticity in each pair stimuli. The upper panel shows results on the achromatic background, and the lower panel shows results on the red background.

Evidence of an asymmetry is clear in both figures. In the upper panel of each figure, the mean response time was longer when the less red stimulus served as the target. In the lower panel of each figure, the asymmetry is reversed. When the stimuli were presented against the red background, the mean response times were longer when the redder stimulus served as the target except for the color pair on the right in Figure 5, where there was little difference in mean response times. The mean response times also tended to increase as the target and distractor chromaticities were moved away from the chromaticity of the background, indicating that the search was more difficult and that the target and distractor colors were probably less discriminable. The size of the asymmetry also tended to vary with the difference between the stimulus colors and the background colors. It was smallest when the target and distractor colors were most similar to the background color and increased with increasing difference between the stimulus colors and the background colors. These secondary effects of the change in chromaticity difference between the background and target/distractors will be discussed further in the section on modeling color search. The main effect is clear: changing the background color from a dark gray to a dark red as in this experiment reverses the previously known search asymmetry regardless of the difficulty of the search. Mean log search times are typically longer in Figure 4 than in Figure 5, indicating that search was more difficult when the color difference between target and distractors was smaller. 

Experiment 2: Luminance contrast affects color search even when target and distractors differ only in chrominance

In Experiment 1, varying the difference in chrominance between the stimuli and the background and changing the chrominance of the background were shown to affect search performance. In Experiment 2, we wanted to determine whether changing the luminance of the stimuli could affect color search when the target and distractors differed only in their chrominance. Experiment 2 was similar to the first experiment except that the stimuli were less luminous than the background, and there was lower luminance contrast between the target/distractor pairs and the background. The luminance of the stimuli was reduced to 4 cd/m² while the backgrounds remained at a luminance of 5 cd/m². The color pairs with the small chromaticity differences (i.e., pairs consisting of colors 1 & 2, 2 & 3, 3 & 4, and 4 & 5 in Figure 1) were used as stimulus colors on the same achromatic and red backgrounds. The first three color pairs were used with the achromatic background, and the last three color pairs were used with the red background because some observers found the color pair most distant from the background too difficult to complete while maintaining 90% accuracy. Figures 6 and 7 illustrate the appearance of one pair of colors on the white and red backgrounds. 

Figure 6

 

Illustration of more saturated target among less saturated distractors in Experiment 2 with stimuli dimmer than the background. Not drawn to scale.

Figure 7

 

Illustration of less saturated target among more saturated distractors in Experiment 2 with stimuli dimmer than the background. Not drawn to scale.

The same three observers, NB, MC, and ALN, completed two blocks of trials for each condition. Again mean results are shown because results for different observers were similar. The results shown in Figure 8 are in general similar to those shown in Figures 4 and 5. On the achromatic background (upper panel), response times were longer when the less red stimulus served as the target color. On the red background, the asymmetry reversed and the response times were longer when the redder stimulus served as the target stimulus. Response times again increased as the chromaticity difference between the stimulus colors and the background color was increased, and the magnitude of the asymmetry again increased with the difference between the stimulus colors and the background color. Overall, the magnitude of the asymmetry tends to be larger in Figure 8 (mean of .42 log units across all conditions) than in Figure 4 (mean of 0.25 log units across all conditions), though the chromaticity differences between the target and distractor stimuli were the same in the two figures. This result, taken by itself, is ambiguous as to whether the reversed sign of contrast or the reduced luminance difference is the major cause of the increase in search asymmetry. Taken with earlier results (Nagy & Cone, 1996), in which the luminance contrast with the background was even larger, it appears that much of the effect is due to the change in magnitude of the luminance contrast, as opposed to a change in sign of the contrast. However, further experiments would need to be done to confirm the relative importance of sign and magnitude of luminance contrast. 

Figure 8

 

Mean log search times for Experiment 2, plotted against the lower L chromaticity for stimuli that are less luminous than the background. The upper panel shows results on the achromatic background, and the lower panel shows results on the red background.

Experiment 3: Changing the background can make a color search asymmetry disappear

Experiment 2 showed that changing the luminance of the background affected color search even when the target and distractors differed only in chrominance. In Experiment 3, we explored whether a change of background color along one cardinal axis would affect search when the target and distractors differed only along the other cardinal axis. 

In this condition, targets and distractors differed from each other only in S chromaticity, whereas the backgrounds differed from each other only in L chromaticity. Because the L and S axes are thought to represent independent color mechanisms (Lennie & D’Zmura, 1988), it would be expected that changing the background color would have no effect on performance in this condition. Signals in the S cardinal mechanism must be used to distinguish between target and distractor stimuli, and the two background colors produced approximately the same excitation in the S cardinal mechanism. On the hypothesis that observers ignored signals in the L cardinal mechanism, which are irrelevant to the task, and attend only to signals in the S cardinal mechanism, it would be expected that changing the background color from achromatic to red should have no effect on performance. 

In this experiment, the stimulus colors were chosen from the S cardinal axis, as shown in Figure 9. The Xs represent the chromaticities chosen as target/distractor colors, and the open circles represent the achromatic and red background chromaticities, which were the same as those used in Experiments 1 and 2. Color pairs consisted of colors 1 & 3, 2 & 5, and 4 & 6. We increased the chromaticity difference between the colors in a pair as the pair of colors was moved away from the locus of white (.666, 0.016) to keep the difficulty of the search more nearly constant as the S chromaticity of the pair of colors was increased on the achromatic background. Again, each member of each color pair served as target and distractor on both the achromatic background and on the red background. 

Figure 9

 

Chromaticities of stimuli and backgrounds used in Experiment 3 are plotted in the cone excitation diagram. Stimulus colors are represented by Xs and backgrounds are represented by open circles.

On the achromatic background, the stimuli with the lowest S chromaticity appeared approximately achromatic and the stimuli looked increasingly violet in hue as the S chromaticity was increased. Figures 10 and 11 illustrate the appearance of one color pair on the achromatic and red backgrounds. Five observers (KN, TY, DS, MA, and AN) completed two blocks of 60 trials for each target color on each background. Mean log response times for target present trials were calculated for each observer. Mean results for the five observers are shown in Figure 12. Again error bars indicate 95% confidence intervals on the mean across observers. The blue squares indicate results when the more violet stimulus in each pair served as the target, and the white circles indicate results when the less violet stimulus served as the target. The upper panel shows results for stimuli presented against the achromatic background, and the lower panel shows results for stimuli presented against the red background. The results in the upper panel indicate a clear asymmetry consistent with previous results for an achromatic background (Nagy & Cone, 1996). Response times were longer when the less violet stimulus served as the target, and the magnitude of the asymmetry increased as the chromaticities of the stimuli were moved away from the background chromaticity. The mean asymmetry across color pairs is approximately 0.27 log units, about the same size as in Figure 4. 

Figure 10

 

Experiment 3. Illustration of less saturated target along S axis among more saturated distractors on achromatic and red backgrounds. Not drawn to scale.

Figure 11

 

Experiment 3. Illustration of more saturated target along S axis among less saturated distractors on achromatic and red backgrounds. Not drawn to scale.

Figure 12

 

Experiment 3. Mean log search times plotted against the lower S chromaticity in each stimulus pair. The upper panel shows results on the achromatic background, and the lower panel shows results on the red background.

In contrast, the lower panel shows little evidence of an asymmetry, and there was also little effect of changing the chromaticity of the stimulus colors. The mean difference in log response times for the more and less violet target colors was only 0.02 log units. Changing from achromatic to red backgrounds appeared to have a clear effect on the asymmetry, even though the two backgrounds were matched for excitation in the cardinal color mechanism that must have been used to distinguish target and distractor stimuli. 

Experiment 4: Changing the background color can induce a color search asymmetry

Experiments 1–3 have shown that changing the background color can reverse an existing search asymmetry, or make it disappear. In this experiment, we explore whether changing the background color can induce an asymmetry in search for hue. 

Nagy and Cone (1996) found little evidence of an asymmetry for target and distractor stimuli that differed in hue when they were presented against a neutral dark background. In our final experiment, we chose target and distractor stimuli that differed primarily in hue and were similar in chromatic saturation. The stimuli were presented against the same achromatic and red backgrounds used in the first three experiments. The stimuli that served as target and distractor stimuli varied in appearance from a fairly saturated red that was the same chromaticity as the red background field to a fairly saturated blue. Though we did not attempt to equate the apparent saturation of the these stimuli precisely, we did try to chose the end points of the line from which the stimulus colors were chosen so that they were approximately equally saturated on the basis of data from previous search experiments. Thus the stimuli differed primarily in hue with only minor variations in saturation when viewed against the achromatic background. The chromaticities used for target and distractor colors are represented by Xs in Figure 13. The achromatic and red background chromaticities are again represented by open circles. Color pairs consisted of colors 1 & 2, 2 & 3, and 3 & 4. Figures 14 and 15 illustrate the appearance of one color pair on the achromatic and red backgrounds. 

Figure 13

 

Chromaticities of the stimuli and backgrounds in Experiment 4 are plotted in the cone excitation diagram. Stimulus colors are represented by Xs, and background colors are represented by open circles.

Figure 14

 

Illustration of the redder target stimulus among bluer distractors on the achromatic and red backgrounds in Experiment 4. Not drawn to scale.

Figure 15

 

Illustration of a bluer target among redder distractors on the achromatic and red backgrounds in Experiment 4. Not drawn to scale.

Again, each member of each color pair served as target and distractor color on each background. Three observers (CE, PA, and DS) completed two blocks of 60 trials for each condition. The mean log response time was again calculated for the 80 target present trials from the two blocks. Mean results from the three observers are shown in Figure 16. Blue squares indicate results obtained when the bluer stimulus in each pair served as the target, and red circles indicate the results when the redder stimulus served as the target. Error bars indicate 95% confidence intervals. The upper panel shows results for stimuli presented against the achromatic background, and the lower panel shows results for stimuli presented against the red background. On the achromatic background (upper panel), there is little evidence of an asymmetry. The mean log response times were slightly longer (.04 log units or about 10%, not statistically significant) when the bluer stimulus served as the target, but this is a small difference compared to the asymmetries obtained in the previous experiments. The response times for the reddest pair of colors (colors 1 & 2 in Figure 13) were slightly longer than the response times for the bluest pair of colors (colors 3 & 4), but this difference was again small (approximately 0.15 log units, not statistically significant). 

Figure 16

 

Experiment 4. Mean log search times plotted against the lower S chromaticity in each stimulus pair. Results for the achromatic background are shown in the upper panel, and results for the red background are shown in the lower panel.

Results on the red background (lower panel) show clear evidence of an asymmetry for two of the three color pairs. For the reddest pair of colors, most similar to the red background (colors 1 & 2 in Figure 13), there was no asymmetry. However, for the other two color pairs, the magnitude of the asymmetry was again quite large (0.25 log units or more). As in previous experiments, the asymmetry in response times increased as the pair of stimulus chromaticities was moved away from the background chromaticity. Thus, changing to a red background introduced an asymmetry for color pairs that differed primarily in hue and produced no asymmetry on an achromatic background. 

Discussion and modeling of experimental results

For simplicity in discussing the results of these experiments, we use the following notational convention. We refer to an experiment by the trio Background (target, distractors), where “Background,” “target,” and “distractors” represent in words the colors used for these three components of the stimuli. These color words do not represent precisely the colors used, but rather are a mnemonic to capture the essence of the experimental condition. For example, in Experiment 1, we run the conditions Gray (red, pink), Gray (pink, red), DarkRed (red, pink), and DarkRed (pink, red) (i.e., for both gray and red backgrounds, observers search for both a red target among less saturated pink distractors, and a pink target among more saturated red distractors). 

Results clearly show that color search asymmetries depend on the color of the background. Changing the background color can reverse the direction of an asymmetry, abolish asymmetries that occurred when the background was achromatic, and introduce asymmetries where there were none. Asymmetries were often much larger when stimuli were presented against a luminous background than when stimuli were presented against a dark background (Nagy & Cone, 1996) and depend on the luminance contrast between the stimuli and the background. This can be true even when luminance is not a useful cue for distinguishing between the target and distractors. Two different backgrounds that have the same effect on the cardinal color mechanism that one might assume would be used to discriminate target and distractor stimuli (Experiment 3) produce different asymmetries. These results suggest that signals in all three cardinal mechanisms influence the asymmetries and that the cardinal mechanisms do not act independently in the search process. Either observers do not attend only to signals in the cardinal color mechanism that is used to discriminate target and distractors or signals in the cardinal mechanisms interact under the conditions of the search experiments. 

Results support the hypothesis that asymmetries in color search are dependent on the relationship between the stimuli and the background against which the stimuli are viewed, even when the stimuli are easily visible against the background. Any explanation of asymmetries in color search will need to include this relationship as a key component. 

Given that a model of search mediated by independent cardinal color mechanisms seems not to predict these results, one might ask what sort of model does make these predictions. Certainly, the model must take into account the background color relative to the target and distractor colors, as well as the difference between the target and distractors. This dependence on the background color means that most existing models of visual search will not explain these results, as most such models ignore the background color. However, a number of models can probably be adapted to take into account the background color and thus predict these results. We will demonstrate this for two models in the following section. 

The model must explain the following main effects that result from changing the background color from achromatic to red:

•  
  Reversal of the search asymmetry when searching for a saturated reddish target among less saturated distractors of approximately the same hue. [Experiments 1 & 2: Gray (red, pink) is easier than Gray (pink, red), but DarkRed (pink, red) is easier than DarkRed (red, pink).]
•  
  Inducement of a search asymmetry when searching for a target that differs from distractors in hue. [Experiment 4: DarkRed (blue, red) is easier than DarkRed (red, blue), but Gray (blue, red) ≈ Gray (red, blue).]
•  
  Elimination of the search asymmetry when searching for a saturated violet target among less saturated distractors of approximately the same hue. [Experiment 3: DarkRed (blue, desatBlue) ≈ DarkRed (desatBlue, blue), but Gray (blue, desatBlue) is easier than Gray (desatBlue, blue).]

In addition, a model must explain the additional effects:

•  
  Increased search difficulty with increased chromaticity difference between the background and the target-distractor pair (Experiments 1 and 2, and Experiment 4 against a DarkRed background).
•  
  Increase in asymmetry size with increasing chromaticity difference between the background and target-distractor pair, sometimes followed by a decrease in asymmetry size for even larger chromaticity differences with the background (Experiments 1 and 2 — see Figure 17 for a crude measure of the size of the search asymmetry for these experiments — and Experiments 3 and 4, for those conditions where an asymmetry exists).
•  
  Decrease in asymmetry size with increasing luminance difference between the background and the target distractor pair (Experiments 1 and 2, see Figure 17).

Figure 17

 

Ratios of the response times for the more and less saturated targets are plotted against L chromaticity. Square symbols indicate high-contrast stimuli of Experiment 1, and circles indicate low-contrast stimuli of Experiment 2. Error bars indicate 95% confidence intervals.

It is important to note that all of these main and secondary effects are specific to the relationships between the target-distractor pairs and the background colors used for the particular experiments presented in this work. 

In this section, we suggest two candidate models and show that both of them qualitatively predict all but one of the main and secondary experimental results of this study. The first model is the saliency model (Rosenholtz, 1999, 2001a) that originally motivated these experiments. The second model is a modification to a signal detection theory model of visual search. 

Models

Here we give simple intuitive explanations of both the saliency model and the modified signal detection theory (SDT) model. For more details on the saliency model, see Rosenholtz (1999, 2001a). For more details on the basic SDT model, see Palmer, Ames, and Lindsey (1993) and Palmer, Verghese, and Pavel (2001). 

The saliency model

In its simplest form, the saliency model says that it is easier to search for an item if its features are “unexpected,” given the distribution of features in the surrounding display. In the displays typical for visual search experiments, the locations of the target and distractors are chosen randomly from the set of possible item locations. For such displays, the intuition is that a target is easy to search for if its features are essentially “significantly different” from the mean feature in the display, or, put differently, if the target features are “outliers” to the local distribution of features. In particular, to determine the predictions of the saliency model, one first calculates the mean μ and covariance Σ of the distractors, and then computes the saliency, Δ:  

 

(1)

where T is the target feature vector, and (T−μ)’ represents the vector transpose of (T−μ). Observation noise can be incorporated into the covariance matrix, Σ. The simple version of the saliency model predicts that the higher the saliency, Δ, the easier the search task. 

Rosenholtz (2001a) suggested that for color search, the saliency model should count the background as a distractor, with its weight (how many distractors it “counts as”) perhaps proportional to the area of the background relative to that of the other distractors. For the purposes of the predictions in this work, we account for the background in this way. 

For the purpose of making qualitative predictions from the saliency model, one can just look at the magnitude of the saliency, Δ, for different search conditions. We will sometimes do this visually, as shown in Figure 18. Here we plot the target as a T, the distractors as an X, and the background as “B,” all in an appropriate uniform feature space with coordinates (x, y). A uniform feature space is ideally one in which equal distances denote equal discriminability of features. For color, our best guess for such a feature space is something like the CIELUV space, though for the purposes of this work, in which we are typically making qualitative predictions, we will often depict a more intuitive color space. The solid ellipse shown is the 1 σ covariance ellipse. This is a saliency iso-contour; all points along this curve have a saliency of 1. We will sometimes also plot the 2 σ covariance ellipse, shown in a dashed line, for which all points have a saliency of 2, and so on. These ellipses mark distance from the mean of the distractors, and the farther outside these ellipses the target lies, the easier the predicted search. In Figure 18, the target lies just outside the 2 σ covariance ellipse, so the saliency is slightly greater than 2. This suggests moderately easy search. To check the predictions of the saliency model for our experiments, we also need a qualitative measure of the size of the predicted search asymmetry. For that we use the ratio of the saliency when one element of the target-distractor pair is the target versus the saliency when the other element of the pair is the target. 

Figure 18

 

A graphical representation of the saliency model. Plotted are the target (T), distractors (X), and background (B) in a uniform feature space with coordinates (x, y). The solid ellipse shows the 1 σ covariance ellipse, along which the saliency is 1. The dashed ellipse shows the 2 σ covariance ellipse, for which all points have a saliency of 2. The farther the target lies outside these ellipses, the higher the saliency, and the easier the predicted search.

The background contrast signal detection theory model

In the basic SDT model, an observer makes noisy, independent observations of the features of elements in the display. The intuition to use is that the basic SDT model predicts that search becomes more difficult the more likely it is that the observation noise causes the observer to mistake one of the distractors for the target. So, as the target and distractors become more similar, or as the noise in the observations increases, the SDT model predicts more difficult search. 

It is clear from our experiments that one must somehow incorporate the background color into any model of color search. One of the open questions in implementing the SDT model is how to set the observation noise. One way of doing this is to make the noise in observing a given target or distractor color proportional to the difference between that color and the color of the background. This is essentially the “multiplicative noise” of Lu and Dosher (1998). We call the SDT model with this noise model the background contrast signal detection theory (BCSDT) model. 

Sutter, de la Cruz, and Sheft (2000) have shown that the SDT model can predict the search asymmetry that it is easier to search for an oblique line among horizontal distractors than vice versa. This prediction follows naturally from running the SDT model with greater noise in observing an oblique line than in observing a horizontal line. In general, one can take this as a rule of thumb: if there is more noise in observing feature A than in observing feature B, SDT will predict that search for A among B is easier than search for B among A. 

This rule of thumb suggests that a useful qualitative model of the size of a search asymmetry is the ratio of the observation noise for one element of the target-distractor feature pair to the observation noise in the other member of the pair. Without loss of generality, we will look at the ratio of the more noisy element to the less noisy. Then a ratio close to 1 predicts little search asymmetry, whereas a large ratio predicts a large search asymmetry. 

Predictions

Main effects: reversing, creating, and eliminating a search asymmetry

The first main effect that any viable model must predict is the reversal of the asymmetry shown in Experiments 1 and 2. Gray (red, pink) was easier than Gray (pink, red), whereas DarkRed (pink, red) was easier than DarkRed (red, pink). 

First, consider the asymmetry between Gray (red, pink) and Gray (pink, red). Rosenholtz (2001a) has already demonstrated that the saliency model can explain this asymmetry. We reproduce the relevant pictures in Figure 19a and 19c. These diagrams show feature space representations of the search experiment, as described in the section “The saliency model.” The ellipses indicate the Δ=1 saliency iso-contours. The more saturated target falls farther outside this ellipse than the less saturated target, and thus the saliency model predicts easier search for the more saturated target. The predictions for search against a red background should be clear. Figure 19b and 19d are essentially mirror reflections of Figure 19a and 19c, respectively. The saliency model now predicts easier search for the less saturated target. 

Figure 19

 

Saliency model depictions of search for a unique saturation. (a) and (d) show search for a more saturated red target among less saturated pink distractors. (b) and (c) show search for a less saturated pink target among more saturated red distractors. (a) and (c) show search against a dark gray background, whereas (b) and (d) show search against a dark saturated red background. In (a) and (b), the target lies well outside the 1 σ covariance ellipse, and the saliency model correctly predicts easy search. In (c) and (d), the target lies close to the 1 σ covariance ellipse, and the saliency model correctly predicts more difficult search.

The BCSDT model can also explain these results. In Gray (red, pink) and Gray (pink, red), the more saturated red color is more distant from the background than the less saturated pink color. The BCSDT model assumes, therefore, more noise in observations of the saturated color than the unsaturated, and based on the results of Sutter et al. (2000), the BCSDT model predicts that Gray (red, pink) is easier than Gray (pink, red). On the other hand, in DarkRed (red, pink) and DarkRed (pink, red), the more saturated color is more similar to the background and so has less observation noise. Thus the model predicts that DarkRed (pink, red) is easier than DarkRed (red, pink). Both the BCSDT and saliency models predict the reversal of the asymmetry. 

The second main effect was the inducement of an asymmetry in search for a target that differs from the distractors in hue, in Experiment 4. First consider Gray (red, blue) and Gray (blue, red). There is little difference in feature space (see Figure 20a & 20b) between the relationship between the gray background and the bluish versus reddish elements of the target-distractor pair, and both models predict little or no asymmetry. However, against a dark red background (Figure 20c & 20d), the situation again looks like that of Experiments 1 and 2, and both models predict the asymmetry that DarkRed (blue, red) is easier than DarkRed (red, blue). 

Figure 20

 

Saliency model depictions of search for a target that differs from distractors in hue. Distance from the origin represents saturation, whereas angle represents hue. Depth into the page (not shown) indicates luminance. (a) and (b) show search against a gray background. The saliency model correctly predicts no asymmetry. (c) and (d) show search against a dark saturated red background. The saliency model correctly predicts easier search for the bluer target in (c) than for the redder target in (d).

Now consider the final main effect in Experiment 3: the elimination of an asymmetry between DarkRed (blue, desatBlue) and DarkRed (desatBlue, blue), when there exists an asymmetry between Gray (blue, desatBlue) and Gray (desatBlue, blue). In the case of a gray background, the feature space representation of the experiments looks much like Gray (red, pink) and Gray (pink, red) (Figure 19a & b), and both models predict the asymmetry that Gray (blue, desatBlue) is easier than vice versa. 

In the case of a dark red background, both models can predict the lack of asymmetry if the background is sufficiently different from the target-distractor pair in terms of both luminance and chrominance, relative to the difference between the target and distractors. For both models, the relationship of the background to one element of the target distractor pair would, in this case, be very similar to the relationship of the background to the other element of the pair. Thus the models would predict little or no asymmetry between DarkRed (blue, desatBlue) and DarkRed (desatBlue, blue). 

Certainly, the difference between the dark red background and the target-distractor pair is greater than in Experiment 1, in which we saw asymmetries in search for a unique saturation. After all, in Experiment 3, the background differs from the target-distractor pair in hue, luminance, and saturation, whereas in Experiment 1, the difference is only in luminance and saturation. However, when we convert the colors used in our experiments to a more uniform color space (e.g., CIELUV), it seems that the dark red background is unlikely to be sufficiently more distant from the target-distractor pairs in Experiment 3 than from the pairs in Experiment 1, though the distances are somewhat larger in Experiment 3. If the models predict an asymmetry in Experiment 1, they would likely predict the asymmetry that DarkRed (blue, desatBlue) would be easier than DarkRed (desatBlue, blue). Our inability to predict these results with the simple versions of either the saliency or BCSDT models, when these models predict all of the other major and secondary effects (see below) in our experiments, may be due to things such as CIELUV space being a uniform color space over only very short distances, though this seems unlikely. We discuss more intriguing explanations in the Conclusions. 

Secondary effects: results of varying luminance and chrominance difference between elements and the background

In addition to what we have dubbed the “major” effects of our four experiments, a model needs to explain three secondary effects. The first of these is the effect that increasing the chromaticity difference between the target-distractor pair and the background increases search difficulty. We see data consistent with this trend in all of the experiments, though it is most significant in Experiments 1 and 2, and against a dark red background in Experiment 4. That the BCSDT model predicts this is evident from simple intuition; increasing the chrominance difference with the background means greater observation noise for both target and distractors. This increase in observation noise implies more difficult search, because it is more likely that the target will be confused with one of the distractors (see the section describing the BCSDT model). 

That the saliency model also explains this effect is best shown using the pictures in Figure 21. Here we show the Δ=1 and Δ=1.5 saliency iso-contours. As the chromaticity difference with the background increases, the target lies closer to the Δ=1.5 iso-contour, implying lower saliency and increased search difficulty. 

Figure 21

 

Saliency model depictions of the effect of increasing the chrominance difference between the target/distractor pair and the background. Increasing the chrominance difference (b) causes the target to lie closer to the outer, 1.5 σ covariance ellipse, correctly predicting increased search difficulty when compared with the low chrominance difference in (a).

The next two secondary effects concern the size of the asymmetry. In the experiments, we measure the size of the asymmetry by taking the ratio of the reaction time for the more difficult search task to the reaction time for the easier search task. The saliency and BCSDT models model asymmetry size as described above in the description of the models. The first effect on asymmetry size that a model must predict is the increase (and then possible decrease) in asymmetry size with increased chrominance difference between the target-distractor pair and the background, as found in Experiments 1 and 2 (where an asymmetry is present in Experiments 3 and 4, one can also see this effect). The square symbols in Figure 17 show the asymmetry size for Experiment 1, whereas the circle symbols show the asymmetry size for Experiment 2. 

Figure 22 shows qualitative predictions of both the saliency (Figure 22a) and BCSDT (Figure 22b) models, indicating that they can predict this increase and then decrease of the asymmetry size as a function of chrominance difference. Intuitively, in the set up for Experiments 1 and 2, for no chrominance difference, there is little difference between the relationship between the background and one element of a target-distractor pair versus the other. The target-distractor pair is close to being symmetric about a line from the background to the midpoint of the pair. As the chrominance difference increases, this induces a geometric asymmetry between the relationship of the background to one element of the target-distractor pair versus the other, and we expect the search asymmetry size to increase. However, for a really large chrominance difference with the background, relative to the difference between the target and distractors, again the target and distractors have very similar geometric relationships to the background. 

Figure 22

 

Asymmetry size versus chrominance difference between target and distractor pair and background, for the saliency (a) and BCSDT (b) models. Note these predictions are qualitative, and should not be expected to exactly match the plots shown in Figure 15.

This is akin to the argument that when one views objects at great distances, the perspective may be well approximated by orthographic projection. 

The last secondary effect that a model must predict is the difference between Experiment 1 and Experiment 2: a decrease of luminance difference between the background and the target-distractor pair increases the size of the search asymmetry. Figure 17 shows the asymmetry size for Experiments 1 and 2 (compare square and circle symbols). Figure 23 shows qualitative predictions of the saliency (Figure 23a) and BCSDT (Figure 23b) models. Again, we can see that either candidate model can predict this secondary effect. The intuition here is essentially the same as the intuition in the previous paragraph. As one increases the luminance difference between the target-distractor pair and the background, the relationship between the target and the background becomes more similar to the relationship between the distractors and the background. As one decreases the luminance difference without changing the difference between the target and distractors, any asymmetry is enhanced because the target-background geometry becomes more different from the distractor-background geometry. 

Figure 23

 

Asymmetry size versus luminance difference between target/distractor pair and background, for the saliency (a) and BCDST (b) models. Again these predictions are qualitative.

Both of our candidate models, the saliency model and the BCSDT model, can qualitatively predict all secondary effects observed in our experi-ments. It should be noted, however, that earlier work has shown examples of cases in which the saliency model performs better than the basic SDT model for orientation search in dense, highly heterogeneous displays (Rosenholtz, 2001b). A similar argument can be made for motion search. 

Conclusions

Our experiments demonstrate that the color of the background matters in color search. Any model of visual search must take this into account. Previous work has also suggested that the background motion may matter in motion search and that the background orientation may matter in orientation search (for a review, see Rosenholtz, 2001a). We have presented two simple models, the saliency model and the BCSDT model, which predict virtually all results, both the major and the secondary effects, of our experiments. 

Most existing models of visual search cannot predict these results simply because they do not take into account the background color in color search tasks. However, many such models can probably be simply adapted to consider the background color, and thus predict the majority of our results, as with the BCSDT model. 

One might ask, then, whether these results pose particular problems for any models of visual search. On the surface, it might seem more likely that our results would be inconsistent with suggestions that attention is object-based, but in fact the results pose more problems for some of the simple feature-based models of search. 

Models of object-based attention suggest that objects in the display are segmented from the background in a single, pre-attentive step, and that search then proceeds only among those segmented objects. There have been a number of studies showing an advantage for deploying attention within an object (e.g., Duncan, 1984; Egly, Driver, & Rafal, 1994), thus demonstrating that objecthood is an early feature for attention. Furthermore, Wolfe et al. (2002) have shown that the addition of a cluttered background to a search display does not seem to increase effective set size, thus suggesting an early segmentation of target and distractors from even highly cluttered backgrounds. 

The fact that the background color has an effect on search can be consistent with all but the most restrictive object-based attentional models. Neither our results, nor either of the models presented here, require that the visual system fail to segment the target and distractors from the background, nor search the background for the target. The results can instead be explained by assuming that the background color can affect the perception of the target and distractors. For example, the background color could affect the internal noise in perceiving the target and distractor colors, as in the BCSDT model. Alternatively, the background color could be taken into account in computing color statistics for the display, and thus degree of interest for the target, as with the saliency model. Though they have very different stimuli and search tasks, Wolfe et al. (2002) in fact suggest, based on modeling of their results, that “when the background becomes more complex, it seems to take longer to accumulate the information required to identify a selected object. Perhaps the separation of background from item is imperfect and enough background gets included to make identification more difficult.” Our results here are compatible with such conclusions. 

Certain feature-based models of visual search do not fare as well. In the past, studies of visual search have often been consistent with the idea that observers can attend to the feature-coding mechanism that best differentiates target and distractors and ignore activity in other feature coding mechanisms. Depending on the suggested feature-coding mechanism, our results can pose problems for this sort of simple model. 

As an example, many previous psychophysical studies of human color vision have suggested that the cardinal axes in color space represent independent color-coding mechanisms under many conditions (e.g., Nagy, 1999; Nagy & Winterbottom, 2000). Comparisons of the results in Experiments 1 and 2 and the results of Experiment 3 both suggest that signals in different cardinal color mechanisms interact to determine search performance. These experiments are inconsistent with either the notion that observers attend only to signals in the single cardinal color mechanism that best differentiates target and distractors, or the notion that the cardinal axes represent independent color-coding mechanisms. Our results are consistent, however, with other studies that do show evidence of interactions between the cardinal mechanisms. For example, studies of masking (Sankerelli & Mullen, 1997) and color contrast adaptation (Zaidi & Shapiro, 1993; Webster & Mollen, 1994; Singer & D’Zmura, 1994) have suggested that interactions between these mechanisms can occur under some conditions. 

The one result that the saliency model and the BCSDT model seem unlikely to be able to fully explain is the lack of an asymmetry in Experiment 3, between DarkRed (desatBlue, blue) and DarkRed (blue, desatBlue). As both the saliency and BCSDT models explain all the other major and secondary effects, it is worth considering the implications for the apparent inability to explain this lack of asymmetry. The fact that the models predict all the results of the experiments except for this one may be a signal that something interesting is going on in this experiment. 

The main purpose of this work is to suggest that the background matters in visual search experiments, and this has certainly been born out by the results of each of the experiments. However, the lack of an asymmetry between DarkRed (desatBlue, blue) and DarkRed (blue, desatBlue) looks like the sort of result we would expect if the background color did not matter. Ignoring the background color, this pair of experiments is completely symmetrically designed. 

Furthermore, it is interesting that this one experimental pair whose result most looks like it would if the background did not matter is also the experimental pair in which the background is most different from the target-distractor pairs. This dif-ference is both metric-the distance in feature space between the DarkRed background and the blue/desatBlue pair is greater than the distance in other experiments reported in this work-and also categorical and semantic. In Experiment 1, for instance, the background is either DarkRed or Gray, and the target/distractor pairs range from a lighter gray to a lighter red. In Experiment 3, on the other hand, the background is DarkRed and the target/distractor pairs range among different shades of blue. 

Perhaps the lack of a DarkRed (blue, desatBlue) versus DarkRed (desatBlue, blue) asymmetry points to more of a mixed model, in which the background matters for some stimuli more than others. Perhaps, as in the saliency model, the background sometimes counts as a distractor, but counts less if the background is somehow segmented out from the elements — if the background is seen as fundamentally, perhaps categorically, different in some way. A red background seems relevant when the items are red differing in saturation, but maybe does not seem as relevant if the items are blue, differing in saturation. This opens the door for the possibility of all sorts of grouping phenomena that might come into play here (see Schirillo & Shevell, 2000, for evidence that grouping can affect color appearance). For example, what happens if the background is seen at a different depth from the stimulus elements? Will it be less likely to count as a distractor, or otherwise influence the color search task? 

Acknowledgments

This work was partially supported by National Eye Institute Grant RO1EY12528. 

Commercial relationships: none. 

Corresponding author: Ruth Rosenholtz. 

Email: . 

Address: Department of Brain & Cognitive Sciences, NE20-447, MIT, Cambridge, MA 02139.