Reliable identification by color under natural conditions

Abstract

In order to recognize objects on the basis of the way in which they reflect different wavelengths of light, the visual system must deal with the different illuminant and background conditions under which the objects are seen. To test this ability under natural conditions, subjects were asked to name 6 uniformly colored papers. The experiment started by showing subjects six papers simultaneously in a normally illuminated room, and instructing them about how to name them. The papers were easy to differentiate when seen together but they were so similar that subjects only identified 87% correctly when they were presented in isolation under otherwise identical conditions to those during the instruction. During the main part of the experiment subjects walked between several indoor and outdoor locations that differed considerably in lighting and background colors. At each location subjects were asked to identify one paper. They correctly identified the paper on 55% of the trials (well above chance level), despite the fact that the variation in the light reaching their eyes from the same paper at different positions was much larger than that from different papers at the same position. We discuss that under natural conditions color constancy is probably as good as it can be considering the theoretical limitations.

Introduction

The light that is reflected from an illuminated object depends both on its surfaces' reflectance properties and on the illumination of the scene. If we are interested in the object's reflectance properties the fact that the illumination can vary drastically over time and between locations raises a problem for our visual system, since the intensity and spectral distribution of the light that is reflected from the object in question onto the receptors in our eyes will also vary considerably (von Helmholtz, 1866). Extracting the influence of surface reflectance from the light reaching the eye is the problem of color constancy. The main advantages of having color vision are to enhance the detection and identification of objects in the environment (Gegenfurtner & Rieger, 2000; Wichmann, Sharpe, & Gegenfurtner, 2002). For detection, a shift in illumination need not be a problem, but for identification a failure in color constancy could be a hindrance. Color constancy has mainly been studied by matching the colors of two surfaces, which is quite an indirect test of color identification (Foster, 2003). Other methods that have been used to study color constancy include color naming (Hansen, Walter, & Gegenfurtner, 2007; Troost & de Weert, 1991), achromatic adjustments (Delahunt & Brainard, 2004a) and detecting reflectance changes at the time of a change in illumination (Foster & Nascimento, 1994). We here directly test object identification on the basis of color under natural circumstances. 

Probably many factors are involved in achieving color constancy, including various kinds of spatial (e.g., Brenner & Cornelissen, 1991; Brenner, Cornelissen, & Nuboer, 1989; Brenner, Granzier, & Smeets, 2007; Granzier, Brenner, Cornelissen, & Smeets, 2005; Land, 1964; Blackwell & Buchsbaum, 1988; Walraven, 1973) and temporal (e.g., Cornelissen & Brenner, 1991; Lennie & D'Zmura, 1988; Von Kries, 1905) comparisons. For detailed reviews of such factors see Hurlbert (1996, 1999) and Smithson ( 2005). 

The extent to which color constancy is achieved differs between studies, probably because factors such as overall scene complexity (Gelb, 1950; Gilchrist & Annan, 2002; Kraft, Maloney, & Brainard, 2002; Maloney & Schirillo, 2002), or perhaps specific aspects such as the three-dimensional structure (Adelson, 1993; Bloj, Kersten, & Hurlbert, 1999), specular highlights (D'Zmura & Lennie, 1986; Lee, 1986; Yang & Maloney, 2001; Yang & Shevell, 2003), mutual illuminations (Bloj et al., 1999; Delahunt & Brainard, 2004b), shadows (Usui, Nakauchi, & Takebe, 1996) and illuminant gradients (Brainard, Brunt, & Speigle, 1997) can all contribute to color constancy; and their presence differs between studies. Color constancy in the laboratory is often poorer than in our everyday experience, possibly because fewer such factors are available in laboratory displays. Even with all the above-mentioned factors available in a scene, color constancy cannot be perfect because human color vision is based on the comparison of signals of three types of cones in the retina. This constrains the identification of colored surfaces under different illuminations (Foster, Amano, Nascimento, & Foster, 2006; Nascimento, de Almeida, Fiadeiro, & Foster, 2004; Young, 1987) as can be observed when matching clothes. After careful scrutiny in a store a match is accepted under fluorescent lighting, only to experience great disappointment when leaving the store and discovering that the match is no longer acceptable in daylight. In this case, what was a perfect match under one illuminant (fluorescent) is not a perfect color match under another illuminant (daylight), because the reflectance in the store (the actual spectrum of the reflected light) was not the same, only the three receptor stimulations were the same. 

We here test color constancy under natural conditions using stimuli for which color constancy is likely to limit performance: when the differences in reflectance between the surfaces of interest are small so that changes in illumination have a relatively large influence on the light reaching the eyes. We therefore use quite similarly colored objects (here colored sheets of paper), but selected reflectances that are easy to categorize so that memory is not an issue. 

Methods

Subjects

21 subjects (including two of the authors) with normal color vision (Ishihara, 1969) took part in the experiment. This research was approved by the local ethics committee. 

Procedure

For practical reasons, the experiment was performed in three groups of seven subjects. During an ‘instruction phase’ subjects were told how to name the colors of six different test papers that were presented simultaneously on a desk under daylight illumination (see Figure 1). The test papers were white or very slightly gray, green, red, blue, or yellow. After the task was explained to a group of subjects, they walked a tour passing 24 different pre-selected locations (that were different for each group). Each subject had to identify one test paper at each of the 24 locations. At each location, all subjects within a group had to identify the same test paper. The papers were presented in random order but ensuring that each test paper was presented four times. Since there were three groups of subjects this meant that altogether each test paper was presented 12 times. 

Figure 1

 

The papers as first shown to the subjects during the instruction phase. Although this image is obviously not calibrated, it gives an impression of the difficulty of the task.

Subjects were not told that each paper would be presented 4 times. They wrote the name of the color of the paper that they thought was being shown to them on an answer form (“white,” “gray,” “green,” “red,” “blue” or “yellow”). At each location, the experimenter presented the test paper separately to each subject. Subjects were allowed to hold the test paper in their hands and change its orientation. They were allowed to look around as they pleased, so they could compare the test paper's color to the colors of objects in the direct vicinity, but were not allowed to compare the color of the test paper with their white answer form, and they had to remain at the place at which the experimenter had given them the test paper. Subjects were not allowed to talk about the experiment during the tour and were instructed to keep their answer form hidden from the other participants. 

The locations

We used both indoor and outdoor locations (see examples in Figure 2). Two walking tours were carried out inside and near the university campus of the VU University in Amsterdam. The third walking tour was carried out both inside and outside the first author's house located in the south of The Netherlands. The walking tours were performed on days in which the weather conditions were likely to make outdoor color constancy most difficult (blue sky with occasional clouds). About half of the locations were indoors while the other half were outdoors. There were locations in which only artificial illumination was present and ones in which the test papers were illuminated only by natural daylight, as well as ones in which both kinds of illumination were present (the instruction room was one of these). There were outdoor locations that were in the shadow of plants or buildings, and ones in direct sunlight. 

Figure 2

 

Examples of indoor (left) and outdoor (right) locations. The subject decides which paper he (top) or the experimenter (bottom) is holding.

The final location was the one that we had used for the instruction phase. We included this location to see whether subjects would be particularly good at recognizing the papers in the environment in which they had initially seen all the colors. The experiment took about 90 minutes for each group. 

Baseline measurement

Although the difference between the papers was very clear when they were presented simultaneously, identifying them in isolation was quite difficult. In a separate measurement, we tested our subjects' ability to identify the test papers at a fixed place under constant fluorescent illumination (Philips, 38 HF; 50 watt). Five subjects who also participated in the main experiment took part in this baseline measurement. The CIE _xy coordinates of the light reflected by the test papers under these conditions, as measured with a Minolta CS-100A chroma meter, were (0.436, 0.404), (0.432, 0.406), (0.439, 0.402), (0.426, 0.401), (0.441, 0.411) and (0.436, 0.405), for the gray, green, red, blue, yellow and white test paper respectively. 

The procedure was similar to that of the main experiment, but the background was always the same (the gray surface of a table), the illumination did not change between the first simultaneous presentation and the subsequent test presentations, and subjects remained at the same place under constant illumination between the presentations. Thus, performance is unlikely to be limited by failures of color constancy. After presenting all six pieces of paper simultaneously, the experimenter placed one of the six test papers on the table every three minutes, and the subjects had to write down which paper they thought was being presented (i.e. its color). As in the main experiment, each test paper was presented four times, and the papers were presented in random order (24 trials). The three minutes waiting time was chosen to match the time between judgments in the main experiment. Subjects remained in the room during the 3 minutes between presentations. 

Analysis

To illustrate the judgments that subjects made, pie charts of the groups of 7 subjects' responses were made per location and test paper. The corresponding color of the reflected light was indicated for each pie chart. Since subjects could move the papers around and the illumination could change slightly while the members of the group sequentially made their judgments, we measured the color of the reflected light several times at each location for each group (while the subjects were making their decisions) and calculated the average CIE _xyY values. These averages are shown together with the above-mentioned pie charts. The variability between these repeated measurements turned out to be quite small (the median standard deviations while a paper was shown to the 7 subjects of the second group were 0.002, 0.001 and 12.3% for CIE x, y and Y, respectively). 

That performance would not be perfect is obvious because we chose shades of colors that were difficult to distinguish. The question is to what extent performance is worse when the papers are shown at various locations with different kinds of illuminations than when the papers were shown at a single location under a fixed illumination (baseline). To find out, we plotted the percentage of correct responses as a function of the distance in CIE color space between the test papers' CIE _xy coordinates during the experiment and during the corresponding instruction phase. We averaged consecutive groups of 6 presentations (a presentation is a set of 7 responses for a given combination of paper and illumination) after sorting the presentations in terms of the above-mentioned distance. 

Results

The average number of correct responses during the main experiment was 55.8% (ranging between 37.5% and 79.2% for individual subjects; 16.6% is chance level). During the baseline measurement, in which there was no change in illumination or in background color and the subjects were fully adapted to the illumination, 87.5% of the responses were correct. That subjects made errors under these conditions demonstrates how difficult it was to distinguish between our papers' colors. The 5 subjects who participated in the baseline measurement did not perform any better than the other subjects in the main experiment (52.5% correct). 

In Figure 3 each panel presents data for one of the six test papers. The disks indicate the average measured CIE _xy coordinates of the light reflected by the test papers at the different locations. They illustrate the color shifts that the subjects had to deal with. For comparison, the crosses indicate the CIE _xy coordinates of the light reflected from the test papers during the baseline, when all papers were viewed under the same fluorescent illumination (coordinates given in the Methods section). This illustrates how small the impact of the differences in reflectance is in comparison with the impact of the illumination. These coordinates are shown for illustration purposes only; they do not coincide with the values shown during the initial part of the main experiment, which differed for the three groups of subjects because the initial part was conducted in a room with windows (daylight). The measured luminance of the light reflected by the papers varied between 5 and 17100 cd/m ². 

Figure 3

 

Overview of the results. Each graph represents the results for one of the six test papers. The six crosses show the CIE _xy coordinates of the six test papers in the baseline condition (same in all panels). The disks show the average coordinates of the test papers when measured at the 12 different locations (4 locations for each of the 3 groups of subjects per test paper). Pie charts show the distribution of subjects' responses. The location corresponding with the pie chart for one green test paper (middle right graph) is absent because of technical problems when measuring the light during the presentation.

Each pie chart in Figure 3 shows the proportions of responses for one test paper at one of the twelve locations at which that paper was presented (to 7 subjects). The colors in the pie charts correspond with the names that the subjects wrote down. Figure 4 shows how the percentage of correct responses depends on how different the color of the illumination is from the value during the instruction phase. Figures 3 and 4 highlight three aspects of the task. 

Figure 4

 

Performance as a function of the shift in chromaticity with respect to that in the instruction phase. Points are averages for 6 presented papers, with standard errors across the percentages correct for the 6 presented papers and across distances.

First, the influence of the differences between the test papers' reflectance properties on the light that reached our subjects' eyes (distances between crosses in Figure 3) is much smaller than the influence of seeing the test papers at different locations (distances between disks in Figure 3). This illustrates the problem that the visual system is confronted with when having to recognize objects by their color in different settings at different moments (e.g. the papers in the current study). 

Secondly, despite the large differences between the light reflected from the same test paper at the different locations, subjects were often able to recognize the test papers' colors (pie charts in Figure 3). In fact, a large difference between the light reflected during the instruction phase and when tested in the main experiment hardly reduces subjects' performance ( Figure 4). 

Finally, looking at the pie charts of Figure 3, we see that subjects are only slightly biased, if at all, by the color of the light that reaches their eyes from the surface of the paper. For example, when the dominant light that reaches the subjects' eyes is yellowish (top right of panels), we may have expected subjects to often erroneously identify the test paper as being the yellow paper; incomplete color constancy (Brainard, 1998; Hurlbert, 1999) implies that part of the change in the color of the light reaching the eyes is attributed to the surface). 

We also see that quite similar coordinates for the reflected light of a given paper sometimes resulted in quite different responses. For instance, consider the three presentations of the yellow paper in the top right of the figure (CIE _xy coordinates of about 0.45, 0.45). In these three cases the same paper is shown at different places. The illumination at the three locations must have a similar spectral distribution, because the x and y coordinates of the reflected light (as shown in the figure) are almost identical. Nevertheless in one case most subjects judge the paper to be yellow whereas in one of the others most judge it to be green. There are several such examples in Figure 3. In these cases the difference between the judgments presumably arises because of biases introduced by the surroundings (and possibly the overall luminance). 

Table 1 shows the frequency of responses for each of the six test papers. Diagonal cells represent correctly identified test papers. Certain colors were chosen more often than others, and certain pairs of test papers were more frequently confused with each other than others. For example, the green test paper was often identified as being the blue and the yellow and red test papers were often identified as being white. 

Table 1

 

Summary of responses for each paper.

Response	Paper
	White	Yellow	Red	Blue	Green	Gray	Total
White	64	28	32	3	7	5	132
Yellow	3	28	8	0	2	0	41
Red	3	6	29	1	1	7	53
Blue	5	2	4	67	32	8	118
Green	4	19	2	2	31	2	60
Gray	5	1	9	11	11	62	100
Total	84	84	84	84	84	84	504

On average, subjects correctly identified 76% of the test papers when they were shown again at the location at which they had originally been presented. This was significantly better than average (chi-square = 3.85, p < .05). It was slightly (though not significantly; chi-square = 1.88) less accurate than in the baseline, perhaps because the illumination was no longer exactly the same and subjects had been exposed to very different illuminations between the instruction phase and this test (exposure to a wider range of apparent paper colors may influence subjects' memory of the paper colors that were originally seen). 

Discussion

Although the papers were very similar to each other in reflectance, and both the color and luminance of the illumination and the color of the background varied considerably between the locations, subjects were able to identify the colored test papers on more than half of the trials. Their performance did not appear to depend on how different the illumination was from the one under which they were instructed about how to name the papers ( Figure 4). There was a weak tendency at most to be biased by the color of the light that reached the eyes from the surface of the paper. The improved performance when returning to the initial position is probably due to the background being the same as during training. Thus, color constancy is extremely good for real objects presented under natural conditions when the task is to recognize surfaces by their color. 

There are two main reasons why color constancy in general cannot be perfect. First, there are the theoretical limitations of trichromatic color vision, as described in the Introduction section, that constrain the identification of colored objects under different illuminations. Secondly, since any pattern of light reaching the eye could arise from an infinite number of combinations of reflectance and illumination, the visual system has to make assumptions for separating the contributions of illumination from those of reflectance (e.g., Granzier et al.,2005; Granzier, Smeets, & Brenner, 2006). Any such assumption (the gray world hypothesis; the bright is white hypothesis; relying on color contrast; estimating the illuminant from highlights; etc) can be violated. Relying on multiple sources of information (as described in the Introduction section) can make color constancy very robust. However considering the possibility that assumptions are violated constrains the amount of color constancy that can be achieved (even when the assumptions are not violated). Moreover small errors will arise when any of the assumptions are violated. Indeed, we need not visit a laboratory to observe large failures of color constancy: when you attend a movie you view a flat white surface on to which is projected a complicated dynamic pattern of light. You see people, cars, explosions and so on, just as the script of the film predicted. None of these objects or their surfaces are present and yet you ‘see’ them, most of the time forgetting about the only surface truly present, the uniform white screen. 

Acknowledgments

This work was supported by grant 051.02.080 of the Cognition Program of the Netherlands Organization for Scientific Research (NWO). 

Commercial relationships: none. 

Corresponding author: Eli Brenner. 

Email: [e-mail]. 

Address: Van der Boechorststraat 9, 1081 BT Amsterdam, The Netherlands.