Number of perceptually distinct surface colors in natural scenes

Abstract

The ability to perceptually identify distinct surfaces in natural scenes by virtue of their color depends not only on the relative frequency of surface colors but also on the probabilistic nature of observer judgments. Previous methods of estimating the number of discriminable surface colors, whether based on theoretical color gamuts or recorded from real scenes, have taken a deterministic approach. Thus, a three-dimensional representation of the gamut of colors is divided into elementary cells or points which are spaced at one discrimination-threshold unit intervals and which are then counted. In this study, information-theoretic methods were used to take into account both differing surface-color frequencies and observer response uncertainty. Spectral radiances were calculated from 50 hyperspectral images of natural scenes and were represented in a perceptually almost uniform color space. The average number of perceptually distinct surface colors was estimated as 7.3 × 10³, much smaller than that based on counting methods. This number is also much smaller than the number of distinct points in a scene that are, in principle, available for reliable identification under illuminant changes, suggesting that color constancy, or the lack of it, does not generally determine the limit on the use of color for surface identification.

Introduction

Color can be remarkably effective in allowing surfaces in natural scenes—and by extension objects and regions—to be distinguished from each other. Yet, in the limit, how many surfaces can an observer usefully discriminate by virtue of their color in such scenes? 

This kind of question has previously been addressed in an idealized context by calculating, within a suitable three-dimensional color space, the volume of the solid arising from the set of all possible surface spectral reflectances. This solid, known as the object-color solid (Wyszecki & Stiles, 1967), was suggested by Judd and Wyszecki (1975) to contain about 10 million discriminable surface colors, although the provenance of this estimate is a little uncertain (McCamy, 1998; Pointer, 1998). A more grounded analysis by Pointer and Attridge (1998) counted the number of cubic cells of unit side in the object-color solid in CIELAB color space (CIE, 2004a), which produced an estimate of about 2.3 million discriminable colors, later revised slightly downward by Martínez-Verdú et al. (2007) to about 2.0 million. Even so, the object-color solid is a hypothetical construct, containing all spectral distributions, including those having just two values, 0 and 1 (MacAdam, 1935; Martínez-Verdú et al., 2007), and it does not represent the gamut of surface colors to be found in natural scenes. A similar problem of representation occurs with artificial collections of colored materials, including color atlases (Nickerson & Newhall, 1943; Pointer, 1980). 

An important feature of natural scenes is that their spectra are not distributed uniformly. Most natural spectra come from surfaces whose colors under daylight illumination are dominated by browns, greens, and blues, mainly from earth, vegetation, and sky (Burton & Moorhead, 1987; Hendley & Hecht, 1949; Osorio & Bossomaier, 1992; Webster & Mollon, 1997). In addition, not all possible natural spectra are present in any particular scene. The number of discriminable colors averaged over natural scenes might therefore be expected to be much less than 2.0–2.3 million, even though these are the scenes to which observers should be accustomed and for which performance might be expected to be optimal (MacLeod & von der Twer, 2003). 

An estimate of the number of discriminable colors in natural scenes was provided by Linhares, Pinto, and Nascimento (2008), who used a set of 50 hyperspectral images of rural and urban scenes to calculate reflected spectra. Because the coordinates (L*, a*, b*) of CIELAB color space do not define a strictly uniform color space, that is, the Euclidean distance ΔE* = [(ΔL*)² + (Δa*)² + (Δb*)²]^1/2 between points does not correspond to a constant perceptual color difference, Linhares et al. calculated the separations between points using the CIEDE2000 color-difference formula (CIE, 2004a), which largely compensates for the non-uniformity of CIELAB. The number of discriminable colors was estimated by iteratively discarding points that differed from each other by less than one half the nominal CIEDE2000 threshold color difference ΔE ₀₀ = 0.6. The procedure was repeated until there were no points left. Although based on a smaller population than the theoretical object-color solid used by Martínez-Verdú et al. (2007) and Pointer and Attridge (1998), the resulting estimate was almost the same, a total of about 2.3 million colors (Linhares et al., 2008, Table 1), accumulated over the 50 scenes. The estimated number of discriminable colors averaged over individual scenes was inevitably smaller, about 2.7 × 10⁵ (Linhares et al., 2008, Table 1). 

There are, however, two fundamental problems with trying to use counting methods to estimate the number of surface colors an observer can distinguish in natural scenes. First, these methods do not allow for the different relative frequencies of different surface colors within a scene. Certainly, where colors are sparse, there is little risk of confusion, and perceived color is a reliable guide to surface color, but where colors are dense, the risk of confusion is high, and perceived color is a less reliable guide. Second, the assumption in counting methods of a hard threshold boundary separating perceptually discriminable colors represents an all-or-nothing deterministic process. In practice, observer responses are probabilistic, governed by a psychometric function relating the measured difference in colors to the probability of reporting a difference between them. Thus, with the same stimuli, perceptual responses necessarily vary from trial to trial. It is not easy to implement this probabilistic property in a procedure based on counting discriminable colors. 

Fortunately, there is an alternative approach based on information theory (Cover & Thomas, 1991; Shannon, 1948a, 1948b) which incorporates the differing relative frequencies of surface colors in a natural way and which also accommodates observer response uncertainty. A particular information-theoretic quantity, the mutual information, provides an estimate of the number of perceived surface colors that can each be identified with a distinct surface color in the scene. This number is referred to here as the number of perceptually distinct surface colors to avoid confusion with the number of discriminable or discernible colors estimated by Linhares et al. (2008) and Pointer and Attridge (1998). 

In this analysis, colors were represented in a nearly uniform color space and were derived from the reflected spectral radiances from 50 hyperspectral images of natural scenes under a simulated standard daylight with a correlated color temperature of 6500 K, representing approximate average daylight through the visible spectrum. The estimated mutual information yielded an average number of perceptually distinct surface colors of 7.3 × 10³, markedly smaller than that based on counting methods. An implication of this result for estimates of color constancy in natural scenes is briefly considered in the Discussion section. 

Theory

Consider a scene illuminated by the chosen daylight. As outlined earlier, the color of the reflected light from each surface element in the scene can be expressed in CIELAB (L*, a*, b*) coordinates, and a color-difference formula such as CIEDE2000 (Luo, Cui, & Rigg, 2001) or DIN99 (Cui, Luo, Rigg, Roesler, & Witt, 2002) can then be used to correct for non-uniformities (Linhares et al., 2008). Instead, to simplify the analysis, the color of the reflected light was expressed in CIECAM02 space (CIE, 2004b), which has the advantage that perceived color differences represented as Euclidean differences in CIECAM02 coordinates (J, a _C, b _C) correspond to almost constant perceptual color differences (Luo, Cui, & Li, 2006; Melgosa, Huertas, & Berns, 2008). Since CIECAM02 is determined empirically and has a built-in chromatic-adaptation transform (CIE, 2004b), it automatically incorporates any improvements in discrimination performance found in the region of the reference white. The variable J represents lightness and a _C and b _C represent the projections of chroma onto the red–green and blue–yellow hue axes, giving a hue angle h = tan⁻¹(b _C/a _C). Along with several other color spaces, CIECAM02 may depart from uniformity with very small color differences, where CIELAB ΔE* ≤ 1 (Melgosa, Huertas, & García, 2008). 

The fact that CIECAM02 is a color-appearance space is immaterial to this analysis: any other color space would suffice, providing that the same space was also used to represent the psychometric function describing observer response uncertainty. There is a general constraint that the color space should not incorporate any dependence on the spatial structure of the scene—as in, e.g., S-CIELAB (Zhang & Wandell, 1997) or iCAM (Fairchild & Johnson, 2004)—since this analysis is concerned with the distinguishability of surface colors taken individually, not with their distinguishability as modified by accidents of context. 

The triplet (J, a _C, b _C) may be treated as an instance u of a trivariate continuous random variable U with probability density function (pdf) f _U , say. Likewise, the observer's perceptual response may be treated as an instance v of a second trivariate continuous random variable V with pdf f _V , say. Given two particular triplets u and u′, the probability that an observer reports them as being different depends on the corresponding difference v − v′. This dependence is described by the psychometric function Ψ, say. The differences w = v − u and w′ = v′ − u′ are, in turn, instances of a trivariate continuous random variable W with pdf f _W , where V = U + W. The pdf f _W is essentially the derivative of the psychometric function Ψ (DeCarlo, 1998). 

The degree of unpredictability of each of the random variables U, V, and W may be quantified by the differential entropy (Cover & Thomas, 1991). For example, the differential entropy h(U) of U is given by 

$h\left(U\right)=-\int f_U\left(u\right)\log f_U\left(u\right)du,$

(1)

where the symbol h should not be confused with that for hue in CIECAM02. When the logarithm is taken to the base 2, the differential entropy is given in bits. Shannon's mutual information I(U; V) between U and V is given (Cover & Thomas, 1991) by 

$I(U;V)=h\left(V\right)-h(V|U).$

(2)

 

Given the near uniformity of CIECAM02, it may be assumed that Ψ is constant, and therefore W is independent of U, although see MacLeod and von der Twer (2003). Accordingly (Cover & Thomas, 1991), h(V∣U) = h(W), and therefore 

$I(U;V)=h\left(V\right)-h\left(W\right).$

(3)

 

This quantity I(U; V) measures the amount of information that the perceived color provides about the sampled surface colors. It has the useful interpretation as the mean number N of perceived surface colors that can each be identified with a distinct surface color in the scene by virtue of the following relationship (Cover & Thomas, 1991; Shannon, 1948a, 1948b): 

$N=2^{I(U;V)}.$

(4)

 

Notice that the differential entropies h(V) and h(W) may be positive, negative, or zero, depending, not least, on the scale of measurement defined by CIECAM02, but the mutual information I(U; V) is always positive and independent of the scale. 

Methods

Estimates of differential entropies were based on data generated from a set of 50 hyperspectral images of natural rural and urban scenes (Foster, Amano, Nascimento, & Foster, 2006; Nascimento, Ferreira, & Foster, 2002). See Linhares et al. (2008) for thumbnail images. Each hyperspectral image had dimensions ≤1344 × 1024 pixels and spectral range 400–720 nm sampled at 10-nm intervals, providing an effective spectral reflectance r(λ; x, y) at each wavelength λ and position (x, y). The reflectances r(λ; x, y) were obtained by dividing the spectral radiance of the image by the spectral radiance of a small neutral (Munsell N5 or N7) reference surface embedded in the scene and then multiplying by the known spectral reflectance of the neutral surface. The effect of a particular daylight was simulated by multiplying r(λ; x, y) at each point (x, y) by a standard illuminant spectrum, here a daylight with correlated color temperature of 6500 K, fixed over all scenes so that the color gamut of each scene was not confounded by differences in illuminant. The raw spectral radiance images were actually acquired under daylights with correlated color temperatures ranging from about 4400 K to 8200 K. 

The accuracy of the hyperspectral imaging system was described in Foster et al. (2006). The root-mean-square error between estimates of reflectance from the hyperspectral images and telespectroradiometer measurements of test pigments was 1.1%, falling to 0.8–0.9% when allowance was made for a 1-nm difference in wavelength calibration, smaller than the nominal spectral accuracy of both devices. With independent sampling at each wavelength, the imaging system was capable of following the rapid variations in spectral reflectance sometimes found with natural pigments (Jaaskelainen, Silvennoinen, Hiltunen, & Parkkinen, 1994; Vrhel, Gershon, & Iwan, 1994). With an acceptance angle of the camera of about 6 degrees of visual angle, the spatial resolution of the system was at least as good as that of the eye at the same viewing distance. Each pixel was therefore assumed to correspond to a single surface in the scene. Even if at a finer scale its spectral reflectance was a mixture of several spectral reflectances, the mixture would have been undetectable by the eye. See Foster et al. (2006), Sections 2.A and 3.E, for further discussion. 

The triplets (J, a _C, b _C) were calculated at each pixel in each image of each scene according to the CIECAM02 specification with default values, including those for chromatic adaptation (CIE, 2004b). Integrations were performed numerically over 400–720 nm with the given 10-nm sampling interval. 

Estimates of h(V) were not feasible as the values of V are not directly available, but from numerical simulations, it was found that h(U) and h(V) were almost equal, with h(V) on average about 0.1 bits larger than h(U). Hence, in Equation 3, if h(V) is replaced by h(U), and if

$\widehat{h}$

(U) and

$\widehat{h}$

(W) are the corresponding estimates of h(U) and h(W), then I(U; V) may be estimated by 

$\widehat{I}(U;V)=\widehat{h}\left(U\right)-\widehat{h}\left(W\right).$

(5)

 

One direct way to obtain

$\widehat{h}$

(U) is to find an estimate of the pdf f _U and plug the result into Equation 1. Naïve estimates of f _U may be obtained by binning, i.e., partitioning the space of triplets (J, a _C, b _C) from each scene into a finite number of cells (different from the cells of the counting methods described earlier) and counting the frequency of occurrences in each cell (Silverman, 1986). In practice, the number of cells needed for an accurate estimate of the pdf is generally large and the estimate of the frequency of responses will be biased unless the sample size is much larger still than the number of cells. If the sample is not large enough, systematic errors in estimating the entropy can occur (Kraskov, Stögbauer, & Grassberger, 2004). These errors may be minimized by introducing bias-correction terms, but the accuracy of the estimates still depends on binning. 

A different approach that completely avoids estimation of the pdf makes use of k-nearest-neighbor statistics to obtain an estimate

$\widehat{h}$

(U) of the differential entropy h(U). The k-nearest-neighbor estimator due to Kozachenko and Leonenko (Goria, Leonenko, Mergel, & Novi Inverardi, 2005; Kozachenko & Leonenko, 1987) was chosen since it provides results that are efficient, adaptive, and have minimal bias (Kraskov et al., 2004). It was applied to the triplets (J, a _C, b _C) from each scene in an offset form that converges more rapidly and accurately than the original estimator (Foster, Marín-Franch, Amano, & Nascimento, 2009; Marín-Franch, 2009). 

An estimate

$\widehat{h}$

(W) of the differential entropy h(W) associated with observer response uncertainty was obtained from the reported perceptibility of small color differences between matt paint samples (Wang, Luo, Cui, & Xu, 2009). The samples consisted of 10 reference samples and, for each reference, 30 test samples. The recorded proportions of perceptible differences ranged from 0.125 to 1.0, and, at each level, estimates were made of the differences (ΔJ, Δa _C, Δb _C) in (J, a _C, b _C) values. The resulting sample sizes (300) were, however, too small to find a reliable estimate of the differential entropy h(W) by any of the empirical methods mentioned earlier. Fortunately, as the distribution of the differences ΔJ, Δa _C, and Δb _C were each approximately Gaussian, the differential entropy h(W) could be estimated by the differential entropy of a trivariate Gaussian variable; that is, 

$\widehat{h}\left(W\right)=\frac{1}{2}\log \left[\right(2\pi e{)}^3|K|],$

(6)

where ∣K∣ is the determinant of the covariance of W, which was approximately the product of the individual variances σ _J ² σ _a ² σ _b ², since there was little correlation between J, a _C, and b _C. An estimate

$\widehat{h}$

(W) of h(W) was also obtained from the reported acceptability of the color differences between the same samples that Wang et al. used for perceptibility judgments. Perceptibility results are presented in detail; acceptability results in summary form. 

For comparison, an estimate

$\widehat{h}$

(W) of the differential entropy h(W) was also obtained with a hard threshold based on the assumption of a uniform distribution of W over a sphere of radius 0.3, as in Linhares et al. (2008). 

Estimates were made for the average scene and for the union of those scenes, i.e., with surface colors merged over the 50 scenes. 

Results and comment

Figure 1 shows two of the 50 scenes with the lowest and highest estimated entropy

$\widehat{h}$

(U) and correspondingly lowest and highest estimated mutual information. The difference in the color gamuts of the two images is obvious. The corresponding estimates of the number N of perceptually distinct surface colors were 5.5 × 10² and 4.1 × 10⁴. 

Figure 1

 

Images with the lowest and highest mutual information.

Table 1 shows estimates averaged over all 50 scenes and for the union of those scenes, each illuminated by the 6500 K daylight. Entries are for the differential entropy h(U) for surfaces, the differential entropy h(W) for observer responses based on Gaussian perceptibility data and on a uniform distribution (equivalent to a hard threshold of 0.3 in CIECAM02), the corresponding mutual information I(U; V) from Equation 5, and the number N of perceptually distinct surface colors from Equation 4. 

Table 1

 

Estimates of the differential entropy h(U) of surface colors in natural scenes, the differential entropy h(W) of observer responses, the mutual information I(U; V), and the number N of perceptually distinct surface colors, all measured in bits. The estimated standard deviation of h(U) was 1.25 bits. The total number of scenes was 50.

Surface-color entropy h(U)		Observer entropy h(W)		Information I(U; V)	No. of colors N
Average	12.53	Gaussian	−0.29	12.82	7.3 × 103
		Uniform	−3.14	15.68	5.2 × 104
Union	15.14	Gaussian	−0.29	15.43	4.4 × 104
		Uniform	−3.14	18.28	3.2 × 105

With observer responses based on Gaussian perceptibility data, the estimated number of perceptually distinct surface colors was 7.3 × 10³ and 4.4 × 10⁴ for the average and union of the scenes, respectively. 

With observer responses based on Gaussian acceptability data, results were only a little different. The estimated differential entropy h(W) was −0.49 bits, i.e., 0.20 bits less than the −0.29 bits for the perceptibility data (Table 1). The estimated number of perceptually distinct surface colors was 8.3 × 10³ and 5.1 × 10⁴ for the average and union of the scenes, respectively, an increase of 14%. 

As a control on these calculations, the same method of analysis was applied to a flattened version

$f_{\bar{U}}$

of the pdf f _U used to estimate surface-color entropy h(U), to mirror the estimates based on uniform distributions reported by Linhares et al. (2008) and Pointer and Attridge (1998). With the same method of estimation based on the offset Kozachenko–Leonenko estimator, the differential entropy h(

$\bar{U}$

) of

$f_{\bar{U}}$

for the union of the scenes was estimated as 18.42 bits (to be compared with 15.14 bits in Table 1). With a differential entropy of −0.29 bits for observer responses based on the Gaussian perceptibility data (Table 1), the mutual information was estimated as 18.72 bits (Equation 5). With a differential entropy of −3.14 bits for observer responses based on a uniform distribution (Table 1), the mutual information was estimated as 21.57 bits (Equation 5). These estimates of mutual information correspond (Equation 4) to 4.3 × 10⁵ and 3.1 × 10⁶ discriminable colors, bracketing the value of 2.3 × 10⁶ obtained by Linhares et al. (2008) and Pointer and Attridge (1998) for the union over scenes. 

For the mean over scenes rather than the union, the differential entropy of

$f_{\bar{U}}$

was estimated as 14.95 bits (to be compared with 12.53 bits in Table 1), and the mutual information as 15.24 and 18.09 bits for observer responses based on the Gaussian perceptibility data and a uniform distribution, respectively. These estimates of mutual information correspond (Equation 4) to 3.9 × 10⁴ and 2.8 × 10⁵ discriminable colors, also bracketing the value of 2.7 × 10⁵ obtained by Linhares et al. (2008) for the average over scenes. 

Discussion

Traditional methods of estimating the number of discriminable surface colors do not take into account their frequencies of occurrence. Yet, as argued earlier, the number of surface colors an observer can usefully distinguish depends strongly on their relative frequencies in a given scene. It was found here that the number of perceptually distinct surface colors, averaged over 50 natural scenes, was about 7.3 × 10³, more than an order of magnitude lower than the 2.7 × 10⁵ discriminable colors reported by Linhares et al. (2008), with almost the same set of scenes. When these numbers were obtained not as averages over individual scenes but accumulated over the union of all 50 scenes, a similar disparity occurred. The number of perceptually distinct surface colors was 4.4 × 10⁴ and the number of discriminable colors was 2.3 × 10⁶ (Linhares et al., 2008). 

These large differences between the number of perceptually distinct surface colors and the number of discriminable colors are due not only to the non-uniform relative frequencies of surface colors in natural scenes but also to observer response uncertainty. For the union of scenes, the differential entropies for observed and flattened surface-color relative frequencies were 15.1 and 18.4 bits, respectively, a difference of 3.3 bits (a smaller difference of 2.4 bits was obtained with the average over scenes). The differential entropies for observer responses based on the Gaussian perceptibility data and on a uniform distribution, i.e., a hard threshold, were −0.29 and −3.14 bits, respectively, a difference of 2.85 bits. These two factors, the relative frequencies of surface colors in natural scenes and observer response uncertainty, appear to contribute almost equally to the low estimates of the number of perceptually distinct surface colors. 

Nevertheless, as with any estimate of this kind, there are several sources of potential error. First, the empirical psychometric function describing observer response uncertainty was based on relatively small numbers of test and reference samples (Wang et al., 2009), and no identical samples were included that would have allowed the estimation of a true false-alarm rate, and thereby a bias-free measure of discrimination such as d′ from signal-detection theory (Macmillan & Creelman, 2005). 

Second, as noted earlier, the perceptual uniformity afforded by CIECAM02 is not perfect (Melgosa, Huertas, & García, 2008), and color differences do depend on where in color spaces the colors are located (Luo et al., 2006). The assumption of independence between perceptual color difference and surface color leads to a small underestimation of the expected number of perceptually distinct surface colors because h(W) in Equation 3 is larger (Cover & Thomas, 1991) than h(V∣U) in Equation 2. 

Third, the use of a Gaussian model for discriminating surface colors based on perceptible color differences may not be sufficiently accurate. With a better model of discrimination, not necessarily defined with respect to perceptibility, the resulting differential-entropy estimates may be smaller, but without more data it is not possible to estimate precisely by how much. How the psychometric function varies with location in color space may also be important (MacLeod & von der Twer, 2003). This dependence of estimates on the model of discrimination is not unique, of course, to the approach adopted in this analysis: counting methods depend explicitly on the assumed hard threshold for the color difference ΔE* or ΔE ₀₀. 

Finally, these calculations took no explicit account of metamerism, that is, of the fact that surface color does not define a unique surface spectral reflectance, although it does appear implicitly in determining the pdf f _U of surface colors with a particular illuminant. 

A question related to the one considered here is how the number of perceptually distinct surface colors varies with a change in illuminant spectrum. Estimates made elsewhere (Foster et al., 2009) suggest that, for an ideal observer unaffected by any uncertainty in matching, the number of distinct points in a scene that are, in principle, available for reliable identification under an illuminant change from a 25,000 K daylight to a 4000 K daylight is about 1.37 × 10⁵. It is interesting that even with this extreme illuminant change, this number is much larger than the number 7.3 × 10³ of perceptually distinct surface colors, suggesting that color constancy, or the lack of it, does not generally determine the extent to which surfaces may be identified by their color in natural scenes under different illuminants. 

Acknowledgments

This work was supported by the EPSRC (Grant No. EP/E056512/1 and EP/F023669/1). We are grateful to M. R. Luo, H. Wang, and S. Westland for making available the source data on perceptibility and acceptability ratings (Wang et al., 2009); to K. Amano and S. M. C. Nascimento for the use of the set of hyperspectral images; to M. S. Mould and S. M. C. Nascimento for helpful comments and discussion; and to M. Maule, C. Novara, and K. ¯ychaluk for critically reading the manuscript. 

Commercial relationships: none. 

Corresponding author: Iván Marín-Franch. 

Email: [e-mail]. 

Address: Department of Optometry and Visual Science, City University London, Northampton Square, London, EC1V 0HB, UK.