Spatial filtering, color constancy, and the color-changing dress

Abstract

The color-changing dress is a 2015 Internet phenomenon in which the colors in a picture of a dress are reported as blue-black by some observers and white-gold by others. The standard explanation is that observers make different inferences about the lighting (is the dress in shadow or bright yellow light?); based on these inferences, observers make a best guess about the reflectance of the dress. The assumption underlying this explanation is that reflectance is the key to color constancy because reflectance alone remains invariant under changes in lighting conditions. Here, we demonstrate an alternative type of invariance across illumination conditions: An object that appears to vary in color under blue, white, or yellow illumination does not change color in the high spatial frequency region. A first approximation to color constancy can therefore be accomplished by a high-pass filter that retains enough low spatial frequency content so as to not to completely desaturate the object. We demonstrate the implications of this idea on the Rubik's cube illusion; on a shirt placed under white, yellow, and blue illuminants; and on spatially filtered images of the dress. We hypothesize that observer perceptions of the dress's color vary because of individual differences in how the visual system extracts high and low spatial frequency color content from the environment, and we demonstrate cross-group differences in average sensitivity to low spatial frequency patterns.

Introduction

On February 25, 2015, a phenomenon referred to as the color-changing dress went viral on the Internet. A blogger on the website Tumblr posted a photo of a striped dress, asking her relatively small community of readers to vote on the perceived colors of the fabric; some voters saw the dress as blue with black lace stripes, whereas others perceived it as white with gold lace stripes (“Guys-please-help-me,” 2015). The sharp divide among responders (as well as disagreement from widely followed celebrity commentators) fueled a rapid spread of the photo across many online news outlets, and the topic trended worldwide on Twitter under the hashtag #theDress. The huge debate on the Internet also sparked debate in the vision science community about the implications of the stimulus with regard to individual differences in color perception, which in turn led to a special issue of the Journal of Vision, for which this article is written. 

The predominant explanations in both scientific journals (Gegenfurtner, Bloj, & Toscani, 2015; Lafer-Sousa, Hermann, & Conway, 2015; Winkler, Spillmann, Werner, & Webster, 2015) and in popular scientific press articles (Lafer-Sousa, 2015; Macknik, 2015) featured variations of what we term the color constancy hypothesis. The term color constancy refers to the fact that objects maintain a relatively stable color appearance even when viewed under markedly different illumination (Judd, 1940). For instance, a yellow ball viewed in sunlight and then fluorescent light is perceived as “yellow” in both conditions despite illuminants with wildly different spectral compositions. Color constancy would not pose a theoretical problem if the visual system were truly able to sense the spectral reflectance of objects (i.e., the proportion of light an object reflects at each wavelength); however, the visual system has access only to the light reaching the eye. The ability to maintain color constancy despite the limited information in the environment has therefore led to the suggestion that observers infer reflectance by using their prior experience to make a best guess about scene illumination, the nature of the material of the object, and the physical configurations of the world (Brainard et al., 2006; Lotto & Purves, 2000). 

The color constancy hypothesis leads to a seemingly straightforward explanation for the color-changing dress: Observers report color based on their estimation of material reflectance, and individual differences in the appearance of the dress arise because observers differ in how they interpret the illumination and shadows in the picture. According to this explanation, the material of the dress typically appears blue and black even though measured pixel values of the dress image show a range of colors most easily characterized as blue and brown (Lafer-Sousa et al., 2015). Observers who have good color constancy are able to discount the yellowish illuminant and therefore perceive the dress as blue-black (or perhaps blue and brown), whereas those who see the dress as white and gold either have poor color constancy and are unable to discount the yellowish illuminant or perceive the dress as if it is in shadow and are able to discount the shadow. Still other observers who see the dress sometimes as blue and black and sometimes as white and gold have imperfect constancy and can switch their interpretation of the illuminant. Another suggested explanation is that observers are discounting shorter wavelengths based on prior experience and their particular daytime chronotype (Lafer-Sousa et al., 2015) and that the variability in the perception of the dress may be accentuated because observers are more likely to interpret blue as a property of illumination rather than reflectance (Winkler et al., 2015). 

A spatial filtering approach

Although a framework based on illumination and reflectance has some intuitive appeal (particularly for discussing phenomena with the media and public), there are other ways of thinking about spatial aspects of color vision that may be of relevance for understanding the dress. For instance, many researchers have suggested that illumination can be estimated by averaging across portions of the scene (Buchsbaum, 1980; Judd, 1940; Land, 1986); indeed, investigations with “a gray world hypothesis” show that the average of the scene can give reliable estimates of illuminant chromaticity under many conditions (Barnard, Cardei, & Funt, 2002; Khang & Zaidi, 2004). Others have suggested that many aspects of color contrast can be accounted for by lateral inhibition-type processes (D'zmura & Singer, 2001; De Valois, Webster, De Valois, & Lingelbach, 1986), spatial weighting functions (Shapley & Reid, 1985; Zaidi, Yoshimi, Flanigan, & Canova, 1992), or other types of gain control that operate within color channels. Indeed, many brightness/lightness phenomena previously attributed to unconscious inference can be accounted for by spatial filtering approaches that have similarities to older lateral inhibition models (Blakeslee, Pasieka, & McCourt, 2005; Perna & Morrone, 2007; Robinson, Hammon, & de Sa, 2007). 

Following this approach, Shapiro and Lu (2011) and Dixon, Shapiro, and Lu (2014) presented a simple model for brightness/lightness phenomena, shown in Supplementary Movie 1. The model consists of a high-pass filter with a single parameter; the parameter adjusts the cutoff frequency so that when the parameter is small, only a narrow range of high-pass information remains, and the filter acts like a high-pass edge detector. When the parameter is large, only a narrow range of low spatial frequency content is removed, and the filter acts like a process that removes blur from the image. Remarkably, this simple filter can account for the relative brightness of test patches in a natural image and for phenomena often attributed to unconscious inference or to anchoring. In addition, Dixon et al. (2014) showed that the size of the parameter adapts to the size of the objects in the scene. So, objects far from an observer subtend a smaller visual angle than objects close to an observer; hence, a distant object would lead to less low spatial frequency being removed. 

To our knowledge, no one has directly attempted to account for color vision phenomena with the spatial filtering models frequently applied to lightness and brightness (Blakeslee & McCourt, 2012). This is somewhat surprising because there are many reasons to speculate that such models would be useful for color perception. For instance, von Kries transformations and the gray world hypothesis suggest that much of the effect of illumination is contained in the low spatial portion of the image spectra (see Smithson, 2005). The response of double-opponent cells is spatially band pass and that of single-opponent cells is spatially low pass (Conway, 2002; Shapley & Hawken, 2011). Color vision based on these channels would select from different portions of the images, and these portions likely contain different types of chromatic information. Indeed, models that propose that color “fills in” between luminance edges in a scene (see Feitosa-Santana, D'Antona, & Shevell, 2011) in essence divide the image into high spatial frequency “edges” and low spatial frequency “fill-in.” In addition, Werner (2014) recently suggested two types of color constancy processes: a slow type, operating at a global scale for the compensation of the ambient illumination, and a fast type that is locally restricted and well suited to compensation for region-specific variations in the light field. 

Here, we explore how effective the Shapiro and Lu (2011) and Dixon et al. (2014) spatial filtering model is at explaining several aspects of color vision and how this can lead to an explanation about the dress. We show that adaptive high-pass filters can counteract the effects of illumination within many of the images typically used to illustrate the effects of “discounting the illuminant”: (a) Lotto and Purves's (2002) Rubik's cube illusion, (b) the image of the dress under three different simulated illuminants, and (c) an image of a shirt under three different natural illuminants. The demonstrations suggest that we do not need to posit that observers make assumptions about the illuminant—a stage in models of color vision that often lacks quantitative specificity; instead, individual variations in perception of the dress can arise from differences in how observers' visual systems filter the image. 

We use the dress to investigate two hypotheses about the two potential roles for the high-pass channel in color vision. The main hypothesis assumes that the high-pass filter simply acts as a way of discounting the illuminant, as described by Dixon et al. (2014). In this version of events, the filter can be considered a segment of the standard color constancy hypothesis: The filter serves as part of the discounting method (perhaps in conjunction with other information about specularity) so that the observer can make inferences about the reflectance of the material. The alternative hypothesis is that the high spatial frequency content is invariant with illumination, and a low spatial frequency response encodes information that correlates with changes in global illumination. To achieve color constancy, therefore, the visual system does not need to infer spectral reflectance of the material; rather, the visual system primarily needs to give greater weighting to the higher bands of the spatial frequency spectrum. We emphasize that we are not suggesting that the low spatial frequency content is completely discarded from all future processing. Rather, observers seem to balance responses to the low and high spatial frequency color information depending on the situation and, perhaps, the individual. In this way, we attempt to recast the discussion of the dress from perceptual interpretations of reflectance and illumination to the visual system's weighting of color information that arises in the environment at multiple spatial (and temporal) scales. 

Demonstration 1: A high-pass filter can account for the Rubik's cube illusion

The Rubik's cube illusion (Lotto & Purves, 2002), an iconic image for illustrating the effect of illumination on color appearance, has often been recruited to explain the color constancy hypothesis with regard to the dress (Lafer-Sousa, 2015; Macknik, 2015). The illusion (shown in Figure 1a) depicts a cube with multiple colored squares on its surface; the illusion consists of two squares that appear radically different from each other (a brown square on the top of the cube and an orangeish square on the side), even though they have the same pixel value. In an achromatic version of the image (Figure 1c), the two squares appear to have different brightness levels. The variation of the Rubik's cube demonstration in Figure 1e appeared in a Scientific American Mind blog post, which explained the dress phenomenon with reference to a color constancy hypothesis (Macknik, 2015). In this version of the Rubik's cube illusion, the test squares are physically achromatic, and a yellow or blue overlay is added to the cube. 

Figure 1

 

(a) The original cube image (copyright Beau Lotto). Two identical brown squares are placed in direct illumination (black bounding box) or shadow (white bounding box); bounding boxes are present only to identify the measured squares. The square in shadow appears lighter than the square in illumination, even though the two squares have identical pixel values; actual colors shown in two squares to the right of the image. (b) The cube image after filtering at size 60 pixels (the measured diagonal of the individual squares on the surface of the cube). The square in shadow is now physically different from the square in illumination; actual colors shown in two squares to the right of the image. (c) Gray-scale version of the cube. Both squares are identical, mid-luminance gray, and again, the square in shadow appears brighter. (d) The gray-scale cube image after filtering at size 60 pixels. The square in shadow is now physically brighter; actual colors to the right of the image. (e) The cube in the white bounding box and the cube in the black bounding box are both mid-luminance gray. The one under the blue overlay appears yellow, and the one under the yellow overlay appears blue; actual colors below the image. (f) Blue and yellow overlay images after filtering at size 60 pixels. The square in the white bounding box appears yellow postfiltering, and the square in the black bounding box appears blue postfiltering; actual colors below the image.

The color constancy explanation of the Rubik's cube illusion, like the color constancy explanation for the dress, is that the perception of the color of the square corresponds to the visual system's estimate of the reflectance of the cube's material. So, in Figure 1a and c, the top test squares appear darker because observers infer them to represent a less reflective surface in bright light, whereas the bottom squares appear brighter because observers infer a highly reflective surface in shadow (Lotto, 2010). Figure 1e can be interpreted as a particularly powerful extension of this approach because it seems to show that the test squares, like the dress, change appearance depending on the presumed illumination. That is, although the test squares are physically achromatic, the visual system treats the square under blue illumination as having yellow reflectance and the square under yellow illumination as having blue reflectance. 

Here we explore an alternative explanation: All three variations of the Rubik's cube illusion (Figure 1a, c, and e) can be accounted for by removing low spatial frequency content from the image. The particular technique for removing low spatial frequency does not seem to matter (Supplementary Movie 1 demonstrates three different techniques); we use Adobe Photoshop's high-pass filter because it is widely accessible and simple for others to use to replicate our findings. The filter has a single parameter that has the effect of changing the amount of low spatial frequency content removed from the image. For example, a filter radius of 1 removes most spatial frequency information so that all that remains in the image are the thinnest edges; edge extraction of this form is what is usually thought of when high-pass filters are mentioned. Larger filter values shift the filter cutoff so that more low spatial frequency information remains in the image; in effect, the filter removes the blur but leaves the pictorial sense of the image roughly unchanged. 

The results of the application of the high-pass filter to Figures 1a, c, and e are shown in Figures 1b, d, and f, in which the filter size is set to 60 pixels (1.5875 cm), the width of each individual square. Although not standard, we use pixels as opposed to visual angle because we have previously demonstrated that the optimal value of the high-pass filter is related to the relative size of the test object, not to the test object's visual angle on the retina (Dixon, Shapiro, & Lu, 2014). The operations we describe should therefore work on any device or any rescaling of the image, as long as the filter is adjusted to the size of the objects in the scene. In the filtered images, the pixel values of the test squares are no longer equal to each other but instead more closely mimic the perceived values in the original image. What is particularly striking, however, is that when the high-pass filter is applied to Figure 1e, the simulated illumination completely disappears: The high-pass filter removes the illuminant, and the gray squares are now physically yellow and blue. 

One way to understand why the filter successfully accounts for the Rubik's cube phenomenon is to consider the high-pass filter as a process that subtracts the global information from the original image (see Supplementary Movie 2). The global information can be thought of as the result of a low-pass filter or, equivalently, a very large blur filter. So, Figure 1f can be approximated by blurring the image in Figure 1e, subtracting the blurred image from the original image in Figure 1e, and then adding a constant value so that the image values are all positive. The blur does not affect the overlay color in Figure 1e; hence, the overlay colors can be found both in the blurred image of Figure 1e and in the original image and will disappear when the images are subtracted from each other. The simple filtering operation, therefore, approximates an elimination of the illuminant and therefore allows the observer to estimate the reflectance of the material. 

Another way to understand the success of the filter is to consider that—at the appropriate spatial scale—the tests patches are actually physically different from each other. A photometer placed in front of a high-pass filtered version of the Rubik's cube illusion in Figures 1a and c would record that the top square has a physically lower value than the bottom square. Therefore, if we desire a physical value of light that corresponds to perception, our measurements should consider the energy at different spatial bandwidths as well as different spectral wavelengths. We should not assume that the test patches are physically identical to each other just because they have the same pixel values. In this interpretation, then, the patches are different from each other because the visual system simply extracts the information in the image at the appropriate spatial scale; the visual system would not need to use unconscious inference or Bayesian estimation in order to separate the image into material and illumination properties (see Brainard et al., 2006). 

Demonstration 2: Application of a high-pass filter to the dress image

In this section, we demonstrate how a spatial filter like that presented above can be extended to improve the understanding of the dress phenomenon. To do this, we will use stimuli modeled after the Wired magazine image (Rogers, 2015) that appeared in the days following the Internet phenomenon, which consists of the original image and two images using different pixels for a white-balance point in order to create both a white-gold percept and a blue-black percept. The image in Figure 2a re-creates the Wired photo by placing a semitransparent yellow (255, 255, 0 values) and a semitransparent blue (0, 0, 80) overlay on top of the original dress image. The original dress photo is in the center, the image on the left simulates a very strong yellowish illuminant, and the image on the right simulates a very strong bluish illuminant. If observers had perfect color constancy based on an ability to discount the illuminant, then the dress should appear the same in all three panels in Figure 2a. However, the simulated illuminants serve to disambiguate the color for many observers: Under a yellow illuminant, most observers see the dress as white-gold, whereas under a blue illuminant, most observers see the dress as blue-black (evidence for this is shown in the experiment below). The Wired magazine image has therefore been used to illustrate the idea that observers perceive the original dress differently depending on how they interpret the illumination in the original dress image. 

Figure 2

 

Dresses under varying simulated illumination based on Wired magazine illustration. (a) Original dress photo in center, flanked by image under a yellow illuminant (left) and a blue illuminant (right). (b–h) Image in panel (a) filtered with high-pass filter sizes ranging from 256 pixels (b) to one pixel (edge-extracted image) (h). Photograph of the dress used with permission. Copyright Cecilia Bleasdale.

The effect of filtering the original image of the dress through a high-pass filter is shown in Figure 2b through h; a dynamic demonstration can be seen in Supplementary Movie 2. The dress within the image is approximately 170 pixels wide, and we used a range of filter sizes from 256 pixels in radius down to one pixel. With large high-pass filters, as in Figure 2b and c (radius equals 256 and 128 pixels), the resulting images look fairly similar to the original image in Figure 2a. With mid-size high-pass filters, as in Figure 2d through g (radius equal to 64, 32, 16, and eight pixels, respectively), the shading from the overlay seems to disappear, resulting in images that look similar to each other. Figure 2h shows a filter with a radius of one pixel; only in this limiting condition does the high-pass filter behave as an edge detector that completely desaturates the image. 

Figure 2 therefore continues the suggestion that high-pass filters act to remove local averages from images and can be used to simulate the discounting of the illuminant in many images. To illustrate this idea more directly, Figure 3 shows color patches taken from the dresses seen in Figure 2. In each column, the left square corresponds to the color of a pixel taken from a white (or blue) stripe of the dress, and the right square corresponds to the color of a pixel taken from the gold or black stripe of the dress. As with Figure 2, the left column shows the pixel values from the dress with the blue overlay, the middle column shows the pixel values from the original dress image, and the right column shows the pixel values from the dress with the yellow overlay. As described by Lafer-Sousa, et al. (2015), the pixels from the original dress are measured as blue/brown, which can be seen in the color patches for the original image in the unfiltered condition (Figure 3b, first row). 

Figure 3

 

Colored squares from the dress image in all overlay and filter conditions: (a) yellow overlay, (b) original, and (c) blue overlay. Each row shows the results for a different size high-pass filter; the numbers indicate the radius size. The colors for the unfiltered condition are variable across overlay conditions and for filters of size 256 and 128, but relative color constancy occurs in the range of 64–8 radius filters. The squares for the filter size of 1 demonstrate that when the filter is very small, the filter works only as an edge detector, and the colors within the image are gray scale.

As can be seen, the diameter of the filter affects the size of the local average and therefore affects the amount of global information removed. So, in Figure 3, 256- and 128-pixel diameter filters remove the majority of overlay information; however, for all three overlays, the color of the patches does not match the color of the unfiltered dress image. The 64-, 32-, 16-, and eight-pixel diameter filters produce almost identical colors. This means that when viewed through filters of these sizes, the simulated illumination would not affect the color of the original patch. Hence, we can say that the patches would remain relatively color constant for these filter sizes and these illuminants (this point will be a focus of the following section). 

A common (and incorrect) assumption about high-pass filters is that they act only to desaturate the colors of images. Perhaps this assumption is so common because it holds for the extreme case (i.e., radius =1; Figure 2h), where high-pass filters act as edge detectors. For intermediate filter sizes, however, high-pass filters can have a variety of effects on color direction and sometimes even produce color saturation instead of desaturation. For example, in the Rubik's cube illusion, the two achromatic test patches in Figure 1e become more colorful after high-pass filtering (Figure 1f). The reason high-pass filters sometimes produce saturation and sometimes produce desaturation of test patches is that high-pass filters subtract local averages. If the test patch is less saturated than the surround, then the tendency is to saturate the test patch, and vice versa. 

The effect of filter size on the saturation of the test patches in Figure 3 can be seen in Figure 4, which plots the CIE chromaticity of the patches in Figure 3. Panel a corresponds to the yellow overlay, panel b to the original illuminant, and panel c to the blue overlay. The lines connect the chromaticities of each successive filter size; hence, a straight line indicates that increasing filter size leads to desaturation, and a bent line indicates that the filter had a more complex effect. Curiously, decreasing the filter diameter tends to desaturate the blue overlay condition but not the yellow overlay condition. We do not think that this change indicates anything unique about the colors of the overlays but rather is pertinent to the specifics of the dress image. That is, it should be possible to create an image in which the yellow overlay desaturates with filter size whereas the blue does not. 

Figure 4

 

CIE color space values for colored squares in Figure 3 for all overlay and filter conditions: (a) yellow overlay, (b) original image, and (c) blue overlay. The values for the stripe perceived as white or blue are marked as white diamonds with blue borders, and the values for the stripe perceived as gold or black are marked as yellow diamonds with black borders. The CIE values demonstrate that full desaturation occurs only for the filter size of 1; all other filter sizes do not overlay the achromatic (rgb = 128) value marked on the color space.

Demonstration 3: Application of a high-pass filter to fabric under varying illuminants

The analysis described in Demonstration 2 is, in many respects, trivial: The simulated layers of illumination add low spatial frequency content to the image; the high-pass filter operation removes low spatial frequency content from the image. So, even though we have added simulated blue and yellow illumination to the image, it is fairly obvious that we can remove the effects of illumination with a simple filter. Nonetheless, this relatively simple operation may be able to explain the Rubik's cube illusion (Demonstration 1) and many other similar phenomena without requiring knowledge of intrinsic object properties, as would be suggested by proposals that posit cognitive or Bayesian estimation of illumination. 

To show that the operation performed in Demonstration 2 is not just a trick performed by digital addition and subtraction, we demonstrate that the same high-pass filter operation can be used to neutralize the effects of real changes in illumination. Figure 5 shows a shirt with purple and white checks, photographed under three illuminant conditions (General Electric A-19 white, yellow, and blue 25-W light bulbs). Each row depicts a different lighting condition—the white illuminant is seen in a, the yellow illuminant in b, and the blue illuminant in c; the left column shows the unfiltered images of the shirt and background, whereas the right column shows the images after filtering with Adobe Photoshop's high-pass filter with a radius of 300 pixels, approximately the width of the shirt in the original photographs. 

Figure 5

 

Purple checked shirt under varying illumination conditions. The images shown in the left column are the original unfiltered photographs of the shirt under three illumination conditions: (a) shirt under a white illuminant, (b) shirt under a yellow illuminant, and (c) shirt under a blue illuminant. The illuminants cause the fabric to appear dissimilar across conditions. The images shown in the right column have been high-pass filtered with a filter radius of 300 pixels, matching the width of the fabric in the image; illuminant information has largely been discarded, and the fabric now appears similar across conditions.

In the original images (Figure 5, left column), the colors of the shirt and background look quite different from each other across illumination conditions; in the filtered images (Figure 5, right column), the colors appear similar. We stress that the color content in the filtered versions has not been lost, as would be expected when an extreme high-pass filter is applied (like the image in Figure 2h); instead, all three shirts have colors similar to that shown under the neutral illumination (Figure 5a, left column). The demonstration, therefore, shows that a simple high-pass filter can go a long way toward achieving color constancy, even when the low-pass information is added with natural illumination. So, although the use of a high-pass filter in Demonstrations 1 and 2 may seem trivial, because of the addition and removal of digital illumination information, it can also be used to successfully remove real scene illumination in natural images. 

The demonstration suggests two intriguing hypotheses: (a) Neural representations of objects have built into them some degree of color constancy. Any visual representation that does not include low spatial frequency content will be relatively invariant to global changes in illumination. Because objects are spatially bounded, the neural representation of an object itself does not include low spatial frequency content larger than the spatial scale of the object. Hence, any neural representation of an object will not respond much to changes in illumination. (b) The visual system represents color at two different spatial scales. The image output of a high-pass filter is relatively immune to changes in illumination; on the other hand, the converse filter operation (i.e., a low-pass filter) encodes only information about changes in illumination. By dividing the world into different spatial scales, therefore, the visual system can create one representation that is impervious to illumination and another representation that encodes only illumination. Such representations would be analogous to current descriptions of neural responses based on material reflectance and illumination. We will further discuss these hypotheses in the General discussion, but first we will examine the effects of filtering on the appearance of the dress. 

Experiment 1: The effects of filtering on the perceived color of the dress

The demonstrations show two possibilities: first, that the removal of low spatial frequency content can neutralize the effects of the illuminant, and second, that high-spatial frequency content is invariant to changes in illumination. We therefore hypothesized that individual variation in the perception of the dress is related to the observer's response to spatial properties of the image. We measured whether observers change their report of dress color when low spatial frequency content is removed from the image. Online observers viewed each dress from Figure 2a through g individually for all possible illumination and filtering conditions (e.g., the dress under the blue illuminant at filter size 8, as in Figure 2g, rightmost dress image) and selected whether each dress appeared most like white-gold or blue-black. 

Figure 6

 

Labeling of dress under different illuminants. Percentage of observers labeling the dress as white-gold for (a) the original illuminant, (b) the blue illuminant, and (c) the yellow illuminant.

Figure 7

 

Dress similarity rankings. (a) Rankings of dress similarity with original illuminant and blue illuminant. (b) Rankings of dress similarity with original illuminant and yellow illuminant. (c–h) Stimuli used in Experiment 2: paired dresses with original on left and dress under blue illuminant on right, for all filter sizes. (j–p) Stimuli used in Experiment 3: paired dresses with original on left and dress under yellow illuminant on right, for all filter sizes. Photograph of the dress used with permission. Copyright Cecilia Bleasdale.

Method

Participants

A total of 203 observers completed the survey online via Amazon's Mechanical Turk and were compensated. We selected the sample size of 200 after running a pilot survey to determine response rates and used it for all following experiments. The survey closed after 200 submissions. We had an N of 203 because Mechanical Turk allowed three observers who had started prior to closing to complete the survey. Thirty-two observers who failed to respond correctly to catch trials were dropped from analysis. Results were analyzed from the remaining 171 participants. All observers had normal or corrected-to-normal vision and provided informed consent as approved by the American University Institutional Review Board. 

Stimuli and procedure

The images used were 21 individual photos of the dress; each filter size (unfiltered, 256, 128, 64, 32, 16, eight) for each illuminant (original, yellow, and blue) was represented. Each dress image was 450 × 300 pixels and was presented in isolation on the screen, so observers could view only one image at a time. For each image, observers were asked to respond to the prompt, “The colors in this dress are most like” by selecting “white and gold” or “blue and black.” Observers had no time limit for how long they spent on each image. The 21 images were randomized for each observer so that the order was nonsequential, and each image appeared one time, except for the original, unfiltered image, which appeared three times. These three trials were compared as catch trials; observers who varied in response across conditions were excluded from analyses. 

Results

For the unfiltered images, the observer reports of the color of the dress were as follows: for the original image (i.e., no simulated illumination), 70% reported white-gold and 30% blue-black (similar to informal online polls); for the simulated yellow illumination, 85% of participants reported white-gold and 15% blue-black; for the simulated blue illumination, 16% white-gold and 84% blue-black. The results confirm the observation in Demonstration 1 that adding a yellow illuminant creates a shift toward reports of white-gold and adding a blue illuminant creates a shift toward reports of blue-black. 

Removing low spatial frequency content increases the percentage of observers who report seeing the dress as white-gold in both the original condition (an 11% increase) and blue illuminant condition (a 60% increase). Observers were classified into either a white-gold group or blue-black group based on the colors assigned to the original, unfiltered dress image. Figure 6a shows the percentage of observers in each group who labeled the dress under the original illuminant as white-gold, and Figure 6b and c show the same for the blue illuminant and yellow illuminant, respectively. For the white-gold group, the dress under both the original illuminant and the yellow illuminant is labeled as white-gold by more than 98% of observers for each filter size, and the dress under blue illumination is labeled as white-gold by 91% or more of observers after filtering at sizes 64 and smaller. 

Although the graphs for the original and illuminant conditions show a generally steady increase of observers from the blue-black group labeling the dress as white-gold, the yellow illuminant condition shows a large decrease from the unfiltered condition to the size 256 filter. This drop makes sense, however, when compared with the color values seen in Figure 3, in column a. Adding a yellow overlay causes the values of the measured squares to appear white and gold, whereas filtering to 256 brings the values into a blue and brown range. It is not until entering the midrange of filter sizes, 64 and below, that the filter appears to create a color constant range of values in the stimulus. 

Experiments 2 and 3: The effects of filtering and dress similarity

To measure color constancy between illumination conditions, we collected observer rankings of similarity between sets of two dress images; the dress under the original illuminant paired with either the dress under the blue illuminant (Experiment 2) or paired with the dress under the yellow illuminant (Experiment 3); the pairings were done across all filter sizes. We hypothesized that regardless of whether the paired dresses were perceived as white-gold or blue-black by observers, that rankings of similarity would increase as filter size decreases and more low spatial frequency content is removed from the images. We increased the opacity of the overlays for the experimental conditions to simulate more extreme blue and yellow illuminants. In the original images, the opacity was set to 20%, which we increased to 40% for the experimental stimuli, as can be seen in Figure 7 below. 

Method

Participants

In Experiment 2, 207 observers completed the survey online via Amazon's Mechanical Turk and were compensated. Thirty observers who failed to respond correctly to catch trials were dropped from analysis. Results were analyzed from the remaining 177 participants. In Experiment 3, 205 observers completed the survey online via Amazon's Mechanical Turk and were compensated. Eleven observers who failed to respond correctly to catch trials were dropped from analysis. Results were analyzed from the remaining 194 participants. All observers had normal or corrected-to-normal vision and provided informed consent as approved by the American University Institutional Review Board. 

Stimuli and procedure

The images used in each experiment were paired photos of the dress for each filter size (unfiltered, 256, 128, 64, 32, 16, eight). Experiment 2 presented the original dress image and the image under the blue illuminant; Experiment 3 presented the original dress image and the image under the yellow illuminant. In each pairing, the dresses were marked as 1 and 2, and the combined image of the two dresses was 650 × 500 pixels; the stimuli can be seen in Figure 7. 

Each block of paired dresses and questions was presented in isolation on the screen, so observers could view only one image set at a time. For each block, observers were asked to respond to the prompt, “The colors in dress 1 are most like” by selecting “white and gold” or “blue and black”; observers answered the same question for Dress 2. Observers then rated the similarity of the two dresses (1 = very different, 2 = different, 3 = similar, 4 = very similar/identical). In addition to the seven experimental conditions, there were two catch trial blocks: one with the unfiltered dress under the original illuminant, repeated twice, and one with the unfiltered dress under the blue illuminant (for Experiment 2) or yellow illuminant (for Experiment 3), repeated twice; observers who did not select matching colors for identical dresses were dropped. The nine blocks were randomized for each observer so that the order was nonsequential, and each block appeared once; observers had no time limit for how long they spent on each image pairing. 

Results

Regardless of group (white-gold or blue-black) or illumination condition (yellow or blue), removing a higher proportion of low spatial frequencies increased rankings of dress similarity. The results can be seen in Figure 7a for the blue illuminant and 7b for the yellow illuminant. For the blue illuminant, observers in the white-gold group averaged 1.92 for the original pairing, whereas those in the blue-black group averaged 2.81. As filter size decreases and the amount of low spatial frequency information removed increases, both groups increased in similarity rankings: white-gold averaged 3.77, and blue-black averaged 3.7. For the yellow illuminant, observers in the white-gold group averaged 2.66 for the original pairing, whereas those in the blue-black group averaged 2.15. As filter size decreases and the amount of low spatial frequency information removed increases, both groups increased in similarity rankings: white-gold averaged 3.51, and blue-black averaged 3.26. 

Summary and discussion of Experiments 1–3

We have measured the effects of removing low spatial frequency content on the appearance of the dress. If the high-pass filter acts to remove the illuminant (as part of the color-constancy hypothesis), then high-pass filtering should allow observers to see the dress more veridically (i.e., blue-black). We should therefore expect that when low spatial frequency content is removed from the images, more observers would report the dress as blue-black. However, Experiment 1 demonstrates that as more low spatial frequency content is removed from the image, more observers see the dress as white-gold. 

Experiments 2 and 3 set out to give empirical documentation to the question asked in Demonstration 2; that is, can high-pass filtering act to neutralize the effect of the simulated illuminant placed on the dress? As would be expected from the demonstration, as more low spatial frequency content is removed from the image, the dresses under simulated blue and yellow illumination appear more similar to the original dress image. The results lead to the suggestion that individual differences in the perception of the dress could be related to variation in the processing of low spatial frequency content available within the image. 

Experiment 4: Contrast sensitivity and the dress

One obvious question is whether differences in the dress correlate with observers' sensitivity in the low spatial frequency portion of the spectrum. The field of optometry has spent considerable effort examining how individuals differ in the processing of high spatial frequency content, because deficits in the high-frequency region of the spatial spectrum can prevent people from reading, driving, and a host of other social activities. Individual differences in gain control at the other end of the spatial scale must exist but are less documented because they do not cause the same level of social deficit as high spatial frequency deficits (see McCourt, Leone, & Blakeslee, 2015, for a recent investigation into the substantial individual differences in low spatial frequency sensitivity). We therefore looked for differences between observers' perception of the color of the original dress image and measurements of contrast sensitivity. 

Method

Participants

Fifty-three observers (n = 25 white-gold, n = 28 blue-black) completed an experimental measure of contrast sensitivity function (CSF) in the lab. Fifty-one were American University undergraduates who participated in a larger study that included the CSF measurement, for course credit, and two were American University graduate students who are members of the lab. All observers had normal or corrected-to-normal vision and provided informed consent as approved by the American University Institutional Review Board. 

Stimuli and procedure

The stimuli were presented on a Sony Trimaster EL OLED monitor. The luminance levels of the monitor were measured using a Spectrascan 650 and gamma corrected using the driver software packaged with the computer graphics card (Catalyst Control Center on ATI Radeon HD 5970). Observers were seated 24 in. from the monitor. The stimuli were 45° tilted spatial frequency gratings for six conditions of cycles per degree (0.25, 0.5, 1, 2, 4, 8, 16), generated and presented using PsychoPy (Peirce, 2008). Gratings appeared on a gray background for 50 ms, and observers used the left and right arrow keys to indicate which direction the grating was tilted; between responses, a fixation dot appeared. The gratings were presented in three-down, one-up interleaved staircases with eight reversals per spatial frequency (step sizes of 4, 2, 2, 2, 1, 1, 1, 1 db), beginning at 50% contrast. For each condition, the threshold was expressed as the average of the last four reversals. Participants ran the experiment twice during the session; the results for each participant are the averages from the two runs. 

Results

The data are plotted in Figure 8 and reported in Table 1. The results and shape of the functions are similar to other CSF functions reported in the literature (see Hou et al., 2016). Observers in the white-gold group have a higher average contrast sensitivity than observers in the blue-black group at low spatial frequencies; a two-tailed, independent-samples t test across all conditions showed significant differences for 0.25 and 1 cycles per degree, as seen in Table 1. Although there was no significant difference in the location of peak sensitivity (average equal to 0.88 [white-gold] and 1.1 [blue-black], p = 0.59), there was a significant difference for maximum contrast sensitivity between groups, with white-gold observers having an average of 3.95 to the blue-black observers' average of 3.04 (p = 0.04). 

Figure 8

 

Contrast sensitivity functions of white-gold and blue-black observers.

Table 1

 

Contrast sensitivity (1/threshold contrast), standard deviation, T values, and p values for white-gold and blue-black observers.

 

We also fit the results from individual observer data with a function described by Chung and Legge (2016). The function had four parameters (the peak sensitivity frequency, the peak sensitivity, the width of the CSF function on the right, and the width on the left). Of these, only the peak sensitivity parameter showed a significant difference on a two-tailed, independent-groups t test (mean blue-black = 2.82; mean white-gold = 3.51), t(49) = −2.07; p = 0.043. 

Discussion

Prior to the start of these experiments, we noted that sensitivity measurements should not necessarily be correlated with individual perceptual differences of the dress. One reason for this is that the task of judging color is generally a superthreshold task and, as shown for contrasts greater than 5%–10%, many visual psychophysical tasks are contrast independent (see Sperling & Lu, 1998). A second reason is that we think that the relevant spatial scales for this filtering process may be in some way related to the encoding of objects or visual grouping. For example, if Observer A organizes the scene in a manner that encodes a small framework, and Observer B organizes the scene in a manner that encodes a large framework, then Observer A will exclude more low spatial frequency content than Observer B. Two observers may have identical contrast sensitivity, but Observer A's representation of the framework would be different from Observer B's in the low spatial frequency portion of the spatial spectrum. Thus, there are many ways in which observers can have differences in spatial processing but not have any differences in threshold contrast sensitivity. Although the results show significant differences for CSF measurements for white-gold and blue-black observers, the variability of individual scores within a group and overlapping scores between groups make it seem unlikely that this is what is fully driving the group differences. 

General discussion

Most explanations of the dress assume that a central task of color perception is to infer the reflectance of surface material by way of discounting the illumination falling on the object. In this article, we pursue a different approach based on the idea that many aspects of color constancy can be achieved by removing low spatial frequency content from the image. The idea has similarities to other models of color constancy (Buchsbaum, 1980; Judd, 1940; Land, 1986), to physiological explorations of color vision that suggest that cortical double-opponent cells are spatially band-pass (Conway, 2002; Shapley & Hawken, 2011), and to models of brightness/lightness perception based on spatial filters (Blakeslee et al., 2005; Perna & Morrone, 2007; Robinson et al., 2007). We implemented the removal of low spatial frequency content with the simple filter model of Shapiro and Lu (2011) and Dixon et al. (2014). We show that a simple spatial filter can account for (a) the Rubik's cube illusion, an iconic demonstration frequently used to discuss how the visual system discounts illumination; (b) the differences produced by adding simulated illumination to the dress image; and (c) the effects produced by changing real scene illumination on a colored object. 

As noted earlier, the basic principle underlying many of the demonstrations in this article is trivial: First, low spatial frequency content is added to an image, and then the low spatial frequency content is removed with a high-pass filter. Trivial as this may seem, a simple filter can be useful because changes in global illumination primarily have their effects in the low spatial frequency portion of the visual image. The visual system contains many processes that act like high-pass filters (e.g., spatial response of double-opponent cells, changes of ganglion cell response with eye movements, removal of motion blur, representation of objects). If observers differ in one or more of these spatial processes, it is likely that they will also differ in how they encode the effects of illumination in the image and, subsequently, may therefore differ in their reports of the color of the dress. And although this may seem like a reasonable first place to begin inquiry into individual perceptual differences of the color of the dress, we call attention to the fact that the four most recent reviews of color constancy (Foster, 2011; Shevell & Kingdom, 2008; Smithson, 2005; Xiao, 2016) do not mention spatial filters or spatial scale representations. Hence, it should not be surprising that spatial filtering approaches were absent from other discussions of the dress. 

Representation of the color at different spatial scales

The demonstrations in Figures 1, 2, and 3 illustrate the idea that global changes in illumination can be discounted simply by removing the low spatial frequency content. To be clear, we are not suggesting that the visual system completely discards low spatial frequency information, nor do we wish to imply that high spatial frequency representations always are invariant to illumination. Rather, the demonstrations show the utility of considering color information at a range of spatial frequency scales: Color in the high spatial frequency range will encode different functional information than color in the low spatial frequency range. 

We present one more demonstration to illustrate these differences. In Figure 9a, we show multiple copies of the dress in front of a blue-yellow gradient background. The dress itself does not show induction, so all copies of the dress appear fairly similar to each other (whether white-gold, blue-black, or another variation dependent on the observer). Figure 9b shows a high-pass filtered version of the image, and Figure 9c shows a low spatial frequency version of the image. To arrive at these images, we calculated the width of the dresses (about 170 pixels in our image) and then filtered the original image with a high-pass filter with a radius of 170 pixels (Figure 9b) and with a Gaussian blur of 170 pixels (Figure 9c). 

Figure 9

 

Filtering the dress on a gradient background. (a) Original image repeated across yellow-blue gradient background. (b) High-pass filtered version of (a) with filter diameter of 170 pixels. (c) Gaussian blurred low-pass version of (a) with blur diameter of 170 pixels. (d) Combined high-pass and low-pass images with alpha of 0.5. Photograph of the dress used with permission. Copyright Cecilia Bleasdale.

Both Figures 9b and 9c can be thought of as back-pocket approximations of the information encoded by color vision systems with two spatial resolutions. For the high-pass image (Figure 9b), the dresses are physically different from each other. For the low-pass image (Figure 9c), the dresses, rather remarkably, disappear; the filtered image shows only the gradient background. An exclusively high spatial frequency visual system would therefore encode four dresses of differing hue, whereas an exclusively low spatial frequency visual system would not encode the dress at all. 

Our visual system must somehow combine these different types of spatial representations into a coherent image. To simulate the synthesis, we create a new image in Figure 9d by overlaying 9c with a transparency of alpha equal to 0.5 over 9b. In the resulting image, the dresses (not surprisingly) again appear the same, as in 9a. So, although the dress images appear dissimilar at a particular scale after spatial filtering, they appear identical again once low-pass information is added back into the image. Images contain information at multiple spatial scales, and each scale may contribute a different interpretation of the world; any particular visual process could weight the information according to its needs. Variations in weighting of any level of frequency would necessarily result in variations in perception (e.g., observers who perceive very little illumination in the original dress image versus those who perceive high illumination). 

The idea that there are multiple spatial representations is a hallmark of 20th-century psychophysics and physiology (Graham, 1989), and it seems likely that spatial filtering should play a role in color vision. In general, we are agnostic to the particular quantitative and physiological embodiment of spatial frequency attenuation and combination. However, Dixon et al. (2014) showed that the cutoff frequency for the high-pass filter adjusts to information in the scene and seems to be tuned to the size of objects. By tying the size of the spatial filter to objects rather than to a low-level response, we can posit adjustments to spatial channel weights based on other upstream factors that may affect color or brightness, such as attention (Tse, 2005), grouping (Gilchrist & Radonjić, 2010; Xian & Shevell, 2004), and motion (Werner, 2007). There has been some evidence for objects being invariant to changes in distance for some tasks, thereby suggesting visual representation in terms of object spatial frequency and not retinal spatial frequency (Peterzell, 1997; Peterzell, Harvey, & Hardyck, 1989). 

Individual differences and the dress

Many of the arguments about the dress have relied on demonstrations to illustrate possible ways of interpreting the image of the dress and have primarily concluded that the visual system makes an inference about some aspect of illumination, such as direction or color. We have written this article to show that an explanation for these demonstrations can also be cast in terms of information in the image that the visual system can extract through spatial filtering. In a filter-based approach, individual differences in the perception of the dress would arise if observers differ in the amount of high spatial frequency content extracted from the image (or, conversely, the amount of low spatial frequency content attenuated) or if observers give different weights to processes that differently filter high and low spatial frequency content. At the suggestion of a reviewer and the editor, we measured for correlations between observers' contrast sensitivity functions (CSF) and the perception of the dress. We found that the white-gold observers had higher contrast sensitivities at lower spatial frequencies as well as a higher overall peak contrast sensitivity. The result is consistent but not necessary because the differences in spatial response between groups could arise at superthreshold levels. 

We have shown that the dress tends to appear white-gold when low spatial frequencies are filtered from the image. This result seems contrary to what would be predicted if the visual system were attempting to “recover” an illuminant-independent description of the object's reflection. That is, because the high-pass filter in effect removes the illuminant, the prediction should be that after the filter is applied, observers should shift the perception toward blue-black. The white-gold response therefore argues against a theory based on the estimation of the material reflectance. The results suggest that either the filter has an effect that is dissimilar from removing the illuminant or that attenuating the illuminant with a physical filter has the reverse effect of what happens when the brain discounts the illuminant. 

Or, alternatively, the goal of the visual system is not to recover the “true” color; rather, color is encoded by a system that preferentially responds to the high spatial frequency content extracted from the image, and there is no process that infers the material property. We sometimes think the visual system infers the material property because the high-pass information (possibly carried by the spatially band-pass double-opponent cells in the cortex) remains constant and therefore has the same outcome as a putative cognitive function that estimates the material. Although such an approach seems to open new ways to consider individual differences, the predictions that it makes depend on the specification of the linking assumption that connects responses to high spatial frequency content to color perception. For instance, observers could differ in their perception of the dress because they differ in channel tuning (i.e., they differ in the amount of low spatial frequency attenuated; observers, in effect, have different gain controls for the low spatial frequency channel), observers could construct the perception of the dress through a comparison of the low and high spatial frequency responses, or observers could differ in how their perceptual system encodes contrast within the high spatial frequency system. At this time, we cannot differentiate between these specific types of linking assumptions. So, although our results are inconsistent with the standard hypothesis (i.e., if the high-pass filter discounts the illuminant, then as low spatial frequency content is removed from the image, more people should see the dress as blue-black), the data do not reject the alternative hypothesis. 

Limitations

We have shown that a very simple model can account for demonstrations related to the dress and hopefully will give new insights into color constancy. To be clear, we are not suggesting that a simple high-pass filter can explain all brightness illusions and all of color constancy. First, it is unlikely that the filtering process is unitary; spatial filters occur at many different stages of visual processing and are likely to vary for different functional pathways. The notion that the dress occurs along a yellow-blue line in color space (Winkler et al., 2015) may represent differences in filtering for the ancient tritan S-(L-M) system compared with other color systems. We note here that our filter model could also be considered a blunt one-dimensional version of von Kries adaptation (Buchsbaum, 1980), because such von Kries adaptation implicitly examines color contrast over local spatial regions (Khang & Zaidi, 2004). Given our results, one might expect that expanding von Kries adaptation models to account for different spatial scales would be productive, but the testing of any hypotheses in this regard would require experiments beyond the scope of this article. 

Second, we restate the caveat given by Shapiro and Lu (2011) that some brightness illusions cannot be accounted for by a simple filter. Illusions such as the Craik-O'Brien-Cornsweet effect, Long-range Argyles (Flynn & Shapiro, 2014), and the Watercolor Effect (Pinna, Brelstaff, & Spillmann, 2001) show changes that are generated at a thin edge but extend over very large distances. In addition, it is clear that processes that generate brightness (and contrast) occur at several stages of visual processing (Flynn & Shapiro, 2013; Shevell, Holliday, & Whittle, 1992). Furthermore, the addition of low spatial frequency transparent layers to contrast illusions has curious perceptual consequences (Dixon & Shapiro, 2014); a simple high-pass filter cannot fully account for the additional strength of these illusions. 

Third, the filter used for our demonstrations has one parameter: the amount of low spatial frequency content removed from the entire image. A curious fact is that the size of the parameter seems to be tied to the size of the objects in the images (Dixon et al., 2014). This suggests that filtering is not passive and may be part of higher-order functional processes. It could be that other factors often associated with perceptual inferences and top-down processing—scission, grouping, object formation, and so forth—could affect the filters or actually are the filters. As we have noted elsewhere, object representations necessarily exclude low spatial frequency content. 

General conclusion

A long-standing divide in the vision science literature concerns the role of perceptual inferences about the distal stimulus versus the role of early filters in shaping visual response (see Kingdom, 1997, for a concise summary of the historical division). To date, the predominant theories for the dress phenomenon have relied on a strong role for perceptual inferences (i.e., the approaches assume that observers infer material reflectance by estimating and discounting the illuminant). Here, we have tried to show how the dress can, in principle, be accounted for by variations in a simple spatial filter that extracts relevant information from the environment. We have also shown that changing the spatial content in the image leads some observers to switch their color categorization of the dress. The result is consistent with the reports that increasing image size leads to a higher proportion of white-gold responses and that blurring the image of the dress leads to a lower proportion of white-gold responses (Lafer-Sousa et al., 2015). 

The results support the hypothesis that, under some conditions, high spatial frequency content remains invariant to changes in illuminant. Furthermore, if visual subsystems weight spatial information according to the task (e.g., object perception will give a greater weight to high spatial frequency content than to low spatial frequency content), then any subsystem that gives less weight to processes that respond to low spatial frequency content will automatically have some degree of color constancy. Conversely, any subsystem that responds primarily to low spatial frequency content will respond primarily to changes in the global illuminant. Of course, this hypothesis may not be true for complex scenes with multiple illuminants or large amounts of interreflection. 

To be clear, we are not suggesting that a single spatial filter is a complete explanation for color constancy and the dress. We are saying, however, that measurements of luminance and color that disregard spatial frequency scale are often incomplete and can mislead as to what the visual system is capable of encoding. There is a wealth of information in the stimulus that many standard proposals of color constancy seem to ignore; in addition to high spatial frequency content and low spatial frequency content, there are other types of information that the visual system can extract from the environment (see Adelson & Bergen, 1991). For instance, the visual system can separate changes in color and changes in color contrast (Shapiro, 2008; Whittle, 2003); color contrast information remains constant with illumination changes, whereas color information remains constant when backgrounds change (Brown, 2003). 

Attention to these aspects of the stimulus may give insight to many other problems in the color constancy literature that tend to be explained by cognitive mechanisms or Bayesian inferences (see issues raised by Arend & Reeves, 1986). It is likely that individual differences in extracting any of these sources of information are related to individual differences in visual perception. Nonetheless, as pointed out by Brainard and Hurlbert (2015), a complete understanding of individual differences and the dress would necessitate measurements ranging from preretinal mechanisms through neural mechanisms and onto cognitive mechanisms for each observer. Although an overexposed photo of a striped dress may not seem to be the ideal stimulus for understanding the visual system and color constancy, the image may prove a useful starting point to test theories of individual differences in perception at varying spatial scales. 

Acknowledgments

The authors thank Laysa Hedjar and Divya Nigam for coding the experiment and running the observers in the contrast sensitivity study, and Dr. Sherri Geller for editorial help. 

Commercial relationships: none. 

Corresponding author: Erica L. Dixon. 

Email: . 

Address: Department of Psychology, American University, Washington, DC, USA.