Interactions between luminance and color signals: Effects on shape

Abstract

Abstract

Abstract:

Abstract  Although luminance and color are thought to be processed independently at early stages of visual processing, there is evidence that they interact at later stages. For example, chromatic information has been shown to enhance or suppress depth from luminance depending on whether chromatic edges are aligned or orthogonal with luminance edges. Here we explored more generally how chromatic information interacts with luminance information that specifies shape from shading. Using a depth-matching task, we measured perceived depth in sinusoidal and square-wave gratings (specifying close-to sinusoidal and triangle-wave depth profiles, respectively) in three conditions. In the first, as we varied luminance contrast in the presence of an orthogonal chromatic grating, perceived depth increased (consistent with classical shape from shading). When we held the luminance at a fixed contrast and varied the chromatic grating in the other two conditions (orthogonal or aligned), we found large and inconsistent individual differences. Some participants exhibited the expected pattern of enhancement and suppression, but most did not, either for the sinusoidal or square-wave stimuli. Our results cast doubt on the idea that the interaction demonstrates a single high-level heuristic linked to depth perception. Instead, we speculate that interactions are more likely due to early cross-channel masking.

Introduction

Our visual systems allow us to perceive the world in color. To do this requires a complex biological apparatus and extensive processing. A key issue is to understand the advantages that color vision bestows over a simpler achromatic system. It is well known that variations in luminance allow us to obtain information about object shape (e.g., Horn, 1975; Horn & Brooks, 1989; Kerrigan & Adams, 2013; Langer & Bulthoff, 2000, 2001; Pentland, 1982a, 1982b, 1989; Ramachandran, 1988; Schofield, Hesse, Rock, & Georgeson, 2006; Sun & Perona, 1998; Tyler, 1998). In this paper we explore a related phenomenon, where variations in color can affect the amount of shape from shading that luminance delivers. This has been called the color-shading effect (Kingdom, 2003; Kingdom, Rangwala, & Hammamji, 2005; Kingdom, Wong, Yoonessi, & Malkoc, 2006) but is relatively little studied. Here we explore the effect of color on luminance shape from shading across two different kinds of stimuli and a range of naïve observers. Below we review work that has been done in this area before describing our study in detail. 

Natural objects of a uniform color can show gradients of lights across their surfaces. This is due to the changes in surface orientation with respect to the light source. Hence luminance gradients can, in principle, tell us about object shape, assuming a constant known (or inferred) light source. The shape from shading resulting from such gradients has been studied in detail both in terms of modeling what is possible (e.g., Horn, 1975; Pentland, 1982a, 1982b), and by measuring human performance (e.g., Todd & Mingolla, 1983, who showed that observers can extract information about curvature from shaded cylinders). Perceived shape can be affected by changes in the scene that are not related to the object for example, by varying the lighting direction (Nefs, Koenderink, & Kappers, 2005). Three-dimensional shapes inferred from 2-D images can be perceived as ambiguous (e.g., Curran & Johnston, 1996). This is perhaps not surprising as luminance and chromatic gradients in a scene can be due to a host of factors: material or reflectance changes (e.g., the patterning on the surface of a leaf), differential reflection of light due to object shape (which we can interpret as shape from shading), shadows, and interreflections. 

Gradient information is inherently ambiguous, and information from other sources might therefore be helpful in disambiguating what luminance delivers so that shape can be extracted (see recent review by Kingdom, 2008). However, under certain circumstances, luminance and/or chromatic gradients can act as a robust cue to shape. Gradient information can be combined with chromatic information efficiently. Harding, Harris, and Bloj (2012) showed that the relationship between light direction, the object reflectance, and the shape of an object could be quickly learned by an observer. Gradient information can also be combined with binocular disparity cues: Lovell, Bloj, and Harris (2012) showed that this was done optimally and fast (Lovell et al., 2013). 

Color provides a potential way to disambiguate luminance. We know that color and luminance are processed separately at early stages of visual processing (for early work, see Livingstone & Hubel, 1988). As early as the retina, the human visual system separates visual input into one luminance and two chromatic channels. Each of these channels has different acuity, temporal properties, and spectral properties (for a review on pathways focused on color vision see Solomon & Lennie, 2007; Gegenfurtner & Kiper, 2003; and on achromatic channels see Wilson & Wilkinson, 2004). 

When considering the encoding of visual information the brain is usually hypothesized to decorrelate the signal to save computing power and remove redundancy. This efficient encoding strategy has been hypothesized for stereopsis (e.g., see Li & Atick, 1994) but also as the reason for separated luminance and color channels. Statistical independence has been shown between luminance and chromatic edges in natural scenes (Cecchi, Rao, Xiao, & Kaplan, 2010; Hansen & Gegenfurtner, 2009), although the exact value of mutual information might vary depending on how the edges and the correlation metric are computed and also depending on the dataset of natural images used. The data suggests that luminance and color edges do not tend to coincide. This makes sense when one considers the toy example of a cast shadow across a path: The shadow will cause a luminance edge, but hue will remain constant across the edge. As De Valois and Switkes (1983) pointed out, it makes sense for the visual system to use color differences rather than luminance ones to segregate objects from their backgrounds. Conversely, if the material changes, perhaps from stone to grass, luminance and color edges will likely coincide (e.g., see illustration in Kingdom, 2003). 

There are results from the literature suggesting interactions between color and luminance information, with shape information. For example, there is a proven influence of shape on color perception. Bloj, Kersten, and Hurlbert (1999) showed that when the stereo-defined shape of a real folded card (one side white the other red, the chromatic Mach card) was changed by optical means from an open book shape to that of a roof, the perceived color of the white side changed from white to pink. This was suggested as being due to the visual system's interpretation of chromatic and luminance gradients arising from interreflections. In a related phenomena (AMBEGUJAS, see Bergstrom 2004), perceived color was studied for a folded card stimulus that was bistable: Observers alternated between perceiving pairs of panels as forming roof-shaped convexities or book-shaped concavities. The apparent color and brightness of panels changed when an object's apparent shape flipped from one state to the other. In both studies 3-D shape information had an effect on color perception. 

Color and luminance signals segregated into parvocellular (red-green), koniocellular (yellow-blue), and magnocellular (dark-light) channels can potentially interact at different processing stages: retinal, lateral geniculate nucleus (LGN), and cortical (see Horwitz & Albright, 2005; Johnson, Hawken, & Shapley, 2001, 2008; Nassi & Callaway, 2006). Non-oriented filters are monocular and emerge from the LGN. The interpretation of orientation selectivity is a bit more difficult for chromatic information as non-oriented blobs exists in V1. We note that an additional level of complexity has been suggested; there is a possibility of multiplexing of color (red-green) and luminance information in the parvocellular pathway (e.g., see Billock, 1995; Billock & Tsou, 2004). In turn the cortical simple cells will exploit this multiplexing, and it has been suggested that there might be two distinct pathways processing chromatic (red-green) information, one non-oriented and monocular and the other one orientation selective and binocular (Gheiratmand, Meese, & Mullen, 2013). The multiplexing hypothesis is however still controversial (for example, see Lee, Sun, and Valberg, 2011). In order to test interactions between channels and to pinpoint the locus of these interactions, it is common to explore responses to stimuli presented monocularly, dichoptically, or binocularly. There is evidence for two stereopsis channels, one chromatic and one luminance (e.g., Simmons & Kingdom, 1997). Simmons and Kingdom (2002) showed that the two interacted with each other. They concluded that these interactions happened after the disparity of each input was processed, making it a purely cortical process. 

There is also evidence that color information can have an effect on perceived shape and depth. The specific effects of color edges on luminance edges during judgments of shading have been studied by Kingdom's lab using sinusoidal luminance and color patterns that can be aligned or nonaligned (Kingdom, 2003; Kingdom, Rangwala, & Hammamji, 2005). They described the “color-shading effect” as the modulatory effect of chromatic components on the perceived depth of a luminance-defined (shape from shading) modulation. There were two modulatory effects found, an enhancing effect when the chromatic component was nonaligned with the luminance component (originally found when the two were orthogonal in Kingdom, 2003) and a suppressive effect, when the two components were iso-oriented and in phase. It has been suggested by Kingdom (2003) that these effects are the results of heuristics used to disambiguate the origin of luminance variations. When chromatic and luminance edges coincide, the edge is likely due to a reflectance change (perhaps a change in pattern or material). When the edges are not aligned, this is more likely to be because the luminance edges are caused by object shape change. 

The color-shading effect was previously explored using combinations of sinusoids (Kingdom, 2003; Kingdom et al., 2005) or with combined sinusoids and texture (Kingdom, Wong, Yoonessi, & Malkoc, 2006). Kingdom et al. (2005) showed that the effects generalized from red-green to blue-yellow chromatic components. Kingdom et al. (2006) showed that the color-shading effect worked when combining shape from shading and shape from texture.¹ In all these studies the luminance component bore the shape of the object via shape from shading, and the color had specific effects on the shape from shading percept depending on whether color and luminance components were aligned or orthogonal. In the iso-oriented in-phase condition, it was found that chromatic contrast has a suppressive effect on perceived depth of the corrugations. This suppressive effect disappeared when the color and luminance components were out of phase. If the color-shading effect occurs because of heuristics applied by the visual system, it should extend to a wider range of stimuli than those previously explored. In this paper we explored a wider range of stimuli than those previously tested and used a larger population of naïve observers. 

Aim

Our aim was to explore the generality of the heuristics proposed by Kingdom (2003) by testing if the color-shading effect works with different kinds of shape profiles. A straightforward manipulation is to replace sinusoids with a hard-edged pattern, such as a square wave on screen (producing a triangle-shaped profile in depth). This effectively adds some harmonics of spatial frequency (Kingdom & Simmons, 1998). In terms of the shape perceived it turns a corrugated pattern that is close to sinusoidal in depth (Kingdom et al., 2005; Wright & Ledgeway, 2004) into a hard-edge folded triangular wave. It has been shown by Sun and Schofield (2012) that square-wave luminance patterns tend to be perceived as triangular surfaces in depth. 

Another important point of our study was to simplify the experimental design and procedure that had been used before. In previous studies (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006), stimuli consisted of three components: The color sinusoidal component was always paired with a luminance component of the same orientation. Experiments explored how this luminance-color pair affected the depth perceived in a third luminance-only or color-only sinusoidal component (which could have a range of orientations). Yet the logic behind the interpretation of the effects found suggests that the effect should work with only a pair of components. Here we used only two oriented components, one red-green chromatic and one luminance defined. This was done to avoid possible contamination of the results from low-level masking effects which have been described between luminance and color (e.g., Medina & Mullen, 2009). 

Methods

Observers

A total of 12 observers were tested. All observers were naïve to the task and hypothesis tested. All of them had no color deficiency as screened using the Ishihara color plate test (38-plate edition, 1979). Observers also were screened for stereo-vision using the TNO test, and no participants were rejected using those criterion. Observers had normal vision or corrected to normal. Ethical approval was given by the University of St. Andrews ethics committee UTREC (reference PS6135) and followed university guidelines. Participants were given monetary compensation for their time. 

Apparatus

The stimuli were displayed on a CRT color monitor Mitsubishi Diamond Pro (110 Hz, noninterlaced). The CIE-1931 chromaticity coordinates of the red, green, and blue phosphors were x = 0.620, y = 0.340; x = 0.290, y = 0.604; and x = 0.149, y = 0.071, respectively. The gun chromaticity values were obtained by measuring the full spectra of the gun with a Photo Research spectroradiometer PR-650 and then converting into CIE-1931 (2° standard observer) values by multiplying the functions and spectra. The stimuli were generated using Cambridge Research Systems ViSaGe and Matlab. The screen gamma nonlinearity was corrected using a CRS ColorCal colorimeter. Details on stimuli generation as well as the color space used are described in the following sections. 

Participants viewed the screen through a Wheatstone stereoscope. Using sets of mirrors, this stereoscope splits the available visual field on the CRT in two. Effectively the left part of the screen will be only visible to the left eye and the right part will be only visible to the right eye. A chin rest was mounted with the stereoscope to hold the observer's head steady at the chosen viewing distance of 70 cm. 

Stimuli and task: General

Stimuli consisted of combinations of luminance and chromatic sinusoids or square waves, positioned below a random dot stereogram depicting a circular slab at a different depth from its circular background (see Figure 1a). Below we describe the stereo-defined stimuli, represented in the upper half of Figure 1a. Observers were required to adjust the stereoscopic depth-matching stimulus until it appeared to contain the same depth (defined by binocular disparity, Figure 1b) as the peak-to-trough depth depicted by the luminance sinusoid or square wave (defined by shape from shading), see Figure 1c. 

Figure 1

 

(a) Top: Illustration of the matching stimulus used for the experiment. When presented via a stereoscope, observers perceive a circular depth pedestal in the middle of the matching stimulus. Bottom: color-shading stimulus, always identical in both eyes (no disparity). (b) Side view of the disparity profile, the inner disk of the top stimulus is adjustable using a dial. (c) Depth profile, observers were asked to match the depth perceived from shading to the disparity-defined stimulus.

Stimuli: Depth matching

A stereoscopically defined depth pedestal was used to assess the amount of perceived depth in the target-luminance pattern. The stimulus (see top half Figure 1a) was a random-dot stereogram, composed of a circular patch (4° diameter) and made of prerendered circular Gaussian blobs (standard deviation of 0.8 mm on screen or 3.4 min arc). A central circular pedestal (2° diameter) could be adjusted in depth (see disparity profile, Figure 1b). We used a similar protocol to create our Gaussian blobs stimuli than Kingdom et al. (2005), but we did not use subpixel shifts in disparity. The blobs in the random dots stereogram were non-oriented Gaussian envelopes, darker than the background. The blobs with coordinates falling into the inner 2° disks were modulated in disparity, i.e., when the observer adjusted a dial the stimulus was redrawn with the central patch having a new disparity. The background blobs (outer circle) were presented at 0 min arc disparity (see Figure 1b). 

Previous experiments on the color-shading effect used disparity defined matching stimuli that resembled the final percept. As we wanted to compare the depth perceived via sinusoids and square-wave patterns, we chose to use a disk-shaped pedestal. This matching stimulus makes no assumptions about the shape of the perceived object, which can vary between observers for the square wave (see Sun & Schofield, 2012). Observers were asked to turn a dial to match the depth between the central patch and the background (Figure 1c) with the amount of depth they perceived from shape from shading for the luminance grating: peak to trough from the lowest to the highest depth. 

Stimuli: Luminance and chromatic

The test stimuli were composed of one chromatic and one luminance component of the same profile type but different orientation and contrast. The luminance component carried the shape of the object according to shape from shading. In our experiment, the type of pattern used to generate the chromatic and achromatic components could be either sinusoids or square-wave modulations. A square-wave pattern on screen, corresponds roughly to a triangular profile in depth. A sinusoidal pattern corresponds to a corrugated material with an approximate sinusoidal shape. All stimuli used are shown in Figure 2 for both sinusoids and square waves. Below we describe the methodology and color space used to create these patterns and later the experimental conditions tested. 

Figure 2

 

Illustration of the stimuli used. All stimuli are shown for sinusoids and square wave. Each condition is shown on a different row. For each condition one component is fixed and one component is systematically changed. (a) Condition 1, chromatic component fixed and variable luminance contrast in orthogonal orientation. (b) Condition 2, luminance component fixed and variable contrast in orthogonal orientation. (c) Condition 3, luminance component fixed and variable contrast in aligned orientation. During experimental trials conditions were all randomized and sinusoids and square waves were tested in different sessions.

Both chromatic and luminance components had a spatial frequency (fundamental frequency for the square-wave) of 0.75 cpd. Equation 2 below shows the mathematical formula used to generate the sinusoidal modulation, and Equation 3 shows the modification used to obtain a square-wave pattern. To avoid stimulus display artifacts, the two components (i.e., chromatic and luminance) were displayed on different video frames and temporally interleaved. Frame interlacing of a chromatic component is common in color-vision psychophysics (e.g., Kingdom et al., 2005; Michna, Yoshizawa, & Mullen, 2007; Victor, Purpura, & Conte, 1998). This technique has the advantage that it limits any nonlinear interactions on the screen (Victor et al., 1998). It also effectively divides the contrast by two. Contrast values reported in the rest of the paper take this effect into account and represent actual contrast as seen by the participant. The high frame rate of the monitor (110 Hz) allows seamless fusion for the observers. 

Luminance variations were obtained by modulating all cone class receptors in phase. The luminance component, which gives the impression of shape, was always oriented at −45° (left oblique). The chromatic component could be either oriented at −45° (left oblique, aligned with the luminance component) or +45° (right oblique, orthogonal to the luminance component). These two conditions are sometimes referred to in the literature as iso-oriented and cross-oriented, respectively. 

The chromatic component was obtained by setting modulations of inputs to L and M cones to be out of phase (L-M). Thus when L-cone activation is at its highest, M-cone activation is at its lowest, and vice versa. The chromatic component does not change the overall shape of the object; however, we expect it to have a modulatory effect on the perceived depth of the corrugations. 

Color space

Stimuli were created using the cone contrast space as defined by Cole, Hine, and McIlhagga (1993) and Cole and Hine (1992). Each of the three dimensions of this color space (Lc, Mc, Sc) is computed by dividing the cone intensity by the background intensity. Cone contrast is defined for each type of cone by Equation 1:  

 

 

 

 

 

where Δε_L, Δε_M, and Δε_S are variations of the stimulus cone excitation from the background cone excitations, ε_bL, ε_bM, and ε_bS. The background intensity we used was x = 0.282, y = 0.311, and Y = 40 cd/m² in CIE coordinates, the same values used in Kingdom, Rangwala, and Hammamji (2005). 

The long/medium/short photoreceptor activations (LMS cone excitations) for the guns were obtained using a transformation from CIE 1931 to Smith and Pokorny (1975) cone fundamentals. These values were then used to create a linear transform from LMS to gun values (see method in Brainard, Pelli, & Robson, 2002). This allows us to display any triplet of Smith and Pokorny (1975) cone excitations ε_L, ε_M, ε_S, on the screen as long as they fall within the display's gamut. However, as the visual system encodes information using contrast, we defined our stimuli using contrast metrics, as described below. 

Luminance and chromatic components

Each component (luminance or chromatic) was defined as a sinusoidal modulation of L, M, and S cone contrasts. The contrast modulations for the cones were computed respectively using Equation 2 for sinusoids (Csin_L, Csin_M, Csin_S) and Equation 3 for square waves (Csq_L, Csq_M, Csq_S).  

 

 

 

 

 

 

 

 

 

 

 

 

In Equation 2, x and y represent the relative horizontal and vertical distance from the stimulus center. The parameter θ corresponds to the orientation of the sinusoid; θ = 0° produces a vertical sinusoid, θ = −45° a left oblique sinusoid, and θ = +45° a right oblique one (note: for clarity these are expressed in degrees instead of radians). The parameter f represents the spatial frequency (in cycles per degree). The parameter A corresponds to the maximum modulation of the cone contrast. If A = 0 (as it is for the S cone in the red-green chromatic component), then there is no modulation of contrast; the cone excitation will stay at the background value (Equation 1). In this paper when we refer to a stimulus contrast we refer to the amplitude of cone contrast modulation, i.e., parameter A. In the case of luminance modulation, all three classes of cone are modulated with the same amplitude A (i.e., A_L = A_M = A_S), this parameter is equivalent to the Michelson contrast (see Appendix A). 

The luminance components were computed by modulating L, M, and S contrasts in phase. The chromatic (red-green) sinusoid components were computed by having L and M contrast modulated 180° out of phase (|A_L| = |A_M|) and setting A_S = 0. We obtained square waves by transforming the sinusoid using Equation 3. This ensured that the spatial features of the stimuli are preserved between the two conditions. From Equations 2 and 3 we then used Equation 1 to transform cone contrast values into cone excitation (ε_L, ε_M, ε_S); we then transformed LMS cone excitation levels directly into gun values (as described by Brainard et al., 2002). The gun values were subsequently adjusted for linearity and gamma corrected. 

Generally the contribution of L and M inputs to the luminance signal are not equal (Gunther & Dobkins, 2002). Consequently, when L and M cone contrast are out of phase the stimulus might not be equiluminant. In order to correct for this and create truly equiluminant stimuli, we adjusted the amplitude values, A_L and A_M. This adjustment is different for each participant. A control experiment to set equiluminance for each participant was therefore required and was performed for both sinusoids and square-wave stimuli. Full details of equiluminant settings plus data on a minimum motion experiment for each participant can be found in Appendix B. In the remainder of this paper, we identify contrast modulation with the amplitude values before individual participant adjustment. 

Experimental design and procedure

We measured perceived depth in the shape from shading delivered by the luminance component using the method of adjustment. Observers used a stereo-defined patch to adjust the perceived depth. We manipulated several different combinations of luminance and chromatic components, which we will describe below as three conditions. Trials from each condition were randomly interleaved and the task was always the same. The experiment was split into two sessions, one for sinusoids and one for square-wave patterns (the order of which was randomized between observers). Before each session observers performed a minimum motion experiment with identical stimuli to those used during the main experimental sessions (see Appendix B). Observers performed 10 trials per contrast value for each condition and for each pattern profile. Figure 2a is an illustration of all stimuli for both sinusoids and square-wave conditions. 

Condition 1

In order to measure the effect of increasing luminance contrast on perceived depth, we set a fixed color component with a right-oblique orientation (+45°) at a constant contrast amplitude of 0.012. The luminance component was left-oblique oriented (−45°) and tested using six values of maximum contrast (0, 0.01, 0.02, 0.04, 0.08, 0.16; see Figure 2a for illustrations). In this case the luminance contrast measure is equivalent to Michelson contrast (Appendix A) and so these values can be also expressed as percentages. 

Condition 2

The luminance left-oblique component was fixed at contrast amplitude 0.04 and we tested six values of color contrast (0, 0.004, 0.08, 0.012, 0.016, 0.02; Figure 2b). We expected, from Kingdom (2003), Kingdom et al., (2005), and Kingdom et al., (2006), that the perceived depth would increase as the color contrast increased. Note that the fourth stimulus of Condition 2 is exactly the same as the third stimulus of Condition 1 (0.04 luminance component and orthogonal chromatic component at 0.012 contrast amplitude) as can be seen in Figure 2b–c, fourth row. 

Condition 3

The color component was left-oblique oriented and hence aligned with the luminance grating. The values of contrast used were the same as in Condition 2 (Figure 2c). Note that Conditions 2 and 3 are indistinguishable at zero color contrasts; this gives us the perceived depth for luminance only at a fixed contrast of 0.04. From previous work (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006) we expect that aligned in-phase chromatic contrast would suppress perceived depth. 

Data analysis

We analyzed sinusoid and square wave data conditions separately. We recorded matched depth as a function of contrast for each of the three conditions separately for each observer. A function was then fitted to the matched depth versus contrast data, separately for sinusoids and square waves. For each observer, pattern, and condition, we performed a Kruskal-Wallis test on the data to check for a main effect of contrast (luminance or chromatic contrast, depending on condition), see Supplementary File 1 for details and values. 

It is common for luminance contrast mechanisms to show sigmoidal/saturation behavior (e.g., Albrecht & Hamilton, 1982; Dean, 1981). This has been hypothesized to be due to a normalizing mechanism (Carandini & Heeger, 2012) and is used in many perceptual models, sometimes referred as nonlinear transducer behavior (e.g., Legge & Foley, 1980; Schofield, Rock, Sun, Jiang, & Georgeson, 2010). Hence Condition 1 perceived depth data was exclusively fitted with a modified cumulative Weibull function²:  

 

 

Equation 4 has four free parameters γ, the offset parameter, the scale parameter σ, used to scale the functions output on the y-axis, α the threshold parameter, and β the shape parameter. High values of β give step-like functions and lower values (<1) will deliver a more sigmoidal function. We define threshold as the value of α that delivers a value for the function of 63% of maximum. 

For Condition 1 we expect participants to perceive no depth at zero contrast; hence, we do not need an offset, so the fit is performed with three free parameters and γ fixed at zero. For Conditions 2 and 3, if the Kruskal-Wallis test showed a significant effect of chromatic contrast on perceived depth, we would then proceed to fit a curve to the data. If the effect was positive we fitted with a cumulative Weibull, if negative we fitted with a decreasing Weibull, described with Equation 5, the parameters retain the same meaning.  

 

 

The direction of the effect of chromatic contrast was determined by a simple linear fit and the final fit was performed using either Equation 4 or 5, depending on the sign of the slope parameter. Fitting was performed using function nlinfit from MATLAB Statistics ToolBox. We should emphasize that the curve fits provide us with a convenient way to describe and summarize the data; they are not intended as a model that explains the underlying mechanisms of depth modulation perception. 

Results

We will report the raw results for each condition below, followed by a description and discussion of the model fits. 

Condition 1: Fixed color contrast and variable luminance component

In Condition 1 the independent variable was the contrast of the left oblique luminance component. The color contrast was fixed at 0.012. 

Mean matched depth as a function of luminance contrast is plotted in Figure 3 for each observer (sinusoids: black circle, square wave: gray square). For all observers, the increase in luminance produced a significant increase in perceived depth (Kruskal-Wallis tests, χ² (5,54), p < 0.001, see Table 1, Supplementary File 1) for both sinusoids and square waves, except for Participant P5 who was unable to see the 3-D shape of square-wave pattern. For all participants, we obtain similar saturating functions. This allow us to use a Weibull function (Equation 4) to perform the fit to the data, plotted with solid lines on Figure 3. The results are consistent with what we expect from classical shape from shading, where higher amplitude shading corresponds to more perceived depth. Participant 9 is the only individual with a different pattern of responses from the rest of the group. For him, we can see a clear step-like function. On the whole, our observers set maximum depths in the range 1.22 to 4.41 arcmin (mean = 3.38, σ = 0.9). 

Figure 3

 

Results from Condition 1: Mean perceived depth as a function of luminance contrast for each observer (black circle: sinusoids, gray squares: square wave, error bars: standard error of the mean, SEM). The individual curve fits are presented with solid lines.

When comparing sinusoid and square-wave data, the response to the square wave tended to reach the highest perceived depth faster than for sinusoids. These results demonstrate that all our participants were performing the task properly, that is to say responding to the depth shown from the luminance-defined component. As all the conditions were interleaved, we can assume that participants were not switching tasks between conditions. Furthermore, as sinusoids and square waves were tested in different sessions, we can see from the results in Figure 3 that, using our stimuli, shape from shading is a robust phenomenon, with similar maximum depth perceived and similar shapes of fitted function for both sinusoids and square waves. 

Condition 2: Fixed luminance contrast, variable color contrast

In Condition 2, the independent variable was the contrast of the right oblique color component, presented orthogonally to the shape-carrying luminance component. From previous research on the color-shading effect we expected to see an increase of perceived depth with increasing color contrast (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006). 

We found two surprising results. First, for several participants, the chromatic component had no effect on perceived depth in the luminance component (Figure 4). Kruskal-Wallis tests showed that some participants did not show any significant effect of color contrast on perceived depth (see Table 2, Supplementary File 1). Where effects were significant, curves were fit and plotted in Figure 4. Eight participants had a significant color contrast effect (p < 0.05) for sinusoids or square wave or both; the other four participants showed no significant effects at all (p > 0.05). 

Figure 4

 

Results from Condition 2: Perceived depth as a function of color contrast for all observers. (black circle: sinusoids, gray squares: square waves, error bars: standard error of the mean, SEM). When statistics revealed a significant effect of color contrast (see Table 2, Supplementary File 1), the data were fitted with a Weibull function, the individual fits are shown by solid lines.

Second, and perhaps more surprising, of the participants with significant effects, three showed negative effects of color contrast (P4, P6, P9). Thus less depth was perceived as chromatic contrast increased. This is contrary to what was reported previously in the literature (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006) and does not fit the heuristic proposed to explain the effect (Kingdom, 2003). Participant P1 showed no significant effect on the square-wave condition; however, this might be due to lack of statistical power, in which case the direction of the effect would be similar in both sinusoid and square-wave conditions. Participant P12 had an outlier point at contrast zero that could suggest a step-like function and a positive effect akin to its sinusoid condition. However, with the amount of variation in the data (high error bars in this condition), it was found nonsignificant on the Kruskal-Wallis test. 

Condition 3: Aligned fixed luminance contrast and variable color contrast

In this condition, we used the same values of luminance contrast (0.04) and color contrast as in Condition 2; however, the orientation of the color component was changed so that the chromatic and luminance components were aligned (iso-oriented and in-phase). We expected perceived depth to decrease as the contrast of the chromatic component increased. 

Figure 5 shows the results for all participants (and Table 3, Supplementary File 1 summarizes statistical analyses). As before, raw data were analyzed with a Kruskal-Wallis test for each participant and each pattern (sinusoid and square wave). When significant, data were fitted using a Weibull function, either positive or negative (Equation 4 or 5) depending on the direction of the effect. Curve fits for significant datasets are shown in Figure 5. 

Figure 5

 

Results from Condition 3. Perceived depth as a function of color contrast for all observers (black circle: sinusoids, gray squares: square waves, error bars: standard error of the mean, SEM). When raw data showed a significant effect of color contrast (see Table 3, Supplementary File 1) the data was fitted with a Weibull function, the individual fits are presented with solid lines.

Participants P1, P2, P3, and P12 showed a decreased perception of depth with increased color contrast for both sine-wave and square-wave stimuli, consistent with previous literature (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006). However, we also have some participants showing the opposite effect, with a significant increase in perceived depth for both patterns (P7, P10). For the sinusoids, 10 out of the 12 participants showed a significant effect of color contrast. Six of them were negative, as predicted by color shading; the rest were positive. For the square-wave pattern, only seven people showed significant effects of color contrast. Participant 9 showed no significant effect for either the sinusoid or square-wave condition, and Participant 5 saw no depth in the square-wave condition and no significant change of depth in the sinusoid. 

Participant P8 showed a reversal in effect direction between the two stimulus types, delivering a large negative effect for sinusoids wave and a small positive one for the square-wave pattern. In total seven participants showed similar behaviors between sinusoids and square stimulus types. Once again, we have a similar overall pattern of results as for Condition 2: A few participants showed no effect of color contrast on the perceived depth, and some of those with significant effects show a significant increase in the perceived depth in the opposite direction to that expected. 

Discussion

Our aim was to explore how different combinations of chromatic and luminance components affect perceived depth, i.e., to test the effect of color contrast on shape from shading. We compared the effects on shape from shading for both sinusoidal and square-wave patterns. Overall, we found a wide variety of observer responses, not all compatible with the heuristic suggested to explain the color-shading effect, namely that when color and luminance variations are orthogonal, luminance variation is more likely to be due to shape changes, and hence more depth may be perceived. 

In the following sections, we will discuss in detail what we have learned from our experiments. We then suggest that some of our effects, and those of previous studies, could potentially be explained via a hypothesis that is independent of depth perception, using arguments about contrast masking. Alternatively, we will discuss how the individual differences observed could be due to use of heuristics but ones that are idiosyncratic and not generic heuristics based on the statistics of the environment. Finally, we discuss the possibility that our results could have occurred due to luminance artifacts in chromatic masks and discuss how this work links to other recent literature. 

Condition 1: Varying luminance contrast for constant color contrast

We start by discussing results from a baseline condition (Condition 1) that simply tested how depth from shading increased with luminance contrast. For all participants, the zero contrast condition always corresponded to a mean perceived depth of 0 arcmin. This result provides evidence that the participants were correctly responding to the shape from shading and that the color alone did not elicit any depth. For all participants, luminance contrast had a robust effect on perceived depth: increasing the luminance contrast consistently increased the perceived depth. Observers behaved similarly to each other in this condition for both sinusoids and square waves. 

The neutral nature of our matching stimulus, in terms of shape, allowed us to compare the perceived depth directly between the two different stimulus types used here. We observed that the perceived depth saturates with luminance contrast (Figure 3), the speed of saturation varies with the pattern, square waves tending to saturate faster. However, the maximum perceived depth for both stimuli showed a high correlation (r = 0.76) and is similar for the two stimulus types for almost all observers (with the exception of Participant P4). This suggests that the perceived depth might not be linked to the geometry of the object depicted by shape from shading but instead by the luminance contrast. As sinusoids and square waves would be expected to have different perceived contrast (Ginsburg, Cannon, & Nelson, 1980) there should be an advantage toward square waves (more contrast) and we indeed find that square-wave stimuli result in faster saturation (Figure 3). 

Conditions 2 and 3: The effects of a chromatic grating on perceived depth from luminance

Previous work in this area (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006) has suggested that color enhances perceived depth when presented orthogonally to luminance and suppresses perceived depth when aligned. In our hands, the chromatic components had dramatically different effects on different participants. Some showed the effects expected from the previous literature, some no effects, or some the opposite of the expected behavior. In Condition 2 we found a surprisingly high number of participants with nonsignificant effects of color contrast (Figure 4). For Condition 3, we had expected increasing color contrast to result in less perceived depth. This only occurred consistently for three participants (Figure 5). We will discuss each condition in more detail below. 

Condition 2: Orthogonal color component, effects of increasing color contrast

There are two surprising results for this condition. First, not all participants showed significant effects of color contrast on perceived depth (see Figure 4 and Table 2, Supplementary File 1). Participants 2, 3, 8, and 11 showed no significant effects for either sinusoids or square waves. One could hypothesize that the lack of significant effect in both conditions is due to the value of luminance at which they were tested was too small for an effect of color to be present. This would mean that the chromatic component might interact with shape from shading but only over a specific range of luminance contrasts specific to each individual. We know from Condition 1 (see Figure 3 and Table 1, Supplementary File 1) that the depth perceived is directly dependent on the luminance contrast. We also know that the perceived depth saturates more quickly for the square wave. One could therefore further hypothesize that the perceived contrast is actually different between the sinusoid and square. As discussed above, the faster saturation for square-wave stimuli in Condition 1 suggests that this might be true. In this case, the perceived contrast of the chromatic component might be different as well. Thus, individual differences in perceived contrast could account for the large individual differences in perceived depths found. All in all, these results suggest that if a color-shading enhancing effect exists, it only does so for a limited contrast range. 

The second surprising result is the reduction of perceived depth with chromatic contrast, which was found for some participants. In previous work (Kingdom, 2003; Kingdom et al., 2005; Kingdom et al., 2006), suppression of perceived depth has only been observed when the chromatic and luminance components are aligned and in phase. These data therefore speak against a hypothesis based on the heurists suggested previously (Kingdom, 2003) to explain the color-shading effect. 

Could stimulus specific differences explain why our results are not consistent with previous literature? The color-shading effect might be dependent on the luminance contrast at which it is tested. Kingdom et al. (2005) showed a tuning of the color-shading effect depending on the luminance contrast tested, suggesting that there might be smaller enhancement at high luminance contrast. One hypothesis for why our results were mixed is that the luminance values our observers were tested at might have been too high. If so, perceived depth had already saturated at its maximum level, leaving no space for further enhancement. But our choice of contrasts was low enough for this not to provide a convincing explanation. We used luminance contrasts (for Conditions 2 and 3) of 4%. Kingdom et al. (2005) found enhancement for luminance contrasts of 5% and 15%, but less so for 45%. To address this further, we considered all the individual data and looked for correspondence between the luminance level tested in the chromatic conditions (i.e., Figures 4 and 5) and the level of saturation of contrast in Condition 1. If the above hypothesis were correct, then we might expect to see nonsignificant effects or suppressive effects at saturation levels of luminance and increasing facilitatory effects for those observers who were not yet at their saturation level. However, this pattern was not found consistently; some participants showed increases or decreases at saturated value and others nonsignificant effects even though we were testing at their midrange. Thus, the saturation hypothesis does not explain why some participants showed nonsignificant effects of color contrast. 

Condition 3: Luminance and color contrast components aligned

When luminance and color signals were aligned, we expected decreased perceived depth, consistent with a suppression of luminance defined depth at borders that specify both luminance and chromatic changes. This was not consistently found. Compared to Condition 2, more participants showed significant effects of color contrast when the two components were aligned (compare Figures 4 and 5). There was greater depth modulation as a function of color contrast for the sinusoidal pattern than for the square-wave pattern (see Table 3 in Supplementary File 1 and Figure 5). We note also that the direction of the effects, when present, tended to be the same for sinusoids and square waves. 

However, a few participants showed nonsignificant effects of color contrast increase on perceived depth. 

Some participants with significant effects showed the opposite of the expected effect direction, i.e., they showed an increase in perceived depth. One example is Participant 7 (see Figure 5). Thus, once again, we do not find strong evidence for a systematic effect of color that would be consistent with the heuristic suggested to explain the color-shading effect. Furthermore, in the previous section we argued that some decreases in perceived depth might be because the luminance contrast was too high (i.e., the maximum depth was already reached), but this argument does not match the observed data in Condition 3. Participant P4 exemplifies this assessment. Consider their behavior for Condition 1 (Figure 1) and compare with Conditions 2 and 3 (Figures 4 and 5): The values of luminance contrast tested in Conditions 2 and 3 is already in the saturated part of the curve for the sinusoid, and we can see a suppression of perceived depth when color is orthogonal (Figure 4) and an increase when aligned (Figure 5), both significant (see Tables 2 and 3, Supplementary File 1). This is the opposite from what one would expect if perceived contrast was the only issue. 

Difference between our stimuli and previous studies

There were other stimulus details that differed between our study and previous ones. A key difference between our experiment and those performed in other labs was that in our case the depth-suppressing effect was not tested using two chromatic components, one iso- and one cross-oriented, but only one iso-oriented component. Kingdom et al. (2005) showed that a luminance component would have its depth enhanced by an orthogonal chromatic component (either L-M or S modulated) and suppressed by a third color component (either L-M or S modulated) aligned with the luminance component. In our Condition 3, we used only one (aligned) color component. 

We did not test the color shading with three components (two chromatics and one luminance); instead, we used two components (one chromatic, one luminance). We therefore cannot directly conclude that the individual differences we found would apply to the three-component situation. However, we can think of no reason, given the logic of the color-shading heuristics suggested, why different results would be expected. We discuss the possibility of specific observer- and stimulus-dependent heuristics in a following section. 

According to the color-shading effect hypothesis, the alignment of chromatic and luminance should be interpreted as changes in reflectance, and hence perceived depth should be reduced. We think that the kinds of interactions found in previous studies could also be attributed to interactions between the two chromatic components. We explore this idea in more details in the section below on masking. 

Masking: Within channel and cross channel

In this section we draw a parallel between our results and some of the literature on masking, and we consider whether effects attributed to the color-shading effect could instead be related to low-level masking. There are two main points we wish to cover: cross-orientation masking (usually studied within channel) and cross-channel masking (e.g., between red-green and luminance channel), both of which can produce suppressive and facilitatory effects. 

One way to interpret our results is to think in terms of masking. In other words, to what extent chromatic patterns might mask luminance patterns, at stages well before the luminance information is interpreted as depth. Most low-level interactions between sinusoidal luminance, or color, patterns are described in term of masking within channels (e.g., Legge & Foley, 1980; Losada & Mullen, 1994) and between channels (e.g., Chen, Foley, & Brainard, 2000). Below we summarize the literature and relate it to our experiments here. 

Facilitatory and suppressive effects within channels

Facilitation can occur when the mask is at detection level. The facilitation (manifested as a lowered threshold) is then followed by an increase of the threshold at higher contrast. The resulting pattern threshold elevation, as a function of mask contrast, is commonly referred to as the “dipper function” because of this initial facilitation, then suppression effect. This pattern of masking occurs within chromatic (e.g., Losada & Mullen, 1994) as well as within luminance channels (e.g., Bird, Henning, & Wichmann, 2002). 

The strength of the mask is dependent on the characteristics of both mask and target. As a rule of thumb, the more similar mask and target are, the greater the masking effect. This is explained by hypothesizing that test and mask are processed by the same channel and that the mask is adding noise, rendering detection more difficult. For example, masking of a sinusoidal luminance pattern is greatest when the mask and test are about the same orientation (Phillips & Wilson, 1984), but cross-orientation masking can also occur: Targets are still masked when masks are 90° out of phase (Meese & Holmes, 2007; Phillips & Wilson, 1984). 

In color vision, masking follows a different pattern: Masking appears to be isotropic, i.e., not dependent on mask orientation (Medina & Mullen, 2009). An orthogonally-oriented (also referred to as cross-oriented) stimulus usually shows suppressive effects, but masking can be facilitatory (Meese & Holmes, 2007) at low temporal frequency. Medina and Mullen (2009) compared cross orientation masking (XOM) within luminance and chromatic channels. Chromatic XOM was found to be independent of temporal frequency and generally stronger than achromatic XOM, i.e., stronger facilitation at low contrast and stronger suppressive masking at higher contrast. We used only two components per stimulus, one achromatic and one chromatic; Thus, there will be no within-channel masking. XOM could, however, be involved in causing the suppressive effects found for the three-component stimuli used in Kingdom's studies. 

Cross-channel masking

Studies on how color masks luminance or vice versa, cross-channel masking, are few and far between (but see Chen et al., 2000; Gheiratmand & Mullen, 2010; Mullen, Gheiratmand, Medina, & Kim, 2012) and typically conducted at, or near, detection thresholds. Hence we cannot predict from them whether specific aspects of the color-shading effect could be accounted for by low-level masking. There is also some data on cross-channel (red-green chromatic vs. luminance) iso-oriented interaction. Chen, Foley, and Brainard (2000) found that chromatic masks interact, but do not facilitate, luminance target detection when the chromatic component is properly isoluminant (see section on luminance artifacts below), however when it is not they found slight facilitatory effects at low contrast. Iso-oriented masking could potentially be involved in the suppressive effect that we find for some participants (Figure 5) and others have found (Kingdom, 2003; Kingdom et al., 2005). 

However, Cole, Stromeyer, and Kronauer (1990) found relatively no interaction between luminance and chromatic signals using a non-oriented patch but did manage to get facilitation when the pedestal used was slightly suprathreshold; this was dependent on the presence of a surround. They only found facilitation of chromatic detection by luminance pedestal when the two were presented in the same eye (monoptically). Furthermore Medina and Mullen (2007) demonstrated luminance and red-green interactions across eyes: Detection was left untouched but binocular summation was altered by luminance noise. This demonstrates the complexity of cross-channel interactions around threshold. In the next subsection we explore interaction at suprathreshold level in more detail. 

Suprathreshold masking effects

In the suprathreshold domain, Pearson and Kingdom (2002) found that superimposed chromatic (red-green) and luminance Gabors showed suprathreshold facilitation, as opposed to the sub-threshold facilitation showed in detection tasks. This study also showed marked individual differences in stimulus integration (for example, one participant did not show any facilitation but only the suppressive effect with a luminance target and chromatic mask). From the results of our Condition 3, we can hypothesize that some participants with increased perceived depth might misappropriate chromatic contrast for luminance contrast. This suprathreshold combination of contrast might be akin to results described in Pearson and Kingdom (2002). It is possible that the aligned configuration of Condition 3 is perceived as ambiguous; one hypothesis could be that the respective contrasts (luminance and color) get pooled together or color contrast somehow modulates luminance contrast. If this pooling mechanism exists, it could well occur before the shape is computed. 

Gowdy, Stromeyer, and Kronauer (1999) tested XOM with sine- and square-wave patterns. They found that at 0.8 cpd there was larger facilitation for sinusoids than square waves. They concluded that the sharp edges played an important role in the facilitation. They also found that the facilitation was phase dependent. The facilitation was similar at 0° or 180° phase (green on light bar or green bar on dark bar); however at 90° the facilitation was abolished. This is interesting when attempting to interpret the color-shading effect because we know from (Kingdom, 2003) that the color-shading effect is also phase dependent. However, De Valois and Switkes (1983) found suppressive (masking) effects of chromatic masks on luminance targets. Two of their participants showed no phase effects but one did show a large suppressive effect for 0° and 180° (in phase) and a small effect for 90°. In their paper, they postulated that luminance and chromatic masks could have distinct excitatory or inhibitory effects on the detection of luminance targets. 

To sum up, we do not know how the masking data obtained from other studies relates to the perceived suprathreshold contrast that was used in our study and in others on the color-shading effect. However, we note that phase dependence and orientation seems to be similar in masking and suprathreshold data; both can show large individual differences too. It would therefore be interesting, but beyond the scope of this study, to measure how the different chromatic components affects the perceived contrast of the luminance modulation. 

Use of heuristics

Above we have suggested that individual differences could occur due to differences in low-level masking; it is also possible that individuals could be using different high-level heuristics to perform the depth task. In the original paper on the color-shading effect, Kingdom (2003) hypothesized that the interactions observed between color and luminance, in the context of shape from shading, were due to the use of heuristics based on the statistics of the environment (Kingdom, 2008). We would have expected these heuristics to be general, applying to potentially more complex, or indeed, less complex scenes (like the two-component patterns we used here) and used by most participants. However, our data suggest that observers could be using idiosyncratic, high-level heuristics that are not a common feature of everyone's behavior. For example, we know that individuals combine cues differently for shape perception, favoring shading to lesser or greater extents (e.g., Lovell et al., 2012). 

This implies that visual systems might be more flexible than expected and could evolve to match real-world statistics or become biased. Thus, we challenge the generality of the heuristics proposed by Kingdom (2003). 

A role for luminance artifacts?

In any color vision experiment, there is the possibility that chromatic information has not been accurately isolated and that small luminance artifacts could underlie some of the observed behavior. In this section we discuss several reasons why luminance artifacts are very unlikely account for our results. 

First, chromatic aberration can occur due to the differential refraction of the eye; however, it is only a problem at medium and high spatial frequencies (Mullen, 1985). For the spatial frequencies used here, (0.75 cpd) this should not be an issue. 

The second possibility is that despite having tested individual isoluminant points for each participant and contrast value tested, there still exists some luminance artifact. The results of Condition 1, which showed how perceived depth increased with contrast for every observer (Figure 3), provide a compelling case that the depth perceived in our stimulus was mainly driven by luminance contrast. Any manipulation that decreases the perceived contrast of the right oblique luminance component should then also affect the perceived depth. Georgeson and Shackleton (1994) found, for a detection task, that plaids have less perceived contrast than individual gratings and that it was minimal for orthogonal gratings. If there were a luminance artifact at work in our Condition 2 (luminance and chromatic gratings orthogonal to each other), then it should decrease the perceived contrast of the shape-from-shading object. This might be a factor for those of our participants who showed a decrease in perceived depth with color contrast (P4, P6, P9). But for the remaining nine participants, where we found no change or an increase in perceived depth, this is not borne out. Furthermore, in Condition 3, where chromatic and luminance components are aligned, any luminance artifact in the chromatic component should directly add to the luminance one and result in enhanced depth. Such a pattern of data was present for only a few participants, not all the same as those consistent with the prediction for Condition 2 (P4, P6, P7, P10). Overall the pattern of data expected if a luminance artifact was at work was not displayed by the majority of our participants. 

In order to explore the possibility of luminance artifact further we retested two participants, selected to have one with increased and one with decreased perceived depth on Condition 3 (P7 and P11). All depth-matching experiments were repeated (both sinusoid and square wave) but with the phase of the chromatic stimulus altered by 180°. This should have no effect on the orthogonal conditions (Conditions 1 and 2) but would affect Condition 3 if a luminance artifact was present, resulting in a reduced or reversed effect on depth compared with original Condition 3. No such effects were found (Supplementary File 2), hence we were satisfied that luminance artifacts were not responsible for the results of the main experiment. 

Links with second-order variations

The literature on second-order luminance effects is also of interest here. A first-order variation is luminance change across space, what we usually describe as contrast. A second-order variation is a change of contrast across space. The second-order luminance modulation is relevant to the effect of chromatic information on luminance-defined shading for two main reasons. First, because the channels responsible for detection of LM (local mean luminance, first order) and AM (local luminance amplitude modulation), sometimes also referred as contrast modulation, CM (Schofield et al., 2006), interact in a similar way to luminance and (red-green) chromatic channels (see Chen et al., 2000; Schofield & Georgeson, 1999). Second, it has been shown that both these channels have modulatory effects on shape-from-shading processing (for second order luminance: Schofield et al., 2006; and for chromatic components see Kingdom, 2003). Note that this connection has also been made by Kingdom (2008) and by Schofield et al. (2006). 

Dovencioglu, Welchman, and Schofield (2013) showed AM-LM interactions could be learned over time by naïve observers using positive reinforcement. From this work, two conclusions seem important. First, naïve participants might behave very differently until trained and this might extend to luminance and color interactions. Second, it indirectly raises a cautionary note on the use of trained observers or non-naïve (usually authors) trained in psychophysics. Our relatively modest sample size of 12 naïve participants has shown tremendous interindividual differences in the effects of color contrast. However, the effects of luminance contrast on perceived depth were similar across all participants and robust between sinusoids and square waves. It has been suggested, in the case of a different mechanism, motion in depth, that the behavior of the general population can be quite different that of trained “expert observers” (Nefs, O'Hare, & Harris, 2010), and this is very much in line with what we found here. 

Conclusions

We have shown that sinusoidal and square-waves luminance modulations have similar effects on perceived depth. We interpret the faster increase of perceived depth with contrast of the square wave as due to higher perceived contrast between sinusoids and square waves. However the maximum perceived depth was similar for both patterns for 11 of our 12 participants. 

For the chromatic contrast manipulation, both iso- and cross-oriented chromatic contrast increases were perceived with highly individual differences by our test group. Therefore we do not think that a common heuristic is used by the whole population to disambiguate luminance variation that could be delivered by a shape or a reflectance change. Dovencioglu et al. (2013) suggest that a similar relationship can be learned for AM-LM modulation with feedback. Consequently we argue that people usually described in the literature as expert observers might have learnt how to use a common heuristic to disambiguate the luminance information. It is possible that different observers use their own separate heuristics which might not match an optimal behavior, one observer could learn the opposite (e.g., Dovencioglu et al., 2013). The time scale of these interactions is quite telling and could possibly distinguish between feedback loop and bottom-up effect. 

The interactions that we observed for the iso-oriented (aligned) conditions could be considered as akin to masking and an alternate explanation of the color-shading effect could potentially be accounted for with low-level masking, but could also be due to idiosyncratic heuristics, not linked to the statistics of the environment. We think that the next step is to explore the possibility of low-level interindividual differences in the processing of luminance and chromatic contrast. Furthermore a complete account of these interactions should test for suprathreshold inter-channel interactions. These approaches are currently being pursued in our lab to account for the large differences found in the effects of color on shape from shading. 

Supplementary Materials

pdf S1 

pdf S2 

pdf S3 

Acknowledgments

Thanks to Anastasia Kolokolnikova for collecting some of the data and to the members of the vision lab at St. Andrews for inputs on the early version of the manuscript. This research was supported by the Engineering and Physical Sciences Research Council (EPSRC, UK) via grants EP/G038708/1 to JMH and EP/G038597/1 to MB and a Leverhulme Research Fellowship to JMH. 

Commercial relationships: none. 

Corresponding author: Stéphane Clery. 

Email: [e-mail]. 

Address: Vision Lab, School of Psychology & Neuroscience, University of St. Andrews, St. Andrews, UK. 

Footnotes

¹   Note that second order luminance information (changes in local luminance amplitude) can also be used to help disambiguate luminance due to shading from luminance due to pattern (Schofield et al., 2006; Schofield, Rock, Sun, Jiang, & Georgeson, 2010; Sun & Schofield, 2012).

Footnotes

²   The choice of a Weibull function was arbitrary; any sigmoidal-shaped function could have been used to represent the form of the data.

Appendix A: Definitions of Chromatic and Michelson contrast

We show here that Parameter A in the definition of sinusoids (Equation 2) corresponds to Michelson cone contrast. The Michelson contrast definition is based on periodic stimuli in which we take the highest and lowest values of luminance, L_Max and L_Min, respectively to obtain contrast Mc:  

 

 

The Weber definition of contrast Wc is based on variation from background luminance, L_b, of a particular test luminance, L_test:  

 

 

Contrast is usually defined in luminance terms but, for our purpose we use cone excitation values instead, as is common in the literature (e.g., Weber cone contrast, Cole & Hine, 1992; Giulianini & Eskew, 1998; Michelson cone contrast, Gunther & Dobkins, 2002):  

 

where Cone_test is the cone excitation, at the test, of cone type Cone (Cone = L, M, or S), Cone_b is the cone excitation of type C, at the background level, and Δ_cone Δ_c is the difference between background and test. Our cone contrast modulation is defined in terms of Weber cone contrast; that is, modulation from background (Equation 2, here reproduced in Equation A4).  

 

 

 

 

 

 

We demonstrate here that the Parameter A (Equation 2) is equivalent to Michelson contrast for luminance stimuli, i.e., cases in which the three cone classes are modulated in phase by the same amount of cone contrast. The demonstration applies to all three cone types; for clarity we refer to cone type “C” as a generic term here. 

In Equation 2, functions produce periodic waveforms oscillating from −A to +A. This gives us a total amplitude peak-to-trough of 2A. Following from Equation A1, we define the highest contrast peak as C_max = +A and lowest as C_min = −A, this is also the case for square wave by definition (see Equation 3). We can turn these values into cone excitation values, C_test using Equation A3, knowing the intensity of the background (C_b) and the contrast of the test (C_C). Equation A3 can be reformulated as:  

 

 

There are two values we are interested in testing, the highest C_maxL_max and lowest C_minL_min in order to use the Micheslon's contrast equation. From Equation A1, we can get the highest and lowest cone contrast values and we know that C_max = +A and C_min = −A and transform them into cone intensity values. Consequently:  

 

 

 

 

Now that we have both L_max and L_min we can calculate the Michelson contrast, substituting from Equation A6 into Equation A1, we obtain:  

 

 

Appendix B: Equiluminance settings

Chromatic stimuli must be tailored to each individual participant. In order to isolate the chromatic channel and create isoluminant components for each participant, we adjusted amplitude A in Equation 2 for L and M cones (A_L, A_M, respectively) for each value of contrast that we wanted to display. Essentially we added an amount b of L contrast and subtracted the same amount b of M contrast. However because M-cone contrast took a negative value (i.e., to create a red-green modulation) the equations take the following form:  

 

 

 

 

 

 

The values obtained for A_L and A_M are the ones used for the main experiment (Equation 2 main text, or in Equation A4). 

The techniques used to find isoluminance usually rely on the fact that the chromatic channel carries poor motion or flicker information, compared with the luminance channel (Livingstone & Hubel, 1988). Moving or flickering stimuli are adjusted until minimum motion or flicker are obtained. The standard practice is to use a stimulus as close as possible to the one used in the specific experiment for which the isoluminant stimulus is required (i.e., the stimulus should be the same size, orientation, spatial frequency, etc.), as the values at which isoluminance is achieved can vary with stimuli parameter (Anstis & Cavanagh, 1983). 

To obtain a measure of b, participants performed a minimum motion experiment (similar to that described by Anstis & Cavanagh, 1983). The minimum motion stimuli consisted of alternating color and luminance gratings with phase offsets, presented at a rate that resulted in motion perception in the luminance channel. Phase offsets were designed so that if color gratings possessed a luminance artifact, the alternation of the artifact with luminance components would deliver the perception of coherent motion. For example, if the red stripes on the sinusoids were darker than the green ones, then the motion detection system would match them to the dark bars on the luminance frames and, due to the phase offsets between frames, this would create the perception of motion in a specific direction. 

Our aim was to find the equiluminant setting, i.e., the value of b corresponding to the red and green stripes being perceived to have equal luminance. This case corresponds to the minimum motion percept. We used the method of adjustment. By turning a response knob (Cambridge Research Systems CB7), participants added or subtracted equal amounts of contrast to both L and M (effectively updating b in Equation B1) on the chromatic frames. Participants were asked to turn the response knob in the direction opposite to the perceived motion until they perceived minimum motion. The minimum motion was perceived just before the motion changed direction at the point of perceptually equal luminance between red and green stripes. 

Participants were encouraged to take as much time as necessary to make fine judgments. Participants completed twelve settings at each value of the contrast modulation (A). We used the full range of contrasts that were to be tested in the color-shading experiment, and we used both sinusoidal and square-wave stimuli. The results are presented for both sinusoids and square-wave in Figure B1. Note that the results for both type of patterns (sinusoid and square wave) are very similar within all participants. There is however individual differences in matching between participants as expected (e.g., Gunther & Dobkins, 2002). 

Figure B1

 

Results of minimum motion experiment for both sinusoids (black circles) and square-wave (gray squares) patterns. The results are presented in the form of parameters a and b (See Equations A7 and B1).

Appendix C: Additional comments regarding contrast metrics

Chromatic contrast, a, is not expressed as the RMS contrast; however, RMS can be obtained if necessary using the data provided in the paper. First we need to obtain individual cone contrast for L and M (S is zero for the color stimuli), which is obtained by using the adjustment values from the minimum motion experiment. Contrast of L (A_L) is a + b and M A_M is a − b (or |−a + b|, as A is a sinusoidal or square-wave modulation from +A to −A), where b is the additional amount needed to obtain equiluminance for each individual (Appendix B). Essentially, with b we add as much contrast from one cone type as we subtract from the other. From values a and b provided in the text it is possible to obtain any contrast metric (see Brainard, 1996, for more details on contrast metrics) we want e.g., RMS would be $\sqrt{\left[{\left(a+b\right)}^2+{\left(a-b\right)}^2\right]/2}$, simplified to $\sqrt{a^2+b^2}$. 

From the supplemental data on equiluminance we can see that a and b tend to have a monotonic relationship. Further, estimates of b for square-wave and sine-wave stimuli are similar. Replacing our contrast metric by another would essentially rescale the x axis on all plots (if the relationship between a and b is strictly linear as we would expect, and as we measured). For convenience, we presented all of our data using a. The RMS values used are detailed in Supplementary File 3.