Perception of object illumination depends on highlights and shadows, not shading

Abstract

Human observers are able to successfully infer direction and intensity of light from photographed scenes despite complex interactions between light, shape, and material. We investigate how well they are able to distinguish other low-level aspects of illumination, such as the diffuseness and the number of light sources. We use photographs of a teapot, an orange, and a tennis ball from the ALOI database (Geusebroek, Burghouts, & Smeulders, 2005) to create different illumination conditions, varying either in diffuseness of a single light source or in separation angle between two distinct light sources. Our observers were presented with all three objects; they indicated which object was illuminated differently from the other two. We record discrimination performance, reaction times, and eye fixations. We compare the data to a model that uses differences in image structure in same-object comparisons, and outcomes suggest that participants mostly rely on the information contained in cast shadows and highlights. The pattern of eye fixations confirms this, showing that after the first fixation, observers mostly fixate cast shadow areas. However, information in the highlights is rather salient, so it might be available from first fixation, making separate fixations are unnecessary.

Introduction

Appearance is the result of a physical projection of an object or scene on the retina, combined with our brain's perceptual and cognitive interpretation of this projection. We can describe the physical appearance of an object as a combination of the light field, its material properties, and its shape. In return, object appearance also provides us with clues about the illumination, shape, and material properties. However, deriving the illumination, shape, and material properties from images or even from real scenes is by no means a trivial task. Mathematically, there usually is no unique solution to the problem; many combinations of objects and light fields would result in the same images and retinal input. Calculating shape, illumination, and material from images, be it on the retina or in photographs, is therefore an underdetermined problem (Adelson & Pentland, 1996; Belhumeur, Kriegman, & Yuille, 1999; Blake & Bülthoff, 1990; Dror, Willsky, & Adelson, 2004). 

In many cases, it is the shape or material of objects and scenes that we humans are interested in and not the illumination per se. Generally, observers seem to judge material and shape perceptually without giving the ambiguities that are always present any thought. To determine the perceived shape or material, however, observers need to disentangle the effect of illumination on the appearance of objects from the effects caused by shape and material properties. To be able to do that means that assumptions about the light field have to be made. 

A lot of work on the separation of perceived illumination and material has been conducted in the field of color constancy and the perception of lightness and brightness. Typically, a few rendered two-dimensional surfaces are used in so-called Mondrian stimuli that show that illumination and material are actually metameric. Although many results have been achieved in this manner, questions have been raised as to whether these results can actually be extrapolated to more complex or even natural scenes (Gilchrist, 1994; Hurlbert, 1999; Kraft & Brainard, 1999; Radonjić et al., 2016; te Pas & Koenderink, 2004). In the Mondrian case, the relevant information about the illuminant is mainly in the spectral distribution of the light, not the spatial distribution. However, there has been a growing interest in investigating interactions that emerge in complex, typically three-dimensional, scenes (e.g., Boyaci, Fang, Murray, & Kersten, 2007; Braje, Legge, & Kersten, 2000; Kartashova, de Ridder, te Pas, Schoemaker, & Pont, 2015; Kartashova, Sekulovski, de Ridder, te Pas, & Pont, 2016; Ling & Hurlbert, 2004; Marlow, Kim, & Anderson, 2012; Nishida & Shinya, 1998; Obein, Knoblauch, & Vienot, 2004; Pont & Koenderink, 2007; Robilotto & Zaidi, 2004; Xia, Pont, & Heyndericx, 2014). The context provided in the scene can help in the disambiguation between light and material, but even in complex, photorealistic images, material changes are often confounded with illumination changes (Anderson, 2011; Boyaci, Maloney, & Hersh, 2003; Doerschner, Boyaci, & Maloney, 2007; Fleming, 2014; Fleming, Dror, & Adelson, 2003; Maloney, Gerhard, Boyaci, & Doerschner, 2010; Pont, Koenderink, Van Doorn, Wijntjes, & te Pas, 2012; Pont & Te Pas, 2006; Ripamonti et al., 2004; te Pas & Pont, 2005; Toscani, Zdravković, & Gegenfurtner, 2016; Zhang, de Ridder, & Pont, 2015). However, when asked to place photographed real materials in simple natural categories, participants are surprisingly fast, even when stimuli are degraded and materials are taken out of context (Sharan, Rosenholtz & Adelson, 2009, 2014; Wiebel, Valsecchi, & Gegenfurtner, 2013). 

When judging the illumination in a scene, observers rely on several cues. Most of the cues that are used have to do with the way objects reflect the light and thus with object properties and scene layout. They enable the human observer to determine certain lower-order properties of the light field. Texture shading and cast shadows can, for instance, be used to determine the direction and intensity of a light source (Casati, 2004; Koenderink, van Doorn, Kappers, te Pas, & Pont, 2003; Koenderink, van Doorn, & Pont, 2004; Xia et al., 2014). Morgenstern, Geisler, and Murray (2014) determined that observers are attuned to diffuseness levels of natural scenes. Observers even can infer low-level properties of the light in empty space (Koenderink, Pont, van Doorn, Kappers, & Todd, 2007; Schirillo, 2013) from complex photographed real scenes and also from actual scenes (not photographs) by using a light probe (Kartashova et al., 2016; Koenderink et al., 2007; Xia et al., 2014). 

However, human observers are not very good at spotting local inconsistencies in the lighting or at using cues such as shading and shadowing when there are very few shape cues (O'Shea, Agrawala, & Banks, 2010; Ostrovsky, Cavanagh, & Sinha, 2005). Moreover, they can distinguish only a limited number of cases of global light field structures (Kartashova et al., 2015; Kartashova et al., 2016; van Doorn, Koenderink, Todd, & Wagemans, 2012). Appearance of gloss, for instance, is best mitigated by simply shaped highlights, which are usually driven by simple light fields (van Assen, Wijntjes, & Pont, 2016). Perceptually, observers can typically distinguish differences in contrast gloss and in distinction of image gloss; both types depend on the interaction of light with the object and on the surrounding scene (Ferwerda, Pellacini, & Greenberg, 2001). Although gloss is rather complex, observers are typically very fast at recognizing glossy (and other) materials (Sharan et al., 2009, 2014). When cues are not conclusive, observers make assumptions about the light field that are probably based on prior experience, such as that light comes from above (Fleming, 2012; Mamassian, 2004; Mamassian & Goutcher, 2001; Morgenstern, Murray, & Harris, 2011). 

Apparently, when enough cues such as cast shadows, shading, and highlights are available, and the context is rich enough, human observers are rather good at determining basic aspects such as the general direction and intensity of the light. More complex aspects of illumination are often confused with changes in material properties or shape, even in rich contexts. The question we ask here is how well human observers are able to distinguish basic aspects of the illumination that can generate similar but not identical light fields in space, such as the variations of the diffuseness of a single light source and the separation distance between multiple light sources, when we offer a sparse but well-controlled context. Moreover, we would like to know what kind of information observers use to assess these qualities of the light field by also recording eye fixations and reaction times during their assessment of the scene. Boyaci, Doerschner, and Maloney (2006) have shown that highlights, shadows, and shading information are all used in determining lighting conditions in complex scenes. To determine if that is also true in our rather sparse contexts, we ask observers to determine which one of three objects is illuminated differently from the other two. We measure percentage correct, reaction times, and eye fixations on the objects. We compare results of human observers with models that use different types of information about the illumination, specifically information contained in the highlights, shading, or shadow of the object at hand. 

Methods

Stimuli

We used pictures of a teapot, an orange and, a tennis ball from the Amsterdam Library of Object Images database (ALOI; Geusebroek, Burghouts, & Smeulders, 2005) to create six illumination conditions for each object. From the ALOI database we, selected pictures of each object shot with a fronto-parallel camera position, illuminated with a single light source in five different positions: straight above, 15° and 30° to the right and 15° and 30° to the left (for an example, see Figure 1). 

Figure 1

 

Photographs of the teapot (No. 161 from the ALOI database; Geusebroek et al., 2005; illumination conditions l1c1–l5c1 rendered in black and white). These five photographs were used to render the different illumination combinations used in the experiment.

Unfortunately, the illumination conditions in this database do not vary in diffuseness or in separation of the sources. However, because they are so well controlled, we can create new pictures with combined illuminations by a simple superposition of the separate images. Because light follows the superposition principle, the result is physically correct. This can be done with different weights to create a whole range of different illumination conditions. 

We combined these five pictures by superposing them with different weights to create new illumination conditions that resemble either a single source with different diffuseness or two distinct light sources with different separation. To determine the weights, we implemented a Gaussian weighting function with its top (largest weight) in the center and varied the diffuseness (spread) by varying the width of the Gaussian weight function so that at standard deviation, the width is 10°, 20°, or 30° to create the impression of light sources with different diffuseness. By defining as a weight function two Gaussians with constant width of 10°, but with their peaks at a distance from the center, we were able create the appearance of two distinct light source directions with a varying separation angle of 10°, 20°, and 30° (see Figure 2 for a graphic representation of this weighting process). 

Figure 2

 

Graphic illustration of the weighting function for the addition of images. Top row: Relative image weights for diffuseness (for the construction of an image with a diffuse light source with a 10°, 20°, and 30° of angle width, respectively). Bottom row: Relative weights for separation (for the construction of an image with two light sources with a separation of 10°, 20°, and 30° of angle, respectively).

The illumination conditions we created in this way are illumination from a single direction with three values for diffuseness (10°, 20°, and 30° of width of the illuminant) and illumination from two distinct directions with three amounts of separation between the two sources (10°, 20°, and 30° of angle; for the resulting stimuli, see Figure 3). 

Figure 3

 

Photographs of the actual oranges stimuli we used. Top row: One light source with varying diffuseness (from left to right: 10°, 20°, and 30° spread). Bottom row: Two light sources with varying separation (from left to right: 10°, 20°, and 30° apart).

Participants

A total of 16 observers participated in the actual experiment (five men, 11 women). All participants were students of Utrecht University who were paid for their participation. All observers were naïve with respect to the purpose and nature of the experiment. All observers gave written informed consent. Experiments were done in agreement with local ethical guidelines, Dutch Law, and the Declaration of Helsinki. 

Task

In the instruction phase, observers were shown pictures like the ones in the real experiment but of a different object and with large differences in illumination to illustrate the different types of illumination they could expect. They were given only examples; no instructions were given as to how they should look at or compare the images. In the experimental phase, observers were presented with a tennis ball, a teapot, and an orange side by side on a black background on every trial. They had to indicate which of the three objects was illuminated differently from the other two (an odd-one-out task). Participants reported that they found this a relatively easy task to do. There are 30 different illumination combinations (6 × 5; of the odd-one-out illumination and the illumination of the other two objects). For each of these combinations, we have three objects that can be the odd-one-out times six possible layouts of the three objects, yielding 18 different trials per condition. This means that in total, there are 540 trials per participant. The eye fixations of the participants were recorded during the entire experiment. Although we did not instruct participants to react as fast as possible, we did record the reaction time for each condition as well. 

Setup

In the experimental phase, observers were seated with their head in a chinrest at 57 cm from a 22-in. LaCie Blue Electron CRT monitor. Experiments were performed using a PC running the Psychophysics Toolbox for Matlab (Brainard, 1997; Pelli, 1997). Eye fixations were recorded at 52 Hz using a portable, EasyGaze™ eye tracker (Design Interactive, Inc, Oviedo, FL). 

Analysis of data

For each illumination condition, we obtain an average percentage correct and an average reaction time per participant. The eye fixations of the participants were analyzed offline. Fixation detection was done using a custom Matlab program that marked fixations by an adaptive velocity threshold method (Hooge & Camps, 2013). 

From the analyzed fixation locations, we create average heat plots using the output of the probability density function of the fixation locations, based on a smooth kernel density estimate using Silverman's rule to determine bandwidth (this is a standard Mathematica function) per participant for the overall fixation location and for the location of the first five fixations separately. We also compute the percentage of fixations in different areas of interest (AOI) within the stimulus, such as a highlight AOI, a shading AOI, and a cast shadow AOI. For the eye fixations, we kept the AIOs simple and coarse, so that they all span an equally sized area. We used the upper third of the scene to define the location of highlights, the middle third for the location of shading, and the lower third for the location of cast shadows. In this way, the AOIs are the same size; however, there is some overlap in the type of information contained in each AOI. From the heat maps, we can see that the fixations are mainly concentrated in the areas we defined. For both modeling and behavioral analysis, we use somewhat more sophisticated AOIs that were defined by masks that were created by cutting out the relevant parts of the scene in terms of highlights, cast shadows, or shading by hand. 

Results

Psychophysics

We collected both percentages correct and reaction times for all observers in all conditions. To be able to compare results, we plot these in matrix form. On the horizontal axis is the illumination condition of the two objects that have the same illumination (references) and on the vertical axis the illumination condition of the odd-one-out (test). We visualize percentages correct by means of a gray scale, with black meaning 100% correct and white meaning 0% correct (with 33% chance level). The conditions on the diagonal where the conditions of both reference and test were the same, expressed here in white, were not included in the experiment. This way of visualizing enables us to look at patterns of results and compare them. Figure 4 shows the average percentages correct over all observers as well as individual results. Although observers reported that this was a task they could perform, they still made a lot of errors. There is a distinct pattern of errors, in that some illumination conditions are clearly more difficult to distinguish than others. Specifically, although overall performance is low, the percentages correct increase when the difference in diffuseness is larger (lower-left quadrant). However, there is no clear increase in performance when the separation difference is larger (upper-right quadrant). Two distinct light sources and one diffuse light source are frequently mixed up. 

Figure 4

 

Percentages correct for all 30 combinations of illumination conditions. On the horizontal axis is the illumination condition of the two reference objects and on the vertical axis the illumination condition of the odd-one-out (left to right and bottom to top: first three increasing diffuseness of 10°, 20°, and 30° spread, respectively; second three increasing separation of 10°, 20°, and 30°, respectively). (A) Averaged over all 16 participants. (B) Individual data. Black means 100% correct; white means 0% correct.

Although we did not ask them to respond as fast as possible, we also collected average reaction times per condition from our observers. To be able to compare the pattern of results from the average reaction times to the pattern of results of the percentages correct, we visualize the reaction times in the same way, with black meaning the shortest average reaction time and white meaning the longest average reaction time. Again, the same-illumination conditions on the diagonal were not included in the experiment. Figure 5 shows the average reaction times over all observers as well as individual results. Again, there is a distinct pattern, in that some illumination conditions clearly take more time to distinguish than others. Specifically, just like with the percentages correct, the reaction times decrease with larger diffuseness difference (lower-left quadrant). However, there is no clear decrease with decreasing separation (upper-right quadrant). 

Figure 5

 

Reaction times for all 30 combinations of illumination conditions. (A) Averaged over all 16 participants. (B) Individual data. Illumination conditions are ordered in the same way as in Figure 4. Black means shortest reaction time; white means longest reaction time.

The pattern of results is slightly different from the pattern of results we found for the percentages correct; however, there is a strong negative correlation (Pearson correlation = 0.92) between these two measures: Shorter reaction times generally mean more correct judgments. This is visualized in Figure 6, where we plotted the average percentage correct for a particular illumination combination against the average reaction time for the same illumination combination. 

Figure 6

 

Correlation between reaction times and discrimination performance averaged over all observers. Chance performance is at 33.3% correct.

Modeling

To investigate what type of information participants were using to perform this task, we modeled performance of observers based on different assumptions. 

In our first 4 models, termed PixelDifferencesModel, MeanLuminanceModel, BrightestLuminanceModel, and LuminanceSkewnessModel, we take some well-known image statistics and see whether these image statistics can predict the results of our participants. In the PixelDifferencesModel, we assume that observers simply based their judgments on absolute pixel differences between the three images. To model this, for each condition, we took the pixel values of the two images that were illuminated in the same way and subtracted the pixel values of the image with the odd-one-out illumination from them. We averaged the resulting differences. If an observer were to use this strategy, one would obtain the pattern of results that is shown in Figure 7A, where black means highest pixel difference and white means lowest pixel differences between illumination conditions. In the MeanLuminanceModel, we simply took the average luminance of all three images and calculated which average luminance was most different from the other two. If an observer were to use this strategy, one would obtain a pattern of results shown in Figure 7B. A number of studies have reported that the luminance in the brightest part of the image or the skewness of the luminance histogram yields information on material properties such as gloss (Motoyoshi, Nishida, Sharan, & Adelson, 2007; Sharan, Li, Motoyoshi, Nishida, & Adelson, 2008; Toscani, Valsecchi, & Gegenfurtner, 2013, 2017; Wiebel, Toscani, & Gegenfurtner, 2015). These statistics might also be of interest for the perception of illumination, and so we modeled them too to see whether they can predict the results of our participants. For the BrightestLuminance model, we chose not to use the brightest pixel in the image but instead used the 95% quantile of the brightness histogram, because in photographs of natural images, the brightest pixel is usually very artificial because of things such as glare. If an observer were to use this strategy, one would obtain a pattern of results shown in Figure 7C. In the LuminanceSkewnessModel, we took the skewness of the luminance histogram of all three images and calculated which skewness luminance was most different from the other two. If an observer were to use this strategy, one would obtain a pattern of results shown in Figure 7D. 

Figure 7

 

Predictions of performance with the (A) PixelDifferencesModel, (B) MeanLuminanceModel, (C) BrightestLuminanceModel, and (D) LuminanceSkewnessModel. Illumination conditions are ordered in the same way as for Figures 4 and 5. Black means large difference; white means small difference (in arbitrary units).

Pearson correlation coefficients between the model predictions and actual observer data are low: 0.22, 0.024, −0.001, and −0.007 for average percentages correct and −0.35, 0.023, 0.070, and 0.097 for average reaction time. A series of t tests after Fisher z transformation of the correlation data reveals that there is no statistical difference between the correlation coefficients for these models (p values are well above the Bonferroni-corrected threshold of 0.008 for all comparisons). The low correlation coefficients suggests that participants are probably using more specific information to do the task. This is visualized in Figure 8, where we plotted the average percentage correct and the average reaction time for a particular illumination combination against the scaled average pixel differences, mean luminance, brightest luminance, and luminance skewness for that same illumination combination. 

Figure 8

 

(A) Correlation between discrimination performance averaged over all observers and several image statistics models. (B) Reaction times averaged over all observers and the PixelDifferenceModel. (C) Correlation between discrimination performance averaged over all observers and the MeanLuminanceModel. (D) Reaction times averaged over all observers and the MeanLuminanceModel. (E) Correlation between discrimination performance averaged over all observers and the BrightestLuminanceModel. (F) Reaction times averaged over all observers and the BrightestLuminanceModel. (G) Correlation between discrimination performance averaged over all observers and the LuminanceSkewnessModel. (H) Reaction times averaged over all observers and the LuminanceSkewnessModel.

Our second model, termed the IlluminationModel, is a bit more sophisticated. It assumes that observers can extract all relevant information about illumination from the scene. The way we model this is to calculate the difference in pixel values of the images when all three objects are the same. Despite this never actually occurring in the experiment, this approach can reveal how much the images of a single object vary between illumination conditions and so how distinguishable the illumination is when it falls on the same object. We calculate these total pixel differences for all three objects (orange, teapot, and tennis ball) separately and average the resulting three values. If we again plot the highest pixel difference in black and the lowest pixel difference in white, we obtain the pattern of results that is shown in Figure 9A. This correlates nicely with the results of our observers (Pearson correlation coefficient of 0.85 for the percentages correct and of −0.85 for the reaction times). A series of t tests after Fisher z transformation of the correlation data reveals that these correlation coefficients are indeed higher than the correlations shown in Figure 8 (all p values are well below the Bonferroni-corrected threshold of 0.0125). The remaining three models use the same method but are restricted to a specific AOI of the image, containing either the highlight on the object (HighlightModel), the shading on the object (ShadingModel), or the shadow cast by the object (ShadowModel). The pattern of results we obtain for these three models is shown in Figures 9B–D. 

Figure 9

 

Predictions of performance with (A) IlluminationModel, (B) HighlightModel, (C) ShadingModel, and (D) ShadowModel. Illumination conditions are ordered in the same way as in Figure 3.

Clearly, the pattern of results from the shading model correlates less well with the pattern of results we obtained from observers (Pearson correlation coefficient of 0.54 for the percentages correct and of −0.41 for the reaction times). The pattern of results from both the shadow and highlight models correlates highly with the pattern of results we obtained from observers, with the highlight and shadow models correlating best with both the reaction time data (Pearson correlation coefficient of 0.84 for the percentages correct and of −0.90 for the reaction times for highlights, correlation coefficient of 0.87 for the percentages correct and of −0.85 for the reaction times for shadows). This is visualized in Figures 10 and 11, where we plotted the average percentage correct and the average reaction time for a particular illumination combination against the scaled average image information for that same illumination combination. A series of t tests after Fisher z transformation of the correlation data reveals that there is no statistical difference between the correlation coefficients for the HighlightModel and the ShadowModel (p = 0.75 for percentage correct, p = 0.50 for reaction times). However, the correlation coefficient for the ShadingModel is significantly lower than both (p = 0.025 and p = 0.011 for percentage correct and p = 0.0002 and p = 0.0020, respectively, for reaction times). 

Figure 10

 

Correlation between discrimination performance averaged over all observers and the IlluminationDifferencesModel for (A) All AOIs, (B) highlight AOI, (C) shading AOI, and (D) shadow AOI.

Figure 11

 

Correlation between reaction times averaged over all observers and the IlluminationDifferencesModel for (A) all AOIs, (B) highlight AOI, (C) shading AOI, and (D) shadow AOI.

Eye Fixations

Comparing our different models with the observer data suggests that observers used mainly highlights and cast shadows to distinguish the illumination conditions. To see whether we can observe such a preference for highlights and shadows in looking behavior, we analyzed observers' eye fixations. We divided the stimuli roughly into three AOIs: the top part of the images, containing the highlights; the middle part, containing most of the shading; and the lowest part, containing most of the shadows. About 82% of fixations were inside the image regions. We discarded all fixations outside of the image regions. We found that of all fixations within the images, 64% were in the shadow AOI, 27% in the shading AOI, and 9% in the highlight AOI. Analyzing the different objects separately revealed that this distribution did not depend on which object the observers were looking at. 

Although this gives us some quantitative measure that suggests observers fixated the shadows more often, the more interesting information is contained in the location and the order of the first five fixations. We plotted heat maps of first through fifth fixations for all observers in Figure 12. From this, we can clearly see that, apart from the first fixation that is located near the fixation cross, participants look mainly at the cast shadows of the three objects in a systematic way, as if they are comparing them (as well as near the fixation cross). Apparently, they hardly fixate in the locations containing shading and highlights. 

Figure 12

 

Eye fixation density plots combined for all participants. (A) All fixations. (B) All first fixations. (C) All second fixations. (D) All third fixations. (E) All fourth fixations. (E) All fifth fixations. Note that the order of the objects in this example is arbitrary; we combined all eye fixations of the 18 different layouts per illumination combination. Analyzing them separately yields similar results.

Discussion

We investigated how well observers are able to distinguish the diffuseness of the light and the number of light sources in real photographed scenes and what kind of stimulus information is most important for such a task by asking observers to distinguish different illumination conditions in an odd-one-out task. 

Our results show that participants performed above chance for most conditions, and there are systematic variations in performance over conditions. The differences in performance are not predicted well by several simple image statistics such as the average pixel differences, the mean luminance differences, the differences between the brightest luminance in the scenes, and differences in the skewness of the luminance histograms. However, the variations in performance are predicted well by a model that assumes that observers can extract information about illumination from the scene, using the differences in image structure in same-object comparisons. The advantage of such a model is that we can also look at parts of the images containing different types of information about the illumination, such as the highlights, shading, and cast shadows. The resulting simulations show that participants' performance can be explained by assuming they rely either on the cast shadows or highlights or both to compare illumination conditions. This is in line with the fact that in our rather simple stimuli, the correspondence of the cast shadow to the object is obvious, making the shadow a reliable source of information (Mamassian, 2004), apparently not only for complex scenes and light source direction and intensity (Casati, 2004; Koenderink et al., 2003; Koenderink et al., 2004; Xia et al., 2014) but also for matching illuminant diffuseness and source separation. 

When we look at the reaction times, we see that the pattern of results is similar to that of performance: Higher performance correlates with faster reactions. When there is a larger difference in highlight structure or cast shadows of the images, the reaction times are shorter. Shading information is less predictive of participants' reaction times and accuracy. This might be due to the fact that although the images are of real objects, the context they provide is rather sparse and there are very few shape cues (O'Shea et al., 2010; Ostrovsky et al., 2005). Boyaci et al. (2006) showed that all three types of information are used in their scenes; they could rule out the use of shading information in only one participant. However, we show here that it is rather unlikely that using shading information has any advantage for our participants in both reaction time and accuracy. Whether it is likely that participants use cast shadow and highlight information equally is harder to say, because in our models, there is a high correlation (0.90) between the information contained in highlights and shadows. The information contained in the shading does not correlate as well with the highlights and the shadows (0.53 and 0.55, respectively). 

There could be a geometrical explanation for the perceptual difference we find between the use of information between highlights, shading, and cast shadows. From cast shadows and highlights, we can infer mathematically higher-order information about the light field than from shading; shading is due to the diffuse part of the reflectance only. Ramamoorthi and Hanrahan (2001) showed that the diffuse part of the reflectance can be described by the second-order spherical harmonics description of the light field. Thus, from the shading, we can infer only the 0th-, first-, and second-order spherical harmonics components of the light field, which are related to intensity, average direction and strength of the light vector, diffuseness (Xia, Pont, & Heynderickx, 2016a, 2016b), and the squash tensor (see Mury, Pont, & Koenderink 2009). Cast shadows and highlights do allow the inference of higher-order contributions (Bunteong & Chotikakamthorn, 2016; Sato, Sato, & Ikeuchi, 2003). Because we varied both diffuseness of the light source and the number of light sources, and the latter distinction requires higher-order information, this might explain our perceptual results. 

Interestingly, the eye-fixation data reveal that participants primarily look at the cast shadows in favor of shading and highlights. First through fifth fixations reveal that after the first fixation near the fixation point, observers mainly move their eyes between the shadows of the three objects, seemingly comparing them. This supports our notion that shading information is hardly used in this case, but it seems contradictory to the fact that highlight information in the image also predicts both the performance and reaction time data well and also contradicts data from Boyaci et al. (2006). However, it could be the case that observers did not need to make many fixations to extract the information from the highlights but were able to extract this high-contrast information directly at fixation. This is in accordance with results from Sharan et al. (2009, 2014) that show (among other things) that gloss can be categorized within 40 ms. 

Conclusions

We show that observers' performance on illumination matching in simple real scenes is best predicted by a model that uses only highlights and cast shadow information about the illuminant. This was confirmed by fixation data; observers fixated mostly in the cast shadow areas. However, the number of eye fixations in highlight areas is rather low. This could be due to the fact that highlight information might be readily available to the observers, even at larger eccentricities. We conclude that observers rely mainly on cast shadows and possibly also on highlights to distinguish illumination conditions that vary in low-level aspects such as number of sources and diffuseness. This makes sense from a geometrical point of view: Higher-order information on the light field is available only from shadows and highlights, not from shading. 

Acknowledgments

Part of this work has been funded by the EU FP7 Marie Curie Initial Training Networks (ITN) project PRISM, Perceptual Representation of Illumination, Shape and Material (PITN-GA-2012-316746), awarded to SCP. We thank two anonymous reviewers for their valuable comments on the article. 

Commercial relationships: none. 

Corresponding author: Susan F. te Pas. 

Email: [email protected]. 

Address: Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands.