Effects of light map orientation and shape on the visual perception of canonical materials

Abstract

We previously presented a systematic optics-based canonical approach to test material-lighting interactions in their full natural ecology, combining canonical material and lighting modes. Analyzing the power of the spherical harmonics components of the lighting allowed us to predict the lighting effects on material perception for generic natural illumination environments. To further understand how material properties can be brought out or communicated visually, in the current study, we tested whether and how light map orientation and shape affect these interactions in a rating experiment: For combinations of four materials, three shapes, and three light maps, we rotated the light maps in 15 different configurations. For the velvety objects, there were main and interaction effects of lighting and light map orientation. The velvety ratings decreased when the main light source was coming from the back of the objects. For the specular objects, there were main and interaction effects of lighting and shape. The specular ratings increased when the environment in the specular reflections was clearly visible in the stimuli. For the glittery objects, there were main and interaction effects of shape and light map orientation. The glittery ratings correlated with the coverage of the glitter reflections as the shape and light map orientation varied. For the matte objects, results were robust across all conditions. Last, we propose combining the canonical modes approach with so-called importance maps to analyze the appearance features of the proximal stimulus, the image, in contradistinction to the physical parameters as an approach for optimization of material communication.

Introduction

One of the aims of material perception research is to understand how human beings perceive materials in varying lighting environments. The endless combinations of materials and lighting environments pose a difficult challenge on this matter in two important ways, namely, (a) same material under different lights and belonging to different shapes can have a different appearance, and (b) same appearance can be the result of different combinations of lightings, shapes, and materials (image ambiguities). The appearance of materials varies enormously depending on the lighting and shape (Olkkonen & Brainard, 2010), and human observers were found not to be “material constant” if the shape (Nishida & Shinya, 1998; Vangorp, Laurijssen, & Dutré, 2007) or the lighting varies (Dror, Willsky, & Adelson, 2004; Pont & te Pas, 2006). A well-known lighting effect for glossy surfaces that has been found in many studies is that glossy surfaces are perceived as rather matte under very diffuse lighting and glossier under directed lighting (Dror, Willsky, & Adelson, 2004; Pont & te Pas, 2006; Zhang, de Ridder, Fleming, & Pont, 2016; Zhang, de Ridder, & Pont, 2015, 2018) or perceived to have different levels of glossiness under different artificial or natural lighting environments (Adams et al., 2018; Doerschner, Boyaci & Maloney, 2010; Fleming, Dror, & Adelson, 2003; Olkkonen & Brainard, 2010; Motoyoshi & Matoba, 2012; Wendt & Faul, 2017; Zhang, de Ridder, Barla, & Pont, 2019). In a recent study on textiles, the textiles were analyzed optically and categorized into canonical modes, then combined with two canonical lightings (diffuse lighting and collimated lighting), and in a perception experiment found to have a systematic influence on the perception of six material qualities, namely, textured, metallic, silky, shiny, glittery, and soft (Barati et al., 2017). 

In one of our former studies (Zhang, de Ridder, Barla, & Pont, 2019), we asked observers to judge material qualities for a variety of material-lighting combinations. To this end, a system was developed using optics-based models of canonical material and lighting modes that span a wide range of natural materials and lighting. The four material modes employed (matte, velvety, specular, and glittery) were based on optical models that describe the bidirectional reflectance distribution functions (BRDFs) of opaque materials (Barati et al., 2017; Koenderink & Pont, 2003; Ward, 1992), representing, respectively, diffuse scattering, asperity scattering, forward scattering, and meso-facet scattering modes (Zhang, de Ridder, Fleming, & Pont, 2016), spanning a large part of the BRDF space. The three canonical lighting modes employed (ambient, focus, and brilliance light) were based on a mathematical description of the local light field, representing, respectively, the mathematical zeroth-, first-, and higher-order contributions to the spherical harmonic (SH) decomposition of the local light field (Mury, Pont, & Koenderink, 2007). The mathematical basis of this three-component framework for light descriptions, which we use as canonical modes, has a physical meaning as the three components correspond to fully diffuse light (the ambient or zeroth-order SH component, a monopole), directed light from a single direction (the focus or first-order SH component, a dipole), and the fine structure or texture of the light field (the brilliance or sum of the third and higher-order SH components), respectively. These modes represent properties of light that human observers can distinguish (Doerschner, Boyaci, & Maloney, 2007; Kartashova et al., 2016; Morgenstern, Geisler, & Murray, 2014; Schirillo, 2013; Xia, Pont, & Heynderickx, 2017). Moreover, they are known in perception-based lighting design as the basic components of an integral lighting plan (Ganslandt & Hofmann, 1992; Kelly, 1952; Pont, 2009, 2013; Pont, & de Ridder, 2018). They thus span the space of natural light conditions. 

In one experimental condition of the abovementioned study (Zhang et al., 2019), we employed computer renderings of three generic natural lighting environments approximating the ambient, focus, and brilliance lighting modes. On the basis of quantitative metrics of the relative power of their SH decomposition components, the following three lighting maps from the high-resolution USC database (http://gl.ict.usc.edu/Data/HighResProbes/) were selected to represent the three canonical lightings the best: Glacier for ambient lighting (dominated by the zeroth-order SH component), Ennis for focus lighting (dominated by the first-order SH component), and Grace-new for brilliance lighting (dominated by the sum of the higher-order SH components). Rendering the four canonical material modes under these three selected light maps resulted in a “rendered stimuli” set, comparable to the “real stimuli” set created by illuminating the four canonical materials (real objects) under the three canonical lightings. We evaluated the perception of a range of material qualities for both stimuli sets and found the results to be mutually consistent. Thus, using an optics-based canonical approach, we showed that material perception could be varied in a systematic and predictable manner. For example, brilliance lighting (the controlled real light condition and also its virtual metrics-based best match of the natural luminance maps) evoked perceived glossiness, hardness, and smoothness for specular material the most, while focus lighting (again, the real and also the metrics-based best match) evoked perceived roughness and softness for velvety material the most. 

So far, our research into light-material interactions has been confined to one shape (bird) illuminated under a fixed light direction per lighting (Zhang, de Ridder, & Pont, 2015, 2018; Zhang et al., 2016; Zhang et al., 2019). In everyday experience, however, one occasionally observes subtler lighting effects on material appearance, even when the lighting environment remains unchanged. For example, the gloss of an object may not be visible until one moves to a certain location with respect to the direction of the main light source in the room. In this case, changing the viewing angle does not change the global illumination environment, yet the appearance of the object in the image projected to our eyes becomes different due to direction-dependent forward scattering (i.e., specular reflection) of the material and thus triggers a different material perception. Changing viewing or illumination direction to trigger a different percept is often used by lighting designers and photographers to make certain features prominent in the same environment. Marlow and Anderson (2013) found that by varying the light direction of a quite directed lighting, perceived glossiness for specular bumpy objects and surfaces changes significantly. They explained the results using image features, such as contrast, coverage, sharpness of the highlights, and so on. In addition, changing the shape of an object while keeping the material and the lighting environment (illumination map) the same can also influence material perception. For example, it was found that shape can affect material perception (Vangorp, Laurijssen, & Dutré, 2007). Specifically, using a blob-shaped object resulted in a more veridical judgment of glossiness than the usual spherical object. So, although a single glossy sphere can be modeled and rendered easily (without the need for complex self-shadowing and interreflection computations), and it conveniently represents all possible visible surface orientations in one visualization, its global convexity was shown to eliminate certain image features (highlights and lowlights) that are important triggers for perceptual qualities. A blob-shaped object, for instance, might thus be a visually more informative shape for a material probe, in other words, a more visually intelligent shape for material communication. 

In the current study, we applied our optics-based canonical approach for the four materials and three lighting modes (Zhang et al., 2019) and looked into the effects of light map orientation and shape in order to further investigate how to bring out the physical material best—which is an important issue for disciplines and applications involving lighting design and material communication (e.g., computer graphics, design visualizations, webshops, and material selection interfaces). Specifically, we tested the visual perception of our four canonical materials (matte, velvety, specular, glittery) under three metrics-matched natural lighting maps best representing our canonical lighting modes (ambient, focus, and brilliance), namely, the lighting maps Glacier, Ellis, and Grace-new (from the USC database http://gl.ict.usc.edu/data/HighResProbes/). This test was confined to one perceptual quality per material, namely, the corresponding material quality (matte, velvety, specular, or glittery) for each material mode. The lighting modes are expected to give main effects that are material dependent (Zhang et al., 2019). The variation of lighting direction is expected to result in no or minor effects for the Glacier illumination, since that is the best match to ambient illumination, which in its purest form is fully diffuse and nondirectional. The Ennis lighting or best match to focus lighting has one clear average direction and thus is expected to affect material perception, based on the literature. The Grace-new as the best match to brilliance light is expected to result in medium effects, since it is more directed and structured than ambient but less directed than focus. These results are expected to be material dependent. With respect to lighting directions, the effects were expected to be significant for specular and glittery material—since for those materials, the image features (highlights) are strongly dependent on the directions of illumination and viewing, due to the steep variations of their BRDFs (see Nicodemus et al., 1992). In contradistinction, we expected the lighting direction effects to be much subtler for the matte material than for specular and glittery materials, as its BRDF is rather flat (in the ideal case constant) and the shading gradients smooth (considering only smoothly curved shapes). In the case of velvet, the key feature concerns its bright contour, which "sticks" to the silhouette, such that the velvety appearance is also expected to be more robust. To investigate the effect of shape, we implemented, next to our bird shape (Zhang et al., 2016), a blob shape (Vangorp, Laurijssen, & Dutré, 2007) and a sphere shape. We chose these shapes, since they were used in previous material perceptions studies and they represent variations from the simplest smooth (sphere), to a complicated smooth (blob), to a roundish object with sharp edges (bird). To vary only the light map orientations for each combination of shape, illumination, and material, the position of the object was kept fixed relative to the camera during the rendering process. 

Method

Stimuli

Lighting environments and light map orientations

From USC's high-resolution re-creations of Debevec's light probe images (http://gl.ict.usc.edu/Data/HighResProbes/), we selected three light maps (Glacier, Ennis, and Grace-new) that best represent our canonical lighting modes (ambient, focus, and brilliance, respectively). The selection was made by using a combination of a diffuseness metric (Xia, Pont, & Heynderickx, 2017) and a brilliance metric (Zhang et al., 2019), both based on the relative power of their SH decomposition components, and both metrics range from 0 to 1. Specifically, the Glacier map represents the ambient lighting (D_Xia = 1) the best, as it scores highest for Xia's diffuseness metric (D_Xia = 0.83; B = 0.42); the Ennis map represents the focus lighting (D_Xia = 0) the best, as it scores lowest on Xia's diffuseness metric (D_Xia = 0.17; B = 0.71); the Grace-new map represents the brilliance lighting (B = 1) the best, as it scores highest on our brilliance metric (D_Xia = 0.40; B = 0.79). Each light map was rotated vertically and/or horizontally such that the light map orientations varied over three vertical levels (original and ± π/4) and five horizontal levels (original, ± π/5 and ± 2π/5) (see Figure 1). Note that in this article, we label the vertical levels as elevations and the horizontal levels as azimuths, although they do not represent the direction of the main light source in one light map. At the bottom of the original USC's Glacier map, there was a large area of black pattern due to the occlusion of a tripod base. To make it more ambient and natural, we removed the occlusion with Photoshop's “content-aware fill” tool (see Figure 1A). Also, the Grace-new environment was blurred (see Figure 1C) to reduce noise issues in rendering, due to the presence of very small light sources of very high intensity. After blurring, the map still contains many light sources, but noise in rendering is greatly decreased. 

Figure 1.

 

(A–C) Three light maps rotated vertically for three levels and horizontally for five levels (also see supplementary documents, Supplementary Figure S7). (A) The Glacier map, (B) the Ennis map, and (C) the Grace-new map, representing ambient, focus, and brilliance lighting, respectively. Note that in this article, we label the vertical levels as elevations and the horizontal levels as azimuths, although they do not represent the direction of the main light source in one light map. From left to right: the azimuths are –2 π/5, –π/5, 0, π/5, and 2 π/5. From top to bottom: the elevations are –π/4, 0, and π/4 (i.e., the number 8 of each light map was the one with no rotation). These parameters were arbitrarily selected. Also, note that the Grace-new environment was blurred to reduce noise issues in rendering, due to the presence of very small light sources of very high intensity. After blurring, the map still contains many light sources, but noise in rendering is greatly decreased. (D) Examples of the three shapes used in the experiment, rendered with the specular material and the Grace-new light map. From left to right: the bird, the blob, and the sphere.

Rendering process: shapes and material modes

The three-dimensional (3D) model of the bird shape was created in Blender and is the same as the 3D model we used in former studies (Zhang et al., 2016). The blob shape was taken from Vangorp et al. (2007). The usual sphere shape was added as a comparison to the bird and the blob shapes, as shown in Figure 1D. The matte material mode was simulated to resemble a (hypothetical) material with a Lambertian BRDF. The velvety material mode was implemented with the asperity-scattering BRDF model of Koenderink and Pont (2003). The specular material mode was implemented with an isotropic Ward BRDF (Ward, 1992). The glittery material mode was implemented by mimicking the occurrence of multifaceted flakes at the surface of the object (Zhang et al., 2019). 

Rendering was performed in Gratin version 0.3 for Apple Mac OS (Vergne & Barla, 2015) to code and compile the computer rendering program in OpenGL shading language (GLSL) version 410 (see Supplementary Figure S6). 

Irrespective of the choice of material, rendering was performed in RGB (float 32 bits precision), with an orthographic camera, without tone mapping. Rendering was done through ray tracing at 2,000 samples per pixel (spp) unless specified otherwise and considered only direct lighting, a reasonable approximation for the object shapes we consider. We used the same material models as in our previous work (Zhang et al., 2019); we thus refer the reader to our previous study for a detailed description. 

In order to simulate the matte material, an environment prefiltering approach was implemented as it completely removes noise coming from the rendering process of the diffuse component. Specifically, assuming that the shadowing and interreflection effects can be neglected, the diffuse component of the materials may be represented equivalently using a diffuse-filtered version of the illumination environment, as provided on the USC website (http://gl.ict.usc.edu/Data/HighResProbes/). Rendering the diffuse component then simply consists of evaluating the diffuse-filtered environment in the direction of the surface normal and multiplying the result by the colored diffuse albedo. 

For the specular and glittery modes, we employed Monte Carlo integration using importance sampling of the Ward model to speed up convergence (Walter, 2005). We have also included a Fresnel term in the model to improve physical plausibility, using an index of refraction of 1.5, which yields a reflectivity of 4% at normal incidence (typical of dielectrics). For the glittery mode, we used four times more samples (8,000 spp) to capture the fine spatial variations in the flake texture. 

For the velvety mode, we relied on standard cosine-weighted importance sampling to evaluate the asperity scattering model of Koenderink and Pont (2003), which required longer rendering times. 

In Supplementary Figures S1 to S4, we show all stimuli per material, lighting, and shape. The numbers 1 to 15 correspond to the oriented light maps in Figure 1. 

Procedure

At the beginning of the experiment, observers were first shown all of the stimuli (540 computer-rendered images in total; see supplementary materials) twice in a randomized order, to give them an idea about the range of the stimuli and their scale for the rating. They were instructed that in each trial, a question “rate how [..] is the object?” was shown on top of the screen, with [..] displaying one of the four names, namely, matte, velvety, specular, or glittery. 

In each trial, 15 stimulus panels were shown to the observers below the question, with a slider next to each image (Figure 2). The 15 stimulus panels had the same material, shape, and light map but only differed in light map orientations (three elevations and five azimuths). The observers were explicitly instructed that the task was to rate the same material using the same range in different trials, instead of using the full scale in each trial per 15 stimulus panels. This was done to allow analysis per material instead of only per 15 stimuli. Panels were randomly positioned, and all slider bars were initially set at the bottom. The task of the observers was to rate each image by moving the slider bar, representing “not [..] at all” (or “0” within a “0” to “1” range) at the bottom of the slider to “extremely [..]” (or “1” within a “0” to “1” range) at the top of the slider. When clicking the mouse button within the panel of a stimulus image, a horizontal bar would appear superpositioned on the stimulus, which was slightly thinner than the slider bar attached to the right. When dragging the mouse cursor within the panel, both bars moved vertically together. When releasing the mouse button, only the slider bar was shown to indicate the rating, while the thinner bar would disappear. Observers could freely go back and forth to rate any one of the 15 images until pressing the “Enter” key to go to the next trial. 

Figure 2.

 

The user interface of the experiment developed with the Psychophysics Toolbox. In this screenshot of an example trial, the stimulus is the same specular blob-shaped object rendered using the Ennis map for 15 light map orientations. The resulting 15 stimulus panels were randomly positioned in each trial. The question “rate how [..] is the object” was positioned above the stimuli, with [..] displaying one of the four material names, in this case, specular. A slider bar was positioned on the right-hand side of each stimulus panel, initially set at the bottom of the slider. Observers were told before the experiment that the vertical slider scales from “not [..] at all” at the bottom to “extremely [..]” at the top. If a slider was moved, an additional horizontal bar was superimposed on the stimulus image to display the current rating value, as demonstrated in the stimulus at the bottom-right corner. The slider bar next to the stimulus image was attached to the thinner horizontal bar, moving vertically to indicate the ratings when dragging the mouse cursor. When releasing the mouse button, only the slider bar was shown, while the thinner bar would disappear. Observers could freely go back and forth to rate any one of the 15 images until pressing the “Enter” key to go to the next trial. The number of the current trial and the total number of trials were shown in the top-left corner of the screen. Note that the settings shown in this figure are not from any of the observers but are generated for demonstration purposes only, and the stimuli appeared different as shown in supplements due to the process of taking the screenshot.

For each canonical material mode, only the corresponding material term [..] was tested, that is, we asked “rate how matte is the object?” for the stimuli rendered using the matte material mode only. So, for each observer, the experiment contained altogether 4 materials × 3 illuminations × 3 shapes = 36 trials of 15 stimulus panels, in total 36 × 15 = 540 ratings. The experiment took between 40 minutes and an hour per observer. The interface was developed with the Psychophysics Toolbox extensions (Brainard, 1997; Kleiner et al., 2007; Pelli, 1997) in MATLAB R2016b and presented on a linearly calibrated EIZO ColorEdge CG277 (27-in. class calibration color LCD) display. The viewing distance was around 30 cm. 

Observers

Twelve paid observers participated in the experiment. All participants had normal or corrected-to-normal vision and were inexperienced in psychophysical experiments. Participants read and signed the consent form before the experiments. The study was approved by the Human Research Ethics Committee at Delft University of Technology and conducted in accordance with the Declaration of Helsinki and Dutch law. 

Results

Here we present the general results per material mode and, thus, per material quality. We analyzed the rating data per material using a four-way repeated-measures analysis of variance (ANOVA) with three lightings, three elevations, five azimuths, and three shapes being the independent variables. We assumed that each observer used one constant perceptual scale for the same material despite the changes of lightings and shapes across all trials (presented in randomized order) and rescaled the 135 data points (3 shapes × 3 lightings × 15 directions) per material per observer such that the ratings ranged from 0 to 1. For the interpretation of the results, the analysis was confined to the main effects and the first-order interaction effects. We do not present higher-order interactions because we consider those too complicated to be meaningful. In the supplementary materials, we plotted the ratings next to the corresponding stimuli, allowing visual inspection of the stimuli and data (Supplementary Figures S1–S4). Note that since light sources within the three light maps may be located anywhere (e.g., on the side or at the top), it is only meaningful to directly compare the effects of azimuth and elevation within but not across light maps. 

Matte

We did not find any significant main effect (lighting: F(2,  22) = 1.38,  p = 0.27; shape: F(2,  22) = 0.39,  p = 0.69; azimuth: F(1.58,  17.37) = 1.36,  p = 0.28; elevation: F(1.27,  14.00) = 1.10,  p = 0.33) or any first-order interaction effect for the “matte” ratings of the matte material (M = 0.46, SEM = 0.01 for all ratings). This suggests that perceived matteness was independent of the light maps, lighting directions, and shape of the object. This is confirmed in Figure 3, showing that the averaged ratings for the matte material mode were robust across all conditions. 

Figure 3.

 

The averaged matteness ratings of 12 observers per shape (subplot rows), illumination (subplot columns), elevation (x-axis in each subplot), and azimuth (bars for each elevation in each subplot). The error bars indicate ± 1 SEM.

Velvety

Figure 4 presents the averaged velvety ratings for the velvety material mode. A number of trends can be discerned. First, the overall mean of the velvety ratings for Glacier map (0.57 ± 0.02) is higher than that for the Ennis map (0.44 ± 0.02) and Grace-new map (0.44 ± 0.02), a difference that is substantial as confirmed by a significant main effect for lighting environment (F(2,  22) = 5.48,  p = 0.012). Second, there appears to be an effect of azimuth, in particular for Ennis, in that the means of the ratings for azimuth1 and azimuth5 were lower than for azimuth 2, azimuth 3, and azimuth 4. This is in line with the observation that there is a significant main effect of azimuth (F(4,  44) = 9.13,  p < 0.001) combined with a significant interaction effect between lighting environment and azimuth (F(8,  88) = 11.80,  p < 0.001). Figure 5A confirms the abovementioned trend by showing that it can be seen for Ennis while the azimuth has hardly any impact on the ratings for Glacier and Grace-new. Note that the ratings for Grace-new were lower than for Glacier. Third, for Grace-new, the averaged ratings appear to decrease systematically from elevation 1 via elevation 2 to elevation 3, which cannot be seen for Glacier and Ennis. The statistical analysis indicated a significant main effect for elevation (F(1.21,  13.34) = 6.63,  p = 0.019; the assumption of sphericity had been violated: χ²(2) = 10.482,  p = 0.005, and hence the degrees of freedom were corrected using the Greenhouse-Geisser estimate of sphericity ε =  0.61). In addition, there was a significant interaction effect between the lighting and the elevation (F(4,  44) = 11.72,  p < 0.001) in that, going from elevation 1 to elevation 3, the ratings drop for Grace-new only (Figure 5B). Finally, we did not find any significant differentiating effect of shape on the velvety judgments for the velvety material (main effect of shape: F (2, 22) = 1.14, p = 0.34). To summarize, the perceived velvetiness of velvety material was affected by the lighting environments (light maps) and light map orientations, with the Ennis and Grace-new maps reducing perceived velvetiness the most, while perceived velvetiness was found to be robust for the shape of the object. 

Figure 4.

 

The averaged velvetiness ratings of 12 observers per shape (subplot rows), illumination (subplot columns), elevation (x-axis in each subplot), and azimuth (bars for each elevation in each subplot). The error bars indicate ± 1 SEM.

Figure 5.

 

(A) The mean velvety ratings, averaged across observers, shapes, and elevations, as a function of azimuth and per light map. (B) The mean velvety ratings, averaged across observers, shapes, and azimuths, as a function of elevation and per light map. The error bars represent ± 1 SEM.

Specular

Figure 6 presents the averaged specularity ratings for the specular material mode. When comparing the columns in Figure 6, we found that the averaged ratings of specularity were relatively lower for the Glacier map (0.36 ± 0.02) than for the Ennis (0.56 ± 0.02) and Grace-new maps (0.58 ± 0.02). This was confirmed by a significant main effect for light environment (F(2,  22) = 14.05,  p < 0.001). When comparing the rows in Figure 6, we found that the averaged ratings of specularity were relatively lower for the bird shape (0.43 ± 0.02) than for the blob (0.54 ± 0.02) and the sphere (0.54 ± 0.02). This was also confirmed by a significant main effect for shape (F(2,  22) = 8.50,  p = 0.002). We did not find any significant differentiating effect of light map orientations on the specularity judgments for the specular material (main effect of azimuth: F (1.25, 13.74) = 2.70, p = 0.12; main effect of elevation: F (1.10, 12.13) = 3.35, p = 0.089). The significant interaction effect between light maps and shapes (F(4,  44) = 2.82,  p = 0.036) was mainly due to the ratings for the Ennis map systematically increasing from the bird via the blob to the sphere, while for the Glacier and Grace-new maps, the ratings were rather flat with a small peak for the blob shape (Figure 7). To summarize, the perceived specularity of specular material was affected by the lighting environments (light maps) but not by the light map orientations, with the Glacier map reducing perceived specularity the most. Similarly, perceived specularity depended somewhat on the shape of the object with bird under the Glacier map reducing specularity the most and sphere under the Ennis map highlighting specularity the most. 

Figure 6.

 

The averaged specularity ratings of 12 observers per shape (subplot rows), illumination (subplot columns), elevation (x-axis in each subplot), and azimuth (bars for each elevation in each subplot). The error bars indicate ± 1 SEM.

Figure 7.

 

The mean specularity ratings, averaged across observers, azimuths, and elevations, per shape and per light map. The error bars represent ± 1 SEM.

Glittery

Figure 8 presents the averaged glittery ratings for the glittery material mode. Two trends can be discerned. First, the overall mean of the glittery ratings for the bird shape (0.31 ± 0.02) is lower than that for the blob (0.52 ± 0.02) and the sphere (0.52 ± 0.02) shapes, a difference that is substantial as confirmed by a significant main effect for shape (F(2,  22) = 28.42,  p < 0.001). Second, unlike the light map itself, light map orientations play a role in perceiving glitteriness as confirmed by significant main effects for azimuth (F(4,  44) = 48.81,  p < 0.001) and elevation (F(2,  22) = 21.62,  p < 0.001) and a nonsignificant main effect for light map (F(2,  22) =  2.06,   p = 0.15). Specifically, the ratings of azimuth 2 (0.51 ± 0.02) and azimuth 3 (0.51 ± 0.02) were significantly higher than those for azimuth 1 (0.41 ± 0.02), azimuth 4 (0.44 ± 0.02), and azimuth 5 (0.38 ± 0.02) which was mainly due to the ratings for the bird shape (Figure 9). The latter is confirmed by a significant interaction effect between the shapes and azimuths (F(8,  88) = 9.63,  p < 0.001). To summarize, the perceived glitteriness was affected by the shape of the objects, with the bird shape reducing perceived glitteriness the most, as well as the light map orientations, particularly the azimuths. 

Figure 8.

 

The averaged glitteriness ratings of 12 observers per shape (subplot rows), illumination (subplot columns), elevation (x-axis in each subplot), and azimuth (bars for each elevation in each subplot). The error bars indicate ± 1 SEM.

Figure 9.

 

The mean glittery ratings, averaged across observers, light maps, and elevations, as a function of azimuth and per shape. Note that the azimuth variation is confounded with a change of the lighting elevation. The error bars represent ± 1 SEM.

Perceptual effects, quality ratings, and image features

In Supplementary Figures S1 to S4, we show the rating data per material as a function of the azimuth next to the stimuli images per material, from (A) to (C) under the Glacier map, (D) to (F) under the Ennis map, and (G) to (I) under the Grace-new map. At left, we show the corresponding ratings. The numbers 1 to 15 on the x-axis correspond to the oriented light maps as shown in Figure 1. The stimuli images are shown on the right, with the numbers on the bottom-right corner of each stimulus image corresponding to the oriented light maps. The rows represent the three elevations. In these figures, we could make observations about which image features might have triggered the perceptual effects that triggered the quality assessments. Note that tonemapping was used only for presentation in this article but not for the stimuli in the experiments. In the following sections, we will describe our observations in detail per material mode and connect the observations to our results and previous findings in literature. 

Matte

For matte materials, the diffuse shading gradients vary smoothly and do not show sudden (dis)appearances of highlights or other salient features, which can explain that the perception of matteness is quite constant (Supplementary Figure S1). This kind of robustness implies that its invariants can be used to infer shapes based on the shading patterns (Belhumeur, Kriegman, & Yuille, 1999; Koenderink & van Doorn, 1980; Kunsberg & Zucker, 2013, 2018; Narasimhan, Ramesh, & Nayar, 2003). Simultaneously, knowing the shape can help observers with judging the characteristics of the local light field and material (Kartashova et al., 2016; Koenderink et al., 2007; Xia, Pont, & Heynderickx, 2014,2016). As an exception, illusory gloss effects were found for matte materials on bumpy surfaces under collimated lighting (Wijntjes & Pont, 2010), where second-order shading effects were confused with specular highlights. Such effects only appear for quite nongeneric lighting and only in specific cases form a real problem, for instance, endoscope lighting, for which the viewing (camera) and lighting directions coincide (Wu et al., 2010). 

Velvety

The main effect we found for the velvety materials was that velvetiness was sometimes rated low for the Ennis and Grace-new lightings, especially when the main light source was coming from the back of the objects. The chief visual cue of velvet appearance is a thin but very steep luminance gradient at the silhouette, that is, the bright contours along the surface due to surface scattering by the asperities (Koenderink & Pont, 2003). The cue is invariant to lighting directions when using the Glacier environment map (shown in Supplementary Figure S2A–C), as are the ratings, confirming our expectations. Supplementary Figure S2 reveals that, for the other two lightings, all stimuli with relatively low ratings have other image features in common. When using the Ennis environment map, we observe relatively low ratings for stimuli Nos. 1, 5, 6, 10, 11, and 15 of the bird shape (Supplementary Figure S2D), as well as Nos. 6, 10, 11, and 15 of the blob shape (Supplementary Figure S2E) and the sphere shape (Supplementary Figure S2F). This corresponds to the results shown in Figure 5A, which might be due to an ambiguity caused by the directed light source: When the directed light source is behind the object, the asperity-scattering mode's luminance gradient might be confounded with the diffuse- or specular-scattering mode’s gradients. In other words, the bright contour due to asperity scattering in isolation (without diffuse shading over the body) cannot be distinguished from the bright rim that occurs for the combination of backlighting and diffuse and/or specular scattering. The material may then be perceived as matte or even as somewhat specular, rendered using rim lighting. This could also explain the results using the Grace-new environment map under elevation 3 (Supplementary Figure S2G–I and also see Figure 5B). The significant drops in the ratings as a function of elevation correspond to backlighting configurations. 

To conclude, velvetiness seems to require not only bright contours due to the surface scattering but also the co-occurrence of diffusely scattered luminance or smooth gradients over the body. Simply presenting only a “bright contour” on an otherwise dark object will not trigger the perception of velvetiness but instead may trigger the perception of matteness. This corresponds to results from our abovementioned former work (Zhang et al., 2019), in which we found strong interactions between the matte and velvety material modes. 

Specular

The main visual cues for specular materials are the specular highlights. When using the Glacier environment map, the overall ratings for the specular materials were relatively lower (Supplementary Figure S3A–C), corresponding to the results shown in Figures 6 and 7. This was within our expectations as it confirmed previous findings in glossiness perception literature indicating that perceived glossiness reduces under diffuse lighting (Dror, Willsky, & Adelson, 2004; Pont & te Pas, 2006; Zhang, de Ridder, & Pont, 2015; Zhang et al., 2016, 2019), because the highlights will be diffused. Meanwhile, perceived glossiness is also affected by negative contrast of reflections caused by dark parts of the environment generating dark specular reflections or lowlights (Kim, Marlow, & Anderson, 2012). The combination of fine-structured bright highlights and dark lowlights might explain the perceived glossiness in Supplementary Figure S3A–C. 

When using the Ennis environment (Supplementary Figure S3D–F), the most notable image cues are contrast and coverage of specular highlights. The interaction effects show increased ratings for the sphere under Ennis lighting (i.e., the averaged ratings for specularity were highest when combining the sphere shape and Ennis lighting) (Figure 7). This might be due to a clear reflection of the illumination. The window-shaped specular highlight patterns are particularly clearly reflected on the sphere (Supplementary Figure S3F). As a comparison, highlights on the bird (Supplementary Figure S3D) and the blob (Supplementary Figure S3E) deformed in a more complex manner. This confirmed previous findings on glossiness perception, namely, that the shape of highlights may influence glossiness perception (van Assen, Wijntjes, & Pont, 2016). It also shows that when highlights reveal real-world illumination properties, they are less likely to be misperceived as texture and thus could increase perceiving glossiness (Fleming, Dror, & Adelson, 2003). 

When using the Grace-new environment map (Supplementary Figure S3G–I), the ratings in general were relatively high as the highlights are quite visible in most of the stimuli, as expected, and reflect the fine structure of the brilliance lighting. Coverage and contrast of the highlights were mainly varying as a function of the elevation, and the effects of azimuth are not as salient as for the other two light maps, due to the angular structure of the brilliance lighting in the Grace-new environment (primarily many tiny hotspots from above). 

Unexpectedly, we did not find significant effects of light map orientations for the perception of specularity in this study (unlike, for example, in Marlow, Kim, & Anderson, 2012). However, we did find higher-order interaction effects between light map orientation, shape, and the choice of lighting environment. This was not mentioned in the results since the interpretation of these higher-order effects is usually very complex. In the stimuli images, we could observe subtle variations in the specular ratings as the light map orientations varied. Some of these variations might be due to the changes of the contrast and coverage of the high/lowlights with respect to the diffuse shading, possibly in combination with the effect of clipping highlights in some stimuli of the Ennis light map due to the high dynamic range of the lighting environment. 

Glittery

The main visual cues for glitteriness also seemed to be the features of the highlights on the glitters (see Supplementary Figure S4). With simple image processing—namely, thresholding the top 1% brightest pixels (as the glitters) in each stimulus—we could count the number of glitters as a coarse evaluation of the coverage of the highest intensity glitters. As shown in Table 1, significant correlations were found between the numbers of glitters in the stimuli and the glittery ratings per shape and lighting, except for the blob under the Ennis and Grace-new lighting. 

Table 1.

 

The correlations between the numbers of glitters (top 1% brightest pixels in the stimuli) and the perceptual ratings for glitteriness per shape and lighting.

 

General discussion

The main question we pose in this article is how light map orientation and object shape influence the perception of materials in addition to the material reflectance itself and the main modes of the lighting environment. To answer this question, an experiment was set up in which we combined four canonical material modes, three shapes, and three illumination environments and then oriented the illumination environments in 15 different directions (varying across three elevations vertically and five azimuths horizontally). In our rating experiment, we found the following main results: 

• • For matte materials, perceived matteness was robust and constant across all variations (i.e., no effect was found for light map orientation, shape of the object, or lighting mode).
• • For the perceived velvetiness of velvety materials, there were significant effects of light map orientation, which were lighting dependent but shape independent. Such effects were evoked the most under the Ennis light map and hardly under the Glacier and Grace-new light maps. The Glacier light map highlighted velvetiness the most.
• • For specular materials, we found no significant effect of light map orientation (for both elevation and azimuth). The perception of specularity was influenced by light mode and shape, with the Glacier light map as well as the bird shape reducing perceived specularity the most.
• • For the perceived glitteriness of glittery materials, the effects of direction and shape were significant, with the bird shape reducing glitteriness the most. Lighting mode had only an interaction effect, reducing perceived glitteriness the most for the Glacier light map for all elevations and Grace-new for elevation 3.

In a former work (Zhang et al., 2019), we investigated the interaction between material and light modes for one shape only, namely, the bird shape, and for one lighting direction. In that study, we combined the four canonical material modes (matte, velvety, specular, and glittery) with three canonical lighting modes (ambient/ Glacier, focus/Ennis, and brilliance/Grace-new) and found material-dependent lighting effects for nine qualities (matte, velvety, specular, glittery, glossy, rough, smooth, hard, and soft), which were similar to the main lighting effects found in the current study. In particular, the impact of the Glacier light map with respect to the other two light maps was similar: reducing perceived specularity and glitteriness for the specular and glittery materials, respectively, and highlighting velvetiness and, to a lesser extent, matteness for velvety and matte material, respectively. 

We again found a difference between the matte and velvety material modes, on the one hand, and the specular and glittery materials, on the other hand, when considering the effects of shape on the perceived qualities. No systematic effects were found for the matte and velvety materials, whereas both specular and glittery materials showed a reduction in their corresponding perceived qualities for the bird shape with respect to the other two shapes. An explanation for this systematic finding is probably that the BRDFs of specular and glittery material are more peaked than those of velvety and matte materials. If the lightings or shapes vary, the appearances of specular and glittery materials then will change more than those of matte and velvety materials. 

Interestingly, another differentiation was found in the current study, namely, between matte and specular materials, on the one hand, and velvety and glittery materials, on the other hand. This was based on the (in)sensitivity for lighting direction where velvety and glittery materials showed systematic changes in the quality ratings as a function of azimuth and/or elevation, effects that were absent for the other materials. In Zhang et al. (2019), such a differentiation could also be observed, but there it was based on the judgments of roughness and smoothness. In that study, the matte and specular material modes were assessed to be more smooth and velvety and glittery material modes to be more rough. Since image texture (gradients) due to 3D surface corrugations are extremely sensitive to lighting variations (Pont & Koenderink, 2008), these findings might well be related. 

The main question we addressed in this article is how different material and shape and lighting combinations affect perceived material appearance. Endless combinations of materials, shapes, and illuminations may cause a similar appearance while small variations of one of those factors can sometimes cause large variations in appearance. A major challenge is to find a way to predict the appearance within this endless space of possibilities. In order to do so, we need to get a grip onto the proximal stimulus, the image, and its features, in contradistinction to the basic physical parameters that determine them (and in the end we also will understand the relationships between the physics and image features). To this end, we want to bring up the notion of the importance map, which characterizes the contribution of different lighting directions depending on surface reflectance (material) and geometry (shape). When we trace the light rays from the viewpoint to the surface back to the environment, we find that more light rays accumulate in some directions than in other directions. An importance map records this accumulation: Brighter points in the map correspond to directions where more accumulation has occurred. 

Specifically, we show in Figure 10 how multiple light rays are traced back to the same direction of an importance map from the surface of a specular bird (not only from the highlights but also the rest of the surface), appearing as a bright spot in the importance map due to the accumulation. It suggests that that spot in the light map has a strong contribution to the final image, which we call “importance,” whereas darker regions in the importance map correspond to directions in the light map that will hardly be reflected to the viewer (i.e., they are less “important”). Since the importance map only depends on shape, material, and viewing direction, it is independent of the light map (and its variation after rotation). Hence, if we rotate the light map such that the light sources match the brightest regions in the importance map, the imaged object surface brightens at corresponding locations, depending on its shape and material. Rotating the object (which we did not do in the current study) would also impart a change in the importance map, as the surface shape visible from the viewpoint would change as well. 

Figure 10.

 

A schematic demonstration of importance map construction, taking a specular bird and its importance map as an example. Top left: rays are traced from the viewpoint toward the object, hitting the object surface at different locations; depending on the material's BRDF, new outgoing rays are emitted from these locations. Please note that the way we trace light rays is opposite to how light rays transmit from the sources in lighting environments to object and form an image. Top right: multiple outgoing rays from different positions on the object may have the same direction (in orange); they are then accumulated in the same direction in the distant spherical environment (that is, a single point on the spherical map). Bottom: performing this accumulation for all outgoing rays results in an importance map that we store using a latitude-longitude projection. The central portion of the importance map corresponds to rays that have been projected toward the frontal half of the spherical environment (front), while the sides (where the orange cross is located in our example) corresponds to rays projected toward the rear half of the spherical environment (back). Note that the frontal half of the environment is actually located behind the observer/camera. Brighter regions of the importance map correspond to directions for which more rays have accumulated, due to shape, material, or both.

The 12 importance maps corresponding to the four canonical materials and the three shapes we implemented are shown in Supplementary Figure S5. We can immediately see that the importance maps of matte objects are robustly diffused and quite symmetric in all shapes, which can explain why perceived matteness was found to be constant across all lightings and light map orientations. The importance maps of the velvety material are similar to those of the matte material as they are quite diffused too, but different in that they show some fine structures for the blob and bird. On the contrary, the importance maps of specular and glittery materials are clearly different for each shape and show mutually similar asymmetric structures. The importance maps for glittery are more diffuse than those for specular, due to the broadening of the specular peak caused by the distribution of flakes that compose glitter. The (lack of) variations of the importance maps for the shapes explain some of the main effects found in our experiment, namely, that perceived specularity and glitteriness were influenced by shape, while perceived matteness and velvetiness were not. It also directly shows that the importance maps for specular and glittery materials varied in a more fine-grained way than those of velvety or matte materials, which corresponds to the fact that effects were stronger for materials with peaked BRDFs. 

In future work, we would like to explore the use of importance maps to predict how lighting affects image features and thus permits solving problems such as optimizing lighting for material and shape perception. For example, in combination with metrics, they could be used for predicting the strength of image cues such as the sharpness, the contrast, and the coverage of the highlights that trigger glossiness perception (Marlow, Kim, & Anderson, 2012). In supplementary materials, we illustrate the potential of this approach by showing the product between light maps and importance maps. An example is given in Figure 11, where we show the product of the importance map of the “specular bird” with the Ennis lighting environment for two orientations. When the main light source is oriented such that it matches the brightest spot in the importance map (left column in Figure 11), large and bright specular highlights appear in the rendered image. When the main light source is oriented such that it matches a less bright spot on the back side of the importance map (right column in Figure 11), we observe a small specular highlight on the silhouette of the object. 

Figure 11.

 

An example of the predictive power of the product between an importance map and a light map. From top to bottom, we show the importance map for the specular bird, the Ennis environment map for two orientations, the product between light and importance maps, and the corresponding renderings.

Conclusions

In this study, we primarily investigated how light map orientation and shape influence the visual perception of four canonical materials (matte, velvety, specular, and glittery). Specifically, we performed a rating experiment in which, in each trial, we presented observers 15 stimuli images that differed in 15 orientations (three vertical levels and five horizontal levels) of the lighting environment, while having the same material, shape, and lighting environment (lighting mode), and instructed them to evaluate the corresponding material quality. Effects of light map orientation were found for velvety and glittery materials but not for matte and specular materials. Effects of shape were found for specular and glittery materials but not for matte and velvety materials. Effects of lighting mode were found for velvety and specular materials but not for matte and glittery materials. Hence, the perception of matte for matte materials was found to be the only material quality that is robust across all manipulations of lighting and shape. The results confirmed key image features triggering perceived specularity, glitteriness, velvetiness, and matteness. 

Acknowledgments

This work has been funded by the EU FP7 Marie Curie Initial Training Networks (ITN) project PRISM, Perceptual Representation of Illumination, Shape and Material (PITN-GA-2012-316746). 

Commercial relationships: none. 

Corresponding author: Fan Zhang. 

Email: [e-mail]. 

Address: Delft University of Technology, Delft, Netherlands.