Understanding gloss perception through the lens of art: Combining perception, image analysis, and painting recipes of 17th century painted grapes

Abstract

To understand the key image features that we use to infer the glossiness of materials, we analyzed the pictorial shortcuts used by 17th century painters to imitate the optical phenomenon of specular reflections when depicting grapes. Gloss perception of painted grapes was determined via a rating experiment. We computed the contrast, blurriness, and coverage of the grapes' highlights in the paintings' images, inspired by Marlow and Anderson (2013). The highlights were manually segmented from the images, and next the features contrast, coverage, and blurriness were semiautomatically quantified using self-defined algorithms. Multiple linear regressions of contrast and blurriness resulted in a predictive model that could explain 69% of the variance in gloss perception. No effect was found for coverage. These findings are in agreement with the instructions to render glossiness of grapes contained in a 17th century painting manual (Beurs, 1692/in press), suggesting that painting practice embeds knowledge about key image features that trigger specific material percepts.

Introduction

In the last two decades, artists and vision scientists have boosted joint efforts to mutually profit from each other's knowledge (Adelson, 2001; Wade, Ono, & Lillakas, 2001; Cavanagh, 2005; Pinna, 2007; Conway & Livingstone, 2007; Cavanagh, Chao, & Wang, 2008; Melcher & Cavanagh, 2011; Pepperell & Ruschkowski, 2013; DiPaola, Riebe, & Enns, 2013). Via careful observation of the world, painters have developed implicit knowledge of the key image features needed to render different materials, and they have transferred that to the canvas. Vision scientists can thus use artworks produced throughout the centuries to extract these features and understand visual perception. 

Naturalistic paintings offer novel learning possibilities to the ongoing research on gloss perception. When rendering glossy materials, painters did not retrieve the exact reflectance function of the object they were depicting. Most likely, they rather portrayed the optical phenomenon representing its most salient characteristic, namely its specular peak, by applying a bright spot following the curvature of the surface (Cavanagh et al., 2008). Specular highlights are, indeed, the most common monocular cues used by painters to induce a glossy impression (Wendt, Faul, & Mausfeld, 2008; van Assen, Wijntjes, & Pont, 2016). Here we assume “real-world illumination”, allowing reliable and accurate estimations of the gloss (Fleming et al., 2003), from one primary light source (e.g., a window, as was common in 17th century painting studios), and we ignore illumination variations. 

According to Marlow, Kim, and Anderson (2012), the key visible features of a highlight that affect gloss perception are coverage, sharpness, and contrast. They represent respectively the width, steepness, and height of the specular peak of the reflectance function. Perceptual effects of contrast, coverage, and sharpness of the specular reflections were already considered separately in literature (Beck & Prazdny, 1981; Ferwerda, Pellacini, & Greenberg, 2001; Berzhanskaya, Swaminathan, Beck, & Mingolla, 2005), and found to influence gloss perception. Marlow et al. (2012) investigated their combined effect. Following the study of Ho, Landy, and Maloney (2008), Marlow et al. (2012) extended the research on the perceptual interaction between bumpiness and glossiness, as a function of the illumination geometry. By varying the surface reliefs and illumination directions, but keeping constant the reflectance function, they observed variations in perceived glossiness. Such variations could be predicted by modeling the perceived values of contrast, sharpness, and coverage of the specular reflections. In a follow-up study, Marlow and Anderson (2013) tested the efficacy of perceptual ratings of these highlights' features as predictors, by systematically varying their contribution in the stimuli. By tuning the extrinsic factors of surface geometry and illumination, they could manipulate the highlight features. They demonstrated that the weighted combination of the perceived highlight features can generate and predict the glossy appearance of spherical and planar rendered objects. 

Here, we tested if the same holds for glossy materials depicted in paintings. We tested glossiness perception of grapes in Dutch 17th century paintings via a rating experiment, and whether those data can be predicted by the image features of the highlights. To do this, we developed a novel method to semiautomatically compute the features directly from segmented images of the paintings. The reason for preferring paintings from the 17th century over other periods is the accurate rendering of reality that characterizes this age. We chose to study grapes because they represent an accessible starting point, given their more or less spherical shape. Moreover, grapes offer the advantage of having been painted often during the 17th century, allowing us to collect a high number of stimuli. 

In addition to linking perceptual judgments and image analyses, we investigated whether we could find hints to the three highlights' features in the pictorial recipe for grapes contained in a 17th century treatise on oil painting, The big world painted small (Beurs, 1692/in press). This treatise represents one of the most valuable art historical records of the 17th century studio practice. Throughout the six chapters that make up the treatise, Beurs (1692/in press) provides detailed instructions and practical tips on how to render all kind of materials and surface textures. As the recipes of Beurs (1692/in press) were shown to match the painting practice of some of his contemporaries (Wallert, 1999, 2012; De Keyser et al., 2017), we can use this written source to grasp 17th century painters' implicit knowledge. 

Previous work

It is known that gloss perception interacts with the 3D shape of the target object (Vangorp, Laurijssen, & Dutré, 2007; Ho et al., 2008; Marlow & Anderson, 2015), its surface structure (Pont, van Doorn, Wijntjes, & Koenderink, 2015), and the light field in which it is embedded (Fleming et al., 2003; Pont & te Pas, 2006; Zhang, de Ridder, & Pont, 2015; Wendt & Faul, 2017). Certain combinations of illumination directions and surface reliefs can even render a matte Lambertian surface to look glossy (Wijntjes & Pont, 2010). 

The presence of either highlights or lowlights is recognized to be the minimum requisite to convey a glossy impression (Beck & Prazdny, 1981; Berzhanskaya et al., 2005; Kim et al., 2012; van Assen et al., 2016), as long as they are placed at the “right” position on the surface, i.e., along the direction of minimal curvature (Koenderink & van Doorn, 1980; Beck & Prazdny, 1981; Fleming, Torralba, & Adelson, 2004; Anderson & Kim, 2009; Kim, Marlow, & Anderson, 2011). Moreover, highlights with simple shapes, as squares or circles, were found to be more effective than complex ones in producing a glossy impression (van Assen et al., 2016). 

In 1937, Hunter pioneered the idea of the multidimensionality of glossiness, identifying six classes of gloss that differ in their appearance. In doing so, he laid the groundwork for the perceptual dimensions often used in the subsequent investigations on gloss. Ferwerda et al. (2001) revealed the limitations of the dimensions proposed by Hunter, as being defined a priori. They suggested, instead, a psychophysically based model to predict gloss perception. Via multidimensional scaling, Ferwerda et al. (2001) built a visual gloss space, whose perceptually meaningful axes were contrast and sharpness of the reflected image, for the specific set of conditions and stimuli they used. This gloss space should probably be extended with more dimensions, for extensions of the range of stimuli beyond dielectrics. 

One popular approach to understand material perception from low-level image cues involves image statistics. For gloss perception, it has been proposed that the statistical moments of the luminance histogram of the image, such as the skewness, could be used as predictor (Motoyoshi, Nishida, Sharan, & Adelson, 2007; Sharan, Li, Motoyoshi, Nishida, & Adelson, 2008). However, further researches have demonstrated that the skewness is not enough to explain perceived glossiness (Anderson & Kim, 2009; Kim & Anderson, 2010; Kim, Tan, & Chowdhury, 2016), and it fails to account for the influence of illumination geometry (Olkkonen & Brainard, 2010, 2011). 

Wiebel, Toscani, and Gegenfurtner (2015) found an effect of skewness on glossiness, but only for computer rendered stimuli. When they tested photographs of natural surfaces, the main discriminative statistic for glossiness was the standard deviation of the luminance histogram, a measure for the contrast. Given the wide variety of glossy materials present in the world, Wiebel et al. (2015) proposed the use of photographs of real surfaces as alternative to the time-consuming procedure of computer rendering. 

Here, we aim to extend the study of gloss perception to true-to-life paintings, starting with paintings of grapes. We tested the three highlight features proposed by Marlow and Anderson (2013), but instead of relying on human judgment to estimate them, we developed a new method to semiautomatically compute them from the segmented images of the paintings. Previous research concerning the influence of image cues on material perception have hitherto either used human judgments (Marlow et al., 2012; Marlow & Anderson, 2013) or luminance histogram-based moment statistics (Motoyoshi et al., 2007; Sharan et al., 2008). The former approach appears to be motivated by difficulties designing robust algorithms that capture image properties like coverage, contrast, and sharpness of the highlights (but see also Qi, Chantler, Siebert, & Dong, 2014) for segmentation of highlights based on pixel intensity threshold and pixel wise calculation of the features for the case of rendered surfaces with identical parameters settings). Yet, the drawback of relying on human judgments is that there could be interaction effects (e.g., an object appears glossier, causing the contrast to be perceived higher). Please note that the computation of these features also involves more than only the luminance histogram; in order to determine the coverage, sharpness, and contrast, the spatial characteristics of highlights need to be taken into account too, complicating the computation. Therefore, we propose an intermediate approach where human annotations assist the computation of the image properties. Understanding the effectiveness of this approach will not only answer our specific research questions but may also be useful for other studies concerning the relation between image cues and material perception. 

Method

Glossiness rating experiments were conducted on the cropped (A) and original (B) versions of the stimuli. The first experiment (A) was performed as main experiment, while the second (B) was done to investigate the influence of context of the whole painting on the gloss judgment of grapes. The highlights of the grapes were manually segmented from the images of the paintings. The luminance profile of each segmented highlight was semiautomatically extracted to quantify the features. 

Stimuli

The stimuli used were high-resolution, digital images of 17th century paintings (78 in total), downloaded from the online repositories of several museums (see Supplementary Figure S1 for a numbered list of all the squared cut-outs used for rating experiment A. Each image in the list has the embedded link to the relative museum repository website, where the original images used in experiment B can be found). 

For experiment A, the stimuli consisted of squared cut-outs containing the target bunch of grapes (Figure 1, left). The gloss rating experiment B was conducted using images of the entire paintings (Figure 1, right). The segmentation task was performed using the latter (Figure 1, right). 

Figure 1

 

Left, squared cut-out (experiment A); right, whole painting (experiment B). Abraham Mignon, Still Life with Fruit and Oysters, 1660–1679. Downloaded from the online repository of the Rijksmuseum, Amsterdam.

Experimental set up

For both rating experiments and the highlights segmentation task, the stimuli were presented in a darkened room, on an EIZO LCD monitor (CG277), with built-in self-calibration sensor. To ensure color consistency across the experiments, the monitor was calibrated before each session, using the software “Color Navigator 6” (EIZO, version 6.4.18.4). The brightness level was always set to 100 cd/m² and the color temperature to 5500 K. The interfaces of the experiments were programmed in MATLAB R2016b, using the Psychtoolbox Version 3.0.14 (Brainard, 1997; Pelli, 1997; Kleiner et al., 2007). 

Observers

The two rating experiments were conducted with different groups of participants. Nine observers took part in rating experiment A, using the cut-out stimuli containing the grapes. For experiment B, six observers were asked to rate glossiness for the images of the entire paintings. All participants had normal or corrected-to-normal vision. They were naïve to the purpose of the experiments. They provided written consent prior to the experiment and received a compensation for their participation. The experiments were conducted in agreement with the Declaration of Helsinki and approved by the Human Research Ethics Committee of the Delft University of Technology. 

Procedure rating experiments

Rating experiments A and B differed in that either a part of the painting or the whole was shown (Figure 1), and in the number of material properties rated. For both experiments, the images were presented against a black background. Before starting the experiments, participants went through all the stimuli in order to get an overview of the stimulus range. No time limit was given to complete the task. 

Experiment A

In experiment A the squared cut-outs containing the target grapes were used as stimuli. The rating of gloss was part of a larger experiment. Observers were asked to rate five different attributes on a continuous 7-point scale. Apart from the attribute glossiness, they also rated translucency, bloom, three-dimensionality, and convincingness. The ratings of the other four attributes were not considered in the analysis since they are not relevant for the purpose of the current discussion. Before starting the experiment, a written definition of each attribute was provided to the observers, and their understanding of the meaning of glossiness, translucency, and bloom was verified with a paired comparison test. A pair of photographs of real grapes was shown to the participants to test the three attributes, with one photo having the attribute and one not. Observers were asked to choose which one was glossier (or more translucent/bloomy). They were given feedback on the answers and, if they were able to choose the right option, they could start the experiment. The question presented on the screen was “how [attribute] is this bunch of grapes on average?” The attributes were tested separately in five blocks, in a random order (between and within each block). Altogether the 78 stimuli were rated five times, once for each attribute, resulting in 390 trials per observer. 

In the data analysis, the possible differences in rating due to having rated glossiness as first attribute or as last, i.e., after having seen a certain stimulus for the first time or the fifth time, were tested via interrater reliability analysis. 

Experiment B

Rating experiment B, using the entire paintings, was performed to check the assumption that the context of the painting does not play a significant role on judging the grapes' glossiness. The term “glossiness” was explained to the observers, and their understanding was checked as in experiment A, with the same two-alternative choice test. If multiple bunches were depicted, the researcher indicated to the observer the bunch of grapes in the painting to be rated. The images were presented in a random order to each participant. The rating was done on a continuous 7-point scale. 

Procedure image segmentation

For the segmentation analysis of the stimuli, the full images of the paintings were used. The highlights' segmentation was performed by the first author. On average, 17.64 grapes were segmented from the images of the bunches used in the rating experiments, in order to have a representative set of samples. The segmentation procedure consisted of drawing a polygon around the grape's contour (blue line, Figure 2), followed by another polygon around the outline of the corresponding highlight (green line, Figure 2). The image could be freely zoomed in (up to the pixel level) and out in order to perform the segmentation. 

Figure 2

 

Full painting with one grape and its highlight manually segmented. Abraham Mignon, Still Life with Fruit and Oysters, 1660–1679. Downloaded from the online repository of the Rijksmuseum, Amsterdam.

Procedure computation of highlights' features from the images

Since no conventional method can be found in literature on how to compute the image features that are diagnostic for material properties, we propose a novel approach. We developed a series of functions in Mathematica (version 11.2) for the semiautomated computation of the contrast, coverage, and sharpness of the highlights, which were manually segmented from the images. Although we tried to make the algorithm to extract the highlights' features fully automated, a manual inspection of the luminance profiles was still needed to correct the data. Because of the extremely uncontrolled nature of the stimuli, several factors interfered with the analysis of the images. The major factor was that, since we used photographs of old paintings, cracks on the surface of the paint (visible as dark lines) added noise to the pixel wise analysis of the highlights. 

The coverage was calculated as the ratio between the area of the highlight and the total area of the grape. Sharpness and contrast were derived from the luminance profiles of the segmented grapes. We extracted the luminance profile from a cross-section of the segmented grape, centered in the middle of the highlight. The cross-sections covered the width of the grapes and were 3 pixels high; the luminance profiles were averaged over these 3 pixels, smoothing out potential outliers. 

Contrast values were calculated as the Michelson's contrast (Michelson, 1891), taking the maximum and minimum luminance values of the peak profile as shown by the horizontal lines pink and yellow in Figure 3. The choice of the orientation for the extraction of the luminance profile (blue line crossing the highlight on the grape in Figure 3) will be addressed later. 

Figure 3

 

Illustration of the values extraction from a luminance profile for the computation of Michelson's contrast and of the slope on the two sides of the peak. The horizontal lines pink and yellow show how the minimum and maximum values of the highlight's peak were extracted, whereas the oblique lines green and red show the computation of the maximum derivative. Note that the x axis shows the pixel width of the grape from the original image, but for the data analysis all the values were normalized for the pixel width of the grape shown on the screen.

Instead of sharpness we considered the inverse, which we named the blurriness and quantified as  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}{\rm{Blurriness}}={\left( {{{\Delta {\rm{y}}} \over {\Delta {\rm{x}}}} \cdot {{1} \over {\Delta {\rm{y}}}}} \right)^{ - {1}}}=\Delta {\rm{x}}\end{equation}

where Δy is the difference between the maximum and minimum luminance values of the peak and Δy/Δx corresponds to the maximum derivative, taken and averaged over the two sides of the highlight profile (oblique red and green lines in Figure 3). The Δx values were normalized to the visual size of the grapes shown on the screen during the rating experiment A.  

Δx represents the transition area from the background (diffuse scattering) to the highlight (specular reflections). The relationship between Δx and blurriness is illustrated in Figure 4. Δx increases with the blurriness, and thus it is inversely related to sharpness. Throughout the rest of the paper, we will refer to blurriness instead of sharpness. Figure 4 also shows that changing the contrast does not affect Δx. 

Figure 4

 

Illustration of the relationships of Δx with blurriness and contrast. For the sharp circles, the luminance profile shows a step, with Δx = 0. As blurriness increases, the transition becomes more gradual and Δx increases. Δx is not dependent on the contrast.

Because of the irregular shapes of the highlights, the luminance profiles were extracted at 36 different angles, between 0° and 175°, in steps of 5°. The results were averaged over the different angles. Two examples of luminance profiles acquired at 0° and 90° are shown in Figure 5. 

Figure 5

 

Example of a segmented grape and its highlight. The luminance profile above was taken at 0°, the one below at 90°.

The highlights do not only show irregular shapes, but many also show an internal spatial structure. They were often rendered as a window reflection (van Assen et al., 2016). Thus, the inner structure of the window, visible in the reflection, constitutes an additional term of variation in the luminance profile, depending on the angle of computation. This is evident in Figure 5. Extracting the profile either perpendicular or parallel to the internal line of the window drastically changes the shape of the luminance profile. Hence, in some cases the maximum derivative is detected in the middle of the highlight instead of at the outer edges. We assume that the visual system detects the sharpest edges and they do not necessarily need to be the outer ones. 

Finally, the features' values for each bunch of grapes were obtained from the average of the segmented grapes, analyzed as just described. 

Results

Glossiness rating experiments

As mentioned before, the gloss rating experiment was performed once using the squared cut-outs containing the target grapes, and again with the entire images of the paintings. Before comparing the data, we analyzed the internal consistency of the ratings in experiment A. Here, glossiness was rated as part of a larger experiment in which observers were asked to judge also four other attributes, in random order. Their evaluation of glossiness may thus have been biased by the order of the attributes and the number of times the stimuli were seen before rating glossiness. A reliability analysis resulted in a Cronbach's alpha coefficient of 0.83, demonstrating high consistency in the ratings. A Spearman rank test was also performed, showing that all the observers' data were significantly correlated (p < 0.05) with each other. Nevertheless, the reliability analysis between observers of experiment B, who rated only glossiness of the grapes seeing the entire paintings, gave a Cronbach's alpha coefficient of 0.97. T test showed that the two Cronbach's alpha values are significantly different (p < 0.05). This may indicate that increasing the number of material properties to be rated decreased the agreement between observers, but gloss ratings from experiment A are still reliable. 

To minimize possible effects of unequal interval judgments, the data of each observer for both rating experiments were rescaled from the 7-point scale to the 0–1 range before averaging. The average gloss ratings of experiments A and B were correlated, in order to test possible effects of the painting context on the judgment. The trend of the correlation is shown in Figure 6. The ratings resulted in a strong and significant correlation (r = 0.74, p < 0.001). The regression line that best fit the data gives an offset of −0.02 nonsignificantly different from 0, but a slope of 0.96 significantly (p < 0.05) different from 1. This means that the participants of experiment A perceived a wider range of glossiness levels than the ratings used by participants in experiment B. Such systematic effect of the slope may be due to the grapes' bunch size shown in the two experiments. They were clear and close-up in the cut-outs (A), but small when shown in the entirety of big paintings (B). The values of the mean gloss ratings for the two experiments and their standard deviations are reported in Supplementary Table S2. 

Figure 6

 

Scatterplot of the correlation between the average glossiness ratings of the squared cut-outs (experiment A) and of the whole paintings (experiment B). Results show r = 0.74, p < 0.001; the area around the fit line represents the 95% CI.

In Figure 7, a bar chart shows average ratings from experiment A for the three stimuli judged most and the three judged least glossy. We do not know the ground truth of the glossiness levels of the painted grapes, but since the average minimum and maximum levels were more than 0.6 apart, whereas random data would have shown both the minimum and maximum around 0.5, we can conclude that the ratings were internally consistent and the stimuli obviously cover a perceptual well distinguishable range. 

Figure 7

 

Mean ratings for experiment A (cut-outs) of the three bunches of grapes judged most glossy and the three least glossy. The corresponding images are shown below each bar. The error bars indicate the standard errors of the mean.

Glossiness prediction based on the segmented highlights' features

Using the image processing technique described in the method section, we quantified the features of the segmented highlights. To explore the relationships between the features contrast, blurriness, and coverage and the perceived glossiness, we employed principal component analysis (PCA) and multiple linear regression. In the PCA biplot, shown in Figure 8, we can see how the scores, i.e., the images (numbered points; see Supplementary Figure S1 for the image corresponding to each number) were distributed with respect to the variables. The variables represent the three highlight features and the mean gloss rating from experiment A (a PCA biplot representing the relationships between only the three highlight features is shown in Supplementary Figure S2, with the corresponding factor loadings in Supplementary Table S1). To account for the different scales of the variables, we performed the PCA based on the correlation matrix. 

Figure 8

 

PCA biplot showing the scores (images) distribution with respect to the variables (highlights' features and gloss ratings), and the relationships between the variables themselves.

The first two principal components together account for 83.4% of the variance. From the factor loadings (Table 1), we see that the first component is strongly loaded by contrast and perceived gloss in one direction and by blurriness in the opposite direction. This means that glossiness varied positively with contrast and negatively with blurriness. The correlation between perceived gloss and contrast is indeed positive and significant with r = 0.80, p < 0.001, and it is negative and significant between perceived gloss and blurriness with r = −0.61, p < 0.001. On the second component, the variable with the highest loading is coverage. This suggests that coverage was not correlated with glossiness, and indeed r = 0.03, p > 0.05 between glossiness and coverage. Correlation plots for each highlight feature with perceived glossiness are shown in Supplementary Figure S3, and the corresponding values of the average gloss rating and highlight features are reported in Supplementary Table S2. 

Table 1

 

Factor loadings of the first two principal components.

 

To predict the perception of glossiness based on the highlight features, we used multiple linear regression. We found the best fit (Equation 2) for a model carrying contrast and blurriness as significant (p < 0.001) predictors.  

\begin{equation}\tag{2}{\rm{Perceived\ gloss}}= {\rm{0.32}}+{\rm{1.1\ Contrast}} - {\rm{2.05\ Blurriness}}\end{equation}

 

This model explains (r²) 69% of the variance of perceived glossiness. 

Discussion

One aim of the study presented in this paper was to test whether the diagnostic power of highlight features proposed by Marlow and Anderson (2013) could be transferred from computer rendered to painted stimuli. We therefore first measured the perception of glossiness of grapes in bunches, extracted as squared cut-outs from the images of the paintings. Alongside this rating experiment, a second experiment was performed, showing to the observers the entire images of the paintings in order to test whether the context influences the perceived glossiness of the grapes. The strong and significant correlation, and the lack of systematic effect of the fit offset that we found for the average ratings of the two experiments, show that a potential influence due to the context was not critical. However, the systematic effect of the slope indicates that a wider range of ratings was used for the cut-outs compared to the entire paintings. The different sizes of the bunches of grapes shown in the two experiments may have caused this effect. In the cut-outs of experiment A, they were all shown with similar, close-up sizes; thus, smaller variations of glossiness image cues may have been more visible. 

To measure the three highlight features (contrast, blurriness, and coverage), we segmented the grapes from the images and computed the features via image analysis. Contrast and blurriness were found to be the predictors for the best fit model of gloss perception, accounting for 69% of the explained variance. The amount of explained variance (r²) given by Marlow and Anderson (2013) for their set of experiments ranged between 0.91 and 0.97. We cannot make a direct comparison with their r² values because of the fundamental difference with our stimuli and for the method used to quantify the highlight features. However, we assume that the main explanation for our lower r² can be imputed to the uncontrolled nature of the paintings. As a future step, the algorithms we used to quantify the highlight features should be correlated to their perceptual measures (i.e., via human estimation of the cues), in order to validate the psychophysical relevance of our method. 

Contrast and blurriness were found to be the main contributors regarding gloss rendering of grapes in paintings, as shown by the high correlation between the variables in the PCA biplot (Figure 8). Dutch 17th century painters may have been aware of the importance of the highlights' contrast and may have intentionally emphasized such feature by placing a dark line or area along the highlight contour as a pictorial trick (Figure 9). The importance of contrast for rendering glossiness in paintings is also confirmed by the findings of Cavanagh et al. (2008). They found that in paintings the only requirements for highlights on curved surfaces are to be brighter than the surrounding and to be appropriately curved. 

Figure 9

 

Details of two bunches of grapes used as stimuli showing an example of the use of dark lines around the contour of some of the highlights. Left: Nicolaes van Gelder, Still Life, 1664. Right: Pieter de Ring, Still Life with Golden Goblet, 1640–1660. Downloaded from the online repository of the Rijksmuseum, Amsterdam.

Blurriness had the expected negative correlation with gloss perception. For coverage we found no significant effect. This is comparable with what Marlow and Anderson (2013) reported for rendered spheres. They did find an effect of perceived coverage on glossiness, but this effect and the perceived variation of the highlights' coverage were the lowest compared to perceived contrast and sharpness. If the light source has one main direction, only a small part of a spherical surface will be covered by highlights, because a sphere has a uniform distribution of surface normals. Grapes are spherical (or ellipsoidal) objects, and in still life paintings it was common practice to suggest the presence of a single source of light coming from a window, usually placed top left (Mamassian, 2008). Thus, the coverage is rather small and constant throughout the various paintings and the different levels of glossiness. 

From previous works it is clear that the research on gloss perception cannot be reduced to the highlights only, since the appearance of the highlights is influenced by other factors like the illumination field (Fleming et al., 2003; Pont & te Pas, 2006; Zhang et al., 2015; Wendt & Faul, 2017) and the 3D shape of the object (Vangorp et al., 2007; Ho et al., 2008; Marlow & Anderson, 2015). However, it is also known that painters often abstract the rules of physics into an “alternative physics” (Cavanagh, 2005), which allows portraying just the key information for an efficient recognition of the scene, leaving errors and incongruences unnoticed at first glance. 

This is, for example, the case for the congruency between the orientation of the highlights and of the grapes' shapes. It is well known in literature that one of the fundamental requisites for highlights is to be placed at the “right” position on the surface (Koenderink & van Doorn, 1980; Beck & Prazdny, 1981; Fleming et al., 2004; Anderson & Kim, 2009; Kim, Marlow, & Anderson, 2011). Still, when we measured the orientation of an ellipse fitted onto the highlight and that of an ellipse fitted on the grape, we did not find a correlation for the set of bunches perceived as highly glossy. The orientations were found to be more congruent instead (r = 0.57, p < 0.001) for the medium to low glossy grapes. This finding contradicts the literature as well as the physics. Figure 10 shows on the left a photo of a real bunch of grapes and on the right one of the painted bunches considered among the glossiest. In the photo, each grape has its own orientation, as indicated by the black arrows, and their highlights are always coherently aligned (red arrows). The painting, on the other hand, shows visible incongruences. Nonetheless, such inaccurate orienting of the highlights does not seem to hinder the perception of glossiness, nor improve it when they are more coherently aligned on the low and medium glossy grapes. Another discrepancy between the laws of physics and the “physics of paintings” concerns the elongation of the highlights' shape with respect to the distance of the highlight from the center of the grape, which is related to the slant angle of the light direction. Assuming a spherical shape for the grapes, we calculated the highlights' position. We retrieved the light direction as the tilt and the slant angle. With an average of 143° for the tilt angle and of 51° for the slant angle (Figure 11), we could confirm the top-left convention for the illumination orientation, which is a well-known perceptual prior (Mamassian & Goutcher, 2001; Morgenstern, Murray, & Harris, 2011), also found in paintings (Mamassian, 2008; Carbon & Pastukhov, 2018; Wijntjes, 2018). We found that the highlights' elongations were not consistent with the slant angles of the illumination. Nevertheless, this did not influence gloss judgments throughout our stimulus set, as no correlation was found. 

Figure 10

 

Left: Photo of a real bunch of grapes taken by the authors in the lab. Right: A bunch of grapes considered among the glossiest of our set of stimuli (Pieter de Ring, Still Life with Golden Goblet, 1640–1660. Downloaded from the online repository of the Rijksmuseum, Amsterdam). The black arrows indicate the orientation of the grapes while the red arrows show the orientation of the highlights. The arrows were drawn by hand. In the photo, the grapes are oriented differently, but each highlight follows the shape orientation of the grapes; in the painting, the orientations of the highlights appear to be randomly scattered across the bunch, and they are not consistently congruent with the grapes' shape.

Figure 11

 

Polar plot of the average tilt and slant angles. The plot shows the top-left convention of the light orientation used in paintings.

We found that breaking the rules of the orientation congruency and of the elongation of highlights with the light slant do not affect glossiness perception. We assume that this is the case, because the highlights' contrast has the predominant effect in our set of stimuli. 

As will be discussed later, the artistic conventions, including the recipes given by Beurs (1692/in press), state to use white to render the highlights on grapes. This can be an example of the above-mentioned key information representing statistical regularities of real scenes and transferred to the canvas by the painter. In fact, grapes are dielectric materials, so they have specular highlights of the same color as the light source (Klinker, Shafer, & Kanade, 1990; Nishida et al., 2008). Measuring the chroma of the segmented highlights, we found a significant negative correlation with glossiness (r = −0.35, p < 0.01), which means that the more colored the highlights are (which can also be due to ageing and yellowing of the painting), the less glossy the grapes will be perceived. 

One of the next steps would be to include the perceptual attribute of “haze gloss” (Hunter, 1937; Vangorp, Barla, & Fleming, 2017) to the representation of the glossy appearance of grapes. Grapes are naturally covered by bloom, a waxy coating that looks like a whitish matte layer. Usually, it is not evenly spread over the fruit surface, since it can be easily deleted by handling or transportation, and it can also have various thickness, but in general the more bloom is present, the less glossy the fruit appears (Mukhtar, Damerow, & Blanke, 2014; Loypimai, Paewboonsom, Damerow, & Blanke, 2017). However, the highlight can be also placed next to a highly bloomy area, making the role of bloom in tuning gloss perception far from trivial. 

Our findings on the use of the highlights' features to render glossiness of grapes in 17th century painting practice are supported by the painting manual of Beurs (1692/in press). In his recipe for grapes, no instruction can be found on how much of the fruit surface should be covered with highlights. He may not have mentioned it, either because experience and observation would have been enough to get this notion, or because, as we found, the coverage has no significant role in the case of grapes. 

The recipe contains less ambiguous indications for what concerns contrast and blurriness. It states that the highlight should be placed where the surface is not covered with bloom, and it should be painted white. In the area where no bloom is present, the skin color of the grape is visible. Applying a white spot on a colored background mainly affects the contrast. For blurriness, Beurs (1692/in press) specified that care should be taken, when applying the white highlight, to “gently blend it in.” He referred to the edges of the reflection, blending the white of the specular reflection with the color of the diffuse body scattering, resulting in more gradual edges. Since grapes are not mirror-like materials, this procedure would increase the natural appearance of the fruit and thus its convincingness. 

It would be interesting to apply the model, having contrast, blurriness, and coverage of the highlights as predictors, to other glossy materials depicted in paintings, and see whether the contributions of the predictors change. 

Lastly, we showed that it is possible to extract image cues by manually indicating the highlight and the contour of the grape. Using this input, highlight profiles can be generated that contain information about contrast, blurriness, and coverage. To our knowledge, this approach is relatively new and seems a valuable addition to research on visual material cues. Until now, research has either focused on the physical parameters (that lead to the image cues, e.g., Ferwerda et al., 2001), global image statistics (Motoyoshi et al., 2007) or human estimates of cue strength (Marlow & Anderson, 2013). Almost all of these studies were performed on well-controlled computer rendered stimuli. Although our stimuli are clearly also artificial, they are uncontrolled. Our approach can be readily generalized to “natural images,” like the Flickr Material Database (FMD) (Sharan, Rosenholtz, & Adelson, 2014). 

Conclusions

We have measured the amount of glossiness perceived in paintings of grapes from the Dutch Golden Age, a period characterized by the detailed realistic imitation of nature. We have predicted perceived glossiness using the key features of the highlights, which can be observed in the image (Marlow et al., 2012; Marlow & Anderson, 2013). The novelty of our work consisted in the use of uncontrolled stimuli and in the method we have used to measure the features. Contrast, coverage, and blurriness were mathematically defined, and calculated directly from the segmented stimuli. 

Contrast and blurriness were found to be the main predictors for gloss perception. Coverage, on the other hand, was found to have no influence at all. We could find hints to the same conclusions in the painting instructions for grapes given by Beurs (1692/in press). We also found support for the idea that painters used to sacrifice the true physics of light and instead use key factors of the optical phenomena, and that does not affect glossiness perception (or perhaps even enhance it). 

We have shown that the research on gloss perception can be extended to paintings, and eventually also to the study of historical sources. Via image analysis, we have demonstrated that two of the three cues proposed by Marlow and Anderson (2013), were used by 17th century painters to elicit gloss perception of grapes. 

Acknowledgments

This work is part of the research program NICAS “Recipes and Realities” with project number 628.007.005, which is partly financed by the Netherlands Organization for Scientific Research (NWO) and partly by Delft University of Technology. Maarten Wijntjes was financed by the VIDI project “Visual communication of material properties,” number 276.54.001. 

Commercial relationships: none. 

Corresponding author: Francesca Di Cicco. 

Email: . 

Address: Perceptual Intelligence Lab, Faculty of Industrial Design Engineering, Delft University of Technology, Delft, the Netherlands.