How direction of illumination affects visually perceived surface roughness

Abstract

We examined visual estimation of surface roughness using random, computer-generated, three-dimensional (3D) surfaces rendered under a mixture of diffuse lighting and a punctate source. The angle between the tangent to the plane containing the surface texture and the direction to the punctate source was varied from 50 to 70 deg across lighting conditions. Observers were presented with pairs of surfaces under different lighting conditions and indicated which 3D surface appeared rougher. Surfaces were viewed either in isolation or in scenes with added objects whose shading, cast shadows, and specular highlights provided information about the spatial distribution of illumination. All observers perceived surfaces to be markedly rougher with decreasing illuminant angle. Performance in scenes with added objects was no closer to constant than that in scenes without added objects. We identified four novel cues that are valid cues to roughness under any single lighting condition but that are not invariant under changes in lighting condition. We modeled observers' deviations from roughness constancy as a weighted linear combination of these “pseudocues” and found that they account for a substantial amount of observers' systematic deviations from roughness constancy with changes in lighting condition.

Introduction

Roughness is a surface property that can be used to describe materials or to discriminate them, and it can be critical in the classification of materials. For example, Figure 1 shows sandpapers of different “grits.” One can readily discriminate the different grits both visually and haptically. 

Figure 1

 

Surface roughness. These are photographs of pieces of sandpaper of varying grit painted the same shade of gray. Illuminant direction is the same for all images. Notice that these sandpapers vary in perceived roughness.

For our purposes, roughness is an aggregate or statistical measure of the shape, size, and distribution of elements on a surface. It may be classified haptically (Lederman & Klatzky, 2004) or visually (Rao & Lohse, 1993, 1996). Haptic psychophysical data indicate that perceived haptic roughness increases with an increase in the scale of surface elements. Parameters that affect haptic roughness include interelement spacing and grating groove width (Klatzky & Lederman, 1999; Sathian, Goodwin, John, & Darian-Smith, 1989). Visually perceived roughness likely depends on the size and spacing of the many elements of which a textured surface (e.g., sandpaper) is composed. We are interested in the visual assessment of roughness in classes of surfaces similar to sandpaper or stucco. We refer to these surfaces as three-dimensional (3D) textures. 

Unlike 2D textures, the image of 3D textures changes with illumination and viewpoint conditions. Variations in the appearance of different samples of 3D texture can be captured by the Bidirectional Texture Function of Dana, van Ginneken, Nayar, and Koenderink (1999), a function that summarizes the effects of changes in illumination and viewpoint conditions. Here, we examine how visual estimates of roughness, a property inherent to 3D textures, depend on illumination. 

Perception of surface roughness

Haptic perception of roughness in 3D textures is evidently unaffected by changes in illumination conditions and, if visual perception of the roughness of 3D textures were well calibrated to haptic perception of the same textures, we might expect a high degree of roughness constancy across changes in lighting conditions. However, there are also good reasons to expect otherwise. 

First, it is plausible that visual perception of 3D texture is closely related to shape perception, which is not invariant under changes in lighting conditions. There is evidence that the perception of surface relief (Belhumeur, Kriegman, & Yuille, 1999; Berbaum, Bever, & Chung, 1983; Oppel, 1856; Yonas, 1979) and shape estimated from shading (Koenderink, van Doorn, Christou, & Lappin, 1996a, 1996b; Langer & Bülthoff, 2000) are far from invariant under changes in lighting conditions. 

Second, inspection of monocular images corresponding to a 3D texture suggests that perceived roughness varies markedly with lighting conditions. Figure 2 shows computer-rendered images of a monochrome matte surface viewed under two lighting conditions. On the left, the lighting is primarily diffuse; on the right, it is a mixture of diffuse and punctate illumination. Most observers would agree that the surface shown on the right appears rougher than the one on the left. 

Figure 2

 

Rough surfaces. The same faceted surface is shown rendered under two different illuminants. One contains a primarily diffuse illuminant (left) and the other, a diffuse plus a punctate illuminant (right). Note that the images differ substantially; in particular, the right-hand surface appears somewhat rougher.

However, under binocular viewing conditions, the viewer has binocular disparity cues to roughness that are little affected by changes in lighting condition. It is plausible that observers would exhibit a higher degree of roughness constancy with binocular viewing. 

Moreover, recent work examining surface color perception suggests that the constancy of judgments of surface color and lightness perception increases when the scene contains cues, such as shadows and specular highlights, that signal the spatial and spectral distribution of the illumination (Boyaci, Doerschner, & Maloney, 2004; Boyaci, Maloney, & Hersh, 2003; Doerschner, Boyaci, & Maloney, 2004; Ripamonti et al., 2004; Yang & Maloney, 2001). Such judgments improve as more cues to the illumination conditions are provided (Boyaci, Doerschner, & Maloney, 2006; Kraft, Maloney, & Brainard, 2002; Yang & Maloney, 2001). 

We conjecture that failures of roughness constancy observed when a surface is judged in isolation may be due to lack of information about illumination geometry. Koenderink, van Doorn, Kappers, te Pas, & Pont (2003) suggest that cues available in images of surfaces are not sufficient to accurately estimate the azimuth and elevation of a collimated light source. In particular, they found that, for obliquely illuminated textured surfaces, observers could estimate the azimuth of the source to within about 15 deg (if we ignore frequent 180 deg confusions), whereas estimates of elevation were barely better than chance. We know that these ambiguities are almost completely resolved in real scenes containing other objects and landmarks. By adding context to the scene, we could thereby provide more cues—such as shadows and specular highlights—to illumination conditions. This could lead to a greater degree of roughness constancy. 

Preview

In this study, we examine perception of the roughness of 3D textures, binocularly viewed, under changes in the direction of a punctate illuminant. Our goal is to investigate what factors in the scene determine failures in roughness constancy with changes in lighting conditions. We compare conditions where only the 3D texture is available to conditions where we have additional illuminant cues. 

Methods

Stimuli

Coordinate systems

The stimuli were defined using a Cartesian coordinate system ( x,y,z) ( Figure 3). The origin (0,0,0) is in the frontoparallel plane on which we present stimuli (the stimulus plane). The Z-axis lies along the observer's line of sight. The X-axis is horizontal and the Y-axis is vertical. The punctate illuminant direction is represented as a vector P using spherical coordinates ( ψ,ϕ,d). We define elevation ϕ as the angle between P and the projection of P on the XY-plane; it ranges from 0 to 90 deg. The azimuth ψ is the angle between the projection of P onto the XY-plane and the X-axis. 

Figure 3

 

Coordinate systems. A Cartesian coordinate system was used in defining surface patches. The Z-axis lies along the line of sight from the observer to the origin such that the observer's gaze is in the − Z direction. The X-axis lies along the horizontal line tangential to the surface, and the Y-axis lies along the vertical line tangential to the surface. For the punctate illuminant positions, spherical coordinates were used. The illuminant had an elevation ϕ (relative to the surface) and azimuth ψ, and it was located at a distance d from the surface patch. In this study, ϕ was varied, whereas ψ and d were fixed at 180 deg and 80 cm, respectively. The origin of the coordinate system was the center of the surface patch. The observer was always at ( x,y,z) = (0,0,70 cm) or, in spherical coordinates, ( ψ,ϕ,r) = (90 deg,–,70 cm).

Surface patch

Our 3D textured surfaces were composed of randomly oriented Lambertian facets. Figure 4 illustrates how surface patches were constructed. First, an N × N grid of base points was generated in the stimulus plane with width w. The base point coordinates are denoted ( X _ij ,Y _ij ,Z _ij), 0 ≤ i, j ≤ N − 1. Let U _ij ^x , U _ij ^y , and U _ij ^z be independent, uniformly distributed random variables on the interval [−1,1]. The base point coordinates were defined as  

$\begin{array}{c}{X}_{\mathrm{ij}}=-\frac{w}{2}+(j+n_{\mathrm{xy}}{U}_{\mathrm{ij}}^x)\delta \hfill \\ Y_{\mathrm{ij}}=-\frac{w}{2}+(i+n_{\mathrm{xy}}{U}_{\mathrm{ij}}^y)\delta \hfill \\ Z_{\mathrm{ij}}=r{U}_{\mathrm{ij}}^z,\hfill \end{array}$

(1)

where w = 19 cm, δ =

$\frac{w}{N-1}$

, and the grid has N = 20 points on a side. Setting n _xy = 0.49 ensured that no facets would overlap or intersect one another in the jittered base grid. The amount of jitter in depth could take on any of eight distinct values, r = k ²/16 cm, depending on the roughness level k = {1,2,…,8}. The standard deviation of the Z _ij coordinates in a surface with roughness level r was

$\sqrt{3}$

k ²/12 cm. A flat surface, r = 0, would thus mean all Z _ij = 0. The Z _ij coordinates were always 4 cm or less in absolute value. Note that the spacing of the standard deviation of successive roughness levels is quadratic in r. In initial testing, we found that linear spacing led to stimuli that were difficult to discriminate at high roughness levels, and we consequently adopted quadratic spacing. The 3D surface was constructed from triangular facets. Each set of four neighboring grid points ( i, j), ( i + 1, j), ( i, j + 1), and ( i + 1, j + 1) was split into two triangular facets by randomly selecting one of the two diagonals to be connected by an edge. For each value of roughness, illuminant elevation, and context condition (see Illuminant cues section), four random surfaces were generated to minimize the possibility of observers using patterns in the distribution of facets as cues to roughness. 

Figure 4

 

Rough surface model. (a) A 19 × 19 cm regular grid of 20 × 20 base points was constructed. (b) The base points of the grid were jittered randomly in the XY-plane. (c) Uniform random deviations were added to the base points in the Z direction. (d) The surface that was composed of 3D jittered base points was then divided into triangular facets by randomly splitting each 2 × 2 set of base points along a diagonal.

Light sources

Each surface patch was rendered under a diffuse plus punctate illuminant using the RADIANCE rendering software package (Larson & Shakespeare, 1996; Ward, 1994; http://radsite.lbl.gov/radiance/HOME.html). The surface facets were Lambertian with albedo 0.5. They were rendered with interreflections (one ambient bounce) as well as occlusions and vignetting. All surfaces were oriented frontoparallel to the observer. The punctate illuminant had azimuth ψ = 180 deg (light came from the left) and elevation ϕ = 50, 60, or 70 deg (Figure 5). It was located 80 cm away from the surface, and the punctate-total ratio was 0.62. ¹ 

Figure 5

 

Punctate illuminant directions. Three illuminant elevations ( ϕ = 50, 60, and 70 deg) were used in this study.

To prevent cues that may result from an abrupt change in depths of the edges of the surface patch and the wall, we multiplied a 3.6-cm border around the surface patch by a raised cosine function to smooth the edges of the surface in depth. A “woodgrain”-patterned floor was present in both the control and experimental conditions to provide more cues to depth. 

Illuminant cues

There were two context conditions. In Condition I ( Figure 6, top row), the rough surface patch was embedded in a flat wall and the only other visible surface was a textured floor. In Condition II ( Figure 6, bottom row), additional objects were added to the scene, including cubes, prisms, and cylinders resting on the floor, as well as floating spheres. These objects had varying colors, degrees of specularity, and positions in the scene. Each scene was then rendered from two different viewpoints (±3 cm) and viewed binocularly. A representative set of surface patches with varying roughness and illuminant elevation is shown in Figure 7. 

Figure 6

 

Stereo pair of stimuli. Two example stimuli of roughness level r = 2.25 are shown corresponding to Condition I (top row) and Condition II (bottom row). The two images on the left can be used for crossed fusion, and the two images on the right can be used for uncrossed fusion.

Figure 7

 

Examples of stimuli. For each roughness level, four surfaces were generated. Here, one of the four is shown for each roughness level r rendered under each of the three punctate illuminant directions.

Apparatus

The left and right images were presented to the corresponding eye of the observer using two 21-in. Sony Trinitron Multiscan GDM-F500 monitors in a mirror stereoscope ( Figure 8). The screens on these monitors are close to physically flat, with less than 1 mm of deviation across the surface of each monitor. Lookup tables were used to correct display nonlinearities and to equate the luminance on the two displays based on measurements of the luminance values on each monitor made with a Photo Research PR-650 spectrometer. The maximum luminance achievable on either screen was 114 cd/m ². The stereoscope was contained a box with the front face of the box missing wherein the observer was seated with his/her chin placed in a chinrest. The interior of the box was coated with black flocked paper (Edmund Scientific, Tonawanda, NY) to absorb stray light. Only the stimuli on the screens of the monitors were visible to the observer. The casings of the monitors and any other features of the room were hidden behind the nonreflective walls of the enclosing box. 

Figure 8

 

The experimental apparatus. Stimuli were displayed using a Wheatstone stereoscope. The left and right images of a stereo pair were displayed on the left and right monitors. Observers viewed them via small mirrors placed in front of their eyes. In the fused image, the surface patch appeared approximately 70 cm in front of the observer. The distance was also the optical distance to the screens of the two computer monitors, minimizing any mismatch between accommodation and other distance cues.

Additional light baffles were placed near the observer's face to prevent light from the screens reaching the observer's eyes directly. The optical distance from each of the observer's eyes to the corresponding computer screen was 70 cm. To minimize any conflict between binocular disparity and accommodation, we rendered the stimuli to be 70 cm in front of the observer. The monocular fields of view were 55 × 55 deg. The observer's eyes were approximately in line with the center of the scene being viewed. 

The experimental software was written in the C programming language using the X Window System, Version 11R6 (Scheifler & Gettys, 1996), running under Red Hat Linux 6.1 for graphical display. The computer was a Dell 410 Workstation with a Matrox G450 dual head graphics card and a special purpose graphics driver from Xi Graphics, which permitted a single computer to control both monitors. The monitors were synchronized by a common signal from the Matrox board. The rendered stereo image pair was represented by floating point RGB triplets for each pixel of the image. These triplets were the relative luminance values of each pixel. We translated these values to 24-bit graphics codes, correcting for nonlinearities in the monitors' responses by means of lookup tables. 

Procedure

We used a two-interval forced-choice discrimination paradigm in which a test surface patch and a match surface patch were presented, and the observer's task was to indicate which patch appeared to be rougher. The test, under one illuminant, was chosen from the intermediate range of roughness levels (0.25 ≤ r ≤ 3.06 cm, see Figure 7), whereas the match was under a different illuminant and could have any of the eight roughness levels. 

Figure 9 illustrates the sequence of events in a trial. The observer first saw a gray surface containing a central fixation point and four flanking black dots for 500 ms at the same depth at which the patch would appear. The flanking dots were included to help observers maintain fusion. Then, a test or match stimulus appeared for 1,000 ms in the first interval of each trial. The fixation stimulus reappeared during the 500-ms interstimulus interval, and then a second test or match stimulus was presented for 1,000 ms in the second interval. Observers responded by pressing either the right or left mouse button for the corresponding interval that they believed to contain the rougher patch. The next trial was presented immediately after a response was made. The test stimulus was randomly displayed in the first or second interval, and the match was displayed in the other. For each type of comparison (test patch roughness level and test and match illumination conditions), two types of staircases were used (“two-up, one-down” and “one-up, two-down”) to determine the roughness level of the match surface. Points of subjective equality (PSEs; i.e., points at which the match was perceived rougher than the test 50% of the time) were estimated by combining the data from the two types of adaptive staircases. This resulted in a total of 72 interleaved staircases (6 test roughness levels × 3 illuminant comparisons × 2 staircase types × 2 context conditions). Each observer completed 20 trials per staircase. These trials were split into two sessions; the staircases were continued across sessions. 

Figure 9

 

Trial sequence. The observer performed a two-interval forced-choice task in which a test surface appeared in one interval and a match surface appeared in the other interval. When no stimulus was displayed, observers viewed a central fixation point flanked by four dots positioned on a surface that appeared with the same disparity as the stimulus surface so that observers could maintain vergence at the appropriate distance. Observers indicated which patch appeared rougher. The next trial was presented after a response was made. Observers were permitted as much time as necessary to make a response.

Observers

Seven observers participated. Three observers were naive; four (Y.X.H., M.S.L., C.P., and J.F.) were aware of the purpose of the experiment. All observers had normal or corrected-to-normal vision. 

Results

A representative set of psychometric functions collected for one observer is shown in Figure 10. PSEs were obtained by fitting the data with a Weibull distribution and estimating the point at which there was a 50% probability of choosing the match surface as rougher. 

Figure 10

 

Psychometric functions. A representative set of psychometric functions obtained for one observer for the comparison of a test surface patch under illuminant elevation ϕ = 70 deg with a match surface patch under ϕ = 50 deg. The data were fit with a Weibull distribution (black line), and the point at which the match surface appeared rougher than the test surface 50% of the time (PSE) was estimated from a total of 40 trials per psychometric function (blue circles).

Results for all observers are shown in Figures 11 and 12. The 95% confidence intervals for each PSE were obtained by a bootstrap method (Efron & Tibshirani, 1993) whereby each human observer's performance in the corresponding condition was simulated 1,000 times and the 5th and 95th percentiles were calculated. 

Figure 11

 

Results for Condition I. PSEs are plotted for each pair of illuminant conditions for seven observers. The dashed line represents the results expected of a roughness-constant observer. The solid lines are linear fits to the data predicted by the roughness discrimination model. Of the 21 slopes predicted by the linear model, 17 were significantly different from 1 (at the Bonferroni-corrected α level of .007 per test). Error bars represent bootstrapped 95% confidence intervals. (♦ indicates a fit that is not significantly different from the line of constancy.)

Figure 12

 

Results for Condition II. Details as in Figure 11. Of the 21 slopes predicted by linear model, 18 were significantly different from 1 (at the Bonferroni-corrected α level of .007 per test). (♦ indicates a fit that is not significantly different from the line of constancy.)

If observers were roughness constant across changes in illumination, then their true PSEs should lie along the line of roughness constancy (i.e., the identity line) and the measured PSEs should show no patterned deviation from the line. Almost all PSEs fell below the line of roughness constancy. This trend strongly suggests that a surface appears rougher when illuminated from a more grazing angle. 

Additional contextual cues do not improve roughness constancy

Surprisingly, we did not find any significant difference between Conditions I and II (Bonferroni-corrected α level for seven tests [ α ≅ .007] for a z test of difference of slopes for the three illuminant comparisons). Thus, the additional cues to the illuminant direction provided in Condition II did not improve observers' judgments of roughness across varying illumination conditions ( Figure 13). 

Figure 13

 

Results for Condition I compared with Condition II. Estimates of the PSE slopes from Condition I are plotted versus those from Condition II for each observer. Each data point represents a comparison of all surfaces with varying roughness under two different illuminant conditions (e.g., ϕ = 70 vs. 50 deg) for a given observer. The dotted line represents the lines of roughness constancy along which points representing slope values of 1 would lie. The dashed line represents the identity line along which slope values for Conditions I and II would be equal to one another. Notice that almost all points lie inside the area inscribed by the lines of roughness constancy, indicating that almost all PSE slopes were less than 1. Furthermore, all points lie roughly along the identity line.

Given this unexpected outcome, we sought to verify whether the illuminant directions that we used were discriminable. In an experiment using identical methodology and only Condition II stimuli, one subject (Y.X.H.) readily identified the larger illuminant elevation, for a 10 deg difference between illuminant elevation angles, 93% of the time. We cannot explain the lack of effect of the additional cues to illuminant direction provided in Condition II as simply a failure of discrimination. 

A model of roughness discrimination

Most PSEs fell below the line of roughness constancy. Thus, it seems that observers failed to display roughness constancy across varying illuminant directions. We modeled observers' choice data to determine how much of the observers' estimates were altered by the change of illuminant elevation. We assumed that an observer's roughness discriminations were a function of (1) a roughness transfer parameter c (i.e., the slope of a linear fit to the data in Figures 11 and 12), (2) the standard deviation σ of normally distributed noise that causes the variability in observers' judgments, and (3) a noise-scaling parameter γ. We assume that the noise is scaled by r ^γ; thus, if γ = 0, this corresponds to stimulus-independent noise. If γ = 1, this corresponds to Weber's law for our roughness scale r (see 1 for details). We denote the three illuminant conditions by A, B, and C (corresponding to illuminant elevations ϕ = 70, 60, and 50 deg, respectively), resulting in three roughness transfer parameters c _AB, c _BC, and c _AC, plus σ and γ, for a total of five model parameters. We estimated these parameters by maximum likelihood. 

We first tested the hypothesis that the transfer parameters were all equal to 1 for each observer (i.e., roughness constancy). We rejected this hypothesis for 35 of the 42 transfer parameters for all seven observers (slope estimates

$\widehat{c}$

were significantly less than 1 at the Bonferroni-corrected α level for seven tests, α ≅ .007). This outcome is not particularly surprising because we have little reason to expect perfect roughness constancy; however, what is of interest is the magnitude of the failure of roughness constancy across observers. The estimates of the roughness transfer parameter c were 0.78 on average, markedly less than 1. Additionally, most values of γ fell very close to 1 for all observers, suggesting that noise increases in a manner that follows Weber's law with increasing levels of roughness ( Table 1). 

Table 1

 

Estimated roughness discrimination model parameter values for each observer for each condition. Notice that almost all estimated PSE slope values ( c ^) are under 1 and most values of γ ^ are close to 1. Values in boldface indicate a fit that is significantly different from the line of constancy at the Bonferroni-corrected α level of .007.

		Observer
Context condition	Model parameter	C.P.	J.G.	M.F.	M.S.L.	P.J.N.	T.A.	Y.X.H.
I	c ^ 70 deg 60 deg	1.074	0.840	0.826	0.802	0.783	0.688	0.787
	c ^ 60 deg 50 deg	1.037	0.743	0.828	0.991	0.779	0.872	0.757
	c ^ 70 deg 50 deg	0.826	0.599	0.646	0.797	0.582	0.453	0.648
	σ ^	0.462	0.354	0.398	0.479	0.392	0.795	0.269
	γ ^	0.976	0.967	0.944	0.980	0.990	0.979	0.972
II	c ^ 70 deg 60 deg	0.944	0.848	0.825	0.912	0.737	0.808	0.813
	c ^ 60 deg 50 deg	0.947	0.792	0.846	0.834	0.759	0.831	0.780
	c ^ 70 deg 50 deg	0.800	0.708	0.700	0.757	0.559	0.620	0.635
	σ ^	0.397	0.438	0.481	0.365	0.404	0.698	0.299
γ ^	0.980	0.980	0.960	0.981	0.996	0.984	0.974

Testing transitivity

We tested whether the data display transitivity. That is, if one surface under Illuminant A was perceived to be equal in roughness to another surface under Illuminant B and that second surface was perceived to be equal in roughness to yet a third surface under Illuminant C, then the transitivity prediction is that the first surface (under Illuminant A) should match the third surface (under Illuminant C). More specifically, if r _A is the roughness of any surface under Illuminant A, and it matches a surface with roughness r _B under Illuminant B, then  

$r_{\mathrm{B}}=c_{\mathrm{AB}}{r}_{\mathrm{A}}.$

(2)

If we now find the surface roughness under Illuminant C that matches this second surface under Illuminant B, we have,  

$r_{\mathrm{C}}=c_{\mathrm{BC}}{r}_{\mathrm{B}},$

(3)

and, combining Equations 2 and 3, we have the prediction,  

$r_{\mathrm{C}}=c_{\mathrm{BC}}{c}_{\mathrm{AB}}{r}_{\mathrm{A}}.$

(4)

However, we also have the prediction, based on the model developed above, that  

$r_{\mathrm{C}}=c_{\mathrm{AC}}{r}_{\mathrm{A}}.$

(5)

Comparing Equations 4 and 5, we have the transitivity prediction,  

$c_{\mathrm{AC}}=c_{\mathrm{BC}}{c}_{\mathrm{AB}}.$

(6)

Because we have independent estimates of each of the transfer parameters, we can test whether Equation 6 is valid for our data. 

Figure 14 shows the estimated values of slope

$\widehat{c}$

_{70 deg 50 deg} plotted against the corresponding slope predictions

$\widehat{c}$

_{60 deg 50 deg}

$\widehat{c}$

_{70 deg 60 deg} based on transitivity. Most of the data fall along the identity line. We obtained 95% confidence intervals for both prediction and estimate by a bootstrap method (Efron & Tibshirani, 1993). 

Figure 14

 

Results. Estimated slopes, c ^ _{70 deg 50 deg}, compared with slopes predicted by the transitivity rule, c ^ _{60 deg 50 deg} c ^ _{70 deg 60 deg}, for all observers in Conditions I (blue) and II (red). Standard errors represent 95% confidence intervals for each bootstrapped slope. For almost all points, the slopes match the predictions (dashed line).

The estimated and predicted values, their respective percentage errors of prediction

$(\epsilon =\frac{{\widehat{c}}_{60\mathrm{deg}50\mathrm{deg}}{\widehat{c}}_{70\mathrm{deg}60\mathrm{deg}}-{\widehat{c}}_{70\mathrm{deg}50\mathrm{deg}}}{{\widehat{c}}_{60\mathrm{deg}50\mathrm{deg}}{\widehat{c}}_{70\mathrm{deg}60\mathrm{deg}}}\times 100\%)$

, and p values are listed in Table 2. We performed a z test to determine whether the measured and predicted slopes were significantly different from each other for each possible comparison. There was no significant difference between slopes for both conditions (I and II) at the Bonferroni-corrected α level for seven tests ( α ≅ .007) for six of the seven observers, consistent with the claim that observers' judgments are transitive. 

Table 2

 

Estimated values of slope c ^ _{70 deg 50 deg} for each observer and predicted slope c ^ _{60 deg 50 deg} c ^ _{70 deg 60 deg} based on the transitivity rule ( Equation 6). The error of prediction ɛ (see Testing transitivity section for details) represents how much the estimated value of the slope departed from the value predicted by transitivity. The value in boldface indicates a significant difference between slope values for a z test at the Bonferroni-corrected α level of .007 per test.

		Observer
Context condition	Transitivity predictions	C.P.	J.G.	M.F.	M.S.L.	P.J.N.	T.A.	Y.X.H.
I	c ^ 70 deg 50 deg	0.826	0.599	0.646	0.797	0.582	0.453	0.648
	c ^ 60 deg 50 deg c ^ 70 deg 60 deg	1.113	0.624	0.684	0.794	0.610	0.600	0.596
	% ɛ	25.770	3.980	5.630	−0.300	4.640	24.560	−8.750
	p	.001	.540	.453	.972	.480	.041	.126
II	c ^ 70 deg 50 deg	0.800	0.708	0.700	0.757	0.559	0.620	0.635
	c ^ 60 deg 50 deg c ^ 70 deg 60 deg	0.894	0.672	0.698	0.761	0.559	0.671	0.635
	% ɛ	10.480	−5.380	−0.370	0.490	0.010	7.610	−0.002
	p	.131	.473	.966	.941	.999	.497	∼1

Discussion

In this study, we found that observers deviated substantially from the predictions of roughness constancy. Figure 15 shows two rendered surfaces that a typical observer perceived to be equal in roughness. This deviation from roughness constancy held even after additional cues to illuminant direction were provided in the scene. 

Figure 15

 

Two surfaces perceived by one observer (P.J.N.) in this study to be equal in roughness. A frontoparallel view of (a) a test surface with roughness r = 1.56 cm under one illuminant ( ϕ = 70 deg) and (b) a match surface with roughness r = 0.61 cm under another illuminant ( ϕ = 50 deg), which was perceived to be equal in roughness to the surface in (a).

One explanation for the continued bias in responses with additional illuminant cues may be that observers did not have sufficient time to scrutinize the scene. It is possible that observers may have needed more than the 1-s presentation time to process all the information in the relatively complex scene. We tested this possibility: One observer (Y.X.H.) repeated the experiment with each scene presented for 2 s instead of 1. The bias remained: The estimated slopes (

$\widehat{c}$

values) for Conditions I and II were nonzero and less than 1 ( z tests, p < .001). 

Rather, we suggest that the additional cues in the scene do not improve roughness constancy because observers are instead relying on cues contained in the texture itself (as suggested by Koenderink et al., 2003). That is, observers may use the pattern of shading and cast shadows to estimate surface roughness. These cues vary not only with roughness but also with changes in the pattern of illumination, leading to failures of roughness constancy. 

A cue combination model

In an image of a 3D surface, there exist a number of visual cues that are affected by changes in surface roughness. Some of these are invariant under changes in illumination conditions and others are not. An example of the former would be a measure of the depth variance of the surface patch based on disparity estimates. So long as the visual system can accurately estimate disparity values, this information should not be affected by changes in lighting. Let R _d denote the estimate of roughness based on illumination-invariant cues. Note that R _d may be the result of combining multiple illumination-invariant cues. For our purposes, it suffices to lump all such cues together. 

We assume that the expected value of R _d is the true roughness of the surface: E( R _d) = r. That is, R _d is an unbiased cue. If a visual system used only R _d as its roughness estimate, then it would display roughness constancy. However, if the variance of R _d is large, then the observer's estimates would be highly variable from trial to trial. Consequently, the observer might seek to reduce the variance by combining R _d with other roughness cues. These additional cues are necessarily affected by change in illumination, given how we have defined R _d. 

Inspection of the rendered images suggests four physical measures of the scene, which would be affected by changes in roughness r: (1) r _p, the proportion of the image that is not directly lit by the punctate source (the proportion of the image in shadow); (2) r _s, the standard deviation in luminance of nonshadowed pixels in the image due to differential illumination by the punctate source; (3) r _m, the mean luminance of nonshadowed pixels; and (4) r _c, the texture contrast ² as defined by Pont and Koenderink (2005). Texture contrast is intended to be a robust statistic for characterizing materials across lighting conditions. It is less sensitive to lighting changes than the other three measures. Each measure is a function of the true roughness of the surface r and the lighting condition L and can be written r_s(r,L), r_m(r,L), r_p(r,L), and r_c(r,L) to emphasize this dependence. 

Each of these measures is highly correlated with roughness r when only roughness is varied while the lighting condition L is held constant. Increasing roughness, for example, increases the proportion of the scene r _p( r,L) consisting of cast and attached shadows. Correspondingly, the mean image intensity decreases and the variation of facet illumination increases. We have verified that this is the case in our stimuli for all four of these measures. However, when surface roughness remains constant and lighting conditions change, the values of these measures also change. Consequently, the values of these measures confound roughness and lighting condition. 

We assume that the visual system has four available “pseudocues” to roughness, R _s, R _m, R _p, and R _c, corresponding to the four physical measures just defined. We assume that each is an unbiased estimate of the corresponding physical measure; that is, E[ R _p] = r _p( r,L) and similarly for the other three measures. 

We consider the possibility that the visual system errs in using these pseudocues across changes in lighting condition. We assume that cues and pseudocues are scaled and combined by a weighted average (Landy, Maloney, Johnston, & Young, 1995). In viewing a surface of roughness r in lighting condition L, the observer forms the roughness estimate 

$R=w_{\mathrm{d}}{R}_{\mathrm{d}}+w_{\mathrm{s}}{R}_{\mathrm{s}}+w_{\mathrm{m}}{R}_{\mathrm{m}}+w_{\mathrm{p}}{R}_{\mathrm{p}}+w_{\mathrm{c}}{R}_{\mathrm{c}},$

(7)

where the values w_i combine the scale factors and weights and thus need not sum to 1 as weights do. 

In this experiment, observers compare this roughness estimate to the roughness estimate for a second surface patch with roughness r′ viewed under a different lighting condition L′,  

$R\prime =w_{\mathrm{d}}{R}_{\mathrm{d}}^{\prime }+w_{\mathrm{s}}{R}_{\mathrm{s}}^{\prime }+w_{\mathrm{m}}{R}_{\mathrm{m}}^{\prime }+w_{\mathrm{p}}{R}_{\mathrm{p}}^{\prime }+w_{\mathrm{c}}{R}_{\mathrm{c}}^{\prime },$

(8)

to decide which surface was rougher. Consider the situation in which two surfaces are perceived to be as equally rough, that is, R = R′. Subtracting Equations 7 and 8 yields  

$0=w_{\mathrm{d}}\mathrm{\Delta }{R}_{\mathrm{d}}+w_{\mathrm{s}}\mathrm{\Delta }{R}_{\mathrm{s}}+w_{\mathrm{m}}\mathrm{\Delta }{R}_{\mathrm{m}}+w_{\mathrm{p}}\mathrm{\Delta }{R}_{\mathrm{p}}+w_{\mathrm{c}}\mathrm{\Delta }{R}_{\mathrm{c}},$

(9)

where Δ R _i = R′ _i − R _i. We assume that w _d is nonzero (the observer is making some use of illuminant-invariant cues) and rearrange as  

$\mathrm{\Delta }{R}_{\mathrm{d}}=a_{\mathrm{s}}\mathrm{\Delta }{R}_{\mathrm{s}}+a_{\mathrm{m}}\mathrm{\Delta }{R}_{\mathrm{m}}+a_{\mathrm{p}}\mathrm{\Delta }{R}_{\mathrm{p}}+a_{\mathrm{c}}\mathrm{\Delta }{R}_{\mathrm{c}},$

(10)

where a _s = − w _s/ w _d, and so forth. If R _s, R _m, R _p, and R _c were unbiased cues to roughness, then the expected values of Δ R _s, Δ R _m, Δ R _p, and Δ R _c would all be 0 and, as a consequence, E[Δ R _d] = r − r′ = 0. We would expect the observer to be roughness constant on average but that is not what we found experimentally. Observers systematically matched surfaces with very different roughness r ≠ r′ across lighting conditions. 

The expected value of the difference between the pseudocue R _p and R _p′ is the difference between the actual proportion of the image not directly lit by the punctate illuminants in the two scenes. If we denote this difference by Δ r _p = r _p( r′, L′) − r _p( r,L), then we have E[Δ R _p] = Δ r _p. Similarly, E[Δ R _m] = Δ r _m, E[Δ R _s] = Δ r _s, and E[Δ R _c] = Δ r _c. 

We computed the expected values of each of the terms Δ r _s, Δ r _m, Δ r _p, and Δ r _c for each value of roughness and lighting condition by first computing r _s( r,L), r _m( r,L), r _p( r,L), and r _c( r,L) for each possible roughness r and lighting condition L and then taking differences. These were computed using the four stimulus images for each condition. 

To compute r _p, r _m, and r _s, we must determine which pixels in each image are not directly illuminated by the punctate source. To do this, we rerendered our scenes with the diffuse lighting term set to 0 and surface albedo set to 1 and no interreflections among facets. We refer to these rerendered images as punctate-only images. 

Pixels with a value of 0 in a punctate-only image correspond to surfaces that are not directly illuminated by the punctate source (i.e., in shadow). The proportion of shadowed pixels ( r _p) and the other terms based on nonshadowed pixels ( r _m and r _s) are easily computed once we know which pixels in the image are not directly illuminated by the punctate source. We determined the set of shadowed pixels using the left-eye images only. 

Equation 10 posits that the difference in roughness at the PSE is a linear combination of the difference in each of the cues we have identified, each perturbed by error. To test this model, we collapsed Conditions I and II and regressed 36 PSE differences (2 context conditions × 3 illuminant comparisons × 6 test roughness levels) against the differences in the illuminant-variant cues. The resulting regression equation expresses the observers' failures of constancy in terms of the hypothetical light-variant cues,  

$\mathrm{\Delta }{R}_{\mathrm{d}}=a_{\mathrm{s}}\mathrm{\Delta }{r}_{\mathrm{s}}+a_{\mathrm{m}}\mathrm{\Delta }{r}_{\mathrm{m}}+a_{\mathrm{p}}\mathrm{\Delta }{r}_{\mathrm{p}}+a_{\mathrm{c}}\mathrm{\Delta }{r}_{\mathrm{c}}+a_0+\varepsilon ,$

(11)

where the error term ɛ combines all of the errors given by all terms in the model. The results of the regression fit are shown in Table 3. Note that we include a constant term a ₀ in the regression. We will return to this term in the discussion below. 

Table 3

 

Percentage of variance ( R ²) accounted for by combinations of predictors regressed to deviations in roughness (Δ R _d). For each row, we tested the null hypothesis that R ² = 0 for each entry in the table, with a Bonferroni correction for multiple tests. Values in boldface indicate that R ² was significantly different from 0. The overall α of the test was .05. The Bonferroni-corrected level for each was .007 = .05/7. We did not apply Bonferroni correction for all 49 tests as the results in each column are not independent.

VAF	Observer
C.P.	J.G.	M.F.	M.S.L.	P.J.N.	T.A.	Y.X.H.
R 2 c	35	57	45	40	34	25	70
R 2 s	63	41	32	42	19	30	45
R 2 m	2	17	19	16	15	8	18
R 2 p	12	15	24	14	29	2	57
R 2 s,c	71	58	47	43	41	30	80
R 2 m,c	44	58	45	40	34	26	70
R 2 m,s	72	41	33	42	22	30	46
R 2 p,c	73	57	45	43	40	32	78
R 2 p,s	73	48	37	42	37	30	72
R 2 p,m	19	29	37	24	43	10	71
R 2 m,s,c	74	59	47	44	41	30	81
R 2 p,s,c	73	61	51	43	42	32	81
R 2 p,m,c	81	58	45	47	44	32	78
R 2 p,m,s	77	49	40	43	44	30	76
R 2 p,m,s,c	82	66	53	48	44	33	81

In using the variation in the cues from trial to trial to estimate the weight assigned to each cue, we are, in effect, applying the technique used by Ahumada and Lovell (1971), which is the basis of image classification methods. Note that these coefficients do not provide us directly with an estimate of how much weight the observers give each cue. However, we can determine how much each cue or combination of cues contributes to the observer's judgments by comparing the proportion of variance accounted for by each of the 15 possible combinations of predictors (Table 3). The combination of the four predictors of roughness judgments explains 58% of the variance in the data on average over the seven observers (values for individual observers ranged from 33% to 82%). Figure 16 shows ΔR_d (the observer's failure of roughness constancy) plotted against the predicted values

$\mathrm{\Delta }{\widehat{R}}_{\mathrm{d}}={\widehat{a}}_{\mathrm{s}}\mathrm{\Delta }{r}_{\mathrm{s}}+{\widehat{a}}_{\mathrm{m}}\mathrm{\Delta }{r}_{\mathrm{m}}+{\widehat{a}}_{\mathrm{p}}\mathrm{\Delta }{r}_{\mathrm{p}}+{\widehat{a}}_{\mathrm{c}}\mathrm{\Delta }{r}_{\mathrm{c}}$

using regression estimates of the coefficients for the four pseudocues but without the constant term

${\widehat{a}}_0$

. Most values fall close to the identity line. Although the values of

${\widehat{a}}_0$

were significantly different from 0 for some observers, they were relatively small and not patterned across observers. Hence, we recomputed the regression, forcing

${\widehat{a}}_0$

to be 0. 

Figure 16

 

Results. Estimated PSE difference as a function of PSE difference predicted by the linear cue combination model. Notice that most points fall roughly along the identity line. Positive values indicate PSEs falling below the line of roughness constancy. Data from Conditions I and II were collapsed and fitted together.

To summarize, if observers relied solely on illuminant-invariant cues, such as binocular disparity, to make roughness estimates, they would have exhibited no patterned deviations from roughness constancy. Instead, it seems that observers relied on other measures in the scene such as the four pseudocues we considered. These pseudocues do not provide accurate information about roughness across lighting conditions. The visual system's reliance on pseudocues accounts for the systematic deviations away from roughness constancy that we found in our data. In partial mitigation of the visual system's error, we note that these same pseudocues would have been valid cues to roughness had we not varied lighting conditions systematically. 

We do not claim that the four cues we advance are precisely the cues that the visual system uses. Any invertible matrix transformation of the four cues used here results in four alternative cues that would explain our results equally well. If, for example, we replaced R _p by R _p + R _m, R _m by R _p − R _m, and left R _s and R _c unchanged, we would have a new set of four cues that fit the data equally well. Nonlinear transformations of the four pseudocues may better account for the data. 

In particular, we have fit a second model, substituting two pseudocues R _m′ and R _s′ for R _m and R _s. The expected values r _m′ and r _s′ were computed in the same way as for the unprimed versions but using the punctate-only images described above instead of the images that the observer saw. The revised model based on this second set of pseudocues accounts for a markedly larger proportion of the variance (90% on average, ranging from 74% to 95%). We note, however, that it is not obvious how observers could compute estimates of these cues from the images actually viewed. To do so, they would have to effectively discount the effect of the diffuse illuminant on the scene, as well as interreflections. Thus, if observers can compute these alternative pseudocues, then we have found a parsimonious model that predicts their failures of roughness constancy remarkably well. 

Nonetheless, there is a parallel between one of the pseudocues we found and the “blackshot mechanism” of Chubb, Landy, and Econopouly (2004). In studying 2D texture, they found evidence for a visual mechanism that was highly sensitive to very dark regions of the stimulus and that effectively computed a contrast between these regions and the brighter parts of the stimulus. It is possible that the blackshot mechanism plays a role in 3D texture perception, providing an estimate of what we referred to as R_p, the proportion of the scene not lit directly by the punctate source. 

The deception of shadows

It appears that observers' errors in roughness judgments are largely due to the contribution of shadows/shading to the image. Interestingly, the relationship between the visual cue of shadows and the haptic sense of object/surface properties can be traced back to observations made by the 18th century empiricist philosopher Condillac who described the way a “statue” learns to see a sphere:

 

The first time it brings its vision to bear on a sphere, the impression it gets of it stands for nothing but a flat circle, with shadow and light mixed. It does not, therefore, yet see a sphere: for its eye has not yet learned to assess the relief on a surface where shadow and light are distributed in a particular proportion. But it has touch now, and because it is learning to come to the same judgments with vision as it comes to with touch, the statue takes under its eyes the relief that it has under its hands (from Baxandall, 1995).

 

Condillac suggested that the visual mind is educated by association to the active sense of touch much in the same way that we may conceptualize how one learns to discriminate between material properties such as roughness. Condillac seemed to suggest a way of interpreting observers' reliance on pseudocues as a by-product of associative perceptual learning. Anticipating the ideas of Pavlov, Condillac proposed that a new perceptual cue can result from repeated pairing with another cue that elicits a reliable perceptual response. 

Indeed, it has been shown that associative learning can affect perceptual appearance (e.g., Adams, Graf, & Ernst, 2004; Haijiang, Saunders, Stone, & Backus, 2006; Jacobs & Fine, 1999; Sinha & Poggio, 1996; Wallach & Austin, 1954). Specifically, Haijiang et al. (2006) showed that a visual cue that would normally be ineffective can, after training involving pairing with an effective cue, become effective at disambiguating a perceptually bistable display. Similarly, we may have learned to associate shadows, shading, or both not only with depth but also with surface roughness. 

We accept the suggestion that the association between haptic roughness and visual roughness is learned and conjecture that pseudocues arise as a pathology of associative learning. If most comparisons of haptic and visual roughness take place within a single lighting context and not across changes in lighting context, then the visual system may be guilty only of overgeneralizing a valid learned cue beyond its range of applicability. It may be that patterned deviations away from constancy in other perceptual judgments can also be accounted for as the overgeneralization of learned cues that may have been valid in the limited context in which they were learned. 

Appendix:

Roughness discrimination model

Suppose that the observer compares two surfaces; one surface has roughness level r _a and is viewed under Illumination Condition A (first interval) and the other surface has roughness level r _b under Illumination Condition B (second interval). We first assume that the observer's roughness estimate is a transformation of actual roughness that depends on the illuminant,  

$\begin{array}{c}{\rho }_{\mathrm{aA}}=L_{\mathrm{A}}\left(r_{\mathrm{a}}\right)\hfill \\ {\rho }_{\mathrm{bB}}=L_{\mathrm{B}}\left(r_{\mathrm{b}}\right).\hfill \end{array}$

(A1)

On each trial, these estimates are perturbed by normally distributed error with 0 mean,  

$\begin{array}{c}{R}_{\mathrm{aA}}={\rho }_{\mathrm{aA}}+{\varepsilon }_{\mathrm{aA}}\hfill \\ R_{\mathrm{bB}}={\rho }_{\mathrm{bB}}+{\varepsilon }_{\mathrm{bB}}.\hfill \end{array}$

(A2)

We allow for the possibility that the variance of the error depends on the magnitude of perceived roughness, in a manner analogous to Weber's law. Because our choice of a roughness scale was arbitrary, we formulate a generalization of Weber's law. We assume that the standard deviation of the error is proportional to a power function of the perceived roughness level:  

${\varepsilon }_{\mathrm{aA}}\sim N(0,{\sigma }^2{\rho }_{\mathrm{aA}}^{2\gamma }).$

(A3)

Here, σ ² is the variance when ρ _aA equals 1, and γ yields the power transformation. If γ is 1, then Weber's law holds for the arbitrary roughness scale that we use. If γ is 0, then the error is invariant with roughness level. 

We next assume that the observer forms a decision variable Δ on each trial to decide whether the rougher patch appeared in the first or second interval,  

$\begin{array}{c}\Delta =R_{\mathrm{bB}}-R_{\mathrm{bA}}\\ =L_{\mathrm{B}}\left(r_{\mathrm{b}}\right)-L_{\mathrm{A}}\left(r_{\mathrm{a}}\right)+\varepsilon ,\end{array}$

(A4)

where ɛ is normal with a mean of 0 and a variance of σ ²( L _B( r _b) ^{2 γ} + L _A( r _a) ^{2 γ}). The observer responds “second interval” if Δ > 0 and otherwise responds “first interval.” Let p denote the probability of responding second interval. Then,  

$p=\Phi \left(\frac{L_{\mathrm{B}}\left(r_{\mathrm{b}}\right)-L_{\mathrm{A}}\left(r_{\mathrm{a}}\right)}{\sigma \sqrt{L_{\mathrm{B}}}(r_{\mathrm{b}}{)}^{2\gamma }+L_{\mathrm{A}}(r_{\mathrm{a}}{)}^{2\gamma }}\right),$

(A5)

where Φ denotes the cumulative normal distribution with a mean of 0 and a variance of 1. 

We next assume that the roughness transformation functions are linear,  

$\begin{array}{c}{L}_{\mathrm{A}}\left(r\right)=c_{\mathrm{A}}r\hfill \\ L_{\mathrm{B}}\left(r\right)=c_{\mathrm{B}}r.\hfill \end{array}$

(A6)

Substituting Equation A6 into Equation A5 yields  

$p=\Phi \left(\frac{c_{\mathrm{B}}{r}_{\mathrm{b}}-c_{\mathrm{A}}{r}_{\mathrm{a}}}{\sigma \sqrt{(c_{\mathrm{B}}}{r}_{\mathrm{b}}{)}^{2\gamma }+(c_{\mathrm{A}}{r}_{\mathrm{a}}{)}^{2\gamma }}\right).$

(A7)

We simplify Equation A7 by letting c _AB = c _A/ c _B and absorbing extra parameters into σ, yielding:  

$p=\Phi \left(\frac{r_{\mathrm{b}}-c_{\mathrm{AB}}{r}_{\mathrm{a}}}{\sigma \prime \sqrt{r_{\mathrm{b}}}{2}^{\gamma }+(c_{\mathrm{AB}}{r}_{\mathrm{a}}{)}^{2\gamma }}\right).$

(A8)

 

We define the contour of indifference to be the ( r _a, r _b) pairs such that L _B( r _a) = L _A( r _b). These pairs are predicted to appear equally rough to the observer under the corresponding illumination conditions. We refer to this contour as the transfer function connecting Illumination Conditions A and B,  

$r_{\mathrm{b}}={\tau }_{\mathrm{AB}}\left(r_{\mathrm{a}}\right)=L_{\mathrm{B}}^{-1}\circ L_{\mathrm{A}}\left(r_{\mathrm{a}}\right)=\frac{c_{\mathrm{A}}}{c_{\mathrm{B}}}{r}_{\mathrm{a}}=c_{\mathrm{AB}}{r}_{\mathrm{a}},$

(A9)

where c _AB is as defined above. Note that if c _AB = 1, the observer's judgments of roughness are unaffected by a change of illumination condition. That is, the observer is roughness constant, at least for this pair of illumination conditions. 

We cannot directly observe L _A( r) for any Illumination Condition A or estimate the constant c _A in the form of L _A( r) we have assumed. We can, however, estimate the transfer function parameter c _AB from our data. 

Acknowledgments

This research was supported by National Institutes of Health Grant EY08266 and EY16165. We thank Hüseyin Boyaci and Katja Doerschner for help in developing the software used in the experiments described here, which is based on a code written by them, and for many helpful comments and suggestions. 

Commercial relationships: none. 

Corresponding author: Yun-Xian Ho. 

Email: [e-mail]. 

Address: Psychology Department, New York University, 6 Washington Place, Room 957, New York, NY 10003, USA. 

Footnotes

Footnotes

1  The punctate-total ratio is the ratio of the intensity of light absorbed by an infinitesimal test patch facing the punctate light source to the intensity of all light absorbed by the patch (Boyaci et al., 2003). It is a measure of the relative intensities of punctate and diffuse sources.

Footnotes

2   r _c is a modified version of the Michelson contrast computed as follows: 95th percentile of the luminance histogram minus the 5th percentile, then divided by the median luminance.