Detection of light transformations and concomitant changes in surface albedo

Abstract

We report two experiments demonstrating that (1) observers are sensitive to information about changes in the light field not captured by local scene statistics and that (2) they can use this information to enhance detection of changes in surface albedo. Observers viewed scenes consisting of matte surfaces at many orientations illuminated by a collimated light source. All surfaces were achromatic, all lights neutral. In the first experiment, observers attempted to discriminate small changes in direction of the collimated light source (light transformations) from matched changes in the albedos of all surfaces (non-light transformations). Light changes and non-light changes shared the same local scene statistics and edge ratios, but the latter were not consistent with any change in direction to the collimated source. We found that observers could discriminate light changes as small as 5 degrees with sensitivity d′ > 1 and accurately judge the direction of change. In a second experiment, we measured observers' ability to detect a change in the surface albedo of an isolated surface patch during either a light change or a surface change. Observers were more accurate in detecting isolated albedo changes during light changes. Measures of sensitivity d′ were more than twice as great.

Introduction

In everyday three-dimensional scenes, the flow of light is rarely homogeneous. It depends on the location and the spectral and spatial characteristics of light sources within the scene and on the location and orientation of surfaces within the scene that absorb and reemit light, creating shadows and serving as secondary light sources. The resulting flow of light can be summarized as a light field (Gershun, 1936/1939) or, more generally, a plenoptic function (Adelson & Bergen, 1991) across the scene, a specification of the spectral power distribution of light arriving from each direction at each location in the scene. 

Human observers can accurately estimate light field parameters in static scenes under some circumstances (Khang, Koenderink, & Kappers, 2006; Koenderink, van Doorn, Kappers, te Pas, & Pont, 2003; Koenderink, van Doorn, & Pont, 2004; Pont & Koenderink, 2007) and reliably estimate the spatial variation of the light field across a static scene even when probed in locations that lack local cues to the light field (Koenderink, Pont, van Doorn, Kappers, & Todd, 2007). Furthermore, human observers partially discount variation in the light field ¹ in judging matte surface color and lightness (Boyaci, Doerschner, & Maloney, 2004; Boyaci, Maloney, & Hersh, 2003; Gilchrist, 1977, 1980; Ikeda, Shinoda, & Mizokami, 1998; Ripamonti et al., 2004; Snyder, Doerschner, & Maloney, 2005; see Maloney, Gerhard, Boyaci, & Doerschner, 2010). 

In this article, we are primarily concerned with scenes consisting of achromatic matte surface patches illuminated by a single collimated neutral light source. The surfaces vary in orientation and need not be confined to a single plane. When the surfaces are confined to a single plane, we refer to the resulting configuration as a Mondrian (Land & McCann, 1971). When the surface patches differ in orientation, we refer to the resulting configuration as a crumpled Mondrian, ² illustrated on the right side of Figures 1A and 1B. 

Figure 1

 

The effect of surface position and orientation in 2D and 3D. (A, B) The same rendered scene illuminated by a collimated light source. The left side of the scene is composed of coplanar surfaces (a Mondrian) while surfaces on the right vary in orientation (a crumpled Mondrian). The only difference in the two renderings in (A) and (B) is the direction to the light source. In (A), the elevation of the light source is ϕ = 75°, the azimuth is ψ = 190°. In (B), ϕ = 75° (unchanged) and ψ = 325°. The inset illustrates how azimuth and elevation are defined. The effect of the change in light source direction on the left half of the scene is a common overall scaling of the luminance of all surface patches (in this case, a scaling by 1). The ratios of luminance across edges in the Mondrian are invariant. The effect on the crumpled Mondrian (right side) is more dramatic. Edge ratios are not constant and for some pairs of adjacent surfaces one surface increases in luminance as the other decreases. These are two frames taken from Movie 1, which shows the effect of smoothly varying the azimuth of the light with elevation held constant on the Mondrian and the crumpled Mondrian.

Figures 1A and 1B are two frames of a movie (Movie 1). Over the course of the movie, the direction to the collimated light source changes smoothly and periodically: it rotates smoothly around the scene, remaining at a fixed elevation above the ground plane. The angle between the light source and a surface normal to the Mondrian on the left never varies. As a consequence of Lambert's Law (Haralick & Shapiro, 1993, pp. 2–7), there is no effect on the Mondrian's luminances. The lighting change is invisible. The right sides of Figures 1A and 1B and Movie 1 illustrate the dramatic effect of the same changes in light source direction on a crumpled Mondrian. 

 

Movie 1

 

The effect of surface position and orientation in 2D and 3D dynamic versions. The light source remains at fixed elevation, ϕ = 75°, and rotates counterclockwise in azimuth at a fixed angular velocity.

If the direction to the light source varied in elevation, then the effect on the Mondrian would be an overall scaling of the luminance of each surface patch in the Mondrian. The ratio of luminances of any two patches would always be invariant, in particular the ratio across edges where surfaces abut. In the crumpled Mondrian, in contrast, edge ratios are typically not constant as can be seen in Figures 1A and 1B and Movie 1. For some pairs of adjacent surfaces, one surface in the pair increases in luminance while the other decreases, while for other adjacent pairs both luminances increase or decrease together. 

A change in direction or intensity of the collimated source (a light change) will typically alter the luminance of all of the surface patches within a crumpled Mondrian. However, within the space of all possible patterns of surface luminance changes, few are consistent with such a light change. That is, the patterns of luminance change (light transformations) consistent with a light change are highly constrained. Our first goal in this article is to examine human ability to discriminate light and non-light transformations. Before stating this goal precisely, we need to characterize the light transformations expected in Mondrian and crumpled Mondrian scenes. 

Light transformations. Previous studies of changing illumination have typically used Mondrian scenes with homogeneous light fields as stimuli (Craven & Foster, 1992; Nascimento & Foster, 2000). In two-dimensional scenes illuminated homogeneously (Figure 1, left-hand side), the effect of movement of collimated sources is simply a common scaling of the excitations of the three photoreceptor classes. As [ρ _L ^x , ρ _M ^x , ρ _S ^x ] are the excitations at retinal location x in the long-, middle-, and short-wavelength classes (LMS) of photoreceptors, then the effect of a change in illumination is a scaling of photoreceptor excitations within each class (Maloney, 1999) that depends only on the direction and intensity of the collimated source. 

Let this direction be denoted θ(t), the angle with respect to the surface normal n common to all the patches in the achromatic Mondrian, and let t denote time. For convenience, we assume −π/2 < θ(t) < π/2 (the light remains on one side of the surface patch). If a ^j denotes the albedo of patch j = 1, …, n within the Mondrian, then the time varying luminance of each patch is, as a consequence of Lambert's Law (Haralick & Shapiro, 1993, pp. 2–7) 

$L^j\left(t\right)=L\cos \theta \left(t\right)a^j,$

(1)

where L is the luminance emitted by a perfectly reflective matte patch (a = 1) when direction to the light source is parallel to the surface normal common to all the patches. We refer to it as the light source intensity. 

When the collimated source's direction changes, the surface luminances all change in phase across time and the luminance edge ratio between any two adjacent surface patches at locations j and k is invariant, equal to a ^j /a ^k . This can be seen in Figure 1 on the left-hand side. All luminances scale by a common factor. 

Even with more than one collimated light source, each changing its direction and intensity over time, edge ratios in the achromatic Mondrian are invariant. Let L _k (t) and θ _k (t) be the time-varying intensities and directions, respectively, of k = 1, …, m collimated light sources. Then the time-varying luminance of a surface patch with fixed albedo a ^j is 

$L^j\left(t\right)=\left[\sum _{k=1}^m{L}_k\right(t\left)\cos {\theta }_k\right(t\left)\right]a^j.$

(2)

The time-varying quantity in brackets is the same for all patches. As a consequence, edge ratios (and, more generally, ratios of the luminances of any two surface patches) do not change. Moreover, the effect of all of the collimated lights, varying in intensity and direction, is equivalent to that of one collimated light whose direction is orthogonal to the Mondrian and with time-varying intensity, 

$L\left(t\right)=\sum _{k=1}^m{L}_k\left(t\right)\cos {\theta }_k\left(t\right).$

(3)

That is, the pattern of luminances in an achromatic Mondrian illuminated by collimated sources contains very little information about the number, intensities, and directions of the sources. Boyaci, Doerschner, Snyder, and Maloney (2006) refer to this lack of information as the “Mondrian singularity.” 

In an elegant series of articles, Foster, Nascimento, and colleagues have shown that observers are remarkably sensitive to changes in the illumination of Mondrian scenes (Craven & Foster, 1992; Nascimento & Foster, 2000). Their observers discriminated ratio constant light changes from matched non-light changes in which ratios across edges changed. Foster and colleagues suggest that observers rely on the constancy (or lack of constancy) of cone excitation ratios to perform such discriminations (Amano & Foster, 2004; Foster & Nascimento, 1994). 

They conclude that observers are remarkably good at detecting light-induced changes in patterns of retinal excitation and discriminating them from other transformations. They propose that the edge invariance just described and its generalization to the case where lights and surfaces vary in color play an important role in color perception and surface color constancy. 

However, edge ratio invariance is a consequence of restricting attention to Mondrian stimuli. It need not hold for 3D configurations and typically will not. For convenience, we confine attention to scenes where mutual illumination of surfaces and cast shadows play no role. Let ψ(t) denote the time-varying azimuth ³ and ϕ(t) denote the time-varying elevation of the direction to a collimated light source and let ψ _j denote the azimuth and ϕ _j elevation of the jth surface patch, j = 1, …, n. The cosine of the angle between the light direction and the surface normal is then (Green, 1985, p. 11) 

$\cos \theta \left(t\right)=\sin \varphi \left(t\right)\sin {\varphi }_j+\cos \varphi \left(t\right)\cos {\varphi }_j\cos \left(\psi \right(t)-{\psi }_j).$

(4)

When, for example, the light direction varies only in azimuth ψ(t) so that ϕ(t) is a constant, then Equation 4 becomes 

$\cos \theta \left(t\right)=A_j+B_j\cos \left(\psi \right(t)-{\psi }_j),$

(5)

where A _j , B _j are constants determined by the (constant) elevation of the jth surface patch. Combining Equations 1 and 5, 

$L^j\left(t\right)=L{a}^j[A_j+B_j\cos (\psi \left(t\right)-{\psi }_j\left)\right]=C_j+D_j\cos \left(\psi \right(t)-{\psi }_j),$

(6)

where C _j , D _j are constants determined by the elevation and albedo of the jth surface patch and the intensity of the collimated source. All of the luminances of surface patches are linear transformations of one another but with possibly distinct time lags ψ _j . 

We can now interpret the changes in luminance for the crumpled Mondrian illuminated by a single moving collimated source in Figures 1A and 1B and Movie 1. The light source has a fixed elevation and moves at a constant angular velocity in azimuth. Surface luminances change, but some lag, others lead in time. Adjacent regions can change in opposite directions, one reducing in luminance as the other increases. InFigure 2, we plot the luminances of two adjacent surface patches across time as the azimuth of the punctuate source varies cosinusoidally across time. The two surfaces have the same albedo (0.5) but different orientations. The edge ratio of their luminances is not constant across time but varies over more than an order of magnitude. 

Figure 2

 

Light transformations. We illustrate how changing direction to the collimated light source affects two surfaces in a crumpled Mondrian. The azimuth to the light source varies cosinusoidally between ψ = −45° and ψ = +45° while its elevation is always 0°. The orientation of the first surface (solid) was ψ = −30°, ϕ = 0°. Its albedo was 0.5. The orientation of the second surface (dashed) was ψ = +30°, ϕ = +30°. Its albedo was also 0.5. The moving light induces a time-varying periodic luminance L _j (t) in each surface j = 1, 2. The L _j (t) differ by a linear transformation in magnitude and a time lag. Over time, the edge ratio varies by more than an order of magnitude. See text.

In the current work, we address sensitivity to complex light transformations in three-dimensional scenes, in which edge ratios carry no information. Our observers had the task of discriminating light transformations (scene-wide luminance changes induced by change in direction of a collimated source) from statistically matched scene-wide changes that contained the same local scene statistics ⁴ including edge ratio information but were inconsistent with a global light field change. We refer to the latter as non-light transformations. Analogues of the non-light transformations that we use (where many albedos change simultaneously) are present in the natural environment. They occur when autumn leaves are blown in the wind and when milling crowds are viewed at a distance. 

Experiment 1

In Experiment 1, we examined the visual system's sensitivity to the complex luminance transformations that occur on three-dimensional surfaces when scene illumination changes. The stimuli are designed so that edge ratios between pairs of surfaces contained no information relevant to performing the task. The design of the stimuli effectively eliminated any other cues based on local scene statistics. 

Methods

Stimuli

We rendered stereo pairs of scenes containing 8 four-sided achromatic matte pyramids with textured sides, floating in black space. The pyramids were rendered under a combination of a collimated light source and a diffuse light source with one quarter intensity. They appeared in an imaginary 4 × 4 checkerboard arrangement subtending 14.8 DVA on a side with each pyramid's square base 3.3 DVA on a side. We included eight pyramids in the expectation based on work by Pungrassamee, Ikeda, Katemake, and Hansuebsai (2005; see also Ikeda, Pungrassamee, Katemake, & Hansuebsai, 2006) that a larger number of pyramids would lead to greater ease in discriminating light transformations from non-light. We have not evaluated this conjecture. 

The checkerboard arrangement ensured that pyramids did not cast shadows on one another (or anywhere else). Half of the pyramids were concave (pointing away from the observer), half convex (pointing toward the observer), with concavity assignment randomized on each trial. Pyramid heights on each trial were drawn from a uniform distribution of heights corresponding to height/base-width ratios from [0.5, 1.5]. The scenes lacked cast shadows and illumination gradients. The only cue to the spatial distribution of the illumination was the shading of the pyramids (see Figure 3 for an example of a typical scene). Stereo pairs were rendered assuming an interpupillary distance of 6 cm, which was sufficient for all observers to perceive the stimuli in depth. 

Figure 3

 

Rendered matte pyramids used as stimuli. The stimuli consisted of arrays of textured pyramids that were equally likely to be concave or convex and that varied randomly in height. All surfaces were rendered as matte (Lambertian). Observers viewed the stimuli binocularly, in a stereoscope. The left and right images of a typical stereo pair are shown for both crossed and uncrossed fusions. The inset at the top is a view of the scene from a different viewpoint.

Generation. Stimuli for the two conditions were generated in yoked pairs. A new set of random pyramid heights and face albedos was chosen for each yoked pair. Albedos (in percentages) were one of 40%, 50%, 70%, or 80%. Each pyramid face was divided into four triangles with albedos randomly chosen uniformly from the above distribution. The landscape determined by these constraints on albedo and geometry was then rendered twice: first with the collimated light source perpendicular to the center of the ground plane, and second after a rotation of the collimated light source ±5°, ±10°, ±15°, or ±20° from perpendicular along one of two movement directions with equal probability. The movement directions were roughly horizontal and vertical yet rotated 5° off the principle axes of the stimulus. Rendering two light source positions yielded the two frames of a light transformation trial. 

Importantly, monocular images were insufficient to determine light source movement direction because the pyramids were concave and convex; e.g., when the light moved rightward, convex pyramids' faces grew brighter on the right sides while concave pyramids' faces grew brighter on the left side. Observers needed the binocular disparity of the stereo images to determine the depth and orientation of each face and hence the light source movement direction. 

To create a yoked non-light transformation trial, we permuted the color signal assignments of a light trial by the same permutation in both frames of the trial. The geometry of the yoked non-light transformation trial was identical to the light trial, only the assignment of the luminance signals within pyramids differed. A random selection of 4 of the 8 pyramids underwent the 180° permutation. Namely, in a permuted pyramid, the north, east, south, and west faces were colored as the south, west, north, and east faces, respectively, of a pyramid that underwent a light change. Importantly, this permutation preserves edge ratios within pyramids and matches them to those of a light transformation. Only the locations of edge ratios within the scene changed. As a result, if, for example, the collimated light had moved rightward (or downward) in the light change, then it appears to move leftward (upward) in the permuted pyramid, and vice versa. Furthermore, the permutation does not affect the quality of stimulus appearance during the first frame due to the collimated light being perpendicular to the scene in frame 1, so that observers could not correctly classify the trial change type until having viewed both trial frames. 

In a non-light transformation trial, four of the pyramids changed consistent with one light shift, the other four changed consistent with a light shift in the opposite direction, and the resulting scene change cannot be interpreted as a light transformation. In addition to preserving edge ratio changes, permuting the luminance signals by our method matched the light trials to the non-light trials for all other local scene statistics in a precisely defined sense. 

We use the term “local scene statistics” to refer to computations based on luminance values and depth information from a single pyramid. These include ratios of luminance across edges within a pyramid, ratios of luminances of non-adjacent surfaces within a pyramid, and the moments (mean, variance, skew, etc.) of the distribution of luminances within a single pyramid. We constructed the stimuli so that the observer cannot discriminate light and non-light transformations without comparing information across pyramids within a scene. 

Using this procedure, we rendered 120 light transformation trials and 120 non-light transformation trials for each of the four change magnitude levels (with the largest magnitude used only for practice): ±5°, ±10°, ±15°, or ±20° of collimated light source rotation. 

Luminance signal assignment. The luminance of each triangle within a pyramid was rendered following Lambert's Law (Haralick & Shapiro, 1993, pp. 2–7), modified to include an additive light component consisting of the diffuse light. The luminance of the jth triangle with albedo a ^j on frame t was determined by 

$L^j\left(t\right)=L_{\mathrm{c}}\cos {\theta }_j\left(t\right)a^j+L_{\mathrm{d}},$

(7)

where L _c is the intensity of the collimated light source, L _d is the intensity of the diffuse light source, and θ _j (t) is the angle between the jth triangle's surface normal and the direction of the collimated light source at time t. The ratio of the collimated source's intensity to the diffuse source's intensity was fixed throughout the experiment to be 4:1. 

Apparatus and software

Left and right images were presented to the corresponding eyes on two 20.1-in. Dell UltraSharp 2000FP LCDs with 1600 × 1200 pixel resolution placed to the observer's left and right. Two small mirrors were placed in front of the observer's eyes to reflect the images separately from each monitor into the corresponding eye. Luminance measurements from a Photo Research PR-650 spectrometer were taken at five points on each LCD: the center, and at 6.2 DVA to the right, left, above, and below the center. Each measurement was taken by holding the midpoint of the spectrometer lens constant while pivoting the spectrometer housing so the lens' view was centered on the position of interest. Spatial homogeneity and directional independence of the luminance signals were confirmed. Separate lookup tables for each LCD were created using luminance measurements taken at each display's center to correct for non-linearities in the liquid crystal responses and to equalize display values for the two LCDs. The maximum luminance achievable on each display was set to 118 cd/m². Fitted gamma values for a simple power function for both displays were 1.7. 

The apparatus was housed in a box of side 124 cm, which was lined in black flocked paper (Edmund Scientific, Tonawanda, NY) to absorb stray light. Only the stimuli on the screens were visible to the observer. The casings of the LCDs and other features of the room were hidden from view by the enclosing box, and all lights were turned off. The observer viewed the stimuli from a head and chin rest placed on the open front side of the box. Additional light baffles were placed on both sides of the observer's head to prevent the light from the LCDs from reaching the observer's eyes directly. The optical distance from eye to corresponding LCD was 70 cm, and stimuli were rendered to be 70 cm in front of the observer to minimize conflict between binocular disparity and accommodation. A schematic of the stereoscope is shown in Figure 4. 

Figure 4

 

The apparatus. The apparatus was a computer-controlled stereoscope consisting of two LCD displays viewed through mirrors. The virtual scene appeared approximately 70 cm in front of the observers. The optical distance from the observer to the LCD displays (dashed lines) was also 70 cm. The displays and mirrors were placed in a large box lined with black flocked paper.

The experimental software was written in JAVA programming language as an implementation of the FullScreen class available as part of Huseyin Boyaci's Guide to JAVA Programming for Psychophysics. ⁵ The computer was a Dell Optiplex GX745 with an NVIDIA GeForce 7300 GT dual-DVI graphics card. 

Procedure

The experimental task was to discriminate light from non-light transformations. Observers viewed virtual three-dimensional landscapes of four-sided concave and convex pyramids that underwent either illumination changes or matched non-light changes. Illumination changes were positional changes of an out-of-view collimated light source, rendered to be behind the observer. Details of both transformations are specified in the Stimuli section above. A trial was two frames, the first frame 2 s and the second 1 s, with immediate replacement. 

Before explaining the two trial types, we first showed observers a light change trial of the largest change magnitude (20°, later used for training with the response keys) and asked them what had changed in the scene, to which the universal response was that the light moved, and then we showed them the yoked non-light trial and asked them if it too were a change in the lighting. All observers readily discriminated light transformations from non-light induced changes during this initial demonstration. Observers were then told that for the duration of the experiment, they would be asked to discriminate changes in lighting direction from non-light changes, and that the two change types were equally probable throughout the experiment. It was explained that they would be asked to report where they thought the light had moved if they classified the trial as a light change. 

During the experiment, observers categorized each trial as either a “light” or “non-light” change using the numerical keypad. If they wished to indicate that the light had moved, they pressed the corresponding arrow key for light movement direction: “rightward,” “leftward,” “upward,” or “downward.” On non-light trials, they pressed the 5 key. These instructions discouraged observers from falsely identifying a trial as light induced. Feedback was not given. Observers initiated each trial by pressing the space bar and were permitted to break at any point during the experiment. 

Trials were blocked by transformation magnitude and were presented in order of increasing difficulty. Each block contained 240 trials, in which light and non-light transformations of the same magnitude were equally probable and presented in random order. Observers were informed that the two change types were equally probable. Block magnitude was determined by the degree of angular rotation of the collimated light source (±5°, ±10°, ±15°, or ±20°) since non-light transformations were generated in a yoked manner described in the Stimuli section above. Prior to completing the experimental trials, each observer completed at least 96 practice trials of transformation magnitude ±20° with feedback to learn the correct response keys. In total, observers completed 720 test trials. 

Observers

Five experienced psychophysical observers who were unaware of the purpose of the experiment participated. All observers had normal or corrected-to-normal vision. All observers passed a stereo screening test prior to participation, in which they correctly labeled the concavity of pyramids like those in the experiment. Observers were compensated $10/h for their participation. 

Results

Light versus non-light changes

Discriminability. Discrimination performance was quantified separately for each observer at each magnitude of change by d′ from signal detection theory, a measure which is independent of observer bias (Green & Swets, 1966/1973). We considered light transformation trials as signal trials and non-light trials as non-signal trials. Therefore, we defined the hit rate, p _H, to be the probability of a “light” response when a light transformation occurred, and the false alarm rate, p _F, to be the probability of a “light” response when a non-light transformation occurred. If Φ⁻¹ is the inverse of the cumulative unit normal distribution, then 

$d^{\prime }={\Phi }^{-1}\left(p_{\mathrm{H}}\right)-{\Phi }^{-1}\left(p_{\mathrm{F}}\right).$

(8)

A zero value of d′ indicates chance performance, and d′ increases with increased discrimination performance. In our task, a d′ = 1 corresponds to 69% correct, d′ = 2 corresponds to 84% correct, and d′ = 3 corresponds to 93% correct. Ninety-five percent confidence intervals for each d′ estimate were obtained by a non-parametric bootstrap method (Efron & Tibshirani, 1993): each observer's performance in the corresponding condition was simulated 100,000 times and the 5th and 95th percentiles were calculated to construct 95% confidence intervals. Separate d′ estimates and 95% confidence intervals are plotted for each observer in Figure 5. 

Figure 5

 

Discrimination of matched light and non-light transformations. Five naive observers discriminated light from matched non-light transformations at three change magnitude levels corresponding to 5°, 10°, and 15° rotations of a collimated light source. The changes in local scene statistics induced by light and non-light transformations were matched. See text. Chance performance is d′ = 0.

For all observers, performance was significantly above chance at all three levels of change magnitude, p < 0.05. For four out of the five observers, discrimination improved significantly from the lowest signal level (±5° light source rotation) to the highest (±15°). In the following, we report the d′ estimates for each observer and respective p-values for a z-test comparing discriminability at the highest level (±15° light source rotation) with that of the lowest level (±5°). Specifically, we constructed the following general test of increasing performance: 

$z=\frac{d_{15}^{\prime }-d_5^{\prime }}{\sqrt{SD(d_5^{\prime }{)}^2+SD(d_{15}^{\prime }{)}^2}},$

(9)

where standard deviations were computed on the vectors of 100,000 bootstrapped d′ values for each condition (excluding simulated d′ estimates of infinity), and the test lacked degrees of freedom. 

Observer 1 performed near ceiling at each level with large d′ estimates that varied from 2.2 to 3.7, z = 4.38, p < 0.001 (confidence intervals include infinity at the two highest change magnitudes, indicated by the arrows in Figure 5). Observer 2 performed well with d′ estimates increasing from 0.76 to 2.4, z = 5.15, p < 0.001. Observer 3 performed well with d′ estimates increasing from 1.1 to 3.5, z = 7.22, p < 0.001. Observer 4 performed well with d′ estimates increasing almost linearly from 1.7 to 2.6, z = 2.83, p < 0.01. Observer 5 was able to discriminate the two change types above chance at each level and showed marginally significant improvement across the range; d′ estimates increased from 0.73 to 1.1, z = 1.60, p = 0.054. 

Performance of all observers indicates that it is possible to discriminate lighting induced scene changes from light-inconsistent changes despite the matched scene statistics including edge ratio information. Furthermore, observers' performance depended on the distance the collimated light source moved: the greater the rotation, the better the discrimination performance between a valid light transformation and the matched non-light transformation. 

Light source movement direction

Observers were asked to report the perceived motion of the light source whenever they reported that the transformation was light induced. We calculated the percentage of correct direction reports contingent on correctly classifying the change as light induced. These percentages were calculated at each level of transformation magnitude for each observer separately. Because there were four possible movement directions, chance performance is 25% correct. 

All observers were well above chance at reporting movement direction when they perceived that the light had moved. Individual percent corrects are plotted as a function of angular collimated light movement in Figure 6. Observer 1 achieved 100% correct movement discrimination at each change magnitude level. The other observers performed near ceiling. The following are averages over all three change magnitudes: Observer 2, 97%, Observer 3, 97%, Observer 4, 98%, and Observer 5, 93%. While Observer 5 could not discriminate the changes as well as the other four observers, this high percentage indicates that when Observer 5 correctly identified a light change, the movement direction was easy to report. These data demonstrate that when observers perceived changes in the light source position, they could also report accurately how the light had changed. 

Figure 6

 

Direction discrimination of collimated light motion. To report a light transformation, observers pressed 1 of the 4 arrow keys to indicate the direction of collimated light movement. The percentage of correct direction reports out of all correctly identified light trials is shown here for each observer at each level of rotation and direction.

Discussion

Overall, the observers in our experiment achieved very high performance, demonstrating the ease of detecting complex luminance transformations that are lighting induced. On average, d′, which describes the ability to discriminate light changes from non-light changes, was equal to 1.30 at the lowest change magnitude tested, equivalent to the luminance change introduced by a 5° rotation of a collimated light source, and by 15°, performance was nearly at ceiling for 4 of the 5 observers. 

It is possible that the initial round of practice, which was performed with easier stimuli and included feedback, contributed to the overall high level of performance. However, if the visual system were unable to process the more complex luminance transformations induced by light field changes in three-dimensional scenes, no amount of training would be able to achieve such results. Indeed, during the demonstration prior to the experiment, all observers clearly labeled two sample scene changes correctly: a large sweeping collimated light movement, which they easily described as: “the light just changed,” and a non-light change, which observers usually labeled as unpatterned, e.g., “everything just changed.” The purpose of our training was for observers to learn the key presses associated to the different scene changes, which they could already discriminate. 

Further evidence of the task's simplicity for observers is the high probability correct for distinguishing where the collimated light had moved. To report a lighting change, observers had to press one of four arrow buttons to indicate where the light had moved. Otherwise, they pressed a non-arrow key to indicate that no light change had occurred. Because of this response pattern, observers were discouraged from responding “light” unless they saw the direction of the collimated light movement. Most observers identified the lighting direction with high precision at all three change magnitudes. Even the observer with the lowest sensitivity was near ceiling to report light movement direction on the trials on which this observer correctly identified a light change. In conclusion, not only can observers discriminate complex lighting transformations from non-sense scrambled versions matched in luminance ratio information, but also, when they detect a lighting transformation, they can accurately report how the light field changed. 

Experiment 2

Both the light and non-light transformations in Experiment 1 involved changes in all of the visible surfaces in the scene. If observers recognize that the scene is undergoing a light transformation, can they use this information to enhance detection of a concomitant change in surface albedo, in effect “discounting the light transformation”? Observers were required to detect local albedo perturbations during light transformations and during matched non-light transformations. We compared perturbation detection performance under light transformations and matched non-light transformations. 

Methods

Stimuli

The scenes were the same as in Experiment 1. 

Generation. Stimuli for the two conditions were generated as in Experiment 1, except that the scrambling method for non-light trials was altered. Lighting-consistent changes were rendered first as in Experiment 1, and the luminance signals they generated were scrambled to produce a scene undergoing a global non-light change. Instead of rotating luminance signal assignments within half of the pyramids by 180°, we pseudo-randomly swapped luminance signal assignments across the scene. Swapping was limited such that triangles within north and south faces were always swapped from other north and south facing faces, and east and west only swapped from other east and west facing triangles. This was done to maintain the overall direction of luminance change (up/down versus left/right) within the scene between the light and the non-light trials. Swapping was also limited such that triangles with initially dark albedos (40% or 50% reflectance) were traded only with other dark triangles, and light triangles (70% or 80% reflectance) only with other light triangles. This was done to maintain the light–dark alternation between adjacent triangles, which supported strong disparity cues (see Figure 3 for an illustration of the pattern of triangle albedos maintained throughout both experiments). 

The same pattern of swapping was applied to both intervals of the trial to produce a non-light change. As a result, each triangle changed by an amount determined by a valid lighting change, but when viewed all together, the triangles appear to shift randomly in reflectance by differing amounts. This procedure was used instead of the 180° permutation so that even local (within-pyramid) changes would be inconsistent with a lighting change. 

To produce an albedo perturbation, 1 face out of the 16 east and west facing faces in the scene (8 pyramids each with 4 faces) was randomly selected, and the albedos of all four inner triangles were multiplied by either 1 − m (becoming darker) or 1 + m (becoming brighter) during interval two, with probability 0.5. The perturbation magnitude, m, was varied at three levels: 0.50, 0.75, and 1.0, corresponding to 50%, 75%, and 100% albedo perturbation, respectively. Because feedback was never given, m was set to 1.0 during the first block of trials, so that observers would begin the experiment by gaining a sense of obvious perturbation detections. (When m was equal to 1.0, some trials included large perturbations where perturbed pixels sometimes turned black or appeared much brighter than the rest.) Light movement direction was fixed to be a rightward movement from perpendicular with rotation magnitude 15°, and the perturbed face was always either an east or west facing face, selected randomly with equal probability. 

We limited perturbation location to these faces in order to make the discrimination as difficult as possible. Because the light source always moved rightward, east and west facing faces underwent the greatest luminance changes under the global change. If the perturbation had occurred on the north or south facing faces, we would expect performance to be near ceiling regardless of global scene change type because the north and south facing faces are relatively unaffected by the global scene change. 

Using this procedure, we rendered 120 perturbation trials and 120 perturbation-free trials for both light and non-light global changes at each of three perturbation levels corresponding to 50%, 75%, or 100% albedo perturbation. 

Luminance signal assignment. The luminance of each triangle within a pyramid was rendered as in Experiment 1. In terms of Equation 7, if triangle j underwent an albedo perturbation during frame 2, the perturbation magnitude, m < 1, was equally likely to be in a negative direction, 

$L^j\left(2\right)=L_{\mathrm{c}}\cos {\theta }_j\left(2\right)(1-m)a^j+L_{\mathrm{d}},$

(10)

causing the triangle to appear dimmer in frame 2, as it was to be in a positive direction, 

$L^j\left(2\right)=L_{\mathrm{c}}\cos {\theta }_j\left(2\right)(1+m)a^j+L_{\mathrm{d}},$

(11)

causing the triangle to appear brighter in frame 2. θ _j (2) is the angle between the direction to the collimated light source and the surface normal of the jth surface in the second frame (see Equation 7 for explanation of the other terms). 

Apparatus and software

The same apparatus was used as in Experiment 1, and the software was updated to run the parameters of the current experiment. 

Procedure

The experimental task was to detect albedo perturbations. Observers viewed virtual three-dimensional landscapes of 4-sided concave and convex pyramids that underwent either illumination changes or matched changes that were scrambled versions of illumination changes. Illumination changes were positional changes of a collimated light source behind the observer. Details of both transformations are specified in the Stimuli section above. A trial was two frames, the first frame 2 s and the second 1 s, presented with immediate replacement. From frame 1 to frame 2, a global change in the scene's luminance distribution occurred, and 50% of the time an additional face albedo perturbation occurred. Observers categorized each trial as either “perturbation” or “no perturbation.” Feedback was never given. Observers initiated each trial by pressing the space bar and were permitted to break at any point during the experiment. 

Observers were instructed that they would be asked to detect trials on which the reflectance of one pyramid face had changed. Because all observers had participated in Experiment 1, they were already familiar with the two global change types and were told that one of the two global change types would occur on every trial and that their task would be to detect the trials on which an additional local change in one pyramid face occurred. To illustrate what a face perturbation would look like, we demonstrated a very large perturbation trial in which one pyramid face almost turned white. Observers were instructed that perturbation present trials were equally likely as perturbation absent trials and that the perturbed face could occur on any of the faces in the scene at random. It was further explained that the degree of perturbation would vary throughout the experiment. 

Trials were blocked by perturbation magnitude and presented in order of ascending difficulty. Each block contained 240 trials, in which perturbation and perturbation-free trials were equally probable and randomized in order. Global change type was maintained throughout a block. In total, with 2 global change types (light and non-light), and 3 levels of albedo perturbation each, with 240 trials per block, there were 8 blocks for a total of 1,440 trials. 

Observers

Three observers from Experiment 1 participated and were reimbursed at the same rate as Experiment 1; their identification numbers are the same as in Experiment 1 (Observers 4 and 5 from Experiment 1 were not available to participate in this experiment). All were unaware of the purpose of the second experiment. 

Results

Albedo perturbation

Sensitivity. We again used d′ to quantify albedo perturbation detection performance as in Experiment 1. For this experiment, we defined the hit rate, p _H, to be the probability of correctly detecting an albedo perturbation, and the false alarm rate, p _F, to be the probability of reporting a perturbation when none had occurred. Ninety-five percent confidence intervals were computed for each d′ estimate as in Experiment 1. For each perturbation level (0.50, 0.75, and 1.0), we computed two separate discriminability measures: one under global lighting changes, and one under scrambled lighting changes. Each estimate was computed using 240 trials from the same block with 120 signal and 120 noise trials each. Separate noise trials were used to estimate d′ for each perturbation level, in order to maintain independence of the estimates. To compare albedo perturbation detection performance under light-induced changes with performance under scrambled global changes, we constructed the following general test comparing light (L) and non-light (NL) estimates: 

$z=\frac{\sum _{\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{s}}(d_L^{\prime }-d_{NL}^{\prime })}{\sqrt{\sum _{\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{s}}(SD(d_L^{\prime }{)}^2+SD(d_{NL}^{\prime }{)}^2})},$

(12)

where standard deviations were computed on the vectors of 100,000 bootstrapped d′ values for each condition (excluding simulated d′ estimates of infinity), and the test has no degrees of freedom. 

Perturbation sensitivity estimates and 95% confidence intervals are shown as a function of perturbation magnitude for each subject separately in Figure 7. For both global change conditions, d′ increased from the lowest perturbation magnitude to the highest. At all levels of perturbation, d′ was greater when the global change was light-induced than when scrambled, the average ratio f = d′_light/d′_non-light was 1.81. The improvement was significant (z = 6.78, p < 0.001). For the lowest perturbation level, the ratio f was 2.4, also a significant light-induced benefit (z = 4.46, p < 0.001). That is, the subjects' d′ values were more than twice as great in detecting surface albedo changes simultaneous with a light transformation than with a matched non-light transformation. As perturbation magnitude and sensitivity increase, the average ratio f decreases. This decrease is likely a ceiling effect. If performance in the non-light transformation condition is close to perfect, then there is little room for improvement in the light transformation condition. 

Figure 7

 

Detection of albedo changes accompanied by light and non-light transformations. On half of the trials, the albedo of one surface patch in the scene was decreased or decreased and the observer attempted to judge whether a surface albedo change had occurred. Data are collapsed over positive and negative perturbations. The perturbation levels are reported as factors applied to the surface's original reflectance, where 1 indicates that the surface turned black or white. We plot data separately for each perturbation level as a bar chart. The d′ values for the three subjects in the light transformation condition are plotted as white bars, and the corresponding d′ values for the non-light transformation condition are plotted as black bars. Chance performance is d′ = 0. The error bars represent 95% confidence intervals for estimated d′. The mean ratio f between d′ values for light and non-light transformations is inset in each panel. We expect a greater effect for smaller albedo perturbations. For a perturbation level of 50%, d′ values in the light transformation condition were on average 2.4 times greater than in the non-light transformation condition.

Discussion

There was a clear benefit to the observer in detecting surface property changes concomitant with lighting changes as compared to matched global luminance distribution changes irreconcilable with a change in the light field. No feedback or training was given during this experiment. Observers were required simply to identify trials on which a surface change had occurred. All of the observers were familiar with the stimuli and the scene changes from Experiment 1. For all observers, there was a demonstrated benefit of approximately overall 1.8 times greater sensitivity (d′) under the lighting change than under the scrambled version. Importantly, at the lowest perturbation magnitude, 0.5, all observers showed a large benefit in the lighting condition, with on average 2.4 greater detection sensitivity d′. Even Observer 2, whose performance did not vary between conditions at the higher perturbation levels, showed a large increase in d′ at the lowest perturbation level when the global change was lighting-induced. 

The value of the albedo for some of the perturbed surfaces could exceed 1, and we therefore considered the possibility that the observer used the presence of albedos effectively greater than 1 as a cue to the presence or absence of a perturbation. We remind the reader that we are testing whether there is a difference in detection of surface changes under light and non-light transformations, and that due to the yoked design of the stimuli, “super-albedo” cues would be equally available under light and non-light transformations. To address the potential impact of any “super albedo” cues, we reanalyzed the data excluding the increment trials where albedos greater than 1 might occur and including only the decrement (1 − m) trials. On these trials, all albedos fell in the range 0 to 1. 

The d′ benefit for decrement trials alone was 5.20 at the lowest perturbation level of 50%, and 2.68 on average, demonstrating an even larger benefit when albedos remained in the range from 0 to 1. Similarly evaluating the increment trials only, we still found a light transformation benefit of 1.75 at 50% perturbation and 1.72 on average. At the 75% and 100% perturbation levels, all observers were twice as sensitive to decrements than to increments in both global transformation contexts, yet all three observers' sensitivity fell to chance at detecting 50% decrement perturbations in non-light transformation trials. Estimates of d′ under all three analyses (decrement only, increment only, and combined) can be compared in Table 1. We note that we did not reuse signal absent trials in the estimates for the decrement or increment trials, but instead randomly split the signal absent trials from the corresponding blocks between the two estimates to maintain their independence. Finally, the ratio of light transformations to non-light is greatest when the perturbation is smallest (see Figure 7), and we would expect the opposite pattern if “super-albedo” cues played any significant role. 

Table 1

 

Sensitivity (d′) to albedo perturbations analyzed separately: decrement (1 − m) perturbations, increment (1 + m) perturbations, and combined (all). We computed separate estimates of d′ for trials on which decrement perturbations had occurred versus trials on which increment perturbations had occurred. Estimates under global non-light changes are printed in the shaded “NL” rows, and under global light changes in the “L” rows, the same color coding used for the bars in Figure 7. Signal absent trials from corresponding blocks were randomly split between the two estimates to maintain their independence. For comparison, the estimates reported in Figure 7 are shown for the combined (all) case.

		50% Perturbation			75% Perturbation			100% Perturbation
		m = 0.50			m = 0.75			m = 1.0
1 − m	NL	0.25*	0.21*	0.25*	1.36	1.26	0.96	Inf.	3.23	1.92
	L	2.00	1.20	0.48	3.25	1.25	1.46	Inf.	3.03	2.80
1 + m	NL	1.08	0.55	0.51	1.10	0.70	0.26*	1.33	1.48	0.65
	L	1.80	0.85	1.04	2.05	0.62	0.77	2.31	1.56	1.14
All	NL	0.75	0.40	0.38	1.23	0.97	0.60	2.03	2.11	1.09
	L	1.90	1.03	0.79	2.82	0.96	1.09	2.96	2.15	1.70
Observer	1	2	3	1	2	3	1	2	3

 

*Indicates the estimate was not significantly different from zero.

Similarly, the albedo of perturbed surfaces are often outside the range of albedos of nearby surfaces on the pyramid (either darker or lighter) and are always outside the overall initial range across all surfaces in the scene. It is therefore possible that the observer used such “range cues” to aid them in detecting the presence of a perturbed surface. Two considerations argue that they did not. Again, because of the design of the stimuli any such “range cues” would be equally available under light and non-light transformations. Therefore, such cues cannot explain why sensitivity to surface change is greater under light transformations than under non-light transformations (Figure 7). If an observer depended primarily on such cues, we would expect little difference in d′ in under light and non-light conditions, contrary to what we found. Moreover, the ratio of d′ in light conditions to d′ in non-light is greatest when the perturbation is smallest and the perturbation effect on the albedo range is smallest. If range violations were an important cue subserving detection, then we would expect the opposite pattern. 

General discussion

Estimating surface color is a primary function of the human visual system, and discounting the light field is linked with this ability because the light signal entering the eye confounds illumination and surface properties. Previous work demonstrates that observers discount spatial variations in the illumination when judging surface color (Boyaci et al., 2004, 2003; Gilchrist, 1977; Ikeda et al., 1998; Ripamonti et al., 2004; Snyder et al., 2005) and can explicitly estimate parameters of the static light field accurately under some circumstances (Khang et al., 2006; Koenderink et al., 2003, 2004; Pont & Koenderink, 2007). In the current paper, we address the human visual system's ability to track changes in the light field and concomitantly detect surface property perturbations. This is the first paper to our knowledge to study detection of light field changes over three-dimensional scenes. 

Little is known about how the visual system accomplishes the feat of discounting the light field or how it represents the light field, the intensity of light arriving at each point in a scene (Gershun, 1936/1939). Foster has demonstrated that observers are exquisitely sensitive to light field changes in two-dimensional (Mondrian) scenes (Craven & Foster, 1992; Nascimento & Foster, 2000). In a two-dimensional scene with a homogeneous light field, light-induced changes preserve edge ratios in photoreceptor excitations, and Foster and colleagues suggest that observers use changes in edge ratios to discriminate surface color changes from changes in scene lighting (Amano & Foster, 2004; Foster & Nascimento, 1994). Utilizing edge ratios in photoreceptor excitations is an example of a retina-based heuristic for detecting changes in surface color. 

In crumpled Mondrians (scenes where surface patches differ in orientation), a change in the direction to a collimated light source induces a complex pattern of changes in the luminances of surface patches across the scene. We refer to this change as a light transformation. Edge ratios are typically not invariant under light transformations in three-dimensional scenes and consequently changes in edge ratio are not a reliable cue to surface change in such scenes. 

We first evaluated the visual system's ability to discriminate light transformations from non-light transformations on three-dimensional achromatic scenes, viewed binocularly. In our stimuli, the only information available about the light field was conveyed by the shading of achromatic matte surfaces. Our scenes lacked a ground plane, highlights, shadows, and any effects of mutual illumination. In our experiment, observers could not rely on edge ratios to inform their discriminations because edge ratio changes were equated in magnitude between the two conditions, as were local and global mean and variance statistics. Our observers needed to use the three-dimensional structure of the scene to infer the light source's direction. 

One possible outcome of this experiment was that observers would be unable to discriminate light and non-light transformations. This outcome would be consistent with the work of Foster and colleagues (Amano & Foster, 2004; Craven & Foster, 1992; Foster & Nascimento, 1994; Nascimento & Foster, 2000). In contrast, we found that all of our observers were able to discriminate these changes from statistic matched changes reliably when the collimated source direction changed in direction by as little as 5°. Furthermore, when observers perceived a change in the light field, they were able to report how it changed with high accuracy. 

This result demonstrates that the human visual system is capable of integrating across the scene to produce a global estimate of light field parameters. This result is rather remarkable given the tendency for observers to be unable to integrate local visual measurements to form a globally consistent judgment (Ho, Landy, & Maloney, 2006; Koenderink, van Doorn, Kappers, & Lappin, 2002). In our task, observers can only discriminate light transformations from non-light if they combine information from more than one pyramid. Since observers do so, we can conclude that they are comparing information from multiple pyramids in discriminating light field changes and also in detecting concomitant changes in surface albedo. 

In a second experiment, we examined observers' ability to detect changes in albedo of isolated surfaces concomitant with a light or non-light transformation. We in effect tested whether observers' ability to detect light transformations would benefit them in detecting changes in luminance due to changes in albedo of isolated surface patches. We found that observers could discount light transformations and achieve greater accuracy in judging surface albedo changes simultaneous with light transformations than with non-light transformations. For the smallest changes in surface albedo, we found that d′ values were more than twice as great for detecting surface changes accompanying a light transformation than those accompanying a non-light transformation. This outcome suggests that at least a rudimentary representation of the light field is maintained by the visual system and utilized for detecting and discounting changes to the light field. 

In all of our scenes, the lighting consisted of a collimated source and a diffuse, and only the collimated light source moved. One evident task for a visual system engaged in estimating changes in scene lighting or estimating scene lighting in static scenes is to determine how many light sources are present in a scene, the spatial characteristics of each source, and where each is located. One possible approach to this problem would be to develop a Bayesian model of scene lighting (the number, nature, and locations of light sources), which likely includes a prior that biases interpretation toward smaller numbers of light sources. This sort of model would be a natural generalization of previous work concerning light direction (Mamassian & Landy, 2001) and number of sources (Doerschner, Boyaci, & Maloney, 2007). 

We are left with two questions. How exactly does the visual system detect light transformations? In a second article (Gerhard & Maloney, under review), we propose a particular model of detection and estimation of changes in the light field in three-dimensional scenes similar to those considered in the current article. The second question concerns the range of light fields and changes in light fields that the visual system is capable of detecting and discounting in estimating surface albedo (and more generally color). With the exception of Doerschner et al. (2007), this range remains unexplored. 

Acknowledgments

This work was supported by a Humboldt Research Award (Alexander v. Humboldt Foundation) to LTM and Grant EY08266 from the National Institutes of Health, National Eye Institute. We thank Michael S. Landy for helpful comments in the development of the stimulus. 

Commercial relationships: none. 

Corresponding author: Holly E. Gerhard. 

Email: [e-mail]. 

Address: Department of Psychology, New York University, 6 Washington Place, 2nd Floor, New York, NY 10003, USA. 

Footnotes

Footnotes

¹  The idea of discounting the illuminant or, more generally, the light field is usually attributed to Helmholtz based on the following quote from the Treatise on Physiological Optics: “[I]n our observations with the sense of vision, we always start out by forming a judgment about the colors of bodies, eliminating the differences of illumination by which a body is revealed to us.” von Helmholtz (1866/1962, p. 287). In modern usage, it typically refers to reducing the effect of differences of illuminations and therefore admits of degrees.

Footnotes

²  We omit consideration of a third possibility, a shattered Mondrian, where all surfaces share a common orientation but are not coplanar as, for example, when they are staggered in depth.

Footnotes

³  Azimuth and elevation correspond to longitude and latitude on a terrestrial sphere.

Footnotes

⁴  We define the term “local scene statistics” in the Methods section of Experiment 1.

Footnotes

⁵  Available from the author at http://bilkent.edu.tr/~hboyaci/PsychWithJava/index.html.