We report two experiments demonstrating that (1) observers are sensitive to information about changes in the light field not captured by local scene statistics and that (2) they can use this information to enhance detection of changes in surface albedo. Observers viewed scenes consisting of matte surfaces at many orientations illuminated by a collimated light source. All surfaces were achromatic, all lights neutral. In the first experiment, observers attempted to discriminate small changes in direction of the collimated light source (light transformations) from matched changes in the albedos of all surfaces (non-light transformations). Light changes and non-light changes shared the same local scene statistics and edge ratios, but the latter were not consistent with any change in direction to the collimated source. We found that observers could discriminate light changes as small as 5 degrees with sensitivity *d*′ > 1 and accurately judge the direction of change. In a second experiment, we measured observers' ability to detect a change in the surface albedo of an isolated surface patch during either a light change or a surface change. Observers were more accurate in detecting isolated albedo changes during light changes. Measures of sensitivity *d*′ were more than twice as great.

*light field*(Gershun, 1936/1939) or, more generally, a

*plenoptic function*(Adelson & Bergen, 1991) across the scene, a specification of the spectral power distribution of light arriving from each direction at each location in the scene.

*light field*

^{1}in judging matte surface color and lightness (Boyaci, Doerschner, & Maloney, 2004; Boyaci, Maloney, & Hersh, 2003; Gilchrist, 1977, 1980; Ikeda, Shinoda, & Mizokami, 1998; Ripamonti et al., 2004; Snyder, Doerschner, & Maloney, 2005; see Maloney, Gerhard, Boyaci, & Doerschner, 2010).

*Mondrian*(Land & McCann, 1971). When the surface patches differ in orientation, we refer to the resulting configuration as a

*crumpled Mondrian,*

^{2}illustrated on the right side of Figures 1A and 1B.

*light change*) will typically alter the luminance of all of the surface patches within a crumpled Mondrian. However, within the space of all possible patterns of surface luminance changes, few are consistent with such a light change. That is, the patterns of luminance change (

*light transformations*) consistent with a light change are highly constrained. Our first goal in this article is to examine human ability to discriminate light and non-light transformations. Before stating this goal precisely, we need to characterize the light transformations expected in Mondrian and crumpled Mondrian scenes.

*Light transformations.*Previous studies of changing illumination have typically used Mondrian scenes with homogeneous light fields as stimuli (Craven & Foster, 1992; Nascimento & Foster, 2000). In two-dimensional scenes illuminated homogeneously (Figure 1, left-hand side), the effect of movement of collimated sources is simply a common scaling of the excitations of the three photoreceptor classes. As [

*ρ*

_{L}

^{ x },

*ρ*

_{M}

^{ x },

*ρ*

_{S}

^{ x }] are the excitations at retinal location

*x*in the long-, middle-, and short-wavelength classes (LMS) of photoreceptors, then the effect of a change in illumination is a scaling of photoreceptor excitations within each class (Maloney, 1999) that depends only on the direction and intensity of the collimated source.

*θ*(

*t*), the angle with respect to the surface normal

**n**common to all the patches in the achromatic Mondrian, and let

*t*denote time. For convenience, we assume −

*π*/2 <

*θ*(

*t*) <

*π*/2 (the light remains on one side of the surface patch). If

*a*

^{ j }denotes the albedo of patch

*j*= 1, …,

*n*within the Mondrian, then the time varying luminance of each patch is, as a consequence of Lambert's Law (Haralick & Shapiro, 1993, pp. 2–7)

*L*is the luminance emitted by a perfectly reflective matte patch (

*a*= 1) when direction to the light source is parallel to the surface normal common to all the patches. We refer to it as the

*light source intensity*.

*j*and

*k*is invariant, equal to

*a*

^{ j }/

*a*

^{ k }. This can be seen in Figure 1 on the left-hand side. All luminances scale by a common factor.

*L*

_{ k }(

*t*) and

*θ*

_{ k }(

*t*) be the time-varying intensities and directions, respectively, of

*k*= 1, …,

*m*collimated light sources. Then the time-varying luminance of a surface patch with fixed albedo

*a*

^{ j }is

*ψ*(

*t*) denote the time-varying azimuth

^{3}and

*ϕ*(

*t*) denote the time-varying elevation of the direction to a collimated light source and let

*ψ*

_{ j }denote the azimuth and

*ϕ*

_{ j }elevation of the

*j*th surface patch,

*j*= 1, …,

*n*. The cosine of the angle between the light direction and the surface normal is then (Green, 1985, p. 11)

*ψ*(

*t*) so that

*ϕ*(

*t*) is a constant, then Equation 4 becomes

*A*

_{ j },

*B*

_{ j }are constants determined by the (constant) elevation of the

*j*th surface patch. Combining Equations 1 and 5,

*C*

_{ j },

*D*

_{ j }are constants determined by the elevation and albedo of the

*j*th surface patch and the intensity of the collimated source. All of the luminances of surface patches are linear transformations of one another but with possibly distinct time lags

*ψ*

_{ j }.

*light transformations*(scene-wide luminance changes induced by change in direction of a collimated source) from statistically matched scene-wide changes that contained the same local scene statistics

^{4}including edge ratio information but were inconsistent with a global light field change. We refer to the latter as

*non-light transformations*. Analogues of the non-light transformations that we use (where many albedos change simultaneously) are present in the natural environment. They occur when autumn leaves are blown in the wind and when milling crowds are viewed at a distance.

*height*/

*base-width*ratios from [0.5, 1.5]. The scenes lacked cast shadows and illumination gradients. The only cue to the spatial distribution of the illumination was the shading of the pyramids (see Figure 3 for an example of a typical scene). Stereo pairs were rendered assuming an interpupillary distance of 6 cm, which was sufficient for all observers to perceive the stimuli in depth.

*Generation.*Stimuli for the two conditions were generated in yoked pairs. A new set of random pyramid heights and face albedos was chosen for each yoked pair. Albedos (in percentages) were one of 40%, 50%, 70%, or 80%. Each pyramid face was divided into four triangles with albedos randomly chosen uniformly from the above distribution. The landscape determined by these constraints on albedo and geometry was then rendered twice: first with the collimated light source perpendicular to the center of the ground plane, and second after a rotation of the collimated light source ±5°, ±10°, ±15°, or ±20° from perpendicular along one of two movement directions with equal probability. The movement directions were roughly horizontal and vertical yet rotated 5° off the principle axes of the stimulus. Rendering two light source positions yielded the two frames of a light transformation trial.

*Luminance signal assignment*. The luminance of each triangle within a pyramid was rendered following Lambert's Law (Haralick & Shapiro, 1993, pp. 2–7), modified to include an additive light component consisting of the diffuse light. The luminance of the

*j*th triangle with albedo

*a*

^{ j }on frame

*t*was determined by

*L*

_{c}is the intensity of the collimated light source,

*L*

_{d}is the intensity of the diffuse light source, and

*θ*

_{ j }(

*t*) is the angle between the

*j*th triangle's surface normal and the direction of the collimated light source at time

*t*. The ratio of the collimated source's intensity to the diffuse source's intensity was fixed throughout the experiment to be 4:1.

^{2}. Fitted gamma values for a simple power function for both displays were 1.7.

^{5}The computer was a Dell Optiplex GX745 with an NVIDIA GeForce 7300 GT dual-DVI graphics card.

*Discriminability.*Discrimination performance was quantified separately for each observer at each magnitude of change by

*d*′ from signal detection theory, a measure which is independent of observer bias (Green & Swets, 1966/1973). We considered light transformation trials as signal trials and non-light trials as non-signal trials. Therefore, we defined the hit rate,

*p*

_{H}, to be the probability of a “light” response when a light transformation occurred, and the false alarm rate,

*p*

_{F}, to be the probability of a “light” response when a non-light transformation occurred. If Φ

^{−1}is the inverse of the cumulative unit normal distribution, then

*d*′ indicates chance performance, and

*d*′ increases with increased discrimination performance. In our task, a

*d*′ = 1 corresponds to 69% correct,

*d*′ = 2 corresponds to 84% correct, and

*d*′ = 3 corresponds to 93% correct. Ninety-five percent confidence intervals for each

*d*′ estimate were obtained by a non-parametric bootstrap method (Efron & Tibshirani, 1993): each observer's performance in the corresponding condition was simulated 100,000 times and the 5th and 95th percentiles were calculated to construct 95% confidence intervals. Separate

*d*′ estimates and 95% confidence intervals are plotted for each observer in Figure 5.

*p*< 0.05. For four out of the five observers, discrimination improved significantly from the lowest signal level (±5° light source rotation) to the highest (±15°). In the following, we report the

*d*′ estimates for each observer and respective

*p*-values for a

*z*-test comparing discriminability at the highest level (±15° light source rotation) with that of the lowest level (±5°). Specifically, we constructed the following general test of increasing performance:

*d*′ values for each condition (excluding simulated

*d*′ estimates of infinity), and the test lacked degrees of freedom.

*d*′ estimates that varied from 2.2 to 3.7,

*z*= 4.38,

*p*< 0.001 (confidence intervals include infinity at the two highest change magnitudes, indicated by the arrows in Figure 5). Observer 2 performed well with

*d*′ estimates increasing from 0.76 to 2.4,

*z*= 5.15,

*p*< 0.001. Observer 3 performed well with

*d*′ estimates increasing from 1.1 to 3.5,

*z*= 7.22,

*p*< 0.001. Observer 4 performed well with

*d*′ estimates increasing almost linearly from 1.7 to 2.6,

*z*= 2.83,

*p*< 0.01. Observer 5 was able to discriminate the two change types above chance at each level and showed marginally significant improvement across the range;

*d*′ estimates increased from 0.73 to 1.1,

*z*= 1.60,

*p*= 0.054.

*how*the light had changed.

*d*′, which describes the ability to discriminate light changes from non-light changes, was equal to 1.30 at the lowest change magnitude tested, equivalent to the luminance change introduced by a 5° rotation of a collimated light source, and by 15°, performance was nearly at ceiling for 4 of the 5 observers.

*Generation.*Stimuli for the two conditions were generated as in Experiment 1, except that the scrambling method for non-light trials was altered. Lighting-consistent changes were rendered first as in Experiment 1, and the luminance signals they generated were scrambled to produce a scene undergoing a global non-light change. Instead of rotating luminance signal assignments within half of the pyramids by 180°, we pseudo-randomly swapped luminance signal assignments across the scene. Swapping was limited such that triangles within north and south faces were always swapped from other north and south facing faces, and east and west only swapped from other east and west facing triangles. This was done to maintain the overall direction of luminance change (up/down versus left/right) within the scene between the light and the non-light trials. Swapping was also limited such that triangles with initially dark albedos (40% or 50% reflectance) were traded only with other dark triangles, and light triangles (70% or 80% reflectance) only with other light triangles. This was done to maintain the light–dark alternation between adjacent triangles, which supported strong disparity cues (see Figure 3 for an illustration of the pattern of triangle albedos maintained throughout both experiments).

*m*(becoming darker) or 1 +

*m*(becoming brighter) during interval two, with probability 0.5. The perturbation magnitude,

*m,*was varied at three levels: 0.50, 0.75, and 1.0, corresponding to 50%, 75%, and 100% albedo perturbation, respectively. Because feedback was never given,

*m*was set to 1.0 during the first block of trials, so that observers would begin the experiment by gaining a sense of obvious perturbation detections. (When

*m*was equal to 1.0, some trials included large perturbations where perturbed pixels sometimes turned black or appeared much brighter than the rest.) Light movement direction was fixed to be a rightward movement from perpendicular with rotation magnitude 15°, and the perturbed face was always either an east or west facing face, selected randomly with equal probability.

*Luminance signal assignment*. The luminance of each triangle within a pyramid was rendered as in Experiment 1. In terms of Equation 7, if triangle

*j*underwent an albedo perturbation during frame 2, the perturbation magnitude,

*m*< 1, was equally likely to be in a negative direction,

*θ*

_{ j }(2) is the angle between the direction to the collimated light source and the surface normal of the

*j*th surface in the second frame (see Equation 7 for explanation of the other terms).

*additional*local change in one pyramid face occurred. To illustrate what a face perturbation would look like, we demonstrated a very large perturbation trial in which one pyramid face almost turned white. Observers were instructed that perturbation present trials were equally likely as perturbation absent trials and that the perturbed face could occur on any of the faces in the scene at random. It was further explained that the degree of perturbation would vary throughout the experiment.

*Sensitivity.*We again used

*d*′ to quantify albedo perturbation detection performance as in Experiment 1. For this experiment, we defined the hit rate,

*p*

_{H}, to be the probability of correctly detecting an albedo perturbation, and the false alarm rate,

*p*

_{F}, to be the probability of reporting a perturbation when none had occurred. Ninety-five percent confidence intervals were computed for each

*d*′ estimate as in Experiment 1. For each perturbation level (0.50, 0.75, and 1.0), we computed two separate discriminability measures: one under global lighting changes, and one under scrambled lighting changes. Each estimate was computed using 240 trials from the same block with 120 signal and 120 noise trials each. Separate noise trials were used to estimate

*d*′ for each perturbation level, in order to maintain independence of the estimates. To compare albedo perturbation detection performance under light-induced changes with performance under scrambled global changes, we constructed the following general test comparing light (

*L*) and non-light (

*NL*) estimates:

*d*′ values for each condition (excluding simulated

*d*′ estimates of infinity), and the test has no degrees of freedom.

*d*′ increased from the lowest perturbation magnitude to the highest. At all levels of perturbation,

*d*′ was greater when the global change was light-induced than when scrambled, the average ratio

*f*=

*d*′

_{light}/

*d*′

_{non-light}was 1.81. The improvement was significant (

*z*= 6.78,

*p*< 0.001). For the lowest perturbation level, the ratio

*f*was 2.4, also a significant light-induced benefit (

*z*= 4.46,

*p*< 0.001). That is, the subjects'

*d*′ values were more than twice as great in detecting surface albedo changes simultaneous with a light transformation than with a matched non-light transformation. As perturbation magnitude and sensitivity increase, the average ratio

*f*decreases. This decrease is likely a ceiling effect. If performance in the non-light transformation condition is close to perfect, then there is little room for improvement in the light transformation condition.

*d*′) under the lighting change than under the scrambled version. Importantly, at the lowest perturbation magnitude, 0.5, all observers showed a large benefit in the lighting condition, with on average 2.4 greater detection sensitivity

*d*′. Even Observer 2, whose performance did not vary between conditions at the higher perturbation levels, showed a large increase in

*d*′ at the lowest perturbation level when the global change was lighting-induced.

*m*) trials. On these trials, all albedos fell in the range 0 to 1.

*d*′ benefit for decrement trials alone was 5.20 at the lowest perturbation level of 50%, and 2.68 on average, demonstrating an even larger benefit when albedos remained in the range from 0 to 1. Similarly evaluating the increment trials only, we still found a light transformation benefit of 1.75 at 50% perturbation and 1.72 on average. At the 75% and 100% perturbation levels, all observers were twice as sensitive to decrements than to increments in both global transformation contexts, yet all three observers' sensitivity fell to chance at detecting 50% decrement perturbations in non-light transformation trials. Estimates of

*d*′ under all three analyses (decrement only, increment only, and combined) can be compared in Table 1. We note that we did not reuse signal absent trials in the estimates for the decrement or increment trials, but instead randomly split the signal absent trials from the corresponding blocks between the two estimates to maintain their independence. Finally, the ratio of light transformations to non-light is greatest when the perturbation is smallest (see Figure 7), and we would expect the opposite pattern if “super-albedo” cues played any significant role.

50% Perturbation | 75% Perturbation | 100% Perturbation | ||||||||
---|---|---|---|---|---|---|---|---|---|---|

m = 0.50 | m = 0.75 | m = 1.0 | ||||||||

1 − m | NL | 0.25* | 0.21* | 0.25* | 1.36 | 1.26 | 0.96 | Inf. | 3.23 | 1.92 |

L | 2.00 | 1.20 | 0.48 | 3.25 | 1.25 | 1.46 | Inf. | 3.03 | 2.80 | |

1 + m | NL | 1.08 | 0.55 | 0.51 | 1.10 | 0.70 | 0.26* | 1.33 | 1.48 | 0.65 |

L | 1.80 | 0.85 | 1.04 | 2.05 | 0.62 | 0.77 | 2.31 | 1.56 | 1.14 | |

All | NL | 0.75 | 0.40 | 0.38 | 1.23 | 0.97 | 0.60 | 2.03 | 2.11 | 1.09 |

L | 1.90 | 1.03 | 0.79 | 2.82 | 0.96 | 1.09 | 2.96 | 2.15 | 1.70 | |

Observer | 1 | 2 | 3 | 1 | 2 | 3 | 1 | 2 | 3 |

*d*′ in under light and non-light conditions, contrary to what we found. Moreover, the ratio of

*d*′ in light conditions to

*d*′ in non-light is greatest when the perturbation is smallest and the perturbation effect on the albedo range is smallest. If range violations were an important cue subserving detection, then we would expect the opposite pattern.

*light transformation*. Edge ratios are typically not invariant under light transformations in three-dimensional scenes and consequently changes in edge ratio are not a reliable cue to surface change in such scenes.

*d*′ values were more than twice as great for detecting surface changes accompanying a light transformation than those accompanying a non-light transformation. This outcome suggests that at least a rudimentary representation of the light field is maintained by the visual system and utilized for detecting and discounting changes to the light field.

^{1}The idea of discounting the illuminant or, more generally, the light field is usually attributed to Helmholtz based on the following quote from the

*Treatise on Physiological Optics*: “[I]n our observations with the sense of vision, we always start out by forming a judgment about the colors of bodies, eliminating the differences of illumination by which a body is revealed to us.” von Helmholtz (1866/1962, p. 287). In modern usage, it typically refers to reducing the effect of differences of illuminations and therefore admits of degrees.