A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers’ internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest – within a factor of two of the equivalent noise level set by internal noise.

^{1}In these experiments, the target object's reflectance is the task-relevant scene variable, whereas other aspects of the scene are task irrelevant. Observers’ reports are solicited using a variety of methods, including matching (e.g. Burnham, Evans, & Newhall, 1952; Gilchrist, 1977; Arend & Reeves, 1986; Brainard, Brunt, & Speigle, 1997), naming (e.g. Helson & Jeffers, 1940; Olkkonen, Witzel, Hansen, & Gegenfurtner, 2010), scaling (e.g. Schultz, Doerschner, & Maloney, 2006), and nulling (e.g. Helson & Michels, 1948; Jameson & Hurvich, 1955; Chichilnisky & Wandell, 1997; Brainard, 1998).

^{2}We used a two-alternative forced-choice (2AFC) procedure (Figure 1). On each trial, observers viewed a standard image and comparison image, sequentially presented on a calibrated monitor for 250 ms each. The inter-stimulus interval (ISI) was 250 ms (see Figure 1a). The images were computer graphics renderings of 3D scenes. Each scene contained a spherical target object that appeared achromatic. The observers’ task was to report the image in which the target object was lighter. Across trials, we varied the luminous reflectance factor (LRF; American Society for Testing and Materials, 2017) of the target object in the comparison image while keeping the LRF of the target object in the standard image fixed. The LRF is the ratio of the luminance of a surface under a reference illuminant (here, the Commission Internationale de l' Éclairage [CIE] D65 reference illuminant) to the luminance of the reference illuminant itself. The target object LRF was varied by scaling the surface reflectance spectrum of the target object, without changing its shape.

^{3}The temporal order in which the standard and comparison images were presented was randomized on each trial.

^{4}

^{2}. The average luminances of the target object for the 11 LRF levels were 67.0, 68.0, 68.9, 69.8, 70.7, 71.6, 72.5, 73.4, 74.2, 75.1, and 75.9 cd/m

^{2}.

^{2}, -3.2). This exclusion criterion was specified in our preregistered protocol (see Methods: Preregistration).

*z*. This variable depends on the image and is perturbed by noise. We assume that for any fixed image,

*z*is a normally distributed random variable whose mean depends on the target object LRF. For each image, we assume that

*z*is perturbed on a trial-by-trial basis by independent zero mean normally distributed noise, and we assume that the variance of this noise is the same for the response to all images. We refer to the noise that perturbs

*z*for a fixed image as the internal noise and denote its variance as \(\sigma _i^2.\) For each trial of the experiment,

*z*takes on two values,

*z*and

_{s}*z*, one for the interval containing the standard and the other for the interval containing the comparison.

_{c}*z*for each pair of LRFs, µ

_{s}and µ

_{c}, and on the value of \(\sigma _i^2.\) In our experimental design, we have ensembles of images with different backgrounds for each value of the target object LRF and background covariance scalar. The fact that we draw stochastically from these ensembles on each trial introduces additional variability into the value of the decision variable

*z*that corresponds to a fixed target LRF. We call this the external variability, and model it as a normal random variable with zero mean and variance \(\sigma _e^2.\) We assume that \(\sigma _e^2\) depends on the experimentally chosen covariance scalar, but not on the target sphere LRF. Thus, the distributions of

*z*and

_{s}*z*, for a particular choice of target standard and comparison LRF and covariance scalar, are given by

_{c}*P*(

*z*) =

_{s}*N*(µ

_{s},σ

_{t}) and

*P*(

*z*) =

_{c}*N*(µ

_{c},σ

_{t}). Here, µ

_{s}is the mean value of the internal representation to the standard image and µ

_{c}is the mean value of the internal representation to the comparison image. The overall standard deviation σ

_{t}is obtained via \(\sigma _t^2 = \sigma _i^2 + \sigma _e^2\), where \(\sigma _i^2\) and \(\sigma _e^2\) are the variance of the internal and external noise.

*z*and

_{s}*z*, choosing the interval with the higher value of

_{c}*z*as that with the higher stimulus value. The observer's sensitivity depends on the mean values and the variance of

*z*, and is captured by the quantity d-prime:

*d*′ = (µ

_{c}− µ

_{s})/σ

_{t}. D-prime measures the distance between the two distributions in standard deviation units. A value of

*d*′ = 0 corresponds to an inability to distinguish between the standard and the comparison image. Larger values of

*d*′ indicate increasing discriminability.

_{c}− µ

_{s}) is proportional to the difference in the LRFs of the target object in the standard and comparison images (Δ

_{LRF}). That is, (µ

_{c}− µ

_{s}) =

*C*Δ

_{LRF}, where

*C*is the proportionality constant. This yields the following:

_{LRF}that corresponds to that proportion correct. Our choice of 0.76 corresponds to

*d*′ = 1. In addition we can choose

*C*= 1, in essence setting the units for

*z*to match those of the target LRF.

^{5}To change the amount of external noise, we scaled the covariance of the multivariate normal distribution by multiplying its covariance matrix with a scalar. Thus, for our experiments we have:

^{2}is the covariance scalar and \(\sigma _{e0}^2\) is the external noise introduced when the ensemble of images for each value of target LRF has the reflectance of the background objects drawn from our model of natural reflectances.

^{2}should increase monotonically. For small values of σ

^{2}(\({\sigma ^2} \ll \sigma _i^2\)/\(\sigma _{e0}^2\)), the threshold will approach a constant giving \(\log ( {{\rm{\Delta }}_{{\rm{LRF}}}^2} )\sim\;\log ( {\sigma _i^2} ).\) For large values of σ

^{2}(\({\sigma ^2} \gg \sigma _i^2\)/\(\sigma _{e0}^2\)), the quantity \(\log ( {{\rm{\Delta }}_{{\rm{LRF}}}^2} )\) will approach a straight line with slope 1 in the \(\log ( {{\rm{\Delta }}_{{\rm{LRF}}}^2} )\) versus log (σ

^{2}) plot. Fitting the measurements with Equation 4 allows us to check whether the model describes the data, as well as to determine the two parameters \(\sigma _i^2\) and \(\sigma _{e0}^2\). In particular, we can establish the relative contribution of the internal representational variability and external stimulus variability in limiting lightness discrimination. The parameter \(\sigma _{e0}^2\) quantifies how much the variation in background object reflectances intrudes on the internal representation

*z*that mediates the lightness discrimination task. The value of \(\sigma _{e0}^2\) may be compared directly to the intrinsic precision of that representation characterized by \(\sigma _i^2.\)

*z*as the internal noise. The external variation is characterized experimentally by the covariance scalar (together with the underlying model of natural reflectances which is held fixed across the experiments). Once the model parameters \(\sigma _i^2\) and \(\sigma _{e0}^2\) are determined from the data, we can find the covariance scalar \(\sigma _{equiv}^2\) that produces externally generated equivalent noise

*I*, and the receptive field by the column vector

*R*. The entries of

*I*are the radiant power emitted by the monitor at each image location. The entries of

*R*are the corresponding sensitivities of the linear receptive field to each entry of

*I*. The response of the receptive field is given as

*r*=

_{i}*R*+ η

^{T}I_{i}, where η

_{i}is a random variable representing a draw of zero mean normally distributed internal noise (variance \(\sigma _{ri}^2\)) in the receptive field response for a fixed image. We assume that \(\sigma _{ri}^2\) is independent of

*I*.

*I*

_{s0}and

*I*

_{c0}as the standard and comparison images without external noise. External normally distributed noise is added to both

*I*

_{s0}and

*I*

_{c0}, with covariance matrix Σ

_{e}. The external noise need not have zero mean. After incorporation of the external noise, the response of the receptive field to the comparison and standard images is given by the following:

_{e}is a random variable representing a draw of external noise, η

_{i}represents the internal noise, and η is a random variable representing the overall effect of the external and internal noise. Because the receptive field and noise models are linear and normal, η is normal with variance

_{c}− µ

_{s}) =

*R*(

^{T}*I*

_{c0}−

*I*

_{s0}) =

*C*′Δ

_{LRF}. Here,

*I*

_{s0}and

*I*

_{c0}are the standard and comparison images without external noise added,

*C*′ is a constant, and Δ

_{LRF}is as defined is the SDT section above. The second equality follows because (1) the difference between

*I*

_{c0}and

*I*

_{s0}is proportional to Δ

_{LRF}as only the target LRF changes between these two images, and (2) even if the mean of the external noise is non-zero, its effect cancels when we obtain the mean difference in response.

*z*of the SDT formulation developed above. That is, we assume that on each trial the observer chooses as lighter the interval for which the response of the receptive field is greater. Following the development of the SDT formulation, we have the following:

^{2}in the term corresponding to the variance of the external noise, and where Σ

_{e0}denotes the covariance matrix of the external noise corresponding to the level of variation in natural images. Comparing to relation derived in the SDT model (Equation 3), we see that this is the same functional form for the relation between Δ

_{LRF}and σ

^{2}as derived there, where we associate \(\sigma _i^2 = \frac{{\sigma _{ri}^2}}{{{{({C^{\prime}})}^2}\;}}\) and \(\sigma _{e0}^2 = \frac{{({R^T}{{\rm{\Sigma }}_{e0}}R)}}{{{{({C^{\prime}})}^2}}}\).

*R*and \(\sigma _i^2\) that best account for the data, we then find \(\sigma _{e0}^2\) directly by passing the images corresponding to σ

^{2}= 1 through the receptive field and finding the resulting variance. These parameters in turn allow us to compute \(\sigma _{equiv}^2\) and \(\sigma _{enl}^2\) for the LINRF formulation.

*fmincon*. The best fitting parameters were estimated separately for the mean observer and the individual observers.

_{c}= 1, and each region of the surround had sensitivity denoted by v

_{s}. The RF was the same for each of the three cone classes. The RF response was taken as the sum of the L, M, and S RF component responses. Normally distributed internal noise with zero mean was added to the resulting dot product. The variance of the internal noise (σ

_{ri}) and the value of the RF surround sensitivity (v

_{s}) were the two parameters of the model.

_{ri}) and the value of the RF surround (v

_{s}). The mean squared error values obtained as a function of these two parameters were fit with a degree two polynomial of two variables using the MATLAB

*fit*function. The resulting polynomial was evaluated to estimate the parameters with lowest mean square error. These parameters were then used to estimate the internal and external noise standard deviation of the LINRF formulation using the relations: \(\sigma _i^2 = \frac{{\sigma _{ri}^2}}{{{{({C^{\prime}})}^2}\;}}\) and \(\sigma _{e0}^2 = \frac{{({R^T}{{\rm{\Sigma }}_{e0}}R)}}{{{{({C^{\prime}})}^2}}}\) as explained above, where the constant

*C*′ was obtained by solving

*R*(

^{T}*I*

_{c0}−

*I*

_{s0}) =

*C*′Δ

_{LRF}.

^{2}). We measured discrimination thresholds of four human observers at six values of the covariance scalar. The threshold was measured three times (3 separate blocks) for each observer and for each value of covariance scalar. The psychometric functions for each block/covariance scalar value are shown for one observer in Figure 4 and for all observers in Supplementary Figure S3. Inspection of the psychometric functions shows that their slopes steadily decrease with increasing covariance scalar, corresponding to an increase in thresholds.

*N*= 4) is plotted against the log of the covariance scalar. Figure 6 plots thresholds in the same format for the individual observers, with the data averaged over the three blocks for each covariance scalar. The choice to plot the data as log threshold-squared against the log of the covariance scalar was motivated by the relatively simple expression of the SDT model formulation's predictions for this representation (see Equation 4 and the following text). Table S2 provides the thresholds and SEMs from Figure 6 in tabular form.

^{2}= 0) establishes the level of the internal noise, while the way threshold increases with covariance scalar determines \(\sigma _{e0}^2\). The fits determine the parameters of the model as well as allows us to examine how well the model fits the data.

_{i}and σ

_{e0}), for both the SDT model and the LINRF formulation. There is good consistency in the value of σ

_{i}across observers, the model's manifestation of the observations that thresholds for low covariance scalars are similar across observers. There is more variability in σ

_{e0}across observers, corresponding to the individual variability seen in the rising limb of the threshold versus covariance scalar plots.

_{i}for the LINRF formulation are close to those obtained with the SDT formulation (SDT formulation: mean value of internal noise standard deviation across observers 0.0256, value from fit to mean data 0.0256; LINRF formulation: mean value of internal noise standard deviation across individual observers 0.0250, value from fit to mean data, 0.0250).

_{e0}are higher for the LINRF formulation than for the SDT formulation (SDT formulation: mean value of external noise standard deviation 0.0290, value from fit to mean data across observers 0.0294; LINRF formulation: mean value of external noise standard deviation across observers 0.0421, value from fit to mean data, 0.0429). This is consistent with the observation that the SDT formulation underestimates the rise in thresholds with increasing covariance scalar, whereas this rise is captured more accurately by the LINRF formulation, presumably because the latter incorporates the constraint that the reflectance values at each wavelength are physically realizable (i.e. reflectances lie between 0 and 1).

_{enl}) corresponding to covariance scalar σ

^{2}= 1, the level of background object reflectance variation corresponding to our full model of natural reflectance. For the fits to the mean data, this equivalent noise level is approximately 1.7. This as well as values for the individual observers are plotted in the right panel of Figure 7. This tells us that, for our experimental conditions, the variability in the human representations of lightness induced by naturally occurring variation in the background object reflectances is within a factor of two of the limits imposed by the intrinsic precision of that representation. Had the value been closer to one, we would have concluded that the visual system had discounted the effect of variation in the background object reflectances about as required, given the intrinsic precision of the lightness representation. The fact that the equivalent noise level is higher than one but not tremendously so is consistent with the idea that the visual system has a degree of lightness constancy, but that this constancy can be incomplete (see e.g. Gilchrist, 2006; Kingdom, 2011; Murray, 2021).

_{enl}), that is relative to the standard deviation of the intrinsic noise. In this way, we use the intrinsic noise as a benchmark to interpret the magnitude of the equivalent noise from the external variation. We find that the effect of the external variability introduced by variation of background object reflectances in naturalistic scenes is within a factor of two of the intrinsic precision of the lightness representation. More generally, our work provides a method to quantify the effect of variation in a task-irrelevant properties on the perception of task-relevant property, and is thus applicable to understanding other perceptual constancies beyond the lightness constancy we focused on here.

*Neural Computation*, 19, 2610–2637. [CrossRef] [PubMed]

*Science*, 262(December 24), 2042–2044. [PubMed]

*The New Cognitive Neurosciences*, 2nd edition (pp. 339–351). Cambridge, MA: MIT Press.

*Journal of Vision*, 13(7), 1–18. [CrossRef]

*PLoS One*, 12(6), e0180310. [CrossRef] [PubMed]

*Renovations of Center for Historic Preservation*, 98(A), E1477.

*Journal of the Optical Society of America A*, 3(10), 1743–1751. [CrossRef]

*Journal of Vision*, 19(3), 15. [PubMed]

*Vision Research*, 27(11), 1915–1924. [CrossRef] [PubMed]

*Journal of Vision*, 4(9), 735–746. [CrossRef] [PubMed]

*Journal of Vision*, 3(8), 541–553. [CrossRef] [PubMed]

*Color Research & Application*, 14(1), 23–34.

*Journal of the Optical Society of America A*, 15(2), 307–325.

*Annual Review of Vision Science*, 1, 519–546. [PubMed]

*Journal of the Optical Society of America A*, 14(9), 2091–2110.

*Journal of the Optical Society of America A*, 14(7), 1393–1411.

*Journal of Vision*, 6(11), 1267–1281. [PubMed]

*Journal of Vision*, 11(5), 10.

*Encylopedia of Imaging Science and Technology*(pp. 172–188). Hoboken, NJ: John Wiley & Sons.

*The New Visual Neurosciences*, 1, 545–556.

*Annual Review of Vision Science*, 7, 465–486. [PubMed]

*Physiology of the Retina and the Visual Pathway*. London, UK: Arnold.

*Current Biology*, 7, 844–849. [PubMed]

*Annual Review of Neuroscience*, 6, 491–517.

*Proceedings of the National Academy of Sciences*, 108(40), 16849–16854.

*Journal of Vision*, 14(2), 1. [PubMed]

*Nature Communications*, 6, 7900. [PubMed]

*PLoS Computational Biology*, 13(2), e1005281. [PubMed]

*Journal of the Optical Society of America*, 42(9), 597–605. [PubMed]

*Vision Research*, 37, 616.

*Journal of Neuroscience*, 40(4), 864–879. [PubMed]

*Fundamental chromaticity diagram with physiological axes – Parts 1 and 2. Technical Report 170-1*. Vienna, Austria: Central Bureau of the Commission Internationale de l' Éclairage.

*Psychonomic Science*, 1, 369–370.

*Journal of Neuroscience*, 29(20), 6635–6648. [PubMed]

*Journal of Vision*, 19(4), 8. [PubMed]

*Elements of Psychophysics*(H. E. Adler, 1966, Trans.). New York, NY: Holt, Rinehart and Winston.

*Journal of the Optical Society of America A Optical Image Science Vision*, 35(4), B334–B346.

*Vision Research*, 182, 89–100. [PubMed]

*Vision Research*, 51(7), 674–700. [PubMed]

*Journal of the Optical Society of America A*, 9(11), 1880–1888.

*Annual Review of Psychology*, 59, 167–192. [PubMed]

*Journal of Vision*, 9(13):17, 11–16.

*Science*, 195, 185. [PubMed]

*Seeing Black and White*. Oxford, UK: Oxford University Press.

*Vision Research*, 38, 3913–3926. [PubMed]

*Signal Detection Theory and Psychophysics*(Vol. 1). Hoboken, NJ: John Wiley & Sons.

*Journal of Vision*, 14(2), 6. [PubMed]

*Physiological Optics*. New York, NY: Dover Publications, Inc.

*Journal of Experimental Psychology*, 26(1), 1–27.

*Journal of the Optical Society of America*, 38, 1025–1032. [PubMed]

*Journal of the Optical Society of America*, 71(5), 574–581. [PubMed]

*Journal of the Optical Society of America A*, 22(10), 2090–2106.

*Current Biology*, 17(19), 1714–1719.

*Journal of the Optical Society of America A*, 24(8), 2122–2133.

*Computer Vision, Graphics, and Image Processing*, 3, 277–299.

*Current Opinion in Behavioral Sciences*, 30:186, 186–193.

*Tests for colour-blindness*. Tokyo: Kanehara Shuppen Company, Ltd.

*Journal of Vision*, 17(12), 16. [PubMed]

*Journal of the Optical Society of America*, 45, 546–552.

*Electronic Imaging,*2016(20), 1–4.

*Journal of the Optical Society of America*, 33(7), 355–376.

*Vision Research*, 51(7), 652–673. [PubMed]

*Perception as Bayesian Inference*. Cambridge, MA: Cambridge University Press.

*Perception*, 31, 247–263. [PubMed]

*Journal of the Optical Society of America*, 61(1), 1–11. [PubMed]

*Journal of the Optical Society of America A*, 4(2), 391–404.

*Journal of the Optical Society of America A*, 12(2), 250–260.

*Nature Neuroscience*, 2(11), 1010–1014. [PubMed]

*Journal of the Optical Society of America A*, 3(10), 1673–1683.

*Colour Perception: From Light to Object*(pp. 335–358). Oxford, UK: Oxford University Press.

*Journal of the Optical Society of America A*, 11(12), 3113–3122.

*Visual Neuroscience*, 21, 291–294. [PubMed]

*Journal of the Optical Society of America A Optical Image Science Vision*, 35(4), B244–B255.

*Journal of Vision*, 20(7), 28. [PubMed]

*Annual Review of Vision Science*, 7, 417–436. [PubMed]

*Vision Research*, 39(6), 1133–1142. [PubMed]

*Vision Research*, 14(10), 1039–1042. [PubMed]

*Annual Review of Neuroscience*, 35, 463–483. [PubMed]

*Journal of Vision*, 10(9), 16. [PubMed]

*Annual Review of Neuroscience*, 21(1), 227–277. [PubMed]

*Journal of the Optical Society of America*, 6(2), 318–322.

*PLoS One*9(2:e87989), e87989. [PubMed]

*Vision: Coding and Efficiency*(pp. 3–24). Available from https://www.sciencedirect.com/topics/earth-and-planetary-sciences/quantum-efficiency.

*Journal of the Optical Society of America A*, 16(3), 647–653.

*Frontiers in Psychology*, 9, 1250. [PubMed]

*Journal of Experimental Psychology. Human Perception and Performance*, 42(6), 847–865. [PubMed]

*Journal of Vision*, 15(6), 1–19.

*Journal of Vision*, 18(5), 11. [PubMed]

*i-Perception*, 4(6), 437–455. [PubMed]

*Journal of Vision*, 16(11:2), 1–18.

*The First Steps in Seeing*. Sunderland, Massachusetts: Sinauer.

*Vision Research*, 32(4), 631–637. [PubMed]

*Vision Research*, 39, 533–539. [PubMed]

*Annual Review of Neuroscience*, 41, 77–97. [PubMed]

*Journal of the Optical Society of America A*, 14(10), 2633–2646.

*Journal of Vision*, 6(10), 1102–1116. [PubMed]

*Proceedings of the National Academy of Sciences*, http://doi.org/10.1073/pnas.1619487114.

*Proceedings of the National Academy of Sciences,*117(47), 29363–29370, http://doi.org/10.1073/pnas.1912331117.

*Journal of Neuroscience*, 16, 1486–1510. [PubMed]

*Journal of Vision*, 18(13), 19. [PubMed]

*Philosophical Transactions of the Royal Society of London. Series B*, 360(1458), 1329–1346.

*Vision Research*, 24(10), 1233–1246. [PubMed]

*Journal of the Optical Society of America A*, 19(12), 2329–2348.

*Sources of Color Science (1970)*(pp. 120–1126). Cambridge, MA: MIT Press.

*Color Research & Application*, 19(1), 4–9.

*The New Handbook of Mathematical Psychology*. Cambridge, UK: Cambridge University Press.

*Nature*, 373(6516), 694–698. [PubMed]

*i-Perception*, 8(6), 204166951773963.

*Vision Research*, 9(9), 1095–1110. [PubMed]

*Annual Review of Vision Science*, 4, 475–499. [PubMed]

*Vision Research*, 41, 2581–2600. [PubMed]

*Condition 1. Fixed background:*In this condition, the spectra of objects in the background were kept fixed for all trials and for all intervals. We generated 11 images, one at each comparison LRF level.

*Condition 2. Between-trial background variation:*In this condition, the spectra of the objects in the background were the same for the two intervals within a trial but varied from trial-to-trial.

*Condition 3. Within-trial background variation:*In this condition, the spectra of the objects in the background varied between trials as well as between the two intervals of a trial. The background variation corresponded to covariance scalar equal to 1.

*Condition 2a. Between-trial background variation without secondary reflection:*Same as condition 2, but without multiple reflections of light from object surfaces. The light rays only bounce off once from the surfaces before coming to the camera.

*Condition 3a. Within-trial background variation without secondary reflections:*Same as condition 3, but without multiple reflections of light from object surfaces. Condition 3a was the same as the experiment reported in the main paper for covariance scalar equal to 1.