Color and brightness encoded in a common L- and M-cone pathway with expansive and compressive nonlinearities

Abstract

Abstract

Abstract:

Abstract  Lights near 560 nm appear brighter when flickered, whereas lights near 520 or 650 nm appear yellower. Both effects are consistent with signal distortion within the visual pathway—brightness changes at an expansive nonlinearity, and hue shifts at a compressive one. We previously manipulated the distortion products generated by each nonlinearity to extract the temporal properties of stages of the L- and M-cone pathways that signal brightness and color before (early stages) and after (late stages) each nonlinearity. We find that the attenuation characteristics of the early and late stages are virtually identical in both pathways: The early temporal stage acts like a band-pass filter peaking at 10–15 Hz, while the late stage acts like low-pass filter with a cut-off frequency near 3 Hz. We propose a physiologically relevant model that accounts for the filter shapes and incorporates both nonlinearities within a common parvocellular pathway. The shape of the early band-pass filter is consistent with antagonism between center signals and more sluggish and delayed surround signals, while the late filter is consistent with a simple two-stage low-pass filter. Modeling suggests that the brightness change and hue shift are both initially caused by the half-wave rectification and partition of signals into ON and OFF components. However, the hue shift is probably caused by the additional effects of a later nonlinearity that compresses chromatic red and green signals. Plausible sites for the expansive half-wave rectifying nonlinearity are after surround antagonism, possibly from horizontal cells, but the compressive nonlinearity is likely to be after the late filter.

Introduction

Flickering a light of constant time-averaged intensity may enhance its brightness (e.g., Bartley, 1938, 1939, 1951a, 1951b; Brewster, 1838; Brücke, 1848) or alter its hue (e.g., Ball, 1964; Ball & Bartley, 1966, 1971; Bartley & Nelson, 1960; Stewart, 1887; van der Horst & Muis, 1969). Both changes are consistent with nonlinearities that affect the appearance of the visual input by distorting its representation within the visual pathway. Yet, brightness enhancement and hue change seem to be distinct effects with different spectral sensitivities (Ball, 1964; Ball & Bartley, 1971; Petrova, Henning, & Stockman, 2013a, 2013b; van der Horst & Muis, 1969; Walters & Harwerth, 1978). Consequently, it is possible to choose target wavelengths that favor either brightness enhancement or hue change. Accordingly, we used a 560-nm target that changed in brightness when flickered, and a 650-nm target that changed in hue (Petrova et al., 2013a, 2013b). The two effects also seem to be generated by different nonlinearities: The enhancement in brightness is consistent with an expansive nonlinearity (e.g., Petrova et al., 2013b; Wu, Burns, Reeves, & Elsner, 1996), whereas the hue change in the red-green spectral range towards yellow is consistent with a compressive nonlinearity (e.g., Petrova et al., 2013b; van der Horst & Muis, 1969). 

The distinct nature of the two effects led us to consider the possibility that the responsible nonlinearities might lie in different postreceptoral pathways—the one causing hue change lying in a chromatic pathway, and the other causing brightness enhancement in a luminance or brightness pathway. In our two previous papers, we used the nonlinearities to investigate separately putative chromatic and brightness pathways (Petrova et al., 2013a, 2013b). In both cases, we applied a linear-nonlinear-linear “sandwich” model (e.g., Burns, Elsner, & Kreitz, 1992; Burton, 1973; Chen & Makous, 1990; Chen, Makous, & Williams, 1993; Christiansen, D'Antona, & Shevell, 2009; MacLeod, Williams, & Makous, 1992; Marmarelis & Marmarelis, 1978; Spekreijse & Reits, 1982; Stockman & Plummer, 1998; Trimble & Phillips, 1978; Victor & Shapley, 1980; Victor, Shapley, & Knight, 1977) to extract the temporal properties of the early (pre-nonlinearity) and late (post-nonlinearity) stages of each pathway. Using monochromatic 650-nm flicker, we extracted the properties of the chromatic pathway (Petrova et al., 2013b), and using monochromatic 560-nm flicker those of the brightness pathway (Petrova et al., 2013a). Here, we compare the results, and come to the surprising conclusion that although the nonlinearities that change hue and brightness seem to be different, they may lie in the same pathway. In this paper we combine new data with our previous measurements to generate a physiologically plausible model. 

The sandwich model has been used extensively in vision research (e.g., Burns et al., 1992; Burton, 1973; Chang, Kreitz, & Burns, 1993; Henning, Hertz, & Broadbent, 1975; Krauskopf, Wu, & Farell, 1996; MacLeod & He, 1993; MacLeod et al., 1992; Stockman & MacLeod, 1986; Stockman, MacLeod, & Lebrun, 1993; Williams, 1985). In most cases, its application relies on the assumption that internal nonlinearities generate new visual signals within the visual pathway by distorting the visual input signal and so changing its frequency spectrum and hence its appearance (Bedrosian & Rice, 1971). In our previous experiments, the visual input was a contrast-modulated stimulus produced by sinusoidally flickering a light of 650 or 560 nm at f_c Hz and then sinusoidally modulating its contrast at a much lower frequency, f_m Hz (see Figure 1). The frequency components of this stimulus, in addition to the component at 0 Hz corresponding to the mean level around which the light is modulated, consist of three high-frequency sinusoids: one at f_c Hz, the carrier frequency, and two side bands at f_c − f_m and f_c + f_m Hz. There is no component at f_m Hz in this stimulus until the signal encounters a nonlinearity and undergoes distortion. Distortion of contrast-modulated flicker by a typical compressive or expansive nonlinearity produces components at higher harmonics of the carrier and side bands in addition to intermodulation distortion products. The particular intermodulation distortion product that concerns us is the one at f_m Hz, because we see brightness enhancements and hue changes that vary at that frequency. We assume that it is the distortion product at f_m Hz that produces the effects that we see. 

Figure 1

 

Amplitude versus time for one cycle of the modulation of contrast-modulated 650-nm flicker (upper panel) and one cycle of 560-nm flicker (lower panel). The icons above each waveform represent the approximate changes in hue or brightness coincident with the peaks and the troughs of the flicker.

Figure 1 illustrates the basic phenomena and shows, as a function of time, the amplitude of one modulation cycle of the two contrast-modulated waveforms—650-nm light in the upper panel, and 560-nm light in the lower panel. When 650-nm sinusoidal flicker with a carrier frequency, f_c, between 5 and 40 Hz is contrast-modulated at an f_m of 0.5 Hz (and the modulation is high enough for the distortion to be above threshold), observers report a hue change at 0.5 Hz from red, when the flicker contrast is low, towards yellow, when it is high (indicated by the icons above the upper waveform). By contrast, when 560-nm flicker is similarly contrast modulated, observers reported a brightness change at 0.5 Hz that is brightest when the flicker contrast is high (indicated by the icons above the lower waveform). For further details about this work, see Petrova et al. (2013a, 2013b). Their work, and the work reported here, is an extension of comparable measurements by Stockman and Plummer (1998), who dissected the S-cone pathway using a contrast-modulated 440-nm flickering target superimposed on an intense steady 620-nm background (to isolate the S-cone response), and of Wu et al. (1996), who measured brightness enhancement. 

In this paper, we compare the early and late filter shapes estimated from the hue change generated by 650-nm contrast-modulated flicker and the brightness enhancement generated by 560-nm contrast-modulated flicker. We propose a physiologically relevant model that accounts for the shapes of the early and late filters. The model incorporates, within a single scheme, an expansive nonlinearity that distorts the contrast-modulated flicker signal and enhances brightness, and an additional compressive nonlinearity that changes hue. The shapes of early filters, estimated from the distortion of M- and L-cone isolating contrast-modulated flicker, are also modeled. 

Summaries of methods and previous results

The methods for estimating the early (prenonlinearity) and late (post-nonlinearity) filter shapes have been extensively described in our previous paper (Petrova et al., 2013b). We summarize them briefly here. The primary visual stimulus was contrast-modulated flicker. Its temporal waveform, V_m(t), was:  

 

where f_c is the carrier frequency, f_m is the modulation frequency (both in Hz), t is the time (in secs), and m is the overall modulation or Michelson contrast. The factor in square brackets is sometimes called the amplitude modulation and the amplitude modulation in our experiments, at f_m Hz, always varied sinusoidally between one and zero; i.e., V_m(t) was 100% amplitude modulation. 

The flickering component of Equation 1 can be expanded to show that it comprises three sinusoidally flickering terms:  

 

where the components, at f_c Hz, with amplitude 0.5R̄m and two sidebands at f_c − f_m and f_c + f_m Hz with half the amplitude of the carrier, are made explicit. Observers manipulated the overall contrast, m, using the method of adjustment to find thresholds for flicker or for hue shift or for brightness change as appropriate. 

Determination of the early filter shapes

The early filter shapes at 5 Hz and above were estimated using contrast-modulated flicker with a fixed f_m of 0.5 Hz, and variable f_c. The observers' task was to find the threshold for the hue or brightness change at 0.5 Hz as a function of f_c (Hz) by varying the overall modulation of the contrast-modulated stimulus (i.e., a temporal contrast sensitivity function or TCSF for hue or brightness change was measured). Given that the stages after the nonlinearity were always presented with the same low-frequency distortion product (a near-threshold 0.5-Hz change), the late stage does not influence the shape of either TCSF—it only scales them (i.e., shifts the TCSFs up or down on conventional double logarithmic coordinates). Moreover, since the nonlinearity requires the same contrast variation at its input to produce the threshold 0.5-Hz distortion product, the nonlinearity has no influence on the shape of the TCSF and can also be ignored. Consequently, the TCSF for detecting the hue or brightness change at fixed f_m = 0.5 Hz as a function of f_c reveals the shape of attenuation characteristics of the early stages before the nonlinearity. Plotted as log₁₀ sensitivities against log₁₀ frequency, the TCSFs are proportional to the log₁₀ attenuation of the early filter as a function of frequency. 

Figure 2 shows the directly measured results (above 5 Hz) for observer DP (left-hand column) or GBH (right-hand column). Sensitivity to hue change is shown as yellow symbols, sensitivity to brightness change as gray symbols. (The error bars indicate ± one standard error based on the three measurements contributing to each data point.) Results for four time-averaged radiances for the hue and brightness measurements are shown. The time-averaged 650- and 560-nm radiances at each level are roughly equated for L-cone excitation (Stockman & Sharpe, 2000a). Data from the lowest to the highest levels are shown as inverted triangles, circles, diamonds, and squares, respectively, and the results at the different radiance levels have been arbitrarily separated for clarity. Further, at each level, the directly measured 560-nm brightness-change data (gray symbols) have been vertically aligned with the directly measured 650-nm hue-change data (yellow symbols) using a least squares fitting criterion—the point is that, vertical shifts apart, the early filter shapes for brightness enhancement and hue shift at each radiance level are the same. 

Figure 2

 

Estimates of the logarithmic sensitivities corresponding to the attenuation characteristics of the early filter for DP (left-hand panel) and GBH (right-hand panel) obtained from 650-nm hue shift measurements (open and yellow symbols) and from 560-nm brightness change measurements (smaller black and gray symbols)—both plotted as a function of frequency (Hz, logarithmic scale). Data are shown at each of the four L-cone equated, 650/560-nm time-averaged radiances: 9.10/8.26 (inverted triangles), 9.70/8.86 (circles), 10.33/9.51 (diamonds), and 10.93/10.11 (squares) log₁₀ quanta s⁻¹degree⁻². Error bars indicate ±1 SEM. The directly measured 560-nm brightness enhancement data (gray symbols) have been vertically aligned with the 650-nm hue shift data (yellow symbols) using a least-squares fitting criterion. The vertical axis is correct for the 650-nm hue change data measured at the lowest mean radiance plotted as amplitude sensitivities (top set) where the amplitudes are given in quanta s⁻¹degree⁻². For clarity, the amplitude sensitivities at the next three 650-nm mean radiances have been shifted down by an additional 1, 2, or 3 log₁₀ units, respectively. The vertical positions of the early filter estimates, inferred from differences between either the hue change (open symbols) or brightness change (gray symbols) late filter estimates and conventional TCSFs, were determined by the fit of the subtractive model, the final version of which is shown by the continuous red lines. The attenuation characteristics of the central and surround mechanisms assumed in the model are shown by blue and black lines, respectively. The phase delays between the center and surround signals as a function of frequency are shown as the red line in the lower panel of Figure 4. For further details, see text.

Also shown in Figure 2 are indirect estimates of the low-frequency end of the early filter (f < 5 Hz) from both hue-shift (open symbols) and brightness-change (black symbols) experiments. (Direct measurements could not be made at low frequencies, f_c < 5 Hz, because at those frequencies the carrier also produces changes in hue and/or brightness that make the separate detection of the distortion product difficult.) In this low-frequency region, we inferred the shape of the early filter from other measurements by assuming that conventionally measured TCSFs for detecting flicker reflect the multiplicative combination of the early and the late filters. Thus, the logarithmic differences between the TCSFs measured using sinusoidal 650-nm flicker and the late filter estimates measured using 650-nm contrast-modulated flicker (see Figure 3) give the shapes of the missing parts of the early filter shown as large open symbols in Figure 2. Similarly, the logarithmic differences between the TCSFs measured using sinusoidal 560-nm flicker and the late filter estimates measured using 560-nm contrast-modulated flicker give the shapes of the missing parts of the early filter shown as black symbols in Figure 2. (The monochromatic 650- and 560-nm TCSFs used to estimate the low frequency filter shapes are plotted in figures 4 of Petrova et al., 2013a, 2013b.) 

Figure 3

 

Estimates of the logarithmic sensitivities corresponding to the attenuation characteristics of the late filter for DP (left-hand panel) and GBH (right-hand panel) obtained from 650-nm hue shift measurements (yellow and open symbols) and from the 560-nm brightness change measurements (gray and black symbols)—all plotted as a function of frequency (Hz, logarithmic scale). Data are shown at each of the four L-cone equated 650/560-nm time-averaged radiances: 9.10/8.26 (inverted triangles), 9.70/8.86 (circles), 10.33/9.51 (diamonds), and 10.93/10.11 (squares) log₁₀ quanta s⁻¹degree⁻². Error bars indicate ±1 SEM. The directly measured 560-nm brightness enhancement data (gray symbols) have been vertically aligned with the corresponding 650-nm hue change data (yellow symbols) using a least-squares fitting criterion. The vertical axis is correct for the 650-nm hue change data measured at the lowest mean radiance plotted as amplitude sensitivities (top set) where the amplitudes are given in quanta s⁻¹degree⁻². For clarity, the amplitude sensitivities at the next three 650-nm mean radiances have been shifted down by an additional 1, 2, and 3 log₁₀ units, respectively. The vertical positions of the late filter estimates inferred from differences between either the hue change (open symbols) or the brightness change (gray symbols) early filter estimates and chromatic TCSFs were determined by the fit of the low-pass model, the final version of which is shown by the continuous red lines. For further details, see text.

The vertical alignment of the indirect measurements below 5 Hz with the directly measured data above 5 Hz depends on the best fit of a “subtractive” model, which is shown by the continuous red lines and is discussed next. The agreement of the direct measurements above 5 Hz, and the indirect measurements below 5 Hz is very good and one model of the early filter fits both the hue-shift and the brightness-change data over the full frequency range. 

Determination of the late filter shapes

The late filter shapes were also estimated using contrast-modulated flicker but now with a fixed f_c of 30 Hz and variable f_m. The observers' task was again to find the threshold for the hue or brightness change at f_m Hz by varying the overall modulation of the contrast-modulated stimulus, but this time as a function of f_m. Since all the stimuli have the same 30-Hz carrier frequency and f_m ≪ f_c, the three components of the contrast modulated stimulus are similarly affected—at least at low f_m—by the early filter that precedes the nonlinearity. Consequently, any changes in sensitivity with f_m are due mainly to the combined effects of the nonlinearity and the attenuation characteristics of the late filter. 

We have been able to ignore changes in the frequency content of the amplitude-modulated signal reaching the nonlinearity at higher f_m (due to the three frequency components of f_c − f_m, f_c and f_c + f_m Hz being differently attenuated by the early filter). Although such changes might be expected to lead to an overestimation of the sensitivity loss, the contrast of the distortion product at f_m is surprisingly resilient to the effects of the early filter. When f_m is sufficiently large compared to f_c, the early filter changes the waveform from one that is amplitude-modulated at f_m (with two equal amplitude sidebands that are half the amplitude of the carrier in our 100% contrast-modulated flicker: 0.5(f_c − f_m) + f_c + 0.5(f_c + f_m) to one that beats at f_m [dominated, after filtering by the roughly equal amplitude lower sideband and carrier: (f_c − f_m) + f_c, which beat at their difference frequency]. The distortion of these two waveforms in our model produces comparable distortion products at f_m. 

To discount the effect of the nonlinearity (and thus estimate the characteristics of the late filter alone), we needed to know how changes in the modulation of the contrast-modulated flicker at the input to the nonlinearity alter the size of the distortion product at f_m at its output. Petrova et al. (2013a, 2013b) determined the relation using side-by-side hue or brightness matching procedures. The observer was presented on one half of a bipartite matching field with contrast-modulated 650- or 560-nm flicker (with f_c = 30 Hz and f_m = 0.5 Hz) and on the other half with a pedestal of the same wavelength and time-averaged radiance. Superimposed on the pedestal was a 560-nm light that was sinusoidally flickered at 0.5 Hz, the amplitude of which the observer varied to match the hue or brightness change produced by contrast modulation in the other half field. If the contrast-modulated flicker and pedestal were both 650 nm, sinusoidal variation of the 560-nm light was seen mainly as a change in hue, whereas if they were both 560 nm, the variation was seen mainly as a change in brightness. (The results of the matching experiment are plotted for 560-nm flicker in the left-hand panel of Figure 6, from figure 6 of Petrova et al., 2013a, and will be subsequently discussed in detail.) 

Knowing the relation between the modulation and the magnitude of the distortion at f_m Hz, we can correct the TCSFs for detecting the hue or brightness change as a function of at f_m Hz and thus reveal the attenuation characteristics of the late filter. Figure 3 shows such estimates below 5 Hz for DJP (left column) and GBH (right column) based on detecting either the hue shift (yellow symbols) or the brightness change (smaller gray symbols). Log sensitivity is shown as a function of frequency (logarithmic scale); the four different radiance levels are the same as in Figure 2, and the error bars indicate ± one standard error. The results for the different radiance levels have been shifted for visibility. These symbols directly measured data (corrected for the effect of the nonlinearity) all lie at or below 5 Hz, because changes of brightness or hue could be detected only up to an f_m of about 5 Hz. The 560-nm brightness-change data (gray symbols) have been vertically aligned with the 650-nm hue-shift data (yellow symbols) at each level using a least squares fitting criterion. 

At frequencies above 5 Hz, we again inferred the shape of the late filter from other measurements by assuming, as before, that TCSFs conventionally measured with simple sinusoidal flicker reflect the multiplicative combination of the early and the late filters (but in the case of the late filter, the conventional TCSFs that we used were obtained using sinusoidal chromatic flicker). Thus, the logarithmic differences between the chromatic TCSFs and the early filter estimates of Figure 2 measured using 650-nm contrast-modulated flicker yield the indirect estimates of the shapes of the missing parts of the late hue-shift filter shown as large open symbols in Figure 3. Similarly, the logarithmic differences between the chromatic TCSFs and the early filter estimates of Figure 2 measured using 560-nm contrast-modulated flicker yield the indirect estimates of the shapes of the missing parts of the late brightness-change filter shown as black symbols in Figure 3. Their vertical alignment with the directly measured data below 5-Hz is also not specified by the experimental results and the alignments shown in Figure 3 are based on the fits of the low-pass filter model shown by the red lines. The chromatic 650- and 560-nm TCSFs used to derive filter shapes above 5 Hz are plotted in figures 4 of Petrova et al. 2013a, 2013b.) 

Note, that a single model of the late filter fits both sets of data over the full range of frequencies. 

This is a summary of our previous results, which will be reanalyzed in the following sections. For further details about the derivation of the early and late filters estimates, please see Petrova et al. (2013b) for the hue-change measurements, and Petrova et al. (2013a) for the brightness-change measurements. 

The early filters in the chromatic and brightness pathways compared

We begin by comparing the early filter estimates based on the 650-nm hue shift and 560-nm brightness-change measurements shown in Figure 2. If the early processing of hue and brightness were by distinctly different visual pathways, then we might expect the early filter estimates obtained from hue shifts and brightness changes also to be different. However, the estimates of the early filter shapes determined from the 650-nm hue shift and the 560-nm brightness-change measurements are similar—apart from a slight suggestion that the high-frequency slopes of the early filters for GBH are marginally steeper for the detection of a brightness change than for the detection of a hue shift. Overall, though, there are no differences in shape that might indicate the involvement of different pathways as might have been expected if, for example, flicker were encoded in one case by the parvocellular stream and in the other by the magnocellular stream. The small differences that we find are more likely to be due to the L-cone-equated 560-nm and 650-nm flickering lights producing different M-cone adaptation levels and consequently different flicker strengths and phase delays and perhaps differences in surround antagonism (e.g., Stockman, Langendörfer, Smithson, & Sharpe, 2006). It is possible, of course, that the nonlinearities are in distinct pathways but that they are located before the stages at which any differences in attenuation arise. But since the late filters are also similar, this seems unlikely. 

Given the overall similarities, we will treat the hue and brightness based estimates of the early filter in common. 

Central to our modeling of the early and late filters is the idea that the attenuation characteristics of stages in the visual pathway can be modeled by cascades of leaky integrating stages (or buffered RC circuits), the outputs of which decay exponentially after exposure to a brief pulse of light. The amplitude response, A(f), of n cascaded, identical, leaky integrators as a function of frequency, f, is:  

 

where f₀ (Hz) is the “cut-off” or “corner frequency” and τ is the time constant of each leaky integrator in seconds (τ = 1/2πf₀). The approach of modeling the eye in this way as a linear temporal filter has a long tradition (e.g., De Lange, 1952; Ives, 1922; Kelly, 1961; Roufs, 1972; Watson, 1986). 

In our previous papers, we used what we called the “divisive” model in which the band-pass shape of the early filter was modeled by dividing a center temporal response of a cascade of filters by a surround temporal response of another cascade of filters (see Equation A1 in the Appendix) and fitted the model separately to the hue (Petrova et al., 2013b) and brightness (Petrova et al., 2013a) data. The simultaneous fits using the divisive model are shown in Figure A1 in the Appendix and described there. 

Although the divisive model accounts well for the characteristics of the early filter for both hue shift and brightness enhancement (with an R² = 0.997), it was created ad hoc without regard to the characteristics of the retina. Moreover, despite our claim that the early stage should be linear, the model is at least conceptually nonlinear in that it involves divisive interactions (see Oppenheim & Wilsky on linear feedback systems [p. 685 ff]). We now develop a subtractive model more consistent with known retinal properties in which the surround signals subtract linearly from center signals. As before, the signal from the center processor is assumed to be from a cascade of filters:  

 

and the signal from the surround is assumed to be identical to the center one but with additional stages of filtering, thus:  

 

 

(Note that the surround, A_s(f), is written as the center, A_c(f) times the n_s additional surround stages.) Dividing the right hand side of Equation 5 by Display Formula , scales A_s(f) for ease of comparison so that it has the same amplitude as A_c(f) at low frequencies. This also has the effect of balancing the center and surround signals at low frequencies. Consequently, w in Equation 7 represents the relative magnitude of the center and surround signals at low frequencies. 

The phase delays between the center and surround signals (Δθ, degrees) are assumed have three components: the delay caused by the extra n_s filters in the surround, which is n_stan⁻¹(f/f₀), the delay caused by an additional surround delay of Δt (seconds), which is 360Δtf, and a phase inversion of −180° caused by surround antagonism—the three components add:  

 

 

The resultant signal produced by adding together A_c(f) and A_s(f) at the angle determined by their relative phase difference is then:  

  .

 

The value w is a surround scaling factor by which A_c(f) and A_s(f) differ in amplitude at low frequencies (f = 0 Hz) (if w=1, they would be equal in amplitude at low frequencies). This phase dependent (vector) addition is illustrated in Figure 4. 

Figure 4

 

Upper diagram: Vector addition as required in the subtractive model. The center signal A_c (white arrow) and surround signal A_s (gray arrow) are added with a phase shift of Δθ (red arc) to produce the resultant vector A_r (black arrow) with phase shift of ϕ (green arc). The components of the surround signal that are in phase and 90° out-of-phase with the center signal are also indicated. Lower panel: The phase delays as a function of frequency assumed in the subtractive model (red line) are the sum of: (a) the phase delays caused by the additional filter stage in the surround (green line), (b) a time delay (blue line), and (c) subtraction of the surround from the center (black line). The length of A_r depends on Δθ (red line and the corresponding red arc in upper panel) and on the lengths of A_c and wA_s. The joint dependence of A_r and Δθ on frequency is used to model the attenuation characteristics of the early filter. See text for details.

The upper diagram of Figure 4 illustrates the vector addition of A_c (open arrow) and wA_s (gray arrow), which have a relative phase delay of Δθ(f) (red arc) to yield the resultant vector A_r (black arrow), which has a phase delay of ϕ(f) relative to A_c (green arc). (The component of wA_s in the direction of A_c [wA_scos(Δθ)] and the component at right angles to A_c [wA_ssin(Δθ)] are also shown.) The lower panel of Figure 4 shows the overall phase delay as a function of frequency assumed in the final fit of the model (red line). It is made up of the sum of the phase delays due to the extra surround filters (green line), the time delay (blue line), and the phase inversion or inhibition (black line). 

The fits were made simultaneously to the hue and brightness data for both observers using the logarithm of Equation 7. As in the divisive model (see Appendix), a logarithmic shift, k, was allowed to vary with target radiance level, and, as before, an extra arbitrary constant, v, was added to the low-frequency data for each level. The value of v was individually optimized for each set of data to determine the best-fitting vertical alignment of the low- and high-frequency data, thus:  

 

 

Best-fitting versions of the model were obtained using a standard nonlinear, least-squares, curve-fitting algorithm (implemented in SigmaPlot, SPSS) to account for the data obtained for each observer at each of the four time-averaged radiances. The value of k is related to the frequency-independent scaling illustrated by the adjustment dials within the representations of cones shown in Figure 5, below (for further details, see Stockman, Langendörfer, et al., 2006). The subtractive model (like the divisive model in the Appendix) was simplified by fixing those parameters that did not vary systematically with target radiances or across observers: the number of center filters, n_c, the number of surround filters n_s, the delay, Δt, the scaling of the surround relative to the center, w, and the common corner frequency of the surround filter, f_0s. Again, it was assumed the cut-off frequencies of all the center stages were identical. (If, as in preliminary fits, n_c and n_s are separately allowed to take on noninteger values, the best fitting values were 4.01 ± 1.07 and 0.92 ± 0.89, respectively; but in the final fits we fixed them at the nearest integer values, n_c = 4 and n_s = 1.) Both scaling (the vertical logarithmic shift, k) and the center cut-off frequencies were allowed to vary between observers and between levels. 

Figure 5

 

Model. Two networks of L- and M- cones are shown at the left—the upper one with an L-cone driven center and the lower with an M-cone driven center. The characteristics of the centers are those of a cascade of four identical low-pass filters with a radiance-dependent gain control. Each network has a subtractive surround mechanism with a ratio of 2:1 in favor of L-cone over M-cone inputs. Each element of the surround mechanism has an additional low-pass filter stage and a delay. The networks each drive two half-wave rectifying ON- and OFF-subsystems that are recombined in the ways shown to produce distinct red and green unipolar chromatic mechanisms and bright and dark unipolar brightness mechanisms, each of which is preceded by a two-stage low-pass filter. Each unipolar mechanism is assumed to incorporate a smoothly compressive nonlinearity. See text for further details.

Figure 2 shows the center filters (blue lines), surround filters (black lines), and the resultants (red lines) that best fit the combined hue-shift and brightness-change estimates of the early filter. Log₁₀ sensitivity is plotted as a function of frequency (logarithmic scale), for DP in the left-hand column, for GBH in the right-hand column. The early filter estimates and corresponding fits for the four different mean radiance levels have been displaced for clarity of presentation. The same filter shape characterizes the early filter from both the hue-shift and brightness-change experiments. The parameters from the fit are tabulated in Table 1. (The values of v, which are arbitrary, are not given for reasons discussed above.) Like the divisive model described in the Appendix, the subtractive model provides a highly plausible (R² = 0.996) description of the attenuation characteristics of the early filter. 

Table 1

 

Best-fitting parameters for the subtractive early filter model. See text for details.

Parameter	DP	GBH
nc (fixed)	4
ns (fixed)	1
Δt (fixed)	16.84 ± 0.98 ms
w (fixed)	1.00 ± 0.09
f0s (fixed)	18.59 ± 2.19 Hz
f0c 9.10/8.26	22.31 ± 1.62 Hz	16.10 ± 0.98 Hz
f0c 9.70/8.86	32.14 ± 3.04 Hz	23.65 ± 1.72 Hz
f0c 10.33/9.51	46.46 ± 5.96 Hz	32.98 ± 2.85 Hz
f0c 10.93/10.11	44.08 ± 4.47 Hz	37.23 ± 3.60 Hz
k 9.10/8.26	0.00 ± 0.09	0.00 ± 0.07
k 9.70/8.86	0.13 ± 0.13	0.01 ± 0.09
k 10.33/9.51	0.12 ± 0.19	0.09 ± 0.11
k 10.93/10.11	0.82 ± 0.14	0.58 ± 0.13
R2	0.996

Reassuringly, the corner frequencies of the central filters estimated from fits of the subtractive model are consistent with other estimates. For example, the four-filter estimates of Stockman, Langendörfer et al. (2006), obtained by simultaneously modeling changes in modulation sensitivity and phase delay measurements between adaptation levels made in two protanopic observers, yield comparable corner frequencies. (See figure 9, triangles, of Stockman, Langendörfer et al., 2006, in which they are plotted as time constants, τ; to convert from τ in seconds to the corner frequency f₀ in Hz: f₀ = 1/[2πτ]). 

Center-surround antagonism or lateral inhibition has been incorporated in various forms into several psychophysical and physiological models of early visual processing (e.g., Benardete & Kaplan, 1999a, 1999b; Furman, 1965; Kelly, 1971; Rashbass, 1970; Ratliff, Hartline, & Miller, 1963; Sperling & Sondhi, 1968; Victor, 1987; Watson & Nachmias, 1977). It is typically invoked to account for the increased attenuation of contrast sensitivity when a stimulus is low in both spatial and temporal frequency (e.g., Kelly, 1969; Nachmias, 1967; Ratliff, Knight, & Graham, 1969; Ratliff, Knight, Toyoda, & Hartline, 1967; Robson, 1966; Schober & Hilz, 1965). 

Our subtractive model is directly related to the working model proposed by Watson (1986) in his tutorial review chapter, but differs in some important respects: In a typical example of Watson's model, fitted in his figure 6.5(a) to data from De Lange (1958a), the center signal is shaped by a nine-stage low-pass filter (with each stage having a cut-off frequency of 32.22 Hz) and the surround signal by a 10-stage low-pass filter (with each stage having a cut-off frequency of 24.22 Hz). No time delay is allowed between the two signals, so that the phase delays between the center and surround are due to a combination of opponency (180° at all frequencies) and the phase differences produced by the differences between the center and surround filters. In our model the center signal is shaped by a four-stage low-pass filter and the surround signal is shaped by the same filter with an additional stage, but we also allow a time delay that adds to the phase delays produced by the extra surround filter and by opponency. 

Somewhat surprisingly, both versions of the model produce band-pass filters that are very similar in shape—at least over the measurable range of frequencies. But how can such different numbers of stages and cutoff frequencies produce similar shapes? In both models, destructive interference between the center and surround signals causes low-frequency attenuation. But at intermediate and higher frequencies, the models shape the band-pass form in different ways: In Watson's model, the relative strength of the surround signals declines rapidly with frequency, and so has relatively little effect on the shape of the resultant sensitivity function above about 10 Hz. Consequently, the high-frequency slope depends mainly upon the nine center filters. In our model (see Figures 2 and 4), the influence of the surround signals persists to much higher frequencies, and since a fixed time delay causes the phase delay to increase with frequency, their influence changes from antagonism at low frequencies (when center and surround signals are in opposite phase) to facilitation (when center and surround signals are in phase) to antagonism again at yet higher frequencies (when center and surround are again in opposite phase). 

Which version of these model is more likely to be correct? We favor our own model, but with several caveats: First, values in the model fits shown in Figure 2 are sensitive to the starting values of the parameters in the fitting procedure. What we show are the best fits, but other plausible solutions with less destructive interference at higher frequencies can be obtained. Second, had we allowed the cut-off frequencies and the numbers of center and surround filters to vary independently, we could have produced plausible fits with parameters similar to those adopted by Watson (1986). Third, the delays we propose between center and surround are larger than those typically found or inferred between parvocellular center and surround (e.g., Benardete & Kaplan, 1997; Derrington, Krauskopf, & Lennie, 1984; Lee, Martin, & Valberg, 1989; Lee, Pokorny, Smith, & Kremers, 1994; Smith, Lee, Pokorny, Martin, & Valberg, 1992), but not always (Gouras & Zrenner, 1979; Lankheet, Lennie, & Krauskopf, 1998). 

We speculate that the time delays in our measurements are relatively large because they arise in a recursive network of lateral connections (perhaps at the horizontal cell layer) produced by our flicker stimulation over a relatively large retinal area (in neural terms) of 4° in diameter. Indeed, similarly large delays are found in many other experiments using L-, M-, and S-cone flicker using similarly sized fields (Stockman, Montag, & Plummer, 2006; Stockman & Plummer, 2005a, 2005b; Stockman, Plummer, & Montag, 2005; Stockman, Sharpe, Zrenner, & Nordby, 1991; Stromeyer et al., 2000). This is a topic we will return to in future papers. Recursive network inhibition producing attenuation at low frequencies and facilitation at higher frequencies was of course demonstrated and modeled in Limulus eye by Ratliff and colleagues, and linked to visual phenomena in human vision (Kelly, 1969; Ratliff et al., 1969; Ratliff et al., 1967). The characteristic changes in the shapes of TCSFs due to surround antagonism and facilitation suggested by our model can be clearly seen in figure 1 of Kelly (1969), in which TCSFs for 3° and 16° targets are compared. The comparisons shown in Figure 9 below are also highly suggestive, if, as we have suggested below, the S-cone early filter is mainly the central filter, and therefore more comparable to the photoreceptor response (Stockman & Plummer, 1998). Conceivably, however, the large delays might arise from surround signals from large non-classical receptive fields with silent surrounds (Series, Lorenceau, & Fregnac, 2003). 

The late filters in the chromatic and brightness pathways compared

The estimates of the late filter shapes determined from the 650-nm hue shift (open and yellow symbols) and the 560-nm brightness-change measurements (smaller filled and gray symbols) shown in Figure 3 are clearly very similar for both observers. As for the early filters, there are no marked differences in shape that would indicate that distinctly different pathways are involved. Given also the similarities between the early filter shapes estimated from the hue-shift and brightness-change measurements, these results suggest strongly that the detection of the hue shift and brightness change is mediated either by a single pathway or by pathways with very similar characteristics. Given these similarities, we have treated the hue- and brightness-based estimates of the late filter in common and fitted the low-pass filter model defined by Equation 3 to both sets of data simultaneously. (Separate fits can be found in Petrova et al., 2013a, 2013b.) The fits were made using the logarithm of Equation 3 with a logarithmic shift, k, that varied with target radiance level. In terms of the model, k represents a frequency-independent sensitivity loss that is in addition to any losses resulting from the changing corner frequencies of the filters. An extra arbitrary constant, v, was added to the high-frequency data for each level, the value of which was individually optimized for each set of data to determine the best-fitting vertical alignment of the low- and high-frequency data, thus:  

 

 

The model was simplified by fixing the number of stages, n, and their common corner frequency f₀ because those parameters did not vary systematically with target radiance or across observers. The number of stages and the cut-off frequency were assumed to be the same for both observers and were fixed across levels. The number of stages (n) was allowed to take on noninteger values in preliminary fits, but in the final fits we fixed it at the nearest integer values (n = 2). (If n is allowed to take on noninteger values, the best fitting value is 1.98 ± 0.06.) Only the scaling (the vertical logarithmic shift, k) was allowed to vary between observers and between radiance levels. 

The fits of the low-pass filter model to the late filter estimates are shown in Figure 3 as the continuous red lines. Again, the same filter shape characterizes the late filter from both the hue-shift and the brightness-change experiments at all four radiance levels. The parameters from the fit are tabulated in Table 2. (The values of v, which are arbitrary, are not given.) The low-pass filter model provides an excellent (R² = 0.996) description of the attenuation characteristics of the late filter. The poorest fits are those to the high-frequency estimates for GBH at the lowest mean radiance level (top right-hand panel). 

Table 2

 

Best-fitting parameters for the low-pass late filter model. See text for details.

Parameter	DP	GBH
n (fixed)	2
f0 (fixed)	3.15 ± 0.10 Hz
k 9.10/8.26	0.00 ± 0.03	0.00 ± 0.03
k 9.70/8.86	0.54 ± 0.03	0.55 ± 0.03
k 10.33/9.51	1.22 ± 0.03	1.22 ± 0.03
k 10.93/10.11	1.84 ± 0.03	1.96 ± 0.03
R2	0.996

We might have expected the late filter to be approximately independent of radiance level since, presumably, as a late filter, it operates after adaptation to the different radiance levels and is consequently operating at a similar input level in all four cases. 

The agreement between the filter shapes obtained from the hue-shift and the brightness-change measurements suggests that chromatic and brightness information is transmitted either in a common pathway or in two pathways with strikingly similar characteristics. The data do not support our original supposition that the chromatic information is transmitted by a chromatic pathway, whereas the brightness information is transmitted by a faster luminance pathway (e.g., Boynton, 1979; De Lange, 1958b; Eisner & MacLeod, 1980; Guth, Alexander, Chumbly, Gillman, & Patterson, 1968; Luther, 1927; Schrödinger, 1925; Smith & Pokorny, 1975; Walls, 1955). 

The late filter is comparable to the chromatic TCSFs measured by Wisowaty (1981), who asked observers to set the threshold for perceiving the chromatic red-green alternation produced by alternating 546 and 656-nm lights (rather than setting the threshold for seeing flicker of any type) and to the filter that limits brightness and color changes induced into a steady light by a flickering surround (Christiansen et al., 2009; D'Antona & Shevell, 2006; De Valois, Webster, & De Valois, 1986). 

A model of the chromatic and brightness pathways

The early filter from the subtractive model and the two-stage low-pass late filter model provide the two starting components for an overall model of the chromatic and brightness pathways. The other components of the model, illustrated in Figure 5, will be justified in this section. 

The early filter is represented on the left by two clusters of cones (truncated triangles) and their connections. The L- and M-cones are represented by red and green truncated triangles, respectively. The two larger cones, the upper L-cone and the lower M-cone, are central cones and the three smaller cones around each large cone symbolize a network of surround cones in the ratio of two L-cones to one M-cone. Within each cone there are two components: The first is a four-stage low-pass filter pictured within the triangles as an exponential decay assigned the number four. The second is a sensitivity adjustment pictured within each triangle as an adjustment dial. This adjustment is not explicitly used in the present paper, but is important for modeling adaptive changes that do not depend on changes in the time constants of the filters and are frequency independent (Stockman, Langendörfer, et al., 2006). 

Consistent with our subtractive model, the feedback from the surround cones to each central cone incorporates a one-stage filter depicted in gray diamonds as an exponential decay with the number one—this is the additional surround filter. There is also a time delay (depicted within the gray diamonds by “Δt”). (The recursive characteristics of the network, interconnections between many cones, are not included in this model but will be considered in subsequent papers.) The ratio of L- to M-cone numbers in both surrounds is 2:1, approximately the mean L:M cone ratio found in human retina (e.g., Carroll, Neitz, & Neitz, 2002; Cicerone & Nerger, 1989; Hofer, Carroll, Neitz, Neitz, & Williams, 2005; Sharpe, Stockman, Jagla, & Jägle, 2011; Vimal, Smith, Pokorny, & Shevell, 1989). It is this network that determines the early (pre-nonlinearity) filter characteristics. 

The two cone clusters both branch into ON- and OFF-center types, which are approximately half-wave rectified before being re-combined into unipolar “red,” “green,” “bright,” and “dark” channels. Each unipolar channel incorporates a smoothly compressive nonlinearity preceded by similar late two-stage low-pass filters represented within the square boxes by an exponential decay and the number two. We now consider the parts of the model in addition to the early and late filters, and the evidence we have in support of each component. 

If we accept the idea that hue and brightness as we measure them are encoded within the same pathway, then we must somehow incorporate within that single pathway both an expansive nonlinearity to produce brightness enhancement and a compressive one to produce hue shifts from red or green towards yellow. 

Consistent with known retinal physiology, we assume that there are four classes of center-surround postreceptoral mechanisms with L-ON, L-OFF, M-ON, and M-OFF center responses and an opposing surround response derived from random connections, probably via horizontal cells, to nearby L- and M- cones (e.g., Dacey, 2000; Field & Chichilnisky, 2007; Rodieck, 1998; Wässle, 2004). 

The expansive nonlinearity

We propose that the expansive nonlinearity is the result of the partial half-wave rectification that occurs separately within ON and OFF visual pathways. The half-wave rectification is only partial in the sense that ON and OFF neurons can respond slightly below their resting potential or spontaneous firing rate (e.g., Kuffler, 1953), as illustrated by the input-output functions represented in the red or green squares after the initial cone interactions in each of the four pathways in Figure 5. The coordinate systems of the input-output functions show the mean adaptation level at the origin and the black line plots the hypothetical response to a wavelength shift towards the preferred wavelengths of the central cone or away from it (which can be in the positive direction for ON neurons or in the negative direction for OFF neurons). 

Psychophysical support for half-wave rectification comes initially from a reanalysis of the results of experiments carried out to reveal the characteristics of the late filter in which side-by-side brightness matching was used to find the contrast of a low-frequency sinusoid that matched the appearance of a flicker-induced brightness change (Petrova et al., 2013a). The matches for 560-nm contrast-modulated are replotted in the left-hand column of Figure 6 for DP (upper panel) and GBH (lower panel). The contrast of a 0.5-Hz sinusoid that matched the peak brightness of a contrast-modulated waveform that had a carrier frequency of 30 Hz and a modulation frequency of 0.5 Hz is plotted as a function of the overall modulation of the contrast-modulated waveform. Both axes are linear and the error bars indicate ± one standard error (based on three observations) and data for the four radiance levels (log₁₀ quanta s⁻¹degree⁻²) are shown as different symbols. 

Figure 6

 

The left-hand panels shows L- and M-cone contrasts of the 0.5-Hz sinusoidally flickering 560-nm hemifield that matches the appearance of 0.5-Hz contrast-modulated 30-Hz flicker in an adjacent hemifield as a function of the overall modulation (m) of the contrast-modulated stimuli (both axes linear). The right-hand panels shows the corresponding data scaled to indicate the similarity of form across radiance. Upper panels for DP, lower panels for GBH. The time-averaged radiances were: 8.26 (inverted triangles), 8.86 (circles), 9.51 (diamonds), and 10.11 (squares) log₁₀ quanta s⁻¹degree⁻². Error bars indicate ±1 SEM. Original data from figure 6 of Petrova et al. (2013a).

A striking feature of these data is that the curves for different radiances differ by only a scaling factor. This fact is illustrated in the right-hand panels where the data in each panel of the left-hand column have been scaled to align with the data obtained at the 9.51 log₁₀ quanta s⁻¹degree⁻² level (green diamonds). After scaling, all four functions for each observer agree remarkably well. Moreover, except for a few data points, the functions are well described by the best fitting (least squares) straight line fitted to the aligned data (red lines). For DP the slope of the best-fitting line is 0.46 ± 0.02 with an R² value of 0.95; for GBH the slope is 0.44 ± 0.04 with an R² value of 0.87. 

The approximately linear input modulation versus output distortion functions that scale with target radiance of Figure 6 severely constrain the types of nonlinearity that could be responsible for brightness enhancement. We have modeled the behavior of a range of nonlinearities in an attempt to predict the data of Figure 6 using MatLab and Simulink (Mathworks). We had previously suggested that the data in the left-hand panel were consistent with a nonlinearity that was initially expansive but with a hard limit at the highest input levels (Petrova et al., 2013a). However, the reanalysis shown in the right-hand panels of Figure 6 suggests a better hypothesis. 

We propose that brightness enhancement is due primarily to signal rectification in the segregated ON and OFF channels. The half-wave rectifier used in the simulation has the form:  

 

where x is the input to the nonlinearity, y is the output, and offset is the mean level at the input to the nonlinearity at each radiance level—the level just above that at which the signal is half-wave rectified. In the model we allow the output level of the stages before the nonlinearity to rise as the time-averaged target radiance (DC) increases, thus increasing the “offset” level at which the signal is half-wave rectified. In the simulation, we varied this offset level from 0.0 to 2.5 (the units are arbitrary), and at each value varied the modulation of the contrast-modulated input from 0% to 100%, where 100% was the maximum modulation possible (without going below zero before rectification) at any given offset value (for example, at an offset value of one, the minimum and maximum of the contrast-modulated input signal after rectification are one and two, respectively, whereas for one of 2.5, they are 2.5 and five). (Graphs of this form of rectified output as a function of input are shown for three arbitrarily chosen offsets in the panels at the top of Figure 7.) At each combination of offset and modulation, we calculated the contrast of the distortion product at f_m; that is, the ratio of the amplitude of the distortion product at f_m to the mean of the output after the nonlinearity. And this mean includes a distortion component from the nonlinearity as well as the response to the nontime-varying component of the input. A three-dimensional plot of the simulated contrast of the signal at f_m as a function of the offset and input modulation is shown in the lower panel of Figure 7. 

Figure 7

 

The lower panel shows the effect of half-wave rectification on the output contrast of the distortion product at the modulation frequency, f_m, as a joint function of the input modulation, m, (given as a percentage) of a contrast-modulated input and the offset level. The upper three panels show the input/output function of the half-wave rectified nonlinearity for offset levels of 0.5, 1.5, and 2.5. For details, see text.

The surface of Figure 7 depends very little on offset level and, except for very low input modulation, is effectively planar. Clearly, the predictions shown in Figure 7 can account for the main features of Figure 6: The output contrasts are linear functions of the input modulation for input modulation exceeding about 10%, and are independent of the offset level. Moreover, changes in the input modulation, caused by radiance-dependent changes in the sensitivity to the 30-Hz carrier, will simply scale the input modulation versus output contrast functions along the input modulation axis while maintaining the linear slope. Other types of nonlinearity that we modeled did not possess these simple properties. Comparable predictions for a compressive nonlinearity and an expansive nonlinearity (both with clipping at high input levels) are shown, respectively, in figure 12 of Petrova et al. (2013b) and in figure 11 of Petrova et al. (2013a). However, nonlinearities that are smoothly compressive or expansive (with or without clipping at high input levels) and nonlinearities that are smoothly expansive at low input levels and smoothly compressive at high levels, produce accelerating functions of output contrast versus input modulation and are thus inconsistent with the linear characteristics of the scaled matching functions shown in the right-hand panels of Figure 6; half-wave rectification in the segregated ON and OFF signals can account for the approximately linear output contrast versus input modulation functions. 

The scaling factors required to align the data for different radiances in Figure 6 are important. The reciprocals of the scaling factors are directly related to the magnitude of the contrast out of the nonlinearity, and are plotted on semilogarithmic coordinates in Figure 8 for DP (red diamonds) and GBH (dark-red squares). 

Figure 8

 

Relative scaling of the output distortion versus input modulation functions required to align the data shown in the left-hand panels of Figure 6 with the 9.51 log log₁₀ quanta s⁻¹degree⁻² data: (DP, red diamonds, GBH, dark-red squares). The scale factors are compared with the relative 30-Hz M-cone modulation sensitivities of two protanopes ML (green triangles) and MM (inverted dark green triangles) from Stockman, Langendörfer et al. (2006). The equivalent horizontal position of the M-cone sensitivities was determined from the mean M-cone radiance at the M-cone λ_max for the protanopes relative to the mean L-cone radiance at 560-nm for normals (calculated using Stockman & Sharpe, 2000a). The M-cone modulation sensitivities have been scaled to align with the mean relative scaling predictions for DP and GBH below 9.5 log₁₀ quanta s⁻¹degree⁻².

The increase of scaling with target radiance can be very simply accounted for by light adaptation shortening the temporal integration time of the early filter; that is, decreasing τ_c in Equation 4 (and, since f₀ = 1/2πτ_c, increasing f₀). A consequence of the shortening time constant is that the sensitivity of the system to the 30-Hz carrier frequency will increase as the mean radiance increases (see, for review and data, Stockman, Langendörfer, et al., 2006), so that the contrast-modulated 30-Hz signal reaching the nonlinear site, and its corresponding output distortion, will also increase. 

Fortunately, we can estimate the effect of mean input radiance on sensitivity to 30-Hz flicker using 30-Hz modulation-sensitivity data measured separately in two protanopes, ML and MM (Stockman, Langendörfer et al., 2006) as part of a series of modulation-sensitivity and phase-delay measurements carried out to model light adaptation (measured using a 4° diameter, flickering 540-nm target on a 9°, 610-nm background). The results for ML (green triangles) and MM (inverted dark-green triangles) are shown in Figure 8 scaled to align with the relative sensitivities for DP and GBH below 9.5 log₁₀ quanta s⁻¹degree⁻². With the exception of the data point at the highest mean radiance for GBH, the relative increase in 30-Hz cone modulation sensitivity for ML and MM predicts the increase in output distortion remarkably well, strongly supporting our model. Moreover, the data of for ML and MM, which peak and then fall, suggest that the scaling for DP (but not for GBH) decreases at the highest mean radiance not because of clipping, as we previously proposed (Petrova et al., 2013b), but because the 30-Hz signal becomes attenuated at the highest radiances. 

To return to the remaining features of the model shown in Figure 5, we first need to consider how hue and brightness are determined. 

Extracting hue and brightness

The linear nature of the input modulation versus output contrast functions for brightness enhancement suggests that the brightness signal (and thus the 560-nm flicker signal) is relatively unaffected by the compressive nonlinearity that gives the hue shift. Thus, we can reasonably assume that the compressive nonlinearity occurs later than the rectification stage. Furthermore, the compressive nonlinearity seems to be specific to the red-green chromatic signal, leaving the bright-dark signal relatively unaffected. Thus, returning to the model shown in Figure 5, we separate the chromatic and brightness information from the spatially and chromatically opponent postreceptoral signals using the decoding scheme illustrated in the diagram. 

As a consequence of being chromatically opponent and spatially opponent, cells in the parvocellular stream carry luminance and/or brightness information as well as color information (e.g., Ingling & Drum, 1973; Ingling & Martinez, 1983; Ingling & Martinez-Uriegas, 1985; Ingling & Tsou, 1988; Kelly, 1983; Lennie & D'Zmura, 1988; Merigan & Eskin, 1986; Schiller, Logothetis, & Charles, 1990). Simple mechanisms for decoding the luminance and chromatic signals from multiplexed signals have been proposed that difference or sum center-surround chromatically and spatially opponent neurons (e.g., Billock, 1991; Lennie, 1984; Lennie & D'Zmura, 1988; Martinez-Uriegas, 1985; Stockman & Brainard, 2009). After the L-ON, L-OFF, M-ON, and M-OFF signals are formed, the signals can be combined in different ways to produce non-spatially opponent red and green mechanisms and spatially opponent bright and dark mechanisms: Summing the L-ON and M-OFF signals produces a spatially non-opponent RED signal and the summing the L-OFF and M-ON signals produces a spatially non-opponent GREEN signal. In Figure 5, these RED and GREEN signals form the basis of separate unipolar red and green chromatic mechanisms. By contrast, summing the L-ON and M-ON signals produces a spatially opponent achromatic BRIGHT signal, and summing the L-OFF and M-OFF signals produces a spatially opponent achromatic DARK signal. The BRIGHT and DARK signals form the basis of separate unipolar bright and dark brightness mechanisms. 

Our model does not require that red, green, bright, and dark mechanisms be unipolar rather than red-green and bright-dark bipolar mechanisms, but we prefer them to be unipolar. Evidence in support of unipolar mechanisms is discussed by Eskew (2008). 

The compressive nonlinearity and the position of the late filter

The position of the late filter with respect to the compressive nonlinearities in the red and green channels is crucial to the final form of the model. In our first paper on hue shifts (Petrova et al., 2013b), we assumed that the late filter followed the compressive nonlinearity. Thus, an amplitude-modulated flicker signal that modulates the red or green channels will be compressed by this nonlinearity to produce shifts in hue at the amplitude-modulation frequency. Yet, because of the similarities found between the early and late filter shapes for detecting the hue shift and brightness change, we now suppose that the compressive nonlinearity that generates the hue shift and the expansive nonlinearity that generates the brightness change must be in the same pathway. This juxtaposition of compressive and expansive nonlinearities, however, complicates our previously simple hue and brightness models (Petrova et al., 2013a, 2013b). One complication is that the similarities between the two early filter shapes requires that the expansive and compressive nonlinearities must be at a similar position in the common pathway with little or no filtering between them, which we think is unlikely. Another complication is that amplitude-modulated flicker (e.g., 650 nm) that produces a hue shift is distorted by both the expansive and the compressive nonlinearities, yet amplitude-modulated flicker (e.g., 560 nm) that produces only a brightness change is distorted by only the expansive nonlinearity; and despite these substantial differences, both types of flicker yield nearly identical estimates of the early and late filter shapes. 

A simple resolution of these and other difficulties is to place the late filter before the compressive nonlinearities in the chromatic (and brightness pathways) as shown in Figure 5. The distortion of the amplitude-modulated flicker signal that generates the brightness change (with 560-nm flicker) and the hue shift (with 650-nm flicker) can then be assumed to occur primarily at the early half-wave rectifying nonlinearities. As a result, the early filters for both the brightness-change and the hue-shift measurements should be the same—as we find. The late filters then attenuate the high-frequency components of the amplitude-modulated signal, which consequently have little effect at the compressive nonlinearity. By contrast, the low-frequency distortion products generated by the early nonlinearity are passed by the late filter and so reach the late nonlinearities, where, if the signals are large enough, they will be significantly compressed. For example, amplitude-modulated flicker of 650 nm will be distorted by the early nonlinearity to produce a larger red signal than green signal at the amplitude-modulation frequency (since the L-cone modulation at 650 nm is greater than M). At the compressive nonlinearities, the larger red signal will be more compressed than the green signal and thus the hue at the peak of the amplitude modulation will shift towards yellow—as we find. Amplitude-modulated 520-nm flicker will produce larger green signals than red, as a result of which the green signal will be more compressed, and the hue at the peak of the amplitude modulation will again shift towards yellow—as we find. In contrast, amplitude-modulated 560-nm flicker, will produce roughly equal red and green signals, which will be equally compressed to produce no change in hue (and be seen as a change in brightness). 

Compressive nonlinearities are also hypothesized within the bright and dark channels as shown by the dashed lines in Figure 5. Given that the inputs to these nonlinearities are still spatially opponent, large fields, such as the 4° diameter targets used in our experiments, are unlikely to drive the bright or dark channels into their compressive range (except, possibly, at the edges of the target), because for large stimuli, the center and surround signals will be approximately balanced. However these nonlinearities would be expected to affect small targets or high-frequency gratings since with them, the output would be driven into the compressive range. And brightness enhancement is indeed reduced for small flickering spots (van der Horst & Muis, 1969). Given that there are late compressive nonlinearities in both the hue and the brightness channels, having spatial opponency for brightness but not for hue may be key in producing the different types of distortion seen for 560 and 650-nm flicker. 

Late filters in the four unipolar pathways may be to extract the slowly changing mean brightness and mean chromatic signals from the multiplexed chromatically and spatially opponent input signal. 

Although we have formalized the late compressive nonlinearity as a static one, other work in our laboratory suggest that it might well be a dynamic one. 

The early L- and M-cone filters and S-cone filters compared

Stockman and Plummer (1998) used the distortion of contrast-modulated 440-nm, S-cone-detected flicker to measure the characteristics of the early filter in the S-cone pathway. The distortion was seen as a yellowing of the 440-nm target when the contrast modulation was high. As discussed in their paper, the estimate of the early filter was consistent with the cone visual response measured photo-voltaically (Schneeweis & Schnapf, 1995, 1999, see figure 12 of Stockman & Plummer, 1998). 

The mean 440-nm target radiance used in the S-cone isolating experiments was 9.53 log₁₀ quanta s⁻¹degree⁻². If we assume prereceptoral (lens plus macular) filtering of 0.63 log unit at 440 nm (the mean values for a 2° field, see Stockman & Sharpe, 2000a), the effective radiance at 440 nm is reduced from 9.53 to 8.90 log₁₀ quanta s⁻¹degree⁻². Given that 440- and 560-nm, respectively, are near to the S- and L-cone corneal λ_max wavelengths (Stockman & Sharpe, 2000a), the closest level for L-cones in our experiments is the 560-nm mean radiance of 8.86 log₁₀ quanta s⁻¹degree⁻² level (or 9.70 log₁₀ quanta s⁻¹degree⁻² at 650 nm). Figure 9 shows a comparison of the early filter estimates for the DP and GBH using L- and M-cone contrast-modulated flicker at the 560/650-nm mean radiances of 8.86/9.70 log₁₀ quanta s⁻¹degree⁻² and the data of the observers from Stockman and Plummer (1998): AS (dark blue circles) and DJP (blue squares) measured using S-cone contrast-modulated flicker and plotted in both panels. The estimates for the hue and brightness changes for DP (upper panel) and GBH (lower panel) have been replotted from Figure 2 using the same symbols, alignment, and sensitivity scale. The continuous red lines shows the fits of the subtractive filter model also from Figure 2. To emphasize the differences, the S-cone estimates for AS and DJP have been vertically aligned with the L- and M-cone estimates at the highest temporal frequencies. 

Figure 9

 

Estimates of the logarithmic sensitivities corresponding to the attenuation characteristics of the early filter for DP (upper panel) and GBH (lower panel) obtained from hue and/or brightness changes produced by 650-nm (yellow symbols) and 560-nm (filled symbols) contrast-modulated flicker at 560/650-nm mean radiances of 8.86/9.70 log₁₀ quanta s⁻¹degree⁻²—as a function of frequency (Hz, logarithmic scale). The positions of the 650- and 560-nm data and model are as in Figure 2 and the continuous red line shows the fit of the subtractive surround model. The error bars indicate ± one standard error. The data are compared with comparable S-cone measurements for observers AS (dark blue circles) and DJP (blue squares) from figure 12 of Stockman and Plummer (1998) made with a 440-nm target with a mean radiance of 9.53 log₁₀ quanta s⁻¹degree⁻². The S-cone estimates have been vertically aligned with the L- and M-cone estimates at the highest temporal frequencies. The continuous blue line is the center filter assumed in the subtractive model vertically aligned with the S-cone data at 2 Hz and above using a least-squares fitting procedure.

It is clear the early S-cone filter declines with a much shallower high-frequency slope than the L- and M-cone filters and exhibits a much smaller level of low-frequency attenuation. Largely as a result, the L- and M-cone estimates appear more band pass than the S-cone estimates; they also have their peak sensitivity shifted to higher temporal frequencies. 

The shallower high-frequency slope of the S-cone filter is consistent with the nonlinearity in the S-cone pathway being much earlier in the visual processing stream than the nonlinearities in the L- and M-cone pathways, so that the early S-cone filter is subject to fewer stages of temporal filtering (and consequently is more similar to the attenuation characteristics of the photoreceptor). However, the difference in low-frequency attenuation, which suggests that there is less surround inhibition for S-cone signals, suggests an alternative explanation that is consistent with our version of the subtractive model. 

Also shown in Figure 9 as the continuous blue lines, again from Figure 2, are the estimates of the attenuation characteristics of the center response predicted by the subtractive model. These have been vertically aligned with the S-cone estimates at 2 Hz and above. As can be seen, the agreement for DP is remarkably good above 2 Hz. The agreement for GBH is less good, but given the diverse nature of the two filter estimates, the agreement is nonetheless impressive. 

We conclude that the principal nonlinearities in the S-cone and L- and M-cone pathways revealed by flicker may all be soon after the photoreceptor. The L- and M-cone early filter estimates appear to be very different from the S-cone estimate, and unlike the photoreceptor response, simply because they reflect antagonistic interactions with signals from the surround that cause destructive interference at both low and high frequencies and constructive interference at intermediate frequencies. In contrast, the S-cone early filter estimates reflect mainly the center response with relatively little surround interference. The lack of S-cone surround inhibition (for S-cone isolating stimuli) is consistent with the S-cone's being relatively sparse (e.g., Ahnelt, Kolb, & Pflug, 1987; Curcio et al., 1991; Szél, Diamantstein, & Röhlich, 1988), and with the possibility that S-cones may not feedback through the H2 horizontal cells to which they are connected (Dacey, Lee, Stafford, Pokorny, & Smith, 1996), since S-cone feedback responses are not seen in L- or M-cones (p. 189, Dacey & Lee, 1999). 

The model we propose, then, consists of an early band-pass filter common to a (probably parvocellular) pathway that signals both hue and brightness. The principal effect of radiance on the early filter is a shortening of its time constant with increasing radiance. The shape of the early band-pass filter is consistent with antagonism between center signals and more sluggish and delayed surround signals. The brightness change and hue shift may be caused initially by the half-wave rectification and partition of signals into ON and OFF components. A common late filter, unaffected by radiance, is a simple two-stage low-pass filter. The hue shift is additionally due to the effects of a subsequent nonlinearity that compresses chromatic red and green signals more than the balanced signals. Plausible sites for the rectifying nonlinearity are soon after surround antagonism possibly from horizontal cells. Plausible sites for the compressive nonlinearity, since it follows the late filter, may be much later. According to the model, the distortion of the amplitude-modulated signals for both hue and brightness occurs primarily at the early half-wave rectifying nonlinearity, but the hue change depends also on the effect of that distortion at the later compressive nonlinearity. 

We note that the early expansive nonlinearity that we attribute to signal rectification is probably distinct from the compressive nonlinearity revealed using high spatial frequency laser interference gratings by MacLeod, Williams, and Makous (1992), which they attributed to local adaptation within single cones. In our experiments, the mean adaptation level is held constant. 

Supplementary experiments: The early filter estimated from hue shifts and brightness changes of pure M-cone or pure L-cone flicker

Introduction

In a supplementary series of experiments, the early filter shapes were estimated this time using contrast-modulated stimuli that excited either only M-cones or only L-cones. Cone isolation was achieved using the method of silent substitution (e.g., Estevez & Cavonius, 1975; Estévez & Spekreijse, 1974, 1982; Rushton, Powell, & White, 1973a, 1973b) between a pair of superimposed 650-nm and 529-nm lights that flickered in opposite phase and were equated in sensitivity either for the L-cones to produce M-cone flicker, or for the M-cones to produce L-cone flicker. 

If the nonlinearity had been at the cone level, then these measurements would have allowed us, in principle, to measure separately the temporal characteristics of the M- and the L-cones. However, this goal was thwarted by the results reported above demonstrating that the nonlinearity responsible for the hue and brightness changes follows a substantial stage of surround antagonism, which places it at least at or after the horizontal-cell feedback onto cones, and perhaps later. This conclusion was also reached by Christiansen et al. (2009). (Note here that by using steady state adaptation, we minimize the effects of nonlinearities in the photoreceptors that are associated with receptor adaptation—see for a discussion of photoreceptor adaptation, Stockman, Langendörfer, et al., 2006.) Nonetheless, the cone-isolating stimuli are particularly useful, because unlike the monochromatic 560- and 650-nm flicker used previously, they generate chromatic flicker at the carrier frequency, f_c. 

Our results suggest that at low carrier frequencies, chromatic flicker at f_c is directly detected, but at higher frequencies, the hue-change at f_m is detected. 

Methods

Apparatus

The stimuli were produced using a five-channel, computer-controlled Maxwellian-view optical system. The experimental details, including descriptions of the experimental system used to make the measurements, and the calibration procedures, can be found in our companion papers (Petrova et al., 2013a, 2013b). We describe here only the essential characteristics of the methods in this set of supplementary experiments. 

Observers

The same observers participated in the experiments for all papers. Both were authors, one male (GBH) and one female (DP), and both were experienced psychophysical observers with normal color vision and normal (DP) or corrected to normal (GBH) spatial acuity. Observers used the method of adjustment. In each condition and the mean and SEM of three settings are reported. 

Stimuli

Visual stimuli were centrally fixated target discs of 4° diameter. As described above, the discs were flickered and the flicker was “contrast-modulated” with the temporal waveform given in Equation 1, above. The modulation frequency, f_m, was fixed at 0.5 Hz and f_c was varied. The “overall modulation,” m, or Michelson contrast:  

 

where I_max and I_min are the maximum and minimum radiances of the stimulus, respectively, was varied to find the hue and/or brightness change threshold using the method of adjustment. 

Silent-cone substitution was produced by superimposing a 4°, 650-nm light and a 4°, 529-nm light. The two lights were contrast-modulated at the same carrier frequency of f_c Hz and at the same fixed f_m of 0.5 Hz. Although the contrast-modulation (at f_m) of the two lights was in phase, the flicker at f_c was in opposite phase. For L-cone isolating flicker, the four 650/529 nm mean radiances were 9.37/7.51, 9.95/8.08, 10.58/8.71, and 11.20/9.34 log₁₀ quanta s⁻¹degree⁻². In terms of L-cone excitation, these lights are 0.3 log₁₀ unit brighter than the comparable 560-nm lights and in terms of luminance 0.15 log₁₀ unit brighter, and the modulation was reduced from the system maximum of 92% to one of 79% (Sharpe et al., 2011; Stockman & Sharpe, 2000b). (Ideally, these lights should have been equated in luminance, but the differences are small.) For M-cone isolating flicker, the four 529/650 nm radiances were 8.10/8.85, 8.65/9.40, 9.22/9.98, 9.90/10.66, and 10.50/11.25 log₁₀ quanta s⁻¹degree⁻². In terms of M-cone excitation, these lights are about 0.13 log₁₀ unit dimmer than the comparable 560-nm lights and in terms of luminance approximately equal, and the modulation was reduced from the system maximum of 92% to one of 79% (Sharpe et al., 2011; Stockman & Sharpe, 2000b). 

This research adhered to the tenets of the Declaration of Helsinki. 

Results

The observers reported that the contrast-modulated flicker changed in hue at f_m from orange-red to yellow for L-cone stimuli and from yellow-green to yellow for M-cone stimuli. Thus, the most pronounced change for these stimuli was a hue shift rather than a brightness change. They also reported that for f_c < 10 Hz, flicker at the carrier frequency appeared to change in hue from red to green, which is consistent with the cone-isolating flicker's generating a chromatic signal. 

Figure 10 shows the estimates of the attenuation characteristics of the early filter obtained with L-cone contrast-modulated flicker (red symbols) for DP (left-hand panel) and GBH (right-hand panel) compared with the estimates obtained with monochromatic 650-nm (yellow symbols) and 560-nm (black symbols) contrast-modulated flicker. Mean radiance increases from top to bottom. The alignments of the 650- and 560-nm data and the predictions of the subtractive surround model (continuous red lines) are from Figure 2. The L-cone data have been vertically aligned with 650- and 560-nm data at frequencies ≥ 15 Hz, a range over which all three estimates agree extremely well. 

Figure 10

 

Estimates of the logarithmic sensitivities corresponding to the attenuation characteristics of the early filter for DP (left-hand panel) and GBH (right-hand panel) obtained from hue and/or brightness changes produced by L-cone (red symbols), 650-nm (yellow symbols), and 560-nm (filled symbols) contrast-modulated flicker all as a function of frequency (Hz, logarithmic scale). Only directly measured data for frequencies at and above 5 Hz are shown. The continuous red lines show the fit of the subtractive model, shown previously in Figure 2. The alignments of the 650- and 560-nm data are as in Figure 2. The L-cone data have been vertical aligned with the joint 560 and 650-nm data using a least squares fitting procedure at 15 Hz and above. The blue symbols are the chromatic TCSFs ≥ 4 Hz measured using luminance-equated 650- and 530-nm alternating sinusoidal (equiluminant) flicker previously plotted as gray symbols in figure 3 of Petrova et al. (2013b). The chromatic data have been vertically aligned with the L-cone data at low frequencies (over the range where the least squares differences per fitted point are minimized) using a least squares fitting procedure.

At lower frequencies, however, the L-cone estimates for DP deviate substantially from the other estimates, showing marked increases in sensitivity at 5, 7.5, and, at the two highest radiance levels, also at 10 Hz. Those for GBH deviate only at 5 Hz and/or 7.5 Hz at the two highest levels. 

Figure 11 shows the estimates of the attenuation characteristics of the early filter obtained using M-cone contrast-modulated flicker (green symbols) for DP (left-hand panel) and GBH (right-hand panel). Again, the cone data have been vertically aligned with 650- and 560-nm data at frequencies ≥ 15 Hz, a range over which all three (and by extension all four) estimates agree. At lower frequencies, the M-cone estimates for both observers are consistently higher in sensitivity at 5 and 7.5 Hz than the other estimates. 

Figure 11

 

Estimates of the logarithmic sensitivities corresponding to the attenuation characteristics of the early filter for DP (left-hand panel) and GBH (right-hand panel) obtained from hue and/or brightness changes produced by M-cone (green symbols), 650-nm (yellow symbols), and 560-nm (filled symbols) contrast-modulated flicker as a function of frequency (Hz, logarithmic scale). The continuous red lines show the predictions of the subtractive model. The M-cone data have been vertical aligned at 15 Hz and above with the joint 560 and 650-nm data using a least squares fitting procedure. The chromatic data (blue symbols) have been vertically aligned with the M-cone data at low frequencies using a least squares fitting procedure. Other details as Figure 10.

Discussion

For both observers, the estimates of the early filter using cone-isolating flicker consistently show less low-frequency attenuation than the estimates obtained using either 560- or 650-nm monochromatic flicker. These deviations are large and suggest that a different mechanism generates the hue change for cone-isolating flicker at low f_m. The likely nature of this mechanism is suggested by the observers' observations that for f_c < 10 Hz, flicker at the carrier frequency appears chromatic. (Similar effects are also seen with S-cone flicker below 10 Hz, Stockman, MacLeod, & DePriest, 1991; Stockman & Plummer, 1998.) If we compare the low-frequency portions of early filter estimates obtained with cone-isolating flicker with directly measured chromatic TCSFs with sinusoidal stimuli from Petrova et al. (2013b), the agreement is remarkably good. The blue symbols plotted in Figures 10 and 11 are directly measured chromatic TCSFs obtained using luminance-equated 650- and 530-nm alternating sinusoidal flicker previously plotted as gray symbols in figure 3 of Petrova et al. (2013b). The chromatic functions have been vertically aligned using at low frequencies a least squares fitting procedure with either the L-cone data in Figure 10 or the M-cone data in Figure 11 where the data clearly deviate from the subtractive model (continuous red lines). So aligned, the directly measured chromatic TCSFs capture the details of the cone isolating data of Figures 10 and 11. 

Such good agreement suggests a common underlying chromatic detection mechanism. If the detection of the hue change for low-frequency cone-isolating flicker depends on distortion at f_m, as we originally supposed, then according to the sandwich model, the measurements at low frequencies should reveal the shape of the filter before the nonlinearity. How can the early filter have the same shape at low frequencies as the overall chromatic TCSFs? The simplest explanation is that at low f_c, observers base their responses on the chromatic appearance of the carrier at f_c Hz rather than on the distortion signal varying at f_m Hz. In that case, the relevant chromatic detection mechanism for cone-isolating flicker at low f_c may follow both the early and the late filters and be after the compressive red and green nonlinearities in the model shown in Figure 5. This arrangement is supported by the observation that at low carrier frequencies, cone-isolating flicker is seen as a variation in hue from red to green at the carrier frequency, which suggests that individual flicker cycles are resolved by the chromatic system. Consequently, at low f_c, observers may base their responses on the chromatic appearance of the carrier at f_c Hz rather than on the distortion signal varying at f_m Hz, a task which may be mediated central to the late filter. The effective nonlinearity for the hue change is then after the extraction of the perceptual color signal (i.e., after both the early and the late filters), and therefore results in data similar in shape to that produced by the chromatic TCSFs, which are also affected by both filters. 

Comparisons between the L- and M-cone early filter estimates and physiological data

We can gain some insight into the likely location of the nonlinearity by comparing the early filter shapes estimated psychophysically with comparable physiological recordings from monkey retinal ganglion cells and LGN cells. 

The smooth curves in Figure 12 show the early filters shapes predicted by the subtractive model for the four mean 560/650-nm radiance levels, which, for the 560-nm targets, equate to 2.16 (black lines), 2.76 (brown lines), 3.41 (orange lines), and 4.01 (red lines) log₁₀ trolands (as indicated in the right-hand panel). The left-hand panel shows the early filters for DP, the right-hand, for GBH; in both panels numbers proportional to log₁₀ sensitivity are plotted as a function of frequency (Hz, logarithmic scale).We have converted our radiance measurements to trolands here to allow easy comparisons with the monkey data. Further, to make the comparisons clearer, we have aligned the psychophysical estimates at low frequencies, where they all have similar slopes. 

Figure 12

 

Comparisons between psychophysical estimates of the early filter for DP (left-hand panel) and GBH (right-hand panel) and physiological measurements. Log₁₀ sensitivity is plotted as a function of frequency (Hz, logarithmic scale). The early filters shapes predicted by the subtractive model are shown for each observer at the four mean 560/650-m radiance levels of 8.26/9.10 (black lines), 8.86/9.70 (brown lines), 9.51/10.33 (orange lines), and 10.11/10.93 (red lines) log₁₀ quanta s⁻¹degree⁻². The 560-nm radiances are equivalent to 2.16, 2.76, 3.41, and 4.01 log td, respectively. The psychophysical estimates and physiological data have then been vertically aligned between 1 and 5 Hz. The logarithmic vertical scale has been arbitrarily set to zero at the peak of the early filter estimate at the highest experimental level. The physiological data are from four sources: (a) parvocellular (Pa) and ganglion cell data from Lee, Pokorny, Smith, Martin, and Valberg (1990) measured at 2.43 log td (orange triangles) and 3.43 log td (yellow triangles) from the lower right panel of their figure 3; (b) parvocellular (yellow circles) and magnocellular (Ma) (gray circles) ganglion cell data from Lee, Pokorny, Smith, and Kremers (1994) measured at 3.43 log td from the upper middle and upper right panels, respectively, of their figure 6; (c) parvocellular ganglion cell data from Lee, Sun, and Zucchini (2007) (yellow diamonds) measured at 3.43 log td from panel B of their figure 3; and (d) parvocellular (red squares) and magnocellular (white squares) LGN data from Derrington and Lennie (1984) measured at 3.13 log td from averaged data from their figures 7 and 12, respectively (in estimating the troland level for this work from cd/m², we assumed a pupil diameter of 2.5 mm).

The replotted physiological data shown in Figure 12 were all obtained with luminance rather than chromatic flicker, since the luminance data are more comparable with the psychophysical data we obtained with 560- and 650-nm monochromatic flicker. The log troland values in the key at the top of Figure 12 are the values from each paper plus 0.13 log₁₀ unit to account for differences between monkey and human eye (see Virsu & Lee, 1983). The physiological ganglion cell data include one set of parvocellular (Pa) data measured at 2.43 log td (orange triangles) and three sets measured at 3.43 log td (yellow triangles, yellow circles, and yellow diamonds); and one set of magnocellular (Ma) ganglion cell data also measured at 3.43 log td (gray circles). The LGN data include one set of parvocellular data (red squares) and one set of magnocellular data (white squares) both measured at 3.13 log td. To facilitate comparison, the various physiological estimates have been aligned with the psychophysical data between 1 and 5 Hz in each panel. 

The physiological data and early filter estimates in Figure 12 agree plausibly between 1 and 5 Hz. The agreement suggests that the indirect method of estimating the early filter at low frequencies is reasonably robust, and also supports the subtractive model of surround antagonism at low frequencies (see also, for example, Benardete & Kaplan, 1999a, 1999b). The consistent underestimate of the monkey data at 0.5 Hz suggests that the low-pass model of the center filter (i.e., the photoreceptor response) response should, in fact, be slightly band pass, which is also suggested by the S-cone data plotted in Figure 9 and by biphasic suction electrode recordings cone responses (e.g., Baylor, Nunn, & Schnapf, 1987; Schnapf, Kraft, & Baylor, 1987). 

The agreement between the psychophysical and physiological data at higher frequencies is clearly worse. Most of the physiological data were measured at either 3.13 or 3.43 log td, so they should be compared with the early filter estimate for the 3.41 log td level (orange lines). In general, the early filter estimate at this level falls off more steeply than the retinal ganglion cell data and less steeply than the LGN cell data, particularly in the case of the parvocellular cells. However, a relative loss of high-frequency sensitivity for the psychophysical data relative to monkey data should be expected because the signal-to-noise ratio rapidly decreases with increasing frequency (e.g., Lee, Sun, & Zucchini, 2007). Consequently, psychophysical measurements in which the observer has to discriminate a threshold signal from noise will suffer greater losses of sensitivity with increasing frequency than physiological measurements in which, typically, some moderately high modulation of the firing rate is used as the criterion cell response for defining the cell sensitivity. Thus, these comparisons are not at all inconsistent with the notion that the early filter is before the LGN, and therefore in the retina. 

Although the low-frequency attenuation suggested by our early filter shapes is consistent with the ganglion cell data shown in Figure 12, it remains a possibility that the nonlinearity is much later in the visual pathway and that the early filter shapes also reflect cortical feedback to the LGN (e.g., Sherman & Koch, 1986) and/or inhibition within the primary visual cortex (e.g., Priebe & Ferster, 2008). 

Conclusions

The nonlinearities that cause 560-nm flickering lights to change in brightness and 650-nm flickering lights to shift in hue lie in pathways with very similar temporal attenuation characteristics before and after the nonlinearity that distorts the flicker signal. The early filter is a band-pass filter peaking at 10–15 Hz with substantial low-frequency attenuation, which we model as subtractive as antagonism between center signals shaped by four-stage low-pass filter and surround signals shaped by the same filter with an extra low-pass stage and delayed by an extra time delay. The phase delays between the center and surround signals, which increase with frequency from being in opposite phase at 0 Hz, are important in producing the band-pass shape of the early filter by causing destructive interference at low frequencies, constructive interference at intermediate frequencies, and destructive interference again at high frequencies. We model the late filter as a two-stage low-pass filter with cut-off frequencies around 3 Hz. 

The similarity between the early and late filters derived from hue shifts and brightness changes suggests a common pathway for chromatic and brightness processing, which is most likely the parvocellular stream of processing. Brightness in this context is therefore distinct from the second of the two luminance channels usually linked to the magnocellular stream (e.g., Ingling & Drum, 1973; Ingling & Martinez, 1983; Ingling & Martinez-Uriegas, 1983). 

The band-pass filter attenuates mainly low temporal frequencies (and to a lesser extent higher frequencies), while relatively enhancing the temporal frequencies near its peak, thus accentuating its temporal tuning. The attenuation of low frequencies is an important mechanism of adaptation (for reviews, see Hood, 1998; Hood & Finkelstein, 1986). The purpose of the low-pass late filter may be to extract the slowly changing mean brightness and mean chromatic signals from the multiplexed chromatically and spatially opponent input signals. 

The initial distortion of the input signal in the model is caused by half-wave rectification, hypothesized to be due to the segregation of the visual signal into ON and OFF streams. The rectification encodes and enhances oppositely signed time-varying signals at middle to high temporal frequencies after von Kries adaptation at low temporal frequencies—probably at the photoreceptor level—has removed much of the mean (DC) signal (see for review and model Stockman, Langendörfer, et al., 2006). We speculate that hue shifts are caused by smoothly compressive nonlinearities in unipolar red and green chromatic mechanisms that follow the late low-pass filters. The distortion of the amplitude-modulated 650-nm stimuli at the early half-wave rectifier produces a larger red signal than green signal at the amplitude-modulation frequency, which is consequently more compressed by the compressive nonlinearities, so producing a hue shift towards yellow at the peak of the amplitude modulation. By contrast, the distortion of the amplitude-modulated 560-nm stimuli at the half-wave rectifier produces roughly equal red and green signals that are equally compressed by the later compressive nonlinearities, so producing no hue shift and only a brightness change. We suggest that spatial antagonism may prevent the 560-nm flicker signal from being distorted by a late compressive nonlinearity in the brightness pathway, although we have no direct evidence for a compressive nonlinearity in this pathway. 

Plausible sites for the half-wave rectifying nonlinearity are soon after surround antagonism possibly from horizontal cells. Comparisons with physiological recordings show some consistency between the early filter shapes and primate ganglion cell data. The smoothly compressive nonlinearity, which we suppose follows the late filter, could be relatively late in the processing stream. 

Acknowledgments

This work was supported by Biotechnology and Biological Sciences Research Council grant numbers BB/E016235/1 and BB/I003444/1. We thank Rhea Eskew and the two anonymous referees for detailed and important comments on the manuscript, and Barry Lee for helpful comments and his insistence that the compressive nonlinearity could not be early. 

Commercial relationships: none. 

Corresponding author: Andrew Stockman. 

Email: [email protected]. 

Address: UCL Institute of Ophthalmology, University College London, London, England. 

Appendix

The divisive model

The band-pass shape of the early filter revealed by our measurements was modeled by dividing a center temporal response of a cascade of filters by a surround temporal response of another cascade of filters, thus, the amplitude response, A(f), as a function of frequency, f, in Hz is:  

 

where f₀ are cut-off frequencies in Hz, tau are time constants in seconds, n are the number of cascaded filters, and the subscript “c” denotes the parameters of the central cascade of filters, and the subscript “s” denotes parameters of the surround cascade (e.g., Foley, 1994). The fits were made simultaneously to the hue change and brightness data for both observers, using the procedure described in Petrova et al. (2013b). Briefly, fits were made using the logarithm of Equation A1 with a logarithmic scaling constant, k, that could vary with target radiance. In terms of the model, k represents a frequency-independent sensitivity loss that is in addition to any losses resulting from the changing corner frequencies of the filters; it could be attributed merely to increased adaptation as the radiance level increases. As an expedient in fitting low-frequency and high-frequency measurements derived from different procedures, an extra arbitrary constant, v, was added to the low-frequency data for each level, the value of which was individually optimized for each set of data. This constant simply determines the best-fitting vertical alignment of the directly and indirectly measured data, thus:  

 

 

Best-fitting versions of the model were obtained using a standard nonlinear, least-squares, curve-fitting algorithm (implemented in SigmaPlot, SPSS) to account for the data obtained for each observer at each of the four time-averaged radiances. Equations A1 and A2 were fitted simultaneously to the estimates of the functions for both DP and GBH. 

The model was readily simplified by fixing those parameters that did not vary systematically with target radiances or across observers, in this case number of center filters, n_c, the number of surround filters, n_s, and the common corner frequency of the surround filters, f_0s. Also implicit in the use of Equation A1 is the simplification that at any target radiance, the cut-off frequencies of all center stages (f_0c) and all surround stages (f_0s) are the same. In preliminary fits the number of stages in both center and surround (n_c and n_s) were allowed to take on noninteger values but in the final fits, we fixed them at their nearest integer values (n_c = 6 and n_s = 2). (If n_c and n_s are separately allowed to take on noninteger values the best fitting values are 6.26 ± 0.25 and 1.82 ± 0.25, respectively.) Both scaling (the vertical logarithmic shift, k) and the center cut-off frequencies were allowed to vary between observers and between levels. 

The results of the final fit of the model for the early filters are shown as the continuous red lines in Figure A1. The parameters from the fit are tabulated in Table A1. The values of v are not given—these values are arbitrary because they depend on the relative sensitivities estimated at low and high frequencies, one of which was measured directly and the other of which was obtained indirectly by logarithmically differencing other sensitivities measured in different units.) The corner frequencies and sensitivity losses are slightly larger for DP than for GBH. 

Figure A1

 

Estimates of the logarithmic sensitivities corresponding to the attenuation characteristics of the early filter for DP (left-hand panel) and GBH (right-hand panel) obtained from 650-nm hue shift measurements (open and yellow symbols) and from the 560-nm brightness change measurements (smaller black and gray symbols)—both plotted as a function of frequency (Hz, logarithmic scale). Data are shown at each of the four L-cone equated, 650/560-nm time-averaged radiances: 9.10/8.26 (inverted triangles), 9.70/8.86 (circles), 10.33/9.51 (diamonds), and 10.93/10.11 (squares) log₁₀ quanta s⁻¹degree⁻². Error bars indicate ±1 SEM. The directly measured 560-nm brightness enhancement data (gray symbols) have been vertically aligned with the 650-nm hue shift data (yellow symbols) using a least-squares fitting criterion. The vertical axis is correct for the 650-nm hue change data measured at the lowest mean radiance plotted as amplitude sensitivities (top set) where the amplitudes are given in quanta s⁻¹degree⁻². For clarity, the amplitude sensitivities at the next three 650-nm mean radiances have been shifted down by an additional 1, 2, or 3 log₁₀ units, respectively. The vertical positions of the early filter estimates, inferred from differences between either the hue change (open symbols) or the brightness change (gray symbols) late filter estimates and conventional TCSFs, were determined by the fit of the divisive model, the final version of which is shown by the continuous red lines.

Table A1

 

Best-fitting parameters for the divisive early filter model. See text for details.

Parameter	DP	GBH
nc (fixed)	6
ns (fixed)	2
f0s (fixed)	1.22 ± 0.05 Hz
f0c 9.10/8.26	13.52 ± 0.35 Hz	11.45 ± 0.31 Hz
f0c 9.70/8.86	16.17 ± 0.41 Hz	13.88 ± 0.32 Hz
f0c 10.33/9.51	18.86 ± 0.47 Hz	16.37 ± 0.37 Hz
f0c 10.93/10.11	18.22 ± 0.42 Hz	17.44 ± 0.41 Hz
k 9.10/8.26	0.00 ± 0.05	0.00 ± 0.05
k 9.70/8.86	0.35 ± 0.05	0.21 ± 0.04
k 10.33/9.51	0.64 ± 0.05	0.50 ± 0.04
k 10.93/10.11	1.31 ± 0.04	1.05 ± 0.04
R2	0.997