Linear–nonlinear models of the red–green chromatic pathway

Abstract

The mean hue of flickering waveforms comprising only the first two harmonics depends on their temporal alignment. We evaluate explanatory models of this hue-shift effect using previous data obtained using L- and M-cone–isolating stimuli together with chromatic sensitivity and hue discrimination data. The key questions concerned what type of nonlinearity produced the hue shifts, and where the nonlinearities lay with respect to the early band-pass and late low-pass temporal filters in the chromatic pathways. We developed two plausible models: (a) a slew-rate limited nonlinearity that follows both early and late filters, and (b) a half-wave rectifying nonlinearity—consistent with the splitting of the visual input into ON- and OFF-channels—that lies between the early and late filters followed by a compressive nonlinearity that lies after the late filter.

Introduction

We recently reported that the mean hue of L- or M-cone–isolating flicker depended on the shape of the flickering waveform even though the mean (time-averaged) cone excitation remained constant. In particular, L- and M-cone–isolating sawtooth waveforms, varying around a mean yellow-appearing chromaticity, had different hues depending on whether the sawtooth was slowly-rising–rapidly-falling or rapidly-rising–slowly-falling. In general, the hue shifted in the direction of the slowly changing slope, so that the mean hue of L- or M-cone isolating sawtooth flicker that went slowly-redder–rapidly-greener appeared redder, whereas the hue of sawtooth flicker that went slowly-greener–rapidly-redder appeared greener (Stockman et al., 2017b). Measurements of the effects of varying either the modulation of the waveforms or the rates of change of their rising and falling slopes suggested that the shifts in the mean hue depended mainly on the first and second harmonics of the flickering waveforms (Stockman et al., 2017b). 

Panels B and D of Figure 1 show two cycles of the slowly-off and slowly-on sawtooth waveforms (solid black lines) together with the sums of their first and second harmonics (dashed blue and dot-dashed khaki lines, respectively). The shift in mean hue in the direction of the slowly changing phase suggested a nonlinear mechanism in the chromatic pathway that responds differently to the slowly-on and slowly-off sawtooth waveforms. This paper is concerned with modeling and understanding the hue shifts. The key questions that we address are the type or types of nonlinearity involved, and where the nonlinearities lie with respect to the temporal filtering that occurs in the chromatic processing stream. 

Figure 1

 

The black line in the top panel shows a sinusoid (first harmonic or fundamental) of period T and the colored lines show a set of second harmonics in a number of different phases and with half the amplitude of the first. The black lines in the other panels show waveforms used in the experiments together with the sums of their first and second harmonics. The names we use to describe the waveforms and the phase of their second harmonics are: Panel B: slowly-off (0°, blue short-dashed line); Panel C: peaks-align (90°, red solid line), Panel D: slowly-on (180°, khaki dotted-dashed line), and Panel E: troughs-align (270°, green long-dashed line).

Types of nonlinearity

We have proposed two types of nonlinearity that might account for the hue shifts (Stockman et al., 2017b). First, a slew-rate limited mechanism that restricts the rate at which internal representations that determine hue can change. This restriction prevents the visual system from following the rapidly changing but not the slowly changing slope of low-frequency sawtooth waveforms. As a result, the mean output moves in the direction of the slowly changing slope.¹ Second, a compressive (or saturating) nonlinear mechanism that instantaneously compresses hue signals—the larger the signal, the greater the compression—thus shifting the mean output of waveforms that are asymmetrical in extent, such as the rectangular waveforms shown as solid black lines in panels C and E of Figure 1. These waveforms should exhibit hue shifts in the direction opposite from the greater excursion from the mean. 

Temporal filtering in the chromatic pathway

It is difficult to know which of these two broad classes of nonlinear mechanism (and potentially other classes) is more likely to be responsible for the hue shift, because we are uncertain about where the nonlinearities lie in the chromatic processing stream, and in particular where they lie with respect to the substantial temporal filtering that occurs in the chromatic pathway (e.g., de Lange, 1958). Such filtering changes, in effect, the shapes of input waveforms as they are transmitted through the visual pathway by altering the phases and amplitudes of their harmonic components. Consequently, a sawtooth waveform at the input to the visual system is likely to be very different in shape when it arrives at the nonlinear stage that generates the hue shift. In order to understand the type of nonlinearity involved, we propose to exploit our knowledge of the likely shape of the waveform that produces the optimal hue shift when it arrives at a given type of nonlinearity. 

In our most recent work (Stockman, Henning, West, Rider, & Ripamonti, 2017a), we simplified the sawtooth waveforms by presenting only their first and second harmonic components. We then investigated the effect of varying the alignment of the second harmonic on the mean hue shifts. We represent the alignment in terms of the second harmonic's phase delay and adopted the convention that the second harmonic phase delay is zero when both first and second harmonic components are in sine phase. Panel A of Figure 1 shows two cycles of the fundamental component (first harmonic) as the black sinusoid in sine phase together with a set of second harmonics with half the amplitude of the first—the relative amplitudes of the first two harmonics of a sawtooth waveform. Different colored sinusoids show second harmonics at different second harmonic phase delays: 0° dashed blue line, which when added to the first harmonic, produces the slowly-off waveform shown in Panel B; 90° solid red line, which produces the “peaks-align” waveform shown in Panel C; 180° dotted-dashed khaki line, which produces the slowly-on waveform shown in Panel D; 270° long-dashed green line, which produces the “troughs-align” waveform shown in Panel E. 

With no filtering prior to the nonlinearity, the largest mean hue shifts caused by a slew-rate limited nonlinearity, or by a differentiator followed by a symmetrical nonlinearity¹, should occur with the slowly-off and slowly-on waveforms (panels B and D), which are asymmetrical in slope, whereas the largest hue shift with the compressive nonlinearity alone should occur for the peaks-align and troughs-align waveforms (panels C and E), which are asymmetrical in extent (see Stockman et al., 2017a, figures 12 and 13). 

Our measurements of mean hue shift as a function of second harmonic phase delay showed that the largest mean hue shifts towards red or green were roughly independent of frequency and occurred when the second harmonic phase delay was about 15° less than the phase delays of 90° and 270° that produce the peaks-align and troughs-align waveforms (Stockman et al., 2017a). Consequently, with respect to the two-component waveforms at the input to the visual system, the visual system as a whole behaves more like a system that simply compresses hue signals rather than one that limits their rate of change. 

However, these predictions do not take into account phase changes that are introduced before the nonlinearity—phase changes that alter the shape of the input waveform. Indeed, if the preceding filtering is substantial enough, then filtering before the nonlinearity might change the waveform shape from one that is optimal for a compressive nonlinearity to one that is optimal for a slew-rate limited nonlinearity. 

The purpose of this paper is to use modeling to try to understand the location of the nonlinearity or nonlinearities in the chromatic pathway (i.e., whether they are early or late in the pathway) and to determine the type of filtering that precedes or follows the nonlinearity (i.e., whether it is low-pass, high-pass, or band-pass). We eventually arrived at two plausible models: one with a slew-rate limited nonlinearity and the other with a simple compressive nonlinearity. 

The data to be modeled

We begin by assessing how well the models can simultaneously account for three important sets of data taken from two previous papers (Stockman et al., 2017a; Stockman et al., 2017b). First, (a) the chromatic temporal contrast sensitivity functions (TCSFs) that are often used to infer overall temporal processing in the chromatic pathway (e.g., de Lange, 1958; Kelly, 1974), second (b) the functions showing the dependence of the hue shift on the second harmonic phase delay in two-component waveforms, and third (c) the TCSFs for discriminating between the mean hue shift produced by a sawtooth flickering waveform and the same waveform with its polarity inverted. 

The original research adhered to the tenets of the Declaration of Helsinki. 

The experimental details can be found in our previous publications (Stockman et al., 2017a; Stockman et al., 2017b). Briefly, the experiments were programmed in MATLAB with the Cambridge Research Systems toolbox and presented via a VSG2/5 system (Cambridge Research Systems Ltd., Rochester, Kent, UK) on a carefully linearized and calibrated 21 in. Sony FD Trinitron CRT display. Each waveform was calculated as variations about a mean yellow field (CIE x, y color coordinates = 0.410, 0.514, luminance = 43.8 cd/m²), and could be L- or M-cone isolating or equiluminant, chromatic variations. Observers in the chromatic TCSF measurements viewed a circular 5.7° field and in the discrimination measurements compared different waveforms in two 5.7° semicircular fields separated by 0.6° and reported which half-field looked redder. For further details and for other procedures, please see the original publications. 

Figures 2 and 3 show some of the data that the models need to predict as well as some predictions from various models. We begin by describing the data. 

Figure 2

 

Data are shown from two observers: KR in the left-hand panel, and JA in the right-hand panel, together with a number of predictions. The orange diamonds are chromatic temporal contrast-sensitivity functions (TCSFs) in which log₁₀ sensitivity for sinusoidal equiluminant flicker is plotted as a function of frequency (Hz, logarithmic scale). The colored triangles represent sensitivity in the discrimination of slowly-off from slowly-on sawtooth waveforms either for L-cone isolating stimuli (red triangles) or for M-cone isolating stimuli (green triangles). The curves represent the attenuation characteristics of a number of different mechanisms discussed in the text. Note that the vertical position of the chromatic TCSFs relative to the hue-discrimination TCSFs in Figure 3 is essentially arbitrary, since we cannot easily relate the chromatic contrast that was varied to measure the chromatic TCSFs with the cone contrast that was varied to measure the hue discrimination TCSFs.

Figure 3

 

The triangles show the relative phase of the second harmonic in two-component stimuli that produce the biggest hue shift as a function of frequency (Hz, linear scale)—triangles for AS, inverted triangles for KR—red triangles for L-cone isolating stimuli, green triangles for M-cone isolating stimuli. The associated error bars represent the standard error of the estimates. The other symbols, different for three different observers, are the mid points of regions that have the same hue bounded by settings of second harmonic phases where the hue changes. The same data are shown in two locations to reflect the two directions in which the biggest hue changes can occur—one redward and one greenward. The horizontal gray lines indicate the phase that would produce the biggest hue shifts at the input either to the slew-rate limited mechanism (solid horizontal gray lines) or to a saturating nonlinearity (dashed horizontal gray lines). The curves and the right-hand label are discussed in the text.

Chromatic TCSFs

The overall attenuation characteristics of the model should be consistent with the chromatic TCSFs shown in both panels of Figure 2 as orange diamonds: for observers KR in the left-hand panel and for JA in the right-hand panel. These data are replotted from figure 2 of Stockman et al. (2017b) and show log₁₀ chromatic sensitivity as a function of frequency (Hz, logarithmic axis). To obtain these data, the modulation of equiluminant red/green sinusoidal flicker was varied in a two-alternative forced-choice experiment to find the relation between modulation and detection performance. The reciprocal of the modulation corresponding to 75% correct detection was taken as the “threshold” sensitivity and the logarithm of the sensitivity is plotted as orange diamonds in both panels as a function of frequency. The chromatic TCSF is often taken as an indication of the overall attenuation characteristic of the chromatic pathway and represented by a single filter (e.g., Swanson, Ueno, Smith, & Pokorny, 1987). Although the dip in the TCSFs at 3 Hz is found in the data for both observers, and might suggest the involvement of multiple chromatic mechanisms at low frequencies rather than a single low-pass mechanism, the dip is not a consistent feature of other chromatic TCSF measurements (e.g., Kelly & van Norren, 1977; Petrova, Henning, & Stockman, 2013b; Varner, Jameson, & Hurvich, 1984). However, a similar dip is seen in figure 2 of de Lange (1958), and Cass, Clifford, Alais, and Spehar (2009) have identified low-pass and band-pass chromatic mechanisms in masking experiments that might give rise to such a dip. 

Hue discrimination TCSFs

The model should also account for the L- and M-cone hue-shift discrimination thresholds shown in the panels of Figure 2 as red and green triangles (these data are also replotted from figure 2 of Stockman et al., 2017b). The results were obtained by a forced-choice method in which observers were asked to discriminate between the mean hues of matched but slowly-on and slowly-off sawtooth waveforms as the modulations of the waveforms to be discriminated were varied together. We found the modulation corresponding to 75% of the discrimination's being consistently red or green and define discrimination sensitivity as the reciprocal of these modulations; the logarithm of the discrimination sensitivity is plotted in Figure 2 as a function of frequency—red triangles for L-cone isolating stimuli and green triangles for M-cone isolating stimuli. 

The hue discrimination data should reflect both any filtering before the nonlinearity and the form of the nonlinearity (i.e., its input–output function), both of which will affect the visible distortion produced at the nonlinearity. Since the important distortion product is assumed to be mainly a mean or DC hue shift (the size of which will depend on the form of the nonlinearity), filtering after the nonlinearity should not alter the shape of the hue discrimination TCSF. The chromatic TCSFs will, of course, be affected by filtering both before and after the nonlinearity. As discussed below, given that the stimuli in the chromatic TCSF task are all near-threshold stimuli, the nonlinearity itself can, on the basis of small-signal linearity (e.g., Kelly, 1966; Lee, Pokorny, Smith, & Kremers, 1994), be assumed to have a minimal effect on the chromatic TCSFs. 

Second-harmonic phase delays

The phase characteristics of any filtering that occurs before the nonlinear mechanism should be consistent with the second harmonic phase delays that are plotted twice as the symbols in Figure 3. Again, since the visible distortion product after the nonlinearity is assumed to be mainly a DC shift, the second harmonic phase delays after the nonlinearity should be unimportant. In Figure 3, the symbols give the second harmonic phase delays (degrees) relative to the first harmonic in the two-component stimuli at the input to the visual system that produced the biggest mean hue shifts. Their fundamental frequencies ranged from 4 to 16 Hz (Stockman et al., 2017a). The label along the left ordinate of the figure corresponds to these measurements. The estimates are shown twice to illustrate that the greatest hue shifts occur at two different second harmonic phase delays for both L- and M-cone stimuli—one in the red direction and the other in the green direction. The data shown by the triangles for AS and inverted triangles for KR are based on the rectangular duty-cycle matches to mean hue made as a function of second harmonic phase (from figures 7–9 of Stockman et al., 2017a). There, the function relating the matching duty cycle of a rectangular waveform to the phase delay of the second harmonic was sinusoidal in form. From the fitted sinusoids, we extracted the second harmonic phases that produced the greatest mean hue shifts. The error bars are the standard errors of the phase in the sinusoidal fits. 

Data from another experiment are also shown in Figure 3: the half-filled diamonds for observer AS, the half-filled hexagons for VL, and the half-filled squares for RTE are the midpoints between the second harmonic phases of two-component waveforms at which the mean hue changes from green to red or vice versa (the data are from figure 10 of Stockman et al., 2017a). The data are included on the assumption that the midpoints correspond to the maximum hue shifts. 

The horizontal gray lines show, as a function of fundamental frequency, the second harmonic phases that, among the two-component waveforms, would lead to the biggest hue shifts at the input to the nonlinearity—solid gray lines at 0° and 180°, if the slew-rate limiting model were correct, dashed gray lines at 90° and 270° if the saturation model were correct. In Figure 3, the measured second harmonic delays are plotted relative to the second and first harmonic's being in sine phase at 0°. As noted before, the phases producing the largest hue shifts fit neither model; they are about 75° from the optimal phase predicted by the slew-rate limited mechanism and 15° from the optimal phase predicted by the symmetrical saturating nonlinearity. 

The measured estimates of optimal second harmonic phase relative to the first are roughly independent of frequency between 4 and 10 Hz. In order to achieve this independence, the phase characteristics of any successful filter preceding the nonlinearity must approximate a straight line over a substantial range of frequencies above 4 Hz. Note that fixed time delays have straight line phase characteristics, but because they do not change the shape of the waveform they preserve the relative phases of all the harmonics. Plotted as the second harmonic relative to the first, as in Figure 3, a time delay would be a horizontal line at 0°. 

Our approach has been to develop models and to fit them simultaneously to the second harmonic phase delay data and to the chromatic TCSFs. Many models were tried, including low-pass filter cascades, band-pass filters, models with different nonlinearities, and models with different sequences of nonlinearities and filters. Although a simple low-pass filter can predict TCSFs well, it fails because its phase response changes too fast above 4 Hz to predict the data in Figure 3. Models with a single compressive nonlinearity after the early and late chromatic filters fail because filters that can fit the TCSFs of Figure 2, cannot fit the phase-delay data of Figure 3. Below, we describe two models that fit both sets of data. 

Figure 4

 

Illustration of late slew-rate limited model. The early filter arises from a common network of L- and M-cones (small red and green trapezoids, respectively) influencing here the response of an L-cone (large red trapezoid). There follows a two-stage low-pass filter (the late filter) and then a slew-rate limiting stage. For details see text.

We adopt the convention of plotting the filter characteristics in Figures 3 and 5, below, as phase advances (in accordance with the label along the right-hand ordinates of the figure). By using this convention, if the model's predictions are consistent with the data, the filter phase characteristics plotted as advances and the phase measurements plotted as delays will coincide. 

Figure 5

 

Data replotted from Figure 2 together with predictions discussed in the text.

Figure 6

 

Data replotted from Figure 4 together with predictions discussed in the text.

Models

A late slew-rate limited nonlinearity

We initially placed the slew-rate limited nonlinearity after the low-pass temporal filtering in the chromatic pathway; that is, after the filtering implied by the chromatic TCSF, which is low-pass in shape, and evaluated simple low-pass filters composed of n leaky integrator stages. However, a simple low-pass filter is an implausible approximation of chromatic filtering, since, for one thing, it ignores the prominent surround inhibition that occurs in the retina (e.g., Lee, Martin, & Grünert, 2010). 

A more realistic model of chromatic filtering comes from earlier work from our laboratory shown schematically in Figure 4 (Petrova et al., 2013b; Stockman, Petrova, & Henning, 2014). In that work, we measured the magnitude of the distortion products produced by amplitude-modulated flicker in order to dissect chromatic temporal processing into a cascade of an early band-pass filter and a later low-pass filter. Thus, the model allowed us to split filtering in the chromatic pathway into early and late stages. The early filter is shown in the left-hand pink area of Figure 4 as a network of L- and M-cones represented by the interconnected red and green trapezoidal elements. In this example, an L-cone (represented by a large red trapezoid) is opposed by a surround driven by a mixture of L- and M-cones. Two stages of a hypothetical inhibitory horizontal cell network are shown. The spatially distributed inhibitory input from neighboring cones is represented by clusters of smaller trapezoids in the ratio of two L-cone centered contributors to one M-cone centered contributor—the approximate proportion that appears to represent most normal retinae (e.g., Carroll, Neitz, & Neitz, 2002; Cicerone & Nerger, 1989; Hofer, Carroll, Neitz, Neitz, & Williams, 2005; Sharpe, Stockman, Jagla, & Jägle, 2011; Vimal, Smith, Pokorny, & Shevell, 1989). Note that the assumption of a 2:1 L:M cone ratio in this and other models plays no direct role in the modeling. 

The early filters are chromatically opponent by virtue of the horizontal cell feedback having inputs from the opposing cone type—in this example from an M-cone (e.g., Dacey, 2000; Field & Chichilnisky, 2007; Rodieck, 1998; Wässle, 2004). 

The late filter, shown in the blue-gray region of Figure 4, is an n-stage low-pass filter of the form given in Equations 1 and 2 with n = 2:  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}A_l\left( f \right) = {G_l}{1 \over {{{\sqrt {\left( {{f^2} + f_{cl}^2} \right)} }^n}}}{\rm ,}\end{equation}

 

\begin{equation}\tag{2}\varphi_l \left( f \right) = n\,{\tan ^{ - 1}}\left( {{f \over {{f_{cl}}}}} \right){\rm ,}\end{equation}

where Display Formula\(f\) is frequency (Hz), Display Formula\(A_l\left( f \right)\) is the attenuation characteristic (or amplitude response) of the filter, and Display Formula\(\varphi_l \left( f \right)\) is its phase response. The parameter Display Formula\({f_{cl}}\) is the corner frequency (Hz) of each of the n identical, cascaded, low-pass filters and Display Formula\({G_l}\) is the gain of the filter. The corner (or cut-off) frequency, Display Formula\({f_{cl}}\), indicates the frequency above which the filter attenuation begins to increase significantly; it is the frequency at which the attenuation of a cascade of n identical low-pass filters has increased by Display Formula\({\sqrt 2 ^n}\) from its low-frequency asymptotic level (see Equation 1).  

The retinal complex constitutes the early filter and is modeled as four leaky integrating stages (low-pass filters) and two stages of feedforward inhibition (modeled as lead compensators or lead-lag filters; Rider, Henning, & Stockman, 2016). The resulting early filter is band-pass in form, with three free parameters:  

\begin{equation}\tag{3}A_e \left( f \right) = {G_e}{{{{\sqrt {\left( {{f^2} + {{({f_{ce}}(1 - k))}^2}} \right)} }^2}} \over {{{\sqrt {\left( {{f^2} + f_{ce}^2} \right)} }^6}}}{\rm ,}\end{equation}

 

\begin{equation}\tag{4}\varphi_e \left( f \right) = 6\,{\arctan}\left( {{f \over {{f_{ce}}}}} \right) - 2\,{\arctan}\left( {{f \over {{f_{ce}}\left( {1 - k} \right)}}} \right),\end{equation}

where Display Formula\(f\) is again frequency (Hz), Display Formula\(A_e \left( f \right)\) is the attenuation characteristic (or amplitude response) of the filter, and Display Formula\(\varphi_e \left( f \right)\) is its phase response. This somewhat formidable formulation is a compact way of representing a network of four identical low-pass filters and two lead-lag filters all with corner frequency, Display Formula\({f_{ce}}\). The overall gain of the early filter is G_e, and the gain of the feedforward inhibition of the lead-lag filters is k (if k = 1 the lead-lag filter is a standard high-pass filter, while if k = 0, it is an all-pass filter). The lead-lag filter, which is mathematically equivalent to the filter that we called “divisive” in our earlier work (Petrova et al., 2013b; Petrova, Henning, & Stockman, 2013a; Stockman et al., 2014), is designed to capture the lateral interactions that contribute to the early filter as sketched in the left-hand panel of Figure 4. Considered alone, the early filter has three free parameters and the late filter has two but, since the two gains cannot be estimated independently, there are only four free parameters. To capture the overall gain, the gain of either the early or late filter can be set to unity, so the overall model has just four free parameters to be estimated from the data.  

The data to be fitted are the chromatic TCSFs (plotted as orange diamonds for KR and JA in Figure 2) and the estimates of the second harmonic phase delays relative to the first harmonic that, among the two-component waveforms we used, produce the biggest mean hue shifts (plotted as colored symbols in Figure 3). 

The model predictions for the TCSFs are given by the product of Equations 1 and 3, and the phase characteristic of the cascade of early and late filters by the sum of Equations 2 and 4. From that sum, the model predictions for the second harmonic phase advance relative to the first, Display Formula\({\varphi _2}(f)\), can be calculated as:  

\begin{equation}\tag{5}{\varphi _2}\left( f \right) = 6\,{\arctan}\left( {{{2f} \over {{f_{ce}}}}} \right) - 2\,{\arctan}\left( {{{2f} \over {{f_{ce}}\left( {1 - k} \right)}}} \right) + \ 2\,{\arctan}\left( {{{2f} \over {{f_{cl}}}}} \right) - \ 2\biggl[ {6\,{\arctan}\left( {{f \over {{f_{ce}}}}} \right) - 2\,{\arctan}\left( {{f \over {{f_{ce}}\left( {1 - k} \right)}}} \right) + \ 2\,{\arctan}\left( {{f \over {{f_{cl}}}}} \right)} \biggr]{\rm .}\end{equation}

 

We used Equations 1, 3, and 5, varied Display Formula\({f_{ce}}\), Display Formula\({f_{cl}}\), k, and an overall gain factor, G, to simultaneously optimize the fit to the amplitude and phase data for a model with the slew-rate limiting mechanism following the late filter. 

In all the fitting procedures, the model (Equations 1, 3, and 5) is implemented and the values of each of the free parameters (Display Formula\({f_{ce}}\), Display Formula\({f_{cl}}\), k, and G in the present case; n was fixed at integer values) are varied to find the combination that produces the smallest mean square difference between the predictions of the model and the data. The value of n was constrained to take on integer values in the final fits of each model. The fits were made to the phase delays measured in radians, because the phase delays in radians are comparable in magnitude to the log contrasts. The curve fitting procedure was the nonlinear regression implemented in SigmaPlot (Systat Software, San Jose, CA) based on the Marquardt-Levenberg algorithm (Levenberg, 1944; Marquardt, 1963) that minimizes the sum of the squared differences between the data and model predictions. Note that for nonlinear regression, R² as a measure of goodness-of-fit is problematic (e.g., Kvalseth, 1985; Spiess & Neumeyer, 2010), so that as well as giving R² values, we also give the standard error of the regression. Note that since the fits were in radians, the standard errors for the phase delay fits are also in radians. 

For the slew-rate model to predict the phase data, the cascade of early and late filters must advance the phase of the second harmonic relative to the first so that the phase delays of the second harmonic at the input to the nonlinearity are either 0° or 180° as indicated by the horizontal gray lines in Figure 3 labeled “Slew phase.” These fits were again done simultaneously across the amplitude and phase data for all the observers using the same parameters with the exception that a different overall gain factor (Display Formula\(G = {G_e}{G_l}\)) was allowed for KR and JA for the TCSF data. 

For the late slew-rate limit model, the best fitting parameters were: Display Formula\({f_{ce}}\) = 27.67 ± 1.97 Hz, Display Formula\({f_{cl}}\) = 2.08 ± 0.37 Hz, and k = 0.78 ± 0.02. G in log₁₀ units was 11.02 ± 0.15 for KR and 10.81 ± 0.15 for JA. The R² value for the fit was 0.834 and the standard error of the regression was 0.202. 

The predictions for the second harmonic phase are shown separately for the early and late filters in the lower half of Figure 3. The blue line shows the second harmonic phase relative to that of the first for the late (low-pass) filter, the red line for the early (band-pass) filter. However, the crucial prediction, for the cascade of the early and late filter is shown in both the upper and lower sections by the orange lines. As can be seen, the phase advances of the second harmonic produced by the combined filter agree with the second harmonic phase delays in the stimuli that produce the maximum hue shifts. According to this model, the stimuli at the nonlinearity that produce the maximum hue shifts have second harmonic phase delays that closely approximate the slowly-off and slowly-on sawtooth stimuli. 

The blue and red lines in Figure 2 show the attenuation characteristics of the early and late filters derived from the fits. Note that the red and blue lines have been plotted at arbitrary relative positions on the ordinate as their separate gains cannot be estimated independently from the data. The orange line shows the cascaded attenuation characteristics of these early and late filters. As can be seen, the cascade accounts well for the chromatic TCSF data from 6.67 to 13.33 Hz, although it overestimates the measured sensitivity at 2 and 4 Hz. 

The third set of data for which the model should also account are the hue discrimination data shown in Figure 2 as red and green triangles. These results are more difficult to predict since, although the observers reported that they based their judgments on mean hue shifts, we do not know the characteristics of the outputs of the second filter on which the judgments depend. However, in our earlier work (Stockman et al., 2017a) we suggested that the discriminability of the hue shifts depends on the transmission of the second harmonic by the chromatic filter. Consequently, we might expect the shape of the hue-discrimination TCSF to depend on the shape of the chromatic TCSF but at double the fundamental frequency. Then the sensitivity difference between the hue shift discriminability at, for example, 5 and 10 Hz, should depend on the sensitivity difference between the detection of chromatic sinusoids at 10 and 20 Hz, and so on. In terms of shapes, this is equivalent to shifting the logarithmic chromatic TCSF functions (orange lines) by 0.3 log unit along the log frequency scale toward lower frequencies (or on a linear frequency plot halving the frequency), as shown by the red-green dashed lines. The shape of the shifted function agrees with the hue-discrimination data (red and green triangles), which supports the suggestion that the second harmonic transmitted through the late filter is the important factor (but see below). 

An intermediate compressive nonlinearity

For the late nonlinearity to be a saturating nonlinearity, in addition to accounting for the TCSF data, the filters must together delay the phase of the second harmonic relative to the first so that the phase delays of the second harmonic at the input to the nonlinearity are either 90° or 270°. However, we could not find a pair of early and late filters that would account for both the amplitude and the phase data simultaneously, but we could find a combination in which the early filter alone accounts for the phase data while the early and late filter together account for the amplitude data. This suggests that the important hue-shifting nonlinearity might lie between the early and late filters. In this section, we investigate a model with a nonlinearity between the early and late filters. 

The model predictions for the second harmonic phase advance relative to the first, Display Formula\({\varphi _2}(f)\) for the early filter alone can be calculated from Equation 4:  

\begin{equation}\tag{6}{\varphi _2}\left( f \right) = 6\,{\arctan}\left( {{{2f} \over {{f_{ce}}}}} \right) - 2\,{\arctan}\left( {{{2f} \over {{f_{ce}}\left( {1 - k} \right)}}} \right) - \ 2\biggl[ {6\,{\arctan}\left( {{f \over {{f_{ce}}}}} \right) - \ 2\,{\arctan}\left( {{f \over {{f_{ce}}\left( {1 - k} \right)}}} \right)} \biggr]{\rm .}\end{equation}

 

In this model, the early filter (Equation 6) should adjust the relative second harmonic phase at the input to align with the saturation phase predictions shown by the dashed and solid lines in Figure 6, while the cascade of early and late filters (Equations 1 and 3) should predict the chromatic TCSFs shown in Figure 5 by orange diamonds. 

The best-fitting parameters for a model with the saturating nonlinearity between the early and late filters were: Display Formula\({f_{ce}}\) = 20.67 ± 1.22 Hz, Display Formula\({f_{cl}}\) = 2.68 ± 1.49 Hz, k = 0.55 ± 0.02, and G, in log₁₀ units, was 9.27 ± 0.11 for KR and 9.06 ± 0.11 for JA; the fits to the TCSFs are shown as orange lines in the panels of Figure 5. The fit is remarkably good and as good as the fits of the slew-rate limiting model in that the TCSF is well fit over the entire range of frequencies. The R² value of 0.835 and the standard error of the regression was 0.201. 

The model fits to the chromatic TCSF data, shown by the solid orange lines in each panel of Figure 5, are the combined amplitude characteristics of the cascade of the optimized early band-pass filter (red line), and the optimized late low-pass filter (blue line). One unexpected result is that instead of the low-pass late filter's having two stages, the best fit to the chromatic TCSFs was obtained with a single stage, so that n = 1 in Equations 1 and 2. As well as changing its attenuation characteristics, this halves the phase delays caused by the late filter. The model fits to the phase data are shown by the red line in Figure 6. Again, the fits are good. 

Note here that in assuming that the chromatic TCSF can be modeled as the cascade of the early and late filters, we are ignoring any effects of the now intervening nonlinearity. It is generally reasonable to assume that small, near-threshold signals behave approximately linearly (e.g., Kelly, 1966; Lee et al., 1994), although, of course, “threshold” signals at the output of the late filter might have been produced by large amplitude inputs to that filter (we address this issue for half-wave rectification in Appendix 1). 

In this particular model, the distortion signal that underlies the hue discrimination is produced by the nonlinearity prior to the late filter. That distortion is a shift in the mean and apart from the gain of the late filter, which affects all frequencies equally, the attenuation characteristics of the late filter should not affect the hue shift. This means that hue discrimination as a function of frequency should follow the attenuation characteristics of the early filter alone. The early filter's attenuation hardly changes over the 4–13 Hz range (red lines in Figure 5) and is obviously dissimilar to the steeply low-pass shape of the hue-shift discrimination data (red and green triangles in Figure 5). Therefore, this model, with a single intermediate nonlinearity, cannot predict the hue discrimination data. For similar arguments, see other descriptions of the linear–nonlinear–linear “sandwich” models (e.g., Marmarelis & Marmarelis, 1978; Petrova et al., 2013b; Victor, Shapley, & Knight, 1977). 

To preserve the approach of using a saturating nonlinearity, we can turn again to the model originally proposed by Stockman et al. (2014) to explain their flicker distortion results, which is illustrated in Figure 7. As in Figure 4, the left-hand pink region of Figure 7 shows the early filter as cone networks with, in this sketch, an upper network with a central L-cone and a lower one with a central M-cone. Half-wave rectifiers at the output of each network split the networks' outputs into ON- and OFF-signals. The four half-wave rectifiers are represented by the input–output functions shown in the green area of the diagram. The half-wave rectifiers are designed to capture cells with four types of chromatic receptive field shown as concentric circles. In the fovea, where midget parvocellular bipolar cells predominantly contact single L- or M-cones (Calkins, Schein, Tsukamoto, & Sterling, 1994; Kolb & Dekorver, 1991), the four types are: L-ON, M-ON, L-OFF, and M-OFF, but each will vary according to the cone types driving the cells' surrounds (Derrington, Krauskopf, & Lennie, 1984; Dreher, Fukada, & Rodieck, 1976; Lee, Valberg, Tigwell, & Tryti, 1987; Wiesel & Hubel, 1966). The black lines in the squares indicate the half-wave rectification, while color variation in the squares and in the centers of receptive fields indicate the direction of chromatic variation associated with increases and decreases in the cone excitations driving the receptive-field centers. 

Figure 7

 

Model of the chromatic visual system from Stockman et al. (2014). The left-hand side in the pink region shows two networks of L- and M-cone receptors represented as red and green trapezoids, respectively. The upper cluster produces signals that drive ON and OFF centered units with chromatically opponent center and surround (concentric disks) both with L-cone driven centers (L_ON and L_OFF). The lower cluster is similarly divided but with M-cone driven centers (M_ON and M_OFF). All four units are half-wave rectified as indicated in the squares in each pathway. The L_ON and M_OFF units are combined into a “Red” channel because increases in L-cone excitation and decreases in M-cone excitation are generally perceived as changes in hue towards red. The M_ON and L_OFF units are linked in the “Green” channel because increases in M-cone excitation and decreases in L-cone excitation are generally perceived as changes in hue towards green. The signals in the Red and Green channels are then filtered by identical late two-stage low-pass filters before compressive (saturating) nonlinearities.

The half-wave rectifiers, which partition the signals from the L- and M-cone photoreceptors into ON and OFF signals, are now the intermediate nonlinearities between the early and late filters. The half-wave rectifiers are assumed to occur at the first visual synapse where the L- and M-cones jointly contact diffuse and midget ON bipolar cells via sign-inverting synapses and contact OFF bipolar cells via sign-conserving synapses (e.g., Boycott & Dowling, 1969; Boycott & Wässle, 1991; Calkins et al., 1994; Grünert, Martin, & Wässle, 1994; Kolb & Dekorver, 1991; Polyak, 1941; Wässle & Boycott, 1991). Since cones hyperpolarize in response to light (i.e., their response becomes more negative), ON bipolar cells depolarize (their response becomes more positive) as the light is increased, whereas OFF bipolar cells depolarize as the light is decreased. Half-wave rectification in the retina is not complete, since the responses of ON and OFF neurons in the retina can fall below their resting levels to the lower limit of the cell's response (e.g., Kuffler, 1953), but it is complete in the cortex (e.g., Albrecht & Geisler, 1991; Heeger, 1991; Movshon, Thompson, & Tolhurst, 1978). 

In Figure 7, the outputs of the L_ON and M_OFF receptive fields are combined in a unipolar “Red” channel, and the outputs of the M_ON and L_OFF receptive fields are combined in a unipolar “Green” channel before separately reaching identical late filters. Evidence in support of unipolar Red and Green mechanisms as opposed to bipolar red–green mechanisms is discussed by, for example, Eskew (2008), Sankeralli & Mullen (2001), and Wuerger, Atkinson & Cropper (2005). Our previous work suggested that the late filter acts as a two-stage low-pass filter that begins to attenuate chromatic flicker about 3 Hz (Petrova et al., 2013b; Stockman et al., 2014). However, the fits in the previous section suggest that the late filters might be single-stage, and that is how they are depicted in the blue-gray region of Figure 7. Lastly, compressive late or central nonlinearities in the red and green pathways are shown in the right-hand khaki region and represented by a saturating function. 

In the Stockman et al. (2014) model, the half-wave rectifiers were assumed to account for the brightness enhancement that accompanies flicker, while the half-wave rectifiers plus the late saturating nonlinearity together accounted for the hue changes that accompany flicker. The crucial idea was that the flicker distortion at the half-wave rectifiers was then further distorted at the saturating nonlinearity to produce the hue shifts that they measured (for discussion and reviews of these effects, see Petrova et al., 2013a; Petrova et al., 2013b; Stockman et al., 2014). We employ a similar model here. 

For the modified model with two nonlinearities, we again use the early and late filters derived above, since as before, the cascade of early and late filters accounts for the chromatic TCSF and the early filter accounts for the phase data. To reiterate, the model parameters are Display Formula\({f_{ce}}\) = 20.67 ± 1.22 Hz, Display Formula\({f_{cl}}\) = 2.68 ± 1.49 Hz, k = 0.55 ± 0.02, and G = 9.27 ± 0.11 for KR and 9.06 ± 0.11 for JA. As already shown, this model fits the chromatic TCSFs and the phase data, but how can an intermediate nonlinearity made up of half-wave rectifiers and a late nonlinearity made up of a compressive input-output function account for the hue-shift discrimination TCSFs (red and green triangles, Figure 5)? To explain the final version of this model, we start with an illustrative example of the effects of the half-wave rectifier and the saturating nonlinearity on the two-component, troughs-align, L-cone–isolating waveform shown at A in Figure 8. 

Figure 8

 

An illustrative simulation showing an L-cone-isolating stimulus (A). The “troughs-align” stimulus (A) is to be discriminated from the matched “peaks-align” stimulus (A′). The troughs-align stimulus is taken through the model and is passed through the early band-pass filter (denoted by its impulse response as the gray icon) to produce the filtered signal (B). This is then divided into ON and OFF pathways by half-wave rectifiers (denoted by the input–output relations shown as red-green icons) to produce ON and OFF signals (C1 and C2). Signals in both pathways then pass through one-stage low-pass filters (denoted by their impulse responses as the red and green icons) to produce outputs at D1 and D2, which then enter a compressive nonlinearity (denoted by the input–output relations as the gray-red and gray-green icons) that distorts them to produce the final outputs E1 and E2. The black dashed lines in A and B show the mean of the input signal and early filtered signal, respectively. The red and green dashed lines show the means in the ON and OFF pathways, respectively. Both green and red dashed lines in have been plotted in E2 and E1 for ease of comparison. The signal shown in A′ would produce a larger “green” than “red” mean at E2 and E1.

The input signals to be discriminated are shown at the top of Figure 8: in panel A is a two-component (first and second harmonic) troughs-align waveform with a fundamental frequency of 5 Hz, in which the second harmonic is phase delayed by 259° relative to the first, and is half the amplitude of the first. The inverse waveform from which it is to be discriminated has the second harmonic shifted by 180°; it is peaks-align and is shown in panel A′. Increase in the L-cone input is shown as positive in both cases. We trace the troughs-align (A) waveform through the model. The troughs-align signal passes through the early band-pass filter (shown as its impulse response in the gray box) to produce the slightly different signal in panel B. Note that the change in shape is minor because the amplitude responses of the early filter at 5 and 10 Hz are similar (approx. 0.06 log unit difference; red curves in Figure 5), and the phase response is approximately linear in this range (producing a phase delay of 25° at 5 Hz and 61° at 10 Hz). The mean signal is shown as the dashed horizontal line. The filtered signal is then half-wave rectified around the signal mean to produce the ON and OFF signals shown in panels C1 and C2, respectively. The mean signals, which are equal at this stage, are shown as the dashed green and red horizontal lines in C1 and C2, respectively. Then, each of the ON and OFF signals are filtered by the single-stage late filter with a corner frequency of 2.68 Hz to produce the filtered ON and OFF signals shown in panels D1 and D2, again with equal means. Finally, the filtered ON and OFF signals are compressed by an instantaneous saturating nonlinearity. The exact choice of nonlinearity does not materially affect the results (we tried several); here we choose a function (Naka & Rushton, 1966) that is linear for small values and saturates towards a maximum value of 1:  

\begin{equation}\tag{7}{ {r\left( x \right) = \sqrt {{{{x^2}} \over {{x^2} + {x_0}^2}}} ,} } \end{equation}

where Display Formula\(x\) is the instantaneous signal intensity at the late nonlinearity and Display Formula\({x _0}\) is the intensity at which the compressed response is Display Formula\({1 \over {\sqrt 2 }}\) times its maximum. This produces the compressed ON and OFF signals shown in panels E1 and E2. The dashed lines in panels E1 and E2 show both the ON (red dashed lines) and the OFF (green dashed lines) means for comparison. Because of compression, the mean in the ON channel (red dashed line) is higher than the mean in the OFF channel (green dashed line). We assume that the mean hue reported by our observers shifts in the direction of the larger mean signal so that, for this L-cone example with a troughs-align stimulus, the hue shifts towards red, even though the input stimulus modulates around a mean yellow-appearing chromaticity. Notice that the means of the ON and OFF signals are the same until after the final compressive nonlinearity. As noted above, in our experiments, observers viewed the two-component waveform and its inverse in two 5.7° semicircular fields separated by 0.6° and reported which half-field looked redder. Panel A′ shows the inverse waveform of that shown in panel A. Passing the waveform in A′ through the model stages produces mean hue shifts in the opposite hue directions.  

The predictions of this model depend on the fact that when two-component first and second harmonic waveforms are half-wave rectified the relative modulations of the first and second harmonic components (and other components) that appear in the ON and OFF pathways depend on the phase delay between the second and first harmonics before the rectification. In the example shown in Figure 8, the first harmonic of the troughs-align waveform after filtering and rectification is about 60% larger in the OFF pathway than in the ON; for the peaks-align waveform (not shown), the first harmonic is about 60% larger in the ON pathway than in the OFF pathway; and for the rapid-on and rapid-off waveforms the first harmonic amplitudes in the ON and OFF pathways are equal. The late low-pass filter selectively attenuates the higher harmonics, so that the important harmonic components that reach the late saturating nonlinearity are the DC (which is equal in the two pathways), the first harmonic, and to a lesser extent the second harmonic. In general, the ON or OFF pathways with the larger first harmonic (and therefore with the greater excursions away from the mean) will be more compressed, causing a DC shift at the output of the late nonlinearity as shown by the dashed red and green lines in panels E1 and E2 of Figure 8. Crucially, then, since the amount of compression depends on the sizes of the first harmonic reaching the late nonlinearity, it will also depend on the attenuation of that harmonic by the late filter. 

The half-wave rectification alters the amplitude of the components that are present in the input signals and also introduces new harmonic components. Figure 9 shows how the post-rectification amplitudes of the DC (black line), the first (red lines), second (green lines), third (blue line), and fourth (purple line) harmonics in the ON and OFF pathways depend on the relative phase of the second harmonic for two-component first and second harmonic waveforms in the amplitude ratio of 1.0 to 0.5 before the half-wave rectifiers (here we ignore the effect of the early filter, which will produce only minor changes in the amplitude ratio for low fundamental frequencies). The solid lines show the amplitudes of the harmonic components in the ON pathway and the dashed lines those in the OFF pathway all as a function of the phase delay (degrees) in the second harmonic. The DC, third and fourth harmonic components (created by the nonlinearity) have the same amplitude in the ON and OFF pathways so, for those three, only the solid lines are shown. 

Figure 9

 

Effect of half-wave rectification and partition into ON- and OFF-channels on the DC component and the first four harmonics of waveforms from a half-wave rectifier in response to an input waveform that consists only of the first and second harmonics. Amplitude at the output of the half-wave rectifiers is shown as a function of the phase of the second harmonic at the input to the half-wave rectifiers. At the input, the second harmonic had half the amplitude of the fundamental. Solid lines show the amplitudes in the ON pathway, dashed lines, the amplitude in the OFF pathway. The DC component (black line), the third (blue line), and fourth (purple line) harmonics are identical in the ON and OFF pathways but the first (red lines) and second (green lines) differ markedly in the two pathways.

Figure 9 illustrates that for the slowly-off and slowly-on waveforms with second harmonic phase delays relative to the first harmonic of 0° and 180° at the input to the half-wave rectifiers, the amplitudes of the first (fundamental) and second harmonics in the two pathways are equal in amplitude so no hue shift would be produced. But for other phases there are amplitude differences between the ON and OFF pathways. Further, and consistent with the saturation model (see above), the differences are largest at 90° (peaks-align) at which the first harmonics are approximately 60% larger in the ON than in the OFF pathway, and at 270° (troughs-align) where the signals are 60% larger in the OFF pathway. The second harmonics also have their largest difference at 90° and 270° with a roughly four-fold difference between the ON and OFF signals. Note that the first and second harmonics of the half-wave rectified signals vary together as the phase delay of the second harmonic of the input is varied, so that they can synergistically boost their sum in the ON or OFF pathway before the compressive nonlinearity. However, in absolute terms, the second harmonic amplitude is always between 1.5 and 4 times smaller than the corresponding first harmonic, and after the late filter, will be smaller still, so that the half-wave rectifying nonlinearity will mainly affect the size of the first harmonic for these stimuli. 

We can now account for the hue-discrimination thresholds shown as triangles in Figure 5 using this modified model. Since both the early and the late filter attenuate the first harmonic signal that undergoes compression to produce the mean hue shift, the dependence of hue-shift discrimination on frequency should be at least as steep as the chromatic TCSF to which both early and late filters contribute. The discrimination data is in fact steeper, which may suggest that the hue discrimination also depends on the shape of the saturating nonlinearity and its input–output function. The most significant term in Taylor's expansions of a smoothly compressive nonlinearity, such as a logarithmic or power function, is quadratic (see Kelly, 1981). Thus, we might expect the shape of the hue-discrimination TCSF to be the square of the chromatic TCSF (i.e., the sensitivity squared), and that is what we find. The red–green dashed lines aligned with the hue-shift discrimination threshold are the squared and vertically aligned chromatic filter in Figure 5. 

Some of the small deviations we see between our data and the model could reflect retinal inhomogeneities over the relatively large target size of 5.7°. Inhomogeneities in macular pigment density and photoreceptor optical density (Stockman & Sharpe, 1999), for example, could result in cone-isolating stimuli generating small signals in the nominally silenced cone. In addition, since the stimuli extend beyond the fovea, they will also excite parvocellular cells with center responses that derive from more than one cone (e.g., Kolb & Marshak, 2003), which might slightly change the model parameters (e.g., the feedback, k). 

Summary

We are now left with two plausible models that can account for our data. One is a linear–nonlinear model with a slew-rate limited nonlinearity after the early and late filters (Figure 4), and the other is linear–nonlinear–linear-nonlinear model with an intermediate half-wave rectifying nonlinearity between the early and late filters and a compressive nonlinearity after them (Figure 7). We cannot, however, distinguish between these two models on the basis of our existing data. 

A crucial difference between the two models, and a possible way to distinguish them, is their predictions of the effect of higher harmonics on the hue shifts produced by complex waveforms. Higher harmonics that are significantly attenuated by the late filter should have no effect on the hue shift generated at the late slew-rate nonlinearity in the linear–nonlinear model. By contrast, they may alter the hue shift in the linear–nonlinear–linear–nonlinear model because they reach the half-wave rectifier where they can change the relative sizes of the first and second harmonics carried in the ON- and OFF-pathways, and thus affect the hue shift produced at the late compressive nonlinearity (even though, if presented alone, they might be invisible). 

Acknowledgments

This work was supported by grants BB/1003444/1 and BB/M00211X/1 from the BBSRC. We thank Rhea Eskew for helpful advice and comments. 

Commercial relationships: none. 

Corresponding author: Andrew Stockman. 

Email: . 

Address: UCL Institute of Ophthalmology, University College London, London, UK. 

Footnotes

¹  A related possibility is a mechanism made up of a differentiator followed by a symmetrical compressive (saturating) nonlinearity (Cavanagh & Anstis, 1986), but since this mechanism entails similar hue-shift predictions we just consider the slew-rate limited mechanism.

Appendix

For the late slew-rate limited model, we formed the cascade of the early and late filters and calculated the attenuation characteristics of the cascade as the product of the attenuation of the constituent filters and the phase response of the cascade as the sum of the phase response of the constituents. This is an appropriate procedure when nothing but linear systems intervene between the two filters. However, in the model of Stockman et al. (2014) shown in Figure 7, half-wave rectifiers lie between the early and late filters. Unless the input to the rectifiers lie wholly on the linear (unrectified) branch of the half-wave rectifiers, we need to know what effect half-wave rectification may have on our flickering signals in order to determine the predictions of this model. 

First, consider the effect of half-wave rectification on a sinusoidal input of unit amplitude, frequency f, zero mean, and zero phase; that is, Display Formula\(y\left( t \right) = \sin \left( {2\pi ft} \right)\). We can calculate the Fourier series components of the positive and negative half-wave rectified signals, Display Formula\({Y_ + }\) and Display Formula\({Y_ - }\):  

\begin{equation}{ {{Y_ + }\left[ k \right] = {1 \over T} \mathop \int_0^T {y_ + }\left( t \right){e^{i2\pi kft}}\,dt = {1 \over T}\mathop \int_0^{{T \over 2}} \sin \left( {2\pi ft} \right){e^{i2\pi kft}}\,dt} }\quad {\rm (A1a)}\end{equation}

and  

\begin{equation}{ {{Y_ - }\left[ k \right] = {1 \over T} \mathop \int_0^T {y_ - }\left( t \right){e^{i2\pi kft}}\,dt = {1 \over T}\mathop \int_{T/2}^T \sin \left( {2\pi ft} \right){e^{i2\pi kft}}\,dt} } {\rm ,}\quad {\rm (A1b)}\end{equation}

where Display Formula\(T = {1 \over f}\) is the period of the waveform, and the changes in the limits of the integrals on the right-hand side are due to the fact that y > 0 only between 0 and T/2 and y < 0 only between T/2 and T. Solving Equations A1a and A1b for k = 0, 1, 2, etc., gives the two complete Fourier series representations:  

\begin{equation}{ {{y_ + }\left( t \right) = {1 \over \pi } + {1 \over 2}\sin \left( {2\pi ft} \right) - {2 \over \pi }\mathop \sum \limits_{n = 1}^\infty {{\cos \left( {2n2\pi ft} \right)} \over {4{n^2} - 1}} } } \quad {\rm (A2)}\end{equation}

and  

\begin{equation}{ {{y_ - }\left( t \right) = - {1 \over \pi } + {1 \over 2}\sin \left( {2\pi ft} \right) + {2 \over \pi }\mathop \sum \limits_{n = 1}^\infty {{\cos \left( {2n2\pi ft} \right)} \over {4{n^2} - 1}} } } {\rm .}\quad {\rm (A3)}\end{equation}

 

The amplitudes of the various components are equal in the ON and OFF pathways and, with the exception of the first harmonic, are opposite in sign and therefore would cancel if the two signals were summed. The phases of the first harmonic components are identical in the ON and OFF pathways, and the amplitudes are half the input amplitude so that if the positive and negative signals were summed, they would combine to give back just the original unit amplitude sinusoid. The main effects of half-wave rectification of a sinusoid are (a) to introduce Fourier components that are not in the original signal, and (b) to divide the fundamental equally between the ON and OFF pathways. From Equations A2 and A3 we can see that the fundamental is scaled by a constant factor (1/2) and the introduced components are all much smaller than the fundamental component (e.g., the second harmonic amplitude is 42% that of the fundamental, the third is 15%, the fourth is 8%, and so on) so the small signal linearity assumption is still reasonable. 

However, when we consider more complex waveforms—even just adding the second harmonic at a fixed amplitude ratio, as in Stockman et al. (2017a), in this paper, and in Rider et al. (2016)—the half-wave rectifiers' outputs become much more complicated. While it is possible to compute the Fourier components using the same method as for a single sinusoid (Equations A1a and A1b) it crucially requires knowledge of the zero crossings of the waveform. For a single sinusoid, these are simply 0 and T/2, but for the two-component waveform (with amplitude ratio 1:0.5 and relative phase, φ, of the second harmonic) they are the solutions for t of:  

\begin{equation}{ {\sin \left( {2\pi ft} \right) + {{\sin \left( {4\pi ft + \varphi } \right)} \over 2} = 0 } } {\rm .}\quad {\rm (A4)}\end{equation}

 

While Equation A4 looks relatively simple, it requires finding two real solutions of a fourth order polynomial. Although fourth order polynomials are the highest order that can always be solved analytically, the solution is not necessarily simple, and certainly does not seem to be in this instance. It should also be noted that the solution would not generalize in a straightforward manner, for example adding a third harmonic to the input would lead to a sixth-order polynomial. 

Instead of pursuing this method, we used MATLAB's fast Fourier transform routine to find the solutions numerically. Figure 9 in the main text shows the computed amplitudes of the DC and the first four components of the half-wave rectified two-component signals in the ON (solid curves) and OFF (dashed curves) signals. Note that only components that featured in the original waveform (in this case the first and second harmonics) have different amplitudes in the ON and OFF pathways. At 0° and 180° (the slowly-off and slowly-on waveforms of our experiments) all the harmonics in both the ON and OFF pathways have the same amplitude.