Nonlinear dynamics of cortical responses to color in the human cVEP

Abstract

The main finding of this paper is that the human visual cortex responds in a very nonlinear manner to the color contrast of pure color patterns. We examined human cortical responses to color checkerboard patterns at many color contrasts, measuring the chromatic visual evoked potential (cVEP) with a dense electrode array. Cortical topography of the cVEPs showed that they were localized near the posterior electrode at position Oz, indicating that the primary cortex (V1) was the major source of responses. The choice of fine spatial patterns as stimuli caused the cVEP response to be driven by double-opponent neurons in V1. The cVEP waveform revealed nonlinear color signal processing in the V1 cortex. The cVEP time-to-peak decreased and the waveform's shape was markedly narrower with increasing cone contrast. Comparison of the linear dynamics of retinal and lateral geniculate nucleus responses with the nonlinear dynamics of the cortical cVEP indicated that the nonlinear dynamics originated in the V1 cortex. The nature of the nonlinearity is a kind of automatic gain control that adjusts cortical dynamics to be faster when color contrast is greater.

Introduction

Twenty-first century visual neuroscience has revealed that the primary visual cortex, V1, plays a crucial role in the spatial transformation of color signals (Conway et al., 2010; Hurlbert & Wolf, 2004; Jansen et al., 2014; Johnson, Hawken, & Shapley, 2001), rather than having all important color computations deferred until later stations in the cerebral cortex as had been proposed (Zeki, 1983). It makes functional sense that neural computations for color perception should take place early in cortical processing where the spatial layout of the scene is preserved in V1's precise visuotopic map (Wandell & Winawer, 2011). The reason is that, for veridical color perception, the neural mechanisms of color perception must make computations that take into account the spatial layout of the visual scene as well as the spectral reflectances of different surfaces (Brainard, 2004). But there also are likely to be further color computations at higher cortical levels; it seems likely that V1 and inferotemporal cortex (in macaques) cooperate in color perception (for instance Conway, Moeller, & Tsao, 2007; Harada et al., 2009; Komatsu, 1998). 

We investigated the color responses of neuronal populations in human V1 cortex by measuring the chromatic visual evoked potential (cVEP; Crognale, 2002; Crognale, Duncan, Shoenhard, Peterson, & Berryhill, 2013; Murray, Parry, Carden, & Kulikowski, 1987; Rabin, Switkes, Crognale, Schneck, & Adams, 1994; Souza et al., 2008) over a wide range of color contrast. The cVEP's very fine temporal resolution was critically important for understanding response dynamics in cortical color-responsive regions. Evidence that the cVEP reflects V1 color-evoked activity comes from the signal's topography on the scalp, lack of attentional effects, and experiments on cerebral achromatopsia (see Discussion). One important fact about the cVEP is that it is tuned for spatial frequency. The cVEP amplitude is much smaller for lower spatial frequencies than it is at its peak spatial frequency, between 1–2 cycles per degree (CPD), a consistent result across many studies of cVEP (Murray et al., 1987; Porciatti & Sartucci, 1996; Rabin et al., 1994; Tobimatsu, Tomoda, & Kato, 1995). Taken together with results from single-cell recording in primates (Johnson et al., 2001; Schluppeck & Engel, 2002), the spatial tuning of the cVEP suggests it is mainly driven by V1 double-opponent cells that also are spatially tuned. Unlike double-opponent cells, cortical single-opponent cells respond best to patterns of low spatial frequency or to uniform fields of color (Johnson et al., 2001; Lennie, Krauskopf, & Sclar, 1990; Shapley, Hawken, & Johnson, 2014; Thorell, de Valois, & Albrecht, 1984). Therefore, we designed our experiments to favor the double-opponent cVEP signal by using fine color-checkerboard patterns that were equiluminant with the background gray. 

The cVEP waveform exhibited nonlinear dynamics over the range of color contrast we studied. As reported before (Crognale et al., 1993; Porciatti & Sartucci, 1996; Rabin et al., 1994; Souza et al., 2008), we observed a very large reduction in cVEP latency with increasing color contrast. However, even more remarkable was the change in cVEP waveform shape with increasing color contrast, as analyzed in Results. The nature of the nonlinear change of waveform with color contrast suggests that in the human primary visual cortex there is an automatic gain control for color contrast similar in function to the previously studied (achromatic) contrast gain control for luminance contrast (Carandini, Heeger, & Movshon, 1997; Gomes et al., 2010; Ohzawa, Sclar, & Freeman, 1982). The functional implication is that the cortex adjusts its gain and also its dynamics of response to the color contrast in the visual scene. 

Methods

Participants

All observers gave informed consent to participate in this study. The experiments were conducted in accordance with the principles embodied in the Declaration of Helsinki and were approved by the Hunter College/City University of New York and the New York University Institutional Review Boards. 

Ten observers (four male, six female) aged 19 to 48 (M = 26, SD = 9) participated in this experiment. All participants had normal color vision, assessed with the 18-plate series Pseudo-isochromatic Plates for Testing Color Perception compiled in 1940 by the American Optical Company (Buffalo, NY); Farnsworth dichotomous test for color blindness – Panel D15 (The Psychological Corporation, New York, NY); Lanthony's desaturated 15 hue test (Luneau Ophtalmologie, Chartres, France); and the Farnsworth-Munsell 100-hue test for color vision (Munsell Color Corporation, Baltimore, MD). The participants also had at least 20/20 (or corrected to 20/20) visual acuity, measured using a Snellen chart at 114 cm (the distance to the screen during experiments). 

Visual stimuli

A Sony PVM-1741A OLED monitor (Sony Corporation, New York, NY) was used to present the stimuli. The monitor had a diagonal screen size of 42 cm, resolution of 1920 × 1080 and vertical refresh rate of 60 Hz. The screen was calibrated using a Photo Research PR670 Spectrascan radiometer/photometer (Photo Research, Chatsworth, CA), and this was used to calculate a gamma correction to linearize the screen output to ensure complete control of the intensities on the screen. 

The stimulus size was 20 × 20 cm which at a distance of 114 cm corresponded to 10° × 10° of arc subtended at the eye. The stimuli were equiluminant (using textbook values of equiluminance; Wyszecki & Stiles, 1982) color checkerboards that were rectangular-wave modulated from a gray background to color and back to gray (0.5 s on, 1.5 s off; i.e., modulated at 0.5 Hz with a duty cycle 0.25)—so-called appearance-disappearance modulation (illustrated in Figure 1). The spatiochromatic stimulus was chosen to be a checkerboard so that participants could perceive a definite color in the colored checks for hue and saturation scaling experiments (Gordon, Abramov, & Chan, 1994) done in parallel, the results of which will be reported elsewhere. The checkerboard had 32 × 32 checks. Therefore, each check spanned 0.3125° of arc, for which the dominant spatial frequency has a period of 0.3125 × √2 = 0.4419°, giving a dominant spatial frequency of 1/0.4419 = 2.26 cycles per degree, near the peak of the spatial frequency response reported by Rabin et al. (1994). A given stimulus was presented repeatedly in a block of 30 trials (lasting a total of 60 s). 

Figure 1

 

Representation of the appearance/disappearance stimulus with time. For 0.5 s, a checkerboard of squares was visible. In the checkerboard, squares of equiluminant red of a specific cone contrast alternated with squares that were gray like the background. This “on” pattern consisted of 32 × 32 squares covering 10° × 10° of arc subtended at the eye, and was followed by an interval of 1.5-s duration during which the entire screen was uniform gray (“off”). The transition between “on” and “off” patterns occurred suddenly; thus, the temporal modulation signal was a rectangular wave. All checks and the background were the same luminance.

The background gray color corresponded to a color temperature of 5800°K and had CIE xy coordinates [0.324, 0.328]. The pattern color was one of six saturation-levels of red, with chromatic root mean square (RMS) cone contrast ranging from 0.03 to 0.40. The highest saturation red checks had CIE coordinates [0.522, 0.336]. The chromatic excitation purities and CIE coordinates of each of the color contrasts used are provided in Table 1. For all stimuli the luminance was 31 cd/m². 

Table 1

 

RMS cone contrast and corresponding color excitation purity (CEP) and CIE color coordinates for each stimulus presented.

 

Stimulus presentation was controlled using the Psychophysics Toolbox extensions (Brainard, 1997; Kleiner et al., 2007; Pelli, 1997) for Matlab R2012b (The MathWorks, Inc., Natick, MA), which ran on a Dell Inspiron-3847 computer using the Microsoft Windows 7 operating system. To ensure tighter control of timing, particularly for changing images on the screen from one frame to the next, we followed methods similar to those proposed by Scarfe (n.d.). A trigger signal was sent via serial port to the recording system directly before each stimulus was presented. There was a very small constant delay between the trigger and stimulus signals, which did not cause signal jitter and was taken into account when calculating the pre- and poststimulus periods. 

During each experiment the participants were seated such that their eye level was aligned with the center of the screen and the viewing distance was 114 cm. Stimuli were viewed binocularly. There was one block of stimulus presentations for each cone contrast and the blocks were presented in random order. Each participant was asked to focus on the center of the screen, and to blink as little as possible, particularly when a stimulus was visible on the screen. 

Data acquisition

Data were recorded using a BioSemi ActiveTwo system (BioSemi, Amsterdam, Netherlands); with 64 electrodes we obtained the spatial resolution of a 128-channel system by positioning 63 electrodes on the back half of a 128-channel BioSemi electrode cap set up with the extended 10–20 system (based on the Oostenveld and Praamstra 5% System; Oostenveld & Praamstra, 2001). One electrode was positioned at Fpz and all data were re-referenced to Fpz after data acquisition. When aligning the electrode cap, we ensured that the electrode for Oz was correctly positioned at 10% of the inion-nasion distance along the midline of the scalp (see Figure 2 for a diagram showing the electrode placement). The trigger and electroencephalogram (EEG) signals were sampled at a frequency of 2048 Hz, with an open passband from 0–400 Hz. 

Figure 2

 

Expanded 10–20 electrode placements used on the BioSemi 128-channel head cap, based on the Oostenveld and Praamstra 5% system (Oostenveld & Praamstra, 2001). Image based on figure originally provided by Cortech Solutions, Inc. (adapted with permission).

Data analysis

Using functions from the FieldTrip toolbox for EEG/MEG-analysis (Oostenveld, Fries, Maris, & Schoffelen, 2011; http://www.ru.nl/neuroimaging/fieldtrip), we imported the response data for each stimulus and separated them into trials containing a prestimulus period of 100 ms and post-stimulus-onset period of 500 ms. 

The EEG data in each trial were re-referenced with respect to electrode Fpz, and then were baseline corrected with respect to the average voltage across each entire trial. The data were inspected visually (all channels simultaneously on a trial-by-trial basis) to remove blinks and artifacts due to movement or extreme electronic noise transients (greater than 150 μV). At this point, the trial data were baseline-corrected with respect to the prestimulus period before a Discrete Fourier Transform of the cVEP waveform was calculated using a period of 0.5 s, covering the duration of the stimulus. This resulted in a Fourier fundamental frequency of 2 Hz. Note that we chose not to conduct Fourier analysis over the whole stimulus on/off period because our purpose was strictly to focus on the waveform shape over the time when the stimulus was visible. This was partly because the participants tended to blink more after the stimulus disappeared but mainly because the nonlinear dynamics in which we are interested were in this range. The first 100 Fourier harmonics were used to construct inverse FT waveforms. Before the reconstruction step, the data were filtered for 60 Hz noise and its harmonics by setting the amplitudes of the corresponding harmonics to zero. No additional filtering, including any high-pass band filtering, took place. 

Results

cVEPs: Waveforms and topography

The cVEP waveforms were predominantly negative deflections, consistent with earlier reports (Crognale, 2002; Crognale et al., 2013; Murray et al., 1987; Rabin et al., 1994; Souza et al., 2008). At high cone contrast, the cVEP had a large negative peak occurring around 120–150 ms after stimulus onset (Figure 3) replicating what has been reported before about the timing of the cVEP peak and its polarity (Murray et al., 1987; Porciatti & Sartucci, 1996; Rabin et al., 1994; Tobimatsu et al., 1995). Figure 3 depicts the cVEPs of one participant over a range of cone contrasts to illustrate cVEP dependence on cone contrast and the magnitudes of cVEPS. 

Figure 3

 

The cVEP waveform of one participant for a selection of cone contrasts, covering the time period from pattern onset to 1 s after pattern onset. Note that the pattern was visible only from 0–0.5 s. For this particular participant, there was very little off-response and the waveform after the pattern disappeared was mostly reflective of stimulus-entrained alpha. The peak-baseline negativity at maximum cone contrast for the participants ranged from 12 to 53 μV (M = 26.6 μV, SEM = 3.6 μV).

The cVEP was recorded with a dense multi-electrode array (see Methods) from which we could estimate the regions of the cerebral cortex activated by the spatiochromatic stimulus. Figure 4 from one participant shows the electrode topography of the cVEP as a function of time at two values of cone contrast: 0.09 (low cone contrast) and 0.4 (moderately high cone contrast). The cartoon provided by the manufacturer, reproduced in Figure 2, indicates the electrode locations on the head and their conventional designations (e.g., Oz for the most posterior midline electrode). As can be seen in Figure 4, cVEP activation at high contrast peaked over Oz at the peak time of 115 ms. At later times 155 and 175 ms, the cVEP extended laterally to the neighboring electrodes. At high contrast, at the peak time of the cVEP there was no significant activation either of lateral occipital cortex or parietal cortex by the appearance/disappearance of pure color checkerboards. The data at lower contrast were different. At an early time (135–155 ms) prepeak the active region was confined to Oz. But then at 175 ms the activity spread, encompassing both Oz and more lateral posterior electrodes. It is quite remarkable how the topographic pattern changed dramatically with cone contrast. This is one definite indicator of how the cortical response to cone contrast is nonlinear, not simply scaling in amplitude with cone contrast. Cortical topographic data like those in Figure 4 were found in all participants whose results are used in this paper. This supports the hypothesis that most of the cortical activity we studied was generated primarily within primary visual cortex, V1. 

Figure 4

 

Series of topographic maps of the responses over the head, as viewed from above, for one participant, at several different times after stimulus onset, for two different cone contrasts (0.4 on the left and 0.09 on the right). In each topographic plot, the color contours join areas with the same response magnitude. While the responses were only measured at specific electrode positions, the values between electrodes were calculated by interpolation. The range of response voltages (in μV) is given by the scale to the right. The actual electrode locations are indicated in Figure 2.

cVEPs: Nonlinear dynamics with color contrast

Unlike the color-evoked responses measured in subcortical parvocellular- and koniocellular-pathway neurons (discussed below), the cVEP generated in V1 exhibited nonlinear dynamics in responses to different cone contrasts (Figures 3 and 5). Figure 5 depicts cVEP at higher temporal resolution data than those in Figure 3, with data from two other participants. 

Figure 5

 

The cVEP waveforms of two participants for a range of cone contrasts covering the time period from pattern onset to 0.5 s after pattern onset.

To emphasize the waveform's shape instead of its amplitude, we drew Figure 6 in which all the cVEP waveforms are normalized at (negative) peak to the same value (−1). Then the waveform differences at different cone contrasts are quite vivid. As seen in Figure 6, the cVEP for lower cone contrast was (a) slower to rise (Crognale et al., 1993; Porciatti & Sartucci, 1996; Rabin et al., 1994; Souza et al., 2008) and also (b) more prolonged than at higher contrast. These two effects of nonlinear dynamics can be observed qualitatively in the waveforms in Figure 6 as (a) the shift to the left of the waveforms of responses to higher contrast, and (b) the crossing of lower contrast responses by higher contrast responses (i.e., earlier decline to baseline of responses to higher contrast). For instance, note that the responses to 0.18 cone contrast drawn in green, cross the response to 0.09 cone contrast drawn in blue around 170 ms in the upper panel and around 190 ms in the lower panel. This waveform crossing is a qualitative indication of the longer persistence of the response at lower cone contrast. We analyzed these nonlinear dynamics further in a number of ways. 

Figure 6

 

The normalized cVEP responses of two participants for a range of cone contrasts covering the time period from pattern onset to 0.5 s after pattern onset. The original cVEP waveforms were normalized at (negative) peak to the same value (−1).

First, we replicated the shorter latency of cVEP at higher contrast (Crognale et al., 1993; Crognale, Switkes, & Adams, 1997; Porciatti & Sartucci, 1996; Rabin et al., 1994; Souza et al., 2008) by Fourier analyzing the cVEP waveform as described in the Methods section. The largest harmonic amplitude was usually at 4 Hz, so we analyzed the phase shift of the 4-Hz component in the Fourier transform of the cVEP. Results for one typical participant and an average across all participants are given in Figure 7. The individual's data and the pooled data are completely consistent in showing a very large phase advance, greater than or equal to 100° of phase, from low to high cone contrast, replicating the many previous studies that found decreasing latency with increasing cone contrast. 

Figure 7

 

The phase of the 4 Hz component in the Fourier transform of the cVEP plotted as a function of RMS cone contrast for a typical participant (top) and averaged across all participants (bottom). In the graph for a single participant (top), error bars were calculated from the T_circ² statistic of Victor and Mast (1991). For the grand average (bottom) the error bars represent ±1 SEM.

To reinforce that there were qualitative differences in cVEP waveform, we did a more quantitative analysis by analyzing the Fourier representation of the waveforms (see Methods). We analyzed the contrast dependence of the power spectrum of the cVEP only up to the first five harmonics of the fundamental frequency based on an analysis of the cumulative power spectrum (e.g., Jospin et al., 2007), the sum of the power up to and including a specified harmonic, as shown in Figure 8. Figure 8 shows group averaged data over the range of cone contrasts we studied. Most (>80%) of the power of the cVEP was contained in Harmonics 1–5 under all conditions. 

Figure 8

 

Normalized cumulative Fourier power spectrum for a range of cone contrasts for a typical participant.

The analysis of the Fourier amplitude spectra of the cVEP waveforms demonstrates the profound change in response dynamics with cone contrast. Fourier amplitude spectra are drawn in Figure 9 for one representative participant's data. The Fourier spectra were normalized to 1 at peak amplitude. Figure 9 shows that the spectra change with cone contrast; there is much more power in the higher harmonics at high contrast than at low. This change of spectrum with contrast is a nonlinear effect. In a linear system, the spectrum at higher contrast would have the same shape as at low. 

Figure 9

 

Normalized Fourier amplitude spectrum for two different RMS cone contrasts (0.09 on the left, 0.4 on the right) for a participant.

To quantify the change of the amplitude spectra with cone contrast we devised a spectral shape index = H1/([H2 + H3]/2), that is the ratio of the amplitude of the 2-Hz fundamental Fourier component divided by the average of the amplitudes of the second and third harmonics (i.e., 4 and 6 Hz). Plots of the spectral shape index for one participant and for the population are shown in Figure 10. The spectral shape index declines as cone contrast increases because, as shown in Figure 9, the spectral components of H2 and H3 grow more with contrast than does H1. This is further evidence for the nonlinear dynamics of V1 color-evoked signals in the cVEP. 

Figure 10

 

Spectral shape index, defined as the ratio of the first harmonic amplitude to the mean of the second and third harmonic amplitudes, plotted as a function of RMS cone contrast for one participant (top) and averaged over all participants (bottom). In the graph for a single participant (top), error bars were calculated from the T_circ² statistic of Victor and Mast (1991). For the grand average (bottom) the error bars represent ±1 SEM.

Fourier analysis also was used to study the dependence of cVEP response power on cone contrast (Figure 11). Figure 11 depicts response power versus cone contrast averaged across all six participants in this study. The upper panel is for summed power across the first five harmonics (2–10 Hz) of the stimulus period. As we reasoned above, the cumulative power spectra (Figure 8) indicated most response power was contained in these five harmonics so the summed power across them should give a good estimate of total response power (Parseval's theorem). What is notable about the summed power (2–10 Hz) is that it rises steeply between 0.03 to 0.09 cone contrast and then levels off so that response power at 0.09 contrast is already 80% as large as the response to the highest cone contrast used, 0.4. In other words, response power grows sublinear with cone contrast. However, we noticed that the power in the third harmonic (i.e., 6 Hz) grew roughly proportionally with cone contrast. This is further evidence of the dynamic nonlinearity of the cVEP, that different Fourier components have different dependences on cone contrast. 

Figure 11

 

Normalized Fourier power averaged over all participants and plotted as a function of RMS cone contrast. In the top graph, the Fourier power for each participant was summed over the first five Fourier harmonics (2–10 Hz) before normalization and grand averaging. The lower graph shows normalized Fourier power calculated for only the third harmonic (corresponding to 6 Hz) before normalization and grand averaging. Note that the error bars represent ±1 SEM.

Discussion

Population of double-opponent cells as the source of cVEP signals

It is important to discuss the neural origins of the cVEP signal. cVEPs are evoked by equiluminant color modulation; therefore, they must be driven only from subpopulations of cortical neurons that are responsive to color (Schluppeck & Engel, 2002). We found (Figure 4) that for spatial patterns such as checkerboard patterns, the cVEP was localized mostly over posterior occipital cortex, consistent with the hypothesis that the cVEP is generated in V1 cortex (Crognale et al., 2013; Xing et al., 2015). More evidence that the cVEP reflects color-evoked activity in the primary visual cortex is as follows. The cVEP does not vary with attention, a result that strongly suggests it is evoked early in cortical visual processing (Highsmith & Crognale, 2010). Furthermore, normal cVEPs have been recorded in cases of cerebral achromatopsia where color appearance was lost and lesions were observed in ventromedial extrastriate cortex, but V1 responses to color were unaffected by the lesion (Crognale et al., 2013; Victor, Maiese, Shapley, Sidtis, & Gazzaniga, 1989). The combined evidence from source localization, lack of attentional effects, and cerebral achromatopsia indicates that the cVEP is an index of early cortical responses to color (i.e., an index of V1 activity), and the topography supports this. However, the low contrast cVEP indicates more diffuse responses, with activity spreading to lateral posterior cortical areas (Figure 4) at low cone contrast. 

As mentioned in the Introduction, cortical color computations are based on the combined activity of two kinds of cortical cone-opponent neurons, single- and double-opponent cells, and also on the cone-nonopponent neurons that respond strongly to achromatic patterns (reviewed in Shapley et al., 2014). Single-opponent cells integrate and double-opponent cells differentiate color signals across visual space. While single-opponent cells respond to large areas of color, double-opponent cells respond to color patterns (Johnson et al., 2001) and color boundaries (Friedman, Zhou, & Heydt, 2003). Double-opponent cells comprise approximately 80% of all color responsive cells in the output layers 2/3 of macaque V1 cortex (Friedman et al., 2003; Johnson et al., 2001) and that may be why they might contribute most to the cVEP signal. 

Single- and double-opponent neurons have different spatial frequency responses and this fact can be used to test their contributions to the cVEP. As shown by Schluppeck and Engel (2002), single-opponent neurons not only respond to lower spatial frequencies but also their responses cut off at lower spatial frequencies than those of double-opponent cells. The spatial frequency tuning of double-opponent (and also non-opponent) cells are spatially band pass, like the cVEP. There is a spatial frequency range (1–4 cpd) where double-opponent cells respond and single-opponent neurons respond weakly or not at all. In this range, color stimuli are mainly stimulating double-opponent neurons. The red-gray checkerboard we used had a space-averaged chromaticity. The spatially averaged signal was a red of half the cone-contrast of the checkerboard. Such a stimulus should activate single-opponent cells in V1 (Johnson et al., 2001; Schluppeck & Engel, 2002) but it is known from previous work on the cVEP (Murray et al., 1987; Porciatti & Sartucci, 1996; Rabin et al., 1994; Tobimatsu et al., 1995) that the cVEP amplitude in response to such a low spatial frequency stimulus is very small or absent. Therefore, based on this earlier work, we infer that single-opponent signals were not contributing to the waveforms analyzed in this article and that the nonlinear dynamics observed were affecting signals coming from double-opponent cells. 

Locus of nonlinear dynamics in V1

Next we must discuss where in the brain the remarkably large contrast-dependent nonlinearity revealed in Figures 4, 5, 6, 7, 9, and 10 is introduced into the dynamics of color processing. 

To appreciate how remarkable are the findings of cortical nonlinearities in V1 color responses, one must compare the cortical data with the linearity of the responses in the parvocellular pathway that provides input to V1 (Lund, 1988). Consider an example from the neurophysiological literature about responses in the parvocellular pathway (Figure 12; data in Benardete & Kaplan, 1999). Plotted in Figure 12 are the spike rate responses of a single macaque retinal ganglion cell of the P-ON L-M+ type. The stimulus was an L-cone isolating grating pattern modulated by a temporal m sequence, and the response was obtained by cross-correlation of the spike train with the m sequence (Reid, Victor, & Shapley, 1997). For this cell, an increase in L-cone stimulation produced a brief reduction in firing rate. The (m sequence) stimulus was effectively very brief with duration ∼15 ms. Therefore, the responses in Figure 12 can be considered to be temporal impulse responses of the retinal network that excited the retinal ganglion cell under study. The stimulus had three different values of cone contrast: 0.0625, 0.125, and 0.25. The responses were graded and approximately proportional to cone contrast. Most important the response waveform was the same shape at the three cone contrasts and simply scaled in amplitude as contrast rose. As the authors of the study noted (Benardete & Kaplan, 1999), this simple scaling behavior is a sign of linearity. This example is shown to establish the well-accepted point that responses of cells in subcortical parvocellular pathway are (approximately) linear with contrast modulation (Benardete & Kaplan, 1999; Kaplan & Shapley, 1986; Lee, Pokorny, Smith, & Kremers, 1994). Furthermore, there is direct evidence from pattern-reversal electroretinogram recordings that P retinal ganglion cells are as linear in humans as in monkeys (Morrone, Fiorentini, Bisti, Porciatti, & Burr, 1994; Morrone, Porciatti, Fiorentini, & Burr, 1994). 

Figure 12

 

First-order responses of a P ON L-M+ retinal ganglion cell in the macaque monkey retina to chromatic and achromatic gratings at several contrasts. The chromatic gratings used for the responses depicted were L-cone-isolating at 0·145 cpd spatial frequency. The contrasts were 0·0625 (thinnest line), 0·125 and 0·25 (thickest line). As cone contrast increases, the amplitude of the response scales approximately linearly, with no change in the temporal characteristics of the response. Redrawn from Benardete, E. A., & Kaplan, E. (1999). Dynamics of primate P retinal ganglion cells: responses to chromatic and achromatic stimuli. The Journal of Physiology, 519(3), 775–790. Copyright 1999 by John Wiley & Sons, Inc. Adapted with permission.

With the knowledge that in the retina the processing of color-contrast-evoked signals is linear, as in Figure 12, one realizes that the study of the processing of color signals offers a very significant advantage over the study of the nonlinear dynamics of achromatic processing. The retinal and lateral geniculate nucleus sources of achromatic responses, magnocellular pathway neurons, are sped up at higher contrast (Benardete, Kaplan, & Knight, 1992). Speeding up of achromatic responses in the cortex (Carandini et al., 1997) could be at least in part a retinal effect (cf. Freeman, Durand, Kiper, & Carandini, 2002). Since we know that dynamics of response of neurons in the parvocellular subcortical pathway do not vary with contrast, behaving like a linear system, nonlinear effects of color contrast on dynamics must be entirely cortical not retinal if they are driven entirely by parvocellular inputs. 

We measured responses from stimuli in the red direction corresponding to the screen's red phosphor and found nonlinear dynamics. Our phase data, and Rabin et al.'s (1994) phase versus contrast data indicate that L-M and S-cone pathways have the same kind of nonlinearity of phase that we measured in the red direction. Therefore, we need to consider the possibility that nonlinearities in the S-cone subcortical pathway may have given rise to the cVEP nonlinearities that we observed. Tailby, Solomon, and Lennie's (2008) measurements in the macaque LGN revealed no phase advance with contrast for pure S-cone input, and the same behavior was observed in the S-potentials of LGN neurons, indicating linear responses with contrast by koniocellular retinal ganglion cells. Therefore, as with the parvocellular subcortical pathway, the dynamics of neurons in the koniocellular subcortical pathway do not vary with contrast, and the nonlinear dynamics observed must still be cortical in nature. 

Because it is known that there are nonlinear dynamical effects in the magnocellular pathway, it is worth considering whether or not the cVEP signals we measured were influenced by magnocellular inputs to V1 cortex. For fine pattern stimuli like the ones used in this study, magnocellular cells produce negligibly small responses to equiluminant color patterns (Derrington, Krauskopf, & Lennie, 1984; Lee, Martin, & Valberg, 1989; Shapley & Hawken, 1999). While Lee et al. (1989) reported small nonlinear responses to equiluminant color patterns in magnocellular neurons, they observed these responses to color patterns in magnocellular neurons only for stimulus spots of large area, and indeed ascribed them to nonlinear interactions in the magnocellular receptive field surrounds that could only be driven by large stimuli or very low spatial-frequency patterns. We judge that the nonlinear surround responses of magnocellular neurons would not be evoked by the fine checkerboard patterns that we used as stimuli and thus that the cVEP signals we observed were cortical responses driven by parvocellular and/or koniocellular signals. A further piece of evidence against magnocellular involvement in the cVEP is that a laminar analysis of the color sensitivity in V1 found very weak responses to color patterns in the magno-driven input layer 4C-alpha (Johnson et al., 2001). All of the data cited about magnocellular responses to chromatic stimuli were obtained by microelectrode recordings in the macaque visual pathway. Our reasoning in this discussion is based on the assumption that magnocellular neurons in humans are very similar to those in macaque in terms of their insensitivity to fine color patterns. Support for this assumption comes from many psychophysical experiments on the loss of magnocellular function at equiluminance (reviewed in Livingstone & Hubel, 1988). It would be useful to have more direct support for that assumption. 

Another possible way magnocellular signals could corrupt the cVEP signals we measured is by miscalibration of equiluminance in our stimuli. The values of equiluminance used to calibrate our stimuli were based on average normative data while individual equiluminant points differ from the average values. Therefore, what we called equiluminant stimuli for some participants might have contained small amounts of luminance contrast. However, we calculated the amount of luminance contrast possibly introduced by this possible miscalibration and it was small. Furthermore, we have repeated the experiments reported here in other experiments in which we measured equiluminance for each observer individually and obtained the same results on nonlinear dynamics with cone contrast. Therefore, we judge that miscalibration of the equiluminant point had no effect on our inferences about nonlinear dynamics of the cVEP signal. 

One might consider the possibility that the nonlinear responses were due to luminance pathway intrusion from chromatic aberration. While there is degradation of color grating patterns by chromatic aberration for patterns with spatial frequencies >4 cpd (Flitcroft, 1989), our dominant frequency was chosen to be well below this limit. The higher spatial frequencies that would be produced by the sharp edges of the checkerboards should not generate responses in the chromatic system because of the high-frequency cut-off of the chromatic system (Rabin et al., 1994). If there had been luminance artifacts we would have expected to see an accompanying reduction in saturation of the checkerboards. However, from pilot tests with 64 × 64, 32 × 32 and 16 × 16 checkerboards we concluded that the 32 × 32 checkerboards are in the range in which these higher harmonics are not producing significant achromatic artifacts. 

Lastly, Crognale et al. (1997) proposed an explanation for nonlinear effects with contrast that they observed in cVEPs elicited by ramped sinusoidal modulations along the S and LM axes. They suggested the nonlinearities they observed might be related to the contrast dependence of the interaction between the cVEP and intrinsic cortical rhythms. It is difficult to ascertain whether their observations and interpretation are related to ours because their temporal modulation stimulus was so different. Nevertheless, it is possible that the nonlinear dynamical changes we observed may have been influenced by internal cortical dynamics as suggested by Crognale et al. (1997). However, we do not believe that the cortex's alpha rhythm (8–12 Hz) was involved (as Crognale et al., 1997, conjectured with respect to their data). This is because, while all our participants demonstrated nonlinear speeding up of responses with increasing contrast, only a few exhibited alpha entrainment (though it is clearly apparent for the participant's data in Figure 3). Also, the frequency components that were modified by contrast (Figure 9) were mostly lower than alpha. 

Nature of the cortical nonlinearity

The nonlinear dynamics we have described in the cVEP resemble cortical gain control phenomena that have been reported previously for cortical responses to achromatic signals (Albrecht, Geisler, Frazor, & Crane, 2002; Carandini et al., 1997; Ohzawa et al., 1982). Thus it is well known that there are large phase advances and speeding up of time-to-peak of responses of single cortical cells when the contrast of achromatic stimuli is increased (Albrecht et al., 2002; Carandini & Heeger, 2011; Carandini et al., 1997). Such nonlinear dynamics have been ascribed to the action of a cortical contrast gain control, also called normalization. There are different possible models of cortical normalization. Some authors have maintained that normalization is not even a cortical phenomenon but occurs precortically (Freeman et al., 2002), though we remind the reader that such earlier proposals were based on results in the cat cortex and visual pathway where there is no contrast-linear counterpart to the parvocellular input to macaque (and human) V1. Previous authors who discussed cortical mechanisms of normalization emphasized inhibitory mechanisms such as local circuit recurrent inhibition (Carandini & Heeger, 2011; Heeger, 1992). However, it is also possible that modulation of recurrent excitation could be a cortical mechanism for normalization (Carandini & Heeger, 2011; Sato, Haider, Häusser, & Carandini, 2016). As reported by Nauhaus, Busse, Carandini, and Ringach (2009), increasing achromatic contrast reduces the strength of recurrent excitation in cortical circuits by an unknown mechanism, and this could be the kind of mechanism that was observed also by Sato et al. (2016). In short, either recurrent excitation or recurrent inhibition or both in cortical circuits could act to affect response magnitude and also dynamics of visual cortical responses. These mechanisms must also be candidates for explaining the nonlinear dynamics in the cVEP. The double-opponent population could be affected by the same cortical interactions as their neighbors in the cortical circuit, the color-blind nonopponent cells. This would provide a parsimonious explanation for why color contrast produces such large nonlinear dynamic effects; it is color-contrast normalization. 

The functional role of the color-dependent dynamics we have observed could be to adapt the cortex to the prevalent visual scene. When there are only weak color contrasts, the cortical network integrates color signals over a longer time. But when color signals are strong, the cortex relaxes back to baseline more quickly to enhance signal resolution and signal differentiation. This is a strategy similar to how the cortex handles achromatic signals. Normalization has been called a canonical computation (Carandini & Heeger, 2011) and our finding of phenomena like normalization in color supports that idea. 

Acknowledgments

This research was supported by grant PAC-1555773 from the National Science Foundation. We thank Afsana Amir, Chloe Brittenham, Asmaa Butt, Norine Chan, Aneliya Hanineva, Syed Ali Hassan, Patricia Pehme, Carim-Sanni Ridwan, Yoomin Song, Crismeldy Veloz, and Shaneka Whittick for their help with experiments. We especially thank Peter Schuette who coded the software to produce the stimuli used in this work. 

Commercial relationships: none. 

Corresponding author: Valerie Nunez. 

Email: . 

Address: Center for Neural Science, New York University, New York, NY, USA.