Electrophysiological evidence for higher-level chromatic mechanisms in humans

Abstract

Color vision in humans starts with three types of cones (short [S], medium [M], and long [L] wavelengths) in the retina and three retinal and subcortical cardinal mechanisms, which linearly combine cone signals into the luminance channel (L + M), the red-green channel (L – M), and the yellow-blue channel (S-(L + M)). Chromatic mechanisms at the cortical level, however, are less well characterized. The present study investigated such higher-order chromatic mechanisms by recording electroencephalograms (EEGs) on human observers in a noise masking paradigm. Observers viewed colored stimuli that consisted of a target embedded in noise. Color directions of the target and noise varied independently and systematically in an isoluminant plane of color space. The target was flickering on-off at 3 Hz, eliciting steady-state visual evoked potential (SSVEP) responses. As a result, the masking strength could be estimated from the SSVEP amplitude in the presence of 6 Hz noise. Masking was strongest (i.e. target eliciting smallest SSVEPs) when the target and noise were along the same color direction, and was weakest (i.e. target eliciting highest SSVEPs) when the target and noise were along orthogonal directions. This pattern of results was observed both when the target color varied along the cardinal and intermediate directions, which is evidence for higher-order chromatic mechanisms tuned to intermediate axes. The SSVEP result can be well predicted by a model with multiple broadly tuned chromatic mechanisms. In contrast, a model with only cardinal mechanisms failed to account for the data. These results provide strong electrophysiological evidence for multiple chromatic mechanisms in the early visual cortex of humans.

Introduction

Human color vision starts with three types of cones on the retina, each sensitive to short (S), medium (M), and long (L) wavelengths. In the retinal ganglion cells, signals from cones are combined into the achromatic luminance channel (L + M), the red-green channel (L − M), and the yellow-purple channel (S-(L + M)). In the lateral geniculate nucleus (LGN), the three color-opponent channels are for the most part anatomically separated into the magnocellular layer, the parvocellular layer, and the koniocellular layer, respectively (for review, see Conway, Eskew, Martin, & Stockman, 2018; Lee, Martin, & Grünert, 2010; Solomon & Lennie, 2007; Thoreson & Dacey, 2019). These physiological findings fit well with psychophysical results (Krauskopf, Williams, & Heeley, 1982), where three cardinal directions of color space (i.e. L + M, L − M, and S-(L + M)), were defined using a habituation paradigm. 

Although the subcortical stages of color processing from the retina to the LGN have been characterized fairly well, less is known about subsequent processing at the cortical level. Krauskopf, Williams, Mandler, and Brown (1986) proposed that there are higher-order chromatic mechanisms in addition to the three cardinal mechanisms. Since then, a large number of psychophysical studies have found evidence for the existence of higher-order mechanisms, using a wide range of experimental paradigms, such as adaptation (e.g. Webster & Mollon, 1991), noise masking (Gegenfurtner & Kiper, 1992), chromatic discrimination (Krauskopf & Gegenfurtner, 1992), or classification images (Bouet & Knoblauch, 2004; Hansen & Gegenfurtner, 2005). Inconsistent results were also reported in the case of noise masking experiments, as some studies failed to find evidence for higher-order mechanisms (Eskew, Newton, & Giulianini, 2001; Giulianini & Eskew, 1998; Sankeralli & Mullen, 1997; for review, see Eskew, 2009). 

In the noise masking paradigm, observers are required to detect a chromatic target signal embedded in chromatic noises. The color of the target and noise are systematically and independently varied in color space. Detection performances vary depending on color directions of the target and noise. Gegenfurtner and Kiper (1992) observed selective masking at cardinal axes. That is, noise along one cardinal direction masked detection of the target if the target was along the same cardinal direction as the noise, but did not affect the detection if the target was along another cardinal direction. Such selective masking effects at cardinal axes provided additional support for the cardinal mechanisms of Krauskopf et al. (1982). However, the same pattern of selective masking was observed at intermediate axes, which provided evidence for the existence of additional mechanisms on top of the cardinal ones. Most of the follow-up studies found similar results (Cass, Clifford, Alais, & Spehar, 2009; D'Zmura & Knoblauch, 1998; Goda & Fujii, 2001; Hansen & Gegenfurtner, 2006; Hansen & Gegenfurtner, 2013; Li & Lennie, 1997; Lindsey & Brown, 2004), although some failed in finding selective masking (Eskew et al., 2001; Giulianini & Eskew, 1998; Sankeralli & Mullen, 1997; Stromeyer, Thabet, Chaparro, & Kronauer, 1999). Hansen and Gegenfurtner (2013) showed that the inconsistency was due to a restricted choice of color stimuli in cone contrast space in these studies (Eskew et al., 2001; Giulianini & Eskew, 1998; Sankeralli & Mullen, 1997). After adjusting the sampling of colors, Hansen and Gegenfurtner (2013) did observe evidence for higher-order color mechanisms. 

Although the majority of previous studies support the existence of higher-order chromatic mechanisms, much less is known about the exact number and tuning characteristics of these mechanisms. Only a handful of studies have estimated the number of mechanisms, and the resulting numbers ranged from about four (Zaidi & Halevy, 1993), five to seven (Goda & Fujii, 2001; Shepard, Swanson, McCarthy, & Eskew, 2016), and to 16 (Hansen & Gegenfurtner, 2006). The tuning characteristics of higher-order mechanisms are of interest because they are related to the (non-)linearity of the computations. A broad cosine tuning would indicate that the mechanism linearly combines cone-opponent signals, whereas a narrow tuning is typically interpreted as evidence for nonlinear computations. Some studies observed narrow tuning (Gegenfurtner & Kiper, 1992; Goda & Fujii, 2001), whereas others found broad tuning (D'Zmura & Knoblauch, 1998; Giulianini & Eskew, 1998; Lindsey & Brown, 2004; Sankeralli & Mullen, 1997). It has been argued that the nonlinear narrow tuning is mediated by off-axis looking (D'Zmura & Knoblauch, 1998; Hansen & Gegenfurtner, 2006). Off-axis looking describes the behavior that observers dynamically adjust their detection decisions to use the mechanisms that are less affected by the noise, leading to narrow tuning curves. D'Zmura and Knoblauch (1998) designed two-sided noise stimuli to minimize off-axis looking, and they did observe broad tuning with such stimuli (also in Hansen & Gegenfurtner, 2006). 

Neurophysiological studies have found higher-order color mechanisms in the brain as early as the primary visual cortex (V1). In single-unit electrophysiology, it is well established that neurons in V1 are tuned to noncardinal, intermediate color directions (Conway, 2001; De Valois, Cottaris, Elfar, Mahon, & Wilson, 2000; Lennie, Krauskopf, & Sclar, 1990; Li, Garg, Zhang, Rashid, & Callaway, 2020; Liu, Li, Zhang, Lu, Gong, Yin, & Wang, 2020; Wachtler, Sejnowski, & Albright, 2003; see Horwitz, 2020 for a recent review). In later areas, such as V2 (Kiper, Fenstemaker, & Gegenfurtner, 1997; see Gegenfurtner, 2003; Gegenfurtner & Kiper, 2003 for reviews) or IT (Conway, 2014; Conway, 2018), there are also neurons tuned narrowly to both cardinal and intermediate directions. Human functional magnetic resonance imaging (fMRI) studies have found that brain activities from as early as V1 show representations of intermediate colors (Brouwer & Heeger, 2009; Goddard, Mannion, McDonald, Solomon, & Clifford, 2010; Kuriki, Nakamura, Sun, Ueno, Matsumiya, Tanaka, & Cheng, 2011; Kuriki, Sun, Ueno, Tanaka, & Cheng, 2015; Parkes, Marsman, Oxley, Goulermas, & Wuerger, 2009). Despite poor spatial resolution of electroencephalograms (EEGs), previous EEG studies using chromatic stimuli did find evidence for cardinal as well as higher-order color mechanisms (Duncan, Roth, Mizokami, McDermott, & Crognale, 2012; Kaneko, Kuriki, & Andersen, 2020; Rabin, Switkes, Crognale, Schneck, & Adams, 1994). These results suggest that higher-order color representations are formed in the early visual cortex including V1. 

The present study aims to address some of the issues regarding higher-order chromatic mechanisms by measuring neural activities using EEGs in human participants. Here, we investigated noise masking using steady-state visual evoked potentials (SSVEPs), which are oscillatory brain responses to periodic visual stimulation (for a recent review, see Norcia, Appelbaum, Ales, Cottereau, & Rossion, 2015), originating mainly from the primary visual cortex (Di Russo, Pitzalis, Aprile, Spitoni, Patria, Stella, & Hillyard., 2007; Müller, Teder, & Hillyard, 1997; Wittevrongel, Khachatryan, Fahimi Hnazaee, Carrette, De Taeye, Meurs, & Van Hulle, 2018). Instead of measuring detection thresholds with psychophysical procedures in the noise masking paradigm, we recorded SSVEPs in response to a chromatic target embedded in chromatic noises. The target was flickering on-off at 3 Hz, eliciting SSVEP responses at 3 Hz. The change in SSVEP amplitude in the presence of various noise maskers can serve as a neural indicator for the magnitude of masking effect. Our results revealed a selective masking effect at both the cardinal axes and intermediate axes, providing neurophysiological support for multiple chromatic mechanisms in the early visual cortex of humans. 

Method

Participants

Eight observers (7 women and 1 man, age mean = 23.0 years, range = 19 to 24 years) participated in experiment 1. A new sample of eight observers (4 women and 4 men, age mean = 22.6 years, range = 19 to 24 years) participated in experiment 2. They all had normal or corrected-to-normal visual acuity, and normal color vision as tested by Ishihara plates (Ishihara, 2004). They signed written informed consent forms in agreement with the Declaration of Helsinki. The study was approved by the local ethics committee (No. 102772019RT009) of the Shanghai University of Sport. 

Stimulus

Stimuli were displayed using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) in MATLAB (MathWorks, Natick, MA, USA), on a 60-Hz LCD monitor (Eizo Corporation, Hakusan, Japan). With a spatial resolution of 1920 × 1200 pixels, the screen extended 47 degrees horizontally and 29 degrees vertically at a viewing distance of 60 cm. 

All stimuli were presented against a gray background: CIE xyY = (0.26, 0.28, and 56.5 cd/m²). The primaries of our monitor had xyY coordinates of red (0.68 0.31 22.3), green (0.21 0.72 78.7), and blue (0.15 0.051 11.5). The intensity resolution of the graphics card was eight bits. Colors were chosen from the isoluminant plane in the Derrington Krauskopf Lennie (DKL) color space (Derrington, Krauskopf, & Lennie, 1984; Krauskopf et al., 1982). A detailed description of the DKL color space can be found in the appendix of Hansen and Gegenfurtner (2013). The DKL space is device-dependent. The two cardinal axes (L − M axis and S axis) were scaled relative to the gamut of the display, so that the space uses the maximally possible modulation achievable with the display. In the current study, the maximum modulation is normalized to 100% contrast on each axis. To quantify the color space independently of the device, we can convert the contrast of each axis into cone-contrast space (Brainard, 1996; Noorlander, Heuts, & Koenderink, 1981; Smith & Pokorny, 1975). The end points we used for the L – M and S axis in xy coordinates and in cone-contrast (relative to the gray background) are provided in the Table 1. 

Table 1.

 

End points of the L – M and S axis of our DKL color space.

 

The central 9.7 degrees × 9.7 degrees area was used for stimulus presentations. The stimulus (Figure 1) was an array consisting of 16 × 16 small colored squares, each of them extending 0.61 degrees × 0.61 degrees in visual angle. The stimulus was the sum of a 6 × 6 color pattern in the central 6 × 6 region (the target) and a 16 × 16 color pattern (the noise). 

Figure 1.

 

Example color stimuli used in the present study. All pixels had equal luminance. Target colors (40% contrast) along L – M axis were added to the central 6 × 6 squares of the noise (20% contrast) along the same direction at L – M axis (A) or along orthogonal direction at S – (L + M) axis (B). Panel A shows the case where the masking is relatively strong. It is difficult to detect the target when target and noise colors are along the same direction. Panel B shows the case where the masking is relatively weak. It is easy to detect the target when target and noise colors are defined along orthogonal directions.

Colors were sampled symmetrically from a binomial distribution along a certain axis in the isoluminant plane of color space. For example, when creating a stimulus along L – M axis with a 20% contrast, the color of each small square was randomly selected to be either 20% on the red side, or −20% on the green side. Therefore, a colored stimulus in a certain direction consists of colors in that direction and in the opposing direction in the isoluminant plane (e.g. a stimulus at 30 degrees included colors at both 30 degrees and the opposing 210 degrees). 

The target and the noise were flickering independently with square wave modulations, but were spatially and temporally aligned, excluding the possibility that phase offsets may segment the target from noise (see also Hansen & Gegenfurtner, 2006; Stromeyer et al., 1999). The target signal was flickering on and off at 3 Hz, eliciting SSVEPs at 3 Hz and higher harmonics. The noise pattern flickered at 6 Hz, eliciting SSVEPs at 6 Hz and higher harmonics. The target and noise color patterns were resampled every time there was a pattern switch (i.e. 3 times a second for the signal and 6 times a second for the noise), for every flicker cycle in each trial of each observer. In the SSVEP responses, the 3 Hz response was specific to the target signal, and served to indicate the magnitude of neural responses to the target. Higher harmonics (e.g. the 6 Hz response) were a mixture of neural responses to both the target and noise, and were not considered for the analyses presented here. 

Experiment 1 – Contrast SSVEP-response functions

In experiment 1, we measured SSVEPs to the target at several contrast levels, while varying color directions of the target and noise. We used the sweep SSVEP paradigm (Norcia et al., 2015) in which the target contrast was continuously swept over five levels (i.e. 5%, 10%, 20%, 40%, and 80%). The noise contrast was fixed at 20%. Each contrast was shown for 4 seconds, resulting in a trial length of 20 seconds. The direction of noise colors was at 0 degrees, 45 degrees, or 90 degrees. The direction of tested target colors could be either the same (i.e. 0 degrees) or orthogonal (i.e. 90 degrees) to the noise direction. This led to a total of six conditions, three noise colors (0 degrees, 45 degrees, and 90 degrees) time two test colors (same or orthogonal). We ran 10 trials in each condition. Additionally, we included two noise-only trials, where only the noise (without any target) was displayed at 6 Hz at 20% contrast along the L – M axis. The purpose was to check whether the 6 Hz noise would induce a subharmonic response at 3 Hz. The sequence of all trials was randomized. Observers were required to fixate in the center of the stimuli. They were told to blink their eyes as little as possible, even though SSVEP signals are relatively immune to artifacts from eye blinks (Rossion, 2014). 

Experiment 2– Selective masking in the isoluminant plane

After exploring the contrast response function in experiment 1, we settled the contrast of both target and noise at 20% to investigate the selective masking effect in greater detail in experiment 2. In each trial, the target color varied along one of the following six directions: 0 degrees, 30 degrees, 60 degrees, 90 degrees, 120 degrees, and 150 degrees. The noise color was located at -60 degrees, −30 degrees, −15 degrees, 0 degrees, 15 degrees, 30 degrees, 60 degrees, or 90 degrees relative to the target color direction, which led to six (target color directions) times eight (noise color directions) = 48 conditions. Each condition had two trials that lasted for 21 seconds. In a single trial, the target color and noise color were fixed. The sequence of trials was randomized. The participant was asked to remain fixated in the center of the stimulus and to try to minimize eyeblinks. 

EEG recordings and analyses

EEGs were recorded from three electrodes (O1, Oz, and O2) at a 1000 Hz sampling rate (actiCAP, Brain Products, Munich, Germany). We chose to record only these three electrodes because SSVEP responses in this paradigm are mainly located at O1, Oz, and O2, based on our previous results with similar setups (e.g. Chen, McManus, Valsecchi, Harris, & Gegenfurtner, 2019; Chen, Valsecchi, & Gegenfurtner, 2017; Chen, Valsecchi, & Gegenfurtner, 2019). The ground electrode was placed at FPz, and the on-line reference electrode at FCz location. Electrode impedances were below 5 kΩ. 

Customized scripts in Matlab and functions from EEGlab toolbox (Delorme & Makeig, 2004) were used for analyses. In experiment 1, EEGs from each trial (20 seconds) were cut into five consecutive 4-second epochs, each corresponding to one level of target contrast. For each epoch, we removed the linear trend (Bach & Meigen, 1999). Epochs from the same condition (with identical colors and contrast) of each observer were averaged. The average epoch was then converted to the frequency domain by fast Fourier transformation (fft.m in Matlab). The SSVEP amplitude of the target was calculated only at the fundamental frequency (i.e. 3 Hz), as higher harmonic responses might represent a mixture of responses to both target and noise. The average amplitude of nearby four bins was subtracted from peak amplitude at 3 Hz to discount background noise from the calculated SSVEP amplitude. In experiment 2, each trial lasted 21 seconds. The first 0.5 seconds and last 0.5 seconds were discarded. The remaining 20 seconds were cut into five epochs. The remaining analysis procedures were the same as in experiment 1. Data from the recorded three electrodes (O1, Oz, and O2) were averaged in the analysis. 

To reduce between-observer variability for SSVEP responses, we normalized the data by the maximum response in each observer. 

Models and simulations

We used a chromatic line-element model to simulate the observer's response to the chromatic target in noise. The model largely followed Hansen and Gegenfurtner (2006), and similar models were previously used in the domain of color (Goda & Fujii, 2001), motion (Watamaniuk, Sekuler, & Williams, 1989), spatial frequency (Wilson & Gelb, 1984), and curvature discrimination (Wilson, 1985). Our model consists of the following processing stages: 

Stage 1. A certain number of mechanisms (N) is defined on the isoluminant plane in the DKL color space. All mechanisms have the same cosine-shaped sensitivity profile, but have different peak sensitivity angles that are equally spaced on the isoluminant plane. The sensitivity profile of the i^th mechanism at different color angles θ is defined as:  

\begin{eqnarray*}{S_i}\left( \theta \right) = {\rm{\;}}{\left[ {co{s^k}\left( {\theta - {\rm{\;}}\mu } \right)} \right]^ + },{\rm{\;}}\mu = {360^\circ }{\rm{\;}} \times {\rm{\;}}\frac{i}{N} + {\rm{\;}}\eta \end{eqnarray*}

 

The operator [ ] + denotes half-wave rectification that sets negative values to 0. The k determines the tuning width of the profile. In the present study, we fixed k = 1, so that the sensitivity tuning is a broad cosine function. The µ determines the angle with peak sensitivity. The peaks of mechanisms are equally spaced on the isoluminant plane. A noise term η (a normal distribution with mean = 0 degrees and standard deviation = 10 degrees) was added to the peak to simulate the observed variability in the preferred color angles of individual LGN neurons (Derrington et al., 1984). The jittering on mechanism angles was done independently for each small check in the stimulus once in the beginning of the simulation. All mechanism angles were then kept constant during the whole simulation. 

In the present study, we were particularly interested in the following two models: the cardinal model with N = 4 mechanisms along cardinal directions, and the higher-order model with N = 16 mechanisms. We chose N = 16 just as an approximation to the true value (same as in Hansen & Gegenfurtner, 2006). Models with a similar large number of mechanisms (e.g. 12 or 24) produced similar results. 

Stage 2. For each mechanism, we calculated the integrated response to color signals over a particular region. The response of the ith mechanism to the target region (the central 6 × 6 = 36 squares, see Figure 1) is calculated by:  

\begin{equation*}{R_{T,{\rm{\;}}i}} = {\rm{\;}}\frac{1}{{36}}{\rm{\;}}\sum r\left( {x,y} \right){S_i}\left[ {\theta \left( {x,y} \right)} \right]\end{equation*}

 

The (x, y) means the position of a particular square, r is the chromatic contrast, and θ is the color angle of the square. Similarly, the response of the ith mechanism to the background noise region (the remaining 220 squares except the central 6 × 6 region in the stimulus, see Figure 1) is calculated by:  

\begin{equation*}{R_{N,{\rm{\;}}i}} = {\rm{\;}}\frac{1}{{220}}{\rm{\;}}\sum r\left( {x,y} \right){S_i}\left[ {\theta \left( {x,y} \right)} \right]\end{equation*}

 

Stage 3. The response of the ith mechanism to the stimulus given the target and noise is calculated by:  

\begin{equation*}{\rm{\Delta }}{R_i} = {\rm{\;}}\frac{{{R_{T,i}} - {\rm{\;}}{R_{N,i}}}}{{{R_{T,i}} + {\rm{\;}}{R_{N,i}}}}\end{equation*}

 

Stage 4. The responses over all N mechanisms are pooled by computing the L2-norm, with the following equation:  

\begin{equation*}\Delta R = {\rm{\;}}\sqrt {\Delta R_1^2 + {\rm{\;}}\Delta R_2^2 + \ldots + {\rm{\;}}\Delta R_N^2} \end{equation*}

 

Stage 5. The simulation was repeated 1000 times for a given target and noise color combination. The average was used as the predicted response by the model. To allow a direct comparison between model responses and SSVEP responses, we scaled the range and mean of model responses to be the same as the range and mean of SSVEP responses. The scaling was done once for each dataset (i.e. for each subplot of Figures 2C–D, Figure 3, and Figure 5). In Figure 5, the cardinal model responses were scaled to be in the same range as the multiple mechanism model. 

Figure 2.

 

Amplitude spectrums and SSVEP contrast response function. (A) Average spectrum in the condition of target at 0 degrees, 80% contrast, and noise at 0 degrees, 20% contrast. The response at 3 Hz to the target is clearly visible. (B) Average spectrum in noise-only trials, where the noise was at 0 degrees, 20% contrast. The 6 Hz noise induced response at 6 Hz but not at 3 Hz (i.e. no subharmonics). (C) For cardinal axes, the normalized SSVEP response to target as a function of contrast, separately for the target being at same direction as noise and target at orthogonal direction to noise. (D) For intermediate axes, the normalized SSVEP response to target as a function of contrast, separately for the target being at same direction as noise and target at orthogonal direction to noise. The black and red curves in C and D show predictions of a model with multiple chromatic mechanisms (N = 16). Error bars indicate the within-subject 95% confidence intervals (Cousineau, 2005).

Figure 3.

 

Average normalized SSVEP amplitudes of eight observers to flickering targets embedded in noise. Subplots show the results for target colors at 0 degrees, 30 degrees, 60 degrees, 90 degrees, 120 degrees, and 150 degrees. The x-axis indicates the noise direction relative to the target. The red curves denote the model prediction with N = 16 chromatic mechanisms. Overall, SSVEPs are the largest (i.e. indicating weakest masking) when target and noise are close to orthogonal, and are the smallest (i.e. indicating strongest masking) when target and noise are along the same direction. The result patterns are the same between when the target was at cardinal directions (0 degrees and 90 degrees) and when the target was at intermediate directions (30 degrees, 60 degrees, 120 degrees, and 150 degrees).

Results

Experiment 1 – SSVEP contrast response function

We measured the SSVEPs for the target at several contrast levels (i.e. 5%, 10%, 20%, 40%, and 80%), while keeping the contrast of the noise constant at 20%. We fixed the noise color at a specific color direction (either at cardinal axes, i.e. 0 degrees/90 degrees, or at the intermediate axis, i.e. 45 degrees), and tested with the target at the same or orthogonal to the noise direction. 

Figures 2A and 2B show the amplitude spectrum when the target was at a contrast of 80% at the 0 degrees color direction (see Figure 1A), and when only the noise was presented in the same 0 degrees color direction (see Figure 1B). Figure 1A shows a well-defined peak at 3 Hz, indicating a large SSVEP response to the 3 Hz target. In contrast, in Figure 1B, there was no visible response at 3 Hz when only the 6 Hz noise was presented. Therefore, the 6 Hz noise did not produce subharmonic responses at 3 Hz in our paradigm. 

Figures 2C and 2D show the average normalized SSVEP responses of eight observers, as a function of target contrast. Overall, the results revealed a selective masking effect on the contrast response function. That is, stronger masking was observed in the “same direction” condition (solid curve) compared to the “orthogonal direction” condition (dotted curve). Importantly, this pattern was the same irrespective of whether the color was at cardinal axes (see Figure 2C) or intermediate axes (see Figure 2D). 

To further examine the nature of underlying chromatic mechanisms, we used a chromatic detection model following previous studies (Goda & Fujii, 2001; Hansen & Gegenfurtner, 2006). The purpose of the model was not to minimize the deviations of the predictions from the data by systematically adjusting the parameters. Instead, the model simulated the steps in the processing of chromatic target in noise, and gave rise to a tuning curve with predetermined parameters. We aimed to test whether the model could produce similar responses as the SSVEPs. In Figures 2C and 2D, we plotted the prediction of our computational model, which had multiple higher-order chromatic mechanisms (N = 16, see Method section). The model predicts the SSVEP data fairly well. 

Please note that the curves shown in Figures 2C and 2D appear exactly opposite to the psychological masking functions in previous studies (e.g. Figure 5 in Hansen & Gegenfurtner, 2006), because the masking effect in the present study is indicated by a decrease in brain responses to the target, whereas the masking effect in previous psychophysical experiments is indicated by an increase in detection threshold. 

Experiment 2 – Selective masking in the isoluminant plane

In experiment 2, we fixed the contrast of the target and noise at 20%, and explored selective masking in the isoluminant plane in greater detail. Looking at Figure 2, it can be seen that this combination of contrasts avoids the floor or ceiling effects. In each condition, we fixed the color direction of the target and varied the color direction of the noise relative to the target, from −60 degrees, −30 degrees, −15 degrees, 0 degrees, 15 degrees, 30 degrees, 60 degrees, to 90 degrees. In total, we tested six different target colors at 0 degrees, 30 degrees, 60 degrees, 90 degrees, 120 degrees, or 150 degrees. 

The results are shown in Figures 3 and 4, where the average SSVEP responses of eight observers at six target color directions (0 degrees, 30 degrees, 60 degrees, 90 degrees 120 degrees, and 150 degrees) are plotted. The red curves plot the prediction of the higher-order model with multiple mechanisms (N = 16). The predictions did fit the SSVEP data fairly well. In contrast, predictions of the cardinal model failed to do so (for clarity reason, we did not show them in Figure 3, but see Figure 5). Figure 4 re-plots the result of the 0 degrees and 30 degrees target color directions in polar coordinates. For all six target directions, the response curves were remarkably consistent. SSVEP responses were largest when target and noise were orthogonal, and were smallest when target and noise were along the same direction. In other words, the masking effect of the noise on the target depended on their color directions. Masking was weakest when target and noise were orthogonal, and strongest when the target and noise were along the same direction. In particular, this holds true for target directions along both the cardinal axes (0 degrees and 90 degrees) and the intermediate axes (30 degrees, 60 degrees, 120 degrees, and 150 degrees). This result is in agreement with the multiple mechanism hypothesis, and strongly in contradiction of a cardinal mechanism model. 

Figure 4.

 

A polar plot of some of the results shown in Figure 3. (A) Shows the case where the target color was along 0 degrees, whereas (B) shows the case where it was along 30 degrees (as indicated by the thick red line). The distance of each data point to the origin indicates the normalized SSVEP amplitude. Two example stimuli are included in each subplot, with one having orthogonal color directions and the other having same color directions. The bars at the lower right represent the average between-subject standard errors.

To further compare the model with our data, we grouped the data separately for cardinal axes and intermediate axes. Figure 5 shows the overall average SSVEP responses as a function of noise color in two conditions. The red curves show the prediction from the higher-order model with N = 16 mechanisms, which accounts for the cardinal data and inter-mediate data very well. The gray dotted lines are predictions from the cardinal model with N = 4 mechanisms. The cardinal model failed to account for the result, especially for inter-mediate axis (see Figure 5, right panel). 

Figure 5.

 

Overall average of normalized SSVEP amplitudes (8 observers), plotted as a function of the color direction of the noise relative to the target. The patterns are similar between when the target is at cardinal directions (left) and when at intermediate directions (right). The red curves show predictions of the higher-order model with multiple (N = 16) mechanisms, which fits both datasets quite well. The gray dotted lines denote the prediction of the cardinal model (N = 4 mechanisms). Error bars show the between-subject standard errors.

Discussion

The present study revealed selective masking at cardinal axes as well as intermediate axes in a noise masking paradigm with neurophysiological indices. Instead of measuring the detection threshold of a chromatic target embedded in chromatic noise as in previous psychophysical studies, we obtained SSVEP responses to the target. We found that the target elicited the smallest SSVEPs (i.e. strong masking) when the target and noise were along the same color direction, and induced the highest SSVEPs (i.e. weak masking) when the target and noise were along orthogonal directions. This pattern of results was observed both when the target color was along cardinal directions and intermediate directions. Selective masking at intermediate directions is evidence for the existence of higher-order color mechanisms at intermediate axes. Our simulation results suggest that a model with multiple (N = 16) chromatic mechanisms fits the SSVEP data fairly well. In contrast, the cardinal model with N = 4 mechanisms failed systematically to account for the data. Therefore, our neurophysiological results confirm a number of previous psychophysical findings (Cass et al., 2009; D'Zmura & Knoblauch, 1998; Goda & Fujii, 2001; Hansen & Gegenfurtner, 2006; Hansen & Gegenfurtner, 2013; Krauskopf & Gegenfurtner, 1992; Krauskopf et al., 1986; Li & Lennie, 1997; Lindsey & Brown, 2004; Stromeyer et al., 1999), and argue for the existence of higher-order chromatic mechanisms. 

In both experiments, the model with N = 16 mechanisms fit the data very well. We used 16 mechanisms because this number turned out to provide the best fit to a vast amount of psychophysical data in a previous study (Hansen & Gegenfurtner, 2006). However, please note that we are not arguing that the number is exactly 16. In fact, models with similar large numbers (e.g. 12 or 24) led to similar good fits. It has been argued by Shepard et al. (2016), that our procedure of jittering the directions of the mechanisms might increase the actual number of mechanisms to 576 or more. Although this might sound excessive, one has to keep in mind that our model is essentially parameter free (i.e. additional mechanisms do not simply lead to better fits because there are more tunable parameters). It is also surely the case, that at the neural level there is such variability in the preferred color directions. No two neurons would have the exact same preferences. Finally, all our jittering procedure achieves is to smooth the tuning curves, which would otherwise have numerical artifacts when noise and mechanism direction perfectly agree. Whether such a model is realistic could only be determined by measuring the responses of a large number of individual neurons to these stimuli. What our results here show is that the model can capture the SSVEP mass activity of a large neuronal population quite well, and that more than four mechanisms in the equiluminant plane are needed. 

Our model assumed that the sensitivity profile of each mechanism is a broad cosine profile. The resulted response curve, however, is relatively narrow compared to cosine. This is due to off-axis looking, which occurs for one-sided noise as used in the present study. Off-axis looking does not require any attentional strategies of the observer. It simply means that the signal-to-noise ratio might be highest in a chromatic channel different from the signal direction. Hansen and Gegenfurtner (2006) modeled both one-sided noise and two-sided noise, and showed that narrow-tuning and broad-tuning are observed, respectively. This is also what has been found empirically for one-sided and two-sided noise stimuli (D'Zmura & Knoblauch, 1998; Hansen & Gegenfurtner, 2006). The tuning response we observed with SSVEP neural indices is remarkably similar to their psychophysical results. These results are consistent with the notion that cardinal mechanisms are linearly combined to form higher-order chromatic mechanisms. 

In the present study, we choose not to determinate isoluminance for individual observers. It is quite well established that chromatic sensitivity is far better than luminance sensitivity under the conditions we used (Chaparro, Stromeyer, Huang, Kronauer, & Eskew, 1993; Gegenfurtner & Hawken, 1996). Furthermore, luminance noise has been shown to have no masking effect on chromatic targets (Cass et al., 2009; Gegenfurtner & Kiper, 1992; Hansen & Gegenfurtner, 2006; Li & Lennie, 1997; but see Wang, Richters, & Eskew, 2014). Note also that the potential contribution from luminance artifacts is balanced in the design of experiment 2. In experiment 2, the SSVEP response was induced by a target with fixed contrast and color direction, whereas the noise was varied in color direction. Even if there were some luminance artifacts in the target, it is constant across different conditions (i.e. across different data points in each subplot of Figures 3–5). Therefore, luminance contributions cannot explain the results in the present study. 

One may argue that the SSVEP responses at 3 Hz can be also produced by the 6 Hz noise as a “subharmonic.” The lack of 3 Hz SSVEPs in the noise-only condition rules out this possibility (see Figure 2B). The majority of previous SSVEP studies also did not find any subharmonic response (Rossion, Retter, & Liu-Shuang, 2020), with only a few exceptions (Herrmann, 2001; Walter, Dovey, & Shipton, 1946). A recent review has argued that the occasional findings of subharmonics were possibly due to asymmetries in the onset/offset of the stimulation (Rossion et al., 2020). 

If not due to interfaces from subharmonics, what is the underlying mechanism that drives the masking effect of the noise on the target? One of the possible mechanisms is contrast gain control. Previously, a few studies examined the masking effect using SSVEPs, with black-and-white noise or gratings (Candy, Skoczenski, & Norcia, 2001; Tsai, Wade, & Norcia, 2012), and they found that a contrast gain control model could account for the masking effect very well. Despite that we used another type of stimulus (i.e. isoluminant chromatic stimuli), the masking effect could be explained by a similar gain control mechanism. Note that our chromatic detection model predicts the masking functions along a single-color direction quite well (see Figure 2). It does so by divisive response normalization (Hansen & Gegenfurtner, 2006). 

Previous disagreements on higher-order mechanisms came from some studies that did not find selective masking at intermediate axis in noise masking, suggesting that there is no evidence for higher-order mechanisms (Eskew et al., 2001; Giulianini & Eskew, 1998; Sankeralli & Mullen, 1997). Interestingly, the studies that found positive evidence all defined their stimuli in the DKL space (Cass et al., 2009; D'Zmura & Knoblauch, 1998; Goda & Fujii, 2001; Hansen & Gegenfurtner, 2006; Krauskopf & Gegenfurtner, 1992; Krauskopf et al., 1986; Li & Lennie, 1997; Lindsey & Brown, 2004; Stromeyer et al., 1999), whereas the studies that found no evidence defined color stimuli in cone contrast space (Eskew et al., 2001; Giulianini & Eskew, 1998; Sankeralli & Mullen, 1997). Hansen and Gegenfurtner (2013) showed that their failure was likely due to a restricted choice of color in the cone contrast space. The sampled colors were perceptually similar, even though their color directions were orthogonal in the cone-contrast space. After adjusting the sampling procedure, Hansen and Gegenfurtner (2013) did observe selective masking even for stimuli defined in the cone contrast space. 

In a previous study with chromatic visual evoked potential (VEPs), Duncan, Roth, Mizokami, McDermott, and Crognale (2012) studied higher-order mechanisms using chromatic contrast adaptation, and revealed color mechanisms tuned to intermediate directions. Our results show that the SSVEP can be successfully used to measure color tuning in the human visual cortex as well. Previously, this technique was mainly used to characterize retinal color mechanisms (e.g. Jacob, Pangeni, Gomes, Souza, da Silva Filho, Silveira, & Kremers, 2015; Kommanapalli, Murray, Kremers, Parry, & McKeefry, 2014; Kremers & Link, 2008), luminance (e.g. Regan, 1970; Siegfried, Tepas, Sperling, & Hiss, 1965), or the role of color in attention (Martinovic & Andersen, 2018; Martinovic, Wuerger, Hillyard, Müller, & Andersen, 2018). Cortical chromatic mechanisms were mainly studied by investigating the color tuning of single neurons in the monkey brain. Unlike neurons in the LGN, where their preferred colors clustered exclusively at the cardinal axes (Derrington et al., 1984), neurons in cortical visual areas have preferences for a variety of hues (Lennie, Krauskopf, & Sclar, 1990; for review, see Conway, Chatterjee, Field, Horwitz, Johnson, Koida, & Mancuso, 2010; Gegenfurtner, 2003). A group of neurons with similar hue preference may form one higher-order mechanism described in psychophysical studies. 

Earlier fMRI studies focused on cardinal mechanisms, revealing independent coding of colors at the three cardinal axes in the primary visual cortex (Engel & Furmanski, 2001; Engel, Zhang, & Wandell, 1997; Kleinschmidt, Lee, Requardt, & Frahm, 1996; Schluppeck & Engel, 2002). Recently, a series of studies examined higher-order mechanisms by decoding different colors at inter-mediate axes from fMRI signals (Brouwer & Heeger, 2009; Goddard et al., 2010; Kuriki et al., 2011; Kuriki et al., 2015; Parkes et al., 2009). These studies found that intermediate colors could be successfully classified from brain activities in the early visual cortex, suggesting that specific mechanisms are responsible for intermediate colors. 

In summary, there is converging evidence from psychophysics, single neuron recordings, and human brain imaging in support of multiple higher-order chromatic mechanisms. Our study provides firm physiological evidence from humans, and introduces a new way to investigate human chromatic mechanisms by exploiting the well-established noise masking paradigm and the SSVEP technique. 

Acknowledgments

The authors thank Thorsten Hansen for help with the modeling, and Chenxi Liang for help with data collection. 

Funded by Deutsche Forschungsgemeinschaft SFB/TRR 135 (Grant Number 222641018) TP C2, the National Natural Science Foundation of China (31900758), the Shanghai Sailing Program (19YF1445900), and the Shanghai Program for High-Level Overseas Talents (TP2019071). 

The raw data of the current study is publicly available on zenodo.org at DOI:10.5281/zenodo.5077126. 

Commercial relationships: none. 

Corresponding author: Jing Chen. 

Email: [email protected]. 

Address: School of Psychology, Shanghai University of Sport, Shanghai, China.