A comparison of pedestal effects in first- and second-order patterns

Abstract

Abstract

Abstract:

Abstract  The human visual system is sensitive to both luminance (first-order) and contrast (second-order) modulations in an image. A linear-nonlinear-linear model is commonly used to explain visual processing of second-order patterns. Here we used a pattern-masking paradigm to compare first-order and second-order visual mechanisms and to characterize the nonlinear properties underlying them. The carriers were either a high-frequency horizontal grating (8 c/°) or a binary random dot pattern; they were either added to a vertical low-frequency (2 c/°) sinusoidal grating (first-order stimuli) or multiplied by it (second-order stimuli). The incremental discrimination threshold of the target was measured with pedestals whose spatial properties matched those of the target, with the exception of contrast (in the first-order pedestal) or modulation depth (in the second-order pedestal). The threshold function showed a typical dipper shape for both first- and second-order stimuli. The results for the first-order stimuli with different types of carrier and the second-order stimuli with a grating carrier were well explained by a divisive inhibition model in which the facilitatory input was divided by the sum of broadband inhibitory inputs. The results for the second-order stimuli with a random-dot carrier were explained by a modified divisive inhibition model that operated on modulation depth. Our results suggest that divisive inhibition is required to explain visual discrimination in both first- and second-order patterns. However, the source and nonlinearity of the divisive inhibition may be different for these two types of patterns and carrier.

Introduction

It is well documented that the visual system is sensitive to both first-order and second-order variations in an image (Cavanagh & Mather, 1989; Chubb & Sperling, 1988; Harris & Smith, 1992; Lu & Sperling, 1995; Schofield & Georgeson, 1999; Smith, 1994; Sutter & Graham, 1995). First-order information usually refers to a spatiotemporal modulation in luminance or chromaticity that has a corresponding energy in the Fourier domain and is detectable by a linear operator whose receptive field contains excitatory and inhibitory regions similar to those of the simple cells in the primary visual cortex. On the other hand, second-order information, or the modulation of first-order information in an image, has no corresponding energy in the Fourier domain and is supposedly not detectable by a simple linear receptive field (Chubb & Sperling, 1988). For example, a contrast-modulated (CM) image can be constructed by multiplying a high-spatial-frequency sinusoidal grating (carrier) by a low-spatial-frequency one (envelope). When the orientations of the carrier and envelope match, the Fourier transform of this CM image at frequency energy is the sum and the difference of the carrier and envelope frequencies. There is no energy at the envelope frequency in the power spectrum of the CM stimulus; yet a human observer has no problem in perceiving the envelope of CM stimuli. Therefore, a further process is needed to derive the envelope information. 

In the literature, linear-nonlinear-linear (LNL) models are commonly used to explain visual behavior to a second-order stimulus (Graham & Sutter, 1996, 1998, 2000; Sutter & Graham, 1995; Wilson, 1994). These models generally contain several stages. The first stage involves a band of linear filters that are responsive to the carrier wave. Several studies have explored the physiological basis (Mareschal & Baker, 1998) and psychophysical properties (Dakin & Mareschal, 2000; Graham, 1994; Graham, Sutter, & Venkatesan, 1993) of these linear filters. A nonlinear transform is then applied to the linear-filter responses. The nonlinear transform is not well understood at this stage. Since any type of nonlinear calculation can help to derive the envelope information, most researchers have used rectification, for its simplicity. However, Graham and Sutter (Graham, 1994; Graham & Sutter, 1998, 2000) found that the nonlinearity is in accelerating/expansive form. Some have suggested that a gain-control mechanism may be involved after the rectification process (Graham, 1994; Graham, Beck, & Sutter, 1992; Graham & Sutter, 1998, 2000; Olzak & Thomas, 1999; Olzak & Wickens, 1997; Thomas & Olzak, 1996). This contrast gain control pools and normalizes the responses across spatial frequency or orientation information. Finally, another set of linear filters operates on the output of the nonlinear stage and thus is able to extract the envelope (Cavanagh & Mather, 1989; Chubb & Sperling, 1989), and it is usually assumed to have lower spatial-frequency tuning than the first set of filters. 

Several studies have measured sensitivity to the contrast modulation of a 1-D or 2-D carrier, and they found that the modulation transfer function (MTF), the sensitivity across differential-modulation spatial frequencies, is a more low-pass one (Jamar & Koenderink, 1985; Rovamo, Franssila, & Näsänen, 1992; Schofield & Georgeson, 1999; see also Sutter, Sperling, & Chubb, 1995). Like the first-order contrast-sensitivity function, the second-order MTF is also tuned to orientation and spatial frequency (Arsenault, Wilkinson, & Kingdom, 1999; Kwan & Regan, 1998). It has also been shown that lower detection thresholds are found when the spatial frequency of the carrier is higher on the order of three to four octaves than that of the envelope, and that detection threshold does not change with viewing distance (scale invariance; Dakin & Mareschal, 2000), suggesting a linkage between the scale of first-stage and second-stage linear filters. 

Psychophysical studies suggest that, in the human visual system, the analysis of first- and second-order stimuli may involve different mechanisms. In order to compare the first- and second-order mechanisms under equal circumstances, researchers usually use luminance-modulated (LM) patterns as a comparison to contrast-modulated (CM) patterns. CM patterns are constructed by multiplying a high-spatial-frequency carrier or a white-noise carrier by a low-spatial-frequency envelope as the second-order stimulus, while luminance-modulated patterns (LM) are constructed by adding the carrier and the envelope. There is no subthreshold summation between the first- and second-order stimuli (Schofield & Georgeson, 1999). The detectability of the envelope improves with carrier contrast for the second-order patterns but degrades for the first-order patterns (Schofield & Georgeson, 1999, 2003). In addition, the shape of the CM contrast-sensitivity function is independent of the spectrum of its noise carrier, but that of the LM contrast-sensitivity function is dependent on the spectrum of its additive noise mask (Schofield & Georgeson, 2003). In addition to psychophysical evidence, there are supporting electrophysiological studies showing that in cats, neurons in areas 17 and 18 have different spatial-frequency and orientation tuning for first- and second-order stimuli (Baker & Mareschal, 2001; Mareschal & Baker, 1998). This also indicates that a neuron's firing rate reflects a combination of inputs from first-order and second-order processing. The temporal property of the second-order system is slightly sluggish compared with that of the first-order system (Schofield & Georgeson, 2000), and it has a sustained temporal response with dynamic carrier noise and a transient temporal response with static carrier noise (Schofield & Georgeson, 2000). A study of visually evoked potential (VEP) has shown that response latency is longer for CM stimuli than for LM stimuli (Calvert, Manahilov, Simpson, & Parker, 2005). This longer latency found in VEP, which is also found in physiological data (Mareschal & Baker, 1998), might be due to (a) the slower first-stage linear filter and (b) the additional processing stage of the second-order pathway. 

Pattern-masking paradigm

The pattern-masking paradigm has been widely used in estimating the nonlinear properties of first-order visual mechanisms (Chen, Foley, & Brainard, 2000a, 2000b; Chen & Tyler, 2001, 2008; Foley, 1994; Klein, 2006; Nachmias & Rogowitz, 1983; Ross & Speed, 1991). The task of the observer is to determine whether the difference is identifiable between the pedestal alone (intensity C) and the target plus the pedestal (intensity C+ΔC). In contrast discrimination, a typical result of the experiment plots the detection threshold of the target (ΔC) against the pedestal intensity (C), showing a dipper-shaped target threshold versus the pedestal contrast (TvC) function (Chen & Foley, 2004; Kontsevich & Tyler, 1999; Legge & Foley, 1980). That is, as pedestal contrast increases, the target threshold first decreases (facilitation) and then increases (suppression). The TvC function reflects the response characteristics in the visual system, and the slope of the response function at C is inversely proportional to the contrast discrimination threshold (ΔC) at base contrast C (see Model for the details). 

While this pattern-masking approach has been widely used in the study of first-order visual mechanisms, to the best of our knowledge there have been only a few studies that systematically investigate the response function of second-order mechanisms (Kingdom, Prins, & Hayes, 2003; Schofield & Georgeson, 1999). Here, we used pattern-masking methods to systematically measure the threshold functions for second-order pattern vision as well as first-order pattern vision. Furthermore, we investigated how the carrier contrast influenced the TvC function for both first- and second-order pattern vision. In the first part of the study, we measured the TvC function by using a 1-D carrier: a horizontal grating. In this experimental design, we could investigate the nonlinear properties of the early and late linear filters. In the second part of the research, we replicated Schofield and Georgeson's (1999) experimental results by using a 2-D noise carrier and adopted the contrast-gain-control model to fit the data. We also compared the function between LM and CM patterns to see how the model parameters could describe the data. 

In summary, our purposes in this study were to (a) compare the masking effects in first- and second-order pattern vision and (b) characterize a nonlinear transform for second-order vision, both the nonlinearity between the first and second stage of the linear filter and the nonlinearity after the second stage. 

Method

Apparatus

The stimuli were presented on two 17-in. ViewSonic Monitors (Professional series p75f+), which were driven by a Macintosh computer running the OS9 operating system. The lights from the two monitors were combined with a beam splitter placed at 15 cm in front of the right eye of the observers. A Radeon graphics card was used to provide 10 bits of DAC (digital-to-analog converter) depth for each gun. The resolution of the monitor was 800 × 600 pixels, with a refresh rate of 72 Hz. The size of a pixel was 1 min in Experiment 1 and 0.92 min in Experiment 2. The computer program for experimental control was written in Matlab with Psychophysics Toolbox (Brainard, 1997). The input-voltage/output-intensity function for each monitor was measured by a LightMouse photometer (Tyler & McBride, 1997), and the information was used to compute the linear lookup table to correct the nonlinear properties of the monitor and to equalize the mean luminance of the two monitors. The participants viewed the stimuli monocularly. 

Stimulus

In order to compare the first-order and second-order mechanisms under equal circumstances, we used a contrast-modulated pattern (CM) as the second-order stimulus and a luminance-modulated pattern (LM) as the first-order stimulus (Figure 1). 

Figure 1

 

(A) A CM pattern is a combination of a carrier and a sideband. (B) The stimuli used in Experiment 1. (C) The stimuli used in Experiment 2. The first number in the bottom of each stimulus indicates the contrast of the carrier (c), and the second number indicates the envelope contrast (m).

The luminance-modulated pattern LM is the sum of an envelope G(x, y), which is a low-spatial-frequency vertical sinusoidal grating, and a carrier pattern N(x, y), which is either a high-spatial-frequency horizontal sinusoidal grating (Experiment 1) or a 2-D white noise (Experiment 2). That is,  

 

where I₀ is the mean luminance (32.9 cd/m²), c is the contrast of the carrier, and m is the contrast of the envelope. Note that G(x, y) is called the “envelope” here to be consistent with the description for the second-order pattern. The function w(x, y) is an isotropic Gaussian window whose scale parameter (“standard deviation”) was 1.23° in Experiment 1 and 0.96° in Experiment 2. 

The envelope G(x, y) is defined as  

 

where f_e is the spatial frequency of the envelope wave, 2 c/°. 

The carrier N(x, y) in Experiment 1 was a horizontal grating,  

 

where f_c is the spatial frequency of the carrier wave, 8 c/°, which is two octaves higher than the envelope's spatial frequency. The carrier pattern N(x, y) in Experiment 2 was a 2-D white noise, and the size element was 4 × 4 pixels (3.68′). 

The contrast-modulated (CM) patterns comprised the modulation of the contrast of the carrier N(x, y) by the envelope G(x, y). With a Gaussian window w(x, y), it can be described by  

 

 

The meanings of the symbols are the same as those for the LM stimuli (Equation 1). The spatial frequency of the envelope wave is 2 c/°. Note that, to be consistent with the literature, m for the CM stimuli is called modulation depth, rather than contrast as for the LM stimuli. 

Equation 4 can be rewritten as follows:  

 

 

In this way, we separated the carrier pattern N(x, y) from its sideband pattern N(x,y) × G(x,y). This allowed us to present the carrier and sideband on different monitors (Figure 1A) and to manipulate the modulation depth of the CM pattern and the contrast of the LM pattern independently, simply by changing the lookup tables at different monitors. The frequency and orientation information of the sideband image was 8.25 c/° and 23.66° deviated from the orientation of the carrier in Experiment 1. 

Three carrier contrasts c for LM stimuli (40%, 16%, and 0%) and two carrier contrasts for CM stimuli (40% and 16%) were used in the experiments. In Results, we present contrast values in decibels (dB), or 20 times the logarithm of linear contrast. Thus the carrier contrast of 40% and 16% is equivalent to −8 and −16 dB, respectively. 

Our experiments measured the increment threshold (see the Procedure subsection). Thus, for each trial, the observer was to compare a pedestal, or a pattern at base contrast (for LM) or modulation depth (for CM), with another pattern with an increment, or target plus pedestal. The equations for the target plus pedestal are  

 

and  

 

where m is the pedestal contrast (for LM) or modulation depth (for CM) and Δm is the target contrast (for LM) or modulation depth (for CM). 

Procedure

A temporal two-alternative forced-choice (2AFC) paradigm was used to measure the increment threshold of the target. In each trial, the observers were first shown a black fixation dot (6″ × 6″) at the center of the screen for 83 ms, followed by a 200-ms blank. The first interval started after the blank. In each interval, the stimulus was a temporal abrupt pulse with a duration of 139 ms. A 600-ms interstimulus interval, from the offset of the first stimulus to the onset of the second stimulus, separated the two intervals. An auditory tone was given before each interval to alert the observer to the onset of the stimulus. One interval contained the pedestal (Equation 1 for the LM pattern and Equation 5 for the CM pattern) and the other contained the pedestal-plus-target (Equation 6 for the LM pattern and Equation 7 for the CM pattern). The observers were required to determine which interval contained the pattern with a higher contrast (LM) or modulation depth (CM). An auditory feedback was given to the observers after each trial. 

The threshold was measured with the constant-stimuli paradigm. In each run, there was one pedestal contrast (m in Equation 1) or modulation depth (m in Equation 5) and seven levels of target contrast (Δm in Equation 6) or modulation depth (Δm in Equation 7). There were 10 repetitions of each target contrast or modulation depth in each run. The order of target contrast and modulation depth was randomized within the run. The experimental runs were further blocked by pedestal type (LM or CM) and carrier contrast (−∞, −16, or −8 dB). Each block contained several pedestal contrasts or modulation depths for a specific pedestal type and carrier contrast. That is, each block measured one TvC function. There were four repetitions for each pedestal contrast or modulation depth. Thus the psychometric function for a pedestal had 280 data points. The order of runs in each block was randomized. 

The psychometric function Ψ was fitted by  

 

where Φ( ) is the cumulative Gaussian function, and μ represents the mean and σ the standard deviation of the Gaussian function. The value of γ was set at 0.5, which is the guessing rate for the 2AFC paradigm, and λ was constrained between 0 and 0.05, which indicates the rate of finger error. A maximum-likelihood method was used to estimate μ and σ free parameters (Wichmann & Hill, 2001a, 2001b). The detection threshold was defined as the value of μ, and the standard deviation of the detection threshold was estimated by the bootstrapping method. 

Participants

Two observers (CWC and PCH) with corrected-to-normal vision participated in this study. PCH is one of the authors, and CWC was paid and unaware of the purpose of the experiment. 

Results

Experiment 1: Horizontal grating carrier

The TvC functions for the horizontal grating carrier are shown in Figure 2A. The thresholds for the LM stimuli are represented by circles, while those for the CM stimuli are represented by diamonds. The green circles represent the detection threshold for the no-carrier condition (LM only); the red circles, the low-carrier-contrast (−16 dB) LM condition; and the blue circles, the high-carrier-contrast (−8 dB) LM condition. The TvC functions all show a typical dipper shape. That is, the threshold first decreased (facilitation) and then increased (suppression) with pedestal contrast. All three curves overlap with each other. This is consistent with previous results for these conditions, that the target and pedestal are processed independently in the visual system such that the spatial frequency of the target and the pedestal were three octaves apart or that their orientations were orthogonal (Blakemore & Campbell, 1969; Blakemore, Nachmias, & Sutton, 1970). 

Figure 2

 

The experimental results for (A) the grating carrier (Experiment 1) and (B) the noise carrier (Experiment 2). The TvC functions for the LM and CM stimuli are plotted in the same graph for each subject (CWC and PCH). The green circles represent the absence-of-carrier condition. The red circles represent the low-carrier-contrast (−16 dB) LM condition and the blue circles represent the high-carrier-contrast (−8 dB) LM condition. The magenta diamonds represent the low-carrier-contrast (−16 dB) CM condition and the cyan diamonds represent the high-carrier-contrast (−8 dB) CM condition. The solid lines represent the results of the model fits.

The TvC function for the CM stimuli is shown in magenta diamonds for the low-carrier-contrast condition (−16 dB) and cyan diamonds for the high-carrier-contrast condition (−8 dB). The TvC function shows a typical dipper shape, as commonly observed in TvC functions for first-order pattern masking. When there was no pedestal (−∞ pedestal contrast), the detection threshold for lower carrier contrast was about twice (−6 dB) that for high carrier contrast. This result is consistent with previous findings (Schofield & Georgeson, 1999). This difference in threshold decreased as the pedestal modulation depth increased; thus the two TvC functions merged at high modulation depth. 

Experiment 2: 2-D noise carrier

Figure 2B shows the TvC functions for the 2-D noise carrier. The data for LM stimuli are represented by circles. All LM TvC functions show a typical dipper shape. At low pedestal contrasts, the target threshold is higher when the carrier contrast is higher. At high pedestal contrasts, the carrier contrast has little effect, and the TvC functions for different carrier contrasts merge. This carrier-contrast effect was different from that for the grating carrier, in which the TvC functions were about the same for the three different carrier contrasts, suggesting that contrast of the grating carrier has only trivial influence. 

The TvC function for the CM stimuli (represented by diamond symbols) shows that the threshold initially decreased as pedestal modulation depth increased. However, up to the maximum pedestal modulation depth that can be produced by our apparatus, there was no sign of threshold increment at high pedestal modulation depth. This is probably due to the fact that the pedestal modulation depth we used was not high enough to show an inhibitory effect. 

When there was no pedestal, the target-detection threshold at low carrier contrast was higher than the threshold at high carrier contrast. Schofield and Georgeson (1999) showed a similar result. This difference in threshold decreased as the pedestal modulation depth increased for PCH. As a result, the two TvC functions merged at high pedestal modulation depth. This effect is similar to that for the grating carrier. 

Model

General model

Our model for second-order pattern discrimination combines two modeling approaches in the literature: the LNL models that are commonly used to explain the detection of second-order patterns (Cavanagh & Mather, 1989; Chubb & Sperling, 1988; Graham & Sutter, 1996, 1998, 2000; Sutter & Graham, 1995; Wilson, 1994) and the divisive-inhibition models that are commonly used to explain discrimination of first-order patterns. An LNL model usually contains a set of linear filters that operate on the image. The output of each linear filter undergoes a nonlinear operation. A second set of linear filters then operate on the nonlinear-transformed earlier filter outputs. Decision making is based on the response of the second set of filters. In general, an LNL model uses a simple nonlinearity such as rectification (Graham & Sutter, 1998, 2000) or squaring. It is known that a simple nonlinearity cannot explain pattern discrimination for the first-order stimulus (Chen, Foley & Brainard, 2000b; Legge & Foley, 1980). Instead, a sigmoid response function derived from a divisive-inhibition process is required (Foley, 1994). We thus implement rectification and divisive inhibition in our model for the nonlinear operation. 

The most general form of our model contains the following stages (see Figure 3): 

Figure 3

 

Schematic illustrations of LNL model and contrast-gain-control model. (See text for the details.)

1. Early linear filters. The first stage of the model contains a band of linear filters operating on the input images. The sensitivity profile of a linear filter is a wavelet defined by the Gabor function to mimic the receptive field of the neurons in the primary visual cortex. The excitation E′ of the i-th linear filter, which is centered at position (x, y) and has a sensitivity profile f_i, to the j-th image L_j is given by the cross correlation  

 

 

2. Early nonlinear response. The excitation of the linear filter is first half-wave rectified (Heeger, 1992):  

 

The rectified excitation of the i-th linear filter then undergoes a linear transform to produce the i-th early-channel response R_i, given as  

 

Here we drop the notation (x, y) for simplicity. The numerator in Equation 10 is the half-wave-rectified excitation of the i-th filter, E_i,j, raised to a power p. The denominator contains the divisively inhibitory term I_j and an additive constant z. The inhibition term is a nonlinear combination of the excitation of a set of relevant channels (Foley, 1994; Heeger, 1992). That is,  

 

where q is an exponent parameter and w_i is the contribution of the i-th filter excitation to the inhibition inputs. 

3. Late linear filters. A second set of linear filters then operate on the early nonlinear responses to extract the contrast-modulation signal. The excitation of a later linear filter H is the cross correlation between its sensitivity profile ϕ and the early responses R₁. That is,  

 

 

4. Late nonlinear response. The response of the later mechanism T is then the ratio between the half-wave-rectified second-order linear-filter excitation, raised to a power π, and the late divisive-inhibition input K plus a constant ζ. That is,  

 

Again, the divisive inhibition is a nonlinear combination of the linear-filter excitations. That is,  

 

where θ is the exponent parameter and ω_k is the contribution of the k-th filter excitation to the inhibition inputs. 

5. Decision making. In our 2AFC experiment, the task of the observer was to tell which of the two intervals contained the target. The threshold was determined by the rule that the difference between the response to the pedestal-plus-target and the response to the pedestal alone was great enough to overcome the noise in the system. We assume that the decision-making stage has access to both the early (R_i,j) and late (T_k,j) responses. Therefore, there are two possible decision variables:  

 

and  

 

where D₁ and D₂ are the decision variables and σ₁ and σ₂ are the standard deviations of the noise in the early and late mechanisms, respectively. The subscript p+t denotes the pedestal-plus-target interval, while p denotes the pedestal interval. 

We assume that intrinsic noise in the system is uncorrelated with our task and thus that the standard deviation of the intrinsic noise should be a constant. The source of the external noise was the noise carrier. Since the contrast of the noise carrier was fixed for each TvC function, variability in the response produced by the noise carrier should also be a constant. That is, the standard deviations σ₁ and σ₂ are constants. 

A target is detectable when it produces a response large enough to overcome the noise—that is, the decision variable reaches a criterion. For the convenience of computation, we scaled this criterion to 1. That is, at threshold, either Display Formula = max_i(R_i,p+t − R_i,p) or Display Formula = max_i(T_i,p+t − T_i,p) is equal to unity. 

The decision variable is based on the channel showing the largest response difference between the pedestal alone and the pedestal-plus-target intervals in both the early and the late stages. This max rule is reasonable because a system can detect the target if the response of any mechanism within the system reaches the threshold (Green & Swets, 1966; Tyler & Chen, 2000). However, in practice, if the image contained enough information to allow a decision to be made by the early mechanism alone (D₁), we would not compute D₂, for reasons of parsimony. That is, D₂ was implemented in the model only in the conditions where it was impossible for D₁ to reach the criterion. 

The model parameters were optimized by the fminsearch function in Matlab, which applies the Nelder–Mead simplex direct-search algorithm (Lagarias, Reeds, Wright, & Wright, 1998), and by minimizing the squared difference between model predictions and data. 

Implementation

LM patterns with grating carrier and noise carrier

The LM patterns, as defined in Equation 1, are the weighted sum of a carrier pattern and an envelope. Thus, plugging Equation 1 into Equation 7, the excitation of the linear filter is  

 

where Se_j is a constant called the excitatory sensitivity of the mechanism, c_carrier is the contrast of the carrier, and c_env is the contrast of the envelope. The derivation of Equation 16 is shown in the Appendix. The inhibition of the linear output is the same as in Equation 16 but with inhibitory sensitivity (Si_env, Si_carrier) of the mechanism. In addition, we assume that the observer can see the difference between the pedestal-plus-target and the pedestal alone if the responses to these two intervals are sufficiently different in at least one channel. That is, we only need to consider the channel with the largest response difference between the two intervals. Since c_carrier is the same in both intervals, we only need to consider the mechanism that is most sensitive to the envelope, even though it has response to the carrier. In addition, since the modulation of the envelope is greatest at around the center of the image, the receptive field of this mechanism should be centered on the center of the image. Hence (x,y) = (0,0) in Equation 16. In practice, Se_carrier was fixed to 0 for the grating carrier and a free parameter for the noise carrier. The reason for this is that the peak spatial frequency of the grating carrier is two octaves apart from that of the envelope and that the orientation of carrier and envelope is orthogonal, and hence the target should produce little response in the mechanism responding to the envelope. On the other hand, the noise carrier is a very broadband stimulus, whose spectrum overlaps with that of the envelope and thus may produce a response in the envelope mechanism. Hence there were five parameters (Se_env, Si_env, p, q, and z) for fitting data with LM patterns with a sine-wave carrier, and seven parameters (Se_env, Si_env, Se_carrier, Si_carrier, p, q, and z) for LM patterns with a noise carrier. Based on the pilot fitting results, the value of Se_carrier is close to 0; therefore we fixed the value at 0 as well for both carrier types. The best-fit parameters for the grating carrier are shown in Table 1, and for the noise carrier in Table 2. The fits are shown in Figure 2. The model fit the LM-pattern results very well. For the LM pattern with sine-wave carrier, the root-mean-square error (RMSE) was 1.86 for CWC and 2.48 for PCH. This is similar to the mean standard error of measurement: 2.21 for CWC and 1.34 for PCH. For the LM pattern with noise carrier, the RMSE was 2.36 for CWC and 2.36 for PCH. This is at a reasonable range for the mean standard error of measurement: 1.52 for CWC and 1.37 for PCH. Both LM patterns showed that the values of p and q are significantly deviated from 1 and the value of p is larger than that of q, suggesting that the inhibitory pooling is weaker than the facilitation and a simple rectification cannot explain our results. In addition, the value of Se is larger than that of Si, also suggesting that inhibition is weaker than facilitation, which is consistent with previous findings for first-order pattern vision (Foley, 1994; Foley & Chen, 1999). 

Table 1

 

Fitted parameters for Experiment 1. Notes: *The value of the parameters is fixed.

	CWC	PCH
LM
Seenv*	100	100
Sienv	57.90	59.92
Secarrier*	0	0
Sicarrier*	0	0
p	2.50	2.31
q	2.08	1.93
z	12.97	9.40
SSE	55.70	89.23
# datum	30	36
MSE	1.86	2.48
SE	2.21	1.34
CM
Seside*	100	100
Siside	89.00	86.07
Secarrier*	0	0
Sicarrier	3.76	5.90
p	4.13	4.20
q	3.67	3.73
z	323.70	405.68
SSE	65.58	22.29
# datum	23	23
MSE	2.85	0.97
SE	1.50	1.24

Table 2

 

Fitted parameters for Experiment 2. Notes: *The value of the parameters is fixed.

	CWC	PCH
LM
Seenv*	100	100
Sienv	62.54	60.27
Secarrier*	0	0
Sicarrier	10.63	6.06
p	2.70	3.22
q	2.23	2.92
z	22.31	31.78
SSE	54.07	99.05
# datum	23	42
MSE	2.36	2.36
SE	1.52	1.37
CM
Seenv*	100	100
Sienv	145.38	58.46
R	0.13	0.10
k	1.25	1.60
π	2.85	2.63
θ	1.92	2.30
ζ	51.56	17.27
SSE	65.43	41.25
# datum	23	23
MSE	2.85	1.79
SE	1.33	1.32

CM patterns with grating carrier

The contrast-modulated (CM) pattern, as defined by Equation 4, is the modulation of the contrast of carrier information by a vertical Gabor function. Hence, the excitation of the first-stage linear filter for a second-order pattern is, as shown in Equation 9, the convolution of the CM stimuli CM(x, y) and a linear filter whose sensitivity profile f(x, y) is defined by a Gabor function and whose spatial frequency and orientation are the same as those of the carrier (see Appendix for the derivation). The result can be written as  

 

where Se_carrier and Se_side are constants which are related to the sensitivity to carrier and sideband images, respectively; c is the carrier contrast; and m is the envelope-modulation depth. For a CM pattern with a sine-wave carrier, it may be possible to find a mechanism whose Se_side is large enough that the difference in envelope modulation between the two intervals in a trial would allow the early decision variable D₁ to surpass the noise level. Hence, it is not necessary to have the late mechanism involved. As a result, the model for CM patterns with sine-wave carrier is the same as the model for LM patterns. The parameters are shown in Table 1 and the results are shown in Figure 2. The model fit the CM-pattern results well: The RMSE was 2.85 for CWC and 0.97 for PCH. This is a reasonable range compared to the mean standard error of measurement: 1.50 for CWC and 1.24 for PCH. The values of Se_carrier and Si_carrier are small, suggesting that the carrier itself did not contribute much to the response of the detecting mechanism. The model is well fit to the data, suggesting that perceiving a second-order pattern does not necessarily require a second-order mechanism. 

CM patterns with noise carrier

For CM patterns with binary white noise, the carrier changed between trials. Thus we calculated how noise contrast would influence the output of the linear filter. The simulation was done by randomizing the noise pattern for each trial and using the grating carrier as the filter. The simulation results (based on 500 trials) for each noise contrast show that the output from linear filters whose spatial frequency and bandwidth were the same as those of the grating carrier used in the experiment had a linear relationship with the noise contrast (Figure 4A), and the output did not change much with modulation depth (Figure 4B). Thus, we assumed the output from the first stage of the linear filter was  

 

in which de is a fixed value and c_noise is the noise-carrier contrast. In our experiment, two levels of carrier contrast were used, and their relative output was constant under different modulation depths, no matter what kind of nonlinear transform was adapted. Therefore, the output ratio between high and low carrier contrasts can be set to be a constant value, and thus  

 

in which R represents the output after the nonlinear transform and also the input to the second-stage linear filter. Thus we assumed the output for the second linear stage of the linear filter, Display Formula is  

 

in which R_j, j = l or h, is the excitatory input from the first stage of the process. We used seven parameters to fit the data (R_l, k, Se_env, Si_env, π, θ, and ζ). The RMSE was 2.85 for CWC and 1.79 for PCH. This is at the reasonable range of mean standard error of measurement: 1.33 for CWC and 1.32 for PCH. The fitted results showed that the values of π and θ are significantly deviated from 1 and the value of π is larger than that of θ, suggesting that the inhibitory pooling is weaker than the facilitation and a simple rectification between early and late linear filters cannot explain our results. 

Figure 4

 

The simulation results. The simulated output of the linear filter, whose spatial frequency and bandwidth are the same as those of the sine-wave carrier used in the experiment, is plotted against the noise contrast (A) and modulation depth at two carrier-contrast levels (B).

Discussion

The contamination of luminance artifacts in detecting second-order pattern

One might argue that our task for second-order pattern discrimination involved a luminance artifact, and this could be the reason why the CM results can be explained by the first stage of nonlinearity processing. However, this is not the case. First, in order to complete the task, it is necessary to discriminate the modulation depth between carrier + sideband 1 and carrier + sideband 2. Both combinations have the same contrast energy. Therefore, the second-order structure is needed to complete the task. Second, it is known that overall root-mean-square (RMS) contrast of the modulated carrier increases with modulation depth, even though the mean contrast does not. The changes follow the equation  

 

in which c is the carrier contrast and m is the modulation depth (Schofield & Georgeson, 1999). We measured the discrimination threshold for the noise carriers at −16 dB and −8dB and found the contrast needed to be raised to the value Δc + c to 1.41c and 1.22c for −16 and −8 dB, respectively. Thus a detectable RMS contrast by modulation depth m > 1 or 0.98 was needed for low and high carrier contrast, respectively, by setting C_r.m.s. to 1.41c and 1.22c in Equation 21. For the grating-based carrier, the discrimination threshold was also measured, and c > 1 and 0.86 were derived for low and high carrier contrast, respectively. Our experimental results did not use such high modulation depth; therefore, our results cannot be explained by RMS contrast changes. In conclusion, our experimental stimuli were not contaminated by a luminance artifact. 

Is the first stage of contrast-gain control enough to explain the TvC function for the second-order stimuli?

The TvC functions for grating-based LM and CM stimuli can be well described by a divisive-inhibition model. This indicates that the CM discrimination task might be achieved by off-orientation (sideband information) discrimination instead of second-order pattern discrimination, and suggests that a contrast-gain-control type of nonlinear process is involved in the pattern discrimination. Even though the contrast-gain control right after an early linear filter can explain the CM results, it does not mean the late linear filter is not necessary. This is because adding the late linear filter is equal to multiplying a constant value by the output of gain-control processing, and this would not influence the shape of the response function. 

Comparison of TvC functions between the first- and second-order stimuli

Grating-based TvC functions

The influence of the grating carrier was different for LM and CM stimuli. The contrast of the carrier did not change performance in LM target discrimination but did change performance in CM discrimination. The carrier information in the LM pattern can be viewed as a secondary mask for the TvC function, because the carrier information we used was orthogonal and two octaves higher than the envelope's spatial frequency. Therefore, the influence of the carrier was small in LM. Again, the difference in performance between LM and CM demonstrates that the carrier plays a different role in LM and CM patterns. In our model, the output of the linear filters for CM patterns contains two pieces of information: carrier and sideband. The influential factors from the sideband (Se_side) were larger than those from the carrier (Se_carrier), indicating the importance of sideband information. Our fitting results were also consistent with image-classification findings, which have shown that human observers may detect contrast modulation of a sinusoidal carrier by using sideband information (Manahilov, Simpson, & Calvert, 2005). 

Noise-based TvC functions

The TvC functions for LM and CM stimuli were similar and can be explained by divisive inhibition, which occurs at different stages of processing. Both LM and CM stimuli were influenced by noise contrast, but in opposite ways: An increase in carrier contrast decreased the detectability of the LM target but increased the detectability of the CM target. This is consistent with previous findings (Schofield & Georgeson, 1999). 

Conclusions

Our results showed a typical TvC function for both the first-order and second-order stimuli, and our proposed divisive-inhibition model was well able to account for the TvC function for both the first- and second-order stimuli with grating and noise carriers. Combining noise-based and grating-based CM-pattern results, we found that (a) a divisive inhibition is needed after the second linear filter to describe the CM pattern with noise carrier; (b) the exponent for the excitation signal was significantly different from that for the inhibition signal in the late mechanism, accounting for the result for the CM pattern with noise carrier; and (c) the first stage of divisive inhibition is necessary to account for the CM pattern with grating carrier. Thus, rectification between early and late linear filters alone cannot explain the discrimination threshold for the second-order stimuli, and divisive inhibition is necessary to explain the performance. Furthermore, the divisive inhibition is also determined by the first-order information in the stimuli. 

Acknowledgments

This study was supported by 96-2413-H-002-006-MY3 and NSC 99-2410-H-002-081-MY3 to CCC and by NSC-101-2401-H-006-003-MY2 to PCH. A preliminary account of this study, which included the data from the grating-carrier condition, was presented in the 2011 Electronic Imaging meeting and appeared in the conference proceedings as Huang & Chen, 2009. 

Commercial relationships: none. 

Corresponding author: Chien-Chung Chen. 

Email: [e-mail]. 

Address: Department of Psychology, National Taiwan University, Taipei, Taiwan. 

Appendix

Summary of symbols

I₀: mean luminance 

c: contrast of the carrier 

m: contrast of the LM stimuli or modulation depth of the CM stimuli 

E: excitatory output for early linear filter 

I: inhibitory output for early linear filter 

p: exponent of excitation for early linear filter 

q: exponent of inhibition for early linear filter 

z: semisaturation constant for early contrast- gain-control processing 

R: early response 

H: excitation of a later linear filter 

K: inhibition of a later linear filter 

π: exponent of excitation for late linear filter 

θ: exponent of inhibition for late linear filter 

ζ: semisaturation constant for late contrast-gain control 

T: late response 

D₁: decision variable for the early response 

D₂: decision variable for the late response 

Se: excitatory sensitivity parameters of spatial filters 

Si: inhibitory sensitivity parameters of spatial filters 

The equations for modeling the TvC function for LM and CM patterns with periodical carriers

The carrier-only stimulus is used to illustrate the equations used to model our TvC function. The carrier is given by  

 

in which c denotes the carrier contrast. The filter is defined by a Gabor function with the same spatial frequency and orientation:  

 

 

The response of the filter can be given as follows:  

 

in which Se denotes a constant value related to σ and f_x values. Because the LM stimuli were produced by adding two orthogonal gratings, the response of the filters could be given as  

 

 

The CM stimulus used is described as  

 

in which c denotes the carrier contrast and m denotes the envelope contrast. The first stage of the linear filter is given as follows:  

 

 

Then the response of the filter can be re-written as  

 

 

We calculate the carrier and sideband parts separately. The carrier part is the same as mentioned previously:  

 

 

The sideband part can be derived as follows:  

 

 

Therefore, the first-stage output of the linear filter can be written as  

 

 

According to Equation A10, the response for the discrimination task can be viewed as discrimination of the sideband information. Therefore, if the horizontal grating can be used as a cue to do the discrimination task, this could help us to find the nonlinear process between the early and late linear filters.