Equality judgments cannot distinguish between attention effects on appearance and criterion: A reply to Schneider (2011)

Abstract

Whether attention modulates the appearance of stimulus features is debated. Whereas many previous studies using a comparative judgment have found evidence for such an effect, two recent studies using an equality judgment have not. Critically, these studies have relied on the assumption that the equality paradigm yields bias-free PSE estimates and is as sensitive as the comparative judgment, without testing these assumptions. Anton-Erxleben, Abrams, and Carrasco (2010) compared comparative judgments and equality judgments with and without the manipulation of attention. They demonstrated that the equality paradigm is less sensitive than the comparative judgment and also bias-prone. Furthermore, they reported an effect of attention on the PSE using both paradigms. Schneider (2011) questions the validity of the latter finding, stating that the data in the equality experiment are corrupted because of skew in the response distributions. Notably, this argument supports the original conclusion by Anton-Erxleben et al.: that the equality paradigm is bias-prone. Additionally, the necessary analyses to show that the attention effect observed in Anton-Erxleben et al. was due to skew in the data were not conducted. Here, we provide these analyses and show that although the equality judgment is bias-prone, the effects we observe are consistent with an increase of apparent contrast by attention.

Introduction

Whether attention modulates the appearance of several basic stimulus features has been discussed for more than a century and is still debated. Whereas many studies using a comparative judgment to assess the effect of attention on appearance have found evidence for such an effect (Abrams, Barbot, & Carrasco, 2010; Anton-Erxleben, Henrich, & Treue, 2007; Carrasco, Fuller, & Ling, 2008; Carrasco, Ling, & Read, 2004; Fuller & Carrasco, 2006; Fuller, Park, & Carrasco, 2009; Fuller, Rodriguez, & Carrasco, 2008; Gobell & Carrasco, 2005; Hsieh, Caplovitz, & Tse, 2005; Ling & Carrasco, 2007; Liu, Abrams, & Carrasco, 2009; Liu, Fuller, & Carrasco, 2006; Montagna & Carrasco, 2006; Turatto, Vescovi, & Valsecchi, 2007), two recent studies using an equality judgment have not (Schneider & Komlos, 2008; Valsecchi, Vescovi, & Turatto, 2010). The authors of these studies have assumed that equality judgments are superior to comparative judgments because supposedly they are immune to decision biases and have proposed that previous results can be explained by such a bias. A bias could, in principle, originate either from a certain response strategy generated by observers' expectations about the experiment (demand characteristic) or from specific criterion settings. Numerous studies have experimentally addressed these potential concerns with regard to the comparative judgment (see Discussion in Anton-Erxleben, Abrams, & Carrasco, 2010, pp. 15–16). Schneider and Komlos' (2008) model for comparative and equality judgments attribute the potential bias to a change in response criteria. Anton-Erxleben et al. (2010) were the first to test for biases and sensitivity to shifts in perceived contrast of both the comparative and equality judgments, as used in those studies, with and without attention. That study revealed that equality judgments are bias-prone in a criterion settings sense and are also less sensitive than comparative judgments. Nevertheless, with sufficient statistical power, results from both paradigms showed an increase in apparent contrast by attention. 

Schneider (2011) questions this conclusion because the response distributions in the equality judgment are asymmetric (as discussed in Anton-Erxleben et al., 2010, pp. 8–9 and 11). He reanalyzed Anton-Erxleben et al.'s (2010) data with three different models, two of them accounting for asymmetry in the data. He argues that the observed shift of the PSE in Anton-Erxleben et al. was an artifact due to fitting a symmetric model to asymmetric data and suggests that the effect on the PSE disappears when skew is accounted for. 

This argument misses the point made by Anton-Erxleben et al. (2010). That study aimed at investigating and comparing two psychophysical procedures, the comparative and equality judgments. The main conclusion was that the equality judgment is bias-prone and less sensitive than the comparative judgment. That conclusion was based on an experiment without any manipulation of attention (Experiment 1) and would have been the same had there not been an effect of attention with the equality judgment (Experiment 3). Because in the equality judgment criteria can be asymmetric, Schneider and Komlos' (2008) assumption that in the equality paradigm criterion shifts and PSE shifts are orthogonal is no longer justified. Therefore, the equality judgment cannot be used as a control for bias, and the absence of an effect with the equality paradigm (Schneider & Komlos, 2008; Valsecchi et al., 2010) cannot be used as evidence of bias in the comparative judgment. Schneider's (2011) hypothesis that the effect observed in Anton-Erxleben et al.'s Experiment 3 was due to asymmetries over the contrast range is consistent with this argument. Moreover, it is important to note that Schneider did not perform the analyses necessary to support his conclusion that there is no effect of attention on the PSE when a skewed model is used to fit the data. 

Here, we further discuss how asymmetries in the response distributions render the equality judgment bias-prone. In addition, we analyze the results of Experiment 3 with a skewed model and show that attention still shifts the peak of the response distribution, consistent with an effect of attention on appearance. In addition, we respond to a number of specific points raised by Schneider (2011). 

Bias in the equality judgment

Two previous studies investigating the effect of attention on apparent contrast using an equality judgment (Schneider & Komlos, 2008; Valsecchi et al., 2010) have assumed that in the equality task, decision criteria and changes in appearance are orthogonal. However, as Anton-Erxleben et al. (2010), pp. 3, 13–14) explained in detail, this assumption only holds when criteria are symmetric, which is often not the case (Petrov, 2009). Therefore, whereas it is true that control experiments are necessary to distinguish an effect on PSE from an effect on criterion in the comparative judgment, it is not beneficial to replace the one criterion in the comparative judgment with two criteria in the equality judgment. Such control experiments have been conducted in previous studies and have supported the conclusion that an attention effect on the PSE exists (Abrams et al., 2010; Anton-Erxleben et al., 2007; Carrasco et al., 2008, 2004; Fuller & Carrasco, 2006; Fuller et al., 2009, 2008; Gobell & Carrasco, 2005; Ling & Carrasco, 2007; Liu et al., 2009, 2006; Montagna & Carrasco, 2006; Turatto et al., 2007). The logic behind and findings of these control experiments was discussed in Anton-Erxleben et al. (pp. 2 and 15 and elsewhere, e.g., Carrasco, 2009). 

In the present data set (taken from Anton-Erxleben et al., 2010, Experiment 3), lower contrasts were more likely than higher contrasts to be judged as equal to the standard, creating slightly skewed distributions. Schneider (2011) argues that observers might guess more at low contrasts. However, it is important to note that with the comparative judgment, with the exact same stimuli and only a different instruction, the exact same observers did not exhibit such an asymmetry (Experiment 4). Furthermore, Valsecchi et al. (2010) found similar asymmetries studying appearance of motion speed using the same type of equality judgment, even though all of their stimuli were clearly visible at 60% contrast. Therefore, the asymmetry is unlikely to be due to reduced visibility of the low-contrast stimuli. Instead, it is likely due to an asymmetry in the two criteria, which is indistinguishable from a differential guessing rate: The idea that observers guess more and observers adopt a more liberal criterion for responding “same” are logically equivalent. Thus, there is no evidence for increased guessing at low contrast. 

Whatever the source of the asymmetry, Schneider (2011) suggests that it causes a shift of the low-contrast flank of the psychometric function with the attentional cue and thus explains the effect, without the need to assume an effect of attention on perceived contrast. It is not clear why either a change in criterion or an increased guessing rate should vary with the attentional cue in a way that would create a systematic shift of the psychometric function. Indeed, Anton-Erxleben et al. (2010) presented evidence that the asymmetry exists with no cue to differentiate the two stimuli (Experiment 1) and therefore cannot be attributed to the cue alone. In the next section, we investigate if the attentional effect on the PSE still exists when the data are fit with a model that takes skew into account. 

The attention effect is still evident with the skewed model

Schneider (2011) fits three models to the data from Experiment 3 of Anton-Erxleben et al. (2010): the response model used in Schneider and Komlos (2008), a skewed Gaussian model, and a third model that adds an asymmetric guessing rate to his original response model. 

He reports a correlation between the skew of the distribution and the attentional effect on perceived contrast (derived from fitting the data with a symmetric function). However, such a correlation is not sufficient to conclude that attention does not affect appearance. The correlation is not informative as to the effect of attention on PSE. Instead, it is essential to test if the PSE parameter derived from these fits changes among the attentional conditions (standard cued, neutral, and test cued) before one can conclude that the skew is the sole source of the effect. 

When fitting the third model (equality judgment with asymmetric guessing rate), only the criteria and the asymptote are allowed to vary among conditions but not the location (α) parameter. Because keeping this parameter fixed biases the model against revealing shifts of the peak of the response distribution, this analysis does not allow the conclusion that attention does not affect the PSE. Only if the location parameter were free and were empirically shown to not vary among attentional conditions could one conclude that there is indeed no change in appearance with attention. It is not sufficient to just state that the model with the asymptote fits the data better than a symmetric model and therefore conclude that the location of the PSE does not change. 

The guessing rate model is problematic because the asymmetric guessing rate is based on speculation. There is no empirical or theoretical evidence for differential guessing at low and high contrasts. An asymmetric guessing rate is indistinguishable from asymmetric criteria in a signal detection framework. In the original model used by Schneider and Komlos (2008), however, asymmetric criteria cannot be implemented because the α parameter would become degenerate. Furthermore, the guessing rate implies a high-threshold model, which is inconsistent with the principles of signal detection theory (e.g., Wickens, 2002, pp. 131–139). Therefore, here we focus on the skewed Gaussian model, which accounts for the asymmetry in the distributions while making fewer theoretical assumptions than the guessing rate model and is consistent with the signal detection theory. 

In a one-way repeated measures ANOVA, we test if the peak of the function shifts with the attentional cue, using the values (c _max) reported by Schneider (2011) for the fit with the skewed Gaussian. P-values are Greenhouse–Geisser corrected when appropriate. We find a significant effect of the cue (p = 0.043, n = 9). Pairwise t-tests reveal that there is a significant difference between the test-cued and standard-cued conditions (p = 0.0215, one-tailed), between test-cued and neutral conditions (p = 0.0395, one-tailed), and between the neutral and standard-cued conditions (p = 0.018, one-tailed). Figure 1 shows the relation of c _max in the test- and standard-cued conditions to c _max in the neutral condition for all observers. The cue also significantly modulates the α parameter (p = 0.037, one-way repeated measures ANOVA, n = 9), and the scaling parameter h (p = 0.006) but not the standard deviation σ or the skew γ (p > 0.1). Thus, using a skewed model, we arrive at the same conclusion as in Anton-Erxleben et al. (2010): that attention shifts the PSE consistent with an increase in apparent contrast. Whereas it is possible that biases exist with any psychophysical procedure, there is no evidence that a bias underlies the reported attentional effects. 

Figure 1

 

PSE (c _max) in the test-cued (*) and standard-cued (Δ) conditions as a function of the PSE in the neutral condition for each observer in the equality paradigm (Experiment 3 in Anton-Erxleben et al., 2010), derived from fitting with Schneider's (2010) skewed Gaussian model.

Specific points

Here, we include our reply to individual points, following the order in which they appeared in Schneider (2011). 

(1) Schneider (2011, p. 8) describes that in Experiment 3 of Anton-Erxleben et al. (2010), the response distributions for the test-cued and standard-cued conditions completely overlap at high contrasts and differ only at low contrasts. However, it is misleading to judge the effect of attention just by looking at the overlap of the curves, because in the equality paradigm, the overlap is not only determined by a shift in PSE but also by the standard deviation parameter and the amplitude (which are determined by observers' criteria). Figure 2 illustrates this point: In Figure 2A, two hypothetical psychometric functions with a PSE shift of ∼5% contrast and no changes in amplitude or standard deviation are shown. In the other figures, either the amplitude (Figures 2B and 2C), the standard deviation (Figures 2D and 2E), or both (Figures 2F and 2G) change. Depending on which of the two curves has higher standard deviation and/or amplitude, the overlap of the curves is in the low- or high-contrast range. Note that the PSE shift, which is the measure for a change in apparent contrast, is the same in each of these hypothetical examples. In the data of Experiment 3 (Figure 4A in Anton-Erxleben et al., 2010), this point is illustrated by the neutral condition, which has a higher amplitude than the other two conditions and overlaps with the test-cued condition only at lower contrasts. 

Figure 2

 

Hypothetical effects of attention on the distribution of “same” responses in an equality task with different changes in amplitude and standard deviation but the same shift in PSE between test-cued (blue) and standard-cued (red) conditions. (A) This figure shows two hypothetical psychometric functions with a PSE shift of ∼5% contrast and no changes in amplitude or standard deviation. In the other figures, either (B, C) the amplitude, (D, E) the standard deviation, or (F, G) both change.

(2) Schneider (2011, p. 8) states that the test contrast most often selected as equal to the standard is actually the standard and states that this is the case for all conditions and observers. A close look at the individual data reveals that this is in fact not the case (see his Supplementary Figure S1). 

(3) Schneider (2011, p. 12) suggests that the differences in skew between Schneider and Komlos' (2008) and Anton-Erxleben et al.'s (2010) data might be due to differences in the quality of observers and that observers “with the smallest skew exhibited veridical contrast perception, unaffected by the attentional cue.” However, we do not find a correlation between skew γ and c _max (Spearman's rho = 0.184, 0.017, −0.4 for the neutral, test-cued, and standard-cued conditions, respectively, all p > 0.1). Furthermore, we do not find a significant effect of the attentional cue on the skew, whereas the cue significantly affects c _max (see above). Therefore, the statement that an attentional effect on contrast perception only exists when the skew is large is not supported by the data. 

(4) Schneider (2011, p. 12) argues that Anton-Erxleben et al.'s (2010) suggestion that differences in statistical power could account for the lack of an attention effect in Schneider and Komlos (2008) “… is dubious because adding more points to the psychometric function only weakly improves the estimation of the PSE, and the error in the group PSE is determined primarily by the between-subjects error rather than the error in fitting the psychometric function to each individual subject's data. The estimations by the equality judgments in Schneider and Komlos were almost exactly zero and there is no reason to expect under-sampled data to be biased.” This statement is misleading for several reasons: (A) Anton-Erxleben et al. (2010) used a within-subjects design, so that the error in fitting the psychometric function to each individual subject's data is important. (B) Whereas there are diminishing returns by oversampling the psychometric function, there is no reason to assume that the difference in number of points and number of samples between the two studies is in the range in which adding points/samples does not further improve the fit. This assertion is speculative; it has no basis in the data, especially as Schneider and Komlos did not report goodness of fit. 

(5) Schneider (2011, p. 12) states that “… even if the systematic underestimations of the PSE reported in Anton-Erxleben et al. (2010) were not simply due to low-contrast noise, as suggested by the present study, and therefore might have actually occurred in Schneider and Komlos (2008), it would be hard to imagine that such underestimations, which were constant over contrast in Anton-Erxleben et al. (2010), would have exactly canceled out the effects of the cue that varied with contrast in Schneider and Komlos (2008), being larger for the lower than the higher contrasts.” Given that Schneider and Komlos did not measure a baseline, it is impossible to know if their data show a similar consistent underestimation of the baseline PSE with the equality experiment as reported by Anton-Erxleben et al. Note that Valsecchi et al. (2010) also found that in an experiment without attention (Experiment 4) the PSE deviates from the POE. Thus, it is unwarranted to assume that the no attention condition yields a PSE that matches the POE. The variation with standard contrast that Schneider and Komlos (2008) report is small compared to the noise, so the underestimation of the neutral PSE could have canceled the attentional effect, but without a baseline, this is a matter of speculation. As discussed by Anton-Erxleben et al. (pp. 4 and 15), without measuring at least two conditions, one with and one without attention, Schneider and Komlos' data are not interpretable and the magnitude of the attention effect cannot be estimated. 

(6) With respect to independence of parameter estimates in the equality judgment, Schneider (2011, p. 14) states that it is “not surprising that this amplitude is correlated with σ. Anton-Erxleben et al. (2010) are thus incorrect in stating that the independence of the equality judgment parameters is not justified.” This statement mischaracterizes Anton-Erxleben et al.'s discussion of parameter dependence. It is obviously not surprising that amplitude and width are correlated. Our discussion of parameter dependence is important because of the less intuitive observation that the PSE is not independent from these other two parameters. Whereas this effect was marginal in our study, it was statistically significant in Valsecchi et al. (2010). As discussed by Anton-Erxleben et al. (p. 14), this observation is the reason why the assumption of parameter independence is not justified. 

(7) Schneider (2011, p. 13) states that there is no data or theoretical reason to think that attention should alter appearance. In particular, he (A) states that contrast gain is only one of the possible physiological effects of attention and (B) cites a recent neurophysiological study reporting that attention and contrast are encoded separately in the cortex. 

(A) This argument implies, first, that a change in contrast appearance could only occur via a contrast gain mechanism, which is an unwarranted assumption. Second, it implies that of the different possible physiological effects, the one underlying the psychophysical task employed by Anton-Erxleben et al. (2010) and other studies (Carrasco et al., 2008, 2004; Fuller et al., 2009, 2008; Hsieh et al., 2005; Ling & Carrasco, 2007; Liu et al., 2009; Schneider & Komlos, 2008) is not contrast gain. This assumption is equally unwarranted. 

Schneider (2011) argues that whereas earlier neurophysiological studies supported a contrast gain mechanism, the normalization model of attention (Reynolds & Heeger, 2009) shows that contrast gain is only one possible mechanism by which attention affects neuronal firing rates. This statement does not capture the core idea of the model, i.e., that both contrast gain and response gain are variants of the same computational mechanism (Reynolds & Heeger, 2009). Indeed, this model was developed to reconcile seemingly disparate studies supporting contrast gain and response gain modulation (Martinez-Trujillo & Treue, 2002; Maunsell & McAdams, 1999; Reynolds, Pasternak, & Desimone, 2000; Treue & Martinez Trujillo, 1999) and has been supported by human psychophysical data (Herrmann, Montaser-Kouhsari, Carrasco, & Heeger, 2010). 

Whereas it has been argued that a change in contrast appearance can be linked to contrast gain (Reynolds & Chelazzi, 2004; Treue, 2004), the three neurophysiological models of attention that are most widely accepted—response gain, contrast gain, and additivity—employ an increase of neuronal firing rates (e.g., Reynolds & Heeger, 2009; Treue, 2001). The difference among these models' predictions is in the magnitude of this effect as a function of stimulus contrast. Furthermore, neuronal tuning curves are similarly scaled by increasing stimulus contrast (e.g., Albrecht & Hamilton, 1982; Geisler & Albrecht, 1997; Holub & Morton-Gibson, 1981; Sclar & Freeman, 1982; Skottun, Bradley, Sclar, Ohzawa, & Freeman, 1987) and by allocating attention to the stimulus location (Maunsell & McAdams, 1999, 2001; Treue & Martinez Trujillo, 1999). This similarity between the effects of contrast and attention is orthogonal to the question of whether attention acts via contrast or response gain. Thus, it is not necessary to assume contrast gain in order to predict an attentional enhancement of apparent contrast. Once neuronal firing rates are increased, an enhancement due to attention and an enhancement due to higher stimulus contrast are indistinguishable, unless some kind of corollary signal is assumed. 

Contrary to Schneider's (2011) claim that there is no reason to expect an attentional modulation of contrast appearance, the finding that attention alters appearance fits very well with previous neurophysiological studies indicating that attention increases contrast sensitivity (e.g., Carrasco, 2011; Luck, 2004; Reynolds & Chelazzi, 2004; Treue, 2004). Consistent with these findings, some recent psychophysical and neuroimaging studies support an enhancement of contrast responses (Buracas & Boynton, 2007; Li, Lu, Tjan, Dosher, & Chu, 2008; Murray, 2008). Note that some psychophysical studies support an alternative view in which attention merely reduces noise without enhancing the signal (Foley & Schwartz, 1998; Palmer, Verghese, & Pavel, 2000; Shaw, 1980; Smith, 2000). 

(B) Pooresmaeili, Poort, Thiele, and Roelfsema (2010) reported that in area V1, effects of attention and contrast on the neuronal response are additive, and it is possible to decode both an attention and a contrast signal from the neuronal response. They conclude that a code for contrast independent of attention would permit veridical representation of contrast, so that attention need not alter perceived contrast. This conclusion does not preclude the possibility that attention modulates perceived contrast. Note that their results are based on responses in V1, an early stage in visual processing, whereas perception of stimulus contrast in a behavioral task is likely not based solely on V1. 

(8) Schneider (2011, p. 13) argues that an effect of attention on contrast appearance would not be desirable. There are two problems with this statement. 

(A) The expected benefit of an effect does not determine whether or not it occurs. Such an argument is not scientific. As examples for “undesirable” effects that nevertheless exist, a number of visual illusions interfere with “veridical” perception (e.g., Adelson, 2000; Backus & Oruc, 2005; Chubb, Sperling, & Solomon, 1989; Cornsweet, 1970; Gilchrist, 1988; Hermann, 1870). 

(B) Attention can also have “undesirable” effects. For example, it can impair visual performance (Hein, Rolke, & Ulrich, 2006; Ling & Carrasco, 2006; Yeshurun & Carrasco, 1998). 

(12) Schneider's (2011) statement that a more salient stimulus in his Figure 8 does not look higher in contrast is uninformative about an effect of attention on perceived contrast. 

Conclusion

In summary, we have demonstrated that the equality paradigm is bias-prone. Given that the original justification for using the equality judgment was that it is immune to criterion shifts (Schneider & Komlos, 2008), there is no reason to accept it as a control experiment for bias in the comparative judgment. The equality paradigm is not superior to the comparative judgment: Rather, by replacing one criterion with two that can be asymmetric, it becomes prone to biases that can affect the location of the PSE. Whereas the comparative judgment is potentially bias-prone as well, control experiments have been used successfully to distinguish an effect of attention on the PSE from an effect on the criterion. Furthermore, as empirically demonstrated (Anton-Erxleben et al., 2010), the equality judgment is noisier and therefore less sensitive than the comparative judgment. Nevertheless, setting the caveats aside, even with the equality paradigm, we find an effect of attention on the PSE that is consistent with an increase in apparent contrast, whether or not we fit a model that takes into account skew in the data. Thus, the new analyses presented in this reply and by Schneider (2011) do not change any of our original conclusions. 

Acknowledgments

We thank two anonymous reviewers for helpful comments. This research was supported by a Feodor Lynen Research Fellowship, Alexander von Humboldt Foundation, and NIH 1F32EY021420-01 to Katharina Anton-Erxleben and NIH EY016200 to Marisa Carrasco. 

Commercial relationships: none. 

Corresponding author: Marisa Carrasco. 

Email: [e-mail]. 

Address: Center for Neural Science, New York University, 6 Washington Place, 8th Floor, New York, NY 10003, USA.