Classification images and bubbles images in the generalized linear model

Abstract

Abstract

Abstract:

Abstract  Classification images and bubbles images are psychophysical tools that use stimulus noise to investigate what features people use to make perceptual decisions. Previous work has shown that classification images can be estimated using the generalized linear model (GLM), and here I show that this is true for bubbles images as well. Expressing the two approaches in terms of a single statistical model clarifies their relationship to one another, makes it possible to measure classification images and bubbles images simultaneously, and allows improvements developed for one method to be used with the other.

Introduction

Classification images and bubbles images are psychophysical tools that recover some of the features observers use to identify images (Ahumada, 1996, 2002; Gosselin & Schyns, 2001; Murray, 2011). These two methods were invented independently, and despite their similarities, they have developed along independent paths. Previous work has shown that classification images can be calculated using the generalized linear model (GLM), and here I show that this is true for bubbles images as well (McCullagh & Nelder, 1989; Abbey & Eckstein, 2001; Knoblauch & Maloney, 2008; Dobson & Barnett, 2008). Expressing the two approaches in terms of a single statistical model clarifies their relationship to one another, makes it possible to measure classification images and bubbles images simultaneously, and allows improvements developed for one method to also be used with the other. 

In broad terms, the classification image and bubbles methods are similar: they both introduce random fluctuations into stimuli and measure the influence of these fluctuations on observers' responses. One important difference is that the most common versions of these methods require different kinds of noise. The theory of classification images has mostly been developed assuming additive Gaussian noise (Ahumada, 2002; Murray, Bennett, & Sekuler, 2002), whereas current methods for bubbles images require sparse multiplicative noise (Gosselin & Schyns, 2001). 

Classification images are usually calculated using weighted sums of noise fields (Ahumada, 1996, 2002), but they can also be estimated using the GLM (Knoblauch & Maloney, 2008). The GLM approach has the advantage that it does not make strong assumptions about the stimulus noise. Consider an experiment where the stimulus g is a signal, s₁ or s₂, shown in an additive noise field n and the observer makes an identification response r = 1 or r = 2. Using the GLM we can model the influence of the noise fields on the observer's responses as follows. 

 

 

 

Here Φ is the standard normal cumulative distribution function. ₁ is a vector of regression coefficients that encodes the classification image for s₁ trials and β_1,0 is a constant that encodes the response bias on s₁ trials. ₂ and β_2,0 encode the corresponding quantities for s₂ trials. We represent g, s_i, n, and β as column vectors, and we use ^T to denote matrix transposition, so x^Ty is the dot product of x and y. 

Equations (1) and (2) are GLM models of the observer's responses, and many statistical software packages have routines for making maximum-likelihood estimates of the regression coefficients β_i. The GLM does not make strong assumptions about the statistical distribution of the stimulus noise, so it can be used to measure classification images using noise fields from non-Gaussian distributions (Abbey & Eckstein, 2001) and even from the natural variations in some classes of stimuli, such as faces (Macke & Wichmann, 2010). 

The template matcher, a longstanding model of visual decisionmaking (Peterson, Birdsall, & Fox, 1954; Green & Swets, 1974), is an instance of the GLM. A template matcher performs a two-alternative identification task by taking the dot product of the stimulus g with a template w, adding an internal noise z that has standard deviation σ_z and making a response according to whether the resulting decision variable exceeds a criterion a: 

 

In a classification image experiment with noisy stimuli g = s_i + n, the template matcher's response probabilities are: 

 

 

 

 

 

 

 

Comparing Equations (1) and (5) shows that for a template matcher, the GLM's regression weights are β₁ = w/σ_z and Equations (2) and (7) show that β₂ = w/σ_z as well. Thus the GLM recovers the template w up to a scale factor. 

We can also calculate bubbles images using the GLM. In a bubbles experiment the signal s₁ or s₂ is modulated by a multiplicative noise field n, so the stimulus is s₁ ∘ n or s₂ ∘ n, where ∘ is the componentwise (Hadamard) product, (x ∘ y)_i = x_iy_i. The bubbles method assumes that each stimulus region contributes to some degree to an observer's correct responses. Each signal has an associated map of potent information (Gosselin & Schyns, 2002). The potent information map (i.e., the bubbles image) answers the following question: If we modulate the signal intensity at some location, how does this affect the probability of a correct response? This suggests that to calculate a bubbles image using the GLM, we should regress the correctness of the observer's responses against the multiplicative noise that modulates the stimulus: 

 

 

 

The GLM bubbles images are then the estimated regression coefficients β₁ and β₂. (Note that Equations (5) and (7) give the probabilities of response r = 2, whereas Equations (8) and (9) give the probabilities of correct responses, r = 1 or r = 2, depending on the signal.) 

Is this a plausible reformulation of the original bubbles method? For the template matcher in Equation (3), in a bubbles experiment the probability of a correct response is 

 

 

 

 

 

 

 

 

 

Comparing Equations (8) and (12) shows that for a template matcher, the GLM's regression weights are β₁ = −s₁ ∘ w/σ_z and Equations (9) and (14) show that β₂ = s₂ ∘ w/σ_z. For a template matcher, the conventional bubbles method produces a blurred, affinely transformed estimate of −s₁ ∘ w on trials where signal s₁ is shown, and of s₂ ∘ w on trials where s₂ is shown (Murray & Gold, 2004). Of course we can also blur and affinely transform the estimates from the GLM if we wish to. Thus for a template matcher, the case where the bubbles method is understood best and the only case where the expected value of a bubbles image is known, the suggested GLM approach agrees with the original bubbles method. 

Simultaneous classification images and bubbles images

The GLM methods outlined above require additive noise for classification images and multiplicative noise for bubbles images. This is less of a constraint than it may seem because stimulus noise can be described as either additive or multiplicative if we choose its distribution accordingly. A signal times multiplicative noise, s₁ ∘ n_m, can also be described as a signal plus additive noise, s₁ + n_a, if we set n_a = s₁ ∘ (n_m − 1). Similarly, a signal plus additive noise s₁ + n_a can be described as a signal times multiplicative noise, s₁ ∘ n_m, if we set n_m = (s₁ + n_a)/s₁, where / is componentwise division. (This requires s₁ ≠ 0, a detail we return to later.) Of course these transformations change the noise distribution, e.g., homogeneous multiplicative noise is transformed into nonhomogeneous additive noise and zero-mean additive noise is transformed into multiplicative noise with a mean of 1. However, the GLM does not require homogeneous noise, or noise with a particular mean value, so we can simply describe stimulus noise as additive to calculate classification images and as multiplicative to calculate bubbles images. This approach amounts to measuring the influence of a transformation of the stimulus pixels on the observer's responses, as has sometimes been done in classification image studies (e.g., Abbey & Eckstein, 2006), rather than the stimulus pixels themselves. 

To illustrate these methods we ran two simulations, one using white Gaussian noise and one using bubbles noise, and in both simulations we used the GLM to calculate classification images and bubbles images. 

Methods

The signals were 20 × 20 pixel vernier patterns (Figure 1) with a lower line at the same location in both signals and an upper line shifted one pixel to the left or right. In the Gaussian noise simulation the stimulus was a randomly chosen signal, shown at 15% Weber contrast in additive white Gaussian noise with zero mean and a pixelwise standard deviation of 25% contrast. In the bubbles noise simulation the stimulus was a randomly chosen signal shown at 15% contrast, multiplied pointwise by ten randomly placed Gaussian bubbles with standard deviations of one pixel. In both simulations the observer made unbiased responses by taking the dot product of a suboptimal template (Figure 1) with the stimulus, adding univariate internal Gaussian noise, and responding “right” if the resulting decision variable was greater than zero and “left” otherwise. This template resembles a typical classification image for human observers in a vernier discrimination task (Beard & Ahumada, 1998). We scaled the template so that its pixelwise sum-of-squares equaled 1.0. The standard deviation of the internal noise was chosen so that the observer gave 72% correct responses; this required a standard deviation of 25% in the Gaussian noise simulation and 3% in the bubbles noise simulation. In both cases this produced an internal-to-external noise ratio of approximately 1 (Burgess & Colborne, 1988). Each simulation ran for 5,000 trials. GLM fits were made using the glmfit function in the MATLAB Statistics Toolbox, with a probit (i.e., inverse cumulative normal) link function and a binomial response distribution (The MathWorks, Natick, MA). The MATLAB code for these simulations is available online as supporting information. 

Figure 1

 

Signals and model observer's template in the vernier discrimination task.

Results

Figure 2 shows classification images and bubbles images calculated from Gaussian noise. The first row shows classification images for left-signal and right-signal trials, calculated using the conventional weighted sum method (Ahumada, 1996, 2002). In all classification images shown in this article, white pixels indicate regions where positive-contrast noise makes the observer more likely to respond “right” (i.e., signal 2). The second row shows classification images calculated using the GLM by regressing the observer's responses, coded as left = 0 and right = 1, against the Gaussian noise. As expected, both these approaches recover the observer's template (Figure 1). In a bubbles experiment where the noise bubbles have standard deviation σ_b, the bubbles image is effectively blurred by a Gaussian kernel of standard deviation $\sqrt{2}{\sigma }_b$ (Murray & Gold, 2004). We convolved the classification images in this simulation with a Gaussian kernel of standard deviation $\sqrt{2}$ pixels in order to make them comparable to the results of the bubbles simulation reported below, where the bubbles had a standard deviation of 1 pixel. 

Figure 2

 

Classification images and bubbles images from Gaussian noise.

The third row of Figure 2 shows bubbles images calculated using the GLM by regressing the correctness of the observer's responses, coded as incorrect = 0 and correct = 1, against the equivalent multiplicative noise. That is, we took the Gaussian noise fields n_a, redescribed them as multiplicative noises fields n_m = (s_i + n_a)/s_i (using s_i = s₁ or s_i = s₂ as appropriate on individual trials), and regressed the correctness of the observer's responses against these multiplicative noise fields. We ignored stimulus locations where there was no signal (s_i = 0), since at these locations additive noise cannot be redescribed as multiplicative noise; this is not problematic, because bubbles images are empty at no-signal locations anyway, precisely because there is no signal for the noise to modulate. (We return to this point in the Discussion section.) We convolved these GLM bubbles images with a Gaussian kernel of standard deviation $\sqrt{2}$ pixels to make them comparable to the results of the bubbles simulation reported below. In all bubbles images shown in this article, white pixels indicate regions where increasing the signal's contrast magnitude makes the observer more likely to give a correct response. The main result of the Gaussian noise simulation is that the GLM bubbles images in the third row of Figure 2 are similar to the conventional bubbles images in the bubbles simulation described next. That is, using Gaussian noise we recovered the bubbles images that the conventional method recovers using sparse multiplicative noise. 

Figure 3 shows classification images and bubbles images calculated from bubbles noise. The first row shows bubbles images calculated using the conventional bubbles method (Gosselin & Schyns, 2001). The second row shows bubbles images calculated using the GLM by regressing the correctness of the observer's responses against the bubbles noise. We convolved the GLM bubbles images with a Gaussian kernel of standard deviation $\sqrt{2}$ pixels in order to make them comparable to the conventional bubbles images, and because from the low-frequency noise in this simulation we can only expect to recover low-frequency components of the bubbles image. The GLM bubbles images are similar to the conventional bubbles images, and both types are similar to the bubbles images recovered from Gaussian noise in the previous simulation. 

Figure 3

 

Classification images and bubbles images from bubbles noise.

To make a more exact comparison, we plotted each bubbles image, pixelwise, against the theoretically expected bubbles image, b ∗ b ∗ (s_i ∘ w), where b is the bubble used to create the bubbles noise, and ∗ is two-dimensional convolution (Murray & Gold, 2004; Gosselin & Schyns, 2004). We first subtracted a scalar from each bubbles image so that its mean background value, far from the signal, was zero. We fitted symmetric power functions y = k·sign(x)|nx|ⁿ to the scatterplots, where sign(x) equals +1 when x ≥ 0 and −1 otherwise. Figure 4 shows that all three methods matched the theoretical bubbles image reasonably well. The conventional bubbles image differed from the theoretical image by a compressive nonlinearity, whereas the two GLM bubbles images were closer to being directly proportional. 

Figure 4

 

Pixel-by-pixel comparison of bubbles images generated with different methods to the theoretical bubbles image. Red dots show pixels on signal 1 trials and green dots show pixels on signal 2 trials. The blue lines are power functions fitted to minimize the sum-of-squares error.

Finally, the third row of Figure 3 shows classification images calculated using the GLM by regressing the observer's responses against the equivalent additive noise. That is, we took bubbles noise fields n_m, redescribed them as additive noises fields n_a = s_i ∘ (n_m − 1), and regressed these additive noise fields against the observer's responses. We convolved the classification images with a Gaussian kernel of standard deviation $\sqrt{2}$ pixels. The GLM classification images recovered from bubbles noise are similar to the classification images recovered from Gaussian noise, within the constraint that in this simulation there was no noise in locations where there was no signal, and so naturally we did not recover the observer's template at those locations. Thus the classification image from left-signal trials does not show the upper-right lobe of the observer's template, the image from right-signal trials does not show the upper-left lobe, and neither image completely recovers the bottom two lobes. 

Discussion

The GLM allows us to describe classification images and bubbles images in a common framework and illustrates the strong similarities between them. A classification image is found by regressing the observer's responses against stimulus noise, expressed as additive contrast fluctuations. A bubbles image is found by regressing the correctness of the observer's responses against stimulus noise, expressed as multiplicative contrast fluctuations. Over the subset of trials where a single signal was shown, the difference between the observer's responses and the correctness of the observer's responses is trivial: on trials where signal 1 was shown, a “signal 1” response is correct and a “signal 2” response is incorrect. Thus the more substantial difference is that the classification image measures the influence of stimulus fluctuations expressed as additive noise and a bubbles image measures the influence of stimulus fluctuations expressed as multiplicative noise. For this reason, it seems unlikely that the two methods will reveal fundamentally different information about observers' strategies. Viewed as regression methods, they simply measure the covariates differently. This similarity is not a byproduct of formulating the methods in terms of the GLM; in the Appendix we show that it can seen in the original methods as well. 

A more important difference between classification image and bubbles methods, as they have been practiced, is that they have used different kinds of noise. The GLM gives us more freedom in this respect; it allows us to partly disentangle our analysis from the kind of noise we use. As we have shown, we can measure both classification images and bubbles images using Gaussian white noise. There are limits to this freedom of course, as for instance in the bubbles simulation we were unable to recover classification images at stimulus locations where there was no noise. Also, the GLM approach we have suggested for measuring bubbles images where we ignore noise at nonsignal locations does not work well with noise that is correlated across signal- and nonsignal locations. Suppose we use low-pass Gaussian noise that is correlated at two nearby locations, x₁ and x₂, and there is a signal at x₁ but not at x₂. If we ignore the noise at x₂, then the GLM will attribute any effect that it has on the observer's responses to the correlated noise at x₁, introducing an artifact. 

Within such limits, different types of noise may have various advantages, e.g., we may use bubbles noise in experiments where we wish to show only realistic stimulus fragments (Felsen & Dan, 2005; Rust & Movshon, 2005), or we may use Gaussian white noise to include a wider range of stimulus fluctuations. 

The GLM has made several advances possible in classification image methods, including examining strategies in 1-of-n identification tasks, incorporating priors, using statistical tests associated with the GLM, allowing new kinds of noise, and opening the way to related frameworks such as the generalized additive model (Abbey & Eckstein, 2001; Knoblauch & Maloney, 2008; Macke & Wichmann, 2010; Mineault, Barthelmé, & Pack, 2009; Murray, 2011). With the GLM, these advances can be used for calculating bubbles images as well. 

Acknowledgments

This work was funded by the Natural Sciences and Engineering Research Council of Canada and the Canada Foundation for Innovation. I thank Kenneth Knoblauch, Christopher Taylor, and an anonymous reviewer for helpful comments. 

Commercial relationships: none. 

Corresponding author: Richard F. Murray. 

Email: [e-mail]. 

Address: Centre for Vision Research, York University, Toronto, ON, Canada. 

Appendix

The conventional classification image method can be seen as measuring the covariance between additive zero-mean stimulus noise and the observer's responses (Richards & Zhu, 1994; Murray, 2011). If we encode the observer's responses as r = −1 and r = +1, 

 

 

 

 

 

This is the usual formula for a classification image over all trials where one signal is shown (Ahumada, 1996, 2002). The conventional bubbles image can be seen as measuring the covariance between multiplicative noise and the observer's responses. If we encode responses as r = 0 and r = 1, 

 

 

 

The usual bubbles image is E[n|r = 1], which differs from this covariance only by a constant (Gosselin & Schyns, 2001).  

This is all elementary, but it shows that the two methods can be seen as doing very similar analyses, one on additive noise and the other on multiplicative noise, just as in the GLM versions.  

As a result, so long as the noise elements are weak enough that they bias the observer's response only slightly, there will be a close relationship between classification images and bubbles images. Coding responses as r = 1 and r = 2 as in the main text, suppose the observer's response probability on trials where the signal is s₂ can be approximated as 

 

Here p₂ is the probability of the observer giving response r = 2 without noise, n is a vector of additive noise, and u is a column vector that describes how each noise element perturbs the response probability. A template matcher in weak noise can be approximated as in Equation (A6), and in this case u is proportional to the template w.  

In a conventional classification image experiment we show a signal in additive noise and we find the expected value of the noise on each stimulus-response category of trials, e.g., signal s₂ and response r = 2. 

 

 

 

 

 

 

In a conventional bubbles experiment we show a signal in multiplicative noise and we find the expected value of the noise field on correct responses. 

 

 

Here n_add is the additive noise that is equivalent to the multiplicative noise n. As explained in the main text, on trials where the signal is s₂, n_add = (n – 1) ∘ s₂. 

 

 

 

 

 

 

Equations (A9) and (A13) show that under quite general conditions, classification images and bubbles images are closely related. They differ only in that Equation (A9) has u where Equation (A13) has s₂∘u and Equation (A13) includes the mean noise field times a scalar. Thus when we measure a classification image using additive noise we get a function of the observer's coefficients u. (The function depends on what noise distribution we use, e.g., if we use Gaussian white noise then Equation (A9) is proportional to u.) When we measure a bubbles image using multiplicative noise we get the same function of the pointwise scaled coefficients s₂∘u (plus the mean noise field times a scalar). This similarity does not require strong assumptions about the observer, but follows from (a) the fact that in classification image experiments we use noise additively and in bubbles experiments we use it multiplicatively and (b) the assumption that individual noise elements have small enough effects that we can describe the observer's response probabilities using the linear approximation in Equation (A6). 

Footnotes

¹  The GLM does make assumptions about other noise distributions. The GLM given by Equations (1) and (2) assumes that the decision variable is the linear predictor n^Tβ_i + β_i,0 plus a noise source ϵ that is independent from trial to trial, has zero mean, and has a fixed variance that is independent of the linear predictor. It also assumes that the observer's responses are independent Bernoulli random variables.

Footnotes

²  Equations (1) and (2) assume that the observer's decision variable is a locally linear function of the stimulus in the neighborhoods of the two signals. Because the stimuli are noisy, it could occasionally happen that s₁ + n is closer to s₂ than to s₁. A reviewer pointed out that instead of using all stimuli from s₁ trials as covariates in Equation (1), another possibility would be to use all stimuli that are physically closer to s₁ than to s₂ as covariates in Equation (1), regardless of whether they are from s₁ trials or s₂ trials. Similar comments apply to the covariates in Equation (2).

Footnotes

³  1-of-n bubbles images have been calculated using the original bubbles method in a one-against-the-rest manner (Smith, Cottrell, Gosselin, & Schyns, 2005). Murray (2011) discusses some advantages of using the GLM with multinomial response variables.

Footnotes

⁴  The method we have used does not incorporate a prior, so it requires at least as many trials as there are pixels in the bubbles image in order for there to be a unique maximum-likelihood solution. Several methods for imposing priors and other kinds of regularization have been developed for classification images and these can easily be adapted to bubbles images as well (Murray, 2011, pp. 9–10).