Nonlinear Summation Really Can Be Used to Perform AND Operations: Reply to May and Zhaoping

Introduction

May and Zhaoping (this issue) make the case that neurons must be performing multiplication in order to generate the AND gates needed to detect conjunctions and that the method of simply summing nonlinear inputs suggested by Peirce (2007) would not suffice. Their conclusions are based on three main premises: that nonlinear summation ¹ only applies to conjunctions in which the components have lower contrast than the compound (plaids, for example, but not contours), that it cannot explain data collected in cases where the components have the same contrast when presented alone or as a compound; and that it cannot operate for stimuli with very low contrast. Each of those claims is examined in turn below. The first two turn out to be false when one considers the effect of a threshold applied either at the output of the summing mechanism or, equivalently, on the input of any subsequent readout mechanism. The third assertion is true and in full agreement with the findings of psychophysical experiments that indicate a lack of conjunction detection at low contrasts. Rather than showing that nonlinear summation could not support conjunction detection, May and Zhaoping have actually highlighted a piece of indirect evidence that it is indeed the mechanism being used. 

It should be noted that the arguments below do not suggest that nonlinear contrast response functions are solely used for conjunction detection. Possibly, these nonlinearities exist solely to optimize the dynamic range of neurons and reduce redundancy in signals, they support multiple functions, or they are merely an epiphenomenon with no function at all. The argument made here is simply that they could be used to support neural AND gates, as proposed in Peirce (2007). 

Is nonlinear summation useful in detecting conjunctions?

May and Li claim that the use of nonlinear summation is not useful in detecting signals that are not limited by a fixed net contrast, and that even if it were a useful mechanism to detect plaids, it would not be useful for, say, contour integration. Actually, the original article already noted that in those cases where components do not overlap spatially nonlinear summation was not strictly necessary. However, it certainly could facilitate conjunction detection even in those cases. 

The point is illustrated in Figure 1. The center panel shows the response of the nonlinear summation mechanism described in Peirce (2007), as a function of contrast in two input channels. A decision boundary has been added to the plot indicating all points where the response to the compound is greater than the maximal response to any single component presented alone (by at least 5% of the total dynamic range of the cell). This decision boundary might represent the threshold required to trigger a response, either at the spike-generating stage or in some later readout mechanism. It is hopefully very clear that the mechanism would be useful in detecting the conjunction in nearly all cases, excluding only those where one or both components has a low contrast, as discussed below. The only difference between spatially overlapping and spatially separated components is that spatially overlapping components are limited to the lower left portion of the graph. That portion, for the linear summation model (left panel), does not include any correctly detected conjunctions, hence, the necessity for an AND gate (either multiplicative or with nonlinear summation) for these types of compounds. 

Figure 1

 

The outputs of three mechanisms combining two input signals, as a function of the input contrasts. The greyscale indicates a notional response rate ranging 0–100 ips for all three model cells. The solid line indicates a decision boundary for the presence of the conjunction; the set of contrast combinations that are 5 ips greater than the maximum response generated from stimulating either channel alone. For both the nonlinear summation model and the multiplicative combination this decision boundary includes most cases where a conjunction might be thought to exist. Model details: The linear and multiplication models have linear inputs. The nonlinear summation model has sigmoidal (Naka‐Rushton) inputs with C₅₀=0.1, exponent=2.0, in keeping with single‐unit recordings. In all cases the inputs are scaled to a notional maximum response of 100 ips. Models A and B then sum the inputs, while C multiplies them. The outputs of all three are also passed through a moderate output nonlinearity (C₅₀=100 ips, exponent=2.0). This step does not affect the current discussion but serves to constrain the maximal responses of the mechanisms. Complete Python source code for the figure can be found in the Supplementary materials.

The principal reason that May and Zhaoping claim that nonlinear summation cannot support AND operations is that, with strong stimulation in one channel alone, there is always some degree of response; the sigmoidal nonlinearities described cannot force the summed signals back to zero. May and Zhaoping implicitly assume that the decision about whether the conjunction is present or absent is based on whether the mechanism responds at all; there is an implicit boundary set at zero impulses per second. This is natural enough in mathematics but is not the only option, and for neurons, it seems an unrealistic decision boundary. Neuronal responses are noisy, and to allow tolerance to the noise, a readout/decision mechanism should surely set its decision boundary at some reasonable level above zero. The natural choice would be to set it at some level just above the maximum response that can be generated from either component alone. When that is done, the “decision” to be made from the multiplicative AND gate (Figure 1, right panel) and one using nonlinear summation is actually very similar. Furthermore, it takes very little extension (e.g., the addition of a spike-generating mechanism with an appropriate response threshold) to convert this into a “genuine” AND gate, for which there is no response at all when one of the components is not present. The sources that May and Zhaoping cite as existing evidence for multiplication (e.g., binocular obligate cells of Hubel & Wiesel, 1962) could equally be modeled by nonlinear summation of two signals, as in the middle panel of Figure 1, with no need for multiplication. 

The second criticism of the proposed mechanism was that, for the stimuli we used in our plaid (e.g., McGovern & Peirce, 2010; Peirce & Taylor, 2006) and curvature (e.g., Hancock & Peirce, 2008) adaptation experiments, the contrast of components was constant in the compound and component conditions and, therefore, that the sum of responses to the components would exactly equal the whole. This was obviously necessary in those studies, in order to show that adaptation to the “whole” is greater than to the “sum of the parts”. Again, the authors fail to consider the response (or readout) threshold. With this small addition, which I clearly should have made more explicit in the original description of the mechanism, it quickly becomes clear that either high-contrast component alone would fall outside the decision boundary and result in no response, whereas the compound stimulus would cross the boundary and a response would result. 

May and Zhaoping's third assertion is, however, quite correct. The nonlinear summation mechanism will fail to detect conjunctions when the components are presented with very low contrast. As a result, if the visual system did use nonlinear summation in the detection of conjunctions then it might fail, or might have to resort to a different mechanism, when conjunctions are presented at low contrast. 

In fact, that prediction appears to hold true. Certainly, selective adaptation to plaids, the phenomenon that caused me to think about the mechanism in the first place, falls off dramatically with probe contrast; by a Michelson contrast of 0.1, it is swamped by adaptation to the component gratings (McGovern & Peirce, 2010). Similarly, Meese and Freeman (1995) show that plaid patterns tend to be perceived as two overlapping gratings at low contrast rather than as a single coherent pattern. Sarah Hancock has now attempted to collect similar data for the curvature aftereffect (the CAE, as described by Hancock & Peirce, 2008) but found the task of identifying magnitude of curvature too difficult to be able to measure an aftereffect when probes had a Michelson contrast below 0.1 (Hancock, unpublished observation). For radial frequency patterns, sensitivity is relatively constant for probe contrasts between 1.0 and 0.125 but then drops substantially when contrast is reduced further (Wilkinson, Wilson, & Habak, 1998). Similarly, sensitivity to contours in a field of Gabor patches plummets when contrast falls below 0.1 but is roughly constant above that (McIlhagga & Mullen, 1996). May and Zhaoping are correct to point out that for low contrasts the mechanism would predict a failure to detect or accurately discriminate conjunctions. What they fail to point out is that this fits very well indeed with data from numerous psychophysical studies into a wide range of “mid-level” visual tasks. 

The fact that the visual system might perform conjunction detection simply by summing the nonlinear responses that we already know to exist does not mean that we would necessarily want to do that in computational modeling projects. Whereas nonlinear summation would seem easier to implement in neural circuits than multiplication (not requiring the three layers of neurons to perform the Babylonian trick that May and Zhaoping suggest) in mathematics, simply multiplying signals is often the more straightforward option and can result in very similar results, as shown in Figure 1. 

Summary

We may never know which of these methods the visual system actually uses to detect conjunctions of signals. On the other hand, if simply adding the nonlinear signals that we already know to be present results in a physiological approximation of the AND operation, then it seems hard to imagine that the brain does not occasionally make use of such a computation. In a domain where mathematical models are often described as “physiologically plausible,” this particular mechanism would seem “physiologically almost certain” to exist. 

Supplementary Materials

Supplementary PDF - Supplementary PDF 

Acknowledgments

The author would like to thank Sarah Hancock for attempting to collect low-contrast CAE data and David McGovern, Stephen Coombes, and Carl-Magnus Svensson for feedback on the ideas in this manuscript. This work was supported by a grant from The Wellcome Trust (085444/Z/08/Z). 

Commercial relationships: none. 

Corresponding author: Jonathan Peirce. 

Email: [email protected]. 

Footnote

Footnotes

1  Some authors have used the term “nonlinear summation” to refer to models that contain multiplicative terms in the combination step. Here, it is used to refer specifically to a model in which a nonlinearity is applied to input channels that are then combined in a simple (linear) summation step. For present purposes the nonlinearity on these channels is conceived as a simple sigmoidal Naka-Rushton function [R = C ⁿ / (C ⁿ + C ₅₀ ⁿ ), where C is input contrast and C ₅₀ and n are parameters determining the shape of the nonlinearity]. In Peirce (2007), this equation was extended with an additional exponent to allow nonmonotonic response functions to be included, but that is not critical for the discussion here.