Multiplication in curvature processing

Abstract

Multiplication rather than addition of neural signals is believed to underpin a variety of sensory processes, yet the evidence for multiplication is rare. Here we provide psychophysical evidence for neural multiplication in human visual processing of shape. We show that the curvature of a contour is likely detected by a mechanism that multiplies rather than adds the signals from afferent sub-units that detect parts of the curve. Using a novel perceptual after-effect, in which the perceived shape of a sinusoidal-shaped contour is altered following adaptation to a contour of slightly different sinusoidal shape, a pronounced ‘dip’ in the size of the after-effect is found when the adapting contour is broken into segments of a particular length and spacing. Simulations reveal that the presence and shape of the dip is only expected if the afferent sub-units to curvature detectors are multiplied. The after-effect itself is then best explained in terms of the population response of a range of such curvature detectors tuned to different curvatures.

Introduction

A range of visual and auditory processes are believed to be underpinned by multiplication (Barlow & Levick, 1965; Gabbiani et al., 2004; Gabbiani, Knapp, Koch, & Laurent, 2002; McAdams & Maunsell, 1999; Peña & Konishi, 2001; Reichardt, 1957; Salinas & Abbott, 1996; Sun & Frost, 1998). Multiplication is a form of AND-gating and can in principle be implemented via a variety of physiological routes (Koch, 1999; Koch & Poggio, 1992; Mead, 1989). Evidence consistent with multiplication among visual neurons comes mainly from physiological studies of motion processing (Barlow & Levick, 1965; Gabbiani et al., 2002, 2004; Hassenstein & Reichardt, 1956; McAdams & Maunsell, 1999; Treue & Martinez Trujillo, 1999; Treue & Maunsell, 1999). Multiplication is also featured in a number of models of visual processes, such as orientation processing (Wenderoth, Johnstone, & van der Zwan, 1989), stereoscopic depth processing (Burke & Wenderoth, 1989), motion processing (Van Kruysbergen & de Weert, 1994; van Santen & Sperling, 1985, 1984), and curvature processing (Poirier & Wilson, 2006; Zetzsche & Barth, 1990). However, there is little psychophysical evidence for multiplication in vision. Van Santen and Sperling (1985, 1984) showed how one could identify the direction of motion of an apparent motion stimulus consisting of a pair of adjacent vertical bars with different temporal luminance modulations, even when one bar was sub-threshold. Van Santen and Sperling interpreted the finding as evidence for multiplication of the two component signals. 

In this communication we provide evidence for multiplication in the visual processing of curvature. Curvature can be defined at each point along a contour as the rate of change of the slope of the tangent to the contour with respect to the distance along the contour. However in the sinusoidal-shaped contours used here, as with the perturbed-circle radial-frequency patterns commonly used elsewhere (Anderson, Habak, Wilkinson, & Wilson, 2007; Wilkinson, Wilson, & Habak, 1998), curvature, as so-defined, is not a constant along any portion of the curve (which is also true if curvature is defined at each point by 1/radius), unlike for curves that are co-circular. For this reason we define a ‘curve’ as that portion of a contour in which curvature (as defined above) is non-zero everywhere, not necessarily constant, but of constant sign. A curve defined in this way can be characterized by its ‘sag’ and ‘cord’ (Gheorghiu & Kingdom, 2007a, 2008), which leads to a definition of curvature as (proportional to) the product of sag and cord. Both the psychophysics and neurophysiological literature is replete with references to curves, or to the receptive fields of curvature-sensitive neurons, that are not co-circular (Pasupathy & Connor, 1999, 2001, 2002), so our somewhat colloquial definition of curvature is appropriate to existing notions of what are curves. 

Curvature plays an important role in the representation and recognition of shapes. It is important to bear in mind from the outset however that models for curvature detection and discrimination do not necessarily provide an account of how curvature is represented in the brain. Curvature detection (the task of discriminating a curve from a straight line) and discrimination (the task of discriminating two curves) is probably accomplished with a minimum of neural machinery. For example, single unit recordings in area 17 of the cat visual cortex have shown that end-stopped cells can discriminate between different curves (Dobbins, Zucker, & Cynader, 1987, 1989; Versavel, Orban, & Lagae, 1990). However because end-stopped neurons are univariant with respect to both short straight lines and long curved ones, it is unlikely that they are used to represent curvature. Indeed Zetzsche and Barth (1990) have shown that the Dobbins et al. end-stopped model, which involves the nonlinear combination of pairs of V1 simple-cell-like linear filters with different receptive-field sizes, is unable to fully distinguish between a straight and a curved contour, even if modified by common nonlinearities such as rectification, clipping, and thresholding; although the nonlinearities change the form of the response to a straight line, they never cause it to disappear (see Figures 1a and 1b in Zetzsche & Barth, 1990). Another suggested neural mechanism for curvature discrimination is the comparison of responses from pairs of orientation-selective V1 simple cells positioned at different points along the curve (Anzai, Peng, & Van Essen, 2007; Hedgé & Van Essen, 2000; Kramer & Fahle, 1996; Tyler, 1973; Wilson, 1985; Wilson & Richards, 1989). Again however, this may be insufficient to represent curvature. 

The representation of curvature likely involves more elaborate neural machinery than that needed for detection and discrimination and is probably mediated by neurons in higher visual areas (Connor, Brincat, & Pasupathy, 2007; Gallant, Braun, & Van Essen, 1993; Gallant, Connor, Rakshit, Lewis, & Van Essen, 1996; Pasupathy & Connor, 1999, 2001, 2002) that receive inputs from arrays of V1 simple cells whose receptive fields are arranged in a curvilinear fashion (Gheorghiu & Kingdom, 2007a, 2007b, 2008; Poirier & Wilson, 2006). 

We refer to the afferent inputs to a putative curvature detector as ‘sub-units.’ It has been suggested that the sub-units of curvature detectors are multiplied rather than added (Poirier & Wilson, 2006; Zetzsche & Barth, 1990). Multiplication is an AND-gate-like operation, meaning that if any of the afferent inputs are inactive the curvature detector will not respond. However, to our knowledge, there is no psychophysical or physiological evidence that curvature detectors have such properties. 

The mechanism by which curves are represented, as opposed to detected and discriminated is arguably best understood through studies of curvature appearance (Ben-Shahar & Zucker, 2004; Gheorghiu & Kingdom, 2007a, 2008). This is because the perceived curvature of a contour is likely signaled via the population response of curvature-sensitive neurons tuned to different curvatures, just as perceived orientation is believed to be signaled via a population response of orientation-sensitive neurons tuned to different orientations (Dragoi, Sharma, Miller, & Sur, 2002; Dragoi, Sharma, & Sur, 2000). 

An important class of appearance-based psychophysical tool is the visual after-effect. In a visual after-effect, the appearance of a stimulus is altered as a result of selective adaptation to a slightly different stimulus. Such after-effects are believed to be mediated by the mechanisms that represent the dimension of interest, in other words via a population code as described above. 

Our evidence for an AND-gate-like operation (i.e., multiplication) in curvature processing has emerged from studies using two recently discovered visual after-effects of perceived shape: the shape-frequency and shape-amplitude after-effects, or SFAE and SAAE (Gheorghiu & Kingdom, 2006, 2007a, 2007b, 2008). The SFAE and SAAE are the perceived shifts in, respectively, the shape frequency and shape amplitude of a sinusoidal test contour following adaptation to a sinusoidal contour of slightly different shape frequency/shape amplitude. As with other spatial after-effects such as the tilt after-effect (Gibson, 1933; Magnussen & Kurtenbach, 1980; Wenderoth & Johnstone, 1988) and the size or luminance–spatial-frequency after-effect (Blakemore & Sutton, 1969), the perceived shifts in the SFAE and SAAE are always in a direction away from that of the adaptation stimulus. 

Gheorghiu and Kingdom (2007a) provided evidence that both the SFAE and SAAE are mediated by mechanisms sensitive to local curvature, rather than to local orientation or to global shape frequency/shape amplitude. They found that the perceived shifts induced into sinusoidal-shaped test contours by both sinusoidal-shaped and square-wave-shaped adaptors were very similar, which they argued on geometrical grounds was hard to explain on the basis of local orientation adaptation. Additional evidence against local orientation as the adapting feature is the degree of independence of the SFAE and SAAE: adaptor shape frequencies have little or no effect on the perceived shape amplitudes of test contours, and adaptor shape amplitudes have little or no effect on the perceived shape frequencies of test contours (Gheorghiu & Kingdom, 2008). SFAEs and SAAEs are therefore useful for probing curvature encoding in human vision, as they presumably result from changes in the shape of the response distribution of neurons tuned to different curvatures, in a manner similar to that proposed for other spatial after-effects (Clifford, 2002; Georgeson, 2004). 

Readers can experience the SFAE and the SAAE in Figures 1a and 1b. If one moves ones' eyes back and forth along the horizontal markers between the pair of adapting contours on the left for about a minute, and then transfer one's gaze to the central spot on the right, the test contours above and below the fixation dot should appear to have a different shape frequency ( Figure 1a) or a different shape amplitude ( Figure 1b), even though they are physically identical. An important property of both after-effects is that they survive shape-phase randomization during adaptation, as can be experienced in the non-static adaptor version at http://www.mvr.mcgill.ca/Fred/research.htm#contourShapePerception. The reason for randomizing shape phase during the adaptation period is to prevent the formation of afterimages and to minimize the effects of local orientation adaptation. If we assume that the visual system is tiled with curvature detectors, then a sine-wave-shaped contour will stimulate roughly the same number of curvature detectors irrespective of shape phase, in the same way that a luminance grating will stimulate roughly the same number of luminance–spatial-frequency channels irrespective of luminance phase. 

Figure 1

 

Stimuli used in the experiments. One can experience (a) the shape-frequency after-effect (SFAE) and (b) the shape-amplitude after-effect (SAAE) by moving one's eyes back and forth along the markers located midway between the pair of adapting contours (left) for about 90 s, and then shifting one's gaze to the middle of the test contours (right). (c–k) Segmented sine-wave-shaped contours. Segment length is expressed as the proportion of the length of a single cycle of the contour measured along the path of the contour. The corresponding segment lengths are: 0.0336, 0.0597, 0.1062, 0.1888, 0.3358, 0.5972, 1.0619, 1.8884, and 3.3581. Tests were all continuous contours.

Our hypothesis is that curves are encoded by curvature detectors that receive input from a small number of orientation-selective V1 neurons whose responses are combined by an AND-like operation, such as direct multiplication or another operation equivalent to multiplication (e.g., Log–Exp transform or a saturating nonlinearity followed by addition and accelerating nonlinearity). We predict that the response of a curvature detector will be maximal when all its afferent inputs are stimulated and will be unresponsive if any of its inputs are not stimulated. 

The evidence for multiplication in curvature encoding has emerged from the use of sinusoidal-shaped contour adaptors that are broken into segments, with the gap between segments equal to the segment length. The model simulations described below reveal that if the sub-units to a putative curvature detector are multiplied, there is a pronounced symmetrical dip in the response of the detector at an intermediate segment/gap length. No other type of combinatorial non-linearity besides multiplication (or its mathematical equivalent) predicts such a symmetrical dip. We show that a similar-shaped dip is found psychophysically in both the SFAE and SAAE, when measured using adaptors of various segment/gap lengths but continuous test contours. We interpret this result as the first evidence that the sub-units to curvature detectors are multiplied. 

In what follows we first describe the method for obtaining the psychophysical data, second the model simulations that predict the dip in the size of the two after-effects, and third the psychophysical results that test for the presence of the dip. Finally we consider whether other types of non-linear combinations of sub-units can predict the dip. 

Methods

Observers

One of the two authors (EG) and four naive volunteers (AB, LS, BA, KW) participated in the study. Three of the subjects (AB, EG, LS) participated in the measurement of both after-effects. Subject BA participated only in the measurement of the shape-frequency after-effect, while subject KW participated only in the measurement of the shape-amplitude after-effect. All subjects had normal or corrected-to-normal visual acuity. Each subject gave informed consent prior to participation in accordance with the university guidelines. 

Stimuli

The stimuli were generated by a VSG2/5 video-graphics card (Cambridge Research Systems) with 12-bits contrast resolution, presented on a calibrated, gamma-corrected Sony Trinitron monitor, running at 120-Hz frame rate and with a spatial resolution of 1024 × 768 pixels. The mean luminance of the monitor was 42 cd/m ². 

Example stimuli are shown in Figure 1. Adaptation and test stimuli consisted of pairs of 2D sine-wave-shaped contours. The two adaptors and tests were presented in the center of the monitor 3.5 deg above and below the fixation marker. The cross-sectional luminance profile of the contours was odd-symmetric and was generated according to a first derivative of a Gaussian function:  

$L\left(d\right)=L_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}\pm L_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}·C·\exp \left(0.5\right)·(d/\sigma )·\exp [-(d^2)/(2{\sigma }^2\left)\right],$

(1)

where d is the distance from the midpoint of the contour's luminance profile along a line perpendicular to the tangent, L _mean is mean luminance of 42 cd/m ², C is contrast, and σ is the space constant. C was set to 0.5 and σ to 0.044 deg for all experiments. The ± sign determined the polarity of the contour. Our contours were designed to have a constant cross-sectional width, and the method we used to achieve this is described elsewhere (Gheorghiu & Kingdom, 2006). 

The adaptor pair for the SFAE consisted of contours with a shape amplitude of 0.43 deg and shape frequencies of 0.25 and 0.75 c/deg, giving a geometric mean shape frequency of 0.43 c/deg. For the SAAE, the shape frequency of the adaptor pair was 0.43 c/deg, while the shape amplitudes were 0.25 and 0.75 deg, giving a geometric mean of 0.43 deg. The mean shape frequency and mean shape amplitude of the test contour pair were always maintained constant at 0.43 c/deg and 0.43 deg, respectively. 

The adaptors were either continuous sine-wave-shaped contours or segmented sine-wave-shaped contours. In the segmented contours the segments and gaps were of equal length. For clarity, we express the segment length as a proportion of the length of a single cycle of the contour measured along the path of the contour. We used nine segment/gap lengths: 0.0336, 0.0597, 0.1062, 0.1888, 0.3358, 0.5972, 1.0619, 1.8884, and 3.3581. The test was always a continuous sine-wave-shaped contour. In order that the adapting contours of different shape frequency were comparable in terms of their segmented profiles, the segment length ratio of the two adapting contours was the same for all segment lengths (remember segment length is expressed as a proportion of the length along the path of a single cycle). The segment length ratio was 0.5098 for the SFAE adaptors and 1.5295 for the SAAE adaptors. 

Procedure

Each session began with an initial adaptation period of 90 s, followed by a repeated test of 0.5 s duration interspersed with top-up adaptation periods of 2.5 s. During the adaptation period, the shape phase of the contour, as well as the phase of the segmentation, was randomly changed every 0.5 s in order to prevent the formation of afterimages and to minimize the effects of local orientation adaptation. The presentation of the test contour was signaled by a tone. The shape phase of the test contour was also randomly assigned in every test period. The display was viewed in a dimly lit room at a viewing distance of 100 cm. Subjects were required to fixate on the marker placed between each pair of contours for the entire session. A head and chin rest helped to minimize head movements. 

A staircase method was used to estimate the point of subjective equality, or PSE. For the SFAE the geometric mean shape frequency of the two test contours was held constant at 0.43 c/deg while the computer varied the relative shape frequencies of the two tests in accordance with the subject's response. At the start of the test period the ratio of the two test shape frequencies was set to a random number between 0.33 and 3. On each trial subjects indicated via a button press whether the upper or lower test contour had the higher perceived shape frequency. The computer then changed the ratio of test shape frequencies by a factor of 1.06 for the first five trials and 1.015 thereafter, in a direction opposite to that of the response, i.e., toward the PSE. The session was terminated after 25 trials. The shape-frequency ratio at the PSE was calculated as the geometric mean shape-frequency ratio of the tests adapted, respectively, by the lower and higher shape-frequency adaptors, averaged across the last 20 trials. The geometric rather than arithmetic mean is the appropriate way to average ratios (e.g., if one experiment yields a ratio of 10,000 and the other 0.0001, the arithmetic mean misleadingly gives an average ratio near 5000, whereas the geometric mean accurately gives a ratio of 1). Six measurements were made for each condition, three in which the upper adaptor had the higher shape frequency and three in which the lower adaptor had the higher shape frequency. 

In addition we measured for each condition the shape-frequency ratio at the PSE in the absence of the adapting stimulus (the no-adaptor condition). To obtain an estimate of the size of the SFAE, we calculated, for each with-adaptor measurement, the difference between two quantities: the logarithm of the with-adaptor shape-frequency ratio at the PSE, and the mean logarithm of the no-adaptor shape-frequency ratio at the PSE. We then calculated the mean and standard error of these differences across measurements. These standard errors are the ones shown in the graphs. 

The procedure for measuring the SAAE followed the same principle as for the SFAE. The computer varied the relative shape amplitudes of the two tests in accordance with the subject's response, while the geometric mean shape amplitude of the two test contours was held constant at 0.43 deg. 

Results

Model

Our hypothesis is that curves are encoded by curvature detectors that receive input from a small number of orientation-selective V1 neurons whose responses are combined by an AND-like operation, such as direct multiplication or other operations equivalent to multiplication. 

Before showing the psychophysical results, we present a model that is a simple implementation of our hypothesis. The model simulates the output of a putative curvature detector that multiplies the outputs of its sub-units, in response to a contour divided into segments. The model reveals an important consequence of multiplication. Consider a half-cycle, cosine portion of one of the sinusoidal-shaped contours shown in Figures 1a and 1b. Assume that a curvature detector receives input from a small number of sub-units (e.g., four) whose abutting receptive fields are arranged along a curve, without gaps or overlap, as shown in Figure 2a. Each sub-unit sums the energy (e.g., r.m.s. luminance contrast) of the contour that falls within its receptive field, but the responses of the sub-units are multiplied. The response of the curvature detector will be maximal when stimulated by a continuous contour matched to its global receptive-field structure. The response however will drop to zero if any of the sub-units are not stimulated, as multiplication of any number by zero always equals zero. This fact is the reason for using the segmented contours shown in Figures 1c– 1k. The contours in the simulation are of various segment lengths, with the constraint that the gap between segments is always equal to the length of the segment, such that the overall amount of contour energy is the same in every segment length condition. For the model simulation, segment length is expressed in arbitrary units (e.g., pixels). 

Figure 2

 

Example responses from the model simulation for (a) a non-segmented contour and (b–f) segmented contours of segment lengths 0.0336, 0.1062, 0.3358, 0.5972, and 1.8884. In each figure a curvature detector receives input from four sub-units whose receptive fields are arranged curvilinearly and whose responses are multiplied. The curvature detector's receptive field is positioned to be matched to a portion of the contour. The response of the curvature detector for the sample iteration (i.e., one random phase of segmentation) in each figure is shown as the height of the black bar on the right (see text for details). (g) Mean response of the curvature detector estimated over 1000 iterations (i.e., 1000 random phases of segmentation), as a function of segment length. The mean response is similar for both short and long segment lengths, but for a narrow range of intermediate segment lengths, the response collapses to zero.

The results of the model simulation are illustrated for a continuous, i.e., non-segmented contour in Figure 2a, and for a single, randomly chosen segment phase for segmented contours of 5 segment/gap lengths in Figures 2b– 2f. The response of the curvature detector is shown as the height of the black bar on the right of each figure. In the full simulation the shape phase of the sine-wave-shaped contour was fixed so that the contour always passed through the curvature detector's receptive field. However, the phase of segmentation was randomized on each iteration. By randomizing segmentation phase the probability that any point along the path of the curvature detector's receptive field was stimulated would be exactly 0.5 per iteration, for all segment lengths. Note that in Figure 2f, which shows an example response to the longest segment length condition, the particular response shown would only occur on a small proportion of iterations. In this segment length condition segment-phase randomization would result in just as many iterations producing no response whatsoever. Example responses from a continuous sequence of iterations (i.e., segmentation phases) can be seen in the movies shown in 1 or at http://www.mvr.mcgill.ca/Fred/research.htm#contourShapePerception. 

Figure 2g shows the average response of the curvature detector estimated over a large number (1000) of iterations (each with random segmentation phase), as a function of segment length. As one can see, the mean response is similar at both short and long segment lengths, but for a narrow range of intermediate segment lengths, the response collapses to zero. Additional simulations reveal that the position and width of the dip is determined by two factors: (a) the number of sub-units, and (b) the length of the sub-unit receptive field. We simulated the effect of these two factors separately. As an illustration, Figure 3a shows the effect of the number of sub-units ( n = 2, 3, 4, and 5) for a constant sub-unit length ( s = 40). Figure 3b shows the effect of sub-unit length for a constant number of sub-units ( n = 4). These examples show that by increasing either the number or the length of the sub-units, the position of the dip shifts toward longer segment lengths. 

Figure 3

 

Model simulations showing the effect of (a) the number of sub-units ( n = 2, 3, 4, and 5) for a constant sub-unit length ( s = 40) and (b) sub-unit length for a constant number of sub-units ( n = 4). Increasing either the number or the length of the sub-units shifts the position of the dip toward longer segment lengths. (c) Sample average curvature detector responses estimated over 1000 random segmentation phase samples (black symbols) and step-by-step changes in segmentation phase (red symbols), as a function of segment length. (d) Example mean response of curvature detector with standard error of the mean (SEM) showing the variability in the response of the curvature detector estimated over 1000 iterations, each with random segmentation phase. (e) The segment length producing the mean minimum response is plotted as a function of the overall length of the curvature detector's receptive field, for different numbers of sub-units n ( n = 2, black square, dash line; n = 3, gray square, gray line; n = 4, gray circles, black continuous line; n = 5, black circles, black continuous line; n = 6, black triangles, black continuous line).

One might think that a random, rather than step-by-step change in the segmentation phase would produce a different average response in the curvature detector. Figure 3c shows the response of the curvature detector averaged over a large number (1000) of random (black symbols) and step-by-step (red symbols) changes in segmentation phase, as a function of segment length. As one can see, the average response is similar for both random and step-by-step changes. The standard error of the mean (SEM) of the responses for the random-phase condition is shown in Figure 3d for a curvature detector with n = 4 sub-units and sub-unit length s = 40. Similar SEMs were obtained for other combinations of number and length of sub-units. 

For different combinations of number and length of sub-units, we determined the segment length at which the mean response of the curvature detector reached a minimum. In the single-filter model described here, the product of number and length of sub-unit gives the length of the curvature detector's receptive field. Figure 3e shows a linear dependence between the segment length producing the minimum response and the curvature detector's receptive-field length. Knowing the segment length of the contour at which the mean response of the curvature detector reaches a minimum therefore allows us to estimate the approximate length of the curvature detector's receptive field, as a function of the number of sub-units. 

To summarize, our simulations reveal a signature of a curvature detector that multiplies the inputs from its component sub-units. When stimulated with curved contours that are broken into segments, with the gap between segments equal to the segment length, a response function with a pronounced near-symmetrical dip is found at a particular intermediate segment length. 

Before describing the psychophysical results, there is however an important caveat. The model above simulates the average response of a single curvature detector whose global receptive-field structure is closely matched to a half-cycle cosine part of the adapting contour. As such, the model is not a model of the SFAE or SAAE. These after-effects presumably result from changes in the gain of a sub-set of curvature-sensitive neurons whose population response contains the code for curvature. Only some of these neurons will have a global receptive field structure matched to the half-cycle cosine parts of the contour. Therefore although all responding curvature detectors will be subject to the effects of segmentation as described in our matched single-detector model, the model can only make qualitative, not quantitative, predictions about the SFAE and SAAE. 

Psychophysics

If the response of a curvature detector that multiplies its sub-unit responses dips at some intermediate segment length, then it is reasonable to assume that a curvature-based after-effect induced by segmented adaptors will also dip at some intermediate segment length. We tested this prediction in both the SFAE and the SAAE. For each measurement subjects were presented with a pair of adaptors that differed by a factor of three in either shape frequency or shape amplitude (see Figures 1a and 1b). During the test phase subjects were required to adjust the difference between two test stimuli (presented at the same retinal locations as the adaptors) using a computer-controlled staircase procedure, until the point of subjective equality (or PSE) was reached. The size of the after-effect was then given by the ratio of the relevant dimension (shape frequency for SFAE and shape amplitude for SAAE) between the two test stimuli at the PSE. Details of the procedure and data analysis are given in the Methods section. We used two types of adaptors: (i) continuous sine-wave-shaped contours, and (ii) segmented sine-wave-shaped contours (see Figures 1c– 1k). Segment length was expressed as the proportion of the length of a single cycle of the contour measured along the path of the contour. We used nine adaptor segment lengths: 0.0336, 0.0597, 0.0, 0.1888, 0.3358, 0.5972, 1.0619, 1.8884, and 3.3581. The test contours were all continuous. The reason for using continuous test contours was to ensure an ‘even playing field’ for all conditions and to ensure that segmentation only impacted the adaptation process and not the conscious judgment of the relative shapes of the two test contours. 

Figure 4a shows the size of the after-effect for SFAEs (gray symbols) and SAAEs (black symbols) as a function of adaptor segment length. The right, lower panel in Figure 4a shows the SFAE for subject BA and the SAAE for subject KW. The gray and black coarse-dashed lines indicate the SFAE and SAAE obtained with continuous sine-wave-shaped adaptors. As can be seen, the maximum sized after-effects obtained with segmented adaptors are smaller than those for continuous adaptors. More importantly, both SFAEs and SAAEs show a pronounced dip at intermediate adaptor segment lengths. 

Figure 4

 

Psychophysical results. (a) SFAEs (gray symbols) and SAAEs (black symbols) are plotted as a function of adaptor segment length. The gray and black coarse-dashed lines indicate the SFAE and SAAE obtained with continuous sine-wave-shaped adaptors. The fine-dashed lines indicate the no-adaptor conditions. Both the SFAE and SAAE show a pronounced dip for intermediate adaptor segment lengths (0.33 proportion of single cycle length for subjects AB, LS, BA, and KW and 0.59 for subject EG) for both after-effects. (b) Average SFAEs and SAAEs for four subjects (see text for details).

The dip occurs at the same adaptor segment length (0.33 proportion of single cycle length for subjects AB, LS, BA, and KW and 0.59 for subject EG) for both after-effects. The results also indicate that in three subjects (EG, LS, KW) there is a small decline of both after-effects at very short segment lengths (between 0.03 and 0.1). This might be due to the fact that short segment lengths are more broadband in their orientation composition, producing spurious orientations that mask the ‘signal’ orientations. 

In order to compare the size of the dip for different subjects and for the two after-effects, we normalized the data to the maximum sized after-effect for each subject. Figure 4b shows the average SFAE across the four subjects (AB, EG, LS, BA) and average SAAE across the four subjects (AB, EG, LS, KW). Figure 4b indicates that the size of the dip is approximately 0.25 of the maximum for the SFAE and 0.44 of the maximum for the SAAE. 

The experimental results in Figure 4, which showed that the position of the dip for most subjects is about 0.33 of the length along the path of a single cycle, allows us to constrain our model simulations if we make an important caveat. The model simulations are for a single curvature detector, whereas the after-effects we are dealing with presumably result from the operation of multiple curvature detectors, as explained above. The caveat is that any estimate of the length of the curvature detector, which is a product of the number and length of the sub-units, will likely represent an average value of a range of curvature detectors varying in either or both of length and number of sub-units. 

In Figure 3e we obtained plots showing the position of the dip as a function of the receptive-field length of the (average) curvature detector for different numbers of sub-units. To obtain estimates of curvature detector length one simply multiplies the slopes of the plots by 0.33 (the data). Figure 5a plots these estimates as a function of the number of sub-units. The plot in Figure 5a asymptotes at ∼0.65 of the length of a single cycle, for numbers of sub-units equal or greater than 4. Assuming 4 sub-units and a 0.65 asymptote, the length of a single sub-unit is 0.65/4 = 0.163.Figure 5b shows the portion of the adaptor (whose shape frequency and shape amplitude is the geometric mean of the two adaptors) captured by the curvature detector (assuming it has four sub-units and is centered in cosine phase). This estimate is comparable to that derived by a completely different method in a study by Gheorghiu and Kingdom (2007a): ∼0.55 cycle length as defined along the horizontal dimension or abscissa, which in turn corresponds to ∼0.61 of single cycle length along the contour. 

Figure 5

 

(a) The estimated length of the curvature detector as a function of the number of sub-units. (b) The estimated receptive field profile of the curvature detector assuming four sub-units.

In order to pursue the modeling further it is convenient to give the estimated length of the curvature detector in pixel units as in the model simulation. The segment length at the dip minimum of 0.33 is 1 log unit from the smallest segment length used in the experiment (0.033). Given that the smallest segment length in the model was 5 pixels, a value of 0.33 corresponds to 50 pixels for the dip position and an estimated curvature detector length of 100 pixels. Assuming 4 sub-units, this results in a sub-unit length s = 25 pixels. 

A comparison between the psychophysical data ( Figure 4) and the simulations ( Figure 3) shows that the dip in the data does not reach zero and is slightly broader than that predicted by the model. Both these features can be explained by supposing that during adaptation a range of curvature detectors of various receptive-field lengths are stimulated. Let us assume that the range of stimulated curvature detector lengths is ‘centered’ on the length estimated above: n = 4 sub-units of length s = 25 (see Figure 1g). The range of curvature detectors could receive input either from (a) the same number, n = 4 of afferent sub-units but with different sub-unit lengths, or (b) different numbers of sub-units having the same sub-unit length ( s = 25). Figure 6 shows both individual responses (dark gray symbols) as well as average responses (light gray symbols) for (a) 3 stimulated curvature detectors whose sub-unit lengths are s = 23, 25, and 27, respectively; (b) 4 stimulated curvature detectors whose sub-unit lengths are s = 20, 25, 30, and 35, respectively. As can be seen the dips in the combined average responses do not reach zero and are broader than the individual responses, in line with the data. 

Figure 6

 

Example individual responses (dark gray symbols) as well as average responses (light gray symbols) for (a) 3 stimulated curvature detectors whose sub-unit lengths are s = 23, 25, and 27; (b) 4 stimulated curvature detectors whose sub-unit lengths are s = 20, 25, 30, and 35. (c) Average responses from a range of curvature detectors, whose individual responses are from 100 pixel length curvature detectors with n = 4 sub-units of length s = 25, n = 3 sub-units of length s = 33.3, and n = 5 sub-units of length s = 20. As can be seen the dips in the combined responses do not reach zero and are broader than the individual responses, in line with the data.

A broader dip could also result from the average response of a range of curvature detectors with the same receptive-field length but which receive input from different combinations of number and length of sub-units. Figure 6c shows the average response (light gray symbols) from a range of curvature detectors, whose individual responses (dark gray symbols) are from 100 pixel length curvature detectors with n = 4 sub-units of length s = 25, n = 3 sub-units of length s = 33.3, and n = 5 sub-units of length s = 20. As can be seen the dips in the combined average responses do not reach zero and are broader than the individual responses, again in line with the data. 

Discussion

This study is to our knowledge the first psychophysical evidence that curvature detectors multiply the responses of their afferent inputs. Model simulations revealed an important feature of multiplication: a pronounced near-symmetric-shaped dip in the response of a curvature detector to segmented contours of intermediate segment length. We observed just such a dip psychophysically in two contour-shape after-effects induced by segmented contour adaptors. 

It is important to reiterate that the model simulations are not models of either the SAAE or SFAE. Assuming these after-effects are caused by internal gain changes to a sub-set of curvature detectors whose population response signals curvature, a sub-set that not only differs in receptive-field length but also curvature, the psychophysical data are clearly insufficient to constrain a fully fledged multi-filter model of the after-effects that would inevitably contain many free parameters. Thus our model simulations are ultimately qualitative not quantitative. Nevertheless, we were able to show that two features of the single-filter model simulation that did not accord with the psychophysical data, namely a dip that was both very narrow and reached a minimum of zero, could be dealt with simply by combining responses across a range of curvature receptive-field lengths. This is not to claim that this is the only explanation of these features of the data, merely that in principle it could be. 

Note that if curvature detectors linearly summed their sub-unit responses, a flat function of after-effect against segment length would be expected, since during adaptation the curvature detectors would receive, on average, the same amount of stimulation irrespective of segment length. 

In theIntroduction section we mentioned the end-stopped model of Dobbins et al. (1987, 1989) and argued that while end-stopping might be used for curvature discrimination, it is unlikely to be used for curvature encoding. Nevertheless it would be prudent to consider how well the Dobbins et al. model would predict our data. We simulated the model using the same form of input sub-units as in the multiplication model described above. In the simulation, the end-stopped detector received inputs from just two sub-units whose receptive fields were centered at the same location (i.e., they overlapped). The two sub-units had the same orientation but were of different size—40 and 120 pixels in length. Each sub-unit summed the energy (e.g., r.m.s. luminance contrast) of the contour that fell within its receptive field. End-stopping was computed by taking the difference between the responses of the two sub-units. The model prediction is shown in Figure 7. The figure shows that as segment length increases from zero, there is a very slight decrease in response, but the response is characterized primarily by being flat over most of the range of segment lengths. This is because on average the amount of excitation and inhibition between the two receptive fields will be constant as segment length increases. 

Figure 7

 

Simulation of the response of a version of the Dobbins et al. (1987) end-stopped model to the segmented adaptors. The end-stopped detector receives input from two sub-units whose receptive fields are centered at the same location (i.e., they overlap), have the same orientation but are of different size: RF₁ = 40 and RF₂ = 120 pixels. Each sub-unit sums the energy (e.g., r.m.s. luminance contrast) of the contour that falls within its receptive field. The end-stopping is computed by taking the difference between the responses of these two sub-units.

Are there other non-linear models that could produce a similar dip? The most basic type of nonlinearity is thresholding. Let us assume that the sub-units have a threshold ( T) such that below T their response is zero and above T their response is a linear function of stimulation. Simulations of linear summation of sub-unit responses are shown in Figures 8a and 8b. 

Figure 8

 

Simulations of a model in which a threshold T, defined as a proportion of the maximum response, is imposed on each sub-unit response prior to linear summation of sub-unit responses. T = 0.25 (white symbols), T = 0.35 (light gray symbols), and T = 0.5 (dark gray symbols). Results (a) as a function of the number of sub-units ( n = 2, 3, 4, and 5) for a constant sub-unit length ( s = 40). Note that the position of the dip does not change with increasing number of sub-units. (b) As a function of sub-unit length ( s = 40, 60, 80, and 100) for a constant number of sub-units ( n = 4). Increasing the length of the sub-units shifts the position of the dip toward longer segment lengths. For the combination of number and length of sub-units that produces a dip at comparable locations to the psychophysical data, simulations are shown for T = 0.55 (red) and T = 0.6 (blue).

T is expressed as a proportion of the maximum sub-unit response, for T = 0.25 (white), T = 0.35 (light gray), and T = 0.5 (dark gray). Figure 8a shows the results for different numbers of sub-units ( n = 2, 3, 4, and 5) with constant sub-unit length ( s = 40). Figure 8b shows the results for various sub-unit lengths ( s = 40, 60, 80, and 100) with a fixed number of sub-units ( n = 4). The simulations reveal that a sharp dip at an intermediate segment length only occurs for high thresholds ( T = 0.5). A threshold of 0.5 produces a dip comparable in size to that obtained experimentally for the SAAE (∼0.44) but not for the SFAE. However, as Figures 8a and 8b also show, the modeled shape of the response function for these high thresholds is asymmetric, with the left side of the function prominently reduced, unlike the experimental data. A high threshold, such as T ≥ 0.5, is also implausible. It would predict that after-effects would only be obtained at medium and high contrasts. In a previous study we found similar sized SFAEs with adaptor and test contrasts of 0.05, 0.15, and 0.45 (Gheorghiu & Kingdom, 2006). Therefore model simulations of the effects of thresholding, together with considerations of plausibility, argue against high thresholding as the cause of the dip in the after-effects we have observed. 

What of other nonlinearities besides thresholding? Other common nonlinearities that have been used to model early visual processing are rectification, clipping, divisive normalization, nonlinear summation, and a power function followed by summation. We have simulated all these nonlinearities and none show the characteristic response dip found with multiplication. 

Why multiplication? Some studies have argued that linear filtering, even if modified by common nonlinearities such as those described above, is unable to sufficiently distinguish between one- and two-dimensional stimuli, for example a straight versus a curved contour, because although these nonlinearities change the form of the output signal they never cause the output signal (i.e., response to a straight line) to disappear completely (Zetzsche & Barth, 1990—see their Figures 1a and 1b). Multiplication in a curvature detector will have the effect of sharpening the selectivity of the detector to curvature, and this will mean that fewer detector responses are needed to estimate curvature. One can also consider multiplication to be a form of grouping; in the case of curvature processing a means to link up local contour fragments that are parts of curves. 

How might multiplication be implemented? Curvature detectors might implement multiplication explicitly (e.g., direct multiplication of two signals: a · b, or via another mathematical operation that is equivalent to multiplication). For example, multiplication can be implemented via a Log–Exp transform  

$\mathrm{a}·\mathrm{b}=\exp \left(\log \right(\mathrm{a})+\log (\mathrm{b}\left)\right),$

(2)

which is believed to be one of the most basic computational operations in the nervous system (Mead, 1989). Another possibility is the well-known Babylonian trick (Resnikoff & Wells, 1973; Zetzsche & Barth, 1990): 

$\mathrm{a}·\mathrm{b}{=}^1{/}_4\left[\right(\mathrm{a}+\mathrm{b}{)}^2-(\mathrm{a}-\mathrm{b}{)}^2].$

(3)

 

The Babylonian trick underpins a number of motion models, for instance the motion-energy model of Adelson and Bergen (1985) and the elaborated Reichardt motion-detector model of van Santen and Sperling (1985, 1984). 

Appendix A

Examples of dynamic changes in the response of the curvature detectors for various random changes in the phase of segmentation and for different segment lengths ( Movies 1– 5). 

 

Movie 1

 

Segment length of 0.05 (% single cycle length).

 

Movie 2

 

Segment length of 0.1 (% single cycle length).

 

Movie 3

 

Segment length of 0.33 (% single cycle length).

 

Movie 4

 

Segment length of 1.06 (% single cycle length).

 

Movie 5

 

Segment length of 1.88 (% single cycle length).

Acknowledgments

This research was supported by a Natural Sciences and Engineering Research Council of Canada (NSERC) Grant OGP01217130 and a Canadian Institute of Health Research (CIHR) MOP-11554 Grant given to F.K. 

Commercial relationships: none. 

Corresponding author: Elena Gheorghiu. 

Email: [e-mail]. 

Address: McGill Vision Research, Department of Ophthalmology, McGill University, 687 Pine Avenue W., Rm. H4-14, Montreal H3A 1A1, Quebec, Canada.