Integration of sensory evidence in motion discrimination

Abstract

To make perceptual judgments, the brain must decode the responses of sensory cortical neurons. The direction of visual motion is represented by the activity of direction-selective neurons. Because these neurons are often broadly tuned and their responses are inherently variable, the brain must appropriately integrate their responses to infer the direction of motion reliably. The optimal integration strategy is task dependent. For coarse direction discriminations, neurons tuned to the directions of interest provide the most reliable information, but for fine discriminations, neurons with preferred directions displaced away from the target directions are more informative. We measured coarse and fine direction discriminations with random-dot stimuli. Unbeknownst to the observers, we added subthreshold motion signals of different directions to perturb the responses of different groups of direction-selective neurons. The pattern of biases induced by subthreshold signals of different directions indicates that subjects' choice behavior relied on the activity of neurons with a wide range of preferred directions. For coarse discriminations, observers' judgments were most strongly determined by neurons tuned to the target directions, but for fine discriminations, neurons with displaced preferred directions had the largest influence. We conclude that perceptual decisions rely on a population decoding strategy that takes the statistical reliability of sensory responses into account.

Introduction

To discriminate between simple sensory features, the brain relies on the activity of neurons that exhibit preferences for those features. The direction of visual motion is represented by the activity of neurons that respond differentially to different directions of motion. In primate visual cortex, these neurons, predominant in extrastriate visual area MT/V5 (Maunsell & van Essen, 1983; Salzman, Murasugi, Britten, & Newsome, 1992; Salzman & Newsome, 1994), have characteristic bell-shaped direction tuning curves (Britten & Newsome, 1998). A moving stimulus therefore elicits a corresponding bell-shaped population response across these neurons (Britten & Newsome, 1998; Hol & Treue, 2001; Treue, Hol, & Rauber, 2000). This population response is perturbed by variability that is approximately Poisson (Britten, Shadlen, Newsome, & Movshon, 1993). To discriminate between two directions of motion, the brain must decide which of the two alternatives is more likely to have elicited an observed noisy population response (Gold & Shadlen, 2001; Green & Swets, 1966). 

When the two directions to be discriminated are far apart, neurons that respond best to the two alternatives are most informative (Britten & Newsome, 1998; Shadlen, Britten, Newsome, & Movshon, 1996). Neurons with preferences near the two alternatives can also provide useful but less accurate information (Britten & Newsome, 1998; Shadlen et al., 1996). As a result, the appropriate decoding strategy is one that pools the activity of neurons with different preferences with a weighting profile that confers the highest weights to neurons tuned to the two alternatives (Jazayeri & Movshon, 2006). 

In contrast, when discriminating between two nearby directions of motion, neurons tuned to directions near the alternatives respond similarly to both and are not well suited to discriminate between them. Similarly, neurons tuned to directions very far from the two alternatives respond only weakly to either direction and do not provide useful information. Therefore, for fine discriminations, neurons with direction preferences moderately shifted to the sides of the discrimination boundary should be more strongly weighted (Hol & Treue, 2001; Jazayeri & Movshon, 2006; Purushothaman & Bradley, 2005; Regan & Beverley, 1985). In other words, although the weighting profile for discriminating remote directions has maxima at the two alternatives, for fine direction discriminations the brain should pool direction-selective neurons with a weighting profile that has maxima moderately shifted to the sides of the two alternatives (Jazayeri & Movshon, 2006). 

To examine how the visual system pools the activity of direction-selective neurons, we devised a psychophysical experiment in which subjects reported the direction of moving random-dot stimuli in either a coarse or a fine direction discrimination task. Unbeknownst to the subjects, we added subthreshold motion signals in directions other than the two alternatives to perturb the responses of direction-selective neurons. By varying the direction of the subthreshold motion, we reveal the relative contribution of motion signals in different directions to the subjects' choice behavior. The pattern of biases induced by different subthreshold motion signals indicates that for the discrimination of opposite directions, the visual system relies most strongly on the neurons tuned to the two alternatives, but for fine discriminations, neurons with preferences moderately shifted to the sides of the two alternatives have the greatest influence. 

Methods

Five subjects aged 19–35 years participated in this study after giving informed consent. All had normal or corrected-to-normal vision, and all were naive to the purpose of the experiment. Subjects viewed all stimuli binocularly on an Eizo T960 monitor at a refresh rate of 120 Hz driven by a Macintosh G5 computer in a dark, quiet room from a distance of 71 cm. 

In a single-interval two-alternative experimental design, subjects discriminated the direction of motion in a random-dot stimulus. Two experimental conditions were separately tested. In the D180 condition, the two alternatives were leftward and rightward directions, whereas in the D20 condition, they were 10° to the left and right of the upward direction. 

For both conditions, each trial began with the presentation of a fixation point. After 0.5 s, a static dot field was presented below fixation. After another 0.5 s, the dots began to move. Subjects were asked to keep fixation during the presentation of the motion stimulus. After 1 s of motion viewing, the stimulus was extinguished and subjects pressed one of two keys to report the direction of motion and received distinct auditory feedback for correct and incorrect judgments. 

All stimuli were presented on a dark grey background of 11 cd/m ². The fixation point was a central circular white point subtending 0.5° with a luminance of 77 cd/m ². In the D20 condition, two peripheral black bars 0.7° by 0.12°, located 1.6° below and 0.6° to the left/right of the fixation, cued the direction of the two alternatives. The motion stimulus was a field of dots (each dot 0.12° in diameter with a luminance of 77 cd/m ²) contained within a 5° circular aperture centered 5° below the fixation point. On successive video frames, some dots moved coherently in designated directions at a speed of 4 deg/s, whereas others were replotted at random locations within the aperture. The dot-field had an average density of 40 dots/deg ²/s. On every trial, 0%, 3%, 6%, 12%, or 25% of dots moved coherently in the direction of one of the alternatives (i.e., target signal), whereas another percentage of dots could carry a subthreshold motion signal in a direction other than the two alternatives. For the D180 condition, when the subthreshold was present (one condition with no subthreshold signal was also included), its direction could vary around the circle in steps of 30° ( Figure 1a). For D20, the subthreshold signal, if present, only included directions with no downward component ( Figure 1b). Considering all the variables controlled independently (e.g., coherence, stimulus motion direction, presence or absence of subthreshold motion, and its several possible directions), the D180 and the D20 tasks included a total of 110 and 80 randomly interleaved conditions, respectively. In both tasks, feedback was given only based on motion toward the two alternatives and not the subthreshold signals. 

Figure 1

 

Experimental design. (a) Coarse direction discrimination task, D180. Subjects viewed a field of moving random dots and indicated whether its direction was to the left or right (black arrows). On some trials, the stimulus also had subthreshold motion in directions other than the two alternatives (nonblack arrows). (b) Fine direction discrimination task, D20. The two alternatives were 10° to the left and right of upward direction. Subthreshold motion tested only included directions with no downward component. Note that in panel a, 0° is aligned to the right alternative, but in panel b, it marks the upward direction.

To determine the coherence at which the subthreshold signal was not detectable, we did preliminary measurements with a 2AFC motion detection task in which subjects simply reported the presence of motion. For each subject, the coherence at which his or her performance in the motion detection task was near chance (not better than 60% correct) was then used for the subthreshold signal throughout the main experiment. Conservatively, we also asked our subjects about their impression of the motion signal in the random-dot stimulus. Data for the one observer who reported noticing weak motion signals orthogonal to the discriminanda were excluded. 

In the main experiments, subjects completed roughly 5,000 trials in five to seven sessions each lasting less than an hour. Three subjects participated in each experiment. One subject completed both experiments. 

The addition of subthreshold signals might have various effects: (1) It might have no effect on subjects' choice behavior, (2) it could partially mask (or enhance) the target signals and alter subjects' sensitivity, or (3) it might change the overall evidence supporting one or the other alternative and bias the subjects' choices. To tease apart these possible effects, we adapted a standard receiver operating characteristic (ROC) analysis (Green & Swets, 1966) to our experimental design. We assigned one alternative direction to the presence of the signal (i.e., signal + noise) and the other alternative to the absence of signal (i.e., noise alone) and examined the effects of different subthreshold signals on each subject's sensitivity and bias. 

In the D180 experiment, data from pairs of subthreshold signals with directions symmetric with respect to the two alternatives (e.g., 30° and −30°) did not differ systematically and thus were pooled to obtain more reliable fits for the psychometric functions ( Figure 2). To ensure that our estimate of bias induced by different subthreshold signals was not contaminated by subjects' idiosyncratic response bias, individual bias terms were corrected for each subject's overall response bias (average bias across all conditions). When comparing the effect of different subthreshold motions in Figure 4, conditions symmetric with respect to the two alternatives were analyzed together. 

Figure 2

 

Direction discrimination performance in the presence of subthreshold motion. (a) Proportion of right choices (filled circles) and their standard errors (thin error bars) in the D180 experiment as a function of the percentage of coherently moving dots (0%, 3%, 6%, 12%, and 25%) and their corresponding logistic fits (solid lines) for one typical subject. Different colors show different directions of subthreshold motion as coded in the inset. Directions symmetric with respect to each alternative are grouped together (e.g., 30° and −30°). Positive and negative coherence values on the abscissa correspond to rightward and leftward motion, respectively. The filled black squares mark the expected horizontal shift in the psychometric functions had we added subthreshold signals in the direction of the two alternatives (the overall response bias is taken into account; for details, see Methods section). (b) Same as a, for the D20 discrimination task.

We computed standard errors for the hit and the false alarm rates for the ROC analysis ( Figure 3) and the direction-dependent effects of subthreshold signals ( Figure 4) using a bootstrap analysis with 1,000 resamples. 

Figure 3

 

The receiver operating characteristic (ROC) for direction discrimination tasks. (a) The hit and false alarm rates for the D180 condition for one typical subject. Various directions of the subthreshold motion are color-coded with different coherences shown in different symbol sizes as indicated in the inset. Data points corresponding to the two highest coherences are at the top-left corner where hit and false alarm rates are 1 and 0, respectively. The dashed lines show the iso-sensitivity curves with different amounts of bias assuming additive Gaussian noise. (b) The hit and the false alarm rates for the D20 condition with the same format as in panel a, for one typical subject. Higher coherences (smaller symbols) line up with higher iso-sensitivity curves and correspond to better performance (i.e., larger area under the iso-sensitivity curves). For any coherence (i.e., any given symbols size), different directions of subthreshold motion (i.e., different colors) seem to mainly change the bias along an iso-sensitivity curve.

Figure 4

 

Effect of motion signals in different directions in coarse and fine direction discrimination tasks. (a) Biases induced by subthreshold motion signals as a function of their direction shown for three subjects in different gray levels for the D180 discrimination task. The 0° and 180° marked by red arrows are cases in which no subthreshold motion was present and by construction are assigned to 1 and −1. The bias for various directions of subthreshold motion is measured in units of relative coherence (i.e., relative to the coherence for the 0 and 180 conditions). Conditions symmetric with respect to the two alternatives were analyzed together. (b) Same as panel a, for the D20 discrimination task. Here the 10° and −10° (red arrows) are conditions in which no subthreshold motion was present. Note that ordinates have different scales.

Results

Figures 2a and 2b show the choice behavior of two typical subjects in the D180 and the D20 conditions, respectively. Two features are notable. First, for both conditions, performance improved with increasing coherence and was on average better in the D180 condition. Second, although correct and incorrect choices were evaluated independently of whether and in which direction the subthreshold motion was presented, they were influenced by these signals in a direction-dependent manner. 

To characterize the effect of subthreshold signals on choices, we plotted each subject's hit and false alarm rates in an ROC diagram (for details, see Methods section). As exemplified in Figure 3, the main effect of the subthreshold motion was to bias choices without changing sensitivity. We therefore modeled each subject's choice behavior as a function of motion coherence with a logistic function with the same sensitivity and different bias terms for different directions of subthreshold motion. 

For the D180 condition, as the direction of injected motion approached the two alternatives, it caused a greater shift in the fitted psychometric functions ( Figure 2a, solid curves). Note, however, no subthreshold signal biased the subject to the extent that would be expected from subthreshold signals in the direction of the two alternatives ( Figure 2a, filled squares). 

For the D20 condition, however, the largest shift was caused by signals between 30° and 60° away from the discrimination boundary ( Figure 2b, solid curves). Notably the bias for these conditions was stronger than those expected from subthreshold signals in the direction of the two alternatives ( Figure 2b, filled squares). 

One useful way to compare the biases across conditions is to measure them in units of effective motion strength in the direction of the two alternatives. The bias in each condition can be quantified as the horizontal shift in the fitted psychometric functions; that is, the value of the abscissa for which performance is at chance. For instance, consider a subthreshold motion with 2% coherence that moves 30° away from one alternative. If we observe a shift in the psychometric function equivalent to 1% coherence along the abscissa, we would say that this subthreshold signal is half as effective as one with the same coherence that moves in the direction of one of the alternatives. 

Figure 4 shows the relative effect of subthreshold motion in different directions for the coarse and fine discrimination tasks. For the D180 condition, motion toward the two alternatives had the strongest influence on subjects' choices ( Figure 4a). The effect of the subthreshold signals progressively weakened as their direction was made more different from the two alternatives and was weakest for motion signals orthogonal to the axis of discrimination ( Figure 4a). 

For the D20 condition, subthreshold signals moving 20°–50° away from the two alternatives (i.e., 30°–60° away from the discrimination boundary) had the largest influence ( Figure 4b). There are differences among subjects' choice behavior. For instance, for two of our subjects, the biasing effect of subthreshold motion extended to 80° away from the two alternatives and was comparable to that for motion signals in the direction of the two alternatives. Intersubject differences are perhaps an indication of small idiosyncratic differences in integrating motion signals. 

There are modest differences in detection thresholds between cardinal and oblique directions (e.g., Gros, Blake, & Hiris, 1998), and we wondered whether these might contaminate our measure of how subthreshold signals with different directions are weighted. But for the D180 condition, the magnitude of the threshold difference is too small to have much impact on the shape of the bias function, and for the D20 condition, motion along the cardinal axes had little or no effect on choices and thus would not have been affected. The pattern of biases in Figure 4 cannot plausibly be attributed to variations in detection thresholds near the cardinal directions. 

Our sampling of subthreshold motions in 30° steps did not allow us to map precisely the relative effects of different motion signals on choices. We can therefore only conclude that for fine discriminations, signals from directions of motion between 30° and 60° away from the discrimination boundary had the largest influence on subjects' choices. 

Discussion

The utility of sensory information is measured by how well it can be put to use in the control of behavior. One elementary measure of this utility is how well the brain can transform neural responses in sensory representations into perceptual decisions. Here, we asked about this utility in the context of a direction discrimination task. To examine how the brain extracts the information about the direction of motion from its representation across direction-selective sensory neurons, we perturbed the responses of different groups of direction-selective neurons by injecting subthreshold motion signals and tested how those perturbations influenced the subjects' choice behavior (for other uses of perturbation analysis, see Graf, Warren, & Maloney, 2005; Hillis, Ernst, Banks, & Landy, 2002; Warren, Maloney, & Landy, 2004). We found that subthreshold motion signals, despite being irrelevant to the discrimination task, had a significant direction-dependent effect on the subjects' choice behavior. The results suggest that the visual system relies on information pooled across a wide range of direction-selective neurons, and that the relative contribution of different neurons to the pool depends on their preferred direction. 

Another interpretation of our results is that the visual system only relies on the activity of neurons tuned to the two alternatives and our direction-dependent effects are due to the broad tuning of these neurons. But several lines of reasoning argue against this idea. First, as we have argued elsewhere, this scheme is clearly suboptimal as it ignores information from neurons that are tuned to directions other than the two alternatives (Jazayeri & Movshon, 2006). Second, it fails to account for the observation that in fine discrimination tasks neurons tuned away from the two alternatives have the strongest contribution to the subjects' choice behavior (Hol & Treue, 2001; Jazayeri & Movshon, 2006; Purushothaman & Bradley, 2005). Third, direct recording from MT neurons in a study of random-dot motion detection in monkeys has shown that although the activity of neurons tuned to the expected direction have the highest positive correlation with monkey's choices, neurons tuned away from that direction also contribute to the detection behavior in a direction-dependent manner (Bosking & Maunsell, 2004). Fourth, microstimulation studies of area MT in monkeys involved in a direction identification task indicate that perception of motion relies on MT population response and not just the maximally activated neurons (Nichols & Newsome, 2002). 

Our finding is consistent with previous studies that have examined the nature of readout process in direction discrimination tasks. Recording from neurons in area MT of monkeys discriminating opposite directions of motion in random-dot stimuli has shown that the activity of neurons tuned to the two alternatives predicts the choices better than neurons tuned to directions orthogonal to the discrimination axis (Britten, Newsome, Shadlen, Celebrini, & Movshon, 1996). In contrast, when monkeys are engaged in a fine direction discrimination task, neurons with preferences quite far from the two alternatives most strongly determine the choices (Purushothaman & Bradley, 2005). 

Several studies have used electrical microstimulation as a way to perturb the activity of MT neurons to examine the readout strategy employed by the visual system. The results from these studies are, however, inconclusive. MT Microstimulation of MT suggests a population vector decoding mechanism (Georgopoulos, Kalaska, Caminiti, & Massey, 1982) during pursuit tracking (Groh, Born, & Newsome, 1997), a winner-takes-all mechanism in direction discrimination (Salzman et al., 1992; Salzman & Newsome, 1994), and a combination of the two in direction identification tasks (Nichols & Newsome, 2002). 

Although microstimulation is the most direct way of perturbing MT responses, two methodological issues complicate its interpretation (Cohen & Newsome, 2004). First, the perceptual effect of MT microstimulation is unknown. For instance, it is possible that microstimulation creates a motion percept of its own, which, when combined with a visual motion signal of a different direction, could create a different stimulus altogether, one composed of two superimposed directions of motion. Such stimuli could affect subjects' response strategy in unpredictable ways. Second, even if we assume that the subjective experience of motion and the subsequent response strategy remain unchanged because the effect of microstimulation on the profile of activity across MT neurons is unknown, one cannot predict its behavioral consequences. Our study can be compared to these microstimulation studies because we also perturbed the responses of MT neurons. But in our case, these two problems are reduced: First, by keeping the stimulation at a subthreshold level, we ensured that it would not influence the subjects' response strategy; and second, the response profile that a subthreshold motion would elicit is well characterized (Britten et al., 1993). It is also important to emphasize that both the population vector and the winner-takes-all mechanisms are clearly inconsistent with the decoding strategy that monkeys and humans use in fine direction discriminations (Hol & Treue, 2001; Purushothaman & Bradley, 2005). 

Another way to perturb the activity of sensory neurons is to reduce their responses using an adaptation paradigm. It has been shown that adapting the direction-selective neurons raises detection threshold for random-dot motion in a direction-dependent manner: Detectability is most impaired when the adaptor and the target directions are similar and less so when the adaptor and the target direction become progressively less similar (Hol & Treue, 2001). For fine direction discrimination on the other hand, the performance drops most effectively when the adaptor motion is in directions away from the two alternatives (Hol & Treue, 2001). But interpretation of adaptation studies for understanding the sensory decoding can be complicated by the fact that the adaptor might influence the response of neurons at multiple stages of visual processing. Moreover, prolonged adaptation, in addition to decreasing sensitivity, can shift the tuning properties of individual neurons (Kohn & Movshon, 2004) and change their variability (A. Kohn & J. A. Movshon, unpublished observations). 

Discrimination of motion requires a comparison between the activities of two pools of neurons preferring the two alternatives. For opposite directions, because the neuronal pools have little overlap in their tuning, a weak subthreshold motion in one direction would have little effect on the responses of neurons in the other pool and as such, as noted by theoretical studies of sensory decoding, the readout mechanism can be likened to motion detection tasks where the activation of a single pool of neurons has to be detected (Jazayeri & Movshon, 2006). Therefore, we expect the pooling strategy that subjects use for coarse direction discrimination to apply to the detection of direction of motion. Direct recording from MT neurons with different preferences during a motion detection task supports this conjecture (Bosking & Maunsell, 2004). 

When subjects discriminated between opposite directions of motion (D180 condition), their choice behavior was most heavily influenced by motion signals in the direction of the two alternatives. The effect of signals away from the two alternatives progressively weakened and was negligible for directions orthogonal to the axis of discrimination. In contrast, when judging the direction of motion in a fine discrimination task (D20 condition), motion signals in the direction of the two alternatives or very far from them had smaller effect than signals moderately shifted to the sides of the two alternatives. This suggests that the mechanism by which the activity of direction-selective neurons are decoded must be adaptable: For discrimination of opposite directions, the pooled signals must be centered at the two alternatives, but when discriminating nearby directions, the readout mechanism should automatically give the neurons with flanking preferences the highest weight. Thus, when decoding sensory responses, the brain integrates signals from different neurons according to their statistical reliability, so that the more reliable neurons for a given task more strongly determine the final choice. 

We have previously shown that a simple and a biologically plausible readout mechanism that appropriately integrates the activity of individual sensory neurons to compute sensory likelihoods would automatically generate such an adaptable readout strategy (Jazayeri & Movshon, 2006). Our results provide qualitative support for the computations proposed in that model, which in particular predicts the shift in weighting of sensory inputs for different discrimination tasks. Our findings here differ in detail from the predictions of this model. In particular, we found that the most important neurons for fine discrimination preferred directions between 30° and 60° from the discrimination boundary, whereas the model predicted a somewhat larger displacement. This prediction depends on the assumptions made about the tuning of the cortical neurons providing input to the decoding stage of the model. The discrepancy can be resolved within the framework of the model by supposing that the direction-selective neurons our observers used for fine direction discriminations were more narrowly tuned than those we used in the model. 

The statistically efficient population decoding mechanism we propose seems to contrast with the traditional idea that perceptual performance depends on “labeled-line” detectors (e.g., Barlow, 1972). Such models propose that detection, discrimination, and identification of sensory features are mediated by the detectors that are most sensitive to those features; support for this idea comes from results such as those of Watson and Robson (1981), who showed that subjects can detect and discriminate some stimuli with equal accuracy. But the two models are not so different as they might first appear. In our view, the activation of labeled lines directly supports coarse discriminations, but in fine discriminations, observers shift their decoding strategy to optimize performance. It is therefore to be expected that equal performance in detection and discrimination tasks would be observed for coarse discriminations but not for fine ones, which is the pattern of results observed empirically (Watson & Robson, 1981) and predicted from theory (Jazayeri & Movshon, 2006). 

Acknowledgments

This research was supported by an NIH grant to J. A. M. (R01 EY02017). 

Commercial relationships: none. 

Corresponding author: Mehrdad Jazayeri. 

Email: [e-mail]. 

Address: Howard Hughes Medical Institute, National Primate Research Center, and Department of Physiology & Biophysics, University of Washington, Seattle, WA 98195, USA.