Adaptive shape coding for perceptual decisions in the human brain

Abstract

In its search for neural codes, the field of visual neuroscience has uncovered neural representations that reflect the structure of stimuli of variable complexity from simple features to object categories. However, accumulating evidence suggests an adaptive neural code that is dynamically shaped by experience to support flexible and efficient perceptual decisions. Here, we review work showing that experience plays a critical role in molding midlevel visual representations for perceptual decisions. Combining behavioral and brain imaging measurements, we demonstrate that learning optimizes feature binding for object recognition in cluttered scenes, and tunes the neural representations of informative image parts to support efficient categorical judgements. Our findings indicate that similar learning mechanisms may mediate long-term optimization through development, tune the visual system to fundamental principles of feature binding, and optimize feature templates for perceptual decisions.

Introduction

Evolution and development have been proposed to shape the organization of the visual system and result in structural neural codes that support visual recognition (Gilbert, Sigman, & Crist, 2001; Simoncelli & Olshausen, 2001). However, long-term experience is not the only means by which sensory processing becomes optimized. Learning through everyday experiences in adulthood has been shown to be a key facilitator for a range of tasks, from simple feature discrimination to object recognition (Fine & Jacobs, 2002; Goldstone, 1998; Schyns, Goldstone, & Thibaut, 1998). Here, we review how long-term experience and short-term training may interact to shape the optimization of visual recognition processes. While long-term experience hones the principles of organization that mediate feature grouping for object recognition, short-term training may establish new principles for the interpretation of natural scenes. Here, we summarize our recent work combining behavioral and brain imaging measurements to investigate the role of experience in optimizing neural coding for perceptual decisions. In particular, we focus on the role of learning in perceptual judgements under sensory uncertainty, such as when detecting stimuli presented in cluttered backgrounds or discriminating between highly similar stimuli. We ask: what are the learning mechanisms that mediate our ability to detect targets in cluttered scenes and assign new objects to meaningful categories? 

Learning-dependent mechanisms for visual recognition in cluttered scenes

Recognizing meaningful objects entails integrating information at different levels of visual complexity from local contous to complex features independent of image changes (e.g., changes in position, size, pose, or background clutter). To achieve this challenging task, the brain is thought to exploit a network of connections that support the integration of object features based on image regularities that occur frequently in natural scenes (e.g., orientation similarity between neighboring elements; Geisler, 2008; Sigman, Cecchi, Gilbert, & Magnasco, 2001). For example, long-term experience with the high prevalence of collinear edges in natural environments (Geisler, 2008; Sigman et al., 2001) has been shown to result in enhanced sensitivity for the detection of collinear contours in clutter. However, recent work highlights the role of shorter term training in feature binding and visual recognition in clutter. For example, observers have been shown to learn distinctive target features by exploiting regularities in natural scenes and suppressing background noise (Brady & Kersten, 2003; Dosher & Lu, 1998; Eckstein, Abbey, Pham, & Shimozaki, 2004; Gold, Bennett, & Sekuler, 1999; Li, Levi, & Klein, 2004). In particular, learning has been suggested to enhance the correlations between neurons responding to the features of target patterns while decorrelating neural responses to target and background patterns. As a result, redundancy in the physical input is reduced and target salience is enhanced (Jagadeesh, Chelazzi, Mishkin, & Desimone, 2001) supporting efficient detection and identification of objects in cluttered scenes (Barlow, 1990). 

Further, our recent behavioral studies show that short-term experience in adulthood may modify the behavioral relevance (i.e., utility) of atypical contour statistics for the interpretation of natural scenes (Schwarzkopf & Kourtzi, 2008). Although collinearity is a prevalent principle for perceptual integration in natural scenes, we demonstrated that the brain can learn to exploit other image regularities for contour linking (Schwarzkopf & Kourtzi, 2008). In particular, observers learned to use discontinuities typically associated with surface boundaries (e.g., orthogonal or acute alignments) for contour linking and detection. Observers were trained with or without feedback to judge which of the two stimuli presented successively in a trial contained contours rather than only random elements (two-interval forced-choice task). Behavioral performance (i.e., detection accuracy) improved for the trained contours, and this improvement was maintained for a prolonged period (i.e., testing 6 to 8 months after training), suggesting optimization of perceptual integration processes through experience rather than simply transient changes in visual sensitivity. Thus, these findings provide evidence that short-term experience boosts the observers' ability to detect camouflaged targets by shaping the behavioral relevance of image statistics. 

Our recent studies combining behavioral and brain imaging measurements (Zhang & Kourtzi, 2010) indicate two routes to visual learning in clutter with distinct signatures of brain plasticity (Figure 1). We show that long-term experience with statistical regularities (i.e., collinearity) may facilitate opportunistic learning (i.e., learning to exploit image cues), while learning to integrate discontinuities (i.e., elements orthogonal to contour paths) entails bootstrap-based training (i.e., learning new features) for the detection of contours in clutter. In particular, observers were trained on contour detection (two-interval forced-choice task) with auditory error feedback, or simply exposed to contour stimuli while performing a contrast-discrimination task (i.e., judging stimulus contrast) unrelated to contour detection. Learning to integrate collinear contours was shown to occur through frequent exposure, generalize across untrained stimulus features, and shape processing in higher occipitotemporal regions implicated in the representation of global forms. In contrast, learning to integrate discontinuities (i.e., elements orthogonal to contour paths) required task-specific training (bootstrap-based learning), was stimulus-dependent, and enhanced processing in intraparietal regions implicated in attention-gated learning. This is consistent with neuroimaging studies showing that a ventral cortex region becomes specialized through experience and development for letter integration and word recognition (Dehaene, Cohen, Sigman, & Vinckier, 2005), while parietal regions are recruited for recognizing words presented in unfamiliar formats (Cohen, Dehaene, Vinckier, Jobert, & Montavont, 2008). Taken together, these findings suggest that opportunistic learning of statistical regularities shapes bottom-up object processing in occipitotemporal areas, while learning new features and rules for perceptual integration recruits parietal regions involved in the attentional gating of recognition processes. 

Figure 1

 

Learning to detect contours in cluttered scenes. (A) Examples of stimuli: Collinear contours in which elements are aligned along the contour path and orthogonal contours in which elements are oriented at 90° to the contour path. For demonstration purposes only, two rectangles illustrate the position of the two contour paths in each stimulus. (B) Average behavioral performance across subjects (percent correct) before and after training for collinear and orthogonal contours presented at trained and untrained orientations. (C) fMRI responses for observers trained with orthogonal versus collinear contours. fMRI data (percent signal change for contour minus random stimuli) are shown for trained contour orientations before and after training on orthogonal versus collinear contours. Adapted with permission from Zhang and Kourtzi (2010).

Learning-dependent changes for shape categorization

Assigning novel objects into meaningful categories is critical for successful interactions in complex environments (Miller & Cohen, 2001). Extensive behavioral work on visual categorization (Goldstone, Lippa, & Shiffrin, 2001; Nosofsky, 1986; Schyns et al., 1998) suggests that the brain solves this challenging task by representing the relevance of visual feature for categorical decisions rather than simply the physical similarity between objects. Neuroimaging studies have identified a large network of cortical and subcortical areas in the human brain that are involved in visual categorization (Ashby & Maddox, 2005; Keri, 2003) and have revealed a distributed pattern of activations for object categories in the temporal cortex (Hanson, Matsuka, & Haxby, 2004; Haxby et al., 2001; O'Toole, Jiang, Abdi, & Haxby, 2005; Williams, Dang, & Kanwisher, 2007). 

Yet, using fMRI to isolate the neural code for flexible category learning is fraught with complexity. First, learning-dependent changes measured by BOLD may reflect different underlying changes in neural selectivity (i.e., sharpening of neuronal tuning to a visual stimulus): There may be enhanced responses to the preferred stimulus, decreased responses to nonpreferred stimuli, or a combination of the two. A number of studies have shown that learning changes the overall responsiveness (i.e., increased or decreased fMRI responses) of visual areas to trained stimuli (Kourtzi, Betts, Sarkheil, & Welchman, 2005; Mukai et al., 2007; Op de Beeck, Baker, DiCarlo, & Kanwisher, 2006; Sigman et al., 2005; Yotsumoto, Watanabe, & Sasaki, 2008). However, considering only the overall magnitude (i.e., univariate), fMRI signal does not allow us to discern whether learning-dependent changes in fMRI relate to changes in the overall magnitude of neural responses or changes in the selectivity of neural populations. Second, to quantify changes in (a) perceptual performance and (b) brain responses due to learning, it is useful to vary stimuli parametrically and test changing levels of performance. However, simple univariate fMRI measures may be insufficient to reveal subtle changes in neuronal responses and the links between perceptual and neuronal states. Here, we discuss recent work using multivoxel pattern classification methods that take into account activity across patterns of voxels to investigate the link between learning-dependent changes in neural representations and behavior. 

In a first set of studies (Zhang, Meeson, Welchman, & Kourtzi, 2010), we tested the mechanisms that mediate our ability to learn visual shape categories by combining behavioral judgments with high-resolution imaging (i.e., smaller voxels for finer grained measurements) and multivoxel pattern classification analysis tools (Haynes & Rees, 2005; Kamitani & Tong, 2005). To gain insight into the way in which visual form representations change with training, we employed a parametric stimulus space in which we could morph systematically between two different stimulus classes (concentric vs. radial glass patterns; Figure 2A). By adding external noise to the displays, and using fine-scale variations in the morphing space, we were able to characterize observers' behavioral discrimination performance before and after training (Figure 2B). Participants were trained to discriminate concentric from radial patterns with feedback in three training sessions, and were scanned on the same task (without feedback) before and after training. Based on fMRI measurements concurrent with stimulus presentation, we were able to evaluate the ability of the machine-learning algorithm to decode the presented stimuli. We were particularly interested in the choices of the pattern classifier in predicting each of the presented stimuli. Using the distribution of choices for each stimulus we generated fMRI-based voxel tuning functions that described the distribution of choices made by the classifier when given data measured in different visual areas (Figure 2C). Thereby we sought to link learning-based changes in behavioral responses and fMRI responses by comparing psychometric functions and fMRI pattern-based tuning functions before and after training. 

Figure 2

 

Learning to discriminate global shapes. (A) Stimuli: Low- and high-signal Glass pattern stimuli. (B) Behavioral performance (average data across observers) across stimuli in the morphing space (as indicated by spiral angle) during scanning. The curves indicate the best fit of the cumulative Gaussian function for high-signal stimuli and low-signal stimuli before and after training. Error bars indicate the standard error of the mean. (C) fMRI pattern-based tuning functions for V1, LO, and V3B/KO: The proportion of predictions made to each stimulus condition in terms of the difference in spiral angle between the viewed stimulus and the prediction. Symbols indicate average data across observers; solid lines indicate the best fit of a Gaussian to the data from 1,000 bootstrap samples. Adapted with permission from Zhang et al. (2010).

Comparing the performance of human observers and classifiers demonstrated that learning altered the observers' sensitivity to visual forms and the tuning of fMRI activation patterns in visual areas selective for task-relevant features. For high signal stimuli (i.e., 80% of the dots in the display were aligned to the stimulus shape, while 20% were randomly positioned), we observed a tuned response across visual areas, that is, the classifiers mispredicted similar stimuli more frequently than dissimilar ones. Consistent with the behavioral results, these tuning functions had higher amplitude for high-signal stimuli than low-signal stimuli (i.e., only 45% of the dots in the display were aligned to the stimulus shape) before training in higher visual areas. However, training on low-signal stimuli enhanced the amplitude while it reduced the width of pattern-based tuning functions in higher dorsal and ventral visual areas (Figure 2C). Increased amplitude after training indicates higher stimulus discriminability that may relate to enhanced neural responses for the preferred stimulus category at the level of large neural populations. Reduced tuning width after training indicates fewer classification mispredictions, suggesting that learning decreases neural responses to nonpreferred stimuli. Thus, these findings suggest that learning of visual patterns is implemented in the human visual cortex by enhancing the response to the preferred stimulus category, while reducing the response to nonpreferred stimuli. 

Learning-dependent changes for optimal decisions

Learning is known to optimize mental templates of object features that are critical for efficient performance in perceptual tasks (e.g., categorical discrimination; Dobres & Seitz, 2010; Dupuis-Roy & Gosselin, 2007; Gold et al., 1999; Li et al., 2004). In recent work (Kuai, Levi, & Kourtzi, 2013), we investigated the mechanisms that the brain uses to optimize templates for perceptual decisions through experience. Observers were trained (three training sessions with feedback) to discriminate visual forms (two classes of polygons) that were randomly perturbed by noise (Figure 3A) and were scanned on the same task (without feedback) before and after training. To identify the specific stimulus components that determine the observer's choice (i.e., the discriminative features), we reverse-correlated behavioral choices and fMRI signals with noisy stimulus trials. This approach has been widely used in psychophysics (for reviews see Abbey & Eckstein, 2002; Eckstein & Ahumada, 2002); however, its application to neuroimaging has been limited by noisy single-trial fMRI signals and the small number of samples that can be acquired during fMRI scans (Schyns, Gosselin, & Smith, 2009; Smith et al., 2008). To overcome these limitations, we used reverse correlation in conjunction with multivoxel pattern analysis. We calculated decision templates based on the choices made by a linear support vector machine (SVM) classifier that decodes the stimulus class from the fMRI data measured on individual stimulus trials. By reverse correlating behavioral and multivoxel pattern responses with noisy stimulus trials, we identified the critical image parts that determine the observer's choice. 

Figure 3

 

Learning optimizes decision templates. (A) Sample pentagon-like stimuli comprising 30 equally spaced Gaussian dots with 0.1° SD. Two classes of shapes were generated by varying the location of the pentagon lines that differed in their length. The top panel shows the stimulus space generated by linear morphing between Class I and Class II polygons (stimuli are shown as a function of percent Class II). The bottom panel shows example stimuli with position noise, as presented in the experiment. (B) Behavioral classification images before and after training, averaged across participants. (C) fMRI classification images for V1 and LO before and after training, averaged across participants. Red indicates image locations associated with a Class I decision, while blue indicates image locations associated with a Class II decision. Adapted with permission from Kuai et al. (2013).

Our findings showed that observers learned to integrate information across locations and weight the discriminative image parts. Classification images based on the observers' performance after training showed marked differences between image parts associated with the two stimulus classes (Figure 3B). In contrast, we did not observe any consistent image parts associated with the two stimulus classes before training, ensuring that the classification images reflected the perceived differences between classes rather than local image differences between stimuli. 

Having characterized the behavioral decision template, we used fMRI to determine where in the visual cortex this template is implemented. Our results show that classification images in the lateral occipital (LO) area, but not in early visual areas, revealed image parts that were perceptually distinct between the two stimulus classes (Figure 3B). Importantly, there was little information about this perceptual template before training, ensuring that fMRI classification templates reflect the perceived classes rather than stimulus examples. Comparing classification images derived from behavioral and fMRI data showed that similar image parts became more discriminable between the two stimulus classes after training, suggesting correspondence between behavioral and fMRI templates. Further, we found that these templates were also tolerant of stimulus size changes. In particular, after training, we tested observers' performance for stimuli that were presented at different sizes (1.5 or 2 times larger) from the trained stimuli. Our findings showed that behavioral and fMRI classification images for stimuli of trained and untrained size were highly similar, suggesting that learning tunes representations of discriminative image parts in higher ventral cortex that are tolerant to image changes rather than specific local image positions. 

Summary

Understanding the midlevel representations that support shape perception remains a challenge in cognitive neuroscience and computer vision. Computational and experimental approaches (Marr, 1982; Pasupathy & Connor, 2002; Wilson & Wilkinson, 2014) are converging in uncovering how object structure is encoded in the human visual cortex. Yet, object representations are surprisingly adaptive to changes in environmental statistics, implying that learning through evolution, development, but also shorter-term experience during adulthood, may shape the object code. Here we propose that adaptive shape coding in the visual cortex is at the core of flexible decision making. We demonstrate that learning optimizes feature binding for object recognition in cluttered scenes, and tunes the neural representations of informative image parts for efficient categorical judgements. Thus, understanding the brain dynamics (i.e., interactions between large-scale networks and local cortical circuits) that support adaptive neural processing in the human brain emerges as the next key challenge in visual neuroscience. 

Acknowledgments

This work was supported by a Wellcome Trust Senior Research Fellowship to AEW (095183/Z/10/Z) and grants to ZK from the Biotechnology and Biological Sciences Research Council (H012508), a Leverhulme Trust Research Fellowship (RF-2011-378) and the People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/2007-2013/ under REA grant agreement no. PITN-GA-2011-290011. 

Commercial relationships: none. 

Corresponding author: Zoe Kourtzi. 

Email: [e-mail]. 

Address: Department of Psychology, University of Cambridge, Cambridge, UK.