Preferential responses to occluded objects in the human visual cortex

Abstract

How do we see an object when it is partially obstructed from view? The neural mechanisms of this intriguing process are unclear, in part because studies of visual object perception heretofore have largely used stimuli of individual objects, such as faces or common inanimate objects, each presented alone. But in natural images, visual objects are typically occluded by other objects. Computational studies indicate that the perception of an occluded object requires processes that are substantially different from those for an unoccluded object in plain view. We studied the neural substrates of the perception of occluded objects using functional magnetic resonance imaging (fMRI) of human subjects viewing stimuli that were designed to elicit or not elicit the percept of an occluded object but were physically very similar. We hypothesized the regions that are selective for occluded objects, if they exist, will be differentially active during the two conditions. We found two regions, one in the ventral object processing pathway and another in the dorsal object processing pathway, that were significantly responsive to occluded objects. More importantly, both regions were significantly more responsive to occluded objects than to unoccluded objects, and this enhanced response was not attributable to low-level differences in the stimuli, amodal completion per se, or the behavioral task. Our results identify regions in the visual cortex that are preferentially responsive to occluded objects relative to other stimuli tested and indicate that these regions are likely to play an important role in the perception of occluded objects.

Introduction

The neural mechanisms by which we perceive visual objects have long been a mystery. But recently, great strides have been made in understanding the neural mechanisms of visual object perception (see Grill-Spector & Malach, 2004). Human functional magnetic resonance imaging (fMRI) studies have revealed numerous regions of the extrastriate cortex, the part of the visual cortex involved in the higher order processing of image information. Many of the extrastriate regions are thought to be selective for various object categories. For instance, the human temporal cortex is known to contain regions selectively responsive to faces, body parts, inanimate objects, and other object categories (Grill-Spector, Knouf, & Kanwisher, 2004; Grill-Spector, Kourtzi, & Kanwisher, 2001). 

So far, neural mechanisms of visual object perception have been studied mainly using individual visual objects presented alone (e.g., a face, or a hammer, by itself in the absence of occlusion or background clutter). However, visual objects rarely occur in isolation in natural visual scenes. For one thing, it is common for one object to obstruct, or occlude, another in natural images. Therefore, understanding how we see occluded objects is critical to understanding how we see under natural viewing conditions. 

The mechanisms by which we perceive occluded objects remain poorly understood, although many studies have used stimuli that featured occluded objects (Fang & He, 2005; Lerner, Hendler, & Malach, 2002; Murray, Kersten, Olshausen, Schrater, & Woods, 2002; Olson, Gatenby, Leung, Skudlarski, & Gore, 2004; also see Discussion). One previous study that directly addressed this issue, Lerner et al. (2002), found that the responses to occluded objects in a key object-selective region in the ventral object processing pathway, lateral occipital complex (LOC), is intermediate between the responses to unoccluded objects and scrambled objects. This appears to be consistent with the fact that when an object is occluded, the corresponding retinal image contains less explicit information about the object than when the object is unoccluded but more shape information than the image of a scrambled object. Responses to static occluded objects have not been reported in the dorsal pathway (see Discussion), although many regions in this pathway, notably a broadly defined region referred to as the dorsal object-selective foci or dorsal foci (DF), are known to be responsive to unoccluded objects (Gilaie-Dotan, Ullman, Kushnir, & Malach, 2002; Grill-Spector, Kushnir, Hendler, & Malach, 2000). 

However, psychophysical and computational studies indicate that the perception of an object partially occluded by other object/s entails additional information processing steps not involved in perceiving an unoccluded intact object (Nakayama, Shimojo, & Silverman, 1989; Yuille & Kersten, 2006). In order to perceive an occluded visual object, the visual system must identify object boundaries, distinguish actual boundaries of a given object (“intrinsic” boundaries) from those resulting from occlusion (“extrinsic” boundaries), assign the resulting partial views (or surfaces) to individual objects and amodally fill-in, or “explain away” the missing information about the occluded parts of a given object of interest (see 1 for a detailed analysis of the requisite computational steps). The fact that the perception of occluded objects requires additional information processing steps raises the possibility that the perception of occluded objects may be correlated with neural activity distinctly different from activity associated with an unoccluded object, a notion also supported by previous electroencephalographic (EEG) and fMRI studies (see Johnson & Olshausen, 2005; Murray, Foxe, Javitt, & Foxe, 2004; Murray, Imber, Javitt, & Foxe, 2006; also see Rauschenberger, Liu, Slotnick, & Yantis, 2006). Note that perceiving an occluded object from partial views is distinctly different from being aware or conscious of an object that is rendered invisible by an occluder (Hulme & Zeki, 2007), which the present study does not address. 

In this study, we focus on brain activity associated with the overall process of completing a single whole object using partial views arising from occlusion. To do this, we used a stereoscopic manipulation in which an occluded object is perceived when the occluder is seen as being in front of the object but not when the same “occluder” is seen as being behind the object (Nakayama et al., 1989). In this case, the two conditions are physically very similar (although not identical; see Methods for details). Nonetheless, only one of the two conditions supports the perception of an occluded object and other one does not, which we experimentally verified for our stimuli. We hypothesized the regions that are selective for occluded objects, if they exist, will be differentially active during the two conditions. Our results not only reveal a hitherto unknown region in DF responsive to occluded objects, but also show that subregions within LOC and DF are more responsive to occluded objects than to their unoccluded counterparts. 

Methods

Subjects

Twelve adult subjects (seven females; two left-handed; age range, 19–30 years) participated in this study. All subjects had normal or corrected-to-normal vision and had no known neurological or visual disorders. All subjects gave informed consent prior to participating in the study. All protocols used in the study were approved in advance by the Internal Review Board of the University of Minnesota. 

Stimuli

Rationale

Ideally, one would like to compare the brain activity when the same given image of a partially occluded object elicits vs. does not elicit the percept of a whole object. Nakayama et al. (1989) have shown that although this is not feasible, small manipulations in the perceived depth order of the occluder can lead to the above two contrasting percepts depending on whether the occluder is perceived as in front of or behind the same partial views of the occluded object. Thus, the stimuli that lead to the two percepts are very similar but not identical. Our stimulus set included this pair of conditions, along with three other pairs of stimuli designed to control for the stimulus-driven differences in percepts. 

Stimulus set

The stimulus set consisted of eight stimulus conditions ( Figure 1A; see 1 for additional info). Condition 1 was constructed by placing an occluder (an orthonormal planar surface with random, irregular holes in it) in the near depth plane in front of a luminance-defined target object (a “vase-like” object), so that the object was partially visible through the holes in the occluder. The vase-like objects were surfaces of revolution created using the Canvas graphical toolkit (ACD Systems of America, Miami, FL). Different vases varied in their profile and viewpoint (see, e.g., Figures 1A and 1C). In condition 2, the same partial views of the target object were provided, except that the “occluder” was placed in the far depth plane. Solely for convenience, we will refer to the orthonormal planar surface as the occluder in all conditions regardless of its depth plane, although strictly speaking, an occluder is always in the near plane. In either condition, the depth plane of the occluder was specified using stereoscopic disparity (±1 arcmin for 3 subjects, ±20 arcmin for the remaining subjects). The target object had zero disparity in all cases and had no depth cue other than luminance (i.e., shading). The fMRI data obtained using the two sets of occluder disparities were statistically indistinguishable (2-tailed Mann-Whitney test, p > 0.05; not shown). Condition 1 elicited the object completion percept, so that a connected object was perceived behind the occluder ( Figure 1B, left). But when the occluder is in the far depth plane, the same partial views of the object fail to elicit a filled-in percept, instead yielding a percept of a collection of disconnected object patches hovering in depth ( Figure 1B, right). 

Figure 1

 

(A) The stimulus conditions used in the study. The numbers denote the condition numbers, and the icons illustrate an exemplar stimulus for the corresponding condition. The only condition that elicits the percept of an occluded object (condition 1) is denoted by an asterisk in this and subsequent figures. The stimuli should be viewed with red-green anaglyph glasses with the green filter on right. For free-fusion versions of conditions 1 and 2, see 1. (B) Schematic illustration of the interpretations elicited by the first two conditions. The object is perceived as a coherent whole when the occluder is in the near depth plane ( left) but not in the far depth plane ( right), although the two stimuli provide the same partial views of the object. The stimulus in the right panel appears like a collection of disconnected object patches hovering in depth. (C) The blocked design of the fMRI scan. See Methods for details.

The remaining conditions were controls, including conventional intact and scrambled target objects (conditions 3 and 4, respectively), partially occluded scrambled target objects with the occluder in the near- or the far depth plane (conditions 5 and 6), and the near- or the far occluders alone (conditions 7 and 8). Note that conditions 1 vs. 5 and 2 vs. 6 contain the same pairwise disparity and monocular cues to occlusion (see Howard & Rogers, 2002). 

Multiple different stimuli were constructed for each condition using all possible combinations of appropriate object and/or occluder, drawn from a repertoire of 40 different objects (and their scrambled counterparts) and five different occluders each in near and far depth planes. Stimuli were synthesized off-line using the Matlab utility (Mathworks, Natick, MA) and stored on disk. During the actual experiments, the stimulus corresponding to a given condition was drawn randomly without replacement from the corresponding stimulus pool. 

Psychophysical experiments

To ascertain that the stimuli elicited the intended percepts described above, we carried out three psychophysical experiments that were identical to each other and to the fMRI experiment except as noted otherwise. Experiment 1 used a two-alternative forced-choice (2AFC) paradigm. Each stimulus subtended 9° × 9° and was presented against a neutral gray background. During each trial two stimuli, one corresponding to condition 1 and the other corresponding to condition 2 (see Figure 1) were presented sequentially in random order for 0.5 s each with an interstimulus interval of 0.1 s. Subjects performed a 2AFC task in which they reported, using a key press, which one of the two simultaneously presented stimuli contained a coherent object. Within any given trial, only the disparity of the occluder (i.e., near vs. far) differed between the two stimuli. All other stimulus parameters, including the occluded object, the occlude, and the absolute magnitude of disparity, were identical between the two stimuli in any given trial. The object and the occluder during any given trial were randomly drawn from the same repertoire as that used during the fMRI scans (see below). The disparity of the occluder was ±0, 1, 5, 10, 20, or 30 arcmin, depending on the trial. The performance of the subject at each disparity was measured over 40 randomly interleaved trials. Subjects underwent practice trials until they were ready to begin the actual trials. Data from the practice trials were discarded. No feedback was provided. After the experiments, subjects were debriefed so as to obtain additional, albeit qualitative, confirmation that they had experienced the intended percepts. 

In Experiment 2a, stimuli corresponding to conditions 5 and 7 were presented one per trial in random order. Subjects were required to report whether or not they perceived a single amodally completed occluded object. Experiment 2b was the same as Experiment 2a, except that in this case, the subjects reported whether the occluded object was behind or in the same depth plane as the occluder. 

Eight of the 12 subjects participated in Experiment 1, and four subjects participated in Experiments 2a and 2b. 

MRI scans

Stimuli were back-projected via a video projector (refresh rate, 60 Hz) onto a translucent screen placed inside the scanner bore. Subjects viewed the stimuli through a mirror located above their eyes. Functional MRI data were collected using a 3T Siemens Trio scanner with an eight-channel head array coil. Blood oxygen level-dependent (BOLD) signals were measured with an EPI (echo-planar imaging) sequence (TE: 30 ms, TR: 2000 ms, FOV: 22 × 22 cm ², matrix: 64 × 64, flip angle: 75°, slice thickness: 3 mm, number of slices: 24, slice orientation: axial). The bottom slice was positioned at the bottom of the temporal lobes. T1-weighted structural images at the same slice locations and high-resolution 3D structural data set (3D MPRAGE; 1 × 1 × 1 mm ³ resolution) were also collected in the same session before the functional runs. 

The scans were carried out using a blocked design with 20-s stimulus blocks separated by 20 s interstimulus blocks ( Figure 1C). During each stimulus block, stimuli corresponding to a given condition, drawn randomly from the corresponding stimulus repertoire described above, were presented for 500 ms each one after another without intervening gaps. Each stimulus subtended 9.4° × 9.4° and was presented against a neutral gray background. All eight conditions were used during each scan, and all scans (including the retinotopic mapping, see below) were completed in a single session. 

Subjects viewed the stimuli using red-green anaglyph glasses. Subjects were instructed to fixate the fixation spot during the interstimulus blocks. In keeping with many earlier studies of object perception (Grill-Spector et al., 2001, 2004; also see Joseph, Partin, & Jones, 2002; Grill-Spector & Malach, 2004), eye movements were allowed during stimulus blocks, and the subjects were instructed to carefully observe the shape of the object in the stimulus. After the scan, we verified for each subject that the stimuli had elicited their intended percepts. 

Each scan lasted 5 min 40 s, and the subjects were allowed to rest briefly if necessary between scans. The scans were repeated six times each for the first eight subject (who also participated in retinotopic scans, see below) and 4 times each for the last four subjects (who also participated in control Experiment 1, see below). Thus, each scanning experiment lasted about 50–60 minutes, depending on the subject. 

Control experiments: Bold responses during object recognition task

To determine whether task-dependent factors such as attention made a difference, we carried out three control experiments, each involving an object recognition task. The control Experiment 1 was identical to the main scan experiment above, except that only the last four subjects were involved, and they performed a one-back object discrimination task during the stimulus blocks of the scan. All stimuli were presented over a small central fixation spot which the subject was instructed to fixate throughout. At random junctures during each stimulus block, a given occluded object was featured in two successive stimuli. This repetition occurred twice during each stimulus block and featured a different occluded object each time. While maintaining fixation, the subject performed a one-back task in which he or she pressed a button when she perceived this repetition. The scans of control Experiment 1 were repeated 4 times for each subject. 

Control Experiment 2 was identical to Experiment 1, except that a single subject (different from the four subjects in control Experiment 1) was involved and, for practical reasons, the responses to only conditions 1 and 2 could be measured (30 repetitions per condition). Control Experiment 3 used identical to control Experiment 2, except that the target objects were faces instead of vases, and a sixth subject, different from the five subjects in control Experiments 1 and 2, was used. Control Experiment 3 was designed to ensure that our results were not specific to vases (30 repetitions per condition). 

While the subjects were told to maintain fixation in all of the above experiments, it was not possible to monitor their eye movements to ascertain fixation since the subjects wore anaglyph glasses. For this reason, these experiments do not address the effect of fixation on the responses to our stimuli. 

Retinotopic mapping

Retinotopic regions were defined using standard procedures (DeYoe et al., 1996; Tootell et al., 1997, 1998; Wandell, Chial, & Backus, 2000). We presented counterphase (8 Hz) checkerboard wedges (wedge angle, 45°) for 20 s each in one of the 8 radial positions spaced uniformly 45° apart from each other starting at 0° (vertical meridian in the upper visual field). Each radial position was repeated 4–6 times, depending on the subject. The boundaries of retinotopic areas were defined as smoothed contours showing the highest linear correlation with the hemodynamic function corresponding to vertical or horizontal meridian stimulation. Retinotopic mapping was discontinued after the data from the first eight subjects showed no regions of interest in the retinotopic areas (see Results). 

Data analyses

The BOLD data were motion-corrected using SPM2 software (Wellcome Department of Cognitive Neurology, London, UK) and high-pass filtered at 3 cycles per scan, co-registered with the anatomical data and transformed into the Talairach space using BrainVoyager (Maastricht, The Netherlands). Regions of interest (ROIs) selective for the Intact Object–Near Occluder condition were defined, separately for each subject, at p < 10 ⁻⁴ (uncorrected for multiple comparisons) using the Intact Object–Near Occluder > Intact Object–Far Occluder contrast (i.e ., condition 1 > condition 2). Object-selective ROI were similarly identified using the Intact Object–No Occluder > Scrambled Object–No Occluder contrast (i.e ., condition 3 > condition 4). The response of a given ROI to a given stimulus was measured as average percent BOLD signal change. To do this, the BOLD signal across all voxels of a given ROI for the given subject was averaged across all repetitions of the stimulus using an 8- to 20-s window. The percent signal change was calculated relative to the BOLD activity using a similar time window during the interstimulus blocks during the same scan and subsequently averaged across scans within a given subject and across subjects where appropriate. 

Tests of significance

For hypothesis-driven comparisons, we used the non-parametric Mann–Whitney test. For data-driven pairwise comparisons, we used Tukey's Honestly Significant Difference (HSD) test rather than the more stringent Bonferroni correction since the former is more appropriate for multiple pairwise comparisons (Crawley, 2002; Klockars & Sax, 1986; Toothaker, 1993). 

Cross-validation

For response comparisons involving condition 1 or condition 2, selection bias is a potential concern since these conditions were involved in defining the ROIs in the first place. To ensure that our results were not attributable to such selection bias, we used a cross-validation approach (see Baker, Hutchison, & Kanwisher, 2007; Baker, Liu, et al., 2007), in which we defined ROIs using one half of the data from each subject and compared the responses across conditions using the other half of the data. Other methods of dividing the data into two halves (e.g., assigning all the data from given subject randomly into either half) yielded qualitatively similar results (not shown). 

Results

Psychophysical characterization of stimuli

The stimulus set consisted of three pairs of occlusion stimuli and one pair of non-occlusion stimuli (see Figure 1). In order to ascertain that the stimuli elicited the key intended percepts, we carried out three psychophysical experiments under viewing conditions that mimicked those in the scanner as closely as possible (see Methods). In Experiment 1, we determined the occluder disparity at which the two stimuli that constituted the main contrast in the fMRI experiments, conditions 1 and 2, elicited the intended percepts. For condition 1 (denoted by an asterisk in all figures), the intended percept was that of a single, connected object amodally completed behind an occluder (see Figure 1B, left). Condition 2, which provided the same partial views of the occluded object, was nonetheless expected to yield a percept of a group of disconnected object fragments hovering in front of the occluder (see Figure 1B, right). During each trial, two stimuli corresponding to the two conditions were presented at a given disparity, and subjects performed a 2AFC task in which they picked the stimulus that contained an occluded object (see Methods for details). As expected, all eight subjects performed at chance levels when the occluder had zero disparity ( d′ analysis, p > 0.05; data not shown). All subjects showed significant discrimination performance ( p < 0.01) at occluder disparities of ≥10 arcmin. Five subjects performed at p < 10 ⁻⁴ when the disparity was ±1 arcmin. Subsequent psychophysical and fMRI experiments (see below) were carried out for each subject using a disparity at which he or she discriminated between the two conditions at p < 10 ⁻⁴. 

In Experiment 2a, we tested whether conditions 5 or 7 (the two other conditions that also contained a near occluder) elicited amodally completed percepts. Subjects were required to report whether they perceived, though the holes in the occluder, a single, connected object behind the occluder, or several disconnected fragments, or neither percept. All four subjects in this experiment reported perceiving a single, coherent object in a large majority of trials (≥99% for all subjects; 100% for two subjects). In Experiment 2b, we ascertained that condition 5 was perceived as a single, connected textured surface amodally completed behind the occluder. Condition 7 did not elicit an amodally completed percept, but instead the occluder was perceived as a single surface with black patches corresponding to the holes in the occluder. 

Patterns of response selectivity for occluded objects

We measured BOLD responses to each of the eight stimuli using a blocked design (see Figure 1C and Methods for details). To determine the brain regions, if any, that were preferentially responsive to occluded objects, we compared the BOLD response to condition 1 against the response to condition 2. 

Figure 2A shows two sets of foci ( red voxels) significantly more responsive to condition 1 than to condition 2 (Intact Object–Near Occluder > Intact Object–Far Occluder contrast, p < 10 ⁻⁴ without correction for multiple comparisons). For convenience, we will refer to these loci as occluded object-selective (OO-selective) loci. We define “preferential” or “selective” response as the larger response to a given stimulus (condition 1 in this case) relative to response/s to other relevant stimulus/stimuli (condition 2 in this case). 

Figure 2

 

Regions of selectivity for occluded objects. (A) Clusters ( red) that showed significantly higher responses to the Intact Object–Near Occluder condition than to the Intact Object–Far Occluder condition (i.e., condition 1 > condition 2, p < 10 ⁻⁴, uncorrected for multiple comparisons) in a representative subject are shown on the flattened surface of either hemisphere. The retinotopic areas ( dotted lines) were defined as described in Methods. The gray rectangle in either hemisphere denotes the area shown in the corresponding panel in B in slightly reduced form. (B) Overlap ( yellow) between the OO-selective regions ( red) and the UO-selective regions ( green). The UO-selective regions were defined using the Intact Object–No Occluder > Scrambled Object–No Occluder contrast (condition 3 > condition 4, p < 10 ⁻⁴ uncorrected). See Results for details.

One of the OO-selective foci was located in the lateral occipital region and the other more dorsally in the caudal intraparietal region (for Talairach coordinates, see Table 1A). No systematic hemispheric differences were apparent for either focus. 

Table 1

 

Talairach coordinates of the activation foci (center of activation ±SEM).

Location	x	y	z
(A) Intact Object–Near Occluder > Intact Object–Far Occluder contrast (OO-selective foci)
Left hemisphere
Lateral occipital	−38 ± 4	−74 ± 3	5 ± 7
Parietal	−22 ± 4	−78 ± 4	27 ± 4
Right hemisphere
Lateral occipital	41 ± 2	−71 ± 3	−4 ± 4
Parietal	32 ± 3	−71 ± 5	27 ± 5
(B) Intact Object–No Occluder > Scrambled Object–No Occluder contrast (UO-selective foci)
Left hemisphere
LOC	−40 ± 4	−63 ± 3	−4 ± 5
DF	−27 ± 3	−80 ± 3	27 ± 4
Right hemisphere
LOC	41 ± 2	−59 ± 7	−5 ± 3
DF	27 ± 3	−75 ± 3	25 ± 3

We compared the occluded object-selective (OO-selective) foci with the conventional unoccluded object-selective (UO-selective) foci identified using the Intact Object–No Occluder > Scrambled Object–No Occluder contrast (condition 3 > condition 4, p < 10 ⁻⁴; green and yellow pixels in Figure 2B). Note that this contrast is orthogonal to that used for identifying OO-selective foci. This contrast revealed previously known object-selective foci in the lateral occipital complex (LOC) and posterior intraparietal region, both of which were identifiable from their Talairach coordinates (see Table 1B) and also from their location relative to prominent anatomical landmarks. As noted in the Introduction section, the object-selective foci at comparable Talairach coordinates in the posterior intraparietal region have been referred to as the dorsal foci (DF; see Gilaie-Dotan et al., 2002; Grill-Specter et al., 2000), a notation we will adopt here for convenience. The OO-selective clusters substantially overlapped the corresponding UO-selective foci in both LOC and DF (yellow voxels in Figure 2B; also see Table 1), indicating that the two sets of foci were closely related. 

No other foci elsewhere in the brain were consistently identifiable across subjects (not shown). Murray et al. (2002) have found that in primary visual area (V1), the response is suppressed in response to the intact object compared to the scrambled object. However, in our case no significant V1 cluster (defined as ≥6 contiguous voxles using the comparable condition 3 > condition 4 contrast) was evident in either hemisphere in 9 of the 12 subjects. In the remaining three subjects, the V1 responses consisted of negative BOLD responses (NBRs) outside the stimulated visual field (as defined by the response to conditions 3 and 4). This NBR may represent “vascular steal,” suppression of neural activity, an active blood flow control system, or rebound from lateral inhibition (Chen, Tyler, Liu, & Wang, 2005; Shmuel, Augath, Oeltermann, & Logothetis, 2006; Shmuel, Augath, Oeltermann, Pauls, & Logothetis, 2003; Shmuel et al., 2002; Smith, Williams, & Singh, 2004; Wade, 2002). Overall, the V1 effect did not vary significantly across the stimuli, but did vary significantly across subjects (2-way ANOVA, stimuli × subjects; stimuli, p > 0.05; subjects, p < 0.05; interaction factor p < 0.05; data not shown), indicating that this was not a clear-cut, stimulus-driven effect. 

Effect of object recognition task on responses to occluded vs. unoccluded objects

The above experiments allowed passive free viewing of the objects because many earlier studies have shown reliable activation of higher visual areas using this paradigm (Grill-Spector et al., 2001, 2004; also see Joseph et al., 2002; Grill-Spector & Malach, 2004). Judging by the above results, this paradigm is adequate for revealing at least some OO-selective regions (see Discussion). But it remains possible that the preferential responses to occluded objects revealed by this paradigm are somehow only apparent in the absence in absence of a behavioral task. To explore this possibility, we carried out three control experiments in which the subject had to perform an object recognition task while maintaining fixation. It should be emphasized that the purpose of this experiment was not to delineate the effects of these behavioral parameters on OO-selective responses but solely to ascertain that the OO-selective responses persist during a behavioral task. 

In control Experiment 1, we scanned four subjects while they performed a one-back object discrimination task using the same set of eight stimuli as above (see Methods for details). The results for all four subjects as a group are illustrated in Figure 3 (see legend for details). 

Figure 3

 

Comparison of responses during passive viewing vs. object recognition task. The results shown are from the four subjects who performed both the main experiment (passive viewing, left column) and control Experiment 1 (object recognition task, right column). This figure shows the results of the group analysis of the data from all four subjects overlaid on the flattened brain surface of one of the four subjects. Voxels that show significant activation ( p < 10 ⁻⁴, uncorrected for multiple comparisons) by either contrast in either experiment are shown in red. The arrows in the right column denote selected clusters of significant activation evident during the object recognition task but not during the passive viewing ( left columns).

For the OO-selective contrast (condition 1 > condition 2, top row), the foci of activation during the behavioral task ( top right) largely overlapped, albeit were more extensive than, those from those obtained using passive fixation (top left), indicating that the OO-selective foci revealed using the passive fixation experiment are likely related to the perception of occluded objects, as opposed to simple bottom-up processing of occluded stimuli. Note that the responses were more widespread during the object recognition task than during passive viewing. Moreover, many foci were activated during the object recognition task but not during the passive viewing ( arrows). However, these may or may not represent novel foci that are selectively activated during object recognition task (as opposed to during passive viewing) because these data were collected from a subset of the subjects. Qualitatively similar results were obtained using a different subject while she performed an object recognition task using only conditions 1 and 2 (control Experiment 2) and in another subject while he performed a viewpoint discrimination task using occluded face stimuli (control Experiment 3) (data not shown). 

Together, the results of these control experiments suggest that the results in the main experiment were not idiosyncratic to the vase stimuli and the passive viewing paradigm used in that experiment. Note that this does not necessarily mean that behavioral task has no effect on the responses to occluded objects, but only that the responses revealed in main experiment above were unlikely have been a consequence of the stimuli and the paradigm used (see Discussion). 

Responses to the stimulus set in the OO-selective foci in LOC and DF

We examined the responses of the two OO-selective foci, as defined by the Intact Object–Near Occluder > Intact Object–Far Occluder contrast, to all eight stimuli using a series of post hoc tests. Figure 4A (top) shows the average time course of the BOLD response of the OO-selective focus in LOC. The average percent changes in the BOLD response to individual conditions are shown in Figure 4A (middle). Condition 1, the OO condition, elicited the largest signal change, at 1.16 ± 0.08 (SEM) %. As expected from the contrast used, this response was significantly larger than the response to condition 2 (condition 1 > condition 2; 1-tailed Mann-Whitney test, p < 10 ⁻⁸). However, the response to the OO condition was also significantly larger than the response to the unoccluded object (condition 1 > condition 3; 1-tailed Mann–Whitney test, p < 0.01). Note that this latter result is not necessarily expected since this ROI was defined solely using the condition 1 > condition 2 contrast, and the response to condition 3 played no part in defining this ROI. 

Figure 4

 

The time course ( top) and the average magnitude ± SEM ( midde, bottom) of the responses in LOC ( panel A) and DF ( panel B). In either region, the response to each condition was calculated across all voxels that were significantly more responsive to the Intact Object–Near Occluder condition than to the Intact Object–Far Occluder condition (condition 1* > condition 2) within each subject and averaged across subjects. The red stripes and the intervening blanks in the top panel denote stimulus and the interstimulus blocks, respectively. Results shown at top and middle in either panel were obtained using the entire data set from each subject. Results shown at bottom are from a cross-validation analysis in which the ROIs were defined using half of the data from each subject, and the responses of the ROIs were calculated from the remaining halves of the data. See Methods for details. The arrows in the bottom panel show p values for selected a priori comparisons using the Mann–Whitney test.

The response to the OO condition was larger than the response to any other stimulus, including the response to either type of occluder presented alone (Tukey's HSD test, p < 0.05 in all cases; see Methods for additional info). This indicates the response of this focus to the OO condition was not solely attributable to stimulus properties, including the presence of the target object (cf. condition 1 vs. 3), disparity sign of the occluder (cf. condition 1 vs. 5/7), partial views of the object (cf. condition 1 vs. 2), or the presence of the occluder (cf. condition 1 vs. 5). Note that the response was not attributable to amodal completion per se either since condition 5, which supports amodal completion of a background textured surface (see above) but does not elicit a shape percept, elicited a significantly smaller response (condition 1 > condition 5; 1-tailed Mann–Whitney test, p < 10 ⁻⁵). 

To determine whether the preferential response to the OO condition relative to the unoccluded condition was attributable to selection bias, we carried out cross correlation analyses using non-overlapping halves of data (see Methods for details). The results from this analysis were qualitatively similar ( Figure 4A, bottom). 

It should be noted that in comparisons involving occluded stimuli on the one hand and unoccluded stimuli on the other, the presence of the occluder is confounded by the presence of non-zero disparities, which may have contributed to the preferential responses to the OO condition. However, this confound was equally true for all conditions with an occluder, notwithstanding which the response to one of them (OO condition) was larger than the responses to the remaining conditions. Thus, this confound is unlikely to have been a significant contributor to the preferential responses to the OO condition. 

The pattern of responses of the DF focus across the stimuli ( Figure 4B) was substantially different from response pattern of the LOC focus (2-way ANOVA, conditions × foci; foci, p < 0.05; foci-conditions interaction, p < 0.05). The OO stimulus also elicited the largest response in DF (at 2.08 ± 0.25%) and, as expected, this response was significantly larger than the response to condition 2 (1-tailed Mann-Whitney test, p < 0.01). However, the response to the OO stimulus was also larger than the response to the conventional unoccluded object (i.e., condition 1 > condition 3; p < 0.01). Together, these results indicate that the LOC and the DF foci are each selectively responsive to occluded objects, but with somewhat different patterns of responses across the stimulus conditions. 

As indicated above, in both LOC and DF, the OO-selective foci overlapped the larger object-selective foci (as determined by the Intact Object–No Occluder > Scrambled Object–No Occluder condition). We studied the responses of these larger object-selective regions to our stimuli. In LOC ( Figure 5A), the response to the conventional unoccluded object (condition 3) was significantly larger than the response to the corresponding scrambled object (condition 4), as expected from the contrast used for defining these ROIs. This response was also significantly larger than the response to each of the other conditions (Tukey's HSD test, p < 0.05), except the response to the OO stimulus ( p > 0.05), indicating that activity in the object-selective LOC focus is likely correlated with the object percept regardless of its occlusion status. In the object-selective DF focus ( Figure 5B), the response to the unoccluded object was significantly larger than the response to scrambled object, as expected. However, DF response to the unoccluded object was statistically indistinguishable as that to many control stimuli (condition 3 vs. condition 1, 2, 5, or 6; Tukey's HSD test, p > 0.05 in all cases), indicating that when the responses to the control stimuli were taken into account, the response of the greater DF focus was no longer selective for objects. 

Figure 5

 

Average responses (± SEM) of object-selective ROIs in LOC ( panel A) and DF ( panel B). In either region, the response to each condition was calculated across all voxels that were significantly more responsive to the Intact Object–No Occluder condition than to the Scrambled Object–No Occluder condition (condition 3 > condition 4; denoted collectively by the green and yellow voxels in Figure 2B) within each subject and averaged across subjects. See Methods for details. Cross-validation analysis yielded similar results (not shown).

Together, these results indicate that the lack of OO selectivity in the greater LOC and DF foci arises because of a complex pattern of relative increases in the responses to non-OO stimuli. 

Discussion

Role of LOC and DF in the processing of occluded objects

Our results provide the first evidence for preferential responses to occluded objects in the visual cortex to our knowledge. In the OO-selective foci in both LOC and DF, the response to occluded objects was significantly larger than the response to either the object or the occluder by itself. The response in neither focus was attributable solely to low-level stimulus parameters, such as the properties or the presence of the occluder, or to amodal completion per se. This is consistent with previous EEG studies that show that border completion is dissociable from object recognition (Murray et al., 2006). While it is conceivable that the response to occluded objects was due to the potentially greater difficulty of perceiving an occluded object vs. perceiving an unoccluded object, this factor is unlikely to have played a major role, both because these effects were evident even with passive viewing and because the other conditions with an occluder (e.g., condition 2 or 5) elicited significantly lower response under the same viewing conditions. Thus, our results collectively indicate that these foci are selectively responsive to occluded objects and hence are likely to play an important role in the perception of occluded objects. 

The significance of our results is also that they reveal that the OO-selective foci are a part of the previously known object processing pathway. The selectivity for occluded objects presumably reflects the additional information processing required for the perception of such objects. As noted earlier, in order to perceive an occluded object, the visual system must distinguish between the image segments that belong to the occluder vs. to the occluded object and perceptually “explain away” the missing information about the occluded object (Kersten & Yuille, 2003; also see 1). However, our results do not allow us to determine the precise role of these foci in this process, given the limited spatial and temporal resolution of the BOLD response. Nonetheless, both these foci are likely to play a high-level role in this process, because the responses in both foci largely parallel the intended percept and not the low-level stimulus parameters, and because both these foci are closely associated with high-level, non-retinotopic regions known to play key roles in object perception (Grill-Spector et al., 2001, 2004; Kourtzi & Kanwisher, 2000; Spiridon, Fischl, & Kanwisher, 2006). 

Previous studies of occlusion

Although occlusion (or interposition) has been a subject of a large number of psychophysical studies (e.g., see Fang & He, 2005; Nakayama et al., 1989; for reviews, see Howard & Rogers, 2002; Komatsu, 2006), not many studies have addressed the neural mechanisms of occlusion. As noted earlier, Lerner et al. (2002) have examined the BOLD responses to partially occluded line drawings. While they did not find occlusion-related effects in DF, they did report that in LOC, the response to occluded objects was higher than the response to scrambled objects, but significantly lower than the response to unoccluded objects. The reason why they failed to find preferential responses to occluded objects is hard to ascertain since their study differed from ours in a number of important respects, including the type of objects (animals or unfamiliar novel objects in their case vs. familiar inanimate objects in our case) and occluders (vertical bars vs. random holes) used. However, it is plausible that the type of analytical contrasts used was a major contributing factor to the difference between the two studies. In our case, evidences for preferential responses to occluded objects were revealed using the Intact Object–Near Occluder vs. Intact Object–Far Occluder contrast (Figures 2 and 3), which was not possible in case of Lerner et al. (2002) since their stimulus set was limited to intact unoccluded object, intact occluded object and scrambled occluded object. Thus, in their case the spatially localized preferential responses to occluded objects were presumably averaged out across the entire object-selective LOC region (see our Figure 5). The data in Figure 5 also potentially explain why Lerner et al. (2002; also see Lerner, Harel, & Malach, 2004) did not find occlusion-related effects in DF since the larger DF region responds comparably to the three classes of stimuli used by them. 

Some procedural differences between the two studies are worth noting because they appear not to have made a substantial difference. Lerner et al. (2002) required that the subjects covertly name the objects while fixating, whereas our main experiment was simpler in that it involved only passive viewing. The fact that we found preferential responses to occluded objects using a simple task without a fixation requirement indicates that attentional and fixation controls are not critical for finding the preference for occluded objects. On the other hand, we found that the responses were qualitatively similar when the subject did perform an attention-demanding task. These two results together indicate the results obtained using passive viewing are not an artifact of this paradigm. This is consistent with the fact that, in the macaque inferotemporal cortex, form representation is virtually unaltered by fixation vs. free viewing (DiCarlo & Maunsell, 2000). Nonetheless, it remains possible that instituting these controls may uncover additional and/or larger foci for the processing of occluded objects (see Behrmann, Geng, & Shomstein, 2004; Komatsu, 2006; Murray et al., 2004; Rauschenberger & Yantis, 2001; Rauschenberger et al., 2006). Note, however, that the possibility that instituting a more elaborate behavioral paradigm may reveal additional complexities of the underlying mechanisms does not undermine the importance of revealing the existence foci preferentially responsive to occluded objects in the first place, as our results do. 

Murray et al. (2002) have reported that activity in V1 is lower when an occluded object is perceived as a coherent whole than when it is perceived as a collection of incoherent image elements. The reduced V1 activity may reflect the “residual” sensory signals after most of the image information is “explained away” through feedback from higher areas (Murray et al., 2002). It is intriguing that we did not observe this reduced activity in V1. One possible reason for this is that the reduced V1 activity is retinotopic and therefore dissipated in our case. Another possibility is that the reduction in activity is substantially greater when the subject is actively engaged in an object recognition task, as opposed to during passive viewing used our main experiments. Although we did not observe the reduction in our control experiments either (which involved an object recognition task), it remains possible that this effect can be uncovered using different tasks and/or larger data sets. Further experiments are needed to clarify this phenomenon. 

Olson et al. (2004) have reported that in areas MT+ and the intraparietal sulcus (IPS), a moving gray-and-white disk that smoothly disappears behind a static occluder elicits an intermediate level response between those when the disk abruptly disappears and when it is not occluded at all. It remains to be seen whether and to what extent these foci are functionally related to the ones we find (and vice versa). On the one hand, shape perception of static occluded objects may be different from the perception of dynamic occlusion (see Grill-Spector & Malach, 2004). On the other hand, the neural mechanisms of perceptual awareness of an object rendered invisible by occlusion may be different from the perception of a partially visible objects used in our experiments. It is also possible that static occlusion defined by disparity may be processed differently than those defined by other static occlusion cues. Tyler, Likova, Kontsevich, and Wade (2006) recently compared BOLD responses to visual scenes with depth borders defined by disparity, motion, luminance borders, line contours, or illusory phase borders and found that area KO is preferentially responsive to depth structures defined by disparity or motion. The extent to which the responses in our case, especially in DF, are related to the use of disparity as the depth cue (see Brouwer, van Ee, & Schwarzbach, 2005, Gilaie-Dotan et al., 2002; Tyler et al., 2006) remains to be determined. 

Hulme and Zeki (2007) recently investigated the neural mechanisms of perceptual awareness of objects (faces or houses) that were nearly fully occluded, so that the subjects had little direct sensory perception of the object but were nonetheless conscious of its presence. They reported that while premotor areas are preferentially activated when the subjects were conscious of an otherwise occluded object, the activity of fusiform face areas (FFA) and the lateral occipital region were invariant regardless of whether the objects were visible or not. It remains to be seen whether or to what extent the neural mechanisms of awareness of an object in the absence of direct perception addressed by Hulme and Zeki (2007) are related to those involved in completing a single whole object using partial views arising from occlusion addressed by the present study. 

Future directions

Many previous studies have shown selectivity for various object categories in LOC (see Grill-Spector & Malach, 2004). Thus, object categories such as faces, body parts, inanimate objects, and scenes are all processed in distinct, albeit closely related, subregions of LOC (Grill-Spector et al., 2000, 2001; Kourtzi & Kanwisher, 2000). The selectivity revealed by our study is novel in that it is not based on object category but instead on the occlusion status of an object, i.e., whether a given object is occluded or not. 

Our findings straightforwardly suggest three major topics for future research. First, does selectivity for occluded objects also exist for other object categories, e.g., for occluded faces or occluded body parts, in LOC and/or DF? Either scenario would have important implications for the functional organization of the object processing pathway. As noted earlier, our experiments using viewpoint discrimination of occluded faces do indicate the existence of preferential responses to occluded faces. 

Second, natural visual scenes contain many computational complexities other than occlusion, including (but not limited to) visual clutter, variations in viewpoint, illumination, size, position, and the available cues to object shape and identity. From the computational viewpoint, these scenarios entail several distinctly different types of information processing (Kersten, Mamassian, & Yuille, 2004; Yuille & Kersten, 2006). Does the visual system contain regions selective for these information processing requirements? There is some evidence that among the object-selective regions, ventral occipital regions (LOa/pFs) do contain representations of object shape sensitive to viewpoint or illumination, although not to size and position (Fang & He, 2005; Grill-Spector et al., 1999). Welchman, Deubelius, Conrad, Bülthoff, and Kourtzi (2005) have reported that foci within LOC are selective for combinations of binocular disparity and perspective than to either cue alone, although it is unclear whether these foci are functionally distinct from those that are selective for either cue alone. 

Third, what is the role of prior knowledge in perception of occluded objects? Previous studies indicate that perception of unoccluded objects in impoverished stimuli, such as two-tone images of faces and objects, is greatly facilitated by exposure to the corresponding full grayscale stimuli, and that the temporal cortex plays an important role in this learning (Dolan et al., 1997; Tovee, Rolls, & Ramachandran, 1996). Mechanisms by which the visual system similarly uses prior knowledge of objects to perceive visual scenes in which the impoverishment of sensory information arises from occlusion remain to be explored. 

These issues highlight the fact that much remains to be learned about how the visual system deals with the various computational challenges posed by natural visual scenes. The results of our study raise the prospect that understanding how the visual system deals with these complexities will be a fruitful area of future research. 

Conclusions

Specific regions in both the temporal and parietal visual processing pathways are preferentially responsive to occluded objects. These regions are likely to play an important role in the perception of occluded objects. 

Appendix A

Computations involved in the perception of an occluded object

The perception of an occluded object is part of the more general image parsing problem (Yuille & Kersten, 2006). Figure A1 illustrates key computational steps. The left panel of Figure A1 shows an image that can be interpreted in one of two ways. One the one hand, it can be seen as a collection of three rectangular, possibly textured or shaded, patches superimposed on a uniform gray planar background. Alternatively, it can be seen as a shaded ellipsoid behind a gray plane with three rectangular apertures. If one cross-fuses the left and right images of Figure A1 (left and right images to the right and left eyes, respectively), the first interpretation becomes quite compelling due to the fact that stereo information provides evidence for three rectangular patches floating above a gray background, and that the surface boundaries of the three patches are intrinsic to the patches, as shown in red in panel B1. The perceptual “explanation” of the image data (panel C1) is that of three, possibly scrambled, patches floating in the foreground (blue outlines) above a gray plane (orange outline). On the other hand, if one cross-fuses the two images in Figure A2, the second interpretation becomes compelling. In this latter case, the edges of the three rectangles become intrinsic to the gray planar surface (shown in red in panel B2), and one thus interprets the rectangular patches as holes. These rectangular edges are now extrinsic to the region inside the aperture, and because these edges are no longer bound to the internal patch regions, the regional and partial edge information (curved object fragments within the rectangles) provides candidate data to be “fit” by a single closed object, i.e., in this case, an ellipsoid. This object detection stage may involve both amodal completion (i.e., filling in of the ellipsoid based on Gestalt principles, e.g., of good continuation), as well as access to high-level familiar models, such as “ellipsoid.” The perceptual “explanation” of the image data (panel C2) is that of an ellipsoid (blue outline) floating behind a gray plane (orange outline). Note that both the disparity data and a high-level hypothesis about the form of the occluder can be used to “explain away” those parts of the image that do not belong to the target object (Yuille & Kersten, 2006). 

Figure A1

 

Key computational steps in the perception of an occluded object.

Figure A2

 

Free fusion stereograms.

In some sense, the second interpretation (panel C2) is the simpler or more economical description, and it is known that single, closed volumetric objects play a dominant role in object perception (Arguin & Saumier, 2004; Moore & Engel, 2001). Furthermore, as unoccluded stimuli, closed volumetric forms are particularly effective at activating object processing regions of the visual pathway, such as LOC (Moore & Engel, 2001). 

Free fusion stereograms

Figure A2 shows stereograms corresponding to conditions 1 and 2 that can be fused either divergently ( top) or convergently ( bottom). The resulting percepts are illustrated in Figure 1B. 

Readers who experience problems achieving fusion can follow these standard instructions. View the stimuli with your eyes level and about 20 inches (51 cm) from the image. Hold up a finger about 6 inches (15 cm) from your face and centered at the middle of a given stereo pair. Focus your eyes on the finger tip. When the stereo pair appears as two (blurred) pairs, slowly move your finger in a straight line between your face and the stereo image while maintaining focus on the finger until the middle pair of the images fuse into one. Without moving your finger, slowly shift the focus from your finger to the image so that the fused image in the middle appears three-dimensional while the finger appears single but blurred. With some practice, you should be able to remove your finger without losing stereo fusion. 

If you find this difficult to do with the images for convergent fusers, try those for divergent fusers, and vice versa. 

Acknowledgments

We thank Yi Jiang for valuable help with fMRI data collection. Drs. Huseyin Boyaci, Xingchuan Chen, Katja Doerschner, and Sheng He made helpful comments on the manuscript. We are grateful to Drs. Joseph Rausch and Paul Schrater for useful discussions. This work was supported by NEI grant R01 EY015261 and ONR grant N00014-05-1-0124 to DK. Office facilities were provided by the Center for Cognitive Sciences at the University of Minnesota. The 3T scanner at the University of Minnesota Center for Magnetic Resonance Research is supported by BTRR P41 008079 and by the MIND Institute. 

Commercial relationships: none. 

Corresponding author: Jay Hegdé. 

Email: [e-mail]. 

Address: University of Minnesota, Department of Psychology, 75 East River Parkway, Minneapolis, MN 55455, USA.