Recovery of a crowded object by masking the flankers: Determining the locus of feature integration

Abstract

Object recognition is a central function of the visual system. As a first step, the features of an object are registered; these independently encoded features are then bound together to form a single representation. Here we investigate the locus of this “feature integration” by examining crowding, a striking breakdown of this process. Crowding, an inability to identify a peripheral target surrounded by flankers, results from “excessive integration” of target and flanker features. We presented a standard crowding display with a target C flanked by four flanker C's in the periphery. We then masked only the flankers (but not the target) with one of three kinds of masks—noise, metacontrast, and object substitution—each of which interferes at progressively higher levels of visual processing. With noise and metacontrast masks (low-level masking), the crowded target was recovered, whereas with object substitution masks (high-level masking), it was not. This places a clear upper bound on the locus of interference in crowding suggesting that crowding is not a low-level phenomenon. We conclude that feature integration, which underlies crowding, occurs prior to the locus of object substitution masking. Further, our results indicate that the integrity of the flankers, but not their identification, is crucial for crowding to occur.

Introduction

One of the central functions of the visual system is to recognize objects. Crowding is a striking example where this fails. In the presence of nearby flankers, an otherwise easily recognizable peripheral object cannot be identified (Bouma, 1970; Toet & Levi, 1992). However, the ability to detect the object is not impaired (Pelli, Palomares, & Majaj, 2004). This phenomenon of crowding provides us with a handy tool to uncover the mechanisms involved in object recognition. Most theories of crowding argue for a two-stage process (Chung, Levi, & Legge, 2001; He, Cavanagh, & Intriligator, 1996; Levi, 2008; Levi, Hariharan, & Klein, 2002; Pelli et al., 2004). In the first “feature detection” stage, features are registered independently without any mutual interference. In the second “feature integration” stage, the registered features are pooled together for object recognition. If flankers are too close to the target, features of both are integrated—resulting in crowding. Here we investigate the locus of crowding as a means to determine the locus of feature integration, an essential step in object recognition. 

Despite more than a decade of research, it is not clear where crowding takes place (for a review, see Levi, 2008). Reports differ principally in the level of processing at which crowding is assumed to act. The region of target–flanker interaction in crowding is large (half the target eccentricity), suggesting that it might not be very early in the visual processing stream (Pelli et al., 2004). Dichoptic crowding is as strong as monocular crowding (Flom, Heath, & Takahashi, 1963), indicating that the locus is certainly beyond the retina (or LGN). Recent evidence shows that crowding is not a low-level process involving feature interference (Pelli et al., 2004). Other evidence has shown that it does not occur prior to V1: crowded gratings whose orientation was unreportable could still elicit orientation aftereffects (He et al., 1996). However, another study (Blake, Tadin, Sobel, Raissian, & Chong, 2006) found that crowding leads to a partial reduction in the strength of the orientation aftereffect suggesting that crowding might also act to some extent at earlier levels. There have been some reports claiming that crowding can act at higher levels of visual processing (Huckauf, Knops, Nuerk, & Willmes, 2008; Louie, Bressler, & Whitney, 2007). In short, the locus of feature integration is unknown (Levi, 2008). 

To determine the locus of crowding, we turn to the surprising phenomenon of target recovery in backward masking. An effective mask prevents the report of a target presented prior to it. However, if this first mask is then followed by a second mask, the target can be recovered (Dember & Purcell, 1967; Robinson, 1966). Inspired by the target recovery findings, we mask the flankers in a crowding paradigm using three kinds of masks—noise, metacontrast, and object substitution masks, to determine if a crowded target can be recovered. These masks all have different characteristics. In a general sense, there is a progression among these masks as to where and when they interact with their targets; noise masks interact at a low level, metacontrast masks interact at an intermediate stage, and object substitution masks interact at a high level of visual processing. This allows us to disrupt flanker processing in crowding at different stages of the visual hierarchy. Target recovery at some but not other stages would indicate where in the visual processing stream crowding takes place. 

Experiment 1: Masking the flankers

Noise masks—spatially overlapping random dot patterns presented at short stimulus onset asynchronies (SOAs)—are thought to interact with their targets at relatively early stages of visual processing (Breitmeyer & Ganz, 1976; Kahneman, 1968; Macknik & Livingstone, 1998; Rolls & Tovee, 1994; Sperling, 1963). Metacontrast masks—spatially non-overlapping but closely placed contours presented at moderate SOAs—are also thought to interfere with their targets at early levels of visual processing (Alpern, 1953; Anbar & Anbar, 1982; Breitmeyer & Ganz, 1976; Bugmann & Taylor, 2005; Kahneman, 1968; Werner, 1935). However, there is evidence that higher-level processes, such as attention and object based processing, play a role as well (Neumann & Scharlau, 2006; Ramachandran & Cobb, 1995; Tata, 2002). At short to moderate SOAs, as used in this study, the metacontrast mask and target signals are thought to be integrated by apparent motion processes (Bischof & Di Lollo, 1995; Burr, 1984; Kahneman, 1967). Finally, object substitution masks—spatially non-overlapping masks that have a common onset with their targets but a delayed offset—are thought to interfere at a late stage of visual processing (Enns, 2004; Enns & Di Lollo, 1997; Tata, 2002). Here, it is thought that the representation of the target is erased and replaced by that of the mask, leading to a failure in target identification. 

Features of an object are registered independently in early visual cortices. These features are selected and integrated over some area to form a representation that is then recognized. To determine the locus of this feature integration process, we mask flankers in a crowding paradigm with various types of masks. As described earlier, noise masks act early, by perhaps interfering with or suppressing the registration of features. Therefore, there are few or no surviving flanker features to combine with target features, and we would expect target report to return to uncrowded levels. Metacontrast masks (at the settings used in this experiment) act via a different mechanism but integrate with the target and thus also degrade feature registration. Once again, we would expect target recovery. Object substitution masks, on the other hand, do not interfere with feature encoding, but act much later, by replacing the representation of the stimulus. In this case, we might not see any target recovery since the flanker features may not be degraded and may be available to combine with the target features, resulting in undiminished crowding. To preview our results, this pattern of target recovery is what we find: The target is recovered when the flankers are masked with noise or metacontrast masks, but not with object substitution masks, implying that crowding takes place after the stage where noise masks interfere with their targets but before the stage where object substitution masks become effective. 

It is known that crowding is most effective when the flankers resemble the targets (Kooi, Toet, Tripathy, & Levi, 1994). The flankers we use are highly similar to the targets and, as we will show, create significant crowding. In contrast, the masks we use are highly dissimilar to the targets and, again as we will show, create little crowding on their own. Therefore, if the masks we use degrade or integrate with the flankers before crowding takes place, the flankers will no longer be similar to the targets, reducing crowding. Whereas, if masking alters the perceptual appearance of the flankers after crowding has occurred, no reduction in crowding should be seen. On the other hand, all mask types might alter perceptual appearance of the flankers so that they are now dissimilar to the target. If so, the target should be recovered under all masking conditions. However, when perceptual appearance is altered in all cases, if the target were recovered in some masking conditions but not others, it would indicate that perceptual similarity might not be the main requirement for crowding. Only the differences in the timing and level of the masks' influence, irrespective of their effect on appearance, would explain the results. 

Methods

Observers

Nineteen observers (including the first author) aged 22–34 years with normal or corrected-to-normal vision took part in this study. Of these 19 subjects, 11 participated in the noise masking block, 7 in the metacontrast masking block, and 11 in the object substitution masking block. Seven observers took part in the baseline unmasked, unflanked target identification block. 

Stimuli

The target and flankers in the crowding display were grayscale block C's with the gap in the C pointing in one of four directions (up, down, left, or right) presented on a uniform gray background (luminance = 34.2 cd/m ²). The C's, with a contrast of 20%, were hollow squares of size 1.5 deg with a gap, of size 0.5 deg, in one side. The target was presented at an eccentricity of 9 deg in the lower visual field. Four flanker C's were presented around the target (see Figure 1). The center-to-center distance between the target and the flankers was set at 4 deg (for two subjects, it was 2.7 deg and 3.75 deg, respectively, since they did not show crowding at 4 deg). These separations are within Bouma's (1970) bound for crowding. 

Figure 1

 

Recovering a crowded target: Conditions. Shown here is the procedure adopted in the four conditions tested for each mask type in Experiment 1. On the left is a single panel depicting the crowded condition: a target presented at 9 deg eccentricity was flanked by four flankers, with the entire display presented for 25 ms. The display was the same for all masking blocks. On the right is a 3 × 3 panel depicting the other three conditions for each mask type. Each row depicts the procedure for one of three conditions: masked flankers, masks only, and delayed masks. Each column depicts the procedure for a given mask type: noise, metacontrast, and object substitution masks. The presentation duration for each frame is indicated at the top of that frame.

Noise masks were square patches of size 2.2 deg. Each patch was made up of square dots of size 0.2 deg. The mask appeared as a tessellated surface of dark and light gray dots. The contrast between the dark and the light gray dots was 40%. Metacontrast masks were circles that did not overlap the flanker C's but fit around them snugly—the spatial separation between the inner edge of the annulus and the outer contour of the C was less than 0.25 deg. The circles, drawn with the same contrast (20%) as the flankers, were 2.4 deg in diameter. We used a circular mask instead of a tightly fitting square one because the latter would crowd the target whereas the former would not (Kooi et al., 1994; Levi, 2008). However, a circular mask might also result in weaker masking than a square one (Sherrick & Dember, 1970; Werner, 1935), but this is known to be the case only with a foveal presentation. Using peripheral presentation Enns and Di Lollo (1997) showed that contour proximity did not have much of an effect in terms of masking strength. They did find, however, that the temporal window of masking was narrower. We therefore used a shorter SOA (25 ms) than the standard metacontrast masking SOA of 50–100 ms. Object substitution masks were a set of four small squares located at the corners of an imaginary square of size 2.2 deg. Each tiny square subtended 0.3 deg. The four squares could sit around the corners of the flanker C. The squares were dark gray with a contrast of 40%. 

Procedure

There were three blocks in Experiment 1, each testing the effect of a different kind of mask ( Figure 1). The procedure in each block was similar except for the means to produce the specific masking that was being applied. There were four conditions in every masking block—two main and two control (see Figure 1). The main conditions were the crowded condition and the masked flanker condition. In the crowded condition, the standard crowding configuration, a target surrounded by four flankers, was used. This display was presented for 25 ms. In the masked flanker condition, the standard crowding display was presented for 25 ms. The flankers were then masked with one of the three types of mask. Noise and metacontrast masks were presented for 25 ms after an SOA of 25 ms. Object substitution masks appeared along with the crowded display but persisted for a total of 150 ms. These intervals were chosen based on pilot testing and a reading of the relevant literature. A study of the masks' effectiveness was conducted in Experiment 2. 

To ensure that the masks themselves were not interacting with the target in any way, we used two control conditions— masks only and delayed masks only conditions. In both these conditions, the target was presented with masks only. No C flankers were presented. In the masks only condition, the masks were presented at the locations of the flankers simultaneously with the target. In the delayed masks only condition, the masks were presented exactly as in the masked flanker condition, except that no flankers were presented. If target identification is unaffected by the masks alone, then crowding seen with the flankers and masks cannot be attributed to the suppressive effect of masks. The four conditions were randomized within each block. Feedback was provided. The task was to report the direction of the gap in the central target C by means of a key press. Thirty-two trials were run for each condition for each kind of masking. Finally, we asked observers, in a separate block, to report the orientation of an unmasked, unflanked C presented at the same location as the target in the main experiment. This performance acts as a baseline to determine if the target identification was fully recovered in the various masking conditions. 

Results

A repeated measures ANOVA was conducted for each block separately to ensure that the conditions differed from each other. We then ran paired t-tests within each block to compare performance between conditions. Alpha values were adjusted conservatively (Bonferroni correction) to allow for multiple comparisons. Figure 2 plots the average performance in each of the four conditions for each masking block. Baseline performance [0.81 ± 0.04 (average ± SEM)] for an unmasked unflanked target is plotted as a horizontal line in each graph. As can been seen in the plots, the target was strongly crowded in the crowded condition in all blocks. 

Figure 2

 

Recovering a crowded target: Results. Shown here are the results for each of the four conditions tested for each mask type in Experiment 1. Of importance is the comparison between crowded and masked flankers conditions. An improvement in performance in the masked flankers condition relative to the crowded condition indicates recovery of the crowded target. The horizontal black line in each graph (indicated by the arrow in the lower left graph) shows performance in the unmasked unflanked target block. The dashed gray lines above and below it indicate 1 SEM on that task. Upper left: noise masking: Masking the flankers with noise masks relieves crowding to a large extent. Upper right: metacontrast masking: Metacontrast masking of flankers relieves crowding. Lower left: object substitution masking (OSM): OSM of flankers does not lead to recovery of a crowded target suggesting that crowding must occur prior to the stage OSM becomes effective. Error bars = 1 SEM.

The crowded target was recovered almost completely when the flankers were masked by noise masks [crowded = 0.58 ± 0.05; masked flankers = 0.74 ± 0.05; t(10) = 8.86; p < 0.001]. Performance in the masked flanker condition was indistinguishable from unmasked, unflanked performance [ t(16) = 1.05; p > 0.3]. This is all the more interesting because the flankers and the target were presented for the same duration in the two conditions, yet interfering with flanker processing appears to block the crowding that the flankers would normally produce. Similarly, metacontrast masking of the flankers substantially improved the performance for the crowded target [crowded = 0.48 ± 0.06; masked flankers = 0.64 ± 0.06; t(6) = 4.2; p < 0.05]. However, although the masking induced recovery of the crowded target was significant, it was not complete—performance did not equal that of an uncrowded target [ t(12) = 2.35; p < 0.05], suggesting that, in this case, the residual of mask flanker interactions could still produce some crowding. In both masking conditions, the target report was similar when the target was surrounded only by the masks (without the flankers) or by nothing (baseline). This suggests that the masks per se did not have any direct effect on the target and that target recovery was mediated by the effects of the masks on the flankers. 

No recovery was seen when the flankers were masked by object substitution masks [crowded = 0.53 ± 0.03; masked flankers = 0.50 ± 0.05; t(10) = 0.74; p = 0.48]. Interfering with flanker processing at a high level does not seem to prevent crowding from occurring. We can conclude that crowding has already occurred by the time object substitution masking becomes effective. These findings place constraints on the level at which crowding occurs. 

Experiment 2: Testing mask effectiveness

Experiment 1 showed that noise and metacontrast masking of flankers significantly reduces crowding but object substitution masking does not. The fact that target recovery is seen in some cases but not in others could plausibly be attributed to varying effectiveness of the different masks. Perhaps noise and metacontrast masks are more effective in masking the flankers than object substitution masks. If object substitution masking is weaker, the flanker signals might be sufficiently strong to crowd the target. In this experiment, we tested the effectiveness of the three kinds of masks. 

Method

Observers

Five observers (including the first author) participated in Experiment 2. 

Stimuli and procedure

The stimuli were the same as in Experiment 1. Figure 3 depicts the procedure followed. There were four conditions, based on the type of mask used— noise, metacontrast, object substitution, and no masking. The four conditions were intermixed. Four C's were presented simultaneously for 30 ms in the same locations as the flankers in Experiment 1. No central target was presented. On each trial, depending on the condition, these four targets were masked by one kind of mask, under the same protocol as in Experiment 1: Noise and metacontrast masks lasting 30 ms were presented at an SOA of 30 ms; object substitution masks lasting 150 ms were presented at an SOA of 0 ms (simultaneous onset with the targets). In the no masking condition, no mask was presented. After an SOA of 500 ms relative to target onset (thus equating memory requirements across all mask conditions), a dot probe was presented at the location of the target to be identified. The probe was a tiny bright green square that subtended 0.3 deg. The color, size, and SOA of the probe were chosen so as to avoid any possible masking effect of the probe on the target and to ensure good visibility. The no masking condition served as the baseline condition for reporting the target. This setup enabled us to directly assess the effectiveness of each mask at the tested SOA/duration. 

Figure 3

 

Mask effectiveness: Procedure (left). Four targets were presented for 30 ms at the locations of the flankers in Experiment 1. All four targets were masked. The mask SOA was 30 ms for noise and metacontrast masks, and the mask duration was 150 ms for object substitution masks (with an SOA of 0 ms). A cue was presented at an SOA of 500 ms (relative to target onset) at the location of the target to be reported. The cue here is shown as a white square in the last frame; in the experiment, it was green. We show only metacontrast masking for illustration. Results (right). Performance in the three masking conditions (noise, metacontrast, and object substitution) and the no masking control condition is plotted. There were no significant differences in mask effectiveness; all three mask types masked the targets equally. Error bars = 1 SEM.

This format was chosen for several reasons. This arrangement most closely approximates the condition in Experiment 1 and gives us confidence about whether the flankers in that experiment were indeed masked. On the other hand, other alternatives, such as presenting single targets (or identical targets in flanker locations), cannot be used to test mask effectiveness, as no object substitution masking is observed under those conditions (Enns & Di Lollo, 1997; Tata & Giaschi, 2004). Further, this setup equates all factors, such as memory demands, across mask types, except the masking procedure itself, thus serving as a direct test of mask effectiveness. 

Results

All three mask types were equally effective ( Figure 3). For all three masks, performance was barely above chance when the four targets were simultaneously masked irrespective of the type of mask used (statistically indistinguishable from chance), whereas observers identified the targets well above chance when they were not masked. Further, none of the pairwise comparisons among the three masking conditions were significant. Thus, the lack of target recovery in the case of object substitution masking of the flankers cannot be due to less effective masking of the flankers by such masks. 

Discussion

We investigated, in a crowding paradigm, whether a crowded target could be recovered by interfering with the processing of its flankers. Masking the flankers with either noise patches or metacontrast rings significantly relieved crowding: almost completely with noise masking and partially with metacontrast masking. However, object substitution masking of the flankers did not produce any target recovery. Experiment 2 confirmed that the masks were all equally effective in masking the identity of the flankers. These results suggest that the interaction between the flankers and the target in crowding has not yet occurred by the time metacontrast and noise masks interfere with the flankers but has occurred before object substitution masks render the flankers unreportable. 

Crowding is a consequence of excessive pooling of features at the feature integration stage of object recognition (Chung et al., 2001; He et al., 1996; Levi, 2008; Levi et al., 2002; Pelli et al., 2004). Our results indicate that this feature integration, whether an automatic bottom–up process (Pelli et al., 2004) or a function of selective attention (He et al., 1996), occurs before object substitution masking becomes effective. 

In our investigation, noise and metacontrast masking of the flankers would have resulted in the degradation of the flanker signal. Our results show that this degraded flanker signal has a much weakened crowding effect. The process by which the flanker signal is degraded might have been different for the two masks. Noise masks might act by linear or non-linear summation of mask and flanker signals (Kahneman, 1968) or by lateral inhibition of target signals (Rolls & Tovee, 1994), thus reducing the signal-to-noise ratio of the flankers. On the other hand, there is substantial evidence that integration of the mask signals with those of the flankers in metacontrast masking occurs by processes underlying apparent (or low-level) motion (Bischof & Di Lollo, 1995; Burr, 1984; Kahneman, 1967). In either case, it seems that the masks' signals are integrated with that of the flankers. It has been posited, however, that backward masking might not be an early bottom–up process. For example, Fahrenfort, Scholte, and Lamme (2007) argue that feedback from higher areas is essential for perception, and this re-entrant processing is interrupted by masking. This would suggest that backward masking occurs higher up or later than what we have outlined here. However, the lack of feedback might be a consequence of weak feed-forward signals caused by masking. Furthermore, there is direct electrophysiological evidence that backward masking results in an abrupt attenuation of target signals (Rolls & Tovee, 1994; Rolls, Tovee, & Panzeri, 1999), indicating that the interference occurs early. 

On the other hand, in object substitution masking, the masks persisted for a much longer duration (150 ms) than the flankers (25 ms) and little or no mask flanker integration is seen (Enns, 2004). In this case, it can be argued that the representations of the masks replace those of the flankers but do so apparently too late to reduce the effects of the flankers on the target. Our results suggest that crowding did not occur when the signals of the flankers were degraded by integration with those of the masks as this left no flanker features to integrate with target features. However, in object substitution masking, the flanker features are not degraded before the stage at which they integrate with target features—crowding is therefore produced by flankers that are then quickly overwritten themselves and rendered not reportable. 

As mentioned earlier, similarity plays a significant role in crowding—the more similar the flankers are to the target, the more they crowd the target (Kooi et al., 1994). To explain our results, it could be argued that noise and metacontrast masks reduce the crowding because they alter the appearance of the flankers, whereas object substitution masks do not change the appearance of the flankers. Or it could be that all three masks equally alter the appearance of the flankers, but the specific timing and level at which they disrupt flanker processing is crucial in explaining our results. In terms of appearance, the three cases turned out not to be different. Subjectively, most subjects could not tell whether they saw any flankers under all three masking regimens; they were unaware that any flankers were presented along with the masks. Thus, we cannot attribute the reduction of crowding to a change in the similarity of appearance between targets and flankers. The reduction in crowding was seen for only two types of masking, the noise and metacontrast masks, but the appearance was dramatically altered for all three types. Our results therefore suggest that it is the timing and level of the masks' influence on the flankers that determines the strength of the crowding they produce. 

Recently, an fMRI adaptation study was conducted to determine the locus of object substitution masking (Carlson, Rauschenberger, & Verstraten, 2007). The authors found that when a first target was rendered unreportable by object substitution masking, the second presentation of that same target produced an unadapted response in lateral occipital cortex (LOC); but when the first target was reportable, a reduced (adapted) response was seen in LOC for the second presentation. These changes were not tracked by early visual areas (V1). This suggests that object substitution masking suppressed the representation of the first target in LOC so that the second presentation of the target escaped any adaptation effect. Our results indicate that crowding occurs prior to the region where object substitution masking becomes effective and therefore prior to LOC. Previous studies (Blake et al., 2006; He et al., 1996) have shown that crowded targets can produce measurable aftereffects such as the orientation aftereffect, suggesting that crowding occurs to a large extent in sites beyond the primary visual cortex. Our experiments here suggest that the site of crowding falls between V1 and LOC. This places bounds on the locus of crowding and hence on the locus of feature integration. 

Conclusion

Masking the flankers in a crowding paradigm results in the recovery of the target when the masks interfere with the flankers at early stages of visual processing but not when they interfere later. This suggests that crowding, and thus feature integration, is not an early process but occurs before the stage of object substitution masking. 

Acknowledgments

This research was supported by an NIH grant (EY02958) and a Chaire d'Excellence grant to PC. An NIH grant (EY04432) to Denis Pelli partially supported RC. We thank Charles Stromeyer III, Nate Blanco, and Sarah Rosen for helpful suggestions on this manuscript. 

Commercial relationships: none. 

Corresponding author: Ramakrishna Chakravarthi. 

Email: [e-mail]. 

Address: Psychology and Neural Science, NYU, 6 Washington Place, New York, NY 10003, USA.