Crowding and attention in a framework of neural network model

Abstract

In this article, I present a framework that would accommodate the classic ideas of visual information processing together with more recent computational approaches. I used the current knowledge about visual crowding, capacity limitations, attention, and saliency to place these phenomena within a standard neural network model. I suggest some revisions to traditional mechanisms of attention and feature integration that are required to fit better into this framework. The results allow us to explain some apparent theoretical controversies in vision research, suggesting a rationale for the limited spatial extent of crowding, a role of saliency in crowding experiments, and several amendments to the feature integration theory. The scheme can be elaborated or modified by future research.

Introduction

Visual crowding is a deterioration of the perception of a target object caused by other objects nearby (Bouma, 1970; Andriessen & Bouma, 1976). Crowding is primarily observed in the visual periphery where it is a main factor that limits pattern recognition (Pelli & Tillmann, 2008; Levi, 2008). 

There are many theoretical ideas about the mechanisms of crowding and not much agreement among the researchers (Balas, Nakano, & Rosenholtz, 2009; Greenwood, Bex, & Dakin, 2009; Van den Berg, Roerdink, & Cornelissen, 2010; Herzog, Sayim, Chicherov, & Manassi, 2015; Francis, Manassi, & Herzog, 2017; Harrison & Bex, 2017; Manassi & Whitney, 2018; Rosenholtz, Yu, & Keshvari, 2019). A long-lasting question is about the role of attention (Intriligator & Cavanagh, 2001; Pelli, Palomares, & Majaj, 2004; Põder, 2006; Strasburger, 2007). Whereas some authors see crowding as an implication of certain properties of attention, others reject any important relationship between these. 

In this article, I attempt to clarify relations between visual crowding and different forms of attention. I analyze crowding and attention in the context of a simple computational framework for vision—a feedforward convolutional neural network (CNN). It appears that several puzzling results from crowding experiments can be naturally explained by simple attentional mechanisms combined with standard feedforward network. 

A classic neural network model

Based on much neurobiological data and state-of-the-art machine vision, we can describe a standard object recognition system as a hierarchical feedforward neural network (Hubel & Wiesel, 1965; Fukushima, 1980; Riesenhuber & Poggio, 1999; Krizhevsky, Sutskever, & Hinton, 2012; Kubilius, Bracci, & de Beeck, 2016; Figure 1). It consists of a number of feature maps at several levels of processing. A feature map is an array of local “feature detectors.” Features at higher levels become more complex in terms of local pixel values but more relevant for object recognition tasks. A number of different features is larger at higher levels. Some extent of spatial pooling occurs at every level—receptive fields become larger, and precise absolute position of features is discarded. A few of the highest levels consist of nonspatial feature maps. Crowding is a natural consequence of spatial pooling within this architecture. 

Figure 1.

 

A standard neural network model of vision. Approximate levels of basic attention mechanisms are indicated.

A puzzle of Bouma's law

Many studies of visual crowding have found that crowding zones have radius approximately 0.5 of the eccentricity of the target object (Bouma, 1970; Pelli et al., 2004). Assuming that crowding is an effect of spatial pooling, this value should reveal the size of respective receptive fields. On the basis of these findings, crowding has been frequently attributed to a single level of processing where the size of receptive fields best fits the given dependence on eccentricity (V2, or V4; Freeman & Simoncelli, 2011; Motter, 2018). However, this simple account of crowding seems to be at odds with some knowledge about object recognition systems. 

Position-invariant object recognition with feedforward neural networks apparently requires cumulative spatial pooling over the full visual field. Standard CNNs contain several steps of pooling, with the final output collapsed across spatial coordinates. This implies some crowding-like interference from irrelevant objects located anywhere in the visual field. In reality, available data indicate that zones of interference in CNNs may extend over the whole input image, and target-distractor distance has only a modest effect (Lonnqvist, Clarke, & Chakravarthi, 2020). Similarly, a systematic increase throughout the levels and very large receptive fields at the highest levels have been found in the primate visual system (Gross, Rocha-Miranda, & Bender, 1972; Rolls, 2000). Given these observations, it seems puzzling that spatial extent of crowding in human vision is limited to a relatively small fraction of the visual field with radius 0.5E. 

An explanation quite naturally follows from Intriligator and Cavanagh's (2001) idea that crowding reflects a maximum resolution of spatial attention. When a “spotlight” of attention has radius of 0.5E, we can voluntarily select available information within that zone and exclude everything outside of it (Figure 2). Therefore attentional selection can eliminate any potential crowding caused by stimuli at larger distances. This role of spatial attention has been demonstrated in the inferotemporal (IT) cortex of monkeys (Zhang, Meyers, Bichot, Serre, Poggio, & Desimone, 2011). Because the “spotlight” of attention cannot be reduced beyond 0.5E, crowding within that zone cannot be eliminated. 

Figure 2.

 

Crowding zones and spatial resolution of attention. Classic crowding zones with radius 0.5E according to Bouma's law (top), spatial resolution of attention (minimal spacing that allows attending individual items) according to Intriligator and Cavanagh (2001, center; reprinted with permission from Elsevier), “uncrowded” display of rotated Ts (all inter-object distances are at least 0.6E, bottom). While fixating the central cross, we can voluntarily attend every single letter and avoid crowding from others.

Still, there exist some conditions when crowding zones may be either much smaller (e.g., Põder, 2006), or much larger (Vickery, Shim, Chakravarthi, Jiang, & Luedeman, 2009). “Too large” crowding zones can be likely explained by incomplete focusing of attention in certain conditions. Reduced extent of crowding is the topic of the next part. 

The idea to relate the limited extent of crowding to attention leaves unexplained why the minimum radius of the attentional spotlight is 0.5E. Possibly, that could be explained by some properties of natural environment or limits of brain capacity. 

Finally, we must note that human peripheral vision and convolutional neural networks are not directly comparable. The standard CNNs with their built-in position invariance approximate object recognition in human central vision. In human peripheral vision, not only critical distance of crowding but also many other important characteristics vary with eccentricity (e.g., Strasburger, Rentschler, & Juttner, 2011). Up to now, very few studies have tried to implement these eccentricity dependencies in neural network models (Chen, Roig, Isik, Boix, & Poggio, 2017; Han, Roig, Geiger, & Poggio, 2020). 

Saliency-based selection

It is well known that crowding is significantly reduced when a target differs from the flankers by some simple visual feature (for example, has a different color, e.g., Kooi, Toet, Tripathy, & Levi, 1994; Põder, 2007). This effect is even larger when the number of flankers is increased (Põder, 2006; Malania, Herzog, & Westheimer, 2007; Manassi, Sayim, & Herzog, 2013). With many homogeneous flankers, crowding virtually disappears. Põder (2006) proposed an explanation based on two mechanisms of spatial selection. In addition to voluntary attention with relatively low spatial resolution, there is a saliency-based bottom-up selection with much higher resolution. Most likely, it operates at lower levels of processing, possibly at V1 (Li, 1999; Li, 2002). Multiplication of the effects of these two selection mechanisms produces experimentally observed crowding zones (Figure 3). 

Figure 3.

 

Reduced crowding with large number of homogeneous flankers (Põder, 2006). Examples of stimuli (top), experimental results (center), and illustrative model with two selection mechanisms (bottom).

However, this is not the full story. There is a top-down feature-based attention that can bias selection of salient objects. For example, it is not difficult to select either a blue or orange pop-out bar presented among homogeneous green bars (Figure 4). With a heterogeneous background, however, there is no pop-out, and selection of even a single target color is nearly impossible. Little is known of how this combining of bottom-up and top-down selection is implemented in neural networks. 

Figure 4.

 

Combination of bottom-up saliency and top-down feature-based attention. When presented in the periphery, it is not difficult to attend to either orange or blue salient bar (left). With heterogeneous distractors, it is difficult to attend to a single blue bar (right).

Researchers have designed many different crowding displays that produce results not consistent with Bouma's law and simple pooling models (Livne & Sagi, 2007, 2010; Saarela, Sayim, Westheimer, & Herzog, 2009; Manassi, Sayim, & Herzog, 2013; Rosen & Pelli, 2015; Doerig, Bornet, Rosenholtz, Francis, Clarke, & Herzog, 2019; Figure 5). Frequently, a notion of grouping has been evoked as an explanation (Livne & Sagi, 2010; Herzog et al., 2015; Francis et al., 2017). However, grouping itself is not well understood, and its relation to crowding is ambiguous too. I suggest that saliency could be a more productive idea. There exist neurobiological data on saliency computation in biological vision (Knierim & van Essen, 1992; Sillito, Grieve, Jones, Cudeiro, & Davis, 1995; Nothdurft, Gallant, J. & Van Essen, 1999, Li, Their, & Wehrhahn, 2000), and several methods to compute saliency maps of images have been proposed (Itti & Koch, 2000; Zhang, Tong, Marks, Shan, & Cottrell, 2008; Bruce & Tsotsos, 2009). Relative saliency of the target and flankers could directly predict performance in crowding experiments. 

Figure 5.

 

Various examples of stimuli from crowding experiments that appear to reject simple pooling models. Number, homogeneity, and regularity of flankers have strong effects on target perception (Malania et al., 2007; Saarela et al., 2009; Livne & Sagi, 2010; Manassi et al., 2013; Rosen & Pelli, 2015).

It appears that that many “complex” results from crowding studies are naturally explained by a simple lateral-inhibition–based saliency model. Illustrative examples are given in Figure 6. Here, saliency of an item i was calculated as \({s_i} = {e_i} - \mathop \sum_j {k_{ij}}{e_j}\), where e_i is activation of an item before lateral inhibition and k_ij is inhibitory interaction between items i and j. 

Figure 6.

 

Examples of saliency maps based on simple lateral inhibition rules that predict the effects of (A) number of flankers (Põder, 2006), (B) configuration of orientations (Livne & Sagi, 2007), and (C) shape homogeneity/heterogeneity (Manassi et al., 2016).

To account for Põder (2006) results, we must assume that inhibitory interactions are stronger between items of the same color, as well as between those spatially close to each other. This mechanism suppresses signals from the flankers within a group of the same color, while differently colored target receives much less inhibition. With a larger number of flankers, the total suppression affecting each one is stronger. Suppression is less at the corners and edges of a group. Thus increasing the number of same-color flankers both reduces saliency of the flankers and moves salient flankers further from the target (Figure 6A). To reproduce the experimental results, the saliency model must be combined with an ordinary crowding zone/a window of top-down spatial attention (as shown in Figure 3). 

Livne & Sagi (2007) found an effect of global configuration of flankers on crowding. For example, a smooth circle of flanking Gabors caused much less crowding than a sun-like configuration. Simple orientation- and proximity-based inhibition cannot explain these results. Interestingly, however, Cavanaugh, Bair, & Movshon (2002) have reported on an interaction of orientation and relative position in surround suppression in V1. In addition to the simple effect of orientation tuning, radially oriented surround patches caused stronger suppression compared to tangentially oriented ones. Applying this rule to Livne and Sagi (2007) stimuli produces a stronger inhibition for the target and less inhibition for the flanker signals in “sun” as compared to “smooth” configuration (Figure 6B). 

Manassi et al. (2016) used two different shapes as flankers in a Vernier offset discrimination task. Using a row of flanking squares, the crowding effect decreases with the number of additional squares. However, a similar row consisting of two alternating shapes (squares and stars) produced a strong crowding effect. The results are naturally predicted by lateral inhibition/saliency model with a reasonable assumption that two identical items inhibit each other more heavily than different items. (Additionally, I suppose that inhibitory effect of the Vernier on two other items is negligible because of its smaller size) (Figure 6C). 

Pooling at higher levels

Higher levels of the network pool over very large receptive fields. Without focused attention, effective pooling zones must be much larger than 0.5E (up to the full visual field). Global pooling can be studied by visual search (Shaw, 1980; Palmer, Ames, & Lindsey, 1993; Palmer, Verghese, & Pavel 2000), or whole report experiments (e.g., Kyllingsbaek, Valla, Vanrie, & Bundesen, 2007). In usual search task, all stimuli in a display are relevant. The results of search experiments should reveal, which kind of information can be pooled, and which rules of pooling are used at highest levels of vision. 

In a recent article (Põder, 2017), I analyzed combined effects of crowding and set size in visual search experiments (Figure 7). A simple multiplicative model well describes the results, consistent with independent effects of different levels of processing. However, there is some correlation between two effects, across studied features and conjunctions of features. This could be expected because features in higher levels are based on the features from preceding levels. However, more detailed studies of features and pooling mechanisms over multiple levels are required. I suggest that regarding global/central capacity limitations, local lateral inhibition, and crowding effects as different instances of spatial pooling within a multilevel neural network may help to move toward a general theory of computations in vision. 

Figure 7.

 

Combined efffects of local and global interference (Põder, 2017). Examples of stimuli where interitem distance and set size were varied (top). Example of the results of an individual experiment (middle). Symbols depict data and lines are model fits. Set sizes: 2 = blue circles, 4 = red rectangles, and 8 = green triangles. Results plotted in the space of global capacity limitations and crowding distance (bottom). Green markers represent simple features (S, size; O, orientation; C, color), blue markers = conjunctions, and red markers = rotated Ts.

Amendments to the feature integration theory

Integration of visual features by means of spatial attention (Treisman & Gelade, 1980) has been an influential theory in vision research. However, it appears that artificial neural networks can recognize objects without anything like this. Some researchers have suggested that the feature integration function of attention is unnecessary (e.g., Rosenholtz, Huang, & Ehinger, 2012). I argue that spatial attention has an important role, and, with slight modifications, the feature integration idea fits well the neural network model too. 

The original feature integration model (Treisman & Gelade, 1980) supposed that “spotlight” of attention binds visual features by selecting one object at a time. Assuming minimum radius of spotlight /attentional window is 0.5E, it is very likely that several objects fall into it simultaneously. How will feature integration work in that case? Crowding studies suggest that human vision uses probabilistic selection of features within an attentional window, with probabilities determined by a weighting function of the window (Põder & Wagemans, 2007; Vul & Rich, 2010; Figure 8). 

Figure 8.

 

Reporting a target surrounded by flankers with different features reveals independent sampling of features within an attentional window. Displays from Põder and Wagemans, 2007 (left; target in the center of a group), and Vul and Rich, 2010 (right; target indicated by spatial cue; reprinted by permission of SAGE Publications Inc).

The classic feature integration model deals with conjunctions of different features having the same spatial positions (Treisman & Gelade, 1980; Wolfe, Cave, & Franzel, 1989). However, the perception of feature configurations, or relative position of features, is likely the most attention-dependent task for human vision (Bergen & Julesz, 1983; Wolfe, 1998a; Põder, 1999; Gilden, Thornton, & Marusich, 2010; Palmer, Fencsik, Flusberg, Horowitz, & Wolfe, 2011). Simple spotlight model cannot explain the role of attention in this type of tasks. However, there is a plausible way to extend the classic model (Figure 9). A small set of neurons with spatially shifted integration fields can encode position of a given feature within an area covered by these receptive fields (e.g., Baldi & Heiligenberg, 1988). These coarse feature-position signals carry some position information even when pooled over the full visual field. Relative position of two features can be decoded when there are no irrelevant similar features in a display, or when signals from irrelevant features are excluded by spatial attention. 

Figure 9.

 

A hypothetical mechanism of encoding spatial positions and decoding of relative positions. Relative position information is available over a wide area, but similar features within the attentional window will make it ambiguous.

According to the traditional view, there is a set of “simple” visual features that are registered automatically/preattentively in every location of the visual field. The exact number of simple features is unknown, but usually, 10 to 20 candidates have been listed (e.g., Wolfe, 1998b). 

However, we can, without focused attention, perceive quite complex differences in visual images, including statistical properties of feature distributions, or “gist” of scenes (Ariely, 2001; Torralba & Oliva, 2003). Portilla and Simoncelli (2000) had to adjust about 700 “features” (mostly spatial correlations of wavelets—a kind of simple features) to make two images indiscriminable without focused attention. Although a part of these complex features could be redundant or unnecessary for peripheral vision (Balas, 2006; Ackermann & Landy, 2014), additional features could be needed to account for some results (Wallis, Funke, Ecker, Gatys, Wichmann, & Bethge, 2019). Anyway, the number and complexity of preattentive features are likely much larger than supposed in early studies. 

Conclusions

In this study, I attempted to combine the main psychophysical findings on visual crowding, attention, and central capacity limitations with hierarchical neural network model. I suggest that the similar principles of pooling and selection, at various levels of visual processing, can explain different psychophysical phenomena—visual crowding and central capacity limitations. An important factor seems to be the level where spatial attention is applied. Regardless of similar principles, the exact computations may vary across levels. For example, different pooling rules (averaging, max, correlation) may dominate at different levels of processing. 

Recent crowding studies have reported many results that contradict simple pooling over fixed receptive fields. Different explanations from qualitative grouping account to relatively complex computational models have been proposed. I believe that a notion of saliency could be useful here. There is a lot of evidence for similarity-dependent lateral inhibition at different levels of biological visual systems. Similar calculations have been used in machine vision as well (e.g., Jarrett, Kavukcuoglu, Ranzato, & LeCun, 2009). Several simple cases from crowding experiments apparently fit well to this simple model. In more complex cases, saliency can be calculated at several levels of visual processing, and candidate objects at different levels may compete for access to further processing. 

The present study suggests how classic ideas of attention and feature integration could be related to modern neural network models of vision. I suppose that the “simple” features to be combined should be a bit more complex, and the “spotlight” of attention has a minimum radius of about half of eccentricity. The amendments do not contradict the majority of earlier results because complex models can be reduced to simplified versions when traditional simple stimuli are used. 

Acknowledgments

The main points of this study were presented at the Workshop on Visual Crowding, June 23–24, 2019, Murten, Switzerland. I thank the organizers of this event. 

Commercial relationships: none. 

Corresponding author: Endel Põder. 

Email: [e-mail]. 

Address: Institute of Psychology, University of Tartu, Näituse 2, 50409 Tartu, Estonia.