The influence of clutter on real-world scene search: Evidence from search efficiency and eye movements

Abstract

We investigated the relationship between visual clutter and visual search in real-world scenes. Specifically, we investigated whether visual clutter, indexed by feature congestion, sub-band entropy, and edge density, correlates with search performance as assessed both by traditional behavioral measures (response time and error rate) and by eye movements. Our results demonstrate that clutter is related to search performance. These results hold for both traditional search measures and for eye movements. The results suggest that clutter may serve as an image-based proxy for search set size in real-world scenes.

Introduction

A critical task in human vision is to locate a sought-after object in the visual environment. In traditional research on visual search, a crucial variable has been set size, or the number of items in the display. In search within well-controlled psychophysical displays, set size accounts for a large proportion of the variance in search time (Wolfe, 1998). 

However, in pictures of natural scenes, it is far more difficult to identify a priori the number of “items” in the image that might reasonably be counted in determining set size. In fact, Neider and Zelinsky (2008) have argued that the concept of object-based set size has no clear meaning in real-world scenes (see also Bravo & Furid, 2004). Rosenholtz, Li, and Nakano (2007) noted that in real-world scenes, set size would seem to be better captured by something like object parts than by whole items. Rosenholtz and colleagues proposed that “clutter” may provide a stand-in or proxy for set size in natural scenes (Rosenholtz, Li, Mansfield, & Jin, 2005; Rosenholtz et al., 2007). Rosenholtz et al. operationalized clutter using three image-based measures: feature congestion, sub-band entropy, and edge density. Feature congestion can be thought of as local variability in specific image features such as color, orientation, and luminance. Sub-band entropy can be thought of as a measure of the efficiency with which an image can be encoded while maintaining perceived image quality (similar to the method used in JPEG image compression). Finally, edge density is a count of the number of edges in the image. An advantage of each of these measures is that they can easily be determined for complex scenes (compared to object-based set size) and can be precisely quantitatively specified. 

Rosenholtz et al. ( 2007) conducted a series of visual search experiments in complex full-color images to compare the relationship of each of the three clutter measures to search time. The experiments used images of geographic maps, and participants searched for either embedded Gabor patches or small arrows in each image. Though feature congestion accounts for the additional effect of color variability on search performance that the other measures do not capture, all three measures predicted search times relatively well, supporting the idea that image-based clutter can provide a stand-in for set size in scene search. 

In the present study, we sought to extend the research in this area in two important ways. First, we wanted to investigate the relationship between clutter and search in photographs of real-world scenes. Real-world scenes have particular properties that differentiate them from other complex images (Henderson, 2003; Henderson & Ferreira, 2004). Second, because search in complex scenes typically requires eye movements (Castelhano & Henderson, 2005; Henderson, Weeks, & Hollingworth, 1999; Neider & Zelinski, 2008), we wanted to examine the influence of clutter on both overall search time and on eye movement behavior. We specifically asked whether clutter, as assessed with the three image-based measures proposed by Rosenholtz et al. ( 2007), correlates with search performance in real-world scenes as assessed both by search performance and by eye movements. 

To investigate these issues, we used a difficult visual search task in which viewers were asked to search for and discriminate small Ts and Ls embedded in photographs of real-world scenes (Brockmole, Castelhano, & Henderson, 2006; Brockmole & Henderson, 2006; see also Wolfe, Oliva, Horowitz, Butcher, & Bompas, 2002). If search becomes more difficult in more cluttered real-world scenes, as has been found in complex images of maps (Rosenholtz et al., 2007), then we should observe an inverse relationship between degree of clutter and search efficiency. 

Methods

Participants

Sixteen Edinburgh University first year Psychology students each took part for £6. 

Apparatus

Eye position was sampled at 1000 Hz with an SR Research Eyelink 1000 eyetracker. The experiment was controlled with Experiment Builder software. Viewing was binocular but only the right eye was tracked. Saccades were detected using SR Research's saccade detection algorithm. This algorithm used a 17-sample model with a velocity criterion of 50°/s and a minimum amplitude of 0.5°. Fixations were defined as any time when the eyes were not in a saccade or blink. No minimum duration criterion was set for fixations. Images were presented on a 21″ CRT with a refresh rate of 140 Hz. 

Stimuli

Participants were presented 60 unique full-color 800 × 600 pixel 24-bit photographs of real-world scenes (subtending a visual angle of 25.7° × 19.4°) from a variety of sources (e.g., on-line and collections of laboratory members) depicting a variety of indoor and outdoor scene categories. Scenes did not include people or animals in order to minimize distractions from the search task. A gray letter T or L (Arial 9-point font) was embedded in each scene (Brockmole & Henderson, 2006). Letters were used to ensure that the target location could not be predicted by local or global scene content. The letter subtended 0.3 deg horizontally and vertically. Placement of each search target was random although positions close to the screen center were avoided. To ensure that the search target would be identifiable and would not be mistaken for an edge in the image, any search targets randomly positioned on the border between regions differing in color or texture were jittered a few pixels to a nearby constant region. 

Procedure

Participants were instructed to search the scenes as quickly as possible for a small gray superimposed letter. Once found, participants indicated whether the target was a T or L by pressing the appropriate button on a joypad (Microsoft Sidewinder). Before beginning the main experiment, participants were shown two example scenes with the search targets highlighted and then given three practice trials. The main experiment only began once participants understood the search task. During the main experiment, in the event that the letter was not found, the trial timed out after about 6000 ms. ¹ 

Results

To investigate search behavior as a function of clutter, we computed the three measures of clutter proposed by Rosenholtz and colleagues using the algorithms and code described by Rosenholtz et al. (2007). Figure 1 presents a visual illustration of feature congestion and edge density for two of the scenes used in the present study. (It is not possible to convert sub-band entropy into an equivalent graphical representation.) 

Figure 1

 

Examples of two scenes used in the experiment (Panels A and B), and a depiction of edge density (Panels C and D) and feature congestion (Panels E and F) for those scenes. The edge density and feature congestion maps for the two scenes were each generated using the same scales.

To assess the relationship between each measure of clutter and search performance, we examined seven dependent measures derived from search behavior and analyzed these measures as a function of the three clutter indexes. The measures were search efficiency assessed by search response time and missed targets; search initiation time assessed by latency for first saccade following onset of the scene; mean fixation duration; and mean saccade amplitude. In a secondary analysis, we examined fixation duration as a function of the local clutter value around each fixation point. 

Search efficiency

We first examined two standard measures of search efficiency, response time, and error rates, as a function of clutter. The results can be seen in Figures 2 and 3. Each data point in the figures (and entered into the statistical analyses) represents the value of the dependent variable averaged over participants for one scene. All three clutter indexes were significantly related to search performance as measured by response time ( R ² = 0.27, 0.18, and 0.28 for feature congestion, sub-band entropy, and edge density, respectively, p's < 0.001), though numerically the correlations for feature congestion and edge density were larger. In the case of search failure, defined as trials in which the target was not located, all three clutter indexes were significantly related to search performance ( R ² = 0.16, 0.28, and 0.29, all p's < 0.005). 

Figure 2

 

Search response time (ms) as a function of scene clutter.

Figure 3

 

Search failure (missed targets) as a function of scene clutter.

Eye movements

During search in real-world scenes, viewers typically make a series of eye movements as they seek the target. Therefore, in addition to response time and error rates, we also investigated the degree to which eye movements during search reflect visual scene clutter. Figure 4 presents a representative scan pattern of one participant in one search scene in the present study. 

Figure 4

 

Example of a scan pattern during search. The blue circles depict fixations (circles scaled by fixation duration) and the yellow arrows depict saccades. The yellow box at the end of the scan pattern surrounds the target letter (the box was not presented to the participant).

Search initiation time

We first examined the influence of clutter on the latency of the initial saccade following the appearance of the scene. This measure serves as an index of the time taken by the viewer to begin the first scanning eye movement. Increasing set size has previously been shown to increase the time taken to search an array and the duration of the initial fixation (Zelinsky, 2001). We might expect that the time to begin the search would increase with clutter since it might take more time to choose a potential first saccade target. In agreement with this hypothesis, initial saccade latency showed a significant positive relationship with both sub-band entropy and edge density (R² = 0.11 for both clutter measures, p's < 0.05); feature congestion did not produce an effect (R² = 0.01, ns). These results are shown in Figure 5. 

Figure 5

 

Search initiation time (ms) as a function of scene clutter.

Fixation duration

To investigate the relationship between clutter and eye movements during search, we examined mean fixation durations. Given that fixation durations reflect moment-to-moment processing difficulty during both scene memorization (Henderson & Pierce, 2008; Henderson & Smith, in press) and scene search (Henderson & Smith, in press), we might expect that a more cluttered scene would lead to longer average fixation durations. To test this prediction, we looked at the influence of clutter on mean fixation durations for the 5534 fixations within the first 1 s of search (Figure 6)². We observed a relationship between clutter and fixation duration for all three measures of clutter (R² = 0.08, 0.17, and 0.14 for feature congestion, sub-band entropy, and edge density, respectively; all p's < 0.05). 

Figure 6

 

Mean fixation duration over the first 1 s of viewing as a function of scene clutter.

Saccade amplitude

We also examined the mean saccade amplitude as a function of the three clutter measures. As with fixation duration, we looked at initial saccade amplitude and saccade amplitude over the first 1 s of viewing. Unlike fixation duration, there was no relationship between clutter and saccade amplitude for any measure of clutter or any measure of amplitude (all R ² < 0.025, ns). 

Influence of local clutter on fixation duration

Given that we observed an influence of global scene clutter on fixation durations, we sought to examine the nature of this effect more carefully. Specifically, we examined the degree to which clutter around the current fixation point determined the duration of that fixation. To investigate this question, we defined square regions of two sizes (30 pixels and 100 pixels; approximately 1 and 3.3 deg of visual angle, respectively) around each fixation point and determined the fixation duration for each of the 5534 fixations in the first 1 s of viewing. In contrast to global clutter, we observed no relationship between local clutter and fixation durations for regions of either size (all R ² = 0.00, ns; see Figure 7). Together with the global clutter data, these results demonstrate that fixation durations are influenced by global scene clutter even when they are not influenced by local clutter around fixation. 

Figure 7

 

Fixation duration as a function of scene clutter surrounding the fixation point, for the first 1 s of viewing. Panels A and B show data for a region of 1 deg around fixation and panels C and D show a region of 3.3 deg. Panels A and C show feature congestion and panels B and D show edge density.

Influence of local clutter on fixation location

Finally, we examined the degree to which clutter around the current fixation point changed as a function of ordinal fixation number (1st, 2nd, 3rd, etc. fixation) for the first 6 fixations produced by participants following scene presentation. We again defined square regions of two sizes (30 pixels and 100 pixels; approximately 1 and 3.3 deg of visual angle, respectively) around each fixation point. We then determined the average degree of clutter for each fixation (blue lines in each panel of Figure 8) and for a random sample of locations within each scene (green lines). We found that the first viewer-determined fixation (Fixation 1 on the blue line in each panel of Figure 8) tended to be centered on a region of significantly higher clutter than would be predicted by chance (feature congestion, 1 deg,: t(15) = 9.774, p < 0.001; 3.3 deg.: t(15) = 7.516, p < 0.001; edge density, 1 deg.: t(15) = 7.085, p < 0.001; 3.3 deg.: t(15) = 6.43, p < 0.001). By the next fixation the local clutter is indistinguishable from the baseline. 

Figure 8

 

Average local clutter surrounding the fixation point as a function of ordinal fixation number, for the first 6 fixations. Top panels show data for a region of 1 deg around fixation and bottom panels show a region of 3.3 deg. Left panels show feature congestion and right panels show edge density. The blue curves represent clutter around viewer fixations. The green curves represent clutter around an equivalent number of randomly chosen locations within the same scenes. Error bars show 1 standard error of the mean.

These results suggest that viewers started their search by fixating a more cluttered scene region. Exactly why they did this, and why the tendency dropped off after the first fixation, is unclear. The tendency to first fixate a cluttered region could be a conscious strategy to initially check a region of the scene in which the search target would be harder to find. However, the tendency could also be an artifact of visual saliency, since feature congestion and edge density are constructed from the same visual features that are assumed to contribute to saliency (e.g. Itti & Koch, 2001). The local saliency around fixation has sometimes been found to be higher for the first viewer-determined fixation following scene presentation than subsequent fixations (Foulsham & Underwood, 2008; Parkhurst, Law, & Niebur, 2002). This relationship between clutter and saliency, both in terms of how the factors are modeled and how we as viewers process and respond to them, requires future investigation. 

General discussion

The purpose of the present study was to investigate whether clutter, an image-based measure of visual complexity, can serve as a proxy for set size in real-world scene search. Rosenholtz et al. (2005; see also Bravo & Farid, 2008) observed a relationship between several measures of clutter and search in complex images of maps. We sought to extend this work to search in photographs of real-world scenes, using both traditional search measures and eye movement behavior as dependent measures. 

Overall, although there was some variability across clutter measures depending on the specific analysis, our results demonstrate that clutter does correlate with both global search efficiency (measured by search time and search failure) as well as with eye movement behavior during search. The latter result is novel and provides the first direct evidence that eye movements during scene search are influenced by the degree of clutter present in the scene. 

It is interesting that edge density, despite failing to capture color variability in an image, does not do significantly worse than feature congestion and sub-band entropy in predicting search efficiency. In fact, edge density was the only clutter measure to significantly correlate with all of the reported dependent measures. This effect is generally consistent with the observation that eye movements during scene viewing are very similar for color and gray-scale versions of the same pictures (Henderson & Hollingworth, 1998). The edge density result is also interesting in light of the finding that edges correlate with fixation location in real-world scenes better than luminance contrast (Baddeley & Tatler, 2006) and the finding that color is not a strong correlate of fixation location (Tatler, Baddeley, & Gilchrist, 2005). As in the case of feature congestion, visual saliency is a more complex measure than edge density because it takes additional image variability including color into account. The present results converge with those Baddeley and Tatler (2006) and Tatler et al. ( 2005) in suggesting that the most important image property for predicting search efficiency at both a macro (e.g., response time) and micro (e.g., eye movement) level of analysis may be edges. 

The influence of clutter in the present experiment, though statistically significant, generally accounted for a relatively small amount of the variance in each of the measures of search. Also, the influence of clutter was relatively small compared to the influence of set size typically observed in standard visual search tasks (Wolfe, 1998). Why was the influence of clutter not more pronounced? We suspect that there may be several reasons. The measures of clutter proposed by Rosenholtz and used here may be only an approximation of perceived clutter, which may also take into account other factors beyond those included in the three measures. In addition, other image effects such as crowding, masking, and so forth are very likely to play a role in determining search difficulty. Higher level image features related to scenes, such as those proposed by Torralba and Oliva (2003) in their spatial envelope theory (e.g., degree to which the scene is open–closed, natural–artificial, and near–far), could also be important for search in natural scenes. Finally, higher level cognitive factors like the semantic consistency of the objects in the scene, scene coherence, the degree to which the scene is familiar, and so on may all play a role in search efficiency (Henderson, 2007). These higher level factors are clearly more difficult to capture with image-based indexes. Both additional image features and higher level factors would contribute error variance to search performance and so reduce the correlations of search with clutter. 

In summary, we found that clutter correlates with search performance in real-world scenes. Furthermore, we have provided the first evidence that clutter also predicts eye movement characteristics during real-world search. These data converge with those presented by Rosenholtz et al. in showing that an image-based proxy for search set size can be related to search performance in real-world scenes. 

Acknowledgments

We thank Ruth Rosenholtz for generously providing her Matlab code for Feature Congestion, Sub-band Entropy, and Edge Density, and for her helpful discussion of this research. We also thank the other members of the Visual Cognition Lab and three anonymous reviewers for their critical comments. This project was supported by a grant from the Economic and Social Science Research Council UK (RES-062-23-1092) to JMH. 

Commercial relationships: none. 

Corresponding author: John M. Henderson. 

Email: [e-mail]. 

Address: Psychology Department, 7 George Square, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom. 

Footnotes

Footnotes

1  An additional manipulation was included in each trial, but this was irrelevant for the purposes of the current study. This manipulation involved presenting a brief luminance onset following the first 1000 ms of scene exploration. After the onset, the scene remained in view for 5000 ms. The data presented here did not include saccades and fixations immediately following the sudden onset.

Footnotes

2  The data pattern did not differ when fixations over the entire trial were included, but the onset at 1 s could potentially affect the next few fixations as shown by Brockmole and Henderson ( 2006), so we based the reported analyses only on fixations prior to the onset event.