How spatial frequencies and color drive object search in real-world scenes: A new eye-movement corpus

Abstract

When studying how people search for objects in scenes, the inhomogeneity of the visual field is often ignored. Due to physiological limitations, peripheral vision is blurred and mainly uses coarse-grained information (i.e., low spatial frequencies) for selecting saccade targets, whereas high-acuity central vision uses fine-grained information (i.e., high spatial frequencies) for analysis of details. Here we investigated how spatial frequencies and color affect object search in real-world scenes. Using gaze-contingent filters, we attenuated high or low frequencies in central or peripheral vision while viewers searched color or grayscale scenes. Results showed that peripheral filters and central high-pass filters hardly affected search accuracy, whereas accuracy dropped drastically with central low-pass filters. Peripheral filtering increased the time to localize the target by decreasing saccade amplitudes and increasing number and duration of fixations. The use of coarse-grained information in the periphery was limited to color scenes. Central filtering increased the time to verify target identity instead, especially with low-pass filters. We conclude that peripheral vision is critical for object localization and central vision is critical for object identification. Visual guidance during peripheral object localization is dominated by low-frequency color information, whereas high-frequency information, relatively independent of color, is most important for object identification in central vision.

Introduction

Searching for an object in a visual scene is a vital task we perform countless times in our daily lives. During search, we make about three eye movements per second called saccades, which rapidly shift our gaze to new points of interest in the scene. This is necessary because high-acuity vision is only achieved in the fovea, the central 2° around fixation; toward the visual periphery, resolution falls off rapidly (Jones & Higgins, 1947; Wertheim, 1894). Thus, fine-grained scene information, which is carried by high spatial frequencies, is processed best in central vision (from the point of fixation up to about 4–5° eccentricity, cf. Larson & Loschky, 2009). Central vision is necessary for analyzing details in a scene, identifying objects (Henderson & Hollingworth, 1999; Henderson et al., 2003), and establishing object memory (Geringswald et al., 2016). Low-resolution peripheral vision, on the other hand, is best suited for processing coarse-grained information, which is carried by low spatial frequencies. Peripheral vision is mainly used for rapid reorienting and selecting new regions of interest as saccade targets. Central and peripheral vision can therefore be considered to serve different tasks (Gilchrist, 2011). However, there are not many studies directly testing the consequences of the inhomogeneity of the visual field for scene perception, and not much is known about the different contributions of central and peripheral vision to object search in real-world scenes (but see, e.g., Nuthmann, 2013, 2014; Nuthmann & Malcolm, 2016). 

The present study investigates central and peripheral vision during object-in-scene search. We were interested in whether the two parts of the visual field differ in the use of fine- and coarse-grained information and whether color or brightness contrasts modulate its use. To this end, we attenuated high or low spatial frequencies in central or peripheral vision during search using gaze-contingent low-pass or high-pass filters, respectively. Given the different sensitivities of the central and peripheral visual field to certain spatial-frequency bands (e.g., Hilz & Cavonius, 1974), differences in search performance and eye-movement behavior can be expected depending on which frequencies are filtered in central and peripheral vision. 

Previous studies show that saccade target selection during scene viewing is modulated by the available spatial-frequency information across the visual field: Saccades preferentially target unfiltered scene regions, as peripheral filtering decreases and central filtering increases saccade amplitudes (Cajar et al., 2016, 2016; Foulsham et al., 2011; Laubrock et al., 2013; Loschky & McConkie, 2002; Loschky et al., 2005; Nuthmann, 2013, 2014). These saccadic changes have been shown to reflect attentional modulations: Peripheral filtering induces tunnel vision with a narrowing of the attentional focus, and central filtering induces an attentional bias toward the periphery (Cajar et al., 2016). In scene memorization tasks, these saccadic modulations were larger when spatial frequencies that the respective part of the visual field is most sensitive to were attenuated, that is, with central low-pass and peripheral high-pass filtering (Cajar et al., 2016, 2016; Laubrock et al., 2013). Fixation durations, on the other hand, did not increase as much or even decrease compared with unfiltered scene viewing in these conditions. Durations increased more with central high-pass and peripheral low-pass filtering, which preserve critical spatial frequencies, suggesting that fixation durations only prolong when the available information can be processed in a reasonable amount of time (Cajar et al., 2016, 2016; Laubrock et al., 2013). This behavior can be explained by a computational model for fixation durations during scene viewing that assumes parallel processing of central and peripheral information during fixation with dynamical interactions between central and peripheral processing (Laubrock et al., 2013). 

These previous findings imply greater scene processing difficulty when high spatial frequencies are attenuated in central vision and low spatial frequencies are attenuated in peripheral vision. The results derive from scene memorization tasks, however, with hardly any effects of filter type on task performance (Cajar et al., 2016; Laubrock et al., 2013). Although we assume that viewers adjust their fixation duration in order to maintain a given performance criterion, we therefore cannot be sure to what extent the available information in central and peripheral vision was actually used or needed for the task. With object-in-scene search, Nuthmann (2013, 2014) showed that fixation durations increase both with central and peripheral low-pass filtering, suggesting that high frequencies are needed in both scene regions for efficient search. Thus, in the present experiment, object-in-scene search was given as a task as well, which implies more top-down control and predictability of what viewers do during each fixation (Henderson et al., 2009; Tatler et al., 2011). Search involves two processes that possibly occur in parallel. First, viewers need to analyze the fixated stimulus to decide whether it is the target. This process involves object identification at least to the degree that a rejection decision is possible. Second, if the fixated stimulus is not the target, viewers need to analyze the peripheral stimulus to choose a new fixation location that likely contains the search target (van Diepen et al., 1998). The importance of different spatial frequencies for accomplishing these tasks during fixation is assessed better with a search than a memorization task. 

Search studies show that viewers locate scene regions with features similar to the search target (Hwang et al., 2009) or regions likely containing the target according to scene context (Neider & Zelinsky, 2006; Spotorno et al., 2014). Consequently, search is more efficient when the target is cued with pictures rather than words (Malcolm & Henderson, 2009; Nuthmann & Malcolm, 2016) and when targets are located at predictable compared with unpredictable locations with respect to scene context (Malcolm and Henderson, 2010). Higher search efficiency in these studies was reflected in both shorter scanning times and shorter verification times. Scanning time reflects the time it takes to locate the target in the scene (i.e., fixate it for the first time), and verification time reflects the time it takes to verify the identity of the target once it has been fixated. Nuthmann and Malcolm (2016) also showed that removing color from the scene increases scanning and verification times and impairs performance by decreasing search accuracy and increasing search times. Castelhano et al. (2008) reported that target typicality only modulated verification times but not scanning times in object search arrays, with faster verification when targets were more (proto)typical for their object category. Moreover, attenuating high spatial frequencies has been shown to impede search performance differently when applied to central or peripheral vision, with peripheral low-pass filtering increasing scanning times and central low-pass filtering increasing verification times (Nuthmann, 2013, 2014). These results suggest that peripheral vision aids object localization, whereas central vision aids object verification. 

The present work aims to extend the findings by Nuthmann (2013, 2014) by qualifying whether low or high spatial frequencies are more important in central and peripheral vision for object-in-scene search. To this end, we applied gaze-contingent high-pass or low-pass filters to central or peripheral vision while viewers searched for target objects that were either present or absent in the scenes. Additionally, we investigated search in both color and grayscale scenes to assess the effect of color on the importance of low or high spatial frequencies in different parts of the visual field, thus extending the findings by Nuthmann and Malcolm (2016) on the contributions of color in central and peripheral vision during scene search. 

We expected high frequencies, which are best resolved in central vision, to be critical for object identification, as they aid object–background segregation and analysis of detail. Search performance and eye-movement behavior should thus be more impaired by central low-pass filtering than central high-pass filtering. This hypothesis is challenged by the fact that high-pass filtering inherently attenuates features like luminance, color, and contrast in addition to low spatial frequencies (see Figure 1); thus, to the naive observer, high-pass filtering might appear more artificial and disruptive for processing than low-pass filtering. When applying spatial-frequency filters only to central vision, however, research indicates that high-pass filtering is more beneficial for scene categorization (Peyrin et al., 2003) and for eye-movement control (e.g., Cajar et al., 2016) than low-pass filtering. Object verification should take substantially longer with central filtering, especially low-pass filtering, whereas peripheral filtering should not impair target verification once the target has been located. 

Figure 1.

 

Illustration of the five filter conditions. The red cross indicates the current gaze position. In this example, the target stimulus is the watering can (which stands on the lawn in front of the shed with the predictable location condition and on the roof of the shed with the unpredictable location condition). The top panel shows the original stimulus in the unfiltered control condition. Below, the four filter conditions are illustrated with cropped and zoomed-in versions of the original stimulus (indicated by the red frame) to better illustrate the filter effects. (Middle row, left) Central low-pass filter. (Middle row, right) Peripheral low-pass filter. (Bottom row, left) Central high-pass filter. (Bottom row, right) Peripheral high-pass filter. Unfiltered image retrieved from http://info.ni.tu-berlin.de/photodb/ (see Mohr et al., 2016).

In contrast, we assume that peripheral vision is critical for object localization. We thus expected scanning times to prolong only slightly with central filtering, which slows down object rejection, but to prolong severely with peripheral filtering, which impairs saccade target selection. Since high frequencies can hardly be resolved in peripheral vision, search performance and eye-movement behavior should be more impaired with peripheral high-pass than low-pass filtering. 

Because color is an important feature for object search and identification (Hwang et al., 2007), we expected search to be more difficult in grayscale scenes than color scenes. Although color sensitivity decreases with increasing eccentricity, cones are spread throughout the retina, with their density not decreasing much beyond 10 degrees (Wells-Gray et al., 2016), so that color can still be used quite effectively in the periphery (e.g., Abramov et al., 1991; Hansen et al., 2009; Johnson, 1986; see also Nuthmann & Malcolm, 2016). The absence of color in peripheral vision should therefore increase scanning times, and its absence in central vision should increase verification times. In grayscale scenes, the inherent attenuation of color with high-pass filtering plays no role, so high-pass filtering was expected to be less impairing than low-pass filtering compared with color scenes. On the other hand, search with low-pass filtering should be more difficult per se when color is missing as a feature. We therefore expected smaller differences between filter types in peripheral vision and larger differences in central vision when searching grayscale scenes compared with color scenes. 

Methods

The search experiment was part of a large scene corpus study with N = 200 participants, where each participant inspected 90 scenes for a memorization task in one session and 120 different scenes for an object-in-scene search task in another session (data and analyses scripts are available on Open Science Framework, DOI:10.17605/OSF.IO/JQ56S). Session order was counterbalanced. 

Participants

The two hundred participants (38 male, mean age: 22.6 years, range: 16 to 40 years) were students at the University of Potsdam or pupils at local schools. Participants had normal or corrected-to-normal vision and normal color discrimination. They were naive as to the purpose of the experiment and received course credit or monetary compensation for their participation. The experiment conformed to the Declaration of Helsinki. Participants gave their written informed consent prior to the experiment. 

Apparatus

Stimuli were presented on an iiyama VisionMasterPro 514 monitor with a resolution of 1,024 × 768 pixels and a refresh rate of 150 Hz. Stimuli and response collection were controlled with MATLAB (The Mathworks, Natick, MA) using the Psychophysics Toolbox (Brainard, 1997; Kleiner et al., 2007) and the Eyelink Toolbox (Cornelissen et al., 2002). Viewers were seated 60 cm (23.6 in.) away from the monitor with their head stabilized by a head-chin rest. Gaze position of the dominant eye was tracked during binocular viewing with the EyeLink 1000 system (SR Research, Ontario, Canada). 

Stimuli and design

Stimuli were 120 images of real-world scenes from the BOiS database (Mohr et al., 2016) resized to 1,024 × 768 pixels, subtending a visual angle of 38.2° × 28.6°. Of these images, 99 depicted indoor and 21 depicted outdoor scenes. Each scene could be presented in one of three versions: with the target object present at a predictable location regarding scene context (e.g., a watering can standing on the lawn in a garden; see Figure 1), an unpredictable location (e.g., a watering can standing on the roof of a shed), or absent from the scene. 

For each scene, low-pass and high-pass filtered versions were prepared in advance. Filtering was realized in the Fourier domain with Gaussian filters. Cutoff frequencies for low-pass and high-pass filters were 1 c/° and 9 c/° respectively (cutoffs were defined as the half power point, where the filter response is reduced to 0.5, that is, −3 dB in the power spectrum, which corresponds to 1/\(\sqrt{2}\) in the amplitude spectrum). Thus, low-pass filtering attenuated spatial frequencies above 1 c/° and high-pass filtering attenuated spatial frequencies below 9 c/°. These cutoffs are near the maximal sensitivities of magno- and parvocellular cells, respectively, in the lateral geniculate nucleus (Derrington & Lennie, 1984). To give the reader a rough estimate of how far off into the periphery the cutoff frequencies were visible, we calculated the maximal eccentricities they were visible at according to the cortical magnification principle. On the superior (worst contrast sensitivity) and temporal (best contrast sensitivity) half-meridian of the visual field, the cutoff frequencies should become invisible at the eccentricities of 41.5° and 73.3° respectively for spatial frequencies of 1 c/° and 6.8° and 9.9° for spatial frequencies of 9 c/°. For details about how these values were calculated, we refer to Cajar et al. (2016, p. 188). 

For gaze-contingent filtering in the central or peripheral visual field, a foreground and a background image were merged in real time using alpha blending. With central high-pass filtering, for example, the foreground image was the high-pass filtered version of the scene and the background image was the original scene. A two-dimensional hyperbolic tangent with a slope of 0.06 served as a blending function for creating the alpha mask. The inflection point of the function corresponded to the radius of the gaze-contingent window, which was 5°, roughly dividing central from peripheral vision (see also Larson & Loschky, 2009; Loschky & McConkie, 2002; Nuthmann & Malcolm, 2016). The alpha mask was centered at the current gaze position and defined the transparency value, which constitutes the weighting of the central foreground image at each point. At the point of fixation, only the foreground image was visible; with increasing eccentricity, the peripheral background image was weighted more strongly until it was fully visible. 

Two filter locations (central/peripheral visual field) were crossed with two filter types (low pass/high pass), yielding four filter conditions: central low pass, central high pass, peripheral low pass, and peripheral high pass (for example stimuli, see Figure 1). A control condition without filtering served as a baseline. This resulted in 24 trials per condition in the search task. Half of those trials were target-absent and the other half target-present trials; of the latter, half of the trials presented the target at a predictable and the other half at an unpredictable location. For half of the participants, scenes were presented in their original color version; for the other half of the participants, the same scenes were presented in grayscale. 

A Latin square design ensured counterbalancing of condition–scene–target location assignments across participants. Scenes were presented in random order. 

Procedure

At the beginning of the experiment and after every 15 trials, a 9-point calibration was performed. Each trial started with a fixation check where a fixation point was presented at the center of the screen. The viewer’s gaze had to stay within an area of 1.5° × 1.5° around the fixation point for 200 ms to pass the fixation check. After a successful check, the actual trial started; if the check failed three times, a recalibration was scheduled. 

Three example trials familiarized participants with the task and the gaze-contingent window procedure. Each trial started with a picture cue of the target object on a black background presented for 2 s. After that, a black cross was presented in the center of the screen to ensure viewers always started exploring in the center of the image. After the cross was fixated, the scene was revealed. Viewers were instructed to search for the target object in the scene and to decide as fast as possible whether the object was present or absent in the scene by pressing the left or right button of the computer mouse, respectively. The response deadline of 60 s was never actually reached in the experiment. 

Data preparation

Saccades were detected in the raw time series of gaze positions using a velocity-based algorithm (Engbert & Kliegl, 2003; Engbert & Mergenthaler, 2006) with a relative velocity threshold of six standard deviations and a minimum duration of eight data samples. A total of 68 trials (0.28%) were removed owing to poor calibration or too much data loss. Single fixations and saccades were removed if they neighbored eye blinks or were outside the monitor area. If the first or last trial event was an ongoing saccade, it was also removed. Since the present study focuses on target present trials, target-absent trials were excluded from analyses. Thus, 11,966 trials remained for analyses of task performance, and 115,390 fixations and 105,915 saccades remained for eye-movement analyses. For analyzing fixation durations, the last ongoing fixation of each trial was excluded. 

Data analyses

Data from trials with target objects at contextually predictable and unpredictable locations were collapsed for all analyses. For effects of target predictability on search performance and eye movements, see Figures 4 to 6 in the Appendix. 

Separate models were run for color and grayscale scene data. Search times and eye movements were analyzed with linear mixed-effects models (LMMs) and search accuracy was analyzed with binomial generalized linear mixed-effects models (GLMMs) with a logit link function. Both LMMs and GLMMs are implemented in the lme4 package (Bates et al., 2015), which is supplied in the R system for statistical computing (version 3.6.0; R Core Team, 2018). Besides fixed effects for the experimental manipulations, (G)LMMs account for random effects (i.e., variance components) due to differences between participants and scenes, which reduces unexplained variance. We assumed random intercepts for participants and scenes for all models. Fixed-effects parameters were estimated via treatment contrasts, testing the effects of each of the four filter conditions against the unfiltered control condition. Selected comparisons between filter conditions were done post hoc using paired t tests. To test interactions between spatial frequency and color filtering, we ran supplementary analyses with color as an additional fixed effect in the LMMs, providing main effects for color versus grayscale scenes and interactions between color and spatial-frequency filtering effects. For the GLMM on search accuracies, only a main effect of color was added to the model, as a model with additional interactions yielded no stable estimates. In order to keep the presentation succinct, we only report significant effects from these supplementary analyses in the Results section. 

All GLMM analyses yield regression coefficients, standard errors, z values, and p values for fixed effects. LMM analyses only yield regression coefficients, standard errors, and t values, because the degrees of freedom are not known exactly for LMMs. For large data sets, however, the t distribution has converged to the standard normal distribution for all practical purposes (Baayen et al., 2008, Note 1). Thus, t statistics exceeding an absolute value of 1.96 were considered statistically significant at the two-tailed 5% level. 

Since the distributions of all eye-movement variables and search times were positively skewed, variables were transformed before model fitting to approximate normally distributed model residuals. To find a suitable transformation, the optimal λ-coefficient for the Box-Cox power transformation (Box & Cox, 1964) was estimated with the boxcox function of the MASS package (Venables & Ripley, 2002), with y^(λ) = (y^λ − 1)/λ, if λ ≠ 0 and log (y), if λ = 0. For all variables except saccade amplitudes, λ was near zero and the log-transformation was chosen; for saccade amplitudes, the exact λ was chosen as a transformation (λ = 0.22 for both color and grayscale scene data), yielding considerably better distributions of model residuals compared with log-transformed data. 

Results

Search performance

In general, a similar pattern of results was observed with color and grayscale scenes regarding the effects of spatial-frequency filtering on search accuracy and search times (see Figure 2). Performance was overall better, however, when searching color scenes than grayscale scenes, reflected in higher accuracies (b = 0.62, SE = 0.09, t = 6.88, p < .001) and shorter search times (b = −0.22, SE = 0.04, t = −5.39). 

Figure 2.

 

Search performance for color and grayscale scenes. (Top row) Mean search accuracies. (Bottom row) Mean search times. Error bars represent within-subjects 95% confidence intervals with Cousineau-Morey correction (Cousineau, 2005; Morey, 2008).

Search accuracies

In the unfiltered control condition, mean search accuracy was 83% with color scenes and 78% with grayscale scenes. These mean accuracies are rather low, as they reflect collapsed data from search trials with targets at predictable and unpredictable locations, and viewers performed worse with the latter (see Figure 4 in the Appendix). When searching color scenes, accuracy decreased with all filter conditions. The decrease was pronounced with central low-pass filtering (b = −1.45, SE = 0.11, z = −12.92, p < .001) and mild in all other conditions (b = −0.38, SE = 0.12, z = −3.20, p = .001 for peripheral low-pass filtering; b = −0.38, SE = 0.12, z = −3.24, p = .001 for peripheral high-pass filtering; b = −0.37, SE = 0.12, z = −3.14, p = .002 for central high-pass filtering). The decrease in accuracy with central low-pass filtering seemed even more dramatic when viewers searched grayscale scenes (b = −1.78, SE = 0.11, z = −16.27, p < .001), where accuracy dropped to chance performance. With all other filter conditions in grayscale scenes, accuracy did not differ from the control condition (b = −0.13, SE = 0.11, z = −1.17, p = .240 for peripheral low-pass filtering; b = −0.13, SE = 0.11, z = −0.18, p = .854 for peripheral high-pass filtering; b = −0.18, SE = 0.11, z = −1.56, p = .119 for central high-pass filtering). 

Search times

With all filter conditions, search times increased compared with the unfiltered control condition, showing similar patterns for color and grayscale scenes. The increase in search times was weakest with central high-pass filtering (b = 0.16, SE = 0.02, t = 8.40 for color scenes and b = 0.12, SE = 0.02, t = 6.15 for grayscale scenes) and strongest with central low-pass filtering (b = 0.43, SE = 0.02, t = 22.18 for color scenes and b = 0.55, SE = 0.02, t = 28.20 for grayscale scenes). The increase in search times with central low-pass filtering was considerably higher when searching grayscale than color scenes (b = −0.13, SE = 0.03, t = −4.17), complementing the stronger decrease in search accuracy with grayscale scenes. The increase of search times with peripheral filters was intermediate between the increase with central low-pass and high-pass filters (b = 0.25, SE = 0.02, t = 13.12 and b = 0.34, SE = 0.02, t = 17.72 for peripheral low-pass and high-pass filters in color scenes, respectively; b = 0.22, SE = 0.02, t = 10.96 and b = 0.22, SE = 0.02, t = 11.41 for peripheral low-pass and high-pass filters in grayscale scenes, respectively). The increase in search times with peripheral high-pass filtering was stronger when searching color scenes compared with grayscale scenes (b = 0.12, SE = 0.03, t = 3.88). Post hoc comparisons showed a significant difference between filter types with peripheral filtering in color scenes (t(99) = 4.04, p < .001, d = 0.09) but not in grayscale scenes (t(99) = 0.38, p = .702, d = 0.01). 

Search epochs

In order to better understand the effects of spatial-frequency filtering on subprocesses of search and thus the effects on task performance, we decomposed the search process into scanning time (time until first fixation on the target) and verification time (time from first fixation on the target until manual response). Scanning time reflects target localization, that is, the actual search process, and verification time reflects target identification plus the manual response (see also Castelhano & Henderson, 2008; Malcolm & Henderson, 2009, 2010; Nuthmann, 2013, 2014). For classifying fixations as being on the target object or not, a polygon was drawn outlining the target and then enlarged by 1°. Thus, fixations landing 1° to the edge of the object still counted as target fixations, which accounts for oculomotor errors and eye tracker inaccuracies but ensures that near-foveal processing of the target is still possible. 

As with search accuracies and search times, search epochs showed a similar pattern for color and grayscale scenes but somewhat worse performance when searching grayscale scenes, reflected in longer scanning and verification times (see Figure 3). 

Figure 3.

 

Search epochs and eye movements with the five filter conditions for color and grayscale scenes. (Top row) Mean scanning times (left) and mean verification times (right). (Middle row) Mean fixation durations (left) and mean number of fixations (right). The inset figures with number of fixations display the mean number of fixations during verification time only. (Bottom row, left) Mean saccade amplitudes. Error bars represent within-subjects 95% confidence intervals with Cosineau-Morey correction (Cousineau, 2005; Morey, 2008). (Bottom row, right) Changes in saccade direction in angular degrees between successive saccades during verification time only. Changes of 0° and 180° reflect forward and backward saccades, respectively; positive and negative changes reflect upward and downward saccades, respectively.

Scanning times

Overall, scanning times were slightly shorter when searching color scenes compared with grayscale scenes (b = −0.08, SE = 0.04, t = −2.04). With both color and grayscale scenes, scanning times increased considerably with peripheral filtering but only slightly with central filtering compared with the unfiltered control condition. In color scenes, scanning times did not increase with central low-pass filtering (b = 0.02, SE = 0.03, t = 0.77) and only slightly with central high-pass filtering (b = 0.06, SE = 0.03, t = 2.10). With peripheral filtering, however, scanning times increased strongly (b = 0.43, SE = 0.03, t = 15.41 for peripheral low-pass filtering; b = 0.56, SE = 0.03, t = 20.15 for peripheral high-pass filtering), with significantly longer scanning times for peripheral high-pass filtering than low-pass filtering (t(99) = 3.61, p < .001, d = 0.13). Similar effects emerged with searching grayscale scenes. Scanning times increased slightly with both central low-pass and high-pass filtering (b = 0.06, SE = 0.03, t = 2.13 and b = 0.08, SE = 0.03, t = 2.82, respectively) but increased more strongly with peripheral filtering (b = 0.40, SE = 0.03, t = 13.67 for peripheral low-pass filtering; b = 0.35, SE = 0.03, t = 12.09 for peripheral high-pass filtering). The increase in scanning times with peripheral high-pass filtering was stronger in color scenes than grayscale scenes (b = 0.20, SE = 0.05, t = 4.36). Contrary to searching color scenes, there was no effect of filter type on scanning times when searching grayscale scenes, neither with central nor peripheral filtering (see Figure 3, top left graph). 

Verification times

Verification times were rather long in the present experiment, because scenes were complex, targets were located at unpredictable locations in half of the trials, and verification times included the manual response to the target. 

Verification times increased in most filter conditions compared with search in unfiltered scenes. Overall, verification times were shorter when searching color compared with grayscale scenes (b = −0.20, SE = 0.05, t = −3.71). For search in color scenes, verification times increased slightly with peripheral filtering (b = 0.06, SE = 0.03, t = 2.08 for peripheral low-pass filtering; b = 0.07, SE = 0.03, t = 2.34 for peripheral high-pass filtering). Verification times increased more strongly with central high-pass filtering (b = 0.19, SE = 0.03, t = 6.62) and drastically with central low-pass filtering (b = 0.69, SE = 0.03, t = 24.12). This pattern was similar when searching grayscale scenes, with no effect of peripheral low-pass filtering on verification times (b = 0.03, SE = 0.03, t = 1.11) and a moderate increase of verification times with high-pass filtering in central and peripheral vision (b = 0.11, SE = 0.03, t = 3.94 and b = 0.09, SE = 0.03, t = 3.28, respectively). The strong increase of verification times with central low-pass filtering compared with the control condition (b = 0.86, SE = 0.03, t = 28.97) was even higher when searching grayscale scenes than color scenes (b = −0.17, SE = 0.04, t = −4.06). Post hoc comparisons showed no difference in verification times between peripheral filter types with grayscale scenes (t(99) = 1.88, p = .006, d = 0.06). 

Eye movements

As with task performance and search epochs, effects of filtering on eye-movement behavior were similar with search in color and grayscale scenes, with stronger modulations of behavior when searching grayscale scenes (see Figure 3). 

Fixation durations

In both color and grayscale scenes, mean fixation durations increased with all filter conditions compared with the unfiltered control condition. Central filtering led to a stronger increase of fixation durations than peripheral filtering. For search in color scenes, central low-pass and high-pass filtering prolonged fixation durations similarly (b = 0.09, SE = 0.01, t = 13.34 and b = 0.10, SE = 0.01, t = 14.50, respectively). Numerically, the increase was stronger with high-pass than low-pass filtering, but the difference between central filter types was only marginally significant (t(99) = 1.79, p = .076, d = 0.02). The increase of fixation durations with peripheral filtering compared to the control condition was stronger with peripheral low-pass filtering (b = 0.06, SE = 0.01, t = 9.17) than with peripheral high-pass filtering (b = 0.04, SE = 0.01, t = 6.67). This difference between peripheral filter types was statistically significant (t(99) = 2.51, p = .014, d = 0.02). When searching grayscale scenes, there was no effect of filter type on fixation durations (see Figure 3), which increased similarly with central low-pass and high-pass filtering (b = 0.07, SE = 0.01, t = 12.70 and b = 0.08, SE = 0.01, t = 13.20, respectively) and with peripheral low-pass and high-pass filtering (b = 0.06, SE = 0.01, t = 8.93 and b = 0.06, SE = 0.01, t = 9.18, respectively). There was no main effect of color and there were no interaction effects between spatial-frequency filtering and color on fixation durations. 

Number of fixations

Viewers needed fewer fixations to find the target in color than in grayscale scenes (b = −0.22, SE = 0.03, t = −6.23). With color scenes, mean number of fixations was highest with central low-pass filtering (b = 0.28, SE = 0.02, t = 13.72) and peripheral high-pass filtering (b = 0.30, SE = 0.02, t = 14.89), with no significant difference between the two (t(99) = 0.73, p = .465, d = 0.02). Peripheral low-pass filtering also increased the number of fixations considerably (b = 0.20, SE = 0.02, t = 9.90), whereas central high-pass filtering increased number of fixations only slightly (b = 0.04, SE = 0.02, t = 2.06) compared with the unfiltered control condition. With grayscale scenes, a similar pattern emerged. Central high-pass filtering increased the number of fixations only slightly (b = 0.04, SE = 0.02, t = 2.01), whereas central low-pass filtering increased the number of fixations strongly (b = 0.43, SE = 0.02, t = 21.50), the increase being a lot higher than with color scenes (b = −0.16, SE = 0.03, t = −5.03). Also contrary to searching color scenes, the increase in the number of fixations with peripheral filters was similar with low-pass and high-pass filters (b = 0.18, SE = 0.02, t = 8.90 and b = 0.17, SE = 0.02, t = 8.67, respectively). Peripheral high-pass filtering did not increase the number of fixations as strongly compared with the control condition as it did in color scenes (b = 0.12, SE = 0.03, t = 3.99). 

The inset figures for mean number of fixations (see Figure 3) show the mean number of fixations during verification time. It appears that viewers make considerably more fixations during object verification with central low-pass filtering than with any of the other filter conditions, especially when searching grayscale scenes. 

Saccade amplitudes

The modulation of saccade amplitudes by spatial-frequency filtering was similar with color and grayscale scenes, although overall, amplitudes were shorter when searching color scenes (−0.02, SE = 0.01, t = −3.96). Saccade amplitudes were not affected by central high-pass filtering (b = 0.01, SE = 0.004, t = 1.94 for color scenes and b = −0.001, SE = 0.004, t = −0.26 for grayscale scenes) but strongly increased with central low-pass filtering (b = 0.14, SE = 0.004, t = 36.55 for color scenes and b = 0.16, SE = 0.003, t = 47.29 for grayscale scenes). Peripheral filtering, on the other hand, shortened saccade amplitudes compared with the control condition: For searching color scenes, the decrease in saccade amplitudes was slightly stronger with high-pass (b = −0.06, SE = 0.004, t = −14.78) than with low-pass filtering (b = −0.05, SE = 0.004, t = −11.63). This difference between peripheral filter types was statistically significant (t(99) = 2.50, p = .014, d = 0.01). For searching grayscale scenes, saccade amplitudes also decreased (b = −0.05, SE = 0.004, t = −15.01 for low-pass filtering and b = −0.05, SE = 0.004, t = −13.23 for high-pass filtering), but the difference between peripheral filter types was not significant (t(99) = −1.59, p = .116, d = −0.01). 

Changes in saccade direction

We expected target verification to be more difficult when central vision was impaired by spatial-frequency filters (especially low-pass filters) than when central vision was left intact. Viewers might therefore adopt different strategies for identifying the potential target object in the different filter conditions. To reveal viewers’ exploration strategies during target verification, we inspected changes in saccade direction in the different filter conditions, that is, saccadic angles between two successive saccades. The bottom right graph in Figure 3 shows the distributions of changes in saccade direction for the different filter conditions during target verification. Angles of 0° reflect forward saccades and angles of 180° reflect backward or return saccades; positive angles reflect upward and negative angles reflect downward saccades. Distributions show that central low-pass filtering involved a considerably higher amount of return saccades during target verification than the other filter conditions and the control condition, especially with search in grayscale scenes. To a weaker extent, this effect also emerged with central high-pass filtering. 

Discussion

In the present study, we investigated the importance of spatial frequencies and color in central and peripheral vision for eye-movement control during scene search. Results show considerable differences in both search performance and eye-movement behavior between high and low spatial-frequency filters, especially with search in color scenes and filters in central vision. Overall, performance and eye-movement behavior were impaired more strongly when searching grayscale scenes than color scenes (see also Nuthmann & Malcolm, 2016). Interestingly, color information was particularly important in the low-frequency band in peripheral vision. 

Peripheral filtering

Compared with searching unfiltered scenes, peripheral spatial-frequency filtering slightly decreased search accuracies with color scenes but not at all with grayscale scenes. Search times, however, increased considerably with both color and grayscale scenes. This effect mainly derives from an increase in scanning times, indicating that it took longer to locate the target object in the scene. Object verification time, once the target object was located, was hardly affected by peripheral filtering. This finding is not surprising given that central vision is critical for object identification (Henderson et al., 2003) and information in the central visual field was unaltered. The increase in scanning times was caused by a combination of shorter saccade amplitudes and more as well as longer fixations. The decrease in saccade amplitudes replicates previous findings (Cajar et al., 2016, 2016; Foulsham et al., 2011; Laubrock et al., 2013; Loschky & McConkie, 2002; Loschky et al., 2005; Nuthmann, 2013) and likely reflects a shrinkage of the attentional focus to the unfiltered central region (Cajar et al., 2016). This tunnel-vision effect also accounts for the higher number of fixations (see also Foulsham et al., 2011; Nuthmann, 2013, 2014), as the scene is scanned for the target in smaller steps. Fixation durations increased because saccade target selection took longer with higher peripheral processing difficulty. In color scenes, as expected, search performance and eye-movement behavior were affected more severely by high-pass than by low-pass filtering, reflected in longer search and scanning times, a higher number of fixations, and slightly shorter saccade amplitudes. Fixation durations were longer with low-pass filtering, suggesting that more time was invested for selecting a new peripheral saccade target when the available information was easier to process (see also Cajar et al., 2016, 2016; Laubrock et al., 2013). 

Surprisingly, the benefit for low spatial frequencies in peripheral vision disappeared completely in all search and eye-movement parameters when viewers searched grayscale scenes. This result suggests that the benefit for object search with peripheral low-pass filtering mainly arose from the availability of coarse-grained color information (which is strongly attenuated with high-pass filtering) rather than the availability of coarse-grained luminance information. This result is compatible with Hwang et al. (2007), who found that color dominates visual guidance when searching for scene patches and reduces the guidance by other stimulus dimensions such as intensity or contrast. However, a strong influence of color in peripheral vision might be specific to scene search, since in a previous study with scene memorization and peripheral object detection in grayscale scenes, we found considerable differences in task performance and eye-movement behavior between peripheral low-pass and high-pass filters (Cajar et al., 2016). 

Central filtering

With spatial-frequency filtering in central vision, effects were rather different. First, object localization was hardly affected, whereas effects on object identification were pronounced (see also Nuthmann, 2014). Second, search performance and eye-movement behavior depended strongly on filter type in both color and grayscale scenes. With central high-pass filtering, search accuracies were not affected at all in grayscale scenes and only slightly decreased in color scenes compared with the unfiltered control. Search times did increase with high-pass filtering but with the least increase of the four filter conditions. The longer search times derive from a combination of slight to moderate increases of scanning and verification times. Saccade amplitudes were not affected at all by high-pass filtering, and the number of fixations hardly increased. Higher processing difficulty with high-pass filtering was mainly counteracted by increasing fixation durations, thus taking more time to process the fixated stimulus. Overall, performance and eye-movement behavior with central high-pass filtering were nearest to normal. Although high-pass filters inherently attenuate luminance, contrast, and color in the scene, they affected object localization and verification only slightly when applied to central vision, which is an interesting new finding of the present study. 

Central low-pass filtering, on the other hand, turned out to be the most detrimental filter condition. Search accuracies decreased drastically and even dropped to chance performance when searching grayscale scenes. Search times also increased more than in all other conditions. It appears that this decrease in search performance originates from the difficulty of identifying the target object once it had been found: Scanning times increased only slightly, as with central high-pass filtering, but target verification times increased drastically compared with the control and all other filter conditions. The effect was larger when searching grayscale scenes. Eye-movement behavior was also modulated strongly by central low-pass filtering, with increased fixation durations, number of fixations, and saccade amplitudes. The increase in saccade amplitudes replicates previous findings (Cajar et al., 2016, 2016; Laubrock et al., 2013; Nuthmann, 2014), suggesting that viewers show an attentional bias toward the visual periphery with a low-pass filtered center (Cajar et al., 2016). The number of fixations increased with central low-pass filtering mainly because more fixations were made during target verification than in all other conditions, especially with grayscale scenes. Furthermore, a higher amount of return saccades during target verification was observed compared with the other conditions, as was reported by Henderson et al. (1997) with a central scotoma in an array of line drawings of objects. These findings likely reflect a strategy of saccading back and forth between the target object in the strongly blurred center and neighboring scene regions to try and confirm object identity with the help of the less blurred periphery. Almost all effects were stronger when searching grayscale scenes, where search performance dropped down to chance level. Thus, object identification without high and medium spatial frequencies was extremely challenging and nearly impossible when color was unavailable as a diagnostic object feature. 

In contrast with peripheral filtering, where effects of filter type completely disappeared with grayscale scenes, differences between central filter types were even larger with grayscale than color scenes. Thus, the specific spatial-frequency content in central vision became even more important when color was removed from the scene. This new result of the present study suggests that the proposed dominance of color over other stimulus dimensions during search (Hwang et al., 2007) might hold for the process of object localization in peripheral vision but not for object identification in central vision. 

Conclusions

The present work demonstrates that object identification in central vision mainly depends on high spatial frequencies and is nearly impossible when based solely on low frequencies. For object localization in peripheral vision, the type of available spatial-frequency information seems less important, whereas color information is critical for facilitating search. We corroborate previous findings that central vision is critical for object identification and peripheral vision is critical for object localization. Our results highlight the different roles of central and peripheral vision to object-in-scene search and provide a basis for characterizing the contributions of different spatial-frequency bands for search. 

Acknowledgments

This work was funded by Deutsche Forschungsgemeinschaft (Grants LA 2884/1 to J.L. and EN 471/10 to R.E.). The authors thank Petra Schienmann and their student assistants for their help during data collection and Lisa Buchwald for object annotations in the scenes. 

Commercial relationships: none. 

Corresponding author: Anke Cajar. 

Email: [e-mail]. 

Address: Department Psychologie, Universität Potsdam, Potsdam, Germany. 

Appendix

The following figures show the effects of target predictability (i.e., target location is either contextually predictable or unpredictable) on search performance (Figure 4), search epochs (Figure 5), and eye movements (Figure 6). 

Figure 4.

 

Search accuracies (upper row) and search times (lower row) for the five filter conditions in color scenes (left) and grayscale scenes (right), with targets at contextually predictable or unpredictable locations. Error bars represent within-subjects 95% confidence intervals with Cousineau-Morey correction (Cousineau, 2005; Morey, 2008).

Figure 5.

 

Scanning times (upper row) and verification times (lower row) for the five filter conditions in color scenes (left) and grayscale scenes (right), with targets at contextually predictable or unpredictable locations. Error bars represent within-subjects 95% confidence intervals with Cousineau-Morey correction (Cousineau, 2005; Morey, 2008).

Figure 6.

 

Fixation durations (top row), number of fixations (middle row), and saccade amplitudes (bottom row) in the five filter conditions for color scenes (left) and grayscale scenes (right), with targets at contextually predictable or unpredictable locations. Error bars represent within-subjects 95% confidence intervals with Cousineau-Morey correction (Cousineau, 2005; Morey, 2008).