Guidance toward and away from distractors in repeated visual search

Abstract

When searching for two targets consecutively in the same display, participants use memory of recently fixated distractors that become the target in the second search to find that target more quickly. Here we ask whether participants are also using memory for fixated distractors that do not become the target. In Experiment 1 we show that search is faster overall in the second search regardless of whether or not the second search target was fixated in the first search. We replicate this effect in Experiment 2 for different display sizes and further show that the effect is a result of the prioritization of locations that are more likely to contain the target. This suggests that representations of the fixated distractor items are retained across the two searches and that these representations can be used flexibly to optimize search performance. Furthermore, this suggests that the short-term memory processes that support search across consecutive searches not only facilitate guidance toward the target but also allow distractors to be excluded from the search process.

Introduction

When we search for a target object in our visual environment, we may have the feeling that we can easily keep track of which locations we have inspected and which we have not. For instance, if we search for a pencil on our desk, we may be able to avoid reinspecting locations previously tagged as “pencil-less” and guide search to noninspected locations. Moreover, we may even be able to remember some of the objects that are positioned at the previously inspected locations and use this information for upcoming searches in the same environment. That is, if we have to search the desk again for a new target object (perhaps the pencil sharpener), we may find it faster in this subsequent search if we had inspected it during the pencil search. 

In general, there is ample evidence that memory processes support visual search. Previous research has indicated that we can memorize a limited number of inspected items and locations during a single search in order to make search more efficient (e.g., Dickinson & Zelinsky, 2007; Emrich, Al-Aidroos, Pratt, & Ferber, 2010; Gilchrist & Harvey, 2000; Hollingworth, 2012; Körner & Gilchrist, 2008; Lleras, Rensink, & Enns, 2005; McCarley, Wang, Kramer, Irwin, & Peterson, 2003; Williams, Henderson, & Zacks, 2005). For instance, McCarley et al. (2003) showed that participants are less likely to reinspect items if they were presented at previously inspected locations. This held true for about three to four most recently inspected items and suggests the involvement of short-term memory (STM) during search. 

Memory supports search not only during a single search but also if the same display is presented repeatedly. In this case, it has been shown that search performance and efficiency improve with repetition (e.g., Hollingworth, 2012; Hout & Goldinger, 2010; Solman & Smilek, 2010, 2012; see, however, Kunar, Flusberg, & Wolfe, 2008). For instance, Solman and Smilek (2010) had participants search 60 times in a repeated, unrepeated, or partially repeated display and showed that search performance and search efficiency improved when the display was repeated completely compared with the other conditions. In addition, repetition benefits were found not only after several searches in the same display but even when the display was repeated only once (Höfler, Gilchrist, & Körner, 2014; Howard, Pharaon, Körner, Smith, & Gilchrist, 2011; Körner & Gilchrist, 2007). When participants searched the same display twice consecutively, performance improved in the second search compared with the first search (see Körner & Gilchrist, 2007). (In this paradigm, the second target was announced auditorily after the response to the first target, so there was no visual transient between the searches. This is different from other repeated visual search paradigms; e.g., Hout & Goldinger, 2010; Solman & Smilek, 2010). Furthermore, analysis of the eye movements indicated that participants remembered most recently inspected items from the first search better than less recently inspected items. That is, if one of the most recently inspected items of the first search became the target in the second search, participants were able to find this target faster than when it was a less recently inspected item. This effect of target recency is evidence that STM plays a part not only in single searches but also in repeated visual search. 

A framework to explain this improved performance in a subsequent search for items inspected in a previous search is provided by object file theory (Kahneman, Treisman, & Gibbs, 1992; Treisman, 1998). According to this theory, an object file that includes all relevant information about the object is generated for each attended item. This information typically consists of the perceptual features of the object (e.g., color or identity) and a spatial index (i.e., where it is located in the scene). This spatial index is automatically activated if the perceptual features of an object are retrieved. In case of repeated visual search, it can be assumed that the identity and the location of an attended item are elements of each object file because both identity and location information have been shown to be necessary for finding a previously attended item faster in a subsequent search (Höfler et al., 2014). At the beginning of a subsequent search, when a new target identity is presented, it may be matched against the identities of the object files created during the previous search. If this comparison is successful (i.e., the second target identity matches one of the identities in the object files), the corresponding information about the location of the respective item is retrieved and used to return to this item faster. This results in the performance benefit for the upcoming search. 

But what if the second target was never attended during the first search? In this case, the matching of the identity of the new target with item identities of the object files would fail. This seems to imply that the location information in the object files is worthless because no direct information about the location of the new target is available. However, the object files formed during the first search may support the second search indirectly: The mismatch ensures that object files—and their locations—are rejected as possible targets and thus become less likely to be attended during the second search. Hence, at the beginning of the second search, search should be directed toward locations that had not been attended so far because the new target will be at those locations with a greater probability. In this way, a search benefit for a second target may arise even if that target was not attended in the first search. In the General discussion, we consider alternative guidance mechanisms for nonattended targets. 

In the present study we investigated whether the object files that are established during a first visual search can be used to guide a second search in the manner described previously. Specifically, we tested whether the second search is directed back to already attended items if the second search target was attended during the first search. Moreover, we investigated whether search is directed away from items that were attended in the first search when these items do not become the target in the second search. To this end, we asked participants to search the same letter display twice successively for different target letters. We expected to find a benefit for the Search 2 target when it had been fixated previously during Search 1. In particular, we expected that a fixated Search 2 target would be found faster compared with the Search 1 target. This would replicate earlier findings (Höfler et al., 2014; Körner & Gilchrist, 2007). However, we also expected a benefit for nonfixated Search 2 targets. That is, nonfixated Search 2 targets should be found faster than the Search 1 target. We tested these two hypotheses by comparing the number of fixations for Search 2 with respective measures from Search 1 as a baseline (Experiment 1). To investigate the hypothesized benefit for noninspected Search 2 targets, we tested whether previously inspected items were avoided in the second search if the Search 2 target was absent (Experiment 2). 

Experiment 1

We hypothesized that information stored in object files after a previous search may be used not only to guide search back to already inspected items if the target is among them but also to direct a subsequent search away from these items. That is, we expected a search benefit for both a Search 2 target that was fixated during the previous search and a Search 2 target that was not fixated before. Specifically, in the second search, both fixated and nonfixated Search 2 targets should be found faster than a Search 1 target. 

Method

Design

Participants had to search the same 10-letter display twice consecutively for two different target letters (Search 1 vs. Search 2). The target letter was present (P) on half of the searches and absent (A) on the other half of the searches. This resulted in four search conditions: absent–absent (AA), absent–present (AP), present–absent (PA), and present–present (PP). In order to investigate how noninspected items are processed during Search 2, we varied the fixation status of the Search 2 target (search condition PP) experimentally. That is, in search condition PP, an item presented as a Search 2 target had been either fixated during Search 1 or not on half of the trials. Such a variation in the fixation status for Search 2 targets was not possible for search condition AP because in these trials participants typically inspected all items in Search 1. All manipulations were made within subjects. We measured the number of fixations until the Search 2 target was fixated in Search 2 for the first time. 

Participants

A total of 16 students (14 females) either were paid €20 for participation or received class credit. Their mean age was 22.9 years (SD = 2.7; range = 19–30 years). All participants had normal or corrected-to-normal vision (contact lenses). 

Apparatus

An EyeLink II eye tracking system (SR Research, Ottawa, Ontario, Canada) was used to collect the data. The EyeLink II system consists of two personal computers (PCs) and a head-mounted eye tracker on which three miniature cameras are placed. Two cameras track the position of the eyes and one tracks the position of the head to compensate for head movements. The stimuli were presented on the display PC, which was connected to the experimenter's PC via an Ethernet link. The display PC was positioned inside a soundproof, darkened booth in which the participant was seated. The experimenter's PC received eye movement data from the eye tracker and was positioned outside the booth. Eye movements were recorded with a sampling rate of 500 Hz. In each block the eye that produced the better spatial resolution (typically better than 0.31°) was recorded. Displays were presented on a 21-in. cathode ray tube monitor with a resolution of 1,152 × 864 pixels and a refresh rate of 75 Hz. Viewing distance was approximately 63 cm. A chin rest was used to minimize head movements. The velocity threshold for saccade detection was set to 35°/s, and the acceleration threshold was set to 9500°/s². Manual responses were collected using a Microsoft (Redmond, WA) game pad that was connected to the experimenter's PC. Participants had to press the left trigger for an absent response and the right trigger for a present response with their respective index finger. 

Stimuli

For each trial, 10 uppercase letters were sampled randomly from a set of 12 letters (A, E, F, G, H, K, M, O, R, S, W, and X) and presented in Arial font (bold; size = 0.32°) on a computer screen. In order to minimize the identifiability of the letter from periphery (see, e.g., Bouma, 1970), the letters were surrounded by an annulus (0.18° thick). We demonstrated earlier (Körner & Gilchrist, 2007) that letter identification did not differ reliably from chance when fixation was more than 3° away from an item. Stimuli were presented in white on a black background. The letters were placed on the intersections of an imaginary 6 × 6 grid; the size of a grid cell was 3.6°. The letter position deviated randomly from the intersection within ±0.23° in both horizontal and vertical directions. Each fixation recorded by the eye tracker was online allocated to an item using a minimal distance criterion (i.e., the actual fixated item was defined as the item that had the smallest Euclidean distance from the actual position of the gaze at that time). If two or more subsequent fixations were assigned to the same item, these fixations were collapsed online into one item fixation. For the analysis presented here, eye movement data were reanalyzed offline using the same procedure. 

Procedure

The sequence of events of a trial is illustrated in Figure 1. At the beginning of each trial a fixation disc was presented randomly at a position where a letter would appear in the search display. When the fixation was registered by the experimenter, a placeholder display was presented. The placeholder display was identical to the following search display except that each letter was replaced by the hash symbol (#). After 500 ms, the placeholder display was removed and the search display was presented in which the hash symbols changed into letters.¹ Simultaneously, a target letter was announced through loudspeakers that were positioned at each side of the monitor. Participants had to search for this target (Search 1 target) and press the right trigger of the game pad if the target was present and the left trigger when it was absent. Immediately after the response a new (second) target letter was announced through the loudspeakers, marking the start of Search 2. In search condition PP, the Search 2 target was chosen from the items that had already been fixated within the previous one to eight fixations of Search 1 on half of the trials (fixated Search 2 target). On the other half of the PP trials, the target was chosen from the nonfixated items (nonfixated Search 2 target). 

Figure 1.

 

Experiment 1. Sequence of events in a trial. Participants had to search the same letter display twice consecutively for two different targets. The arrows in the display indicate a potential scan path in Search 1. In Search 1 present trials, the fixation status of the Search 2 target (“H”) was manipulated so that it was either fixated during Search 1 (fixated Search 2 target) or not (nonfixated Search 2 target). In the example, “H” would be a nonfixated Search 2 target.

The position of both the Search 1 target and the Search 2 target was controlled for proximity. This is, the Search 1 target was presented at a position with the smallest deviation from a Euclidean distance of 10.8° relative to the first fixation (i.e., the position of the fixation disc). In Search 2, the item with the smallest deviation from a distance of 10.8° relative to the last fixation in Search 1 (which was the starting point of Search 2) became the target. If no item met this criterion, the target was chosen randomly from the whole set of items and the respective trial was excluded from analysis. In order to minimize the possibility that participants might implicitly learn that the target was presented a certain distance from the current fixation, in Search 1 the target was presented at the position of the fixation disc in six trials of each block. Likewise, the Search 2 target was presented on the last fixated item of Search 1 (i.e., the starting point of Search 2) in trials in which no suitable Search 2 target could be selected. As in Search 1, these trials were excluded from analysis. 

Each participant completed eight blocks of 70 trials each. The blocks were spread across two sessions of four blocks each. Participants completed four practice trials before the first block. There was a break of about five minutes between blocks. Participants were told to respond as quickly and accurately as possible. The total duration of the experiment (Session 1 and Session 2) was about three hours. 

Results and discussion

Error rates and manual response times

Data from 8,958 trials were collected (16 participants × 560 trials; two trials were lost due to technical problems). Incorrect trials (4.3% on average; individual participant range = 0.54%–9.8%) were excluded from further analysis. Additionally, trials in which the target was placed at the position of the fixation when the search started or when the target was chosen randomly were excluded. In all, analysis of response times and number of fixations included 7,392 trials. 

We averaged the mean response times for target presence and search across individual means. On average, target-absent search lasted 3730 ms (SD = 558) in Search 1 and 3680 ms (SD = 734) in Search 2. Target-present search lasted 2327 ms (SD = 236) in Search 1 and 1943 ms (SD = 241) in Search 2. A 2 × 2 repeated measures analysis of variance (ANOVA) with target presence and search as factors showed a main effect for target presence, F(1, 15) = 168.88, p < 0.001, η_p² = 0.92, and search, F(1, 15) = 22.43, p < 0.001, η_p² = 0.60. Also the interaction was significant, F(1, 15) = 15.44, p < 0.001, η_p² = 0.51. Thus, target-absent searches were faster than target-present searches. This is a standard result in serial visual search. More importantly, response times in Search 2 were faster than those in Search 1. This effect was modified by the interaction. Post hoc comparison (t tests, Bonferroni-Holm corrected) showed that Search 2 was faster in target-present searches, t(15) = 13.1, p < 0.001, η² = 0.92, but not in target-absent searches, t < 1. 

Number of fixations

Averaged across individual means, participants fixated 6.9 items (SD = 0.3) in Search 1 target-present searches and 5.3 items (SD = 0.4) in Search 2 target-present searches. If the target was absent, participants fixated 11.6 items in Search 1 (SD = 1.0) and 11.2 items in Search 2 (SD = 1.6). A 2 × 2 repeated measures ANOVA with target presence and search as factors showed a main effect of target presence, F(1, 15) = 384.16, p < 0.001, η_p² = 0.96, and search, F(1, 15) = 78.90, p < 0.001, η_p² = 0.84. Also, the interaction was significant, F(1, 15) = 34.52, p < 0.001, η_p² = 0.70. Post hoc comparison (t tests, Bonferroni-Holm corrected) showed that for target-present searches, Search 2 searches were faster than Search 1 searches, t(15) = 18.15, p < 0.001, η² = 0.96. For target-absent searches, this difference was only marginally significant, t(15) = 2.09, p = 0.054, η² = 0.23. Thus, Search 2 target-present searches required fewer fixations compared with Search 1 target-present searches, whereas there was hardly any difference between Search 1 and Search 2 in target-absent searches. Together, these results nicely reflect the findings of the manual response times. Because of this concordance, we now focus on the eye movement data investigating the effect of target fixation status (old vs. new). Analysis of manual response times for the target fixation status revealed the same pattern of results and can be found in Table A1 of the Appendix. 

Search 2 target fixation status

For the analysis of the fixation status of the Search 2 target, we analyzed trials with a target present in both searches (condition PP). There were 3,475 such trials. A preliminary analysis showed that Search 1 was longer for a fixated Search 2 target than for a nonfixated Search 2 target. We therefore selected for analysis trials in which more than one but fewer than 10 items had been fixated during Search 1 (2,969 trials). This ensured that Search 1 duration was comparable for fixated and nonfixated Search 2 targets. 

We then analyzed the number of fixations until the Search 2 target (fixated or nonfixated) was inspected in Search 2 for the first time (search condition PP; see Figure 2) as well as the number of fixations until the first inspection of the Search 1 target as a baseline (M = 5.3, SD = 0.1). A one-way ANOVA showed a significant difference between these conditions, F(1.28, 19.15) = 20.18, p < 0.001, η_p² = 0.57. In particular, a fixated Search 2 target was found after 4.2 fixations (SD = 0.4) during Search 2; a nonfixated Search 2 target was found after 4.4 fixations (SD = 0.7). A t test showed that there was a reliable difference between baseline and fixated Search 2 targets, t(15) = 10.40, p < 0.001, η² = 0.88, and baseline and nonfixated Search 2 targets, t(15) = 4.74, p < 0.001, η² = 0.60, but no difference between fixated and nonfixated Search 2 targets, t < 1 (Bonferroni-Holm corrected). Thus, both previously fixated Search 2 targets and previously nonfixated Search 2 targets were found faster than baseline. This confirms our hypotheses that there is a benefit in the second search not only when the target was inspected during the first search but also when the target was previously not inspected. 

Figure 2.

 

Experiment 1. Number of fixations until first target fixation in Search 1 and Search 2 (fixated and nonfixated targets).

With regard to fixated Search 2 targets, the findings of Experiment 1 replicated the results of Körner and Gilchrist (2007). There was a search benefit in Search 2 compared with Search 1: The target was found faster in Search 2 than in Search 1. With regard to target fixation status, results showed that participants were able to find a Search 2 target in a subsequent search faster not only when they had inspected it before but also when they had not inspected it previously. In the Introduction, we argue that this benefit results from a match with an object file or a mismatch, respectively. If the Search 2 target matches the identity of an object file, this match activates the corresponding location information and directs search back to this item, resulting in a benefit for fixated Search 2 targets. However, for noninspected Search 2 targets, the matching process against the object files always arrives at a negative result, which leads to a rejection of the identities (and, consequently, the corresponding locations) of these items as possible targets. Hence, the second search is (at least at the beginning) directed away from these items toward thus-far nonattended items. 

We have demonstrated a benefit for Search 2 targets regardless of whether or not they were attended during Search 1. In Experiment 2, we not only replicated these findings in a broader context but also examined situations in which no Search 2 target at all was present in the subsequent search. 

Experiment 2

The results of Experiment 1 revealed a search benefit not only for Search 2 targets that were attended during Search 1 but also for Search 2 targets that were not attended. This suggests that a match of information with the object files generated during Search 1 can be used to guide search back to these items, whereas a mismatch results in a rejection of such items and, as a consequence, in prioritization of any unattended item. If this is true, we expected that, in general, hitherto noninspected items are attended earlier in the second search. More precisely, if the Search 2 target is absent, distractors that were not fixated during the first search should be fixated earlier in the second search than distractors that were fixated before. Although Experiment 1 would allow us to test this assumption by analyzing refixation rates in such trials, we wanted to test this possible rejection of attended items in absent searches more directly. In particular, for these trials, after the end of Search 1 we selected one distractor for Search 2 (probe distractor) that had either been fixated during Search 1 (fixated Search 2 probe distractor) or not (nonfixated Search 2 probe distractor) and compared the time it took participants to fixate this probe distractor depending on its fixation status in Search 1. We expected that nonfixated Search 2 probe distractors should be fixated earlier than fixated Search 2 probe distractors. As in Experiment 1, we also analyzed whether there is a benefit for both a previously (during Search 1) fixated and a nonfixated Search 2 target with regard to Search 1. 

A further improvement in Experiment 2 is that we controlled the number of items that were inspected in the first search with a saccade-contingent procedure. In a design such as that used in Experiment 1 in which the same display has to be searched twice, the probability of refixating an inspected or noninspected item in the second search depends on the number of items inspected during the first search (i.e., the longer the first search, the fewer noninspected items left for the second search, and vice versa). In Experiment 2, we therefore made sure that participants found the Search 2 target (if it was present) whenever they had searched the display halfway through. Finally, we also varied the set size in Experiment 2 in order to investigate the effect of guidance for smaller and larger displays. 

Method

Design

Participants had to search the same letter display consisting of 5, 11, or 17 items twice consecutively for two different target letters (Search 1 vs. Search 2). The target letter was present (P) on half of the searches and absent (A) on the other half of the searches. This resulted in four search conditions: absent–absent (AA), absent–present (AP), present–absent (PA), and present–present (PP). As in Experiment 1, we varied the fixation status of the Search 2 target (fixated Search 2 target vs. nonfixated Search 2 target; search condition PP). In addition, we varied the fixation status of one of the distractors if the Search 2 target was absent (search condition PA). In particular, we varied the fixation status of one specific distractor—the probe distractor—such that this probe distractor was an item that had either been fixated during Search 1 (fixated Search 2 probe distractor) or not (nonfixated Search 2 probe distractor). Such a variation in the fixation status for probe distractors was possible only for search condition PA because in these trials participants typically did not have to inspect all items in Search 1, which was a prerequisite for the selection of a noninspected target or probe distractor for Search 2. 

We used a saccade-contingent technique (McConkie & Rayner, 1975) to ensure that the target of the first search was found after the display had been searched halfway through. This ensured that the items were divided into two sets of equal size regarding fixation status (i.e., fixated vs. not fixated) after Search 1 and thus controlled the probability of refixating an inspected or noninspected item in Search 2. In order to prevent participants from noticing this regularity, we intermixed the different set sizes (5, 11, or 17 items) within a block. 

All manipulations were made within subjects. We measured the number of fixations until the Search 2 target (search condition PP) or probe distractor (search condition PA) was fixated in Search 2 for the first time. 

Participants

A total of 16 participants (13 females) either were paid €35 for participation or received class credit. Their average age was 25.6 years (SD = 5.0; range = 20–38 years). All participants had normal or corrected-to-normal vision (contact lenses). 

Apparatus and stimuli

The apparatus and the stimuli were the same as in Experiment 1 with the following exceptions. For each trial, 5, 11, or 17 uppercase letters were sampled randomly from a set of 19 letters (A, E, F, G, H, I, K, L, M, O, P, R, S, T, U, V, W, X, and Z). 

In Search 1 present searches (search conditions PP and PA), the Search 1 target was not present in the display at the beginning of the first search (i.e., there were only distractors in the display). At a specific time in the search the target letter replaced one of the distractor letters. The replacement occurred during the saccade that followed the fixation of two (display size 5), five (display size 11), or eight (display size 17) different letters as soon as the saccade started from the item fixated last and crossed an invisible boundary to another item (using a minimal distance criterion). The target letter then replaced the distractor letter at the position to which the saccade was directed. (The target never appeared at an item position that was inspected earlier during search.) This procedure ensured that participants fixated the target letter after three (display size 5), six (display size 11), or nine (display size 17) fixated distractors. Because of saccadic suppression (Dodge, 1900; Ross, Morrone, Goldberg, & Burr, 2001), the participants were usually not aware of this change in the display. To control for such awareness, we conducted a postexperimental interview (see below). 

Due to the online presentation of the Search 1 target, there is no natural baseline with which to compare Search 2. As an alternative, we computed a potential target (Search 1 probe distractor) in Search 1 target-absent searches (search conditions AP and AA). For this, we used the same algorithm as that used for the target in Experiment 1 (i.e., the Search 1 probe distractor was the item closest to the desired distance of 10.8° relative to the first fixation in the display) and analyzed how long it took to fixate this Search 1 probe distractor as the baseline. 

Procedure

The procedure was the same as in Experiment 1 with the following exceptions. If Search 1 was a target-present search (search conditions PA and PP), the target was computed in the way described previously and presented on the display when the participant had searched it halfway through. If Search 1 was a target-absent search (search conditions AA and AP), a distractor item was chosen as the Search 1 probe distractor. In both cases, participants had to search for the Search 1 target and press the appropriate button to indicate its presence or absence, after which the second search started. If Search 2 was a target-absent search (search condition PA), a distractor item was chosen as the Search 2 probe distractor. Similar to regular Search 2 targets, it had been either fixated during Search 1 (fixated probe distractor) or not (nonfixated probe distractor) on half of the trials, respectively. Participants were not informed about the existence of the Search 2 probe distractor and instead searched for the announced target. After the second manual response, the display was cleared and a new trial started. The positions of the target and the probe distractor were controlled for distance (see Experiment 1). 

Each subject participated in 12 blocks of 78 trials (i.e., 936 trials per subject). The blocks were typically spread across two sessions of four blocks each. Participants performed four practice trials before starting block 1. There was a break of about five minutes between blocks. Participants were told to respond as quickly and accurately as possible. Display size, target fixation status, and search condition were varied within each block. 

Postexperimental interview and target fixation duration

After the experiment, the participants were asked whether they had noticed any display changes and, if so, whether they were aware of when exactly the change happened. The interview started with general questions (e.g., “Did you notice anything unusual during the experiment?”) that became more suggestive (“Did anything change in the display?”; “Did you notice that the target letter appeared in some of the trials?”). Four of the 16 participants did not notice any display changes. Of the 12 remaining participants, two reported some “abnormalities” in the display during the experiment itself but were not sure if changes were occurring or what was actually changing. (When participants mentioned an “abnormality” during the breaks in the experiment, they were told to ignore this as a technical problem.) All other participants mentioned that they noticed “something” only when they were told after the end of the experiment that there were actual changes during search. In this case, we also asked them to estimate how often such changes occurred. Participants reported that changes occurred rarely or once in a while, about two times per block, or three times at most. (Note that display changes happened more than 400 times per participant.) 

If saccade-contingent presentation impaired target processing, one would expect longer target fixation durations. A comparison of target fixation durations in Search 1 showed that participants did not fixate the target longer in Experiment 2 than in Experiment 1. Fixation durations were actually somewhat shorter in Experiment 2 (Experiment 1: M = 406 ms, SD = 74; Experiment 2: M = 343 ms, SD = 51). We can therefore assume that the participants were not aware of the extent and regularity of our manipulation and that the results reported here are not affected by it. 

Results

Data from 14,967 trials were collected (nine trials were lost due to technical problems). Only trials in which the target was present during Search 1 (i.e., search conditions PA and PP) were analyzed. This was the case for 7,290 trials. From these trials, all incorrect trials were excluded (8.7% on average; individual range = 1.1%–16.0%). All trials in which the Search 1 target was placed at the position of the fixation disc or the Search 2 target was placed at the position of the last fixated item were also excluded (7.5%). In all, 6,165 trials (84.6%) were included in the analysis. 

Search 1 target and probe distractor fixation

Averaged across individual means, participants found the Search 1 target after 3.1 fixations (SD = 0.1) for display size 5, 6.2 fixations (SD = 0.1) for display size 11, and 9.3 fixations (SD = 0.2) for display size 17. (The respective manual response times can be found in Table A1 of the Appendix.) These numbers showed that the duration of Search 1 was controlled successfully. The small variations (as reflected in the standard deviations) occurred because, in some cases, the display change was triggered too early due to technical reasons or because an item that had already been fixated was refixated at the time when the target should have appeared. In this case, the display change was postponed until a suitable item was fixated. Because there was no natural baseline with which to compare Search 2, we used a probe distractor in Search 1 target-absent searches (search conditions AP and AA) to provide the baseline (see Apparatus and stimuli). This Search 1 probe distractor was fixated after 3.3 fixations (SD = 0.2) for display size 5, 6.5 fixations (SD = 0.4) for display size 11, and 9.3 fixations (SD = 0.5) for display size 17. These numbers were virtually identical to the number of fixations needed to find the Search 1 target (see above) and demonstrated the intended purpose of the probe distractor. 

Search 2 target fixation status

Figure 3 shows the number of fixations until the first Search 2 target fixation depending on fixation status and display size and the number of fixations until the first fixation of the Search 1 probe distractor (baseline) for comparison (see Table A1 for the analysis of manual response times). The number of fixations clearly increased as a function of display size. More importantly, there was only a small difference (averaged across display size) between fixated Search 2 targets (M = 5.1, SD = 0.9) and nonfixated Search 2 targets (M = 4.5, SD = 0.9), whereas it took more fixations to fixate the Search 1 probe distractor (M = 6.4, SD = 0.3). Accordingly, a 3 × 3 repeated measures ANOVA with display size and target fixation status (fixated and nonfixated Search 2 targets, including Search 1 probe distractor as baseline) as factors showed a main effect of display size, F(2, 30) = 929.27, p < 0.001, η_p² = 0.98, and a main effect of target fixation status, F(1.25, 18.82) = 22.86, p < 0.001, η_p² = 0.60 (Greenhouse-Geisser corrected), but no interaction, F < 1. Post hoc pairwise comparisons (Bonferroni-Holm corrected) showed that, besides an effect of display size, it took fewer fixations to find both the previously fixated and the nonfixated Search 2 targets than the Search 1 probe distractor. However, there was no difference between fixated and nonfixated Search 2 targets. Participants found the target equally fast regardless of whether or not they had fixated it before. This replicates the respective finding from Experiment 1 and extends it to different display sizes. 

Figure 3.

 

Experiment 2. Number of fixations until first target fixation in Search 2 for fixated Search 2 targets (solid line) and nonfixated Search 2 targets (dashed line) depending on display size (search condition PP). The dotted line represents the baseline performance in Search 1 based on the analysis of the Search 1 probe distractors.

Search 2 probe distractor fixation status

For search condition PA, we analyzed the number of fixations until the Search 2 probe distractor was fixated first (see Figure 4). Again, the number of fixations increased as a function of display size. There was a difference (averaged across display size) between fixated Search 2 probe distractors (M = 5.9, SD = 0.6) and nonfixated Search 2 probe distractors (M = 4.7, SD = 0.9), whereas it took more fixations to fixate the Search 1 probe distractor (M = 6.4, SD = 0.3). A 3 × 3 repeated measures ANOVA with display size and Search 2 probe distractor fixation status (fixated vs. nonfixated Search 2 probe distractors vs. Search 1 probe distractor as a baseline) as factors showed a main effect of display size, F(2, 30) = 1199.87, p < 0.001, η_p² = 0.98, and fixation status, F(1.2, 18.05) = 21.38, p < 0.001, η_p² = 0.59. Also, the interaction was significant, F(2.3, 34.5) = 7.69, p < 0.001, η_p² = 0.34. Post hoc comparison (one-way ANOVAs and follow-up post hoc pairwise comparisons, Bonferroni-Holm corrected) showed differences between fixated Search 2 probe distractors and Search 1 probe distractor (baseline) for display sizes 5 and 11 but not for display size 17. Hence, the benefit in Search 2 for already fixated items disappeared for larger displays. However, nonfixated Search 2 probe distractors were fixated earlier than the Search 1 probe distractor for all display sizes (all ps < 0.001). Most importantly, for all display sizes nonfixated Search 2 probe distractors were fixated earlier than fixated Search 2 probe distractors (ps < 0.05). 

Figure 4.

 

Experiment 2. Number of fixations until first Search 2 probe distractor fixation for fixated probe distractors (solid line) and nonfixated probe distractors (dashed line) depending on display size (search condition PA). The dotted line represents the baseline performance in Search 1 based on the analysis of the Search 1 probe distractor.

The results of this experiment showed that, for all display sizes, Search 2 targets that had not been fixated during Search 1 were found as fast as Search 2 targets that had been fixated during Search 1. With respect to distractors, nonfixated Search 2 probe distractors were fixated earlier than fixated Search 2 probe distractors across all display sizes. 

General discussion

The aim of this study was to investigate how memory for previously inspected distractors can be used to guide a subsequent visual search in the same display. In particular, we assumed that search may be guided not only back to already attended distractors if the target is among them (as already demonstrated by Körner & Gilchrist, 2007) but also away from these items if the target is not among them. To this end, we had participants search the same display twice consecutively for two different targets. The results showed that participants found a Search 2 target faster than a Search 1 target regardless of whether or not they had inspected the Search 2 target during Search 1. This search benefit was approximately of the same magnitude. In addition, participants preferred searching through noninspected items at the beginning of the second search if the Search 2 target was not among the previously inspected items. Previously noninspected distractors were fixated earlier in Search 2 compared with previously inspected distractors. 

Körner and Gilchrist (2007; see also Höfler et al., 2014) showed that a Search 2 target was found faster than the Search 1 target when the same display was searched twice. This benefit is due to STM processes: Recently inspected items of Search 1 are remembered across searches, and if one of these items becomes the Search 2 target, search is guided back to those items. This target recency effect was robust against short delays between the consecutive searches, and, most importantly, it relied on both the identity and location memory for recently inspected items (Höfler et al., 2014). Participants have to remember which items they inspected during Search 1 and where these items were located in order to find a recently fixated Search 2 target faster. 

In the present experiments, we wanted to take a step forward and tested whether and how information about previously fixated items can flexibly be used to enhance search performance in repeated search. That is, we investigated whether the very same information can also be used to guide search away from previously inspected items that do not become the second target. Indeed, the current experiments revealed that there was a benefit in Search 2 not only for targets that had been inspected during Search 1 (replicating previous findings from Körner & Gilchrist, 2007) but also (even to the same extent) for items that had not been inspected previously. 

But how can these findings be explained from a theoretical point of view? We have argued that object file theory (Kahneman et al., 1992; Treisman, 1998) provides one theoretical framework to explain the benefits for both inspected and noninspected Search 2 targets. If the Search 2 target matches the identity of an object file, this match activates the corresponding location information and directs search back to this item, resulting in a benefit for attended targets. This explains not only the current findings for fixated Search 2 targets but also previous findings with regard to the target recency effect (Körner & Gilchrist, 2007). In contrast, if the matching of the identity of a nonattended Search 2 target with the object files fails (which is the case whenever the Search 2 target is not in the display or has not been inspected previously), the corresponding location information in the object file may be rejected too. As a consequence, this location is marked as a nontarget and is less likely to be immediately reattended during the subsequent search. In turn, nonattended items will be prioritized. Such prioritization results in previously nonattended items being inspected earlier in the second search than items that were previously inspected. 

The fact that search is guided to noninspected items by discouraging the reinspection of previously inspected items has been reported in other paradigms. For instance, in the preview paradigm (Watson & Humphreys, 1997), participants are allowed to view a part of the search display before they start the search. The search display itself includes old items (presented in the preview) and new items (not presented in the preview). Results typically show that a target (which is never presented during the preview) is found faster in a display with preview compared with a display without a preview. Hence, memory for the items in the preview allows participants to restrict search to those items they have not seen before. In a visual search paradigm with real-world objects, Yang, Chen, and Zelinsky (2009) manipulated the novelty of a distractor such that a novel distractor was presented among old distractors in each trial. Results showed that participants fixated this novel distractor preferentially compared with familiar distractors. This implies that search was guided away from the familiar objects toward novel ones. 

As described previously, object file theory provides one explanation for the guidance both toward and away from items within one coherent framework. In the context of research on STM, such a flexible usage of memory was also proposed by Woodman and Luck (2007). They demonstrated that the contents stored in STM can be used in a task-dependent manner as either a template for selection or a template for rejection of an item (see also Arita, Carlisle, & Woodman, 2012). Woodman and Luck had participants store an item in STM in preparation for an upcoming memory test while they searched for a target in the display. In the search display, one of the items shared the same color as the item held in STM. The results showed that if this matching item was never the search target it was avoided during search. In contrast, if it was occasionally the target, attention was actively guided to this item, making the search more efficient. These findings indicate that memory content can be used quite flexibly in order to perform a concurrent task. 

However, it is also possible that two separate mechanisms (rather than one flexible mechanism) are responsible for the guidance back and away from previously fixated items. Search may be guided back to previous items because of STM processes, whereas the guidance away might be due to a different process. For instance, Sawaki and Luck (2011) showed that distractors that match the content of working memory were actively suppressed if they were not relevant for the current task. They had participants store one item in working memory while task-irrelevant probes were presented. One of these probes shared the same color with the memorized objects. The electroencephalogram showed a so-called distractor positivity component in the event-related potential for this memory matching probe, indicating attentional suppression of it. 

Another mechanism that discourages the immediate reinspection of recently visited items and that could therefore account for our findings on nonfixated targets is inhibition of return (IOR; see e.g., Klein, 2000). IOR has been demonstrated in visual search (e.g., Klein, 1988; Klein & MacInnes, 1999; MacInnes, Hunt, Hilchey, & Klein, 2014; Thomas & Lleras, 2009; although see Hooge, Over, van Wezel, & Frens, 2005) and is said to be a “foraging facilitator” (Klein & MacInnes, 1999). Regarding the current experiments, recently inspected items of Search 1 would be inhibited and, hence, search would be guided to noninspected items at the beginning of Search 2. However, we consider it unlikely in the current paradigm that IOR promotes the rejection of recently inspected items if a new Search 2 target is present. Using a probe paradigm, Höfler, Gilchrist, and Körner (2011) tested whether, and how, IOR operates during or across two consecutive searches of the same display. Their results showed that while IOR was acting within each of the two searches, at the end of the first search the inhibition of recently inspected items was reset. They argued that such a reset of IOR is one factor in why participants are able to find previously inspected items faster in a subsequent search. (Note, however, that a probe paradigm can disrupt natural search behavior; see MacInnes et al., 2014.) However, if IOR is working for previously inspected items and search is hence guided away from these items, this does not explain why a target is found faster when it is among those items. For this case, a separate mechanism would be needed to overrule the inhibition and guide search back to previously inspected items. We therefore think that object file theory is a more economical and coherent way of explaining how Search 2 is guided. It explains both the guidance toward and the guidance away from previously inspected items within the same framework. 

In the current study we investigated how the information gained during a previous search changed search behavior in a subsequent search. What we did not test, however, was how much information could be maintained across searches. Previous research has demonstrated that memory in visual search is capacity limited (e.g., McCarley et al., 2003). In previous articles (Höfler et al., 2014; Körner & Gilchrist, 2007), we also demonstrated that there is memory for only about the four most recently inspected items across two consecutive searches. Therefore, the guidance reported in the present experiments may be limited by a capacity of four items held in STM. Given that we did not distinguish how well (if at all) previously inspected items are represented in STM, the present effect of guidance toward targets may be an underestimation. Suppose we considered only those cases in which the Search 2 target was inspected within the four most recent fixations and was therefore well represented in STM. Then the search benefit should be more pronounced than reported here. That is, regardless of display size, the number of relevant items in Search 2 could be restricted to the four items present in STM. In contrast, for nonfixated Search 2 targets, the effect of guidance away may be slightly overestimated: If we assume a memory for four items, this would reduce the display size in Search 2 only by these four items. For example, a display size of 10 in Search 1 would reduce to a relevant display size of six in Search 2 because the target of Search 2 is not among the four items in STM. 

We have repeatedly shown that, due to memory processes, search can be guided back to items recently inspected in a previous search if one of these items becomes the target in a subsequent search of the same display (Howard et al., 2011; Körner & Gilchrist, 2007) and have investigated the boundary conditions of this effect (Höfler et al., 2014). Here we extended this effect and demonstrated that memory for previously inspected items can be used even to guide search away from inspected items if the target is not among them. This shows that information stored in memory about inspected items is used flexibly to facilitate a subsequent search. Finally, using object file theory as a theoretical framework allowed us to explain how such guidance toward and away from distractors improves search efficiency. 

Acknowledgments

This work was supported by the Faculty of Natural Sciences of the University of Graz (M. H.) and the Austrian Science Fund (FWF; P19707 G-17; C. K.) and is printed with the support of the University of Graz. We are grateful to Susanna Haberschek and Christian Poglitsch for collecting the data. 

Commercial relationships: none. 

Corresponding author: Margit Höfler. 

Email: [email protected]. 

Address: Department of Psychology, University of Graz, Graz, Austria. 

Footnotes

¹  In a control experiment with six participants, we found that the duration of the placeholder display did not affect search performance in a subsequent search. Participants searched for a target letter in a display comparable with the display of Experiment 1. Before the search, a placeholder display was presented for either 500 ms (short duration) or a variable duration between 2000 and 3250 ms (long duration; average duration = 2,611 ms). Each participant completed 210 trials in blocks of either short or long placeholder durations. Neither search response times (short duration: M = 2755 ms, SD = 138; long duration: M = 2838 ms, SD = 141) nor the number of fixations (short duration: M = 9.6, SD = 0.4; long duration: M = 9.8, SD = 0.8) during search were affected by the duration of the placeholder display.

Appendix

Experiment 1: Target fixation status

To investigate the effect of target fixation status (fixated or nonfixated) with respect to manual response times, we analyzed the individual mean response times in Search 2 for search condition PP and compared them with the response times for Search 1 (present searches) as a baseline (see Table A1). A one-way ANOVA showed a significant difference between these conditions, F(1.39, 20.89) = 16.31, p < 0.001, η_p² = 0.52. Post hoc t tests (Bonferroni-Holm corrected) showed that there was a reliable difference between baseline and fixated targets, t(15) = 7.85, p < 0.001, η² = 0.80, and baseline and nonfixated targets, t(15) = 3.34, p < 0.01, η² = 0.43. There was a tendency for fixated targets to be faster than nonfixated targets, but this difference was statistically not significant, t(15) = 8.08, p = 0.06. 

Table A1.

 

Mean (SD) manual response times (ms) until first target fixation in Search 1 and Search 2 (fixated and nonfixated targets) for Experiments 1 and 2. Search 1 response times for Experiment 2 are based on target present searches.

 

Experiment 2: Target fixation status

For each display size, we analyzed the individual mean response times until the Search 2 (previously fixated or nonfixated) target was fixated in Search 2 for the first time and compared this with the response times for Search 1 (present searches; see Table A1). A 3 (display size) × 3 (target fixation status) ANOVA showed a significant main effect of display size, F(1.33, 19.87) = 460.18, p < 0.001, η_p² = 0.97, and a main effect of target fixation status, F(1.38, 20.05) = 13.99, p < 0.01, η_p² = 0.47 (Greenhouse-Geisser corrected). The interaction was not significant, F < 1. Post hoc comparisons (Bonferroni-Holm corrected) of the target fixation status showed a reliable response time difference between Search 1 and fixated targets, p < 0.01, and between Search 1 and nonfixated targets, p < 0.001. The difference between fixated and nonfixated targets was not significant, p = 0.24.