A working memory account of refixations in visual search

Abstract

We tested the hypothesis that active exploration of the visual environment is mediated not only by visual attention but also by visual working memory (VWM) by examining performance in both a visual search and a change detection task. Subjects rarely fixated previously examined distracters during visual search, suggesting that they successfully retained those items. Change detection accuracy decreased with increasing set size, suggesting that subjects had a limited VWM capacity. Crucially, performance in the change detection task predicted visual search efficiency: Higher VWM capacity was associated with faster and more accurate responses as well as lower probabilities of refixation. We found no temporal delay for return saccades, suggesting that active vision is primarily mediated by VWM rather than by a separate attentional disengagement mechanism commonly associated with the inhibition-of-return (IOR) effect. Taken together with evidence that visual attention, VWM, and the oculomotor system involve overlapping neural networks, these data suggest that there exists a general capacity for cognitive processing.

Introduction

Visual exploration of the environment involves an active and alternating sequence of saccadic eye movements and gaze fixations for detailed processing of visual information. The visual search paradigm has been a valuable approach to studying how visual attention is allocated when multiple competing stimuli are present (Eckstein, 2011). Oculomotor search, in particular, has been crucial in contributing to our understanding of the dynamic attentional processes involved in selecting sequences of saccade targets (Findlay & Gilchrist, 2003). While most studies of visual search have focused on how saccade target selection proceeds based on each object's attentional priority, only a few have considered the additional retrospective observation that previously examined objects are seldom refixated (e.g., Gilchrist & Harvey, 2000; Peterson, Kramer, Wang, Irwin, & McCarley, 2001). This effect may reflect a crucial process in visual search that prevents the eyes from returning to previously examined locations, thereby improving visual search efficiency. Efficiency measures typically associated with prospective attentional allocation, such as response time, number of fixations, and fixation durations, may actually also indirectly reflect a retrospective search retention process. 

Search retention is often described as being imperfect and objects are occasionally refixated. The rate of refixation is significantly lower than that expected from a retentionless system (Gilchrist & Harvey, 2000; Peterson et al., 2001) and the probability of refixation increases with intervening fixations (McCarley, Wang, Kramer, Irwin, & Peterson, 2003). Together, these findings suggest that search retention has a limited capacity. This retention process is often thought to be mediated by an attentional disengagement mechanism based on studies of the inhibition-of-return (IOR) effect (Klein & MacInnes, 1999). This mechanism is thought to temporarily inhibit attention from returning to locations of previously attended items. Concurrently, other studies have suggested that visual working memory (VWM)—the process by which visual information is actively maintained over a short duration and used to guide future behavior—is tightly coupled with saccade target selection. VWM is more precise for saccade targets than nontargets (Bays & Husain, 2008; Shao et al., 2010) and its contents can be used to guide saccades (Hollingworth & Luck, 2009). The ability to inhibit saccades to sudden-onset distracters is poor when memory is concurrently loaded (Van der Stigchel, 2010), suggesting that working memory (WM) prevents saccades to irrelevant stimuli. The relationship between active vision and memory also extends to long-term memory representations (Ryan, Hannula, & Cohen, 2007; for review, see Hannula et al., 2010). Memory recall improves with increasing frequency of fixation (Loftus, 1972). Subjects can recall details of previously fixated objects even when not explicitly instructed to do so (Castelhano & Henderson, 2005). Moreover, amnesic patients with hippocampal damage display different visual behavior during scene viewing as compared to healthy controls (Ryan, Althoff, Whitlow, & Cohen, 2000). Finally, initial abstract scene information (i.e., gist) can be retrieved and used to guide future eye movements (Castelhano & Henderson, 2007). 

Given the evidence for the obligatory use of memory representations to guide visual behavior, it is difficult to reconcile how a separate nonmemory-related attentional disengagement mechanism may be involved in the retention process. Teasing apart IOR-related and VWM processes is hampered by the fact that both would be similarly manifested in visual behavior: Objects are not refixated once they have been looked at. In fact, IOR- and memory-related mechanisms in active vision are often discussed interchangeably (e.g., Bays & Husain, 2012; Gilchrist & Harvey, 2000; Shore & Klein, 2000). It is crucial, then, to provide a unifying account of search retention. For the purposes of this paper, we define search retention as the process by which previously fixated items are not refixated. We additionally define the IOR-related mechanism as a passive low-level process that temporarily inhibits previous saccade targets as a consequence of attentional disengagement (Klein, 2000). Conversely, we consider VWM to be a cognitive process that requires resources to actively maintain encoded information (Baddeley, 2003). We tested the hypothesis that the retention process in active vision involves memory-specific mechanisms by directly relating VWM performance in a change detection task to visual search efficiency in an overt search task in the same subjects. If memory processes are involved in active vision, then VWM capacity (Alvarez & Cavanagh, 2004) should limit active visual search efficiency, including search retention. To the best of our knowledge, this is the first study to assess the relationship between VWM and refixations in visual search. We report how refixation rate, and visual search efficiency in general, was well predicted by VWM capacity. We discuss how the relationship between retention in active vision and VWM is suggestive of an account in which previous interpretations for IOR-related mechanisms in active vision may, instead, be consequences of memory processes that operate over short delays. 

Methods

Forty-one adults (28 female) aged 18–35 years (M = 23.6 years, SD = 4.68) were recruited from the Rotman Research Institute participant pool. All were right-handed, had normal or corrected-to-normal vision, and reported no history of neurological or psychiatric disease. All participants provided informed written consent prior to the study and received monetary compensation. Approval was obtained from the Baycrest Research Ethics Board and procedures complied with the Declaration of Helsinki. 

Participants performed a visual search task and a change detection task in the same experimental session, with task order counterbalanced across participants. Eye position was monitored with an EyeLink II infrared camera system (SR Research Ltd., Oakville, ON, Canada) at a sampling rate of 500 Hz during both tasks. A nine-point calibration procedure was performed at the beginning of every block of trials, and was accepted if the error at any calibration point was ≤1° and if the mean error for all nine calibration points was ≤0.5°. Stimuli were presented on a 19-in. CRT computer monitor (Dell M991, 1024 × 768 resolution, 75 Hz; Dell, Inc., Round Rock, TX) at a distance of 25 in. using custom programs written in Experiment Builder (SR Research Ltd.). Luminance and standard color values were measured using a Minolta CA-100Plus photometer (Minolta Co., Ltd., Osaka, Japan). All stimuli were matched for luminance (∼15 cd/m²) and presented on a black background (<0.01 cd/m²). 

Visual search task

In the visual search task (Figure 1A), each trial began with a central white fixation cross measuring 1° × 1°. Participants were required to fixate this cross for 200–500 ms. The fixation cross then disappeared with the simultaneous presentation of the visual search array of 32 gray outlined squares (0.5° × 0.5°), each with a gap measuring 0.25° on one of its sides. The array was composed of one target (square with left or right gap) and 31 distracter (squares with top or bottom gaps) stimuli, each presented randomly in one of 64 locations generated by an 8 × 8 grid having 5.5° of spacing. Target identity (left or right) was also randomly selected on each trial. To minimize the structural coherence in the stimulus arrays, stimuli were individually and randomly jittered up to 2° in any direction. Participants were instructed to find the target as quickly and as accurately as possible, and to report the target identity with a button press. Each participant performed 384 search trials (four blocks of 96). 

Figure 1

 

(A) Example visual search trial. Participants searched freely for a target (left or right gap) among 31 distracters (top and bottom gaps) and were required to report the direction of the target gap using a button response. (B) Example change detection trial. Participants were presented with a memory array (two, three, four, six, or eight colored squares) followed by a retention interval. A test array was then presented and participants were required to report whether or not a color change had occurred using a button response. Eye position, as denoted by the dashed lines, was monitored in both tasks.

Trials were aborted if a button press was not made within 7000 ms of array presentation. This trial duration was selected to discourage sequential inspection of every item in the search array (i.e., preplanned scanning strategies). Both the variability in the duration of the initial fixation cross presentation and the randomization/jittering of stimulus locations were also implemented to minimize the possibility of preplanned search strategies. Participants were debriefed at the end of the experimental session to assess whether they employed preplanned search strategies. 

The visual search task was preceded by a perceptual discrimination task to ensure that participants were able to discriminate between the four search stimuli (top, bottom, left, and right gaps). On each trial, participants maintained fixation on a central cross. A single stimulus (0.5° × 0.5°, gap = 0.25°), randomly selected from the set of four possible stimuli, was presented at an eccentricity of 9° along the horizontal axis of the display (left or right visual field, randomly). Participants were instructed to press one of four buttons (up, down, left, or right arrows) as quickly and as accurately as possible to indicate the location of the gap within the square. Each participant performed 20 discrimination trials. All participants performed at >80% correct in this task. 

Change detection task

In the change detection task (Figure 1B), participants indicated whether the color of a single stimulus in an array of peripherally presented squares had changed following a retention interval using a button press. Each trial began with a central white fixation cross (1° × 1°) presented for 200–500 ms. The memory array was then presented for 500 ms, which consisted of colored squares (1° × 1°) presented randomly in one of eight concentric locations at an eccentricity of 10°. This was followed by a retention interval of 1200 ms during which only the central fixation cross remained on. Finally, a test array having the same set size and spatial configuration as the memory array was presented. 

The fixation cross remained on for the duration of the trial and participants were required to maintain fixation within a 2° × 2° window. Set size was varied randomly between two, three, four, six, and eight items. The color of each square was randomly chosen from a set of nine predetermined colors. In half of the trials, one square in the test array had a different color than in the memory array. Each participant performed 150 change detection trials (30 of each set size in two blocks of 75). 

Data analysis

The data from one participant were eliminated from all analyses due to a conspicuous reading-like strategy in the visual search task. This participant successfully completed only ∼50% of search trials within the 7000 ms time limit and her strategy was corroborated by her answers during the debrief. The data from a second participant were also eliminated from all analyses because she initiated eye movements in visual search trials at a significant delay (over 3 SDs greater than the mean) as compared to other participants. Her data suggested that she planned a search strategy prior to her first eye movement and this was confirmed by her verbal self-report at the end of the experimental session. The remaining 39 participants did not display such strategies, nor did they report them, and their data were included for analysis. Trials in which fixation was broken while the fixation cross was presented in either task, or in which an eye movement was not made following array presentation in the search task, were not analyzed. 

Visual search efficiency was assessed using several gaze-related measures. For each participant, we computed the average number of fixations per trial to assess the extent to which the display had to be explored before the target was found. We also computed each participant's average fixation duration to assess how much processing time was required to extract foveal information. Related to these measures, we also computed the time it took to initially fixate the target (search time). Search time was calculated using both correct and incorrect trials. Critically, we assessed search retention ability by computing their probability of refixation (P[refixation]). P[refixation] was calculated for each participant as the total number of refixations on distracter stimuli divided by the total number of distracter fixations across all trials. Consecutive fixations on the same distracter and refixations of the target were not considered refixations. We included several additional measures to assess the speed and accuracy of button responses. These included (a) the accuracy of the button response (search accuracy), (b) the time between display presentation and the button response (search RT), and (c) the probability of not making a button response within the allotted time (P[T not found]). 

The difference in latency between forward saccades (those to new items) and reverse saccades (those to items just fixated) in the search task was computed. Forward saccades were those having an angle within ±5° of the saccade preceding it, and return saccades were those having an angle of 180° ± 5° to the preceding saccade. Latency differences were only considered if saccade amplitude was within ±30% of that of the preceding saccade. 

Individual capacity limits of VWM ability were assessed in two different ways. First, we assumed an item-limit hypothesis by computing a VWM item-limit from the change detection task using Pashler's k (Pashler, 1988): k = N(h − f / 1 − f), where k is the item-limit, h and f are the hit and false alarm rates, and N is the set size. Second, we assumed a continuous-resource hypothesis by estimating each individual's mean precision at set size 1 using a variable-precision model as described by Keshvari, van den Berg, and Ma (2013). As we did not deliberately manipulate the magnitude of change in the change detection task, it was assumed that each change was of equal magnitude. 

A multivariate partial least squares (PLS) analysis was performed to explore the relationship between a participant's VWM ability and visual search efficiency. PLS correlation is comparable to a canonical correlation, where the goal is to derive linear combinations of two or more groups of variables that maximally relate the groups (Abdi & Williams, 2013). PLS analysis results in a series of latent variables (LVs) that express the different relationships between groups. Performance measures in the change detection task were taken as sensitivity (d′) for each set size. Performance measures in the visual search task were chosen to represent search efficiency: search accuracy, P[refixation], P[T not found], search time, and fixation duration. Because both the number of fixations per trial and search RTs were highly correlated with search time (Pearson correlation, r = 0.84, p ≪ 0.001 and r = 0.91, p ≪ 0.001, respectively), we considered these three to be redundant measures and only included search time in the PLS analysis. 

For the PLS analysis, the covariance between the visual search and change detection datasets was first computed across subjects to form a cross-covariance matrix. Singular value decomposition was then performed on this cross-covariance matrix to produce LVs, each containing two elements: (a) a weighted linear combination of change detection performance variables that, as a whole, covary with search performance (where the weights are referred to as “saliences”), and (b) a singular value (similar to an eigenvalue) that is the strength of the covariance. The first element of each LV was then multiplied by the raw change detection performance measures to yield individual subject change detection scores. Finally, the correlation between change detection scores and search performance was calculated. Statistical significance of each LV was assessed using a permutation test with 1,000 permutations (Edgington, 1995). The reliability of the saliences was assessed using a bootstrap estimation (500 samples, with replacement) of the standard error (Efron & Tibshirani, 1993). The salience:bootstrap standard error ratio approximates a z-score. Bootstrap estimation was also used to derive 95% confidence intervals for the LV correlations with behavior. 

Results

Participants were highly accurate in performing the visual search task (M = 98.3%, SD = 2.0%, range = 88.3%–100%). The probability of being unable to complete a trial in the allotted time was generally low (M = 0.034, SD = 0.033, range = 0–0.135). Participants were quick to initiate visual search, with initial saccadic reaction times occurring on average 198 ms (SD = 28 ms, range = 133–288 ms) following display onset. Table 1 summarizes other search performance measures. Only a small percentage of trials were completed after a single fixation (M = 8.6%, SD = 4.2, range = 3.5–21.6) and search time varied with the number of fixations per trial (Pearson correlation, r = 0.84, p ≪ 0.001), suggesting that participants made serial shifts of attention during search and that the task promoted active vision. Crucially, refixations occurred with a low probability (M = 0.050, SD = 0.018, range = 0.017–0.084), suggesting that participants were able to retain objects they had already examined. There was no temporal delay for return saccades and, instead, a slight decrease in latency for return saccades was observed (M = 183 ms, SD = 22 ms) as compared to forward saccades (M = 191 ms, SD = 22 ms; t test, t[38] = −2.88, p < 0.01). 

Table 1

 

Visual search performance measures across participants (n = 39).

	M (range)	SD
Fixations per trial	9.3 (5.9–13.4)	1.9
Fixation duration (ms)	213 (177–265)	22
Search time (ms)	2186 (1403–3078)	416
Search RT (ms)	2760 (1756–3878)	478

In the change detection task, performance accuracy decreased with increasing set size, as seen in Table 2, one-way ANOVA, F(4, 194) = 87.6, p ≪ 0.001, while response times increased, F(4, 194) = 6.0, p < 0.001. Correspondingly, participants showed decreasing d′with increasing set size, F(4, 194) = 75.5, p ≪ 0.001. These data suggest that participants had a limited capacity for remembering items over a short delay. 

Table 2

 

Change detection performance measures across participants (n = 39).

	M ± SD (range)
Set size	2	3	4	6	8
Accuracy (% correct)	93.9 ± 7.6 (69.2–100)	88.3 ± 8.8 (63.2–100)	80.9 ± 8.6 (61.1–100)	68.3 ± 11.1 (45.0–89.5)	62.2 ± 7.9 (47.4–77.8)
Response time (ms)	773 ± 149 (564–1141)	821 ± 169 (357–1177)	876 ± 174 (596–1209)	918 ± 177 (576–1403)	953 ± 241 (524–1915)
Hit rate	0.84 ± 0.11 (0.50–0.93)	0.80 ± 0.13 (0.40–0.93)	0.69 ± 0.13 (0.33–0.92)	0.53 ± 0.19 (0.20–0.91)	0.39 ± 0.15 (0–0.80)
False alarm rate	0.10 ± 0.04 (0.07–0.25)	0.12 ± 0.06 (0.07–0.30)	0.12 ± 0.06 (0.07–0.37)	0.18 ± 0.14 (0.07–0.75)	0.19 ± 0.13 (0.07–0.71)
Sensitivity (d′)	2.37 ± 0.48 (0.84–2.93)	2.14 ± 0.54 (0.97–2.97)	1.74 ± 0.47 (0.79–2.85)	1.05 ± 0.63 (−0.46–2.10)	0.68 ± 0.43 (−0.06–1.56)

The nature of VWM capacity limits has been highly debated in the literature. Some have argued for a model that assumes storage using discrete slots, in which a limited number of items can be stored in WM at any given time (Pashler, 1988). Alternatively, VWM has been argued to be governed by limited shared resources that are divided between all of the items to be stored (van den Berg, Shin, Chou, George, & Ma, 2012). While recent evidence has suggested that the continuous-resource model may provide a better description of performance in WM tasks (Ma, Husain, & Bays, 2014), item-limit models are still widely used (Luck & Vogel, 2013). In this paper, we consider both to be a possibility and therefore assess VWM ability using both types of models. First, we measured a VWM item-limit (k) while assuming an item-limit hypothesis. Next, we measured VWM precision at set size 1 (J̄₁), while assuming a continuous-resource hypothesis. The memory item-limit (k) for our participants was estimated to be, on average, 2.44 items (SD = 0.63, range = 0.65–3.56). Memory precision (J̄₁) was estimated to be 190.4 (SD = 130.8, range = 9.2–521.7). 

To assess whether VWM ability determined performance in the search task, participants' VWM capacity was correlated with their probability of refixation in the search task. Figure 2 shows how a greater VWM capacity was associated with a lower probability of refixation when VWM capacity was assessed assuming an item-limit model (Figure 2A; Pearson correlation, r = −0.49, p < 0.01) or a continuous-resource model (Figure 2B; r = −0.36, p < 0.05). These results suggest that, no matter which model of VWM we assume, the retention of fixated objects during visual search is related to VWM ability. 

Figure 2

 

Each participant's probability of refixating a previously fixated distracter in the active visual search task is plotted as a function of their VWM capacity as determined by the change detection task, where capacity was computed as (A) an item-limit (k) and (B) as precision (J̄₁).

To further avoid any assumptions on the nature of the limitations of VWM, we also performed a PLS correlation analysis by relating raw change detection performance measures to visual search efficiency. PLS has the added advantage that it is well-suited for data in which variables within a group are themselves correlated (McIntosh & Misic, 2013). One significant LV (p < 0.01) was obtained and displayed a pattern describing decreased VWM performance across set sizes. The reliability of this pattern is depicted as the salience to bootstrap-derived standard error in Figure 3A. This pattern of VWM performance was negatively correlated with the accuracy of visual search responses, and positively correlated with the probability that the target would not be found, search time, and the probability of refixation (Figure 3B). This LV, however, was not correlated with the average fixation duration. The ratio of the salience to bootstrap standard errors were <−3.0 for d′ in set sizes 4 and 6 (Figure 3A), suggesting that change detection performance in these conditions best predicted visual search efficiency. This was likely because our participants expressed the greatest variability in VWM performance at these set sizes. In other words, set sizes 4 and 6 together may have best captured the limits of VWM capacity across individuals. Regardless of the way in which we quantified VWM capacity, we found increased VWM ability to be associated with greater visual search efficiency. 

Figure 3

 

(A) Bootstrap ratios (salience:bootstrap standard error) of the first LV (p < 0.01). (B) Correlation between the first LV and visual search efficiency metrics. Error bars are 95% CI.

To assess whether the relationship in VWM ability and visual search efficiency was due to differences in perceptual ability or strategy, we performed a set of control analyses. First, we examined whether individuals with greater VWM capacity were able to process more items parafoveally during visual search. Individuals with better parafoveal processing may not need to fixate the most proximal stimuli, and saccade amplitudes for these individuals are expected to be larger than for individuals with poor parafoveal processing. We found no correlation between the mean saccade amplitude for each participant and their VWM capacity (for k: Pearson correlation, r = 0.22, p = 0.17; for¯J₁: r = 0.09, p = 0.60), suggesting that the better visual search efficiency in individuals with higher VWM capacity was not due to differences in parafoveal processing. This was additionally supported by the results of our perceptual discrimination task, which indicated that our participants' discrimination of our search stimuli at an eccentricity similar to what would be considered an adjacent location in the search display was near perfect (see Methods). Second, we examined whether individuals with greater VWM capacity were planning their search path prior to the initiation of the first saccade by relating the median saccadic response time in the visual search task for each participant to their VWM capacity. Again, we found no correlation between saccadic response time and VWM capacity (for k: r = 0.11, p = 0.47; for ¯J₁: r = 0.19, p = 0.25), suggesting that the greater visual search efficiency exhibited by higher capacity individuals was due to differences in information processing within an ongoing trial rather than differences in visual processing before the first eye movement. 

Discussion

Using both a change detection and an active visual search task, we show that visual search efficiency is partially mediated by VWM. In the search task, participants retained information about previously fixated items, as evidenced by a low probability of refixating those items. The aversion to previous spatial locations was not accompanied by a temporal aversion: The latencies of return saccades were not longer than those of forward saccades. We show that individual differences in VWM capacity predict refixation probability in oculomotor search. Importantly, individual differences in VWM capacity also predicted other aspects of search efficiency including response accuracy, probability of target acquisition, and overall search time. 

While visual search is often considered a paradigm for studying the allocation of attention, several previous studies have specifically explored the role of memory in this complex task. Horowitz and Wolfe (1998) showed how subjects were still able to detect search targets when the items in the display were randomly displaced within a trial. They suggested that information about the search display does not accumulate over the course of a trial, and that attention is not guided by information about previously attended items. A number of studies have since refuted this interpretation, and instead have reported how information is, in fact, retained over the course of a trial (Gibson, Li, Skow, Brown, & Cooke, 2000; Gilchrist & Harvey, 2000; Peterson et al., 2001), as well as across trials in visual search (Chun & Jiang, 1998; Thomson & Milliken, 2012; for review, see Hollingworth, 2006). It has been argued that the discrepancies between these studies and Horowitz and Wolfe's (1998) may have to do with whether items are displaced to new locations or ones that were previously occupied (i.e., locations in memory; Kristjansson, 2000). Notably, WM capacity has been shown to predict performance in covert search tasks in which the eyes did not move: High capacity individuals have faster button response times (Poole & Kane, 2009; also see Al-Aidroos, Emrich, Ferber, & Pratt, 2012) and shallower search slopes (D. E. Anderson, Vogel, & Awh, 2013; Sobel, Gerrie, Poole, & Kane, 2007) than individuals with low capacity. We add to this literature by using an overt search task (where the eyes were allowed to move) to demonstrate how VWM ability impacts refixation rates and, more generally, visual search efficiency. Our findings therefore indicate that VWM plays a role in active vision. 

We took steps to minimize the possibility of preplanned (nonmemory) search strategies. These included randomization of the global display configurations and the randomization of display onset times. Moreover, participants likely did not plan an entire search path before beginning search as their saccadic response times were short (<200 ms) and these were not predicted by an individual's VWM capacity. The majority of our participants did not display or report obvious pre-planned scanpaths or strategies, and we eliminated the data from two participants who did. A more thorough analysis of scanpaths and search strategies and how they relate to VWM ability will have to be explored in the future. While the possibility of a nonmemory mechanism in search retention cannot be entirely ruled out, it is unlikely that two separate mechanisms mediate the retention of previously fixated items. Instead, given the evidence for obligatory coupling of memory to saccade locations (Shao et al., 2010), as well as the tight coupling of memory and selective attention (Awh & Jonides, 2001), it is more likely that previous observations of visual search retention that were attributed to an IOR-related mechanism were, in fact, observations of the major role that memory plays in active vision. Evidence for an attentional disengagement account in search has come from observations that subjects do not refixate previously fixated items (Gilchrist & Harvey, 2000). Conversely, in selective attention studies, the IOR effect is measured as a temporal delay in returning attention to a previously attended location in both overt (Klein & MacInnes, 1999) and covert (Posner & Cohen, 1984) attention tasks. When using temporal delay as a measure for attentional disengagement, latency differences are inconsistently reported (Abrams & Pratt, 1996; Tipper, Weaver, & Watson, 1996), and effects are especially minimal in active visual search (Bays & Husain, 2012). In line with these previous observations, we found no delay in return saccades. Interestingly, the IOR effect has been considered by some to be more closely aligned with a memory-based account rather than an attentional disengagement one (Lupiáñez, Martín-Arévalo, & Chica, 2013). This view is additionally supported by observations that delays in responding during covert attention tasks decay with the number of attended locations (Snyder & Kingstone, 2000) and are eliminated when VWM is concurrently occupied (Castel, Pratt, & Craik, 2003). Notably, temporal delays in saccades are observed when they are made to remembered locations (Belopolsky & Theeuwes, 2009; but see Wong & Peterson, 2013), suggesting a role for VWM in the IOR effect. Such latency differences may be a secondary consequence of the primary goal of preventing refixations, with task and motivational differences influencing the existence of a delay (Bays & Husain, 2012). 

WM is often framed as an online workspace, the contents of which are temporary, quickly accessible, and dynamically updated using both new and retrieved information (Baddeley, 1992). It is difficult to discern from our data the nature of these VWM representations during search. They may be representations for items or locations recently attended/fixated. They may also be reactivated representations, such as gist information initially extracted about the search display, retrieved from memory. In this framework, refixations occur as a consequence of those items or locations having been displaced from the VWM workspace. This is distinct from an attentional disengagement mechanism that would involve a simple sliding temporal window of inhibition. While the probability of refixation would still increase with time, in the memory-reliant case, refixation probability would also depend on other factors affecting memory encoding and retention, such as stimulus salience and object complexity. This may explain why reports of visual search retention capacity range from just a few intervening fixations before a refixation (Gilchrist & Harvey, 2000; McCarley et al., 2003) to many intervening fixations (Dickinson & Zelinsky, 2007; Peterson et al., 2001). Future research is needed to elucidate the contents of the VWM workspace during visual search. 

Our findings add to a broader literature on how WM performance predicts a variety of different cognitive abilities including reading comprehension (Daneman & Merickle, 1996), decision making (Ester, Ho, Brown, & Serences, 2014), and reasoning ability (Halford, Cowan, & Andrews, 2007). These relationships may be a consequence of the strong links between WM and executive control. WM capacity is commonly thought to depend on an individual's ability to control attention (Engle, 2002). For example, top-down attentional control may impact early stages of perceptual encoding (Gazzaley & Nobre, 2012), or serve to inhibit irrelevant representations and responses (Lustig, Hasher, & Zacks, 2007). In both cases, individual differences in attentional control are manifested as differences in WM performance. Taken together with our findings that VWM capacity predicts multiple aspects of oculomotor search performance, these studies suggest that there exists a general capacity limit dictated by the brain's limited resources (Franconeri, Alvarez, & Cavanagh, 2013). These resources likely arise from a common source and are therefore shared across different stages of visual processing (D. E. Anderson et al., 2013). In our task, then, individuals with higher VWM capacity may be better equipped to process and maintain incoming information to guide visual behavior. This is supported by neuroimaging evidence that describes a substantial overlap between the neural networks responsible for spatial WM and covert search (E. J. Anderson, Mannan, Rees, Sumner, & Kennard, 2010; also see Fusser et al., 2011; Silk, Bellgrove, Wrafter, Mattingley, & Cunnington, 2010), as well as between networks responsible for covert attention and saccades (Corbetta, 1998), IOR and covert attention, and IOR and saccades (Mayer, Seidenberg, Dorflinger, & Rao, 2004). Different cognitive functions are thought to arise from the same brain structures by variation in their dynamic interactions with other structures in the network (i.e., neural context; McIntosh, 1999; also see M. L. Anderson, Kinnison, & Pessoa, 2013). The same resource limitations would therefore exist across different contexts, or cognitive processes, of highly overlapping neural networks. 

Conclusions

Our results add to a growing literature on the interplay between attention and memory in guiding active vision. In such a dynamic process, memory and attention act in parallel on each fixation (i.e., encoding retrieval and attending in cycle) to influence the selection of the next saccade goal. 

Acknowledgments

This work was supported by grants from the Canadian Institutes of Health Research (CIHR) to J. D. R. and the James S. McDonnell Foundation to A. R. M. We are grateful to W. J. Ma for providing the code for the variable-precision model. We thank D. McQuiggan and L. Watt for technical assistance, and M. D'Angelo for constructive feedback. K. S. holds a Fellowship Award from the CIHR. 

Commercial relationships: none. 

Corresponding author: Kelly Shen. 

Email: [e-mail]. 

Address: Rotman Research Institute, Baycrest, Toronto, Canada.