Using eye movements to explore switch costs in working memory

Abstract

Abstract

Abstract:

Abstract  Updating object locations in working memory (WM) is faster when the same object is updated twice in a row compared to updating another object. In analogy to repetition priming effects in perceptual attention, this object-switch cost in WM is thought of as being due to the necessity to shift attention internally from one object to another. However, evidence for this hypothesis is only indirect. Here, we used eye tracking and a classic model of perceptual attention to get a more direct handle on the different processes underlying switch costs in spatial WM. Eye-movement data revealed three different contributors to switch costs. First, overt attention was attracted initially towards locations of the previously updated object. Second, longer fixation periods preceded eye movements between locations of different objects as compared to (previous and new) locations of the same object, most likely due to disengaging and reorienting focal attention between objects. Third, longer dwell times at the to-be-updated location preceded manual responses for switch updates as compared to repeats, probably indicating increased uncertainty between competing sources of activity after the actual attention shift. Results can easily be interpreted with existing (perceptual) attention models that propose competitive activation in an attention map for target objects.

Introduction

The role of working memory (WM) in day-to-day functioning extends beyond the maintenance of information as many of the tasks we perform require us to manipulate or update the representations held in WM. Updating can be targeted at only a subset of the information in WM. For example, when performing a multidigit addition, people often break the calculationdown into more manageable steps, performing operations on some digits whilst maintaining the others in WM without interference. The process underlying such an operating step on a subset of internal representations has been compared conceptually to the allocation of attention in perceptual tasks (Oberauer & Bialkova, 2009), giving rise to the development of WM models such as Oberauer's three-embedded components model (Oberauer, 2002, 2009). In this model, there is an activated part of long-term memory (peripheral component), a region of direct access (central component), and a focus of attention into which objects have to be moved to be updated. The mechanisms underlying such object selection into the focus of attention have yet to be established; here, we try to characterize their functioning in spatial WM by hypothesizing that they are directly comparable to visuospatial shifts of attention and can thus be examined with overt attention shifts (i.e., eye movements). 

The existence of a selective attention mechanism (in the form of a focus of attention) in WM is supported by four lines of evidence (for a recent review, see Oberauer & Hein, 2012). The most relevant of these for the present study comes from object switch costs in WM updating tasks. These tasks consist of modifying an existing object in memory according to a sequence of instructions. Garavan (1998) investigated updating latencies in a task in which participants were required to repeatedly update separate counts of two different shape categories according to individually presented shapes. Participants were faster to indicate they had updated the count when updating the same count twice in a row (repetition, e.g., adding to the count of triangles on two successive updates) as opposed to updating a different count to the one previously updated (switch, e.g., a triangle followed by a square). Note that the process of mentally updating the object is common to both repetition and switch updates; it is the selection of the object to-be-updated that differs. Switch costs have similarly been observed in arithmetic digit updating (Oberauer, 2002, 2003) and spatial-updating tasks (Kübler, Murphy, Kaufman, Stein, & Garavan, 2003; Oberauer & Bialkova, 2009). Based on such observations, Oberauer and Bialkova (2009) proposed that the single-chunk focus of attention (FoA) in working memory serves a function analogous to the focus of attention in perception, namely to select a single object or event for processing at the expense of other objects or events. Indeed, repetition priming/switch costs are well studied in visual attention tasks (for a recent review see Kristjánsson & Campana, 2010). To date, it remains unclear whether such an analogy between a focus of attention in perception and in WM also means that both types of selective attention are based on a similar (or even identical) mechanism. This leads us to the main question of the present research: What are the mechanisms underlying selective attention to internal (WM) representations and how do they contribute to the switch costs observed? 

One avenue for exploring this question draws upon methods commonly used to examine attention allocation to external (sensory) sources of information, namely eye tracking. Spivey and Geng (2001) used eye movements as an indicator of attention to memorized items: When asked to recall the identity of an item presented previously in an array, unprompted participants tended to saccade to where the object had been located earlier. Further, participants' eye movements closely corresponded to presented locations during a retention period in a serial spatial recall task, and the occurrence of fixations between consecutive items correlated with higher performance at recall (Tremblay, Saint-Aubin, & Jalbert, 2006). 

Premotor theories of attention (e.g., Rizzolatti, Riggio, Dascola, & Umiltá, 1987) suggest that spatial attention shifting consists of programming a saccade to a location, which is then executed in overt shifts of attention or inhibited in covert shifts of attention (i.e., shifts of attention in the absence of eye movements). Spatial shifts of attention and saccadic eye movements co-occur in most natural viewing conditions, with covert shifts to a location shortly preceding a saccade and attention to locations other than that of the saccade target being impaired (Hoffman & Subramaniam, 1995). Specifically, during saccade execution, participants are unable to discriminate items reliably in areas other than the target location of the saccade (Deubel & Schneider, 1996). Accordingly, it can be assumed that a saccade to one location is incompatible with a concurrent shift of attention to an item held in memory at an alternative location. Evidence supporting such an assumption comes from work by Awh, Jonides, and Reuter-Lorenz (1998), in which spatial memory performance is impaired when a secondary perceptual discrimination task requires a shift of spatial attention to an alternative location. 

The aim of the current study was to characterize how attention is oriented to, and between, internal object representations using a memory-updating task similar to that used by Kübler et al. (2003), while recording participants' eye movements. Participants memorized the locations of two objects in a grid and updated their locations (i.e., mentally shifted an existing object in memory from one location to another) one update at a time according to arrow cues presented in the center of the screen. 

The advantage of this object-switching paradigm for examining selection in WM is that we can manipulate the prioritization of individual representations in WM from moment-to-moment while holding memory load constant. To update one object's location while holding the other object's location in WM unchanged, participants must orient their attention to the object to be updated. We carried out two experiments with this paradigm. In the first, participants were instructed to return to the location of the arrow cue between updates, though this instruction was not enforced. As not all of our participants consistently followed our instructions, we made the appearance of each cue contingent upon participants' gaze returning to the center of the display in the second experiment, giving us tighter control over the starting point of the eyes at the beginning of each updating step. 

As the mechanisms underlying shifts of attention in WM are yet to be unequivocally defined, we adopted the framework of three interactive attention networks outlined by Posner (2008), Posner and Petersen (1990), and Posner and Rothbart (2007) as a working hypothesis. This framework identifies separable attention components of alerting to the to-be-updated object, the actual orienting (shift) of attention, and of executive attention. Using this framework allows us to generate predictions about different mechanisms involved in attention switching and decompose each individual updating step into several key processes. 

We assumed that eye movements would reflect attentional shifts toward task-relevant locations, as evidenced by previous studies (e.g., Awh et al., 1998; Tremblay et al., 2006). If so, the moment at which participants made a saccade to the new location of the target to be updated should be directly associated with the speed of their manual reaction times (RTs). Accordingly (and considering the large switch cost in reaction times in WM), we expected saccades to arrive at this new target location later on switch updates than on repeat trials. We assume that the physical shifting of the focus of attention from one object location to the next should be relatively fast, but the prior selection of an object currently not in the focus of attention, as required for switch trials but not repeats, should be slow. If we express this in terms of Posner's attentional framework, then alerting to a new object would constitute the mechanism of processing the new cue information and matching it to the object to be updated. If this cue was not valid (i.e., did not correspond to the object currently held in the FoA), then reorientation would be required. Under such circumstances, the attentional focus needed to be disengaged from the current object and shifted to the new, a process that is obligatory on switch updates but not on repetitions. Based on this, we predicted that (a) participants would be faster to initiate a saccade on repetition updates (irrespective of landing position), and/or (b) the first saccade would be more likely to be the actual target for the update on repetition updates. Note that such an analysis assumes that object disengagement in switch trials occurs as a function of the update. However, as already implied in Prediction B, participants may not immediately switch their attention between objects in response to the onset of a switch cue. To account for such a possibility, as well as to investigate saccade behavior across the entire updating step, we further compared self-initiated switches of attention between objects (between-object saccades) to saccades between locations reflecting the same object (within-object saccades). If between-object switches took longer than within-object switches, this would be further direct evidence for a single-item focus of attention in working memory as suggested by Oberauer (2002, 2009; see also McElree, 2006; McElree & Dosher, 1989; Wickelgren, Corbett, & Dosher, 1980). 

Whereas alerting to the new cue and the subsequent disengagement concerns behavior in the early stages of the update, processes of executive attention are more likely, though not exclusively, to manifest closer to the response. According to Posner (2008), Posner and Petersen (1990), and Posner and Rothbart (2007), the executive control network is associated with processes underlying the resolution of conflict/uncertainty between different sources of attention, thus playing a crucial role in error detection and decision-making (Fan et al., 2009). If so, executive processes should occur after the attentional orienting shift itself had occurred (i.e., after saccade execution to the new target location), and attention was prioritized at the new target location over potential distracter locations. We predicted that such uncertainty would result in longer fixation times at the new target location prior to manual responding on switch updates, as switches set higher demands on executive attention (including the actual decision). Note that this effect could also be expressed in terms of choice response models of reaction times, in which evidence is accumulated until a response threshold is reached (e.g., Brown & Heathcote, 2008; Ratcliff, 1978). Indeed, dwell time has been linked to evidence accumulation and decision making in the context of inhibition of return eye-movement tasks (Farrell, Ludwig, Ellis, & Gilchrist, 2010). Applying this logic to our scenario, switch updates would be expected to either have a higher response threshold or disruption associated with attention switching would slow the rate at which the threshold is reached. 

While we mainly focus on the analysis of eye movements to further our understanding of the attentional processes involved in switch costs in WM updating, the contributions of the three different attentional network components suggested by Posner (2008), Posner and Petersen (1990), and Posner and Rothbart (2007) should also be visible in RT distributions. Drift rate (the rate at which a threshold is reached in choice RT models) has been linked to particular characteristics of RT distributions in empirical work and simulations. For example, the ex-Gaussian distribution is a convolution of a Gaussian and an exponential function, represented by the three parameters mu, sigma (corresponding to the mean and standard deviation of the Gaussian component, respectively), and tau (representing the mean and standard deviation of the exponential component). This distribution provides a good fit to typically positively skewed RT data. It has been argued that shifts in the Gaussian component of the distribution represent the insertion of an additional cognitive/motor process whereas an increase in the tau component indicates slowing of decision processes (Hockley, 1984). Further, simulations by Matzke and Wagenmakers (2009) link drift rate to large effects on the tau parameter. Therefore, if switching to a new object in WM involves the insertion of an additional processing step, such as attentional disengagement and orientation to a new object, then RTs on switch trials should differ from RTs on non-switch trials primarily in a larger mu parameter (i.e., a shift of the Gaussian component). In contrast, if switch trials were characterized by more difficult, and therefore slower, decision processes (e.g., due to increased uncertainty about the target item), then the distributions of switch RTs and of repeat RTs should differ primarily in the tau parameter. 

Our analysis thus addresses the following points in order: 

•  
  To link our task back to earlier studies, we examine first the standard switch cost effect in manual RTs (Manual reaction times) and perform an ex-Gaussian analysis of the distributions in repetition and switch trials to establish whether differences are driven by shifts in the mu and/or tau components, thought to represent an additional cognitive process or a slowing in decision making, respectively.
•  
  Then, we provide a general overview of fixation behavior in our task (Moment-to-moment distribution of fixation locations), summarizing participants' shifts of attention over the course of the updating step.
•  
  Next, we address our hypotheses concerning disengagement and orienting in the early stage of the update by examining latency and landing position of the first saccade (First saccade-landing positions).
•  
  We establish whether the initial allocation of attention (i.e., the first saccade) is predictive of the manual RT for its respective update, using multi-level modeling (Multilevel modeling).
•  
  Afterwards, we address our hypotheses concerning decision-making processes in the later stages of the update: After identifying a point at which the required updating operation has been performed, we examine if attention switching is still associated with an increased duration prior to the manual response and whether further attention shifts or increased dwell time underlie this behavior (Executive/decision-making components of the switch cost).
•  
  Next, we try to isolate the actual object switch cost from one object to the other (Examining object switch costs within updating steps).
•  
  Finally, we examine the relationship between patterns of eye movements across the trial sequence and their relationship with the correct identification of the objects' final locations (Predicting correct recall from saccade patterns).

Method

Participants

In Experiment 1, 13 undergraduate psychology students, aged 18–25, from the University of Bristol participated for course credits; in Experiment 2, 15 psychology students, aged 18–41, participated. All had normal or corrected-to-normal vision. Participants gave their informed written consent prior to participation in accordance to the Declaration of Helsinki, and the experiments were approved by the local Ethics Committee. 

Design and procedure

Stimulus presentation and data recording were conducted using MATLAB 2008b with PsychToolbox 3.0.8. (Brainard, 1997; Pelli, 1997), presented on an 18.1-in monitor with a 1280 × 1024 resolution. Participants were required to mentally update the position of two objects in a 3 × 3 grid through a sequence of mental shifts. The objects were a red circle (CIE x = 0.62, y = 0.35, L = 24.5 cd/m²) and a blue circle (CIE x = 0.15, y = 0.09, L = 13.5 cd/m²). Each update consisted of moving one of the circles by one square in a direction indicated by an arrow in the central grid square. The directions of movement were selected at random with the constraints that the shifted circle stayed within the grid and did not occupy the same position as the other circle. Half the updating steps in a sequence were object repetitions in which the update was performed on the same circle as the one shifted on the previous step; the other half were object switches in which the update was performed on the other circle. The order of switches and repetitions was pseudorandomized at the beginning of each trial. These factors were designed to make each operation unpredictable and remove any potentially confounding association between the operations and locations. The first update in each sequence, which could not be categorized as repeat or switch, was excluded from analysis. At the end of each trial, participants were required to indicate the final positions of both circles. A trial was counted as correct only if both final circle positions were recalled correctly. 

At the participant's viewing distance of 57 cm, the grid subtended a visual angle of 25° horizontally and vertically. It was drawn in black lines on a white background (127 cd/m²). The two circles, presented only at the beginning of each trial in the grid square of their initial position, were 2.8° in diameter. Updates were indicated by a centrally-presented, colored arrow (same colors as the two circles), 0.56° in height and width, alternating with a 0.56° fixation cross. 

A diagram of the trial sequence can be seen in Figure 1. A single trial consisted of the presentation of the grid with the starting positions of the red and blue circles for 2000 ms, which was then replaced by a blank grid with a central fixation cross for 500 ms (with a modification described below for Experiment 2). This was followed by a centrally-presented arrow indicating the target and direction for the first updating step which remained on screen until the participant pressed the Enter key to indicate that they had completed the required mental shift of the object. In Experiment 1, each arrow was followed by a fixation cross appearing for a fixed period of 500 ms, after which the next arrow was displayed. In Experiment 2, the arrow was similarly followed by a fixation cross, but this time presentation was restricted to 460 ms followed by 40 ms (20 successive gaze samples) in which the eye tracker evaluated whether fixation fell in the center square of the grid presenting the next arrow. This ensured that participants indeed fixated the center square before starting the next updating step and that the interstimulus intervals were of comparable length in both experiments. After training, all participants returned eye gaze to the center square as requested in Experiment 2. 

Figure 1

 

Trial sequence. Circles and arrows appeared in red and blue. Arrows indicating updates are not drawn to scale. Note that we use the term “trial” to refer to the entire sequence from the initial presentation of the two objects to the memory probes at the end of the updating and update to refer to each presentation of the arrow cue to which reaction time was measured.

Trial sequences consisted of 9, 11, or 13 updating steps to discourage anticipation of the end of the sequence. After the last updating step, the prompt “What was the final location of the red circle?” was presented for 1000 ms, followed by the grid with the numbers one to nine matching the position of the number keys on the response box. Participants reported the current position of the red circle by typing the appropriate number. After a response was given, the final position of the blue circle was probed in the same manner. Testing was completed in a single session lasting approximately one hour. The session consisted of one practice and seven test blocks of nine trial sequences each, with each block containing three of each sequence lengths in a random order. 

Participants were instructed to respond to each updating step as quickly as possible but not until they had completed the update. The recall stage at the end of the sequence remained untimed in both experiments. Participants were monitored during Experiment 2 to ensure that they returned to the center promptly between updates so that they could not spend additional time rehearsing or consolidating the items by intentionally delaying the presentation of the next arrow cue. 

In both experiments, participants were fitted with an Eyelink II head-mounted eye tracker (SR Research, Ltd.), and cameras were adjusted to obtain a clear image of both pupils. Eye movements were recorded based on tracking the movement of the center of the pupil at a rate of 500 Hz, with an accuracy of 0.4° (or better) of visual angle. Nine-point grid calibration and validation were performed before beginning each block, and drift correction was performed in between trials. Participants were asked to hold the response keypad in the palm of their nondominant hand and use the index finger of their dominant hand to enter their responses. Responses were given on a handheld keypad containing the numbers zero through nine and an Enter key, arranged as commonly found on the right side of a standard keyboard. A chin rest was used to maintain a fixed head position; and during the final recall phase, when participants were required to select one of multiple options on the key pad, they were instructed to raise the response key pad into their field of view if needed while keeping their head stable. Participants were given no specific instructions regarding their eye movements aside from to return to the center between updates to view each arrow cue. 

Analysis

Only data from each participant's dominant eye were analyzed, which corresponded for most participants to the eye with the best spatial eye-movement measurement accuracy. The dominant eye was assessed using the “hole-in-the-card test.” Eye movements and fixations were identified online and recorded for later analysis. A change in eye position with a minimum velocity of 30°/s and minimal acceleration threshold of 8000°/s² defined the onset of a saccade. A fixation began after the velocity fell below this value for five successive samples. 

Whereas it is typical in eye-movement research to discard saccades made before a certain latency threshold as anticipatory rather than stimulus driven, we retained all saccades made in between each arrow onset and the participant's response in our analysis. This decision was based on the assumption that our trials were series of updating steps building on each other, and we therefore did not expect behavior on each presentation to be independent of previous behavior. In such a scenario, early shifts of attention might be relevant to performance. 

Eye movements were grouped according to the following five criteria for possible landing positions: 

•  
  New: saccades to the new (updated) position of the to-be-moved circle
•  
  Passive: saccades to the position of the unmoved circle
•  
  Old: saccades to the previous position of the to-be-moved circle
•  
  Center: saccades to the center of the grid
•  
  Random: saccades to any other location on the screen

When the center square coincided with the new, the passive, or the old location, saccades to the center were reclassified under the according category exclusively (new, passive, or old). As the remaining saccades to the center square were reinforced by the task design (people returned back to the center at the end of an updating step to await the next instructions) but are of no theoretical interest, center saccades were not included in our analyses of saccade frequencies. Note that excluding updating steps in which the center was occupied by an object did not impact on the effects reported, so the updates themselves were retained where possible. In the multilevel analysis, the center category was similarly restricted to saccades that were made exclusively to the center (i.e., when the center did not coincide with a new, passive, or old location). 

To characterize the relationship between eye movements and participants' manual responses, we used multilevel regression with the nonlinear mixed effects (NLME) package (Pinheiro, Bates, DebRoy, & Sarkar, 2011), in R (R-Development-Core-Team, 2011). This technique provides coefficients for predictors in a similar way to standard linear regression (fixed effects) as well as describes the variability between group level factors (random effects). This allows us to account for performance at the level of each individual updating step, rather than using summary statistics from each participant whilst accounting for the similarities within, and difference between, each participant's performance. In addition to the standard examination of median RTs, RTs were further analyzed by fitting an ex-Gaussian function for each subject's data separately for switch and repetition updates using the egfit MATLAB function (Lacouture & Cousineau, 2008). 

Results

Only data from correct trials were included in the analysis. Data of the two experiments were combined for the purposes of analysis, including experiment as a factor to account for possible differences induced by experimental design. These effects and interactions did not reach significance and, for the sake of space, are not reported. Further, only the effects relevant to our hypothesis are reported; a full report is available at request. 

Task performance

Accuracy of final report

A trial was only counted as correct if both object locations were correctly identified. On average, participants were accurate in 79.08% (SD = 15.94) of test trials. A 2 (Experiment) × 3 (Trial Length) mixed analysis of variance (ANOVA) showed a significant effect of trial length (9, 11, 13) on accuracy, F(2, 52) = 4.12, MSE = 3.32, p = 0.022. A planned linear contrast between the sequence lengths revealed no significant linear trend in memory performance, F(1, 26) = 2.46, p = 0.129. 

Manual reaction times

A 2 (Experiment) × 2 (Update Type) mixed ANOVA was performed on participants' median RT for switch and repetition updates, revealing a significant switch cost (repeat: M = 979 ms, SD = 300 ms; switch: M = 1141 ms, SD = 307 ms), F(1, 26) = 80.25, MSE = 0.005, p < 0.001, in line with expectations for this task from the literature (Garavan, 1998; Oberauer, 2002, 2003). 

Decomposing the RT distribution allowed us to assess the possibility of multiple, or specific, effects of object switching on the speed of participants' updates. Paired t tests between conditions for each component of the ex-Gaussian fitted to reaction times showed a significant increase in mu from 717 ms (SD = 220) on repetition updates to 799 ms (SD = 226) on switch updates, t(27) = 7.08, p < 0.001, indicative of an additional process involved on switch updates. The tau parameter, associated with evidence accumulation and decision-related processes, increased from 384 ms (SD = 181) in repetition updates to 482 ms (SD = 197) in switch updates, t(28) = 6.57, p < 0.001. A significant effect was also found for sigma, t(28) = 2.40, p = 0.023, increasing from 82 ms (SD = 33) on repetition updates to 94 ms (SD = 34) on switches, indicating a greater degree of variability in the Gaussian component of the RT distribution. As such, the switch cost appears to be reflected in all three components of the ex-Gaussian distribution, though the size of the effect is notably larger for mu and tau than for sigma (see Figure 2). 

Figure 2

 

Ex-Gaussian probability density functions fitted to reaction time data for repetition updates (gray) and switch updates (black).

Eye-movement data

Overall frequencies

One participant's eye data were excluded due to excessive blinking (>2.5 SD from the sample mean). A 2 (Experiment) × 2 (Update Type) mixed ANOVA showed that participants made significantly more saccades per update on switches (M = 2.76, SD = 1.22) than on repetitions (M = 2.35, SD = 1.05), F(1, 25) = 39.03, MSE = 0.059, p < 0.001. This effect emerged as a main effect of update type in all of the following analyses and thus is not subsequently reported. 

Moment-to-moment distribution of fixation locations

To investigate these updating dynamics in more detail, we plotted where, on average, participants fixated on a moment-to-moment basis (see Figures 3 and 4). Figure 3 shows the percentage of correct updates each location was fixated on in 10 ms time bins. Note that, as trials have different durations, fewer and fewer trials contribute to this representation toward the end of the plots in Figure 3. To be able to compare the relative progression of events across updates of differing durations, we replotted the distributions as a percentage of a normalized update length (Figure 4). This reveals which locations participants attended to toward the point of responding. Note that for the purpose of the plots in Figures 3 and 4 and in contrast to our usual definition of center saccades, fixations were only categorized in object locations (i.e., new, passive, etc.) when the objects did not occupy the center square. This allows us to see that participants moved away from the center square (see Figure 4E) and arrived at the new location towards the end of both repetition and switch updates (see Figure 4A). If participants visited the old location, this was most likely to happen halfway through the updating step; visits to the passive and random locations did not seem to follow a particular temporal order. 

Figure 3

 

Percentage of correct updates on which eye movements visited each location, plotted in 10 ms bins. Gray lines represent repetition updates, black lines represent switch updates. A: new location, B: old location, C: passive location. D: random locations, E: center location. Note that the y-axis scales differ strongly for different locations.

Figure 4

 

Percentage of correct updates on which eye movements visited each location, normalized to update response time. Gray lines represent repetition updates, black lines represent switch updates. A: new location, B: old location, C: passive location. D: random locations, E: center location. Note that the y-axis scales differ strongly for different locations.

This visualization of the data thus suggests that completion of the updating step relies on a similar overall sequence of events for both switch and repetition updates, i.e., arrival at the old location peaked halfway through the update on both switches and repetitions, and the new location was reached towards the end of the update. 

First saccade-landing positions

Two further observations can be made in Figures 3 and 4: First, the main difference between switch and repeat trials with respect to saccade location distributions was found in the first half of the updating process, namely before the actual peak at the old location (Figure 4B). In switch trials, more saccades seemed to go first to the passive location (Figure 4C). Second, as all saccade landing positions are plotted in a normalized way relative to response, moving from the old to the new location as well as responding after the new location had been reached seemed both to be longer in absolute times for switch updates as compared to repeats (this can also be viewed in the later peaks in Figures 3a and 3b). To quantify these different observations, we started with analyzing the landing positions of the first saccade for each update with a 2 (Experiment) × 2 (Update Type) × 4 (Landing Position) mixed ANOVA, including only those saccades which were made to one of the four target locations (new, passive, old, or random). The dependent variable was the percentage of total correct updates per participant on which the according location was the destination of the first saccade. 

A significant effect was shown for landing position, F(2.07, 51.62) = 43.43, ε = 0.69, MSE = 118.69, p < 0.001. G.G. Bonferroni post-hoc comparisons showed that the new location (M = 24.64%, SD = 9.60) was a more frequent target for the first saccade than the passive (M = 12.94%, SD = 4.70, p < 0.001) and random locations (M = 6.13%, SD = 3.81, p < 0.001), respectively. Also, first saccades to the old location (M = 20.25%, SD = 7.83) were more frequent than those to the passive (p = 0.001) and random locations (p < 0.001). No other differences between the landing positions reached significance. 

We expected that participants would be more likely to saccade initially to the to-be-updated object (i.e., old or new location) when repeating operations on the same object. This behavior, a direct reflection of the participants' disengagement of an old object location, should be visible in the interaction between updating type and landing position. As predicted, a significant interaction was shown, F(1.98, 49.61) = 12.17, ε = .66, MSE = 37.49; p < 0.001. Participants made significantly more first saccades to the new location in repetition updates (M = 28.23%, SD = 12.28) than in switch updates (M = 21.04%, SD = 7.52, p < 0.001). The reverse pattern was observed for the passive location, which was more frequently visited first on switch updates (M = 14.76%, SD = 4.97) than on repetitions (M = 11.12%, SD = 5.44, p < 0.001). Similarly, random locations were the target of first saccades more frequently in switch updates (M = 7.07%, SD = 4.75) than on repetitions (M = 5.18%, SD = 3.20, p = 0.002). For the old location, in contrast, the difference between repetitions (M = 20.79%, SD = 8.73) and switches (M = 19.72%, SD = 9.49, p = 0.56) did not reach significance. This bias towards the new location on repetition updates and the passive location on switch updates is in line with our hypothesis that participants do not necessarily reorient to the correct object on switch updates. 

To assess whether this differential reorienting pattern for the first saccade was also reflected in differential saccade latencies, thus accounting for at least some of the switch costs observed, we compared the median latency of participants' first saccades of each updating step between switch and repetition updates A 2 (Experiment) × 2 (Update Type) mixed ANOVA revealed no significant difference between repetition (M = 402 ms, SD = 68 ms) and switch updates (M = 410 ms, SD = 67 ms), F(1, 25) = 2.91, MSE = 238.36, p = 0.10. The onset latency of the first saccade therefore does not seem to reflect any cost of object switching. 

Multilevel modeling

Even though overall the new location was visited similarly frequently for both updating types, the observation that the first saccade landed more frequently on locations other than the one of the to-be-updated object (new location) in a switch as compared to repetition updates raised a further question. Does the location of the first saccade reliably predict updating speed (as measured in reaction times)? To answer this question, we used a multilevel regression to establish the relationship between the first saccade landing position and RT on the level of each update. A logarithmic transformation was applied to the RT data to reduce the skew in the dependent variable's distribution. Coefficients for categorical predictors (landing position for the first saccade, update type, and experiment) were dummy coded to reflect changes from the reference category (new location, repetition updates, and Experiment 1, respectively) within each one of the factors. We fitted a series of models through several steps, starting with a simple model including all main effects as fixed effects and participant as a random effect. The second step added all two-way interactions as fixed effects (Model 2). The third step added all relevant three-way interactions as fixed effects. The fourth step involved assessing the impact of different random effects to the best combination of fixed effects from the previous steps (Model 1a). After each step, predictors which did not improve model fit were discarded, though main effects were included if they contributed to an interaction. Those with additional letters indicate the best model from each stage after removing predictors that did not improve model fit. Finally, the change in fit was assessed for each predictor in the model produced in the preceding steps. Model fit was assessed through the comparison of three criteria; the Akaike information criterion (AIC), the Bayesian information criterion (BIC), and the maximum likelihood of the model, from which the aforementioned figures were computed. Predictors were included only if they showed improvements in all three of these criteria over the more parsimonious model. 

The analysis treated each updating step as a case and included all steps on which a response was made from all updates with a correct final result (N = 13,517). All models used log-transformed RTs as the dependent variable and first-saccade location, update type (repeat vs. switch), experiment (one and two), and their interactions as predictors. We investigated several competing models, shown in Table 1. Numbered models reflect full models from each stage described above (i.e., Model 1 includes all main effects, Model 2 includes all two-way interactions, etc.). Those with additional letters indicate the best model from each stage after removing predictors that did not improve model fit. 

Table 1

 

Model-fit comparisons for models estimating RT from landing position of first saccade. Notes. Models lower loglik, AIC, and BIC values indicate better fit. F = First saccade location, UT = Update type, E = Experiment, PPT = Participant. N = 13517. Removing predictors from Model 3 to maximize fit produced a model identical to 2a.

Model	DF	Loglik	AIC	BIC	Fixed	Random
1	10	−4221.57	8463.14	8538.26	F, UT, E	PPT
1a	9	−4221.57	8461.14	8528.75	F, UT	PPT
2	21	−4204.10	8450.20	8607.95	F, UT, E, E*F, E*UT, F*UT	PPT
2a	11	−4221.04	8464.09	8546.72	UT, F, E, UT*E	PPT
3	26	−4192.76	8437.51	8632.82	F, UT, E, E*F, E*UT, F*UT, F*UT*E	PPT
4	11	−4165.40	8352.80	8435.42	F, UT	UT, PPT

The best fitting model (Model 4 in Table 1) included two fixed effects (first saccade location and update type) and two random effects (update type and participant). These random effects reflect variation between participants in their RTs and variation in the effect of update type on RTs across individuals. Note that including either “experiment” or its interactions as predictors does not improve the model fit. 

Parameter estimates for Model 4 are shown in Table 2. As already observed, object switching led to an increase in RTs compared to repetitions, indicated by the estimate for update type. Irrespective of update type, relative to trials with first saccades to the new location, trials with first saccades to any other locations had slower RTs (as reflected in positive parameter estimates in Table 2; each estimate reflects the change in log RT relative to a first saccade to the new location). The only exception to this rule were trials in which no saccades were made, in which case RTs were faster than updates with first saccades to the new location (as reflected in the negative estimate). In addition, though trials with first saccades to the old location had slower RTs relative to trials with first saccades to the new location, this difference was smaller than that for trials with first saccades to passive, center, or random locations (note that the confidence intervals of the estimates for the “first landing position: old” do not overlap with the estimates for the other locations). Thus, it becomes apparent that to speed up the actual updating process (as reflected in faster reaction times), the time at which saccades to particular locations occur in the sequence of saccades predicts RTs. 

Table 2

 

Fixed effect estimates (top) and random effect variance estimates (bottom) for best-fitting model (see Model 4, Table 1) predicting RT from landing position of first saccade. Notes: Estimates reflect size of the effect on the log transformed RTs. Degrees of freedom is 13484 for all t scores. ***p < 0.001.

Parameter	Estimate	95% CI		T
		Lower	Upper
Fixed
Intercept (new landing position, repetition update)	6.879	6.782	6.976	138.59***
Update type (switch)	0.137	0.107	0.167	9.04***
First landing position: passive	0.104	0.085	0.124	10.65***
First landing position: old	0.048	0.031	0.065	5.59***
First landing position: center	0.088	0.072	0.103	10.91***
First landing position: none	−0.150	−0.173	−0.126	12.50***
First landing position: random	0.097	0.069	0.125	6.90***
Random
Intercept	0.256	0.195	0.334
Update type	0.073	0.053	0.100
Residual	0.327

Executive/decision-making components of the switch cost

If we assumed that eye movements reflect the target of WM attention allocation at a particular moment in time during an updating step, then making a saccade to the new location would reflect the actual goal of the updating process itself, i.e., the focus of attention in WM was not only on the current target but had already updated its location. To gain insight into this actual updating process, the subsequent analysis focuses on a subset of trials only, namely on those for which a saccade to the new location occurred at some point during the updating. In particular, we investigate what happened both before and after the saccade toward the new location. We assumed that arrival at the new location reflected the point in time at which the new location of the to-be-updated object had been determined, and therefore the required updating process was completed. If this assumption is correct, then the object-switch cost in manual RTs should arise solely from participants arriving at the new location later in switch updates, as their attention was directed to other locations first (e.g., the passive location) and the switch cost arose solely due to processes influencing the early stages of the updating step. 

We therefore reexamined the effect of update type on both the time until the first arrival of the eyes at the new location as well as the time taken to respond after the eyes had reached this location. Indeed, participants took on average significantly longer to arrive at the new location on switch updates (M = 737 ms, SD = 152) than on repetitions (M = 631 ms, SD = 123), t(26) = 7.03, p < 0.001. Arrival at the new location accounted for about 100 ms of the total 160 ms switch costs observed, in line with our frequency analysis for the location of the first saccade reported earlier: In switch trials, participants were more likely to go to alternate locations initially before arriving at the new location. 

Intriguingly, however, this analysis left 60 ms and thus more than a third of the switch cost unaccounted for. Indeed, comparing the average time required between arrival at the new location and the participant's manual RT, participants also took longer to respond after making the saccade to the new location on switch updates (M = 607 ms, SD = 244) in comparison to repetitions (M = 533 ms, SD = 257), t(26) = 7.03, p < 0.001. Given that participants made more saccades to other locations before reaching the new location in switch as compared to repeat trials, did participants also make more saccades to other locations afterwards? If so, this would most likely suggest different spatial updating strategies for switch and repeat trials. For instance, it could be that, after computing the new location of the to-be-updated object, participants switch attention back to the other (passive) object to rehearse it (e.g., Awh et al., 1998; Tremblay et al., 2006), and they might do this more often on switch than on repeat trials (Figure 4C suggests that they converge to similar levels at the time of response, though differ earlier on in the update). Alternatively, instead of making additional saccades, participants might spend longer at the new location before responding on switch trials as compared to repetition trials, in line with predictions on dwell times within attention models, indicating re-engagement with a new target (Posner & Petersen, 1990) or increased uncertainty in decision making (Brown & Heathcote, 2008; Ratcliff, 1978). 

The first of these possibilities (more saccades to other locations following fixation of the new location) was examined with a 2 (Experiment) × 2 (Update Type) × 4 (Landing Position) mixed ANOVA on the percentage of trials on which participants made a saccade to the locations of interest after having already made a saccade to the new location. The effect of landing position reached significance, F(1.10, 27.40) = 6.947, ε = .37, MSE = 216.24, p = 0.012, with post-hoc comparisons showing slightly more saccades to the passive location (M = 10.71%, SD = 15.67) than repeated saccades to the new (M = 4.38%, SD = 5.24, p = 0.082) location, saccades back to the old location (M = 3.44%, SD = 3.39, p = 0.075), or to random locations (M = 3.61%, SD = 4.28, p = 0.075). Also, the effect of update type reached significance, F(1, 25) = 13.75, MSE = 2.38, p < 0.001, indicating that participants were indeed more likely to saccade to alternate locations before responding on switch updates (M = 5.79%, SD = 6.95) compared to repetition updates (M = 5.01%, SD = 6.62). Note that this value reflects the average over all locations, which differ from each other as seen in Figure 5. Therefore, the average is not representative of the total percentage of updates in which a saccade is made to any location. Moreover, this effect was comparably small, with the difference amounting to less than 1% of updates included. As such, it seems unlikely that such an effect could account for the remaining 60 ms of switch costs. More importantly, the Landing Position × Update Type interaction did not reach significance, F(1.88, 46.90) = 1.09, ε = .63, MSE = 2.36; p = 0.361, thus revealing no bias towards any particular location as, e.g., expected for any kind of rehearsal strategy (see Figure 6). 

Figure 5

 

Landing positions of first saccades on correct updates combined across Experiments 1 and 2. Error bars indicate ±1 SEM corrected for within-subject comparisons (Bakeman & McArthur, 1996). White bars represent repetition updates, gray bars represent switch updates.

Figure 6

 

Landing positions of saccades made after the first saccade to the new location. Please note that the new location here indicates updates in which participants had left the new location to return subsequently to it. Error bars indicate ±1 SEM, corrected for within-subject comparisons. White bars represent repetition updates, gray bars represent switch updates.

As saccades to other locations were relatively infrequent following arrival at the new location on switch updates compared to repetition updates, an alternative is that longer time dwell times at the new location prior to response contribute to delayed RTs on switch updates. This was assessed with a 2 (Experiment) × 2(Update Type) mixed ANOVA on participants' average total time per update spent fixating on the new location on updates on which a saccade was made to that location. A significant effect for update type was found, F(1, 25) = 14.22, MSE = 2082.30, p < 0.001, with more time (47 ms) spent fixating the new location on switch updates (M = 421 ms, SD = 123) than on repetitions (M = 374 ms, SD = 140). Thus, dwell time at the new location was indeed significantly longer on switch as compared to repeat trials. 

Examining object switch costs within updating steps

While the analyses so far allowed us to say when most of the switch costs occurred during the updating step, they did not really address the question of whether these costs were associated with having to swap the focus of attention between objects. In other words, we don't know whether these costs are related to moving a new item into the focus of attention independently of the other processes involved in an updating step. If so, this would corroborate WM models that propose privileged access to a single item (e.g., Oberauer, 2002, 2009). 

We therefore examined whether participants would take longer to prepare a saccade toward a location associated with an object not held in the focus of attention as compared to a saccade between locations reflecting the same object. To estimate this, we compared within-object saccades (old location to new, new to old) with between-object saccades (old to passive, passive to old, new to passive, passive to new). For each participant, we averaged the fixation durations for all occurrences of the first location associated with these types of combinations (e.g., averaging the fixation durations on the old location when followed by a saccade to the new location, along with fixations on the new location when followed by the old location for within object combinations). These were then analyzed with a 2 (Update Type) × 2 (Combination Type) repeated measures ANOVA. The mean fixations durations, along with the average frequency of these combinations, are shown in Table 3. (Please note that one participant, who made no within-object saccades on switch trials, was excluded from this analysis). A significant main effect was found for combination type, F(1, 25) = 21.92, MSE = 1743.72, p < 0.001, with significantly longer fixations preceding between-object saccades (M = 307 ms, SD = 86) in comparison to within-object saccades (M = 268 ms, SD = 71). The effect of update type, F(1, 25) = 0.041, MSE = 2931.59, p = 0.84, and the interaction, F(1, 25) = 0.76, MSE = 3046.53, p = 0.392, did not reach significance. Therefore, we do show evidence of longer fixation times (on average 39 ms) preceding shifts of attention to different objects in WM. To see how this might impact updating type (repeat vs. switch updates), we ran a further 2 (Update Type) × 2 (Combination Type) ANOVA on the total number of saccades made in these combination types. As known from previous analyses, more saccades were made on switch updates (M = 60.29, SD = 50.18) than on repetitions (M = 49.69, SD = 41.11), F(1, 25) = 14.32, MSE = 203.9, p = 0.001. Neither the interaction, F(1, 25) = 2.14, MSE = 234.82, p = 0.155, nor the main effect of combination type, F(1, 25) = 1.28, MSE = 4253.67, p = 0.268, reached significance. However, the simple fact that there were overall more saccades made for switch updates means that there is a larger net cost from object switching in switch updates. 

Table 3

 

Mean fixation duration (ms) for first saccade in combinations for within-object saccades as compared to between-object saccades by update type. Standard deviations shown in parentheses, average frequency of observations on which the means are based are displayed in italics.

Update type	Combination type
	Within	Between
Repetition	263 ms (72) 44.65	311 ms (115) 54.73
Switch	274 ms (75) 50.84	303 ms (85) 69.73

An additional prediction that arises regarding specific combinations of saccades is that of an increase in old to new saccades on switch updates due to the need to select the item not in the FoA prior to updating it. Whilst a trend does emerge for this combination to occur on more switch updates (M = 20.72%, SD = 16.65) than repetition updates (M = 18.20%, SD = 13.14), this difference does not reach significance, F(1, 25) = 1.67, MSE = 50.61, p = 0.21. Thus, these data do not implicate eye-movement combinations specifically in refocusing the FoA to a new WM item (though note that variability between participants at this level makes it difficult to draw firm conclusions). 

Predicting correct recall from saccade patterns

In addition to the analysis reported in the main text, in which we link the pattern of eye movements to manual reaction times in individual updating steps, we also analyzed whether the pattern of eye movements across the trial sequence (from initial object presentation to the memory probes) were predictive of correct identification of the objects. (Note that in contrast to the analyses described previously, which include only trials with correct recall, the analysis presented here entails all trials). The prediction of accurate recall from attending to remembered locations relates more closely to previous work examining the role of eye movements in rehearsal (e.g., Awh et al., 1998; Tremblay et al., 2006). 

To address this, we fitted a hierarchical logistic regression to our data, using the lme4 (Bates, Maechler, & Bolker, 2011) package in R. The dependent variable for this analysis was the correct recall of both items for a trial sequence (1 = correct, 0 = incorrect). The predictors were the percentage of updates in the sequence in which at least one saccade was made to the new, old, passive, and random locations. Experiment (0 = Experiment 1, 1 = Experiment 2) and trial sequence length were also included as predictors. The estimates for this model can be seen in Table 4. Both the percentage of updates in which participants made a saccade to the new location (associated with an increased likelihood of correct recall, indicated by the positive coefficient) and random locations (associated with a decrease in the likelihood of correct recall, indicated by the negative sign on the coefficient). 

Table 4

 

Fixed effect estimates (top) and random effect variance estimates (bottom) for hierarchical logistic regression, predicting correct recall of objects in a trial sequence based on percentage of updates in which a saccade was made to each location. Notes: **p < 0.001, *p < 0.01. AIC = 1444, BIC = 1487, log likelihood = −714.

Parameter	Estimate	Std. error	Z
Fixed
Intercept	2.266	0.637	3.556**
New	0.011	0.004	2.872*
Old	−0.003	0.005	−0.637
Passive	0.005	0.004	1.172
Random	−0.037	0.004	−8.245**
Trial length	−0.052	0.041	−1.271
Experiment 2	−0.014	0.488	−0.028
Random (variance estimates)
Intercept	1.384

Note that we interpret saccades to random locations here in a slightly different way than in our previous analyses. Here, we could assume that participants are still attempting to update and track the objects, but the increase in random saccades may reflect participants orienting attention to where they erroneously believe the objects to be. In correct trials, in contrast, presumably they do not believe an item to be located there, as they correctly identify the final locations at the end of the sequence. As these are qualitatively different reasons for random saccades, it is difficult to determine if there is a relationship between saccades to nontarget locations and interference with, or forgetting of, items. Also, note that our task is not designed to produce errors. However, outcomes of the analysis presented here further support the relationship between overt shifts of attention and attention to items in working memory. 

Summary of results

Our findings reveal that participants showed an RT cost when they needed to switch between objects held in WM to update relative to when items were repeated. In the ex-Gaussian distribution of RTs, these effects were largest in the mu and tau components. Moreover, saccade data revealed a tendency to shift gaze to the location an item had been moved to in WM (the new location) during the updating step. When exactly during an update a saccade was made to the new location turned out to be an important indicator for the time it took participants to update the new object location in WM and to respond. This was revealed as follows: (a) On repetition updates, the first saccades were more likely to land on the new locations of the to-be-updated item; on switch updates, in contrast, the first saccades were more likely to land on the location of the previously manipulated item (passive location) and on random locations; (b) Multilevel modeling revealed that RTs of updating steps were longer when the first saccade landed on a location other than the new one. Furthermore, about a third of the total switch cost was due to processes that occurred after participants had apparently identified the target location of the update, such as spending a longer time dwelling in the new location prior to response. Finally, (c) we showed that switching attention between objects in WM within updates was associated with increased fixation times prior to the saccade, in line with ideas about reallocation of the focus of attention between different objects in WM. 

Discussion

The aim of the current study was to use eye tracking to understand the allocation of attention underlying object switch costs frequently identified in the WM literature (Garavan, 1998; Oberauer, 2002, 2003). The reaction time switch cost in the present WM updating tasks averaged 162 ms across our two experiments. Analyzing updates in which eye movements were made to the actual target location of the update, we show that arrival at this location differs between switch and repeat trials by approximately 100 ms, which we attribute to an increase in the likelihood of orienting first to alternate locations on switch updates. Further, we show that another 47 ms of the switch cost are accounted for by an increase in dwell time on the targeted object's updated location prior to responding. Together, our results provide insight into previously unobservable processes underlying attention orienting to internal representations. 

There was a strong tendency of eye movements towards the new locations of the to-be-updated object at one point or another during the course of an updating step. In both experiments, saccades to this new location were almost twice as frequent as those to other locations, including saccades to the previous (old) location of the object. If the new location was reached earlier within an updating step (i.e., already in the first saccade), this seemed to be advantageous, as manual RTs were reduced for such updates. Indeed, this held true for both repeat and switch trials. Such a correspondence between eye movement behavior and RTs supports our assumption that saccadic shifts do indeed provide insight into the order of attention shifts to objects within WM as previously suggested in the context of rehearsal (Awh et al., 1998; Brandt & Stark, 1997; Spivey & Geng, 2001; Tremblay et al., 2006). Similarly, we show that eye movements to the locations of memorized objects, and a lower rate of saccades to random locations, predicts accurate recall in our task. Our approach differs from these earlier studies in that we are not looking purely at recall. In our experiments, eye movements do not simply reflect returning to the source of previously presented information, as our task entails both the selection and manipulation of a subset of WM content. Here, we therefore argue that eye movements to on-screen locations reflect dynamic shifts in priority in spatial WM. 

Drawing on models of attention networks (Posner, 2008; Posner & Petersen, 1990; Posner & Rothbart, 2007), we dissected the switch cost relative to its contributions by different attention processes such as alerting, disengagement from the previous target, shifting of attention, and executive attention (such as subsequent engagement with the new target and decision making). We now discuss these in turn. 

Altering and orienting

We hypothesized that there were two potential effects in relation to early processes concerning alerting to a new cue and disengaging from the current object in the focus of attention, an additional process required on switch updates. An initial alerting effect would predict longer first-saccade latencies on switch updates, as the cue does not match the current attentional focus, though we did not observe this and therefore exclude differences in alerting between repeat and switch trials. Instead, we saw a difference in the target location of the first saccade: First saccades were more likely to be oriented directly towards the new location on repetition updates, whereas the passive (and random) locations were the target of first saccades more often on switch updates relative to repetitions. As such, our data suggest that the time taken for the focus of attention to arrive at the desired location is, at least in part, attributable to selecting the target of attention deployment. Participants do not seem to consistently disengage from the previous target (in switch trials, the passive location) and reorient to the new object on the onset of a switch cue. 

Before being able to conclude that this lack of disengagement is due to a spillover of activity at the location of the object held previously in the FoA in WM, analogous to an attentional activation map (see e.g., Desimone and Duncan, 1995), we have to consider an alternative explanation. The difference of the first-saccade landing position might reflect an intentional strategy by participants to refresh or rehearse the passive object before switching to the to-be-updated object. For the following reasons, the latter explanation seems unlikely: First, it cannot account for the increased number of saccades to random locations during switch updates. Second, it is not clear how a pure rehearsal account would manifest in more of such behavior on switch updates; as repetition updates entail that the passive item has been out of the FoA for a longer period of time, active rehearsal would be expected to occur more frequently in repetition updates. This is not to say that refreshing or rehearsing behavior doesn't play a role in our task, but that the motivation to do so requires consideration of how attention switching impacts on the state of WM representations in a way that prompts this behavior. Indeed, our moment-by-moment analysis indicates that the comparably infrequent saccades to the passive location during repetition trials occur most likely toward the end of the updating process and after the new location had been visited (see Figures 3 and 4), an observation more likely to reflect rehearsal strategies. Moreover, as indicated by Figures 4 and 6, such rehearsals after the actual updating do not differ substantially between switch and repetition updates. 

Moving objects in and out of the focus of attention in WM

Our approach to decomposing the switch cost suggests that there are multiple contributing factors to the overall RT cost. This raises the question of how to specifically address a key prediction from single item capacity FoA models (e.g., Oberauer, 2002, 2009), namely, whether both objects can be accessed equally at any given moment in time or whether there is a cost of bringing a different item into the focus of attention. To examine this, we compared fixation durations preceding within-object saccades and between-object saccades. We observed increased fixation times preceding between-object saccades, indicating a cost associated with selecting and orienting attention towards a new object and providing further evidence for a single-item FoA (Oberauer, 2002, 2009). Rather than participants switching their attention between objects when prompted by cue onset, we were able to identify participants' self-initiated shifts of attention between objects. We acknowledge that a comparison of between-object saccades as compared to within-object saccades assumes comparable processes within the categories (for example, we equate a passive-to-old saccade with a new-to-passive saccade, though these may reflect different phenomena). However, the within-object category is notable for its substantial contribution of old-to-new saccades (as indicated in Figure 6, participants rarely return to the old position after arriving at the new). These transitions presumably reflect some aspect of the updating process itself, where the new location is computed from the old location, in itself a potentially time-consuming process. 

A key part of our interpretation here is that with the need to disengage from one object and shift attention to the to-be-updated object switch updates entail an additional process over repetitions. This process insertion has predictable consequences for the RT distribution, primarily manifesting in a shift in the mu component (Hockley, 1984). Our ex-Gaussian modeling of the RT distributions showed that the object-switch cost are reflected in all three components mu, sigma, and tau, but are more pronounced in the rightwards shift in the Gaussian component (mu) as well as an extended tail of the distribution (tau). This replicates the findings of Oberauer (2006), who described effects of switching on both components. We assume that the effect of switching on mu reflects the insertion of an additional cognitive/motor process, such as the shift of the FoA from one item to another. This manifests in initial saccades to the item previously held in WM on switch updates as well as longer fixations preceding saccades between objects. A similar interpretation of mu effects has been given in serial scanning models of memory retrieval (Hockley, 1984; McElree, 1998; McElree & Dosher, 1993; Ratcliff & Murdock, 1976) and in visual search (McElree and Carrasco, 1999). 

However, the cost related to the actual orienting from one object to the other in switch as compared to repetition updates seems to account for only a proportion of the entire switch cost. 

Executive attention: Engagement decision time

In addition to the effect of mu, we also observed a strong effect on the tau component of the ex-Gaussian distribution. Further, when analyzing the subset of updates on which participants made a saccade to the new location (which we take as the point at which the target of the update has been identified), participants still took longer from the point of arrival to this location to respond on switch updates compared to repetitions. 

One interpretation of these effects is that switching introduces more uncertainty into the updating process; if so, then engaging with a new target and/or reaching a threshold for response would take longer after an attention switch. Support for such a suggestion comes from models of RT distributions such as the linear ballistic accumulator model (Brown and Heathcote, 2008), in which slower evidence accumulation (as reflected in a lower drift rate) translates in an extended tail of the RT distribution. Further, this suggestion of increased uncertainty is supported by simulation data reported by Matzke and Wagenmakers (2009), in which drift rate was shown to primarily impact the tau parameter with a smaller effect shown for sigma, both of which we see in our data. Alternatively, it has been suggested that increases in tau are consistent with factors that influence only a subset of trials (e.g., McElree, 1998; Ratcliff & Murdock, 1976). It could be that updates in which participants initially moved their eyes to the passive or random locations (which are comparably infrequent) contribute to this extended tail. It is important to note caution with regards to associating the components of the ex-Gaussian distribution with specific cognitive processes, as manipulations to parameters of models of cognitive processes (e.g., the diffusion model in a two-choice task) can effect multiple components. Nevertheless, the shifts in the components we report are consistent with our broader account, whereas their absence would not be. To clarify the relationship between task behavior, eye movements, and the components of the RT distribution, future work would have to pursue computational modeling of this task to assess the impact of manipulating these factors independently. 

Relationship to other forms of attention switching

Our task incorporates two of the five types of attention switching characterized in the previous WM literature (Wager, Jonides, and Reading, 2004), namely object and spatial switching. Whereas we discuss switching attention in respect to the Posner attentional framework developed for spatial attention, the locations in our task are bound to discrete objects/stimuli (with their according object characteristics). Consequently, some aspects of the processes we discuss for our task (e.g., disengagement) are not necessarily spatial in nature but intrinsically contain object switching components. In contrast to our task, many of the previous examinations of switch costs in WM were based on counting or n-back tasks (e.g., Garavan, 1998; Oberauer, 2006) without any (or a controlled) spatial component. Despite this difference in spatial and object contributions to the switch cost, it is tempting to speculate that the same or analogous processes of disengagement from elements and resolution of interference apply. Indeed, Oberauer (2006) observed switch costs of a comparable magnitude to our two-object task in an n-back task as well as corresponding effects on the ex-Gaussian distribution. 

The three remaining attention switching categories outlined by Wager et al. (2004) (switching between features within an object, switching a response rule, and task switching) are also likely to share some of the components highlighted above. Indeed, a recent task switching study, utilizing eye tracking (Longman, Lavric, & Monsell, in press), has shown already that when the tasks are spatially separated and participants are given short preparation times, participants were more likely to initially orient to the location of the old task on switch trials (comparable to our effect shown for the passive location on first saccades). 

In the context of working memory models, different (though analogous) systems have been proposed to underlie task switching and object switching, with the former mediated by declarative WM and the latter by procedural WM (Oberauer, 2009). However, as illustrated by Longman et al. (in press) and elsewhere (Meiran, Chorev, & Sapir, 2000; Rubinstein, Meyer, and Evans, 2001) and supported by our data, a greater consideration of the overlap between the components involved across attention switching tasks would be beneficial. 

The relationship between selection in WM and perceptual attention

If one accepts the above dissection of the object-switch costs in WM into separate components within Posner's attentional framework (Posner, 2008; Posner & Petersen, 1990; Posner & Rothbart, 2007), one can then speculate on the underlying neural mechanisms along the same lines—not only for response uncertainty and shifting attention between objects, but also for shifting spatial attention in WM in general by drawing upon the notion that selective attention is competitive, a key component of visual attention models (Desimone & Duncan, 1995), eye movements (Zelinsky, 2008), and shared spatial maps underlying perception and WM (Theeuwes, Belopolsky, & Olivers, 2009). Indeed, Theeuwes et al.'s (2009, figure 5) framework to account for inhibitory effects in saccades (e.g., inhibition of return and saccade curvature), consisting of a pre-oculomotor attention map, shared by perceptual attention and WM, a saccade map, and an inhibitory control system provides an excellent starting point. In this framework, changes in activity in the pre-oculomotor attentional map feed into the saccade map, guiding eye movements to task-relevant items. The control system can concurrently inhibit items either at the attentional map level or in the saccade map. Similarly, in our conception of a WM system shifts in the FoA moderate activity in the attention map: At the beginning of each update n, the location reached by the preceding update is likely to have the highest activation—this is the new location of the preceding update n − 1, which attracted the most saccades. When the present update n is a repeat update, that location is the old location of the object to be shifted. Shifting the object would involve shifting the peak of activation in the attention map to the object's new location (new location). Consequently, both the old location and the new location should have high activation peaks, thus attracting saccades toward them. Switch updates, in contrast, start out with an activation peak at the location of the now passive object (which was the new location on update n − 1). This peak of activation needs to be shifted first to the old location of the new object—thereby switching the focus of attention to the new object—and then to its new location. Thus, during switch updates, three locations compete for saccades, namely the passive location, the old location, and the new location. Consequently, we observe more initial saccades to the passive location for switch as compared to repeat trials. It is this additional shift in priority that we would then associate with the mu component of the RT distribution, and the noise to be resolved created by a more distributed pattern of activity in switch updates that we would attribute to the effect on tau. Such an interpretation of our data entails that switch costs in WM updating tasks can be decomposed into three factors. First, competition in the attentional activation map attracts the eyes to update-irrelevant locations and thus increases the number of eye movements for a subset of trials. Second, an initial object disengagement and reorientation process is required to target the new item. Third, uncertainty is higher for switch trials, requiring a longer dwell time after the object has already been updated to give a response. Whilst the overall switch cost is a combination of these, only the second of these three factors appears to constitute the actual moving of objects in and out of the focus of attention in working memory. 

Conclusions

The aim of this work was to pursue the use of eye tracking as a tool to characterize the mechanisms underlying orientation to internal representations in a spatial-updating task. We do not attribute any particular functional role to the eye movements themselves in regard to these findings but understand them as an epiphenomenon; rather, we treat the observed eye movements as a window to the underlying processes (which we also assume to apply in the absence of eye movements). We show that predictions derived from models of perceptual attention can account for selective attention in WM and that patterns in eye movement data closely reflect the effects predicted by these models. This correspondence lends support to recent proposals in the literature that emphasize a close link between perceptual and working memory-related attention systems (e.g., Awh, Vogel, & Oh, 2006; Nobre et al., 2004; Olivers, Peters, Houtkamp, & Roelfsema, 2011; Theeuwes et al., 2009) and that they constitute overlapping, or analogous, mechanisms. 

Acknowledgments

Craig Hedge was supported by a Ph.D. studentship from the Biotechnology and Biological Sciences Research Council, UK. The authors thank Professor Klaus Oberauer (University of Zurich, Switzerland) for discussions and comments on an earlier version of this manuscript. 

Commercial relationships: none. 

Corresponding author: Craig Hedge. 

Email: [e-mail]. 

Address: School of Experimental Psychology, University of Bristol, Bristol, U.K.