Subtle eye movement metrics reveal task-relevant representations prior to visual search

Abstract

Visual search is thought to be guided by an active visual working memory (VWM) representation of the task-relevant features, referred to as the search template. In three experiments using a probe technique, we investigated which eye movement metrics reveal which search template is activated prior to the search, and distinguish it from future relevant or no longer relevant VWM content. Participants memorized a target color for a subsequent search task, while being instructed to keep central fixation. Before the search display appeared, we briefly presented two task-irrelevant colored probe stimuli to the left and right from fixation, one of which could match the current target template. In all three experiments, participants made both more and larger eye movements towards the probe matching the target color. The bias was predominantly expressed in microsaccades, 100–250 ms after probe onset. Experiment 2 used a retro-cue technique to show that these metrics distinguish between relevant and dropped representations. Finally, Experiment 3 used a sequential task paradigm, and showed that the same metrics also distinguish between current and prospective search templates. Taken together, we show how subtle eye movements track task-relevant representations for selective attention prior to visual search.

Introduction

One of the most important features of human vision is that it is selective and adaptive. It flexibly samples the environment on the basis of what is relevant to our current goal. In order to do so, the brain has to maintain some representation of what we are currently looking for. This representation, referred to as attentional template or search template, is thought to be activated in visual working memory (VWM; Carlisle, Arita, Pardo, & Woodman, 2011; Desimone & Duncan, 1995; Duncan & Humphreys, 1989; Hollingworth, Matsukura, & Luck, 2013; Mannan, Kennard, Potter, Pan, & Soto, 2010; Wong & Peterson, 2011), from which it guides selection of matching stimuli in the outside world. 

Although the activation of the search template can be inferred from the guidance occurring during search itself (Chen & Zelinsky, 2006; Wolfe, 1994; Wu & Remington, 2003), researchers have been looking for ways of investigating the template prior to search, when the presumably active VWM representation can be investigated without interference from the search process. One classic demonstration here has been the contingent attentional capture effect demonstrated by Folk, Remington, and Johnston (1992). In various versions of their paradigm, participants look for a target that can appear in one of four different locations. The target is defined by a specific feature, for example the color red. The crucial manipulation occurs before the target display, when an irrelevant cue is presented that either appears at the target location (valid cues), or at any of the other locations (invalid cues). Importantly, the cue may carry the target feature (in this case also red), or a different feature (e.g., green). The typical finding is that although cues are not predictive of the target location, valid cues lead to faster search response times (RTs) than invalid cues, but only when the cue carries the task-relevant color. In other words, attention is inadvertently guided towards objects that match the current search template. Moreover, this representation must already have been active prior to the search display, given the fact that the cues were presented beforehand. 

Nevertheless, as a measure to investigate the activation of task-relevant representations prior to search, the classic contingent capture paradigm also has its drawbacks. First, at a methodological level, the cue validity effect is a rather indirect measure that is reflected in the manual response to the search display, rather than prior to the search. The search RT reflects the final outcome of a number of processing steps that could be problematic for interpretation. For example, cue validity effects may also partly stem from priming (Belopolsky, Schreij, & Theeuwes, 2010) or postselection processes (Adamo, Pun, & Ferber, 2010). Furthermore, to get a reliable cueing effect on the search itself, the cues are typically presented just prior to the time of search display onset, which begs the question whether observers were attending the cue because they were expecting to have to search for the same color at that same moment. The question thus remains how task-relevance affects attentional bias during search template maintenance, before the actual task takes place. 

Second, at a conceptual level, the contingent capture literature so far has made a distinction between stimuli that are task-relevant versus stimuli that are not, and thus they are either represented by VWM or are not represented at all. It has not made a distinction between memory representations that are currently task-relevant versus memory representations that are not. Note that working memory is not only used to guide current tasks, but also to maintain future goals. Real life activities more often than not consist of sequences of tasks, with multiple consecutive goals. Yet so far, little is known on the effect of prospective memories in visual attention tasks. Note that in many studies, including the standard contingent capture task, task-relevance is being confounded with memory presence, in that the search target is not only the relevant item, but also the only item that observers need to remember in the first place. Contingent capture effects may thus be driven by VWM maintenance per se, rather than the direct relevance for search (e.g., Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005; however, see Downing & Dodds, 2004; Houtkamp & Roelfsema, 2006; Woodman & Luck, 2007). 

To tackle these issues, the present study investigated if subtle, inadvertent eye movements while observers are trying to fixate can provide a useful measure to online track the currently active attentional template, during the VWM maintenance period prior to the search. Given the rapidity with which the eyes respond, and the close coupling with visual attention (Deubel & Schneider, 1996), the eyes arguably provide a more direct and cleaner measure of current task priorities. Capture of eye movements by task-relevant features has been observed during search (Schreij, Los, Theeuwes, Enns, & Olivers, 2014; Wu & Remington, 2003). Moreover, studies have demonstrated that VWM content can bias simple orienting saccades, including in cases when doing so is counterproductive (Hollingworth et al., 2013; Mannan et al., 2010; Olivers et al., 2006; Soto et al., 2005; Wong & Peterson, 2011). However, in all these studies this bias was observed during the target selection task, when observers were instructed or required to make eye movements. Here we were interested in biases prior to selection, while observers try to maintain fixation. Moreover, none of these earlier studies directly compared the effects of current and prospective search goals on eye movements. 

A measure we were particularly interested in was the directionality of microsaccades. Microsaccades are small, saccade-like movements with amplitudes < 1° visual angle that occur during attempted ocular fixation and are thus implicit (Engbert & Mergenthaler, 2006). There has been debate on whether microsaccades are penetrable by top-down mechanisms such as attention (Rolfs, 2009). However, attentional biases have been observed in microsaccades between 200–400 ms following centrally presented symbolic cues indicating the task-relevant location (Engbert & Kliegl, 2003; Hafed & Clark, 2002; Laubrock, Kliegl, Rolfs, & Engbert, 2010; Meyberg, Werkle-Bergner, Sommer, & Dimigen, 2015), suggesting that microsaccades could provide an index of covert attention. Recently, it was shown that even spontaneous microsaccades seem to reflect such shifts in covert attention (Hafed, 2013; Yuval-Greenberg, Merriam, & Heeger, 2014). Whereas these previous studies have shown that microsaccades are sensitive to spatial attention, it is unclear whether they are also sensitive to feature-based attention mechanisms, when selection is driven by task-relevant features rather than cued locations. In our setup, visual stimulation was always spatially symmetrical (i.e., bilateral), and the only difference between the left and the right probe was that one of them could match the color of the current target template. An effect of such color matches on microsaccades would demonstrate that microsaccades are not only sensitive to stimulus-driven and spatial attention, but also to top-down feature relevance. 

Here we assessed a number of eye movement metrics to investigate the activation of attentional templates well in advance of search, when only VWM representations are active. To this end, we asked observers on each trial to remember a target color for a subsequent search task. The core procedure is illustrated in Figure 1A. During the delay period between the memorandum and the search display, we presented observers with a task-irrelevant probe display consisting of two laterally presented colored disks, one of which could match the search target. Importantly, we instructed our participants to maintain fixation until the search display and ignore the probe. Nevertheless, our prediction was that probes matching current search templates would capture attention, with subtle, yet measurable consequences for the eye movement system, even though the location of the matching color probe was irrelevant for the upcoming visual search task. Experiments 1 and 2 demonstrate the basic principle and show that template-matching probes attract more and larger eye movements than nonmatching probes. In Experiment 3 we then use our eye movement measures to dissociate current relevance from prospective relevance. To this end, in Experiment 3 observers always had to remember two colors, one for the immediately upcoming search (the current search template), and one for a subsequent second search (the prospective search template). The prediction was that even though both colors had to be remembered, eye movements would only be affected by current relevance, and not future relevance, consistent with the idea that VWM can maintain multiple items, but can at the same time assign different levels of priority (Olivers, Peters, Houtkamp, & Roelfsema, 2011). This would truly demonstrate that contingent capture is driven by current task relevance and not by memory maintenance per se. 

Figure 1

 

Visual search tasks (A) Task design of Experiment 1. We used different lines instead of colors for display purposes only. Participants had to remember the color of the item presented at the start of the trial, the search template. In between the search template presentation and visual search display, we briefly presented a probe. We used different probe conditions in which the template-matching color could reappear left (Match Left) or right (Match Right) of fixation or not (Irrelevant). Participants were instructed to ignore the probe and to make a saccade as fast as possible towards the target in the search display. (B) The task design of Experiment 2. The task was very similar to the task in Experiment 1, except we now presented two disks to remember at the start of the trial, followed by a cue that indicated which disk was to be searched for, while the other could be dropped from memory. (C) The task design of Experiment 3. Participants now had to remember both colored disks, and search for them consecutively: the current search template and prospective search template, respectively. As a result, we added two probe conditions: one condition containing both the current and prospective color (Both) and a condition with the prospective and an irrelevant color (Prospective). Depicted are only the probe conditions with the “relevant” item (current and prospective search template) presented left of fixation, but in the experiment these could also be presented on the right of fixation.

Methods

Participants

We tested 20 participants (nine males, 11 females; M = 24.7 years of age, SD = 2.98) in Experiment 1, 20 participants (six males, 14 females; M = 23.1 years of age, SD = 3.11) in Experiment 2 and 28 participants (nine males, 19 females; M = 25.4 years of age, SD = 2.56) in Experiment 3. Sample size was larger for Experiment 3 as the number of conditions was larger with fewer trials per condition. All participants were recruited at the Vrije Universiteit. They all had normal or corrected-to-normal vision, normal color perception, and gave written informed consent. The protocol was approved by the Scientific and Ethical Review Board of the Faculty of Behavioral and Movement Sciences of the Vrije Universiteit. 

Stimuli, design, and procedure

Experiment 1

Figure 1A shows the task design for Experiment 1. Each trial started with a fixation dot (1000 ms), followed by the memory item that consisted of a centrally presented colored disk (1.1° visual angle) which remained on the screen for 1000 ms. Participants had to remember the color of this disk for a subsequent visual search. We will refer to this memory item as the search template. The color of the search template was randomly selected out of six colors (RGB and luminance): red (200,0,0 and 13 cd/m²), green (0,140,0 and 14 cd/m²), blue (0,90,255 and 15 cd/m²), pink (170,68,131 and 14 cd/m²), purple (160,40,230 and 16 cd/m²), and orange (178,88,0 and 15 cd/m²). Crucially, after a temporal jitter of 1500–1900 ms (steps of 100 ms), we briefly presented a probe (150 ms). 

The probe display contained two colored disks positioned left and right (3.3° visual angle) from the center of the display. The disks had the same size as the search template. The critical manipulation was that one of the two disks could have the same color as the search template, either on the left (“Match Left”) or on the right (“Match Right”). The other color was an irrelevant color randomly selected from the remaining colors (“No Match”). As a control condition, the probe could also contain two irrelevant colors that were not presented at the start of the trial. This third condition will be referred to as the Irrelevant probe condition. The probe conditions were randomly presented but occurred equally often within blocks. 

After another fixation period of 2500 ms, the search display, which consisted of six differently colored disks positioned in a circle (4.4° visual angle), appeared. Participants had to make an eye movement as fast as possible to the disk that matched the search template. After a correct fixation on the search template (i.e., within 2.2° visual angle around the center of the matching disk and within 800 ms), the search display disappeared and the next trial started. 

Participants were seated 75 cm from a computer screen with their head positioned on a chinrest. Stimuli were presented on a LCD monitor (1680 × 1050 pixels, 120 Hz). Stimulus presentation was controlled with OpenSesame (Mathôt, Schreij, & Theeuwes, 2012).The experiment started with 12 practice trials. Each trial started with a drift correction. The real experiment then consisted of 10 blocks of 30 trials with a duration of approximately six minutes This resulted in 100 trials per probe condition. Participants were allowed to take small breaks in between blocks. In total the experiment took on average 90 minutes. 

We instructed participants to be as fast and accurate as possible, to maintain fixation throughout the trials, and only make an eye movement during the search display. Importantly, we instructed participants to ignore the probe and try to keep fixation during the probe presentation. 

Experiment 2

The task used in Experiment 2 was the same as in Experiment 1, except that we now presented two vertically positioned disks at the start of each trial (see Figure 1B) instead of only one centrally presented disk to ensure no spatial overlap between the search item and the probe disks. The disks were presented for 1000 ms with a visual angle of 2.2° from fixation. After a fixation of 500 ms, a spatial cue was presented, which was an upward or downward pointing arrow (0.2° visual angle). The cue was presented for 500 ms and indicated which of the two disks the participant had to keep in memory for the upcoming search and would thus be the search template. The other, uncued colored disk could be dropped from memory and ignored. We refer to this uncued item as the dropped item. In the probe (150 ms), the search template could reappear together with an irrelevant color or the probe could consist of the dropped item and an irrelevant color. In the latter case, due to a programming error, the dropped item always appeared on the left and the irrelevant color on the right (none of the participants reported to have noticed this). Additionally, we shortened the fixation after the probe, from 2500 ms to 2000 ms and changed the colors to make them more equiluminant. (RGB and Luminance): red (100,0,5 and 4 cd/m²), green (0,80,0 and 6 cd/m²), blue (0,55,80 and 3 cd/m²), pink (80,50,60 and 5 cd/m²), purple (80,0,80 and 3 cd/m²), and orange (80,60,0 and 6 cd/m²). 

The experiment started with 30 practice trials. The experiment consisted of 10 blocks of 30 trials (resulting in 100 trials per probe condition), with a duration of approximately six minutes each, and the participant was allowed to take small breaks in between blocks. Each block started with a drift correction. The overall experiment took on average 75 minutes. We provided similar instructions as in Experiment 1. 

Experiment 3

Figure 1C shows the procedure for Experiment 3, which was the same as for Experiment 2, except for the following: Participants now had to remember and search for both colored disks presented at the start of the trial. The cue now indicated the order in which the colors had to be searched for. The cue pointed to the colored disk that participants had to search for first (i.e., the Current search template). The uncued colored disk then needed to be searched for in the second search display (referred to as the Prospective search template). In order to compare attentional capture by the two types of search templates, we added a probe condition containing both the current and prospective search templates (i.e., Both) and a probe condition containing the prospective search template and an irrelevant color (i.e., Prospective). As in Experiment 1 and Experiment 2, we also included a probe condition with the current search template and an irrelevant color (i.e., Current) and a probe condition containing two irrelevant colors (i.e., Irrelevant). The matching colors could be presented left or right from fixation. We used the same colors as in Experiment 1. 

Participants were seated 75 cm from a computer screen with their heads positioned on a chinrest. The experiment started with 10 practice trials. The experiment consisted of 14 blocks of 35 trials each and took approximately six minutes. All probe conditions occurred five times per block, resulting in 70 trials per probe condition in total. The participant was allowed to take small breaks in between blocks. Each block started with a drift correction. The overall experiment took on average 100 minutes. We applied similar instructions as in Experiment 1. 

Eye movement recordings and analysis for all experiments

Eye movements of the left eye were recorded with an EyeLink 1000 tracker (SR Research, Oakville, ON, Canada), with 1000 Hz temporal and 0.2 spatial resolution. Each experiment started with a nine-point eye-tracker calibration and validation. 

Preprocessing and further analysis were performed with Matlab (Mathworks). First, the data was downsampled to 500 Hz and epoched into trials of −3 s to 2 s around probe presentation. Trials containing a blink between 200 ms prior and 200 ms after the probe presentation were removed (Yuval-Greenberg et al., 2014; Experiment 1, M = 4.5%, SD = 10%; Experiment 2, M = 5.3%, SD = 7.5%; and Experiment 3, M = 14.3%, SD = 15.2%). We based our saccade detection on the algorithm by Engbert and Kliegl (2003) and Engbert and Mergenthaler (2006). Saccades were defined as monocular outliers in 2D velocity space within a window of 500 ms and 1000 ms around probe presentation. Thresholds for saccade detection were set to be six times the SD of the eye movement velocity of all data samples in the epoch, using a median-based estimate of the SD (Yuval-Greenberg, Merriam, & Heeger, 2014). Minimum saccade duration was set to six samples (12 ms). We smoothed the data using a moving average with a five-sample (10 ms) window to suppress noise. The minimum allowed fixation duration between two saccades was set to 25 samples (50 ms); if saccades were closer together, then the largest saccade of the cluster was kept. Saccades with a visual angle larger 5.4° (center of the probe disks) were removed from analyses (over all experiments < 1% of the saccades). We only analyzed the first saccade that occurred > 100 ms after the probe onset. 

To calculate the saccade rate (number per second) per participant per probe condition, we first used a sliding window of 100 ms for a period of −500 ms before and 1000 ms after the probe. Since we did not expect any long lasting effects of the task-irrelevant probe, we did not analyze beyond 1000 ms. This revealed a strong overall modulation, regardless of condition, of the saccadic rate within a time window of 100–500 ms after the probe (see the results of Experiment 1). We took this time window as the basis for subsequent analyses of probe-induced biases. To this end, we calculated the proportion of saccades towards the search template matching color for each probe condition for each individual participant, collapsed across left and right depending on which side the search template was depicted (cf. Engbert & Kliegl, 2003). We decided to use proportion of saccades instead of total number of saccades since the total number of saccades differed quite extensively between participants. This did not qualitatively affect our results. 

Additionally, we calculated, within this time-window, the saccadic distance in visual angle of saccades directed towards the matching side and no matching side for the Match Left and Match Right probe conditions. For the Irrelevant probe condition, the distance was simply computed as the average distance across leftwards and rightwards saccades. We used the saccadic distance to select microsaccades (< 1° visual angle). 

Finally, we more precisely tested the temporal profile of our directionality effects within the chosen 100–500 ms time window. To this end, we assessed when the saccadic rate towards the matching side was higher as compared to the no-match side and the rate in the Irrelevant condition using a sliding window on the saccade rate. We used group-level permutation testing with cluster correction to statistically test across conditions. For every sample point, we computed t values of condition-difference in saccadic rate. These t values were thresholded at a p value of 0.05, yielding clusters of significant differences. Next, the sign of the power values was shuffled across participants in 2,000 iterations. At each iteration, t tests were performed on these shuffled data. We saved the sum of t values within the largest cluster into a distribution of summed cluster t values. This distribution reflected the null-hypothesis of no difference between conditions. Finally, we summed the observed (nonpermuted) cluster t values, and thresholded them using the null-hypothesis distribution (specifically, the percentile that corresponded to the same p value as used for the initial t test; e.g., the 95th percentile for p < 0.05). This approach ensures a correction for multiple comparisons by taking into account clusters of rate modulations that can be expected by chance (Maris & Oostenveld, 2007). 

Results

Experiment 1

Behavioral performance

First, we assessed whether the presentation of the probe had an effect on the behavioral performance. Table 1 shows the mean accuracy scores and median RTs for the search task. A repeated-measures ANOVA with Probe condition (Match Right, Match Left, and Irrelevant) as within-subjects factor. We did not observe an effect of the probe on search performance: Percentage Correct, F(2, 38) = 0.87, p = 0.428; Reaction Time for correct trials, F(2, 38) = 1.33, p = 0.276. We used the behavioral performance to include only correct trials for the eye movement analyses to ensure that participants held the correct color in memory. 

Table 1

 

Mean search accuracy and the median search RTs for Experiment 1 (N = 20).

 

Eye movement results

Time course of the overall rate of saccades:

Next, we calculated the rate of occurrences of saccades around the probe presentation per probe condition using a sliding window of 100 ms, taking only the first saccade into account (see Methods). As can be seen in Figure 2, we observed a clear overall modulation of the saccade rate, marked by an initial drop shortly after the presentation of the probe followed by a steep increase in rate at 300–400 ms, and a return to baseline at around 500 ms. This pattern is typical (e.g., Engbert & Kliegl, 2003). We therefore chose 100–500 ms after probe presentation as our time window of interest. 

Figure 2

 

The time course of the probe-induced saccadic rate. The probe was presented at time zero. Further analyses are based on the indicated time window of 100–500 ms after probe presentation. The thick lines indicate the mean saccade rate, and the shaded areas indicate the standard error (SEs) across subjects.

More saccades towards probes matching the search template:

Figure 3A shows the proportion of saccades to the matching probe. Comparing this proportion to chance (0.5) using a t test revealed that more saccades were made to probes that matched the search template than towards the irrelevant color: Match versus Chance, t(1, 19) = 3.76, p = 0.001. Fifteen out of the 20 participants showed a bias towards matching probes. 

Figure 3

 

Saccade directionality distribution and saccadic distance results of Experiment 1. (A) Proportion of saccades after probe presentation (100–500 ms). After probe presentation, saccades were significantly biased towards template-matching probes. (B) Saccadic distance: after probe presentation for Match, No Match, and Irrelevant conditions: The saccades towards matching probes were significantly larger as compared to the No Match and Irrelevant probe conditions. (C) Saccadic rate for Match, No Match, and Irrelevant conditions as a function of time. The thick lines indicate the mean, and the shaded areas indicate the SEs across subjects. This also shows that more saccades were directed towards the matching side of the probe, but that this was largely confined to an early window of ∼100–250 ms, depending on the exact comparison. Horizontal bars show the time window of significant biases for Match versus No Match, and Match versus Irrelevant, for all saccade types, and for microsaccades only (cluster-corrected). Error bars indicate SEs, ** p < 0.01, *** p < 0.001.

Larger saccades directed towards probes matching the search template:

As an additional measure, we also computed the mean saccadic distance for the saccades made in each probe condition, shown in Figure 3B. A repeated-measures ANOVA with Probe condition (Match, No Match, and Irrelevant) as within-subjects factor revealed a reliable main effect, F(2, 38) = 22.38, p < 0.001. Paired t tests revealed that saccades towards the matching side were significantly larger than towards the nonmatching side: Match versus No Match, t(1, 19) = 4.79, p < 0.001 (18 out of the 20 participants showed this effect), and also larger than towards irrelevant probes when no matching probe was present, Match versus Irrelevant, t(1, 19) = 4.88, p = < 0.001 for 19 out of the 20 participants. There was a hint that saccades towards the nonmatching side when there was a matching probe present were smaller than saccades towards nonmatching probes when no match was present: No Match versus Irrelevant, t(1, 19) = 2.36, p = 0.029 (12 out of the 20 participants), but this did not survive the Bonferroni corrected α level of 0.05/3 = 0.0167. 

Temporal profile of the probe matching effects:

Finally, we wanted to assess more precisely when the probe matching effect occurs, and specifically whether this was reflected in microsaccades (< 1° visual angle). Figure 3C shows the saccadic rate for all saccades that were directed to the matching color (Match), the no matching color (No Match), or the completely irrelevant probe (Irrelevant), as a function of time, using a 100-ms sliding window. Using a cluster permutation test at p < 0.05 cluster-corrected (see Methods), we tested when the saccadic rate differed significantly between probe conditions. As can be seen in Figure 3C, the saccadic rate towards the matching side started to increase reliably relative to the nonmatching side and relative to the irrelevant probe condition in similar time windows of respectively 140–254 ms and 144–250 ms. We then ran the analysis on microsaccades (< 1° visual angle) only. Probe-matching effects were also observed in the microsaccadic rate, in that significantly more microsaccades were directed towards the matching side than to the nonmatching side (96–252 ms after the probe) or to completely irrelevant probes (82–250 ms). 

Taken together, these results indicate that the active search template is reflected in subtle yet reliable eye movement metrics. Despite the fact that observers were required to maintain fixation throughout the delay period, they made significantly more saccades in the direction of task-irrelevant but template-matching probes, compared to nonmatching probes. Moreover, when observers made a saccade, it was also larger in size when it was directed towards the matching color compared to the nonmatching color. Moreover, the time course analysis revealed that the preference for template-matching probes was predominantly present in an early time window of ∼100–250 ms, and specifically in microsaccades. 

Experiment 2

It may be questioned whether the results of Experiment 1 were really related to the search goals of the observer, or were caused by priming due to stimulus repetition (e.g., Bichot & Schall, 2002). To control for this, we ran a second experiment, in which we presented two instead of one colored disks at the start of the trial. A spatial cue presented after the two disks indicated which of the two colors would be the search template (see Figure 1B and Methods section). Both colors could reappear in the probe, but only the cued item was now task-relevant, while the other, uncued color was never searched for and could thus be dropped from memory. We ran the same analyses as in Experiment 1. 

Behavioral performance

Similar to Experiment 1, we found no effect of the probe on behavioral performance of the search task: Percentage Correct, F(2, 38) = 0.92, p = 0.408; Reaction Time for correct trials, F(2, 38) = 0.10, p = 0.815, see Table 2. Again, we used the behavioral performance to include only correct trials for the analyses on the probe to ensure that participants held the correct color in their memory. 

Table 2

 

Mean search accuracy and median search RTs for Experiment 2 (N = 20).

 

Eye movement results

More saccades towards the probes matching the search template:

As in Experiment 1, we confined the analysis to the time window of 100–500 ms on the basis of the overall modulation of saccadic rate (regardless of probe condition). Figure 4A shows the proportion of saccades towards the cued color probe, which, as in Experiment 1, was reliably higher than chance: Match versus Chance, t(1, 19) = 2.96, p = 0.008, with 12 out of 20 participants showing a bias. Thus, more saccades were directed towards the cued color than towards the uncued color when both these colors reappeared in the probe. 

Figure 4

 

Saccade directionality and distance of Experiment 2. (A) Proportion of saccades: Significantly more saccades were directed to the matching color versus the nonmatching color. (B) Saccadic distance: The saccades towards the search template matching color (Match) were significantly larger than towards the nonmatching color (No Match) or when made in the Irrelevant probe condition. (C) Saccadic rate for Match, No Match, and Irrelevant conditions. The thick lines indicate the mean, and the shaded areas indicate the SEs across subjects. More saccades were directed towards the matching side in the probe, especially between ∼150–250 ms after the probe. We also observed a significant difference in rate between Irrelevant and No Match conditions. Horizontal bars show the time window of significant biases for Match versus No Match, and No Match versus Irrelevant, for all saccade types, and for microsaccades only (cluster-corrected). Error bars indicate SEs, ** p < 0.01, *** p < 0.001.

Larger saccades towards probes matching the search template:

Figure 4B shows the saccadic distance for saccades made to different probe types. As in Experiment 1, these were entered in a repeated-measures ANOVA, which revealed a reliable main effect for probe condition, F(2, 38) = 21.71, p < 0.001. Follow-up comparisons showed that saccades directed to the side of the matching probe were larger compared to the nonmatching side: Match versus No Match, t(1, 19) = 4.91, p < 0.001 (19 out of the 20 participants displayed this effect); and when the probe contained two irrelevant colors: Match versus Irrelevant, t(1, 19) = 4.66, p < 0.001, (for 17 out of the 20 participants). The saccades were equal in size for the No match and Irrelevant conditions, t(1, 19) = 0.50, p = 0.626. 

Temporal profile of the probe matching effects:

Experiment 1 suggested that these probe-matching effects follow a specific temporal profile, and occur also when only including microsaccades. To confirm this temporal profile, here too we investigated the saccadic rate for saccades directed to the matching color (Match), the non-matching color (No Match) or the probe containing only irrelevant colors (Irrelevant) as a function of time. As can be seen in Figure 4C, the saccadic rate towards the matching side was reliably increased relative to the nonmatching side in the time-window 154–282 ms, a time window that was similar as was observed in Experiment 1 (∼140–250 ms). Moreover, we observed that the saccadic rate towards the nonmatching side was decreased relative to the irrelevant probe condition in time window 154–248 ms. When only taking into account microsaccades (< 1° visual angle), similar effects were observed, with again significantly more microsaccades being directed towards the matching side than the nonmatching side (156–182 ms), and less towards the nonmatching side compared to the irrelevant probe condition (144–250 ms). 

In sum, the overall results confirm Experiment 1: Based on eye movement metrics, we can derive which memory content is active as the search template. Specifically, template-matching probes attracted both more and larger saccades, even though the probes were task-irrelevant, unpredictable, and observers were instructed to maintain fixation. Experiment 2 further shows that these results cannot be explained through priming, since both relevant and irrelevant memory items were presented prior to the probe. Relevance was only determined after the items had already disappeared, and only the relevant item (search template) modulated eye movement metrics, while a dropped (uncued) item did not. 

Experiment 3

In Experiment 3 we directly compared a currently relevant search template to a prospectively relevant search template, by having participants perform two consecutive search tasks. Note that the prospective item is also held in memory, but it only becomes relevant later, after the first search task. The question then is whether, prior to the first search, the eye movement metrics as reported here are sensitive to this distinction between currently and prospectively relevant memory representations. 

Participants were again asked to remember two colors, followed by a spatial cue. Rather than telling the observer which item to keep (and thus which one to drop), the cue now indicated the order and thus the relative priority of the colors that needed to be searched for in the two consecutive search tasks. Thus, the current search template was the color that needed to be searched for first, while the color that participants searched for in the second search display will be referred to as the prospective search template. Again, we used probes before the first search display to measure attentional capture by the different types of visual working memory items. In order to further investigate the difference between the relevance of the search templates, we added two more probe conditions: a probe conditions containing both the current and prospective search templates (i.e., Both), and a probe condition containing the prospective search template and an irrelevant color (i.e., Prospective). In all probe conditions the search template could be positioned left or right from fixation in the probe. We predicted that participants would make more eye movements towards the current search template as compared to the prospective search template in the probe. 

Behavioral performance

Initially, we wanted to include only trials on which observers were correct on both the first and the second search. However, this resulted in an exclusion of too many trials: Current, M = 40.1%, SD = 16.4%; Both, M = 38.6%, SD = 16.4%; and Prospective, M = 39.5%, SD = 17.9%. Therefore, we decided to include all trials for this experiment. Note that when performing the same analysis on only correct trials, we observed very similar qualitative results. Still, even when the analyses were performed on all trials, there were two participants with zero saccades in some of the probe conditions in the time-window after the probe, and hence we were unable to calculate the saccadic distance for these conditions of these participants. 

We ran a repeated-measures ANOVA with Search task (Search 1 and Search 2) and Probe condition (Current, Both, and Prospective) as within-subjects factors. The interactions between Search task and Probe conditions were not significant for either behavioral performance measure: Percentage Correct, F(2, 54) = 1.55, p = 0.221; and Reaction Time, F(2, 54) = 0.07, p = 0.930). We did observe strong main effects for Search task: Percentage Correct, F(1, 27) = 18.27, p < 0.001; and Reaction Time, F(1, 27) = 16.13, p < 0.001. Participants were more accurate but slower in the first search compared to the second search (see Table 3). A possible explanation for this pattern could be that during the first search participants still had to maintain the representation for the second color (prospective search item) in mind and were therefore slower. Importantly, as in Experiment 1 and Experiment 2, Probe condition had no effect on behavioral performance: Percentage Correct, F(2, 54) = 2.32, p = 0.108; and Reaction Time, F(2, 54) = 0.10, p = 0.906, confirming that the probe remained task-irrelevant for the search task. 

Table 3

 

Mean search accuracy and median search RTs for Experiment 3 (N = 28).

 

Eye movement results

More saccades towards probes matching the current search template

Figure 5A shows the proportion of saccades towards the color matching side as a function of Probe condition (Current, Both, and Prospective). A repeated-measures ANOVA on the proportion of saccades towards the color matching side with Probe condition (Current, Both, and Prospective) as within-subjects factor, did not support a main effect for Probe condition: F(2, 54) = 2.29, p = 0.111. However, as in Experiments 1 and 2, we were most interested in how the proportion of saccades towards the matching probe side compared to chance level (0.5). First, we replicated the results from Experiment 1 and Experiment 2 in that more saccades were made towards the matching probe as compared to the irrelevant color in the Current probe condition, t(1, 27) = 3.89, p < 0.001 (21 out of the 28 participants showed this effect). Importantly, there was also a bias towards the probe matching the current search template when it was paired with prospective search template, in the Both probe condition t(1, 27) = 4.44, p < 0.001 (22 out of the 28 participants showed this effect). When the probe matching the prospective search template was presented together with an irrelevant color, it also attracted more eye movements than expected on the basis of chance: Prospective probe condition, t(1, 27) = 2.12, p = 0.043 (15 out of the 28 participants showed this effect), but less so than the current search template in the Both condition, t(1, 27) = 2.24, p = 0.033 (21 out of the 28 participants showed this effect), although none of these latter p values survive Bonferroni correction. 

Figure 5

 

Saccade directionality and distance of Experiment 3. (A) Proportion of saccades towards the matching side. After probe presentation significantly more saccades were directed towards the current search template as compared to the irrelevant as reflected in > 0.5 proportion to the match in the Current and Both probe conditions. (B) Saccadic distance. In the Current probe condition, saccades towards the current search template color (Match) were significantly larger than saccades towards the irrelevant color (No Match) and slightly larger when compared to the prospective item (Both condition). The prospective color had no effect on the saccadic distance when paired with an irrelevant color (Prospective condition). (C) Saccadic rate for Match, No Match, and Irrelevant conditions. The thick lines indicate the mean, and the shaded areas indicate the SEs across subjects. The matching side in the probe attracted more saccades and microsaccades than the No matching side, in a time window of approximately 150 to 250 ms after the probe as reflected by the horizontal bars (cluster-corrected). Error bars indicate SEs, *** p < 0.001, ** p < 0.01.

Larger saccades towards probes matching the search template:

Figure 5B shows a similar pattern for the distance of the directed saccades. There were two participants with zero saccades in some of the probe conditions and hence we were unable to calculate the saccadic distance for these conditions for these participants. We compared the saccadic distance of saccades directed towards the matching position versus the no match position, using a repeated-measures ANOVA with Probe condition (Current, Both, and Prospective) and Match/No Match as within-subjects factors. There was a main effect for Probe condition, F(2, 50) = 5.60, p = 0.006, and a main effect for Match/No match, F(1, 25) = 12.99, p = 0.001. We also observed a significant linear interaction for Probe condition × Match/No Match: F(1, 25) = 9.98, p = 0.004. The paired t tests revealed that saccades were larger towards the color of the current search template, when the current search template color was paired with the irrelevant color in the probe: Current, t(1, 26) = 4.20, p < .001 (23 out of the 28 participants showed this effect); and also when paired with the prospective search template color: Both, t(1, 26) = 2.40, p = 0.025 (25 out of the 28 participants demonstrated this effect), although the latter did not survive Bonferonni correction (p = 0.0083). In contrast, the probe condition in which the prospective search template color was paired with the irrelevant color did not show a difference in saccadic distance for saccades directed towards the prospective or irrelevant color: Prospective, t(1, 27) = 0.04, p = 0.970. Moreover, when directly comparing the probe conditions with each other, the saccades towards the prospective search template (Prospective Match) were smaller than the saccades towards the current search template (Match) in the Current and Both probe conditions: Current Match versus Prospective Match, t(1, 27) = 3.39, p = 0.002 (22 of 28 participants); and Both Match versus Prospective Match, t(1, 27) = 3.45, p = 0.002 (21 of 28 participants), while the saccadic distance towards the current search template in the Current and Both probe conditions did not differ: Current Match versus Both Match, t(1, 27) = 0.39, p = 0.702. Thus, in all, saccadic distance towards the template-matching probes was similar in the Current and Both condition, and larger than in the Prospective condition. 

Temporal profile of the probe matching effects:

Figure 5C reveals the time course of the match effects. As in Experiments 1 and 2, we observed reliable differences between the matching and nonmatching side in the Current probe condition, here between 126–236 ms for all saccades, and between 138–236 ms for the microsaccades (Left panel). Similar effects occurred in the Both condition: Where the current and prospective search templates were presented together, more saccades were directed towards the current search template in the time-windows 162–298 ms and 166–264 ms for respectively all saccades and microsaccades (Middle panel). When the prospective search template was presented in combination with an irrelevant color (Prospective probe condition), no reliable match effects were observed (Right panel). 

In sum, we replicated the results of Experiment 1 and Experiment 2 and observed more and larger eye movements towards the current search template, even when the search template was presented with another item represented in visual working memory, the prospective search template. When the prospective search template color itself was paired with an irrelevant color, such biases were weaker to absent. These results demonstrate that subtle eye movement metrics are useful for dissociating the current search template from prospective memories. Furthermore, the increased rate of (micro)saccades towards the current when paired with the prospective probe implies that the current and prospective template representations stand in direct competition with each other in VWM. 

Discussion

In three experiments we demonstrated that stimuli that match current task-relevant representations (“search templates”) in visual working memory (VWM) capture attention during the delay period prior to the search, as indexed by different eye movement metrics. In all three experiments, we observed more and larger eye movements towards template-matching color probes presented before the visual-search task, despite the fact that observers were instructed to keep central fixation until the search display appeared. Experiment 2 confirmed that this effect was related to active task goals and not due to implicit priming. Finally, Experiment 3 showed that this involuntary attentional capture was only apparent when the color was relevant for the current task, when it needed to be searched for first, even when paired with a color held prospectively in memory, for later use in a second visual-search task. These effects of task relevance emerged relatively early after the probe presentation ∼150–250 ms and were also present when only assessing microsaccades (< 1° visual angle). In sum, these results demonstrate that subtle eye movements during attempted fixation provide a useful measure for uncovering task-relevant memory representations during the VWM delay period prior to search. 

Our results are in line with recent models that state that items held in VWM can have different effects on selective attention depending on current task-relevance or priority (Olivers et al., 2011; Zokaei, Ning, Manohar, Feredoes, & Husain, 2014). More specifically, our results support the idea that only a currently prioritized feature captures attention very strongly whereas an item that is important for later on, referred to here as a prospective search template, does not, or at least to a much lesser extent. The prospectively held color did attract slightly more but not larger eye movements when probed together with an irrelevant color but did not survive our multiple comparison corrections. This pattern is reminiscent of an EEG study by Peters, Goebel, & Roelfsema (2009). They presented a prospective memory item in a stream of distractors, and found that the ERP elicited by the prospective memory item could not be distinguished from the ERPs elicited by the other distractor items. In contrast, the search template did inflict a larger ERP. However, it remains elusive how this division of items in VWM is established in the brain and how relevance influences saccade generation, especially since neurons in frontal eye fields (FEF; Stanton, Bruce, & Goldberg, 1995) and superior colliculus (SC; Fries, 1984), regions that are important in generating eye movements, are not themselves color selective (the target-defining feature used here). Therefore, the observed color-memory effects of the template on directionality and saccadic distance were likely generated in color-sensitive sensory systems that project to SC and FEF (White, Boehnke, Marino, Itti, & Munoz, 2009). As we failed to observe strong biases towards the prospective color in the probe, the prospective color might either be represented differently within these color-sensitive regions, be represented in the same way but without projecting to eye movement centers, or be represented elsewhere entirely. It has been suggested that the current item is maintained via sustained neuronal firing in feature-specific sensory populations whereas prospective items might be maintained by sustained firing in other, nonsensory regions (Goldman-Rakic, 1995). Or the prospective item might be temporarily stored via altered patterns of synaptic weights (Mongillo, Barak, & Tsodyks, 2008), which would mean that the prospective memory items are stored in a more passive state, analogous to storage of information in long term-memory. This latter idea is in line with recent fMRI studies that passive memory representations can be reactivated depending on task demands. For example, in a study by Lewis-Peacock, Drysdale, Oberauer, and Postle (2012), it was demonstrated that when multiple items were maintained in memory, the (temporarily) irrelevant item (similar to the prospective search template here) could not be decoded, until it was cued to become relevant to the task again. In contrast, only the item that was within the focus of attention (similar to the current search template here) could be successfully decoded through patterns of voxel activity. Similarly, another fMRI study, by Peters, Roelfsema, and Goebel (2012), suggests that the two different states influence activity in extrastriate visual cortex in opposite directions: Whereas the prospective search template in memory suppressed activity, the current search template in WM enhanced processing of matching visual input. Moreover, Zokaei, Manohar, Husain, and Feredoes (2013) used TMS on motion-sensitive area MT+, and showed that working-memory precision of motion direction was distorted with high-intensity TMS on MT+, but crucially only for the currently prioritized (cued) item, not for uncued memory items. 

At the same time, a number of studies have shown that VWM content can bias even simple orienting saccades, when doing so is either irrelevant to, or even against task goals (Hollingworth et al., 2013; Mannan et al., 2010; Mathôt, Van Heusden, & Van der Stigchel, 2015; Olivers et al., 2006; Soto et al., 2005; Wong & Peterson, 2011). In these tasks observers remembered a color for a prospective memory test, while performing an eye movement-related task in between, in which either that target or a distractor could carry the memorized color. At first sight this may appear to go against our evidence presented here that not-yet-relevant representations guided eye movements to matching stimuli to a much lesser extent. However, first of all in these earlier studies, eye movement biases were measured during the visual search or orienting task itself, when participants were instructed to explicitly make eye movements. In our setup, we measured before the search task, during the VWM delay period. Second, whether or not some biases are found for prospective memories may reflect a graded difference. These earlier studies did not dissociate between current and prospective relevance and their effect on eye movements, which makes it difficult to compare results. Consistent with these earlier findings though, Experiment 3 provided some evidence suggesting that the prospective memory representation affects selection but only in proportion of saccades not in saccadic distance, and not when placed in direct competition with the current template. Moreover, we now only probed before the first search, not before the second search. It is likely that when the prospective item becomes the search template, as is the case for the second search, matching probes before the second search will start to capture attention. Further research could test this switch from being prospectively relevant to being currently relevant, as well as relating biases in saccade direction with visual search performance (White & Rolfs, 2016). 

Note that most of the saccades were smaller than 1° of visual angle. These microsaccades are considered to be involuntarily and implicit. Interestingly, multiple studies have provided evidence that microsaccades can reflect the allocation of covert attention (Engbert & Kliegl, 2003; Laubrock et al., 2010). Specifically, these studies demonstrated that the presentation of a central spatial cue influenced the rate and directionality of microsaccades. We observed effects of the current search template-matching item in the probe on the directionality of microsaccades. This suggests that microsaccades are also sensitive to feature-based selection mechanisms. Recently it was demonstrated that microsaccade direction was sensitive to subjective value and predictive for the upcoming voluntary choice (Yu et al., 2016), as well as for symmetry in an image (Meso, Montagnini, Bell, & Masson, 2016). Meso and colleagues showed that the direction of small saccades (in their case < 2°) was predominantly in the direction parallel to the axis of symmetry. Such findings also indicate that microsaccade directionality is sensitive to other than merely spatial signals. There was however a notable difference between our findings and those of Engbert and Kliegl and colleagues when it comes to the timing of the microsaccades. Whereas Engbert and Kliegl found the microsaccade rate to peak between 300 and 400 ms after cue onset, we found a probe-induced bias that occurs substantially earlier, between 150 and 250 ms post probe onset. A likely explanation for this time difference is the fact that earlier studies used centrally presented symbolic cues to direct attention, which are likely to be processed more slowly than the lateralized, attention-capturing probes used here. 

Conclusions

Taken together, in three experiments we have shown that eye movements provide useful metrics reflecting the activation of task-relevant representations in VWM even when observers try to maintain fixation. This was reflected in more and larger eye movements towards memory-matching color probes during the delay period, although eye movements by and large stayed within the microsaccade range. Specifically, we show that such eye movements distinguish between currently relevant and no longer relevant memories, as well as between currently relevant and future relevant memories. 

Acknowledgments

This work was supported by ERC Consolidator Grant 2013-CoG-615423 from the European Research Council to C. N. L. O. We thank the volunteers for their participation. 

Commercial relationships: none. 

Corresponding author: Anouk M. van Loon. 

Email: [email protected]. 

Address: Department of Experimental and Applied Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.