The dynamics of attentional sampling during visual search revealed by Fourier analysis of periodic noise interference

Abstract

Abstract

Abstract:

Abstract  What are the temporal dynamics of perceptual sampling during visual search tasks, and how do they differ between a difficult (or inefficient) and an easy (or efficient) task? Does attention focus intermittently on the stimuli, or are the stimuli processed continuously over time? We addressed these questions by way of a new paradigm using periodic fluctuations of stimulus information during a difficult (color-orientation conjunction) and an easy (+ among Ls) search task. On each stimulus, we applied a dynamic visual noise that oscillated at a given frequency (2–20 Hz, 2-Hz steps) and phase (four cardinal phase angles) for 500 ms. We estimated the dynamics of attentional sampling by computing an inverse Fourier transform on subjects' d-primes. In both tasks, the sampling function presented a significant peak at 2 Hz; we showed that this peak could be explained by nonperiodic search strategies such as increased sensitivity to stimulus onset and offset. Specifically in the difficult task, however, a second, higher-frequency peak was observed at 9 to 10 Hz, with a similar phase for all subjects; this isolated frequency component necessarily entails oscillatory attentional dynamics. In a second experiment, we presented difficult search arrays with dynamic noise that was modulated by the previously obtained grand-average attention sampling function or by its converse function (in both cases omitting the 2 Hz component to focus on genuine oscillatory dynamics). We verified that performance was higher in the latter than in the former case, even for subjects who had not participated in the first experiment. This study supports the idea of a periodic sampling of attention during a difficult search task. Although further experiments will be needed to extend these findings to other search tasks, the present report validates the usefulness of this novel paradigm for measuring the temporal dynamics of attention.

Introduction

Visual search tasks can be classified according to their level of difficulty. Some are “easy,” characterized by near-zero slopes when measuring reaction time as a function of set size and involve preattentive processes, whereas others are more “difficult,” characterized by positive slopes and specifically involve attention (Treisman & Gelade, 1980; Wolfe, Cave, & Franzel, 1989). Nowadays, one of the main debated questions concerning difficult visual search is whether attention focuses sequentially on the stimuli (Treisman & Gelade, 1980; Wolfe, 1998; Wolfe et al., 1989), acting as a “spotlight” that switches from one stimulus (or group of stimuli) to another (Vanrullen, Carlson, & Cavanagh, 2007), or whether it processes them all at the same time in a continuous or parallel manner (Eckstein, Thomas, Palmer, & Shimozaki, 2000; Palmer, Ames, & Lindsey, 1993). The first hypothesis—but a priori not the second—should predict that difficult visual search involves a periodic temporal dynamic of attentional sampling (Vanrullen & Dubois, 2011). 

Over the past decades, interest in the potential implication of periodic processes in perceptual capabilities has grown steadily. There are different lines of evidence in favor of the idea of “discrete perception” (Stroud, 1956; VanRullen & Koch, 2003), whereby perceptual experience would build on a series of rapidly taken discrete snapshots, giving us a false impression of continuity. Recent studies demonstrated that attention can drive this periodic dynamic (Busch & Vanrullen, 2010; Vanrullen et al., 2007; Vanrullen, Reddy, & Koch, 2005). This suggests that attention could also employ such periodic dynamics during difficult attentional search tasks, as predicted by sequential models of visual search (Treisman & Gelade, 1980; Wolfe, 1994; Wolfe et al., 1989). 

In this study, we introduce a novel technique aimed at measuring the temporal dynamics of attentional sampling during visual search tasks, and we demonstrate its practical application to two standard search tasks: one easy search (L vs. +) and one difficult search (color-orientation conjunction). We used an oscillatory modulation of stimulus information by dynamic visual noise and performed a complex decomposition analysis using Fourier series, similar to a recent study by Gobell and collaborators (Gobell, Tseng, & Sperling, 2004), but in the time domain instead of space. We reasoned that the presence of a sampling periodicity at a particular frequency would support the sequential model of attention deployment and help constrain the underlying neural mechanisms. On the other hand, an absence of sampling periodicity would favor the parallel model of attention. We found a periodic sampling behavior occurring in the difficult task at ∼10 Hz but no corresponding periodicity in the easy search task. Such a periodic attentional sampling naturally supports the sequential model of attention, although it can be a posteriori reconciled with the parallel model by postulating an oscillatory modulation of the efficiency of attention over time. We conclude that our new method can allow the characterization of the temporal dynamics of attention during visual search. Future studies will be needed, however, to test if our finding of a ∼10-Hz attentional periodicity can be generalized to other difficult visual search tasks. 

Methods

Subjects

The age of the participants was between 20 and 36 years. Overall, 29 subjects were included in the experiments (15 women). Nineteen subjects participated in Experiment 1 (14 in the difficult task and 14 in the easy task; nine subjects participated in both tasks) consisting of the evaluation of attentional dynamics during visual search. Seven participants from the difficult task in Experiment 1 also performed Experiment 2. This second experiment tested the relevance of the estimated attentional function of the difficult search task. Ten new subjects (who did not participate in Experiment 1) also performed Experiment 2. Three participants were excluded at this stage of the analysis: two had chance-level performance (percentage correct between 50% and 60%), and one had reaction times more than three standard deviations above the group average. 

Stimuli

Subjects were placed 57 cm from the screen, and their heads were maintained using a chinrest and a headrest. Two tasks were performed: an easy (or “pop-out”) and a difficult visual search task. In the easy search, subjects were asked to report the presence or absence of a “+” sign among L distracting letters (2.2° visual angle). Each letter could be presented randomly in four orientations: 0°, 90°, 180°, and 270° from upright. The difficult task was a conjunction task between color and orientation. Subjects reported the presence or absence of a red grating oriented 30° from upright among red gratings oriented 330° from upright and green gratings oriented 30° from upright (and vice versa). The gratings, with a spatial frequency of seven cycles per degree, measured 3° of visual angle. In both tasks, the target was present in half of the trials, pseudo-randomly determined. The search arrays were displayed for 500 ms followed by an empty response screen. In a preliminary experiment with variable set sizes, we determined a specific set size for each subject to achieve approximately 75% correct using this stimulus duration of 500 ms. This fixed set size was then used in the present experiments. During the easy task, we presented 6 (±2.1) elements (average ± standard deviation over subjects), and we presented 5.4 (±2) elements during the difficult task. Subjects who performed the difficult search task in both Experiments 1 and 2 had the same set size in both experiments. 

Experimental procedure

Experimental logic

In a nutshell, our logic was to apply Fourier series analysis using sine- and cosine-modulated stimulus information at different frequencies and measure attentional dynamics using the inverse Fourier transform. 

In mathematical terms, we assume that there exists an attentional sampling function A(t) that we aim to measure over the interval of 0 to 0.5 s. Our working definition of attention entails that stimulus information S(t) at different moments in time will weigh more or less on the perceptual decision as a function of the value A(t) at that time; in other words, the subjects' perception can be approximated (potentially with an additive or multiplicative constant) by a linear combination of stimulus information and attentional sampling:  

 

For a given temporal frequency ω, we call  

 

 

 

P_cosω, P_sinω, P_-cosω, and P_-sinω can be directly estimated by presenting a visual stimulus whose temporal profile is modulated with a cosine, sine, –cosine, or –sine function (respectively) and measuring the resulting perceptual performance (e.g., d-primes). This is the purpose of Experiment 1 (see Figure 1). In theory, some of these measures should be expected to provide redundant results: because P_cosω = −P_-cosω and P_sinω = −P_-sinω, only one measurement for each pair should (in theory) be sufficient. In practice, however, one can imagine that modulating the visual stimulus at a particular frequency could induce performance changes independently of the exact phase (for example, if faster modulation frequencies cause an increase of arousal or vigilance). Therefore, we systematically measured the effect of each phase modulation (cosine, sine) and their inverse (−cosine, −sine) on performance. Any unwanted factor that would equally affect all modulation phases at a given frequency would then be eliminated in the following equations. 

Figure 1

 

Experimental design of Experiment 1. Examples of four presentation conditions (cosine, sine, −cosine, and −sine) at one given frequency (4 Hz) during the easy visual search task. Observers searched for a + among Ls (the letters were presented randomly in four orientations: 0°, 90°, 180°, and 270° from upright). A trial started when subjects pressed the space bar on the keyboard. The stimuli appeared after a random delay (between 1.5 and 2.5 s) and stayed on the screen during 500 ms. On each of the stimuli, a visual noise patch composed by random dots was displayed by transparency, and its opacity fluctuated according to 1 of 40 conditions randomly intermixed: 10 different frequencies (2–20 Hz in 2-Hz steps) and four phases (cosine, sine, −cosine, and −sine). For all of these conditions, there was always the same amount of stimulus information and visual noise; only the temporal ordering of the display frames varied. A similar procedure was applied to a difficult visual search task (color-orientation conjunction task).

Finally, we define the complex coefficient C_ω as C_ω = (P_cosω − P_-cosω)/2 − i. (P_sinω − P_-sinω)/2. In other words,  

 

Fourier series analysis implies that the attentional sampling function should be proportional to  

 

In other words, measuring the complex coefficients C_ω every 2 Hz (from 2 to 20 Hz) will give us an estimate of the attentional function (note that values greater than 20 Hz are not considered here for practical purposes, although of course they may be relevant for attention). 

Experiment 1: Measuring attentional sampling dynamics

This experiment was intended to put into practice the theory developed in the preceding section, in order to validate our new paradigm. Subjects were placed in front of a gray screen and were asked to keep their gaze on a fixation point at the center. When the subjects pressed the space bar on the keyboard, a fixed number of stimuli (determined separately for each subject as explained above) appeared at random but equally spaced positions on a circle at 14° eccentricity, after a random delay between 1.5 and 2.5 s. On each stimulus element, a dynamic visual noise was superimposed by transparency, consisting of a square filled with dots of random luminance; the square was 1.5 times larger than the stimuli. The dynamic visual noise was applied with a temporal modulation function following 1 of 40 possible conditions (Figure 1): 10 different frequencies (2–20 Hz, by steps of 2 Hz) and four different phases (sine, cosine, −sine, and −cosine), randomly interleaved in different trials of the same blocks. When the modulation value was maximal, the noise was opaque and the stimulus was invisible; when the modulation was minimal, the noise was fully transparent and the stimulus was thus unaffected. Note that for all 40 conditions, there was always the same amount of stimulus information and of visual noise over the course of each trial (in other words, only the temporal ordering of the display frames differed between conditions, not the contents of the frames). 

We evaluated the performances of the subjects by computing d-primes. Based on a complex decomposition analysis, we used these d-primes to assess the dynamics of attentional sampling occurring during one visual search trial, for the difficult and the easy tasks (Figure 2). As explained in the preceding section, this method consists in first combining for each frequency the d-primes for sine and −sine phase conditions and (separately) for cosine and −cosine phase conditions and, second, combining the two resulting estimators to obtain a vector in the complex domain. For each subject, we thus obtained 10 complex vectors for the 10 different frequencies. These vectors were considered as Fourier coefficients, defined by their length (i.e., oscillatory amplitude) and pointing angle (i.e., oscillatory phase). The complex coefficients were then used to compute an inverse Fourier transform, thereby estimating the attentional sampling function in the time domain for each of the subjects. For both tasks, we then computed the average estimated attentional function over all subjects. Finally, we analyzed the amplitude spectra of these two estimated sampling functions. We first computed a fast Fourier transform (FFT) on the grand-average attentional functions (i.e., averaged over all subjects) and looked at the obtained amplitude spectrum. We also calculated this amplitude spectrum for each subject based on his or her individual attentional functions and recomputed the average amplitude spectrum. For both analyses, we evaluated the significance of the measured amplitude spectra by using a Monte Carlo procedure. Surrogate data were created for each subject under the null hypothesis that hit rate and false alarm rate are independent of phase and frequency. The complex decomposition analysis was recomputed for each surrogate (n = 10,000), and the amplitude spectra of surrogate attentional functions (either based on grand-average attentional functions or on individual functions) were used to estimate significance. To achieve this, the 10,000 surrogate amplitude spectra were ranked in ascending order, separately for each frequency. The 9,501th, 9,901th, 9,991th, and 10,000th values were considered as the respective limits of four different confidence intervals (95%, 99%, 99.9%, and 99.99%), which are represented with different colors in the background of the four corresponding graphs. An experimentally observed spectral amplitude value was considered significantly different from the corresponding null distribution with p < 0.05 if it exceeded the 95% confidence threshold (and with p < 0.01 above the 99% confidence threshold, and so on). To take into account the possible problem of multiple statistical comparisons across the 10 frequencies used (2–20 Hz), we adopted a conservative statistical threshold of p < 0.001. For both tasks, we also ran the same analysis, discarding the signals from stimulus onset (0–83 ms) and stimulus offset (417–500 ms). This was aimed at verifying that any periodicity highlighted by the previous analysis would not be due merely to an increased sensitivity to the onset and/or the offset of the stimuli. 

Figure 2

 

Estimation of the attentional sampling function by a complex decomposition analysis. For both visual search tasks, for each subject, we measured the performance modulation induced by visual noise across the 10 different frequencies, for the four phase conditions (sine, cosine, −sine, and −cosine). We then combined the performances across the four phases for each frequency in the complex domain. For each subject, we thus obtained 10 complex vectors, characterized by their angle (phase in the complex domain) and their length (amplitude). These vectors were used as Fourier coefficients, on which we finally applied an inverse fast Fourier transform to estimate the dynamics of attentional sampling in the time domain during one visual search trial. All data in this figure are fictitious and for illustration purposes only.

Experiment 2: Testing the validity of estimated attentional functions

Based on the previous estimates of attentional sampling dynamics for the difficult visual search (see Results and Figure 6), we tested the functional significance of the sampling function on subjects' performances. For two groups of subjects, seven who had also performed the difficult search task in Experiment 1 and seven new subjects, we presented the search arrays (color-orientation conjunction) embedded in dynamic noise with signal-to-noise ratios (SNRs) following the grand-average estimated attentional function (i.e., maximal signal when attention is expected to be maximal; this was called the “test” condition) or the opposite signal (i.e., maximal signal when attention is expected to be minimal; this was called the “control” condition). Whereas in the test condition subjects should perform the search task efficiently, in the control condition their performances would be expected to decrease because stimulus information no longer matches their natural sampling dynamics. We equalized the histograms of SNRs across the two conditions to ensure they had comparable total signal energy and high-pass filtered the attentional function, excluding the 2 Hz component, to ensure that our results would not be driven by the 2 Hz component, which could have reflected a nonperiodic sampling strategy (e.g., increased sensitivity for stimulus onset and/or offset, as found in the easy search estimated function). The two presentation conditions (test vs. control) were randomly interleaved during the experiment. We looked at the performances of the subjects in these two conditions by computing d-primes. We compared test versus control conditions using one-tailed t tests, according to the prediction that a search array presented with an SNR following the estimated attentional function should lead to better search performance than the opposite signal. 

Results

Experiment 1

In a preliminary experiment on the same group of subjects (n = 14 for both tasks), we verified that subjects used the appropriate search strategy for the two tasks that we intended to use in Experiment 1: finding a + among Ls or a conjunction of color (red or green) and orientation (30° or 330° from upright). We presented the stimuli for unlimited durations with a set size randomly drawn between four and eight elements and computed the RT × set size slopes. As expected, the slopes were near zero for the L versus + task: 7.7 ms ± 7.9 ms per element, t(13) = 3.1, p = 0.01, for target present, and 11.6 ms ± 6.7 ms per element, t(13) = 5.5, p < 0.01, for target absent—t tests performed under the null hypothesis that the slopes are equal to zero (i.e., the task involved minimal attention). Slopes were strongly positive for the conjunction task: 70 ms ± 7.7 ms per element, t(13) = 25.6, p < 0.0001, for target present, and 184.6 ms ± 27.4 ms per element, t(13) = 19.1, p < 0.0001, for target absent (i.e., the task involved significant attentional resources). 

In the main experiment, we estimated the attentional sampling dynamics for these two visual search tasks. To do so, we modulated stimulus information by applying an oscillatory visual noise (cf. Figure 1) according to 40 different conditions (randomly interleaved): 10 different frequencies (2–20 Hz, 2-Hz steps) and four different phases (sine, cosine, −sine, or −cosine) at each frequency. In both tasks, we measured for each condition (i.e., for a given phase and frequency) the performance of the subjects in detecting the target by computing d-primes. Then, we applied a complex decomposition analysis to estimate the dynamics of attentional sampling during one trial of visual search (cf. Methods section and Figure 2): at each frequency, d-primes for the different phase conditions were combined to obtain a complex vector in the Fourier domain. An inverse Fourier transform allowed us to estimate attentional dynamics in the time domain (see Methods). 

For the easy search, the resulting grand-average attentional function over 14 subjects (Figure 3) presented large fluctuations in d-prime. To determine the relative power and the exact frequency of these effects, we computed an FFT to obtain the amplitude spectrum of the grand-average attentional function (Figure 4A). The first four frequencies (2, 4, 6, and 8 Hz) were significantly greater than chance (p < 10⁻³, a conservative statistical threshold due to multiple comparisons across frequencies). In Figure 4B, the amplitude spectrum was computed first on each individual sampling function and subsequently averaged. In this case, the spectrum presented an overall decreasing shape, significantly different from chance at frequencies of 2, 4, and 6 Hz. The existence of similar low-frequency components (2, 4, 6 Hz) in the amplitude spectra from both the individual and grand-average sampling functions implies that these components not only present increased amplitude for each subject but also are strongly phase locked between subjects. As we will see, these low-frequency components mostly reflect increased sensitivity to the onset and offset of the stimulus sequence. 

Figure 3

 

Attentional sampling function estimated for an easy visual search trial. Subjects (n = 14) performed an easy task consisting in finding a + among Ls. d-primes were measured for the 40 phase and frequency visual noise presentation conditions. Performances are represented in the top panel as a modulation of d-prime with respect to the average computed over all conditions (dashed baseline). On these performance modulations, we applied the complex decomposition analysis (Figure 2). This analysis was performed for each individual subject. Then, the individual estimated attentional sampling functions were grand-averaged over all subjects (bottom panel). The standard error of the mean is represented in gray.

Figure 4

 

Amplitude spectra of the attention function estimated for the easy search. (A) Amplitude spectrum of the grand-average attentional function. Background p values were calculated using a Monte Carlo procedure (see Methods). The amplitude reaches significance (p value <10⁻³) at the four lowest frequencies: 2 Hz, 4 Hz, 6 Hz, and 8 Hz. (B) Average amplitude spectrum of the individual attentional functions. Background p values were calculated using a Monte Carlo procedure (see Methods). The amplitude reaches significance (p value <10⁻³) at the three lowest frequencies: 2 Hz, 4 Hz, and 6 Hz.

The same analysis was performed to reveal the attention sampling function during the difficult search task (Figure 5). This function was characterized by a more restricted range of oscillatory frequencies: a rapid oscillation appeared to be superimposed on a slower fluctuation. In the amplitude spectra of Figure 6, three specific frequency components stand out, one at 2 Hz, one at 10 Hz, and one at 18 to 20 Hz. These peaks were apparent in the amplitude spectrum of the grand-average function (Figure 6A) as well as on the average amplitude spectrum of individual sampling functions (Figure 6B). As previously, we can thus conclude that these peaks were due to both higher amplitude and stronger phase locking between subjects at these frequencies. In other words, this 10-Hz oscillation at a particular phase likely corresponds to a temporal search strategy that is common to most subjects for this particular task. 

Figure 5

 

Attentional sampling function estimated for a difficult visual search trial. Subjects (n = 14) were asked to report the presence or absence of a red grating oriented 30° from upright among red gratings oriented 330° from upright and green gratings oriented 30° from upright (and vice versa). d-primes were measured for the 40 phase and frequency visual noise presentation conditions. Performances are represented in the top panel as a modulation of d-prime with respect to the average computed over all conditions (dashed baseline). On these performance modulations, we applied the complex decomposition analysis (Figure 2). This analysis was performed for each individual subject. Then, the individual estimated attentional sampling functions were grand-averaged over all subjects (bottom panel). The standard error of the mean is represented in gray.

To summarize, Experiment 1 revealed that perceptual sampling during an easy search (L vs. +) engaged mostly low temporal frequency components, whereas a difficult search (color-orientation conjunction) involved three distinct frequency peaks: 2 Hz, 10 Hz, and 18 to 20 Hz. One might wonder whether the slow frequencies, with performance rising and falling on a time scale of several hundred milliseconds, could reflect a voluntary attentional strategy and/or the effect of onset and offset transients. To test the latter possibility, we recomputed amplitude spectra on the central 333 ms of each 500-ms-long sampling functions (i.e., after truncating the first 83 ms after stimulus onset and the last 83 ms before stimulus offset; Figure 7). For the easy search, no significant frequency remained in the amplitude spectrum of the grand-average sampling function. In the average amplitude spectrum of individual sampling functions, the first three frequencies (3–9 Hz) were significantly greater than chance. As explained before, the absence of corresponding effects in the grand-average sampling function implies that these low-frequency components were not phase locked anymore between subjects. In other words, the previously observed low-frequency effects during the easy search task (Figure 4) appeared to be due in large part to the onset and offset portions of the sampling functions, during which all observers systematically displayed increased sensitivity. For the difficult search, both the amplitude spectrum of the average sampling function and the average amplitude spectrum of individual sampling functions displayed a significant peak at 9 Hz. Together, the results in Figure 6 and Figure 7 suggest that the attentional sampling function during this difficult search task fluctuated periodically at ∼9 to 10 Hz, not just at stimulus onset and offset but also during the central portion of stimulus presentation. In addition, we can speculate that the 18 to 20 Hz component observed in Figure 6 and to some extent in Figure 7 could correspond to a harmonic of this 9 to 10 Hz periodicity. In the next experiment, we aimed to demonstrate the functional significance of the periodic sampling observed in the difficult search task. 

Figure 6

 

Amplitude spectra of the attention function estimated for the difficult search. (A) Amplitude spectrum of the grand-average attentional function. Background p values were calculated using a Monte Carlo procedure (see Methods). The amplitude reaches significance (p value <10⁻³) for four frequencies: 2 Hz, 10 Hz, and 18–20 Hz. (B) Average amplitude spectrum of the individual attentional functions. Background p values were calculated using a Monte Carlo procedure (see Methods). The amplitude reaches significance (p value <10⁻³) for six frequencies: 2 Hz, 8–10 Hz, 16–20 Hz.

Figure 7

 

Control for potential onset-offset (nonperiodic) effects. The analysis from Figures 3 to 6 was replicated with estimated attentional functions truncated for the first 83 ms and the last 83 ms (leaving only the central 333 ms portion of the original 500-ms-long functions). (A) For the easy search task, no frequency reached significance in the amplitude spectrum of the grand-average attentional function (middle; note that the frequency resolution of this analysis is 3 Hz instead of 2 Hz because of the shortened duration of the sampling function). In the average amplitude spectrum of individual attentional functions (right), significance (p value <10⁻³) was reached for three frequencies: 3 Hz, 6 Hz, and 9 Hz. (B) For the difficult search task, three frequencies reached significance in the amplitude spectrum of the grand-average attentional function: 3 Hz, 6 Hz, and 9 Hz. In the average amplitude spectrum of individual attentional functions, significance (p value <10⁻³) was reached for five frequencies: 3 Hz, 6 Hz, 9 Hz, 15 Hz, and 18 Hz, with a peak frequency at 9 Hz.

Experiment 2

The attention function measured in the difficult search task of Experiment 1 is supposed to represent the fluctuations in attention efficiency over time for an average search trial in an average observer. If this is true, one should predict that search patterns systematically presented with an SNR following the estimated grand-average attentional function (test condition) should lead to better performance than with the opposite signal-to-noise modulation (control condition). We directly tested this prediction in Experiment 2. Notice that we high-pass filtered the attention sampling function by discarding the 2 Hz component, thought to correspond to an onset-offset transient effect and therefore not reflecting a true sampling periodicity. Overall, the aim of Experiment 2 was to test whether rhythmic attention sampling significantly contributes to search performance. This experiment was performed on two sets of subjects: seven subjects from the “initial subjects” group (i.e., subjects who also performed the difficult search task in Experiment 1) and seven subjects from the “naive subjects” group (i.e., subjects who had not been included in Experiment 1). 

As previously, we first verified that naive observers used an appropriate search strategy by computing the RT × set size slopes: As expected (and as in Experiment 1), we obtained strongly positive slopes: 88.9 ms ± 41.3 ms per element, t(6) = 2.5, p = 0.04, for target present and 149.5 ms ± 35 ms per element, t(6) = 4.7, p = 0.0034, for target absent. 

In the main comparison for Experiment 2, the subjects' performance (as measured by d-primes) was significantly higher when stimulus presentation followed the estimated sampling function than its opposite (Figure 8, mixed design two-way analysis of variance, within-subject factor “test/control”: F(1, 12) = 8.2, p = 0.014). Moreover, no significant difference was found between subject groups (between-subject factor “initial/naïve,” F(1, 12) = 0.9, p = 0.37), nor any significant interaction between the two factors (“test/control” × “initial/naive”: F(1, 12) = 0.6, p = 0.471). 

Figure 8

 

Testing the behavioral relevance of the estimated attentional function. A difficult search array was presented with dynamic noise fluctuations following two conditions: “test condition,” with a signal-to-noise ratio (SNR) following the previously estimated grand-average attentional function (Figure 5), and “control condition,” with an SNR following the opposite of the estimated grand-average attentional function. We predicted significantly higher performance in the former than the latter case. In both cases, the 2 Hz component of the function was omitted (i.e., the function was high-pass filtered) so as to focus on genuine periodic effects. This experiment was performed on subjects who had previously participated in the first experiment, the so-called “initial subjects” (n = 7), but also on a new set of subjects, the so-called “naive subjects” (n = 7). A two-way analysis of variance (mixed design) revealed a significant difference between control and test conditions, F(1, 12) = 8.2, p = 0.014, but no significant difference between naive and initial subjects, F(1, 12) = 0.9, p = 0.37, nor any interaction between the two factors, F(1, 12) = 0.6, p = 0.471.

In summary, these results reveal that the grand-average attention sampling function estimated for the difficult search task (or, more precisely, its frequency components >2 Hz) is representative of the subjects' sampling strategy in this specific task. This is true not only for the initial subjects, indicating that the result obtained with this new method in Experiment 1 is robust and replicable, but also for the naive subjects, meaning that the estimated attention sampling function for this specific task can also apply to a more general population of human observers. 

Discussion

In this study, we proposed a novel experimental method based on periodic noise interference and Fourier series analysis to characterize the dynamics of perceptual sampling occurring during a difficult task (color-orientation conjunction) and an easy task (L vs. +). This method produces for each task a time course representing the sampling efficiency of attention over the 500 ms of stimulus display. Our major result is the presence of a significant oscillation in this time course for the difficult task but not for the easy one. A simple continuous model of attention sampling, in which perceptual efficiency is constant over time (except possibly for random fluctuations), would have predicted flat attentional sampling functions with no temporal fluctuation and therefore no peak in the corresponding amplitude spectrum. This is not what we observed: Both tasks presented significant peaks in their amplitude spectrum (Figures 4 and 6). Assuming that perceptual efficiency is optimal at one or possibly two specific moments of stimulus presentation, as one would expect for example due to greater stimulus-onset and/or -offset sensitivity, should have predicted increasing and/or decreasing attentional sampling functions and therefore a steadily decreasing amplitude spectrum with a maximum at the lowest frequency (2 Hz). We did observe such a 2 Hz peak in both tasks and confirmed that it was mainly due to stimulus-onset and -offset effects (Figure 7). However, a major unexpected result was the presence of another significant local maximum in the difficult task with a similar phase for all subjects (9–10 Hz; Figure 6), which was absent for the easy task (Figure 4); this peak can be explained only by assuming rhythmic perceptual sampling by attention. 

The presence of a 9 to 10 Hz oscillation in the grand-average temporal function (Figure 6A) could, in theory, be explained either by the existence of an equivalent oscillation (with a similar phase) at the level of each individual subject or by a chance combination of subject-specific peaks and troughs at different moments that would spuriously create an oscillation when averaged over subjects. However, our finding that individual subjects also present a 9 to 10 Hz peak in their temporal functions (Figure 6B) clearly rules out the latter explanation. Rather, it seems that this specific temporal function with its 9 to 10 Hz oscillation at a particular phase captures rather well the attention deployment of a “typical” subject in this search task (as shown also in Experiment 2). 

Such rhythmic sampling in a difficult task (color-orientation conjunction) appears naturally compatible with the classic sequential models of attention deployment: a periodic focusing of attention on different items (or subsets of items) within the search array until the target is found (Treisman & Gelade, 1980; Treisman & Souther, 1985; Wolfe, 1994; Wolfe et al., 1989). However, we cannot fully rule out the alternative parallel model of attention deployment in which all items are sampled at the same time (Eckstein et al., 2000; Palmer et al., 1993). Indeed, it is conceivable that all of the items in the difficult search were processed together within each attentional cycle, even as the efficiency of this processing fluctuated periodically at ∼9 to 10 Hz. Of course, this would represent a substantial alteration of the original parallel model (Eckstein et al., 2000; Palmer et al., 1993), yet such a revision appears necessary to reconcile this model with our experimental findings. Whether attention sampled the entire search array or only a subset of elements, the important conclusion of our study is that it did so periodically, rather than continuously (at least for the specific difficult search task that we used). 

Our novel paradigm appears suitable for revealing the temporal dynamics of attention in at least some visual search tasks. The method is not infallible, however, and it is worth reviewing its main limitations and the possible ways it could be improved in future studies. 

First and foremost, our paradigm relies on Fourier series decomposition, which is an intrinsically linear framework (see the Appendix for further description of the model). As such, we must assume that the search performance on each trial is the product of an attention sampling function and the superimposed noise function and that this attentional sampling function does not systematically vary when the noise function changes (for example, when we apply sinusoidal noise at different phases and frequencies). This assumption of linearity is both a reasonable first approximation in our endeavor to measure the dynamics of attention and an obvious potential limitation of the proposed technique. Indeed, it implies that many nonlinear attentional strategies would not be appropriately captured by the technique. It is worth pointing out, however, that our method was explicitly designed to prevent one common form of nonlinearity that would have resulted from a steady-state masking by the sinusoidal noise (i.e., perception fluctuating in phase with the noise) or from a perceptual entrainment to the stimulus (i.e., perception fluctuating in counterphase with the noise). In these two forms of entrainment, one would expect a general decrease (or increase, respectively) of performance at specific frequencies, regardless of the phase. Yet in our experiments, at each frequency we did not merely measure a response to two sinusoids separated by 90° (cosine and sine, as Fourier analysis would minimally require) but to all four cardinal sinusoids (i.e., cosine and its opposite, sine and its opposite). Therefore, any nonlinear perceptual entrainment or steady-state masking that would have equally affected all four noise sinusoids at a given frequency would have resulted in a null attention modulation at this frequency. To conclude, this type of nonlinear confound alone cannot explain our finding of a ∼10 Hz attentional modulation in the difficult search task. 

Another potential issue with the current version of our paradigm is the possibility that increased perceptual sensitivity at stimulus onset and/or offset could have produced edge effects that would have concealed the true temporal fluctuations of attention. Indeed, there was no noise on the screen before or after our 500-ms sampling window, meaning that the onset and offset stimulus information were potentially free of forward and backward masking, respectively. This potential confound led us to perform an additional analysis in which we removed the edges of our sampling window (Figure 7). Although this removal still produced a ∼10 Hz periodicity in the difficult search task—thereby confirming that this periodicity was not merely an artifact induced by edge effects—it also left no systematic remaining temporal fluctuation in the easy search task. It is plausible, however, that such temporal fluctuations, maybe even periodic ones, would have occurred in the easy search task if the onset and offset transient information had not been directly available to the observers. Therefore, in future studies, we propose to improve the paradigm by starting the sinusoidal noise modulation well before (e.g., 500 ms) and ending it well after the 500-ms critical sampling window during which the search array is actually displayed. This would essentially avoid edge effects by separating the onset and offset transients from our window of analysis. 

Finally, the potential influence of eye movements must be considered. Even though we instructed the subjects not to move their eyes, one might argue that some of the temporal fluctuations observed in our sampling functions could be the result of unwanted saccadic and/or micro-saccadic eye movements. This would imply that these eye movements are time locked to stimulus onset and that they would entail different temporal patterns depending on the exact search task. Saccades and/or microsaccades could not have directly produced the temporal pattern observed in the difficult search task, however, because they tend to occur at lower frequencies, typically less than 5 Hz (Martinez-Conde, Macknik, & Hubel, 2004). It appears more likely, therefore, that the ∼10 Hz periodicity obtained in our difficult search task was a reflection of covert, rather than overt, attention sampling dynamics. 

Conclusions

To summarize, using an innovative psychophysical method, we were able to argue that an easy search task was processed in an apparently continuous mode, whereas a difficult search task was processed periodically at a frequency of ∼9 to 10 Hz. Of course, our conclusions are limited to the two search tasks tested in this study, and further work would be needed to determine if they could be generalized to other easy and difficult search tasks. In addition, a potential limitation of our study is that we sampled attentional dynamics from only 2 to 20 Hz. Values greater than 20 Hz (e.g., in the gamma frequency range) were not considered, although they may be relevant for attention (Bauer, Oostenveld, Peeters, & Fries, 2006; Gruber, Muller, Keil, & Elbert, 1999; Jensen, Kaiser, & Lachaux, 2007; Treue, 2001). A number of recent studies have suggested that attention samples sensory information periodically, at frequencies between ∼7 and 13 Hz, in various other experimental situations, even when only one object is attended, and thus no sequential exploration is required (Busch & Vanrullen, 2010; Vanrullen, et al., 2007; Vanrullen & Dubois, 2011). The present demonstration of an intrinsic periodicity during a visual search task adds significant weight to this emerging view of periodic attention. 

Acknowledgments

This research was funded by a EURYI Award to RV. 

Commercial relationships: none. 

Corresponding author: Laura Dugué. 

Email: [e-mail]. 

Address: CNRS–Cerco, Toulouse, France. 

Appendix

In this article, we postulated that perception could be approximated (potentially with an additive or multiplicative constant) by a linear combination of stimulus information and attentional sampling, that is,  

 

Further, we argued that the variable P in Equation 1 could be approximated by the subjects' behavioral performance as measured experimentally by d′ values. With these assumptions, we could then reconstruct the attentional sampling function A(t) using Fourier series analysis, as a combination of sinusoidal functions weighed by the d′ values recorded experimentally for various frequencies and phases of stimulation (see the Experimental Procedure section in the main text). 

This reasoning, however, does not consider the full sequence of perceptual processes affecting the stimulus display (including the external noise imposed in our paradigm) nor the decision mechanism (potentially subjected to internal noise) by which the perceptual outcome is turned into a yes/no motor response on each trial. Here, we demonstrate within a full perceptual and decisional model (inspired by the visual noise-masking literature; Lu and Dosher, 2008; Solomon, 2002) that d′ is indeed a valid approximation of the variable P in Equation 1. 

Our observer model is depicted schematically in Supplementary Figure S1. It assumes that a time-varying signal pattern signal(t) combined with a time-varying external noise pattern noise(t) is sampled in time according to an attentional sampling function A(t). The resulting pattern is then compared with a target template w via a template-matching operation. The final yes/no decision process returns yes (target detected) whenever the template match, subjected to Gaussian-distributed internal noise η, exceeds a predefined criterion c, that is,  

 

The template-matching and decision processes in Equation 2 are identical to those described by Solomon (2002, equation 2, p. 106). In summary, the observer depicted in Supplementary Figure S1 will respond yes when  

 

and will respond no otherwise. 

Figure S1. Observer model.

 

The time-varying input display (combining stimulus and externally imposed noise) is sampled by an attentional sampling function. The resulting pattern is matched with a target template, and a noisy decision process compares the amount of match with an internal criterion.

Let us now apply this observer model to the conditions of our Experiment 1. On every trial i of our experiment, the display consists in a time-invariant stimulus pattern signal_i and a time-invariant external noise pattern noise_i, both of them amplitude-modulated in time by a sinusoidal function of the same frequency ω_i but opposite phase (let us call φ_i the phase applied to the signal). Within the context of our experiment, Equation 3 can therefore be rewritten as  

 

Because the signal_i and noise_i patterns are time-invariant, the equation becomes  

 

We further simplify this equation by assuming that the (positive) attentional sampling function A(t) is scaled such that its integral over the display interval is equal to 1, thus,  

 

The remaining integral in Equation 6 directly corresponds to the perceptual variable P used in the main text (more precisely, P_cosω, P_sinω, P_-cosω, and P_-sinω are obtained by setting the phase φ_i to 0, −π/2, π, and π/2, respectively). This variable P fluctuates between −1 and 1. When P = 1, reflecting an ideal match between the attentional sampling function and the sinusoidal stimulation for that trial, the external noise terms in Equation 6 are canceled and the response depends solely on the input signal pattern. On the other hand, when P = −1, implying that the attentional sampling function oscillates in phase opposition with the sinusoidal stimulation for that trial, the signal terms in Equation 6 disappear and the response solely depends on the noise. 

As explained above, it behooves us to demonstrate that d′ is a valid approximation for this variable P. For every possible value of P between −1 and 1 (by steps of 0.01), we simulated the model observer's response for 5,000 trials (half of them with a target and half without) using the following parameters. The internal noise η_i was taken from a Gaussian distribution with mean of 0 and standard deviation of 1 (this value was chosen to produce a range of d′ values between 0 and 2, compatible with those observed experimentally). We assumed that the signal pattern was identical to the target template on target-present trials (that is, w^tsignal_i = 1) and was orthogonal to the target template on target-absent trials (that is, w^tsignal_i = 0). The noise pattern was also assumed to be orthogonal on average to the target template but with trial-by-trial deviations in either direction; that is, the dot product w^tnoise_i was taken from a Gaussian distribution with a mean of 0 and standard deviation of 0.5 (this value implies that a fully random noise pattern will match the target template better than the target itself on 2.28% of trials—a rather liberal estimate). Finally, the criterion c was chosen (based on the observed false alarm rates during the experiment) to be c = 0.5. Supplementary Figure S2 illustrates the d′ values of this model observer obtained for various values of the variable P. Based on this near-linear relationship, we can conclude that in our study, d′ was indeed an appropriate experimental approximation of the variable P. 

Figure S2.

 

d′ of the observer model for various values of the variable P (corresponding to the integral in Equation 6). The existence of a near-linear relationship validates our choice of using d′ as an experimental approximation of P.