Sensitivity measures of visuospatial attention

Abstract

Measuring visual sensitivity has become popular to determine the spatial deployment of visual attention. Critically, the accuracy of the measurement depends on the quality of the stimulus used. We evaluated the strengths and weaknesses of six commonly used stimuli for assessing visual attention. While preparing an eye movement to a cued item, participants discriminated a stimulus-specific visual feature, either at the cued location or at other equidistant uncued locations. Stimuli differed in their visual features (digital letters, Gabors, crosses, pink noise, random dot kinematograms, and Gabor streams) and their presentation mode (static or dynamic stimuli). We evaluated these stimuli regarding their temporal and spatial specificity and their impact on saccade preparation. We assessed presaccadic visual sensitivity as a correlate of visual spatial attention and discuss the stimulus-specific time course, spatial specificity, and magnitude of the measured attention modulation. Irrespective of the stimulus type, we observed a clear increase of visual sensitivity at the cued location. Time course, spatial specificity, and magnitude of this improvement, however, were specific to each stimulus. Based on our findings, we present guidelines to select the stimulus best suited to measure visuospatial attention depending on the respective research question.

Introduction

Our complex environment provides us with far more information than we can simultaneously process. To achieve goal-directed behavior, we must selectively filter the vast amount of information with which we are confronted, and extract the most relevant aspects at any given moment. Visual spatial attention functions as a selection mechanism that allows us to prioritize particular locations while ignoring others (Treue, 2001; Carrasco, 2011). Attention biases the neuronal representation of the visual scene, such that the same retinal input elicits different neuronal responses depending on the attentional state of the observer (Hillyard & Anllo-Vento, 1998; Gandhi, Heeger, & Boynton, 1999; Reynolds, Pasternak, & Desimone, 2000; Martinez-Trujillo & Treue, 2002). These modulatory effects of attention on visual processing are also evident at the behavioral level. Attention reportedly benefits visual search (Nakayama & Mackeben, 1989; Carrasco & McElree, 2001), enhances spatial resolution (Yeshurun & Carrasco, 1998; Carrasco, Williams, & Yeshurun, 2002), as well as contrast sensitivity (Lee, Itti, Koch, & Braun, 1999), and alters appearance (Carrasco & Barbot, 2019). 

One approach to assess the behavioral correlates of visual spatial attention is the measurement of reaction times. Since it is well established that stimuli that are presented in the focus of attention are recognized faster than those that appear beyond (Posner, 1980; Posner, Snyder, & Davidson, 1980), one can deduce the spatial deployment of attention from manual response times. This measurement, however, reflects both the time for detecting the stimulus as well as decision and response-dependent processes (Santee & Egeth, 1982), i.e., the time taken to report the decision. Furthermore, the link between reaction times and neurophysiological activity is not well established. In contrast, it is known that spatial attention improves visual perception via oculomotor feedback projections converging onto earlier visual areas (Moore & Armstrong, 2003; Moore & Fallah, 2004; Müller, Philiastides, & Newsome, 2005). 

A more direct correlate of visual spatial attention thus can be obtained by measuring the sensitivity to detect or discriminate visual features (Macmillan & Creelman, 1991). The accuracy of this measurement depends decisively on the quality of the stimulus used and whether its presentation interferes with the normal deployment of attention. In a typical paradigm, a test stimulus is briefly presented among several distractors. Participants are either instructed to detect a specific target feature or discriminate its identity. As attention enhances visual processing, a higher discrimination performance for a particular item reflects the allocation of attention toward it. 

In this context, a variety of discrimination stimuli have been used, both under conditions in which participants had to keep their eyes steady (e.g., Pestilli & Carrasco, 2005; Ling & Carrasco, 2006; Liu, Abrams, & Carrasco, 2009; Störmer, McDonald, & Hillyard, 2009), and in paradigms combining the discrimination task with a movement task (Kowler, Anderson, Dosher, & Blaser, 1995; Deubel & Schneider, 1996; Montagnini & Castet, 2007). This latter case is particularly interesting as it allows to assess both the deployment of attention during movement preparation (while participants still fixate) and the metrics of the following movement. 

Using a dual-task paradigm combining a discrimination task and a saccade task, we compared the strengths and weaknesses of six different stimuli. We evaluated these stimuli regarding their spatial and temporal specificity, and their interference with saccade preparation. Besides experimental codes and collected data sets, we offer guidelines to select the most suitable stimulus for a specific research question. 

Methods

Participants

Ten participants (five females, five males, age 23-28, one author), some of whom had previously participated in similar studies, took part in the experiment. One participant completed only five of the six stimulus conditions (pink noise missing). The protocols for the study were approved by the ethical review board of the Faculty of Psychology and Education of the Ludwig-Maximilians-Universität München (approval number 13_b_2015), in accordance with German regulations and the Declaration of Helsinki. 

Apparatus

The dominant eye's gaze position was recorded using a SR Research EyeLink 1000 Desktop Mount eye tracker (Osgoode, Ontario, Canada) at a sampling rate of 1 kHz. The experiment was controlled by an Apple iMac Intel Core i5 computer (Cupertino, CA, USA) and the experimental software was implemented in MATLAB (MathWorks, Natick, MA), using the Psychophysics (Brainard 1997; Pelli 1997) and EyeLink toolboxes (Cornelissen, Peters, & Palmer, 2002). Stimuli were presented at a viewing distance of 60 cm on a 21-in. gamma-linearized SONY GDM-F500R CRT screen (Tokyo, Japan) with a spatial resolution of 1,024 by 768 pixels and a vertical refresh rate of 120 Hz. 

Main task

The study comprised six experimental conditions: digital letters, Gabors, crosses, pink noise, random dot kinematograms (RDK), and Gabor streams (Figure 1). The conditions differed only in stimulus characteristics (see Stimuli), while experimental design and task were identical. Participants initially fixated a central fixation target, a black (∼0 cd/m²) and white (∼120 cd/m²) bull's eye (radius: 0.4 degrees of visual angle or dva) on a gray background (∼60 cd/m²). Each trial started once we detected stable fixation within a virtual circle (radius: 2.0 dva) centered on the fixation target. Randomly within experimental blocks, four (set-size 4: 1/3 of all trials) or eight (set-size 8: 2/3 of all trials) evenly spaced mask items (M) appeared at a distance of 8 dva from the fixation target. Simultaneously, four or eight white direction indicators (white lines; width: 0.1 dva, length: 0.4 dva) were displayed around the fixation target, pointing toward the items. Between 400 and 800 ms after trial onset and until the end of the trial, the color of one direction line changed to black, cueing the item to which participants had to move their eyes as fast and precisely as possible (Figure 1A). At a randomly selected time between 200 ms before and 200 ms after cue onset, one of the mask items was replaced by a discrimination test target (T) while the others were replaced by distractor items (D). Mask items, test target, and distractors items were specific to each stimulus. Importantly, the test target was equally likely to appear at any position, irrespective of cue direction. At the end of each trial, participants reported their discrimination judgment specific to each stimulus in a non-speeded manner via keyboard button press (right or left button, or left, up, right, or down button for two- and four-alternative forced-choice tasks, respectively). A negative feedback sound followed incorrect responses. To determine the influence of the test target on different saccade metrics, in 6% of the trials no test target was presented (unbeknownst to the observers). Furthermore, to investigate the effect of saccade preparation, in another 6% of the trials no cue was shown and participants were maintained fixation over the whole trial duration. 

Figure 1

 

Experimental procedure and threshold results. (A) Schematic depiction of the stimuli configuration for set-size 4 and 8 (FT: fixation target; D: distractor; M: masks, T: test). (B) Stimuli timing. The fixation target (FT) remained on the screen throughout the whole duration of the trial. Between 400 and 800 ms after the onset of several mask items (M), a cue was presented indicating towards which item to saccade. After different intervals relative to cue onset, for a stimulus-specific duration the mask items were replaced by distractor items (D) and one test item (T). (C) Depiction of static and dynamic stimuli. (D) Threshold results. Colored dots represent individual participants normalized stimulus strength levels at threshold (80% correct discrimination) for each stimulus, for set size of four items (empty dots) and eight items (filled dots).

Participants performed the stimulus conditions in a randomly selected order in three to six experimental sessions on different days. They completed at least 1,620 trials per condition. We monitored correct fixation and saccade execution online and replaced incorrect trials at the end of each block. To maintain a consistent level of discrimination performance across participants and stimuli, the main task of each condition was preceded by a threshold task. 

Threshold task

The threshold task matched its respective main task with the exception that participants maintained fixation throughout the trials. Moreover, they were informed that the test target was always shown at the cued location. We used a procedure of constant stimuli and for each trial selected randomly the test target strength out of seven linear values specific to each stimulus (see the following for details). Note that for the digital letter stimulus we did not vary the test target strength, as for this stimulus neither the size nor the presentation duration affects discrimination performance (Deubel & Schneider, 1996, authors' discretion). 

Participants completed at least 420 trials of the threshold task per condition. By fitting cumulative Gaussian functions to the observed performance at each test target strength level, we determined the test target threshold corresponding to 80% correct discrimination performance. Figure 1D shows the normalized stimulus threshold obtained for each participant, separately for each stimulus and set size. The value 1 refers to the weakest discrimination signal of each respective stimulus (e.g., a tilt angle of 1° for the Gabors) and the value 7 refers to the strongest discrimination signal (e.g., a tilt angle of 25° for the Gabors). 

Threshold values obtained within participants were significantly correlated between the two set sizes of each stimulus (0.95 > r > 0.78, 0.007 > p > 0.001). However, we found a considerable variability between participants, demonstrating the necessity of a threshold procedure before measuring spatial visual attention. Note that we could not reliably estimate one participant's threshold for the RDK stimulus, as the range of test target strength did not match his visual capacity. This participant therefore did not take part in the RDK main task. 

Stimuli

The conditions differed both in the visual characteristics and in the presentation mode, which could either be static or dynamic (Figure 1C). The mask items of static stimuli (digital letters, Gabors, and crosses) changed into one static test target and static distractor items for duration specific to each stimulus. In contrast, dynamic stimuli (pink noise, RDK, and Gabor streams) continuously alternated (Gabor streams) or changed (pink noise and RDK) at a refresh rate specific to the stimulus. See Table 1 for a summary of stimuli properties. Supplementary Movie S1 illustrates a typical trial of the main task for each stimulus. We provide simplified MATLAB codes to produce the six stimuli used in our experiment (https://github.com/nmhanning/StimTest), together with the collected datasets (https://osf.io/x4vb5/). 

Table 1

 

Summary of stimuli properties. The table depicts the main spatial and temporal characteristics of the stimuli used.

 

Digital letters

Stimuli was adapted from Deubel and Schneider (1996). Mask items consisted of static digital “Display Formula”s, which were replaced for a duration of 83 ms by randomly selected digital “Display Formula”s and “Display Formula”s (distractor items) and one “” or “Display Formula” (test target). All characters had the same color (black), dimension (1.4 by 0.7 dva), and text width (0.15 dva). Participants reported whether the test target was an “Display Formula” or a “Display Formula”. The main task was preceded by short practice sessions of 50 trials, each of which the test target location was known. They were repeated until participant's discrimination performance was above 80%. 

Gabors

Stimuli was adapted from Rolfs and Carrasco (2012). Mask items consisted of static noise patches, which were replaced for a duration of 25 ms by vertically oriented Gabor patches (distractor items) and one tilted Gabor patch (test target), rotated either clockwise or counterclockwise relative to the vertical. Noise patches were composed of pixel noise (width ∼0.22 dva), ranging randomly from black to white, windowed by a Gaussian envelope (standard deviation: 1.1 dva). All Gabor patches (frequency: 2.5 cpd; 100% contrast) had the same randomly selected phase and the same Gaussian window as the noise patches. Participants reported the orientation of the test target (clockwise or counterclockwise). The tilt angle of the test target was determined in the threshold task using seven linear steps between 1° and 25° (threshold: 6.52 ± 0.79 dva—mean ± SEM—and 8.76 ± 1.11 dva, for set-size 4 and 8, respectively; see Figure 1D). 

Crosses

Stimuli were adapted from Born, Ansorge, and Kerzel (2012). Mask items consisted of static squared frames, which were replaced for a duration of 100 ms by symmetric crosses (distractor items) and one asymmetric cross (test target), with the vertical bar crossing the horizontal bar to the left or right of its center. All frames and crosses had the same color (black), dimension (1.4 by 1.4 dva), and line width (0.2 dva). Participants reported the offset direction of the vertical bar of the test target (left or right). The offset distance was determined in the threshold task using seven linear steps between 0.01 dva and 0.4 dva (threshold: 0.20 ± 0.05 dva and 0.20 ± 0.05 dva, for set-size 4 and 8, respectively; see Figure 1D). 

Pink noise

Stimuli were adapted from Hanning, Aagten-Murphy, and Deubel (2018). Mask items consisted of dynamic pink noise streams, each composed of randomly generated pink noise (1/f) patches changing at a rate of 60 Hz. For a duration of 83 ms, one noise stream was orientation filtered and displayed a 40° clockwise or counterclockwise tilt relative to the vertical (test target). The other noise streams remained unfiltered (distractor items). Noise patches were created by Fourier transforming uniform white noise, multiplying the noise spectrum with its inverse radial frequency and transforming it back (inverse Fourier transformation). The patches were windowed by a symmetrically raised 2D cosine (radius: 1.9 dva, sigma 0.5). Participants reported the orientation of the test target (clockwise or counterclockwise). The width of the orientation filter α, corresponding to the visibility of the orientation, was determined in the threshold task using seven linear steps between 30 and 90 (threshold: 69.40 ± 3.57 and 70.00 ± 3.40, for set-size 4 and 8, respectively; see Figure 1D). 

RDK

Stimuli were adapted from Szinte, Carrasco, Cavanagh, and Rolfs (2015). Mask items consisted of dynamic patches of dots moving in random directions. For a duration of 100 ms, the dots of one patch showed a coherent motion direction (test target), moving in one of the four cardinal directions (right, 0°; up, 90°; left, 180°; or down, 270°). Dots of the other patches continued moving randomly (distractor items). Each RDK patch was composed of half black and half white dots (radius: ∼0.17 dva), restricted within apertures of a 2.5 dva radius. Dots moved at a constant speed of 5 dva/s (limited lifetime of 83 ms plus an exponentially distributed jitter duration of 67 ms on average). The motion direction of each dot was drawn from a circular normal distribution (von Mises) with a certain degree of concentration K (inverse of the variance of a normal distribution) around one of the four cardinal directions. Test target's and mask items' K was 0 such that all dots moved randomly and incoherently in any direction. The test target was created by increasing the degree of concentration K of the dots in one randomly selected cardinal direction. Participants reported the coherent motion direction of the test target (right, up, left, or down). The coherence of the motion K, corresponding to the strength of the motion signal, was determined in the threshold task using seven exponentially increasing steps between 0.1 and 100 (threshold: 5.89 ± 0.44 and 6.29 ± 0.47, for set-size 4 and 8, respectively; see Figure 1D). 

Gabor streams

Stimuli were adapted from Rolfs, Jonikaitis, Deubel, and Cavanagh (2011). Mask items consisted of dynamic streams of vertically oriented Gabor patches and noise patches alternating at 40 Hz. For a duration of 25 ms, the Gabor patch of one stream was rotated either clockwise or counterclockwise relative to the vertical (test target), while the Gabor patches of the other streams remained vertical (distractor items). To avoid apparent motion effects, after test presentation the streams continued with alternating noise patches and blanks. Noise and Gabors patches were identical to those of the static Gabors. Participants reported the orientation of the test target (clockwise or counterclockwise). The tilt angle was determined in the threshold task using seven linear steps between 1° and 25° (threshold: 8.48 ± 2.07 dva and 9.16 ± 2.29 dva, for set-size 4 and 8, respectively; see Figure 1D). 

Data preprocessing

We scanned the recorded eye-position data offline and detected saccades based on their velocity distribution using a moving average over 20 subsequent eye position samples (Engbert & Mergenthaler, 2006). Saccade onset and offset were determined when the velocity exceeded or fell below the median of the moving average by 3 SDs for at least 20 ms. We included trials if correct fixation was maintained within 2.0 dva from the fixation target until cue onset, if the saccade landed within 2.0 dva from the cued item center no later than 500 ms after cue onset (except no cue trials), and if no blink occurred during the trial. Finally, we excluded trials in which the saccade started before the presentation of the test target had ended. In total, we included 75,132 trials in the analysis (85.4% of the online selected trials, 72.87% of all trials played). 

Behavioral data analysis

To determine the sensitivity to discriminate each test target (d′): d′ = z(hit rate) − z(false alarm rate), we took the percentage of correct discrimination performance as hit rate, and computed the false alarm rate by dividing the percentage of incorrect discrimination by the number of potential incorrect choices (one in the two-alternative forced-choice tasks, three in the four-alternative forced-choice task). Observed correct performance of 100% and 0% was substituted by 99% and 1%, respectively. Performances below chance level (d′ = 0 corresponding to 50% or 25% in the two- or four-alternative forced-choice tasks) were transformed to negative d′ values. 

For all statistical comparisons, we resampled our data and derived p values by locating any observed difference on the permutation distribution (difference in means based on 1,000 permutation resamples). To investigate temporal dynamics, we binned trials as a function of the interval between cue onset and test target offset in a 100 ms moving average (stepping every 20 ms), for test target offsets between 200 ms before and 200 ms after cue onset (Figure 2), or as a function of test target offset relative to saccade onset (100 ms moving average stepping every 20 ms from test target offset −400 to 0 ms before saccade onset, Figure 4). We report uncorrected p values, as all statistical comparisons were planned and conducted between independent sets of data points. 

Figure 2

 

Temporal specificity, attentional pop-out effect, and set-size dependence. (A) Visual sensitivity (d′) as a function of the test target offset to the cue onset at the cued (colored) and uncued locations (gray) for set-size 4 (dashed lines) and 8 (solid lines). Stars indicate the first time at which the sensitivity at the cued location was superior to the average sensitivity in no-cue trials for set-size 4 (framed) and 8 (filled), p < 0.05. Colored/gray areas indicate the SEM. (B) Average sensitivity in the no-cue trials for each condition and set size. Error bars indicate the SEM.

Results

Across all conditions, participants initiated saccades with a mean amplitude of 8.12 ± 0.03°—mean ± SEM—on average 215.50 ± 1.85 ms after the cue onset. In the following sections, we will compare the different stimuli regarding their temporal specificity, their tendency to pop out, the spatial specificity of the observed deployment of attention, as well as their potential disruptive effect on saccade preparation. 

Temporal specificity

We first analyzed the time course of visual sensitivity for each stimulus. As saccades are preceded by shifts of attention to the movement goal (Kowler et al., 1995; Deubel & Schneider, 1996; Montagnini & Castet, 2007), discrimination performance should increase at the cued item compared to uncued items during saccade preparation. For each stimulus and set size, Figure 2A shows the averaged sensitivity (d′) observed for cued and uncued items as a function of test target offset during the 400 ms surrounding cue onset. Irrespective of stimulus and set size, we observed an increase of visual sensitivity at the cued item after cue onset, while the sensitivity at the uncued items remained close to chance level. To quantify the benefit at the cued location, we computed an attentional latency corresponding to the time when the sensitivity observed at the cued item was significantly superior to the averaged sensitivity observed in trials without a cue. For most stimuli and set sizes the attentional latency was between 50 and 150 ms after cue onset (set-size 4 and set-size 8: Gabors: 50 ms and 110 ms; crosses: 110 ms and 150 ms; RDK: 150 ms and 50 ms; Gabor streams: 110 ms and 110 ms), for the digital letters and for the pink noise (but only for set-size 4) the benefit was present before the onset of the cue (digital letters: −30 ms and −50 ms; pink noise: −10 ms and 110 ms). Such a benefit preceding the cue onset could indicate that the test target was not sufficiently masked after its appearance. In this case, the cue could retroactively activate the visual content of the already vanished test signal (Sergent et al., 2013). Note that an analysis with a smaller time bin size (40 ms) gave the same effects as the analysis with 100 ms time bins. 

Attentional pop-out effect

To adequately assess the deployment of visual attention, the test target should not attract attention itself. To estimate each stimulus' vulnerability to such attentional pop-out effects, we evaluated the average visual sensitivity observed in trials without cue presentation (Figure 2B). Since none of the items was cued in these trials, no attention modulation should occur. Any increase in sensitivity relative to chance level can therefore be considered as an indicator of an attentional pop-out effect. 

Among most stimuli, the sensitivity to discriminate the test target without cue was comparably low (set-size 4 and set-size 8: letters: 0.24 ± 0.19 and 0.02 ± 0.11; Gabors: 0.31 ± 0.20 and 0.32 ± 0.13; crosses: 0.04 ± 0.15 and 0.11 ± 0.13; pink noise: 0.22 ± 0.10 and 0.24 ± 0.09; Gabor streams: 0.02 ± 0.21 and 0.12 ± 0.10). For the RDK, however, we observed a notably high sensitivity especially for set-size 4 (RDK: 0.63 ± 0.19 and 0.27 ± 0.11). This indicates that the onset of the test target motion signals could be properly discriminated without the indication of a cue, at least when four RDK patches were used. 

Spatial specificity

Next, we evaluated the spatial distribution of visual attention as measured with each stimulus. For this analysis, we included only trials in which the test target presentation ended within the last 150 ms before saccade onset, as we expected the most pronounced benefit at the saccade target within this time window (Deubel, 2008). Figure 3A shows visual sensitivity at each of the four or eight items as a function of its angular distance from the cue. Irrespective of the stimulus condition, we observed the highest sensitivity at the cued item, which was significantly superior to its neighboring locations both for set-size 4 (0.001 < p < 0.036) and set-size 8 (0.001 < p < 0.024). To quantify the spatial specificity of the attention measurement, we computed the difference between the sensitivity observed at cued location to the sensitivity observed at the adjacent locations for set-size 8 of each stimulus (Figure 3B). We observed the highest spatial specificity for the digital letters (0.89 ± 0.23) and the pink noise (0.91 ± 0.16), followed by the Gabors (0.75 ± 0.25), the RDK (0.65 ± 0.17), and the Gabor streams (0.57 ± 0.20). Crosses showed the weakest spatial specificity (0.45 ± 0.23). 

Figure 3

 

Spatial specificity. (A) Averaged visual sensitivity (d′) for each stimulus as a function of test target position relative to the cue location for set-size 4 (dashed lines) and 8 (solid lines). Data rotated to always represent the cue directed to the right. (B) Difference between the sensitivity observed at the cued location and its adjacent locations for set-size 8 of each stimulus. Error bars and areas indicate the SEM.

Figure 4

 

Impact on saccade parameters. Average saccade amplitude (A) and latency (B) for each stimulus as a function of the test target offset relative to the saccade onset for set-size 4 (dashed lines) and 8 (solid lines). Vertical black lines denote the average cue onset for set-size 4 (dashed lines) and 8 (solid lines). Colored areas indicate the SEM. Stars show significant amplitude differences between set-size 4 and 8 (in panel A), or significant differences between the average latency in trials with test target offset preceding vs. following the cue onset (in panel B); *p < 0.05. Horizontal dashed lines mark the actual saccade target amplitude.

Impact on saccade parameters

Lastly, we investigated the influence of each stimulus' test target presentation on saccade latency and amplitude. Such interference, indicated by a change of these saccade metrics, could suggest that the stimulus used to measure attention affects its deployment. Figure 4 shows the averaged saccade amplitudes and saccade latencies for each stimulus and set size as a function of the test target presentation time. To evaluate any potential impact on saccade preparation, we compared the average saccade latency and amplitude observed for trials in which the test target occurred before and after the cue onset. A latency or amplitude difference between those trials would demonstrate that the test presentation after cue onset, and thus during saccade preparation, affected saccade metrics. As neither saccade latencies (set-size 4: p = 0.236; set-size 8: p = 0.140) nor saccade amplitudes (set-size 4: p = 0.947; set-size 8: p = 0.489) differed depending on whether the test target was cued or uncued, we conducted the following analysis independently of the test target location. 

For all stimuli we observed smaller saccade amplitudes (Figure 4A) when four items were presented compared to eight (set-size 4 vs. set-size 8: digital letters: 8.25 ± 0.12 dva vs. 8.01 ± 0.12 dva, p < 0.001; Gabors: 8.22 ± 0.09 dva vs. 8.01 ± 0.08 dva, p < 0.001; crosses: 8.29 ± 0.11 dva vs. 8.05 ± 0.11 dva, p < 0.001; pink noise: 8.20 ± 0.11 dva vs. 7.95 ± 0.11 dva, p < 0.001; RDK: 8.37 ± 0.11 dva vs. 8.12 ± 0.11 dva, p < 0.001; Gabor streams: 8.22 ± 0.09 dva vs. 7.96 ± 0.09 dva, p < 0.001). Furthermore, for the RDK (pre-cue vs. post-cue: 8.23 ± 0.11 dva vs. 8.19 ± 0.11 dva, p = 0.009) as well as the Gabors (8.16 ± 0.09 dva vs. 8.01 ± 0.08 dva, p = 0.001) we observed slightly reduced amplitudes when the test was presented after the cue. For the digital letters, the crosses, the pink noise, and the Gabor streams, however, we did not observe any difference in amplitude depending on whether the test target preceded or followed the cue onset (0.87 > p > 0.20). This indicates that the saccade amplitude differences observed between set sizes reflect the impact of the display arrangement rather than a disturbance of attentional deployment. 

Moreover, Figure 4B shows that saccade latencies did not differ between trials in which the test target offset preceded or followed the cue onset for the pink noise (precue vs. postcue: 210.6 ± 5.8 ms vs. 211.0 ± 5.5 ms, p = 0.577), and the RDK (219.1 ± 4.5 ms vs. 218.8 ± 4.9 ms, p = 0.858). Gabor streams showed a small reduction of latencies when the test target presentation followed the cue onset (208.9 ± 3.3 ms vs. 205.5 ± 3.4 ms, p < 0.001). In contrast to these three dynamic stimuli, the sudden onset of the static stimulus' test target systematically slowed down saccade execution. In detail, we observed increased latencies when the test target was presented after cue onset for the Gabors (211.0 ± 4.6 ms vs. 220.9 ± 4.9 ms, p < 0.001), the crosses (212.1 ± 4.9 ms vs. 223.0 ± 3.2 ms, p < 0.007), and the digital letters, although for this stimulus the trend did not reach significance (213.3 ± 5.7 ms vs. 219.1 ± 4.9 ms, p = 0.086). Figure 5 reveals that this modulation in saccade latencies is compatible with the phenomenon of saccadic inhibition, whereby a transient change in the scene during a visual task causes a depression in saccadic frequency approximately 90 to 100 ms following the visual change (Reingold & Stampe, 2000; Reingold & Stampe, 2002; Buonocore & McIntosh, 2008). When we analyzed all trials in which the test target was presented after cue onset, we observed for all static stimuli a dip in the distribution of saccade latencies within 90–100 ms after the onset of the test signal. Interestingly, this initial dip in saccade latencies was followed by a second one, and the delay between these two minima directly reflected the test duration of the respective stimulus. This suggests that both the sudden onset of the test signal, as well as its sudden offset (i.e., onset of the mask), inhibited the proper execution of the programmed saccade. No such disruptive effect was observed for the dynamic, constantly changing stimuli. 

Figure 5

 

Distribution of saccade latencies relative to test target onset. Histograms depict the relative probability of saccade onsets for all trials in which the test target was presented after cue onset, combined for set-size 4 and set-size 8 of each stimulus (the sum of the bar heights equals 1). Colored areas in the background depict the test target presentation time. Black arrows (left column) indicate local minima corresponding to saccadic inhibition elicited by the test target onset and offset.

Discussion

We compared six stimuli commonly used to measure visual attention. All tested stimuli follow a common principle by taking the sensitivity to discriminate a stimulus-specific test target as a proxy for the allocation of visual attention. To evaluate saccade parameter, we used a dual-task design and measured the deployment of attention before the execution of a saccade. The stimuli, however, can be and have been also used in typical endogenous or exogenous attention tasks in the absence of eye movements. In summary, while all tested stimuli showed strengths, each revealed potential shortcomings that require a careful selection depending on the characteristics of the planned experiment. 

Threshold procedure

Except for the digital letters, participants performed a threshold task before the main experiment of each stimulus, in which we adjusted the discrimination signal strength of the stimulus-specific discrimination target to the visual capacity of each observer. Although this procedure requires additional testing time, it offers decisive advantages. Threshold tasks increase the intersubject comparability, as after a threshold procedure the performance of each observer in the upcoming main task can be expected to be equal. Furthermore, a threshold procedure ensures an appropriate level of difficulty for each observer, preventing both floor effects (the discrimination task is generally too difficult for an observer) and ceiling effects (the task is too easy), as well as potential learning effects across multiple experiments, all of which impede the successful measurement of the allocation of attention. Remarkably, the digital letter stimulus worked well without threshold procedure, i.e., without the need for individual adjustment. This may reflect the fact that letter discrimination is an over-learned skill in humans, rendering performance robust against stimulus properties such as size and presentation time, and interindividual differences. Thus, if the researcher does not have the possibility to perform a threshold procedure (e.g., in the context of a patient study with limited testing time), they should opt for the digital letter stimulus. 

Attentional pop-out effect

To successfully measure the distribution of attention, the employed test target should only be discriminable above chance when attention is directed toward it. To test any potential popping out of the test target, we randomly intermixed trials in our experiments in which no cue appeared. In these trials, observers kept fixation and attention was not biased toward a particular item, thus visual sensitivity should be close to chance level for discrimination targets that do not pop-out. We observed that the motion direction of the RDK could be discriminated above chance when we used a set size of four items. This issue can be overcome by using a higher set size (Szinte et al., 2015). Except of the RDK stimulus, all other stimuli showed limited attentional pop-out effects. Nevertheless, the observed discrimination performance was also above chance level. In general we suggest to reduce overall baseline performance by increasing the number of distractor items (Baldauf & Deubel, 2008; Jonikaitis & Deubel, 2011; Wollenberg, Deubel, & Szinte, 2018; Szinte, Jonikaitis, Rangelov, & Deubel, 2018). 

Retro-cueing effect

A visual inspection of the temporal dynamics reveals that digital letters are more effective when they are surrounded by close distractors (set-size 8), as with a sparser display (set-size 4), visual sensitivity started rising already before the onset of the direction cue. This observation suggests that crowding by the adjacent items (Bouma, 1970; Pelli, Palomares, & Majaj, 2004) helped to prevent retro-cueing effects, i.e., attentional benefits at the cued location that arise already before cue presentation (Sergent et al., 2013). Likewise, the pink noise stimulus was prone to this effect, with attentional performance rising at the future target already before cue onset when we used a sparser display. Moreover, we recommend to mask the visual stimuli after test target presentation to avoid these effects. 

Saccade parameters

When investigating the effect of each stimulus on saccade parameters, we found that the static stimuli (digital letters, Gabors, and crosses) systematically prolonged saccade latencies: When the discrimination target occurred after cue onset, i.e., within the period of saccade preparation, this sudden change of the otherwise static display affected the saccade execution: Both the sudden onset as well as the sudden offset of the test target delayed the execution of the saccade. This depression in saccadic frequencies approximately 90–100 ms following a transient change in the scene is compatible with the phenomenon of saccadic inhibition (Reingold & Stampe, 2000; Reingold & Stampe, 2002; Buonocore & McIntosh, 2008). Critically, we observed no such effect for the dynamic stimuli (pink noise, RDK, Gabor streams), for which the test target is embedded in a continuously changing display. Given the tight coupling of eye movement preparation and visual attention (Kowler et al., 1995; Deubel & Schneider, 1996; Montagnini & Castet, 2007), it is plausible that interrupting saccade preparation also affects the temporal dynamics of visual attention. Thus, stimuli relying on a sudden onset of the test target might bias what they should measure, the temporal dynamics of visual attention. We thus recommend using dynamic stimuli for precise temporal measurements of spatial attention. 

Stimulus-specific requirements

It is important to note that the temporal precision of the measured effect is determined by the stimulus duration itself. In this regard, Gabors and Gabor streams that allow short presentation of the test target should be considered when the goal is to study the dynamics of attention. 

Another criterion when selecting the stimulus for a given research design can be the stimulus size. Discrimination signals based on orientation judgments (e.g., Gabors) can be perceived at comparably smaller sizes, while identification stimuli like digital letters and motion signals like RDK require larger sizes. Therefore, when the research design requires a high spatial resolution, orientation stimuli should be favored. 

Nevertheless, RDK as well as pink noise have a decisive advantage over the other tested stimuli. Instead of presenting various discrete items or patches, the localized test signal can be embedded within a full-screen stimulus. Although common discrimination paradigms rely on discrete objects that structure the visual field and potentially bias attention to specified locations, a full-field paradigm does not reveal potential test locations and thus allows for an unbiased assessment of visuospatial attention across the visual field (Hanning & Deubel, 2018). 

Conclusion

All tested stimuli allowed for a spatiotemporal assessment of visual attention. The efficiency of the tested stimuli, however, varied with respect to the different criteria examined. What ultimately constitutes the perfect stimulus depends on the respective research question and design. We here provide a set of criteria that should support researchers to select the optimal stimulus for their next behavioral or combined behavioral and neurophysiological study. 

Acknowledgments

The authors are grateful to the current and former members of the Deubel lab for helpful comments and discussions. This research was supported by grants of the Deutsche Forschungsgemeinschaft (DFG) to HD (DE336/5-1 and GI964/1-1) and MS (SZ343/1) as well as a Marie Skłodowska Curie individual fellowship to MS (704537). The authors declare no potential conflict of interest, financial or otherwise. 

Commercial relationships: none. 

Corresponding author: Nina M. Hanning. 

Email: . 

Address: Allgemeine und Experimentelle Psychologie, Department Psychologie, Ludwig-Maximilians-Universität München, München, Germany. 

Supplementary material

Supplementary Movie S1. Demonstration of stimuli and experimental design for a set size of eight items. Instruction: Keep fixation on the central target. As soon as one of the white lines surrounding the fixation target turns black, make an eye movement toward the indicated item. For demonstration purposes, the discrimination target always appears at the cued location in this video.