The dynamics of saliency-driven and goal-driven visual selection as a function of eccentricity

Abstract

Both saliency and goal information are important factors in driving visual selection. Saliency-driven selection occurs primarily in early responses, whereas goal-driven selection happens predominantly in later responses. Here, we investigated how eccentricity affects the time courses of saliency-driven and goal-driven visual selection. In three experiments, we asked people to make a speeded eye movement toward a predefined target singleton which was simultaneously presented with a non-target singleton in a background of multiple homogeneously oriented other items. The target singleton could be either more or less salient than the non-target singleton. Both singletons were presented at one of three eccentricities (i.e., near, middle, or far). The results showed that, even though eccentricity had only little effect on overall selection performance, the underlying time courses of saliency-driven and goal-driven selection altered such that saliency effects became protracted and relevance effects became delayed for far eccentricity conditions. The protracted saliency effect was shown to be modulated by expectations as induced by the preceding trial. The results demonstrate the importance of incorporating both time and eccentricity as factors in models of visual selection.

Introduction

The visual world around us is rich and detailed in information. Yet, our vision is limited, as resolution is high for only a small central area of the retina and drops rapidly toward the periphery (e.g., Osterberg, 1935). The solution to this problem is eye movements, which allow us to select potentially interesting information in the periphery for more detailed subsequent assessment. One mechanism that contributes to selecting the next candidate location for an eye movement is based on saliency—that is, the physical distinctiveness of an object relative to its surrounding (Itti & Koch, 2001; Koch & Ullman, 1985; Parkhurst, Law, & Niebur, 2002; Theeuwes, 1992, 1994). Another mechanism is based on relevance, where selection is guided by the features necessary for the current behavioral goal (Al-Aidroos & Pratt, 2010; Bacon & Egeth, 1994; Castelhano, Mack, & Henderson, 2009; Folk, Remington, & Johnston, 1992; Wu & Remington, 2003); note that selection may also be determined by previous relevance (for overviews, see Awh et al., 2012; Wolfe & Horowitz, 2017). Research has shown that making either a saliency-driven or goal-driven eye movement is very much dependent on the time at which the eye movement is initiated. While saliency information influences visual selection early and only relatively briefly after display onset, the influence of relevance increases with viewing time (Dombrowe, Donk, Wright, Olivers, & Humphreys, 2012; Donk & Soesman, 2010, 2011; Donk & van Zoest, 2008; Godijn & Theeuwes, 2002; Hunt, von Mühlenen, & Kingstone, 2007; Parkhurst et al., 2002; Siebold, van Zoest, Meeter, & Donk, 2013; van Zoest & Donk, 2005, 2006; van Zoest, Donk, & Theeuwes, 2004). 

Whether saliency driven or goal driven, the information needed to guide the next eye movement is, by definition, peripheral to the current fixation (Engel, 1977; Hulleman & Olivers, 2017). However, despite decades of research into visual selection, relatively little is known about how the relative contributions of saliency-driven and goal-driven selection are affected by the eccentricity at which the information is presented. As is well known, eccentricity affects visual performance (Rosenholtz, 2016; Strasburger, Rentschler, & Jüttner, 2011). Performance with regard to eccentric stimuli as compared to centrally presented stimuli suffers more from limited visual resolution (e.g., Curcio, Sloan, Kalina, & Hendrickson, 1990), limited attentional resolution (He, Cavanagh, & Intriligator, 1997), and visual crowding (Bouma, 1970; Lettvin, 1976). Eccentric stimuli are, for example, found less rapidly in visual search (Carrasco, Evert, Chang, & Katz, 1995; Carrasco, McLean, Katz, & Frieder, 1998; Engel, 1977; but see Carrasco, McElree, B., Denisova, & Giordano, 2003; Carrasco, Giordano, & McElree, 2006), if not enlarged to compensate for what is known as the cortical magnification factor (Azzopardi & Cowey, 1993; Carrasco & Frieder, 1997; Carrasco et al., 2003; Horton & Hoyt, 1991; Yeshurun & Carrasco, 1998). On top of these effects there appears to be what is known as a central bias, as observers prefer to attend to central parts of images, whether of abstract arrays or real-world scenes (Bindemann, 2010; Buswell, 1935; Mannan, Ruddock, & Wooding, 1995; Tatler, 2007; Wolfe, O'Neill, & Bennett, 1998). Although these findings suggest that both visual resolution and attentional biases affect visual search performance across eccentricity, there is currently no empirical work on how eccentricity affects the relative contribution of saliency-driven versus goal-driven influences on selection. 

At a theoretical level, eccentricity has so far played a relatively little role in models of attentional guidance. Popular saliency models typically compute saliency uniformly across the visual field, as a property of the world (Borji & Itti, 2013; Itti & Koch, 2001; Itti, Koch, & Niebur, 1998; Navalpakkam & Itti, 2005), whereas models accounting for goal-driven guidance of attention generally do not assume such guidance to be dependent on eccentricity (Cave & Wolfe, 1990; Wolfe, 2012; Wolfe, Cave, & Franzel, 1989; Wolfe & Gancarz, 1996). One exception is one of the versions of the saliency model of Parkhurst et al. (2002) that includes a peripheral reduction of saliency to mimic the naturally occurring drop in visual sensitivity as a function of eccentricity. Including this drop in visual sensitivity provided a better predictor of human eye movement behavior. However, this particular model does not include goal-driven mechanisms and thus makes no prediction with regard to any differential eccentricity effects on saliency-driven versus goal-driven selection. Another exception is the target acquisition model (TAM) by Zelinsky (2008), which is a model of primarily top–down guidance of eye movements. In this model of search, possible target locations are visited by a simulated fovea to reflect the acuity limitations of peripheral vision. Here, too, behavior of the model closely matched the eye movements of human observers. Yet, TAM does not distinguish between saliency-driven and goal-driven influences on selection and whether these are differentially modulated by eccentricity (Zelinsky, 2008). Furthermore, none of these models incorporates the time course of saliency-driven and goal-driven selection. 

The aim of the present study was to investigate whether eccentricity affects the relative dynamics of saliency-driven and goal-driven control of visual selection. For this purpose, we used a task that allows for saliency-driven and goal-driven biases to be separated in time (van Zoest & Donk, 2004; van Zoest & Donk, 2005; van Zoest et al., 2004). Subjects were presented with two orientation-defined singletons, presented among a grid of uniformly oriented background elements (see Figure 1). One of the singletons was always the target and the other one the non-target. We used eye movements as the most direct proxy of selection. Subjects were instructed to make a speeded eye movement toward the target, allowing us to assess goal-driven selection. In addition, depending on the orientation of the background elements, the saliency of the target was either high or low relative to the saliency of the non-target, thus allowing us to assess saliency-driven selection. Importantly, by utilizing the natural trial-by-trial variation in saccade latency (Liversedge, Gilchrist, & Everling, 2011), we were able to measure selection performance as a function of time and separate the relative effects of saliency and relevance across the time courses of selection. In addition, both target and non-target were presented at one of three different levels of eccentricity, thus enabling us to assess any changes in the relative contributions of saliency and relevance over time as a function of the location in the visual field. 

Figure 1.

 

Three examples of the search display for the near, middle, and far eccentricity conditions. In the example displays of the near and middle conditions, the background elements are tilted 10° to the right, making the 30° left-tilted singleton more salient than the 30° right-tilted singleton. In the example display of the middle condition, this is the other way around. Either the left-tilted or right-tilted singleton was the target (this was counterbalanced between participants), making the other singleton the non-target. After a drift correction and an initial fixation display (500 ms) subjects made a speeded eye movement toward the target singleton. The search display remained on until 150 ms after the eye landed within 1.44 dva from one of the two singletons or for a maximum time period of 2000 ms if the eye did not land within 1.44 dva from one of the two singletons.

On the basis of the existing literature, a variety of occasionally opposite predictions can be derived. In line with previous findings, we expected a general increase in saccade latency with eccentricity (Hallett & Kalesnykas, 1995; Wyman & Steinman, 1973). Since saccades become more goal driven with time, it would thus be predicted that, solely on the basis of increased saccade latency, performance would become more goal driven with eccentricity. However, this prediction would only hold when the underlying saliency and relevance computations themselves would not change with eccentricity. For example, it has previously been reported that the speed of visual processing increases with eccentricity (Carrasco et al., 2003; Carrasco et al., 2006), which could possibly compensate for these latency delays. Also, whereas saliency-driven selection depends on the detection of a signal difference (i.e., signal presence), goal-driven selection depends on the discrimination of different signals (i.e., signal recognition). Various studies suggest that detection performance is less affected by eccentricity than discrimination performance (Anstis, 1974; Strasburger & Rentschler, 1996). Accordingly, one would expect an increasing influence of saliency and a decreasing influence of relevance with increasing eccentricity. At the same time, there is also evidence that mechanisms of feature-based attention are fairly constant across the visual field (Liu & Mance, 2011), which would predict little effect on at least goal-driven selection in our experiments. 

In three experiments we show that, although eccentricity did little in terms of overall performance, it affected the underlying time course and therefore the relative contribution of saliency-based selection; while, the time course of goal-driven selection was less clearly affected. Moreover, the data suggest that the saliency effect is modulated by spatial expectations on the range of stimulus eccentricities. 

Experiment 1

The goal of Experiment 1 was to investigate whether the relative contribution of saliency-driven and goal-driven control of visual selection changes across time, as a function of eccentricity. 

Methods

Participants

Twenty subjects participated in the experiment (age range, 18–22 years old; 16 females). All subjects reported normal or corrected-to-normal vision and gave informed consent prior to participation. Subjects received either course credit or a monetary reward for their participation. The protocol was approved by the ethics review board of the Faculty of Behavioral and Movement Sciences and conducted according to the tenets of the Declaration of Helsinki. 

Apparatus and stimuli

Stimuli were presented on a cathode-ray tube monitor with a resolution of 1280 × 1024 pixels and a refresh rate of 75 Hz. Eye movements were recorded using a tower-mounted EyeLink 1000 Plus eyetracker (SR Research, Ontario, Canada). Distance from the screen was kept constant at 50 cm by the use of a chin rest. A fixation cross consisting of two lines (with a stroke width of 0.07 degree of visual angle [dva], extending 0.3 × 0.3 dva) was presented whenever subjects were required to fixate. Stimuli were Gabor gratings, 2 dva in diameter, with a spatial frequency of 1.25 cycles per degree of visual angle presented at 100% contrast. Gabors were presented in a 19 × 19-element square grid (30.5 × 30.5 dva), with a center-to-center distance of 1.7 dva in both the vertical and horizontal direction. Each search display consisted of multiple homogeneously oriented background Gabors, tilted either 10° to the left or 10° to the right, and two singleton Gabors, one of which was oriented 30° to the left and the other 30° to the right. Simultaneously presented singleton Gabors were presented on the array diagonals at one of three possible eccentricities, 4.8 dva (near), 9.6 dva (middle), and 14.4 dva (far) from the center of the display. On each trial, both singletons were presented at the same eccentricity but never in the same quadrant. Participants were instructed to make a speeded eye movement to a predefined target. For half of the participants the target was the left-tilted singleton, and for the other half of the participants the target was the right-tilted singleton. Depending on the orientation contrast relative to the background elements on a given trial, the target could be either more salient (target more salient trials) or less salient (target less salient trials) than the other singleton, the non-target singleton. Note that, as we were interested in the time course, we did not want accuracy to suffer simply because the stimuli became invisible with eccentricity. To ensure that subjects could differentiate the tilt of the singletons at all eccentricities, all subjects completed 96 trials of an adjusted version of the experiment before the start of the main experiment. Here, only one of the two singletons was presented at the farthest eccentricity, 14.4 dva. Subjects were instructed to keep fixation and report whether the singleton was tilted to the left or to the right using the arrow keys. In all other aspects, the experiment was the same as the main experiment. All subjects performed better than 75% correct and therefore participated in the main experiment. 

Design

We used a within-subject design with eccentricity (near, middle, and far) and target saliency (more salient or less salient) as factors. Furthermore, each display contained both a target and a non-target, allowing us to also measure any relevance-based biases. All the different combinations of conditions were equally likely and presented randomly. Subjects completed 1200 experimental trials, divided into 24 blocks of 50 trials each. Feedback regarding saccade latency was provided after each block. A session took approximately 1.5 hours. 

Procedure

Examples of search displays are presented in Figure 1. Before the start of the experiment, a nine-point calibration was performed. Half of the participants were instructed to make a speeded eye movement to the left-tilted singleton, and the other half of the participants were instructed to make a speeded eye movement to the right-tilted singleton. Each trial started with the presentation of a central dot, required for a drift correction. After a space bar press, a central fixation cross was presented for 500 ms, followed by the search display. Subjects were instructed to fixate centrally while the fixation cross was presented and then to move their eyes toward the target singleton as soon as the search display appeared. The search display was presented without the fixation cross to encourage subjects to make a fast eye movement. The search display remained on screen until 150 ms after the eye reached an area within 1.44 dva from one of the two singletons (10% of farthest singleton eccentricity). If participants failed to do so within 2000 ms, the search display disappeared from screen. 

Data analysis

Eye movement data were analyzed offline. Saccade start and end points were defined using the velocity-based algorithm described in Nyström and Holmqvist (2010). For each trial, we calculated the saccade latency and landing position of the first saccade. The first saccade was defined as the first eye movement picked up by the algorithm. Saccade latency was defined as the time between search display onset and the start of the first eye movement. Trials in which the first saccade was initiated earlier than 80 ms were discarded from further analysis, as these were considered not to be driven by either saliency or relevance. The first saccade was assigned to be directed to either one of the singletons if its landing position was located in the corresponding quadrant and less than half of the eccentricity away from the singleton. Trials in which the first saccade were directed to neither one nor the other singleton were also discarded from further analyses. Note that these criteria differed from those used to the determine the end of a trial during the experiment itself. During the experiment, a trial ended if a raw eye gaze sample was less than 1.44 dva from any one of the two singletons. However, this raw gaze sample did not necessarily represent the landing position of the first eye movement. The offline selection ensured that we only included those trials in which the first eye movement was directed to either one or the other singleton. Saccade latency distributions were then calculated based on the remaining trials. However, to obtain reliable estimates of performance and to increase the stability of the model fits, trials were further discarded if the saccade latency fell within the lowest 2.5% of the overall latency distribution or was greater than 500 ms. 

A first analysis determined how saccade latency covaried with item selection (target, non-target; more salient, less salient) and eccentricity. Saccade latencies were averaged separately for whether saccades landed on the more salient or the less salient item and on the target or on the non-target item. These latencies were then entered in a repeated-measures analysis of variance (ANOVA) with singleton saliency (more salient or less salient), relevance (target or non-target), and eccentricity (near, middle, or far) as factors, with α = 0.05. The Greenhouse–Geisser correction was applied if the assumption of sphericity was violated (Huynh, 1978). To investigate how selection performance was overall affected by eccentricity, irrespective of time, we computed the individual averaged proportions of trials in which the eyes went to the target separately for target more salient and target less salient trials, as a function of eccentricity. The overall net saliency effect per eccentricity condition was then obtained by subtracting the proportions of eye movements toward the target in the target less salient trials from those in the target more salient trials. The overall net relevance effect per eccentricity condition was calculated by subtracting the proportions of eye movements toward the more salient non-target (i.e., 1 – proportion of eye movements toward the less salient target) from the proportions of eye movements toward the more salient target. These net saliency and relevance effects were entered into a repeated-measures ANOVA with eccentricity (near, middle, or far) as a factor, with α = 0.05. 

Most important were the analyses of the effects of saliency and relevance across time. For this, we looked at changes in selection performance as a function of saccade latency, using a weighted averaging procedure (van Leeuwen, Smeets, & Belopolsky, 2019). First, the single-subject data were smoothed using a moving Gaussian kernel with a width of 10 ms. Next, each point in the time course (in steps of 1 ms) was assigned a weight based on the number of data points contributing to that subject's latency distribution. These weights were used to calculate the weighted average performance. In doing so, this method compensates for the possibility that some subjects might have very few datapoints contributing to a certain time point. This would lead to an unreliable estimate of performance, which could distort the overall data pattern when simply averaging over participants. In order to examine the effects of saliency across saccade latency, the time course of the proportion of trials in which the eyes moved to the target was compared between the target more salient and the target less salient trials. Note that any difference in the proportions of eye movements going to the target in these different types of trials can only be attributed to the relative saliency of the target. To investigate the effects of relevance across saccade latency, the time course of the proportion of trials in which the eyes went to the more salient item was compared between the target more salient and the target less salient trials. Again, note that any differences in the proportions of eye movements going to the more salient item in these different types of trials can only be attributed to the relative relevance of the more salient item. To test for differences between trial types, we performed paired t-tests corrected for multiple comparisons using cluster-based permutation testing (Maris & Oostenveld, 2007) with 1000 permutations, separately per eccentricity. For a more detailed description of the procedure, see van Leeuwen et al. (2019). 

In order to examine more closely how the time courses of the saliency and relevance effects differed across eccentricity, we calculated difference curves for each eccentricity separately for the proportion of trials with the eyes going to the target and the proportion of trials with the eyes going to the more salient item. That is, for each eccentricity we subtracted the time course of the proportion of trials with the eyes going to the target obtained in the target less salient trials from the one obtained in the target more salient trials to acquire the difference function reflecting the net saliency effect across saccade latency. Similarly, by subtracting the time course of the proportion of trials with the eyes going to the more salient item obtained in the target less salient trials from the one obtained in the target more salient trials, we obtained an estimate of the net relevance effect across saccade latency. To test for differences in the time courses of the saliency effect and the relevance effect among the three eccentricity conditions, we used a jackknife procedure (Miller, Patterson, & Ulrich, 1998) in which we repeatedly calculated the net saliency and relevance effect, leaving each participant out of the analyses once. This resulted in 20 iterations, yielding 20 unique data patterns. For each jackknife, we determined the point in time at which performance reached a certain threshold. To avoid arbitrary selection of a specific value, we sampled different thresholds in the range of 0.1 to 0.5 in steps of 0.1. Threshold values outside this range were not present in at least one of the 20 jackknifes of either the saliency or relevance effect and could therefore not be included in the analysis. To test for differences across eccentricity, we performed adjusted t-tests and F-tests (Miller et al., 1998; Ulrich & Miller, 2001). Note that, because we present the range of jackknife thresholds to avoid cherry-picking, we did not apply multiple comparisons correction. Effect sizes were corrected by adjusting the error variance as described in Ulrich and Miller (2001) and reported as corrected partial eta squared: η_p²_c. 

Results and discussion

Trials in which the first saccade was directed to neither the target nor the non-target (15.0%) and those in which the saccade latency fell outside our latency criteria (11.3%; see Methods) were discarded from further analyses. 

Overall performance

Figure 2 shows the average saccade latency separately for eye movements directed toward the more salient target, the less salient target, the more salient non-target, and the less salient non-target in the three different eccentricity conditions. A repeated-measures ANOVA on the individual averaged saccade latencies with saliency of the selected item (more salient vs. less salient), relevance of the selected item (target vs. non-target), and eccentricity (near, middle, and far) as factors revealed a main effect of saliency, F(1, 19) = 92.38, p < 0.01, η_p² = 0.83, with shorter latencies for eye movements toward more salient items (257 ms) than less salient items (307 ms), as would be expected. Furthermore, we found a main effect of relevance, F(1, 19) = 90.40, p < 0.01, η_p² = 0.83, with shorter latencies for eye movements toward the non-target singleton (266 ms) than the target singleton (298 ms). This may seem somewhat counterintuitive, but it is due to the fact that the faster eye movements are more likely to be erroneous, as we will see later. Finally, we found a main effect of eccentricity, F(1.18, 22.46) = 17.19, p < 0.01, η_p² = 0.47. Average saccade latency increased as a function of eccentricity (near, 269 ms; middle, 280 ms; far, 297 ms), with all pair-wise contrasts being significant (all F > 5.3, all p < 0.05, all η_p² > 0.22). None of the interaction-effects reached significance (all F < 1.6, all p > 0.22, all η_p² < 0.08). 

Figure 2.

 

Average saccade latency in milliseconds as a function of eccentricity (near, middle, and far) plotted separately for the four possible items to be selected: target more salient, target less salient, non-target more salient, and non-target less salient. All error bars reflect 95% within-subject confidence intervals (cf. Cousineau, 2005).

Table 1 summarizes overall selection performance. It shows the average proportions of trials in which the eyes went to the target separately for target more salient and target less salient trials in each eccentricity condition, the corresponding net saliency and relevance effects, and the relevant statistics. The results show that, overall, target detection performance was not modulated by target eccentricity. Selection was affected by saliency. As would be expected, observers were more likely to saccade to the target when it was more salient. Furthermore, this saliency effect was differentially affected by eccentricity. Bonferroni-corrected t-tests revealed that the overall effect of saliency on target detection was lowest in the middle condition. In contrast, the relevance effect did not change as a function of eccentricity. 

Table 1.

 

Selection performance as a function of eccentricity, saliency, and relevance. Note: The saliency effect is defined as p(target|more salient target) – p(target|less salient target). The relevance effect is defined as p(target|more salient target) – [1 – p(target|less salient target)]. Note that p(target|more salient target) equals p(salient singleton|more salient target), and 1 – p(target|less salient target) equals p(salient singleton|less salient target).

 

Thus, in line with previous findings (Hallett & Kalesnykas, 1995; Wyman & Steinman, 1973), we found that saccade latency increased with eccentricity. As argued by Hallett and Kalesnykas (1995), reduced sensory signal strength for peripheral signals may contribute to this overall latency effect, but an important factor also appears to be delays in motor programming (Wyman & Steinman, 1973). Given the increase in saccade latency with eccentricity and given previous research showing that saccades become more goal driven with time, it is then notable that, overall, the relevance effect did not increase with eccentricity. Neither was the overall saliency effect systematically modulated by eccentricity; even though it decreased from the near to the middle eccentricity condition, it increased back to original levels from the middle to the far eccentricity condition. 

Importantly, though, as argued in the Introduction, overall performance potentially obscures differences in the underlying dynamics of selection. The next section therefore assesses how the relative contribution of saliency and relevance changes with saccade latency. 

Saliency as a function of time

Figure 3A shows the time courses of the proportion of saccades toward the target separately per eccentricity for target more salient and target less salient trials (see analysis section in Methods for details). Condition differences are indicated by significant clusters. These condition differences show that, for each eccentricity, eye movements elicited shortly after the onset of the search display were more likely to be directed to the target when it was more salient compared to when it was less salient. This difference disappeared with increasing saccade latency. For the middle eccentricity, we also observed a reversal of the pattern from 450 ms to 500 ms, which, given the size of the effect and the fact that it did not occur in other conditions or experiments, we believe to be largely spurious. 

Figure 3.

 

(A) Proportion of saccades toward the target as a function of saccade latency for the target more salient (blue) and target less salient (green) trials, separately per eccentricity. Shaded areas correspond to 95% confidence intervals. The clusters of time points at which performance differed between the target more salient and target less salient trials are indicated by the blue–green horizontal bars. The bottom of each subplot shows the saccade latency distribution, including a kernel density estimation (KDE; dashed black line). (B, left) Difference functions reflecting the net saliency effect across saccade latency separately per eccentricity. The net saliency effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. Here, 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate different jackknife thresholds. (B, right) The t-values of the jackknife analysis, where the colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference.

To examine the time courses depicted in Figure 3A more closely, we computed the net saliency effects as a function of eccentricity, which are shown in Figure 3B. Overall, the results show that saliency affected performance for approximately the first 200 to 250 ms, after which its influence dropped to zero. Importantly, the data revealed an extended saliency effect for the largest eccentricity. This extended saliency effect was expressed in two ways. First, there was an extended amplitude difference relative to the near and middle eccentricities during a time window of 175 to 275 ms. Second, we used a jackknife procedure (see analysis section in Methods) to investigate whether eccentricity differentially affected the duration of the saliency effect. These analyses showed that the saliency effect was more prolonged in the far condition compared to the middle condition (all tested thresholds) and the near condition (thresholds, 0.1–0.4). Finally, we also observed a brief period in which the amplitude of the saliency effect was higher in the near condition compared to the middle condition (150–200 ms after display onset). No further evidence for differences between these two conditions was observed. 

Relevance as a function of time

In order to examine the effects of item relevance over time, we calculated the weighted average proportions of eye movements to the more salient singleton as a function of saccade latency, separately per eccentricity condition for target more salient and target less salient trials, as shown in Figure 4A. The results show that long-latency eye movements were more likely to be directed to the more salient singleton when it was the target (target more salient trials) compared to when it was not the target (target less salient trials). This was the case for all eccentricities, as indicated by the significant clusters. To examine these time courses more closely, we computed the net relevance effects over time, which are shown in Figure 4B. The results show again that, overall, relevance primarily affects long-latency eye movements. In contrast to the saliency effects, no clear differences between eccentricity conditions were observed. To calculate the durations of the relevance effects we used the same jackknife procedure as described above. This analysis revealed a pattern in which most t-values fell left from zero (corresponding to delays with eccentricity); however, except for one of the thresholds (0.3), these were far from significant. 

Figure 4.

 

(A) Proportion of saccades toward the more salient item as a function of saccade latency for the target more salient (blue) and target less salient (green) trials, separately per eccentricity. Shaded areas correspond to 95% confidence intervals. The clusters of time points at which performance differed between the target more salient and target less salient trials are indicated by the blue–green horizontal bars. The bottom of each subplot shows the saccade latency distribution, including a KDE (dashed black line). (B, left) Difference functions reflecting the net relevance effect across saccade latency separately per eccentricity. The net relevance effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. Here, 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. No significant differences between the relevance effects were observed. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (B, right) The t-values of the jackknife analysis, where the colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference.

Conclusions

Short-latency eye movements were primarily affected by saliency, whereas long-latency eye movements were mostly affected by relevance. These findings are similar to those obtained in previous studies on selection control and support the notion that relative saliency is represented early but also only briefly in the visual system, whereas goal-related influences are delayed but more sustained (Donk & van Zoest, 2008; Siebold, van Zoest, & Donk, 2011). Importantly, we found here that the effect of saliency on visual selection was more protracted for the most peripheral location. 

Quite remarkably, we observed no reliable differences in the effects of relevance with increasing eccentricity. This suggests that the time courses of saliency-driven and goal-driven control are differentially affected in the periphery. Alternatively, power may have been insufficient to pick up subtle effects on goal-driven processing—something we will return to after Experiment 3. In any case, the important result is that the relative contribution to visual selection changes with eccentricity: visual selection becomes more strongly controlled by saliency than by relevance as eccentricity increases. Note again that this difference was not observable in the overall performance (Table 1). The reason is that overall performance was the result of two partly opposing effects—on the one hand, a protracted influence of saliency; on the other hand, an overall delay in selection (as expressed in saccade latency)—and hence greater reliance on goal-driven selection. This also explains why, when considering only overall performance (Table 1), the relative influence of saliency was reduced for the middle eccentricity condition compared to both near and far conditions. Although, as the underlying dynamics show, the saliency effects were actually similar to the near eccentricity condition, observers were overall slower in responding and hence relied relatively more on goal-driven selection (and thus relatively less on saliency). This underlines the importance of assessing the underlying dynamics of selection rather than the end result. 

Finally, even though the saliency effect was prolonged in the far eccentricity condition relative to the other eccentricity conditions, relatively little difference was found between the near and middle eccentricity conditions. One explanation for the absence of any difference between the near and middle eccentricity condition may be related to the specific spatial frequency used for our stimuli, which was relatively low. Sensitivity to spatial frequencies changes with eccentricity (De Valois, Albrecht, & Thorell, 1982; Foster, Gaska, Nagler, & Pollen, 1985; Hilz & Cavonius, 1974; Schiller, Finlay, & Volman, 1976), which may have obscured differential selection effects across eccentricity. To evaluate the generalizability of our results and to see if the spatial frequency characteristics of our stimuli modulated the observed pattern, in Experiment 2 we changed the spatial frequency. 

Experiment 2

Experiment 2 was similar to Experiment 1, except that we doubled the spatial frequency of the Gabor patches from 1.5 to 3 cycles per degree of visual angle. If the patterns of results in Experiment 1 were based on the specific spatial frequency characteristics of our stimuli, we would expect to find a different pattern of results in Experiment 2. More specifically, because sensitivity to higher spatial frequencies is higher close to the fovea (De Valois et al., 1982; Foster et al., 1985; Hilz & Cavonius, 1974; Schiller et al., 1976), we could also observe an eccentricity effect on the time course of the saliency signal for the middle eccentricity. Furthermore, an increased spatial frequency is likely to affect target discriminability and therefore potentially expose any potential eccentricity effects on relevance. 

Methods

Participants

Twenty new subjects participated in the experiment (age range, 18–23 years old; 18 females). All subjects had normal or corrected-to-normal vision and gave informed consent prior to participation. Subjects received either course credit or a monetary reward for their participation. 

Apparatus and stimuli

Experiment 2 was identical to Experiment 1, with the exception that the spatial frequency of the Gabor gratings was increased from 1.5 to 3 cycles per degree of visual angle. This change was also implemented in the pretest assessing whether participants were in principle able to differentiate the tilt of the singletons at the farthest eccentricity. All subjects performed better than 75% correct on the pretest and therefore participated in the main experiment. 

Results and discussion

Trials in which the first saccade was directed to neither the target nor the non-target (10.4%) and those in which the saccade latency fell outside our latency criteria (5.6%; see Methods for Experiment 1) were discarded from further analyses. 

Overall performance

Figure 5 shows the average saccade latency separately for eye movements directed toward the more salient target, the less salient target, the more salient non-target, and the less salient non-target in the three different eccentricity conditions. An ANOVA on the individual averaged saccade latencies with relative saliency of the selected item (more salient or less salient), relevance of the selected item (target or non-target), and eccentricity (near, middle, or far) revealed a main effect of relative saliency, F(1, 19) = 162.77, p < 0.01, η ² = 0.90, with shorter latencies for eye movements toward more salient items (226 ms) than less salient items (268 ms). Furthermore, we found a main effect of relevance, F(1, 19) = 29.73, p < 0.01, η_p² = 0.61, with shorter latencies for eye movements toward the non-target singleton (238 ms) than the target singleton (256 ms). Finally, we found a main effect of eccentricity, F(1.12, 21.29) = 15.25, p < 0.01, η_p² = 0.45, as latency increased with larger eye movements (near, 239 ms; middle, 243; far, 259 ms). These effects are the same as in Experiment 1. Different from Experiment 1 was the presence of a relevance × eccentricity interaction, F(2, 38) = 8.39, p < 0.01, η_p² = 0.31. Looking at Figure 5, we can see that saccade latency was less affected by eccentricity for non-targets than for targets, especially from the near to the middle eccentricity condition. None of the other the interaction effects reached significance (all F < 1.88, all p > 0.19, all η_p² < 0.09). 

Figure 5.

 

Average saccade latency in milliseconds as a function of eccentricity (near, middle, and far) plotted separately for the four possible items to be selected: target more salient, target less salient, non-target more salient, and non-target less salient. All error bars reflect 95% within-subject confidence intervals (cf. Cousineau, 2005).

The results regarding overall selection performance are summarized in Table 1 and show that neither the saliency effect nor the relevance effect varied across eccentricity. Overall, then, the pattern is quite similar to Experiment 1. The fact that overall selection behavior was not affected by eccentricity in spite of the presence of a profound eccentricity effect on saccade latency allows for the possibility that the relative underlying contribution of saliency-driven and goal-driven control over time changed with eccentricity. We assessed this next. 

Saliency as a function of time

Figure 6A shows the time courses of the proportion of saccades toward the target separately per eccentricity for target more salient and target less salient trials. Condition differences are indicated by significant clusters. Figure 6B shows the net saliency effects as a function of eccentricity. The pattern was very similar to that of Experiment 1, as saliency affected performance for the first 200 to 250 ms, after which its effect dropped to zero. Importantly, the data again revealed an extended saliency effect for the largest eccentricity, and the largest eccentricity only. This extended saliency effect was again expressed in two ways. First, there was an extended amplitude difference relative to the near and middle eccentricities during a time window of 175 to 290 ms. Second, the jackknife procedure (see Methods) showed a significant difference between the middle and far conditions for almost all tested thresholds (except threshold 0.1), and a significant difference between the near and far conditions later on in the time course (i.e., the lower thresholds of 0.2–0.3). No differences between the near and middle conditions were observed. 

Figure 6.

 

(A) Proportion saccades toward the target as a function of saccade latency for the target more salient (blue) and target less salient (green) trials, separately per eccentricity. Shaded areas correspond to 95% confidence intervals. The clusters of time points at which performance differed between the target more salient and target less salient trials are indicated by the blue–green horizontal bars. The bottom of each subplot shows the saccade latency distribution, including a KDE (dashed black line). (B, left) Difference functions reflecting the net saliency effect across saccade latency separately per eccentricity. The net saliency effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. Here, 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (B, right) The t-values of the jackknife analysis, where colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference.

Relevance as a function of time

In order to examine the effects of item relevance over time, we calculated the weighted average proportions of eye movements going to the more salient singleton as a function of saccade latency, separately per eccentricity condition for target more salient and target less salient trials, as shown in Figure 7A. The results show that, as in Experiment 1, long-latency eye movements were more likely to be directed to the more salient singleton when it was the target (target more salient trials) compared to when it was not (target less salient trials). This was the case for all eccentricities, as indicated by the significant clusters. Figure 7B shows the net relevance effects over time. When comparing the net relevance effects between eccentricities this time we found relatively weak but reliable evidence for a somewhat earlier onset of the relevance effect in the near condition. This was expressed in two ways. First, in terms of amplitude, the relevance effect was stronger in the near condition than in the far condition for an early time window (170–235 ms). Second, the jackknife analysis showed a similar pattern with reliable benefits for near targets at early-onset (i.e., low) thresholds. This was followed by an episode in which the relevance effect was more pronounced in the middle condition than in both the near condition (380–460 ms) and the far condition (405–450 ms). 

Figure 7.

 

(A) Proportion saccades toward the more salient item as a function of saccade latency for the target more salient (blue) and target less salient (green) trials, separately per eccentricity. Shaded areas correspond to 95% confidence intervals. The clusters of time points at which performance differed between the target more salient and target less salient trials are indicated by the blue–green horizontal bars. The bottom of each subplot shows the saccade latency distribution, including a KDE (dashed black line). (B, left) Difference functions reflecting the net relevance effect across saccade latency separately per eccentricity. The net relevance effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. Here, 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (B, right) The t-values of the jackknife analysis, where colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference.

Conclusions

Experiment 2 was performed to assess the reliability of the results of Experiment 1 and to investigate whether the similarity in time courses across the different eccentricity conditions was related to the specific spatial frequency characteristics of the stimuli in Experiment 1. Importantly, with regard to saliency, we found a pattern of results that was very similar to that of Experiment 1, as, again, the saliency effect persisted longer for the far eccentricity condition compared to the near and middle eccentricity conditions, with again no differences between these latter two conditions. The fact that we replicated the pattern of results makes it unlikely that our results were related to the specific spatial frequency characteristics of our stimuli. Also in terms of relevance, the use of a different spatial frequency did not substantially alter the results across time. As in Experiment 1, the effects of eccentricity on the time course of relevance were relatively minor. Nevertheless, we did find some indication of a difference in the initial onset of relevance effects, as relevance information became available sooner for the nearer conditions. The reason may be the better discriminability of the two targets for more central patterns, which may be a more important factor here than in Experiment 1 due to the higher spatial frequency that we used. We will return to this pattern after Experiment 3, which showed comparable findings. 

What remains is the question why, in Experiments 1 and 2, did eccentricity only modulate the saliency effect beyond the middle eccentricity and hardly affect the time course of relevance at all. One possible explanation for this is that the range of eccentricities was just too small to detect a difference. Related to this, there is the possibility that the duration of the saliency effect is tightly coupled to the exact retinal stimulus locations, to the extent that the near and middle eccentricity conditions fall within a specific zone of retinal locations that is functionally different from that in the far eccentricity condition. In the near and middle conditions, participants were asked to make eye movements of approximately 5 dva and 10 dva, respectively; however, in the far condition they were asked to make a saccade of approximately 14 dva. Note that normally people make eye movements up to about 10 dva, as beyond that range eye movements are typically accompanied by head movements (Bao & Pöppel, 2007; Bao, Wang, & Pöppel, 2012; Land, 2006; Lei, Bao, Wang, & Gutyrchik, 2012; Pöppel & Harvey, 1973). The 10-dva radius region corresponds closely to the macula and includes what is often referred to as the perifovea (not to be confused with parafovea). Macular vision has been associated with both physiological (Provis, Penfold, Cornish, Sandercoe, & Madigan, 2005) and functional (Pöppel & Harvey, 1973) differences in basic sensory processing compared to further eccentricities. It may be the case that this is also expressed in the way in which saliency affects visual selection. If so, we would expect the pattern of results to be tightly linked to the specific spatial range of retinal locations that are used. 

To test this, we conducted Experiment 3. 

Experiment 3

Experiment 3 was identical to Experiment 1, with the exception that we changed the retinal locations of the two singletons from 4.8 dva (near), 9.6 dva (middle), and 14.4 dva (far) to 9.6 dva (near), 14.4 dva (middle), and 19.2 dva (far). This way the range of eccentricities was shifted such that what were originally the middle and far eccentricities (between which an effect was observed) now became the near and middle eccentricities. If the eccentricity effects on saliency-based selection that we found were indeed functionally tied to areas beyond ∼10 dva, then we could expect the time course of saliency-driven selection to also be modulated from the near to the middle eccentricity condition. 

Methods

Participants

Twenty new subjects participated in the experiment (age range, 17–22 years old; 14 females). All subjects had normal or corrected-to-normal vision and gave informed consent prior to participation. Subjects received either course credit or a monetary reward for their participation. 

Apparatus and stimuli

Experiment 3 was identical to Experiment 1, with the exception that the possible locations of the two singletons were changed to 9.6 dva (near), 14.4 dva (middle), and 19.2 dva (far) from fixation. Stimuli were presented on an LG 4K monitor (LG Electronics, Seoul, South Korea) with a resolution of 3840 × 2160 pixels and a refresh rate of 60 Hz. This change was also implemented in the pretest assessing whether participants were in principle able to differentiate the tilt of the singletons at the furthest eccentricity. All subjects performed better than 75% correct on the pretest and therefore participated in the main experiment. 

Results and discussion

Trials in which the first saccade was directed to neither the target nor the non-target (12%) and those in which the saccade latency fell outside our latency criteria (6.7%; see Methods for Experiment 1) were discarded from further analyses. 

Overall saccade performance

Figure 8 shows the average saccade latency separately for eye movements directed toward the more salient target, the less salient target, the more salient non-target, and the less salient non-target in the three different eccentricity conditions. An ANOVA on the individual averaged saccade latencies with relative saliency of the selected item (more salient or less salient), relevance of the selected item (target or non-target), and eccentricity (near, middle, or far) revealed a main effect of relative saliency, F(1, 19) = 304.44, p < 0.01, η_p² = 0.94, with shorter latencies for eye movements toward more salient items (237 ms) than less salient items (288 ms). Furthermore, we found a main effect of relevance, F(1, 19) = 30.96, p < 0.01, η_p² = 0.62, with shorter latencies for eye movements toward the non-target singleton (252 ms) than the target singleton (274 ms). Finally, we found a main effect of eccentricity, F(1.40, 26.40) = 43.54, p < 0.01, η_p² = 0.70, as latencies increased with larger eye movements (near, 248 ms; middle, 264 ms; far, 277 ms). This time, the relative saliency × eccentricity interaction effect was significant, F(2, 38) = 11.10, p < 0.01, η_p² = 0.37. Looking at Figure 8, we can see that the eccentricity effect was slightly less pronounced when the items were more salient compared to when they were less salient. None of the other the interaction-effects reached significance (all F < 0.29, all p > 0.75, all η_p² < 0.01). 

Figure 8.

 

Average saccade latency in milliseconds as a function of eccentricity (near, middle, and far) plotted separately for the four possible items to be selected: target more salient, target less salient, non-target more salient, and non-target less salient. All error bars reflect 95% within-subject confidence intervals (cf. Cousineau, 2005).

The results regarding overall selection performance are summarized in Table 1. As in Experiments 1 and 2, overall performance was quite stable across eccentricity in spite of the presence of a profound eccentricity effect on saccade latency, although as in Experiment 1 we found a reduced overall influence of saliency for the middle eccentricity condition. As before, though, overall performance obscured the underlying time courses of these effects, which we analyzed next. 

Saliency as a function of time

Figure 9A shows the time courses of the proportion of saccades toward the target separately per eccentricity for target more salient and target less salient trials (see analysis section in Methods for details). Condition differences are indicated by significant clusters. Figure 9B shows the net saliency effects as a function of eccentricity. Overall, the pattern is very comparable to that of Experiments 1 and 2, as saliency affected performance for the first 250 to 300 ms, after which it dropped to zero. Importantly, the data again revealed an extended saliency effect for the largest eccentricity, and the largest eccentricity only. This extended saliency effect was expressed in two ways. First, there was an extended amplitude difference in the far eccentricity condition relative to the near and middle eccentricity conditions during a time window of 160 to 310 ms. Second, the jackknife procedure revealed significant differences between the near and far conditions and between the middle and far conditions for all thresholds (except between the middle and far conditions at threshold 0.1). No differences between the near and middle conditions were observed. Thus, again, we observed a protracted saliency effect for the largest eccentricity only. 

Figure 9.

 

(A) Proportion saccades toward the target as a function of saccade latency for the target more salient (blue) and target less salient (green) trials, separately per eccentricity. Shaded areas correspond to 95% confidence intervals. The clusters of time points at which performance differed between the target more salient and target less salient trials are indicated by the blue–green horizontal bars. The bottom of each subplot shows the saccade latency distribution, including a KDE (dashed black line). (B, left) Difference functions reflecting the net saliency effect across saccade latency separately per eccentricity. The net saliency effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. Here, 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (B, right) The t-values of the jackknife analysis, where colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference.

Relevance as a function of time

In order to examine the effects of item relevance over time, we calculated the weighted average proportions of eye movements to the more salient singleton as a function of saccade latency, separately per eccentricity condition for target more salient and target less salient trials, as shown in Figure 10A. As in the previous experiments, the overall pattern shows that relevance primarily affected long-latency eye movements. In addition, we found a very small but reliable effect in favor of selection of targets during an early time window in the middle eccentricity condition, which, given its size and the fact that it did not occur in the other conditions and experiments, we believe to be largely spurious. Figure 10B shows the net relevance effects over time. As in Experiment 2, here, too, we found some evidence for a delay in the onset of the relevance effects when the stimuli were presented farther from fixation. This resulted in a lower initial amplitude of the relevance effect during an early time window for the far condition relative to the near condition (275–360 ms). The jackknife procedure revealed a similar pattern, as the relevance effect in the far condition was delayed at the lower thresholds compared to both the near and the middle condition (thresholds, 0.2–0.4). 

Figure 10.

 

(A) Proportion saccades toward the more salient item as a function of saccade latency for the target more salient (blue) and target less salient (green) trials, separately per eccentricity. Shaded areas correspond to 95% confidence intervals. The clusters of time points at which performance differed between the target more salient and target less salient trials are indicated by the blue–green horizontal bars. The bottom of each subplot shows the saccade latency distribution, including a KDE (dashed black line). (B, left) Difference functions reflecting the net relevance effect across saccade latency separately per eccentricity. The net relevance effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. Here, 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (B, right) The t-values of the jackknife analysis, where colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference between the tested conditions.

Given that the evidence for a reduced relevance effect with eccentricity has so far been relatively weak, we conducted a post hoc analysis in which we collapsed the data across all three experiments and recalculated the net relevance effects over time in order to increase power. Note that this also collapses across the stimulus differences between the experiments. The results of this combined analysis are depicted in Figure 11. This analysis corroborated a delay in onset with eccentricity, as the far condition suffered both in terms of amplitude (relative to the near, 255–340 ms, and middle, 260 ms–365 ms) and in terms of reaching the earlier jackknife thresholds (compared to near, 0.1 and 0.3, and middle, 0.4). Thus, with increased statistical sensitivity, eccentricity-dependent delays in goal-driven processing could be observed. 

Figure 11.

 

(Left) Difference functions reflecting the net relevance effect across saccade latency separately per eccentricity, collapsed across the three experiments. The net relevance effect for the near condition is plotted in red, the middle condition in blue, and the far condition in green. The 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (Right) The t-values of the jackknife analysis, where colors indicate which conditions were compared. Bold markers falling on the shaded region mark the thresholds at which there was a significant difference.

Attentional modulation of the saliency and relevance effects: Inter-trial analyses

The fact that Experiment 3 replicated the pattern of results of Experiments 1 and 2 but for a different range of eccentricities indicates that the effects of neither saliency nor relevance depend on the exact retinal position. We can therefore exclude the idea that the repeated finding of a prolonged saliency effect (and to some extent a delayed relevance effect) at the farthest eccentricity, but not the middle eccentricity, is due to any functional difference associated with absolute eccentricity values, as could, for example, have been associated with macular vision. For the exact same eccentricity value, we found either a prolonged saliency effect (far eccentricity of Experiments 1 and 2) or no such prolongation (middle eccentricity in Experiment 3), depending on the experimental context, plus a similar though much weaker pattern for relevance. 

The question then is what these experiments have in common that may have caused this particular pattern of results. One way to explain the difference between the far eccentricity, on the one hand, and the near and the middle eccentricities, on the other hand, is by assuming that observers adapt their spatial attention to the distribution of potential target positions. Previous research has shown that spatial attention can act like a “zoom lens” or flexible “window” that is either wider or narrower depending on the task (Eriksen & St. James, 1986; Gibson & Peterson, 2001; Theeuwes, 2004). Stimuli within the attentional window have attentional priority over stimuli outside the window (Belopolsky & Theeuwes, 2010; Belopolsky, Zwaan, Theeuwes, & Kramer, 2007; Kerzel, Born, & Schönhammer, 2012). The size of the attentional window is assumed to be dependent on the expected target location in a search task (Belopolsky et al., 2007; Belopolsky & Theeuwes, 2010; Kerzel et al., 2012; Theeuwes, 2004). We therefore speculated, post hoc, that observers may have adopted an attentional window that at least partly adapted to the distribution of the stimuli. Specifically, given that the range of eccentricities was centered on the middle eccentricity, we hypothesized that the width of the attentional window may have overall been tuned such that it encompassed the middle eccentricity. This would occur at the expense of the far eccentricity (outside the window) but not the near eccentricity (within the window), resulting in the current pattern. 

We currently do not know whether the width of the attentional window is based on explicit expectations or on implicit biases caused by previous experience, or both. However, our data provided us with the opportunity to test for the latter, by looking at inter-trial effects, assessing whether the dynamics of saliency-driven selection changes as a function of eccentricity on the preceding trial. Specifically, we predicted that on an individual trial level saliency effects should be more prolonged for more eccentric items when the attentional window is contracted more toward the center as a result of the previous trial, compared to when attention is distributed more widely toward the periphery. To this end, in an exploratory analysis, we collapsed the three datasets and investigated whether the specific eccentricity of the items on trial n – 1 modulated the dynamics of saliency-driven selection on trial n. 

Figure 12A shows the net saliency effect as a function of saccade latency for each eccentricity, with each subplot reflecting the eccentricity condition of the preceding trial. As the graph suggests, the saliency effect on selection with greater eccentricity was relatively most prolonged when the preceding trial contained near targets. This pattern was confirmed when analyzing the jackknife estimates for each eccentricity in the current trial as a function of the eccentricity in the preceding trial. Figure 12B shows these estimates for the standard threshold of 0.5. An ANOVA revealed a main effect of eccentricity in the current trial, F(1.76, 103.7) = 19.56, p < 0.01, η_p²_c = 0.25. Looking at Figure 12B, it is clear that this main effect originated from the overall delay of the saliency effect in the far condition, which we have observed in all saliency effect analyses so far. The main effect of eccentricity in the previous trial was not significant, F(2, 118) = 1.43, p = 0.24. Importantly, there was an interaction between eccentricity in the previous trial and eccentricity in the current trial, Fs(3.37, 198.90) = 2.73, p < 0.05, η_p²_c = 0.04. There was a stronger increase in the duration of the saliency effect with eccentricity when preceded by a near-eccentricity trial than when preceded by a middle- or far-eccentricity trial. Thus, these findings show that the experience on trial n – 1 influenced the duration of the saliency effect on trial n. We propose that these experiences may build up over time and shape the spatial attentional window such that it specifically benefits the near and middle eccentricity conditions. We will further discuss these findings in the General Discussion. Finally, for completeness, we also conducted the same intertrial analyses for the relevance component of selection, but this revealed no interactions (all F < 0.60, all p > 0.72). 

Figure 12.

 

(A) Difference functions reflecting the net saliency effect across saccade latency separately per eccentricity in the current trial, aggregated across trials that were preceded by a near, middle, and far trial, collapsed across the three experiments. The near condition in the current trial is plotted in red, the middle condition in the current trial in blue, and the far condition in the current trial in green. The 95% confidence intervals represent the average confidence interval of the two contrasts (which were very similar). Bold lines indicate where performance differed significantly from zero. Time points where the effect differed significantly between conditions are indicated by the horizontal bars at the bottom of the plot, with alternating colors indicating which conditions were compared. Black horizontal lines indicate which thresholds were tested in the jackknife analysis. (B) Jackknife duration estimates for the 0.5 threshold of the saliency effect of current near, middle, and far trials, plotted separately for trials that were preceded by a near (red), middle (blue), or far (green) trial. Error bars reflect standard deviations. Note that the F values in the repeated-measures ANOVA have been corrected according to Miller et al. (1998).

General discussion

We investigated how eccentricity alters the dynamics of saliency- and goal-driven control of visual selection. Observers were asked to make a speeded eye movement toward a target that was either more salient or less salient. Critical to our research question, we compared the relative contribution of saliency and relevance across three levels of eccentricity (i.e., near, middle, and far). We replicated the findings of earlier studies showing that fast eye movements are mostly driven by saliency whereas slow eye movements are mostly driven by relevance. Most importantly, all experiments showed that the effect of saliency on selection was protracted for the farthest eccentricity. In addition, although less clearly, we found evidence that the effect of relevance on selection was delayed with eccentricity. Thus, over time the contribution of saliency-driven control increases as a function of eccentricity, but the contribution of relevance decreases. 

We once again point out that, although eccentricity affected the dynamics of selection, it did not consistently affect overall performance; that is, on average, across trials, target selectivity remained stable for targets from near to far positions. How can we reconcile this constant performance with an extended underlying influence of saliency and delay in influence of relevance? To understand this, it is important to take into account that saccade latency distributions shifted with eccentricity, as it took longer to initiate a saccade toward more peripheral items. This means that, overall across the different levels of eccentricity, prolonged saliency effects were obscured by longer saccade latencies. Therefore, in addition to reduced sensory signal strength (Hallett & Kalesnykas, 1995) and delays in motor programming (Wyman & Steinman, 1973), another interesting explanation for the increase in saccade latency is that observers tried to maintain the same level of performance in the face of extended saliency effects (in other words, a speed–accuracy tradeoff). In any case, the current results emphasize that it can be more revealing to make use of the distribution of responses and look at the underlying dynamics rather than at the end result of selection. 

Although the effects of eccentricity on the time course of saliency-driven selection were clear and replicated across three experiments, the evidence for an eccentricity-driven delay in the relevance effect is less robust. One reason is that goal-driven mechanisms may be inherently subject to more variance than saliency-driven selection. The latter is a fast, automatic, feedforward process, whereas the former develops more slowly (e.g., Egeth & Yantis, 1997; Müller & Rabbitt, 1989; Posner, 1980) and will be inherently subject to the strength of activation of top–down goals, something that is likely to vary across individuals. Differences in variability are also caused by the fact that saccade latency distributions are strongly skewed toward the right, thus resulting in more data points for the saliency-driven end of selection. Indeed, when we increased the power of our statistical tests by combining the data from the three experiments together, we were able to show a reliable delay with eccentricity for the relevance effect, too, but it is clear that the eccentricity effect here was much weaker than for saliency. 

However, another reason for the lack of a clear effect of eccentricity on goal-driven selection is that we were in essence measuring the effects of feature-based attention. Unlike spatial attention, which by definition enhances specific spatial regions, feature-based attention only optimally serves selection if it operates in a similar fashion across the visual field. In fact, one could argue that feature-based attention is specifically useful for peripheral vision as it helps to locate the relevant target for a next fixation. There is indeed considerable evidence that feature-based attention operates globally (Andersen, Hillyard, & Müller, 2013; Andersen, Müller, & Hillyard, 2009; Forschack, Andersen, Müller, 2017; Liu & Hou, 2011; Saenz, Buracas, & Boynton, 2002; White & Carrasco, 2011) and in a rather constant manner (Liu & Mance, 2011). This is consistent with our pattern of findings, even though we did observe a small but reliable hint of a delay for the farther eccentricity. 

The protraction of the saliency effect occurred only for the far eccentricity condition in all three experiments and did not depend on spatial frequency (Experiment 2) or on the specific range of eccentricities used (Experiment 3). However, the time course was modulated by biases induced by the preceding trial. What kind of model could explain this specific pattern of delays with increasing eccentricity? We propose that repeated exposure to the different eccentricity conditions in our experiment led to an expectation (whether explicit or implicit) of the spatial distribution of the stimuli. This expectation resulted in changes in the spatial bias of attention (Eriksen & St. James, 1986; Gibson & Peterson, 2001; Theeuwes, 2004), such that the observers expanded or centered their attentional window to include primarily the near and middle eccentricities at the expense of the far eccentricity. 

Figure 13 illustrates how such an attentional bias may affect the duration of relative saliency effects. 

Figure 13.

 

Schematic representation of information accumulation over time for a more salient (blue) and less salient (green) item in arbitrary units (a.u.). Dashed lines represent information accumulation for singletons outside the attentional window; solid lines represent information accumulation for singletons inside the attentional window, with attention being modeled as a constant increase in gain. The black vertical line displays the time point at which a particular saccade could have been made (this varied from trial to trial, as illustrated by the saccade latency distribution plotted at the bottom of the figure). As can be seen, net saliency effects (shaded gray areas) are reduced for attended versus unattended singletons.

It shows a schematic representation of evidence accumulation across time for a more salient (blue) and less salient (green) item, following a simple response gain model (Reynolds & Chelazzi, 2004; Reynolds, Pasternak, & Desimone, 2000). The rate at which the evidence accumulates is higher for more salient singletons than for less salient singletons, which is the reason why fast eye movements are mostly saliency driven. As time progresses, evidence accumulation plateaus for both the salient and less salient item, and the preference for more salient singletons diminishes. In addition, one can imagine slower rates of accumulation for more peripheral locations, leading to protracted saliency effects (Staugaard, Petersen, & Vangkilde, 2016; Zhou, Bao, Sander, Trahms, & Pöppel, 2010) (not illustrated here). Importantly, covertly attending to the singletons increases the gain on and thus speeds up the accumulation process, as is expressed in the difference between the solid and dashed lines. As a result, the net saliency effect (shaded area) is reduced for attended versus unattended items. 

At face value, these findings may seem opposite to results from earlier studies investigating the effects of the spatial distribution of attention on saliency effects (Belopolsky & Theeuwes, 2010; Belopolsky et al., 2007; Kerzel et al., 2012). In those studies, observers were asked to look for a shape-defined target while ignoring a more salient color-defined distractor. Through various manipulations, observers were induced to adopt a more or less wide attentional distribution. The results showed that the presence of a distractor slowed down manual reaction times, but this interference effect was reduced to absent when observers had presumably adopted a more narrow attentional window. This appears to go against what we found here—namely, a reduced saliency effect when items presumably fell inside the attentional window. However, those earlier studies relied on manual responses, whereas we measured eye movements, which arguably are driven more directly by attentional orienting mechanisms than are manual reaction times, which may be affected by post-attentional decision making. Moreover, and more importantly, averaging manual reaction times does not provide an indication of the underlying time course of each of the effects. In this respect, it is relevant to note that in these previous studies a narrower attentional window also led to overall considerably slower responses. As our experiments showed, delayed responding resulted in a greater opportunity for goal-driven processes to take over from saliency-driven processes, which could have led to more effective distractor exclusion. In any case, the paradoxical results provide another testament to the importance of looking at the underlying dynamics of selection rather than only the end result. 

Our results also appear to contrast with those reported by Carrasco and colleagues (Carrasco et al., 2003; Carrasco et al., 2006), who found an increase in performance with eccentricity. They used a speed–accuracy trade-off procedure in which participants were asked to manually indicate the orientation of a target Gabor patch (tilted 30° to the right or to the left) which was presented either in isolation or simultaneously with multiple vertically oriented Gabor patches at one of two different eccentricities (4° and 9°). Each stimulus display was only briefly presented (i.e., 40 ms) and was followed after a variable time interval by the presentation of a tone. Participants were instructed to respond within 300 ms after the onset of the tone. By varying the time interval between the stimulus display and the tone, Carrasco and colleagues were able to examine information accrual across time and how this was affected by eccentricity. They showed that, although overall accuracy might have been reduced for more peripheral stimuli, the speed at which information was accumulated was faster. In contrast, we found that larger eccentricity led to slower responses (i.e., longer saccade latencies). We note various differences between the study of Carrasco and colleagues and ours that render a direct comparison difficult. First, there is a difference in stimuli that may be important. Carrasco et al. never presented a target singleton simultaneously with a distractor singleton as we did. Accordingly, differences across eccentricity as observed by Carrasco et al. always reflected variations in the dynamics of target processing, whereas we investigated the outcome of a competition between two singletons—a competition that was modulated by both saliency-driven and goal-driven components which cannot be isolated from the Carrasco et al. studies. This leaves open the possibility of a dissociation between how rapidly stimulus information accrues and how rapidly competition between stimuli is resolved. Second, whereas the Carrasco et al. studies measured manual responses, as paced by the presentation of a tone, we measured eye movements, which were self-paced, reflecting the natural trial-by-trial variation in saccade latency. It could be that these two types of decision are differentially affected by eccentricity. Third, and potentially most important, the presentation duration of the stimulus displays in the Carrasco et al. studies was only 40 ms, whereas our stimuli were presented until one of the singletons was selected. Although such brief, dynamic stimuli may be processed faster at farther eccentricity, the processing of static stimuli may suffer (Carrasco et al., 2006; Hartmann, Lachenmayr, & Brettel, 1979; McKee & Taylor, 1984). As Carrasco et al. (2003) hypothesized, this difference may be related to a higher involvement of the magnocellular as opposed to parvocellular system in eccentric vision. Compared to parvocellular cells, magnocellular cells have a higher speed of conduction and are specifically sensitive to the dynamics of stimulation (Lamme & Roelfsema, 2000; Schmolesky, Wang, Hanes, Thompson, Leutgeb, Schall, & Leventhal, 1998). Nevertheless, in spite of these differences, it is also worth pointing out that Carrasco et al. (2006) found that directing attention speeds up processing at peripheral locations, similar to what we propose accounts for our intertrial effects here. 

The findings reported here are relevant for models of visual selection, which currently lack a dynamic component (Borji & Itti, 2013; Cave & Wolfe, 1990; Itti & Koch, 2001; Itti et al., 1998; Navalpakkam & Itti, 2005; Wolfe, 2012; Wolfe & Gancarz, 1996; Wolfe et al., 1989). For example, most models assume a dominant role for either saliency or relevance. Although this might explain overall behavior in specific tasks, this does not account for the dynamic changes observed in the experiments reported here. An important reason why others might have failed to pick up on any effects of either saliency or relevance is because these processes follow different time courses. We and others before us (Donk & van Zoest, 2008; Godijn & Theeuwes, 2002; Hunt et al., 2007; Ludwig & Gilchrist, 2002; Parkhurst et al., 2002; van Zoest & Donk, 2005; van Zoest & Donk, 2006; van Zoest et al., 2004) have shown that time is a critical factor in the selection process, especially for eye movements. Namely, eye movements executed quickly after display onset are mostly saliency driven while later eye movements are mostly goal driven. In other words, instead of relying on either saliency or relevance to model visual selection, it might be more relevant to predict when selection is driven by saliency and when it is driven by goals. Second, few existing models take eccentricity into account. This is remarkable, considering the profound differences between foveal and peripheral vision (Azzopardi & Cowey, 1993; Curcio et al., 1990; Horton & Hoyt, 1991). It has been found that these differences influence behavior, including visual search (Carrasco et al., 1995; Carrasco et al., 1998; Engel, 1977) and discrimination (Anstis, 1974; Strasburger & Rentschler, 1996). Here, we have shown that they also differentially affect saliency-driven and goal-driven selection. Finally, we have shown that these effects are further modulated by past experience in the range of eccentricities used. This corroborates earlier arguments that there is important information in trial transitions in terms of selection biases (Fecteau & Munoz, 2003; Olivers & Humphreys, 2003; Theeuwes, 2019). 

In conclusion, by investigating saliency-driven and goal-driven selection as a function of both time and eccentricity, we have shown that the contribution of saliency-driven control increases as a function of eccentricity, whereas the contribution of relevance, albeit less clearly, decreases. 

Acknowledgments

The authors thank Artem Belopolsky for help with the SMART analysis. 

Funded by the Dutch Organization for Scientific Research (NWO; grant 453-16-002, to CNLO). 

Commercial relationships: none. 

Corresponding author: Elle van Heusden. 

Email: [e-mail]. 

Address: Department of Experimental and Applied Psychology, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands.