Extrafoveal preview benefit during free-viewing visual search in the monkey

Abstract

Abstract

Abstract:

Abstract  Previous studies have shown that subjects require less time to process a stimulus at the fovea after a saccade if they have viewed the same stimulus in the periphery immediately prior to the saccade. This extrafoveal preview benefit indicates that information about the visual form of an extrafoveally viewed stimulus can be transferred across a saccade. Here, we extend these findings by demonstrating and characterizing a similar extrafoveal preview benefit in monkeys during a free-viewing visual search task. We trained two monkeys to report the orientation of a target among distractors by releasing one of two bars with their hand; monkeys were free to move their eyes during the task. Both monkeys took less time to indicate the orientation of the target after foveating it, when the target lay closer to the fovea during the previous fixation. An extrafoveal preview benefit emerged even if there was more than one intervening saccade between the preview and the target fixation, indicating that information about target identity could be transferred across more than one saccade and could be obtained even if the search target was not the goal of the next saccade. An extrafoveal preview benefit was also found for distractor stimuli. These results aid future physiological investigations of the extrafoveal preview benefit.

Introduction

Visual search has served as a powerful tool to investigate several aspects of visual stimulus processing under controlled laboratory conditions in both humans and monkeys (e.g., Bichot & Desimone, 2006; Davis & Palmer, 2004; Ipata, Gee, Gottlieb, Bisley, & Goldberg, 2006; Mirpour & Bisley, 2012b; Mort & Kennard, 2003; Muller & Krummenacher, 2006). For a long period, most visual search research in humans relied upon reaction time and error rate as measures of search performance using experiments where eye movements were either not studied (A. Treisman, 1993; A. M. Treisman & Gelade, 1980; Wolfe, 2007) or where brief stimuli were used to ensure that search performance was based on information obtained while the eye remained fixed at the center of gaze (Cameron, Tai, Eckstein, & Carrasco, 2004; Verghese, 2001). Similarly, in monkeys, experiments generally required the monkey to maintain the eye at the center of gaze or to foveate the target with its eyes as a response (e.g., Bichot, Rossi, & Desimone, 2005; Mirpour, Arcizet, Ong, & Bisley, 2009; Mirpour & Bisley, 2012b; Motter & Belky, 1998a), usually with the first saccade (e.g., McPeek & Keller, 2001; Schall & Thompson, 1999). In human subjects, there has also been an active research field exploring oculomotor behavior during more naturalistic visual search where the subjects are free to move their eyes as they please (e.g., Eckstein, 2011; Geisler & Chou, 1995; Rao, Zelinsky, Hayhoe, & Ballard, 2002; Tatler, Hayhoe, Land, & Ballard, 2011; Zelinsky, 1996; Zelinsky & Sheinberg, 1997). Similar analysis of eye movements in humans during reading and scene perception has revealed much about the mechanisms that regulate how humans move their eyes while processing complex visual information (reviewed in Rayner, 1998). In order to understand the neural mechanisms operating to guide eye movements during free-viewing visual search, recent studies have begun to report such data from monkeys. Analysis of eye-fixation patterns during free-viewing visual search has revealed a generally similar pattern of results relating search time and the number of fixations (Bisley, Ipata, Krishna, Gee, & Goldberg, 2009; Nothdurft, Pigarev, & Kastner, 2009). Analysis of eye-fixation patterns while free-viewing movie clips and natural images has also shown substantial similarity between human and monkey scanpaths, with a greater importance for low-level visual saliency information in monkeys (Berg, Boehnke, Marino, Munoz, & Itti, 2009; Einhauser, Kruse, Hoffmann, & Konig, 2006; Shepherd, Steckenfinger, Hasson, & Ghazanfar, 2010). Finally, physiological studies in monkeys from areas LIP and V4 (Gee, Ipata, & Goldberg, 2010; Ipata, Gee, Bisley, & Goldberg, 2009; Ipata, Gee, Goldberg, & Bisley, 2006; Ipata, Gee, Gottlieb, et al., 2006) as well as the inferior temporal cortex (IT: Mruczek & Sheinberg, 2007; Sheinberg & Logothetis, 2001) have begun to elucidate the neural responses that lead to saccade-goal selection, target guidance, and target verification during free-viewing visual search. 

Continuing this program of research, in this manuscript, we examine the eye movements of monkeys during a visual search task in which gaze was unconstrained. Studies of reading and form processing in humans have provided evidence that post-saccadic processing of a foveal stimulus can be facilitated by information obtained during presaccadic viewing of that stimulus in the visual periphery during the immediately preceding fixation (e.g., De Graef & Verfaillie, 2002; Demeyer, De Graef, Wagemans, & Verfaillie, 2010; Henderson & Hollingworth, 2003; Kotowicz, Rutishauser, & Koch, 2010; Pollatsek, Rayner, & Henderson, 1990; Rayner, 1998; Schotter, Angele, & Rayner, 2012). Similarly, a study of curve tracing showed that monkeys can use information about a curve's path obtained during the preceding fixation (Khayat, Spekreijse, & Roelfsema, 2004). As part of our effort to understand the mechanisms guiding free-viewing visual search, we wished to ascertain the extent to which a similar extrafoveal preview benefit would manifest itself in overt measures of oculomotor behavior during a visual search task. Only one prior experiment where monkeys searched for targets in a naturalistic visual scene while freely moving their eyes has briefly reported reduced verification times for the search target when the target's eccentricity was lower during the previous fixation (Sheinberg & Logothetis, 2001). The authors inferred that “information useful for identifying the target could be acquired before the eyes foveated the target and that the amount of useful information decreased with increasing eccentricity, most likely because of reduced acuity.” Here, we find robust evidence for the existence of an extrafoveal preview benefit, and we measure and characterize its eccentricity dependence. Further, we find that the extrafoveal preview benefit in our data is obtained even when the search target is not at the saccade goal, indicating that information about target identity could be transferred across more than one saccade. Finally, we also find an extrafoveal preview benefit for distractor stimuli. Our results will aid future physiological investigations of transsaccadic transfer of form information in monkeys during naturalistic search tasks. 

Methods

Subjects

All experimental protocols were approved by the Animal Care and Use Committees at Columbia University and the New York State Psychiatric Institute as complying with the guidelines established in the Public Health Service Guide for the Care and Use of Laboratory Animals. Two male rhesus monkeys (Macaca mulatta) weighing approximately 9–10 kg each had scleral search coils to monitor eye position (Judge et al., 1980) and head restraining devices implanted during sterile surgery under ketamine and isofluorane anesthesia. Both monkeys also had recording chambers implanted on the skull over the left and right parietal cortices. Monkey R had also been trained on a memory-guided saccade task. Behavioral control and data collection (eye-position traces, stimulus events, and bar-release times) were performed on a Dell Optiplex computer using the REX system (Hays et al., 1982) running on the QNX operating system. Eye movements were sampled every millisecond. Visual stimuli were back projected on a tangent screen approximately 75 cm from the monkey in a dimly lit room by a Hitachi CP-X275 LCD projector under the control of a Dell Dimension PC running the GLVEX graphics system. The background luminance was 50 cd/m² (measured with a Minolta LS-110 luminance meter) and the projector refresh rate was 60 Hz. Stimulus timing was calculated by measuring a pulse from a photocell affixed to the back of the screen and illuminated by a small square on the corner of the same video frame as the appearance of any new stimulus. The monkey could not see the photocell or its illumination square. Monkeys were trained to sit in a primate chair with their head fixed and perform the tasks used in this experiment. Rare trials where the photocell failed to register the appearance of the search array were discarded. 

Behavioral task

We trained experimentally naïve monkeys to perform a free-viewing visual search task (Ipata, Gee, Goldberg, et al., 2006; Ipata, Gee, Gottlieb, et al., 2006) in which they were required to report the orientation of a target among seven, eleven or fifteen distractors (Figure 1), with the three set sizes presented in equal proportion. Each trial started when the monkey grabbed two bars, one with each hand. A central spot (a high contrast white square of 0.35β width) appeared, and, if the animal maintained fixation for 1–1.75 s within a 3° × 3° window, the fixation point was extinguished and the stimulus array appeared. The actual fixation accuracy prior to the first saccade was much better than the specified 3° × 3° window: The standard deviation of starting distances for the first saccade from the screen center was 0.6° for both monkeys. The monkeys had 3 s to report the orientation of the search target by releasing one of two bars; 99.9% of reaction times lay below 2 s, indicating that the monkeys were able to solve the task well in time. The monkeys were rewarded with a small drop of water or juice if they released the correct bar for that trial. During the presentation of the search array, no constraints were imposed on the monkey's eye movements, and specifically, they were not required to fixate the target before releasing a bar to indicate their response. 

A search target was present on each trial, and consisted of an upright or inverted T. The monkeys had to release the left bar if the upright T appeared in the array or the right bar if the inverted T appeared. The distractors were black cross-like stimuli, of the same width and length as the target, but whose vertical and horizontal components crossed each other at different levels. Our goal, when designing this study, was to collect a rich dataset that allowed us to measure the influence of different factors spanning a range of issues in free-viewing visual search. Therefore, there are several task manipulations that are not material to the extrafoveal preview benefit that we present here, and which we therefore do not consider in this report. However, we present the complete details of all tasks here. There were two categories of distractors: difficult distractors, in which the horizontal line crossed the vertical line near its top or bottom, and easy distractors in which the horizontal line crossed the vertical line closer to its center. There were three task types, which were run in separate blocks: the target-popout task, where the target was easily detectable as a result of being green and brighter than the distractors, which were black on a gray background; the distractor-popout task where one of the distractors was easily detectable as a result of being green and brighter than the remaining stimuli, which were black on a gray background; and the nonpopout task, where all the stimuli were black on a gray background with identical luminance. The target and distractors were positioned equidistantly along an imaginary circle at an eccentricity of 10°, with one stimulus at the location x = 10°, y = 0°. The target and distractors were 1.0° wide and 2.9° tall. The number of distractors (seven, 11, or 15), target location, and type of target (inverted T or upright T) varied randomly on each trial. The contrast of all the stimuli was well above detection threshold. 

Data analysis

We performed all data analysis using MATLAB (The Mathworks Inc., Natick, MA) and R (R Development Core Team, 2007). Results are presented as mean ± SEM. To plot smooth curves capturing the variation in scatter plots, we used local regression methods based on maximization of local likelihood (locfit: Loader, 1999) extensively, as implemented in the Chronux MATLAB package (http://www.chronux.org). A locally linear tricube kernel with a variable nearest-neighbor span and the quasigamma data family was used for these fits. We chose the nearest-neighbor parameter based on plots of the Akaike Information Criterion at different values of the parameter as well as visually inspecting the fits. We used parameter values of 0.7, 0.9, and 0.6 in Figures 2, 3, and 4, respectively. Local-regression methods are smoothing methods that offer better mean-variance compromises than the more traditional moving average over rectangular windows. As the name implies, the smoothed value of the curve at each point is obtained by performing a weighted local regression including a subset of the data that lies adjoining the point of interest on the X axis; the weights take the shape of a tricube kernel in our case and emphasize the X axis data that lie closer to the point of interest. Larger values for the nearest-neighbor parameter reflect that a larger subset of the data was used for each regression, and lead to greater smoothing. However, our results are very robust to the choice of smoothing parameter and the fits look very similar across a large range of smoothing parameter values. The same is true for other choices of smoothing kernel and data family. We present the results of the fit by plotting fitted means and simultaneous confidence bands: The simultaneous confidence bands are wider than regular confidence bands because they attempt to correct for the inherent multiple comparisons along the X axis. To test whether a single curve (plotting expected value of Y for each value of X) deviated from a horizontal line, we used a permutation test (Bowman & Azzalini, 1997) that we call the “no-effect test.” This assessed whether the effects seen could be predicted as a result of random variability arising from a no-effect model where the expected value of Y was the same for all values of X. We shuffled the relationship between the ordinate and abscissa values and repeated the local likelihood fit and then estimated the statistic D_locfit defined as (−2) × log likelihood. We then calculated the same statistic (D_noeffect) for the null no-effect model where the predicted ordinate value for each abscissa value was simply equal to the global mean of the ordinate values. We then estimated the ratio-statistic (D_noeffect − D_locfit)/D_locfit and repeated the shuffling process 1,000 times to obtain the null distribution for this ratio statistic. By comparing the ratio statistic from the actual data to the percentiles of this null distribution, we obtained a p value for the likelihood that a ratio statistic as large as that in the actual data could be obtained from the null no-effect model. We also used another permutation test to assess whether fitted curves were significantly different for two different groups with the same X and Y parameters, but with different cofactors associated with each group (H. S. Bokil and P. P. Mitra; personal communication, June 11, 2007); we refer to this as the cofactor test. We defined a maximum absolute deviation statistic MAD = max(|f(x) − g(x)|) where f and g are the two fitted curves computed from Groups 1 and 2, respectively; in general f arises from m samples and g arises from n samples. We then pooled the two samples into one large m + n sized sample, randomly assigned m of these to Group 1 and n to Group 2 and recomputed MAD. We repeated this process 1,000 times to estimate the null distribution for MAD, under the assumption of no-group effect; i.e., f = g. We then compared the actual observed value of MAD to the percentiles of this null distribution to obtain a p value for the possibility that a maximum absolute deviation between the two curves as large as the one observed could arise entirely by chance. 

Before analyzing the eye-position data, we first filtered eye-position traces using a rectangular boxcar filter of 5 ms duration, and then detected saccades using an algorithm based on detection of the times at which eye-velocity crossed a threshold that was set by eye (at a level slightly above the random noise in the eye-position signal); this setting was aimed to catch small within-stimulus saccades while keeping the false-positive rate from a purely noisy signal close to zero. We defined saccade onset as the time at which eye velocity crossed the threshold in an upward direction, and defined the end of the saccade as the time at which the eye velocity recrossed the threshold in a downward direction. Additionally, saccades had to be at least 5 ms long in order to qualify. This algorithm performed robustly under our conditions and we verified this by visually inspecting the eye position and saccade traces. The threshold was held constant at 0.1°/ms, and determined solely by viewing the eye traces and before analyzing the data. We note that these threshold values should be interpreted only in context of the filtering parameters that we used while preprocessing the eye traces: For reference, the mean peak velocity for 1°, 2°, 5°, and 10° amplitude saccades in both monkeys were approximately 0.20°/ms, 0.3°/ms, 0.7°/ms, and 1.2°/ms. 

As in a previous study of free-viewing visual search in monkeys (Motter & Belky, 1998a), the monkeys moved their eyes on every trial, directing most saccades to one of the objects in the array and rarely towards blank regions of space in between or away from stimuli. We calibrated our eye-position measuring system at the beginning of each session by having the monkeys fixate a series of fixation stimuli at different points on the screen. We used an iterative procedure to assign the nearest stimulus to each saccade endpoint for a given session: After excluding saccades made after foveating the target, or the last saccade in the trial if the target was not foveated, we first assigned each saccade endpoint in that session to the nearest stimulus based on the raw stimulus location, then collected together the endpoints belonging to each stimulus and used its median as the new estimate of stimulus location and then reassigned each saccade endpoint based on this new estimate. By repeating this iterative procedure twice, we obtained better estimates of the nearest stimulus by reducing the influence of drift and rotational calibration errors. All distances of saccade endpoints to stimulus locations were then calculated using this final corrected set of stimulus locations. For 96.3% and 90.4% of saccades in Monkey Z and Monkey R, respectively, the nearest stimulus assigned to that saccade endpoint did not change as a result of going through the correction procedure. While the actual stimulus to which the saccade was determined to have been targeted rarely changed, mean distances from the nearest stimulus went from 1.32° (SEM = 0.004°) using the raw stimulus location to 1.02° (SEM = 0.003°) using the corrected stimulus location in Monkey Z and from 2.03° (SEM = 0.008°) before correction to 1.00° (SEM = 0.006°) after correction in Monkey R. However, we emphasize that our results remained robust even when we repeated the analysis using simple uncorrected eye positions, and the influence of the correction procedure was minor in practice since the magnitude of calibration error is small compared to the effect sizes we discuss. 

The task required the monkeys to indicate the orientation of the target T, but did not require them to fixate the target. Despite this, on upright-T trials, both monkeys fixated in the target on over 99% of trials. Monkey R showed a very similar behavior during inverted-T trials, again fixating the target on over 99% of trials, but Monkey Z fixated the inverted-T target on only 54% of the trials, a difference that was strongly significant (p < 0.0001, test of proportions). However, despite not fixating the inverted-T target on a substantial proportion of trials, Monkey Z actually performed slightly better overall on inverted-T trials (99.8%) compared to upright-T trials (92.9%) in the distractor popout and nonpopout tasks. Our analysis indicates that while Monkey R searched for, detected, and foveated both the upright and inverted-T target stimuli before indicating his response, Monkey Z only did so on upright-T target trials. On inverted-T target trials, he followed a very successful mixed strategy: He indicated an inverted-T response when he found and fixated the inverted-T stimulus, and also whenever he failed to find a target stimulus (upright T or inverted T) in the display. The strategy was successful because Monkey Z was very good at detecting the upright-T stimulus and therefore if he failed to locate a target stimulus within a few saccades, he could be quite certain that the display contained an inverted T rather than an upright T (since there were no catch trials). We present several additional lines of evidence supporting our interpretation in the Appendix. Keeping this difference in mind, we present the results differently for the two monkeys in Figures 4 and 5, so that we only included trials where the monkeys correctly performed the task based on searching for and finding the target. For Monkey R, we used the results from both upright-T and inverted-T trials (and pooled them since they showed similar dependencies), but for Monkey Z, we only used the results from upright-T trials. Also, to prevent the popout distractor from influencing the analysis, we omitted trials where the saccade being analyzed originated from or was directed to the popout distractor, since these could, in principle, be affected by a different set of mechanisms specific to ignoring the popout distractor (Ipata, Gee, Gottlieb, et al., 2006). Again, the impact of this exclusion in practice was negligible. 

While most saccades landed near one of the stimuli in the search array, both monkeys made “within-stimulus” saccades that started and ended close to the same stimulus. These saccades were usually made after foveating the target (25.3% of target-foveating saccades compared to 5.4% of distractor-foveating saccades for Monkey Z and 44.4% compared to 2.9% for Monkey R). This is probably because mean dwell times on the target (224 ± 0.6 ms in Monkey Z and 261 ± 0.9 ms in Monkey R) were substantially longer than those on the distractor (145 ± 0.6 ms in Monkey Z and 119 ± 0.7 ms in Monkey R) and the probability of a within-stimulus saccade was very highly correlated with the length of the dwell time on that stimulus (p < 0.0001 for both monkeys in a generalized linear model fit). These within-stimulus saccades were small in amplitude (1.15 ± 0.02° in Monkey Z, 0.9 ± 0.01° in Monkey R) and comparable to the dimensions of the stimuli (1.0° wide and 2.9° tall). We did not include these small within-stimulus saccades when counting the number of saccades; only saccades that moved the eye from one stimulus to another were included. In any case, this decision did not have any practical effect on our conclusions, since as mentioned above, only a small proportion of distractor fixations contained a within-stimulus saccade. 

Results

We collected data from two monkeys. Since the extrafoveal preview benefit that we demonstrate here (Figures 1–5) shows similar dependencies in the distractor popout and nonpopout blocks and across the three different set sizes, we merged the data across the two task types and three set sizes and present results for the combined dataset. Monkey Z performed a total of 13,159 trials over 10 sessions (6,116 distractor-popout and 7,043 nonpopout), while Monkey R performed a total of 6,057 trials over five sessions (2,758 distractor-popout and 3,299 nonpopout). Monkey Z released the correct bar on 96.2% and 96.3% of distractor popout and nonpopout trials, respectively; Monkey R released the correct bar on over 99% of trials on both tasks. Because of the low proportion of errors, we limited our analyses to correct trials unless stated otherwise. 

Figure 1

 

Schematic diagram of the main task. Monkeys fixated a central fixation point while holding two bars, one with each hand. An array of search stimuli containing one target (an upward or downward facing T) and seven, 11, or 15 distractors was then presented at an eccentricity of 10°. The monkeys were free to move their eyes as they wished and were rewarded if they released the bar that corresponded to the target stimulus present in the array.

We first present data about the monkeys' search strategy and scanpath in order to provide context to our analyses of the extrafoveal preview benefit. Both monkeys released one of the two bars after search array onset on every trial they initiated successfully. Manual reaction time on the three tasks, defined as the time from search array onset to the release of the bar, increased systematically with set size in a manner similar to that found in humans (Maioli, Benaglio, Siri, Sosta, & Cappa, 2001; A. M. Treisman & Gelade, 1980; Zelinsky & Sheinberg, 1997) and monkeys (Balan, Oristaglio, Schneider, & Gottlieb, 2008; Cohen, Heitz, Woodman, & Schall, 2009); the slopes were statistically significant, as we reported earlier (Bisley et al., 2009). The monkeys moved their eyes on every trial, directing most saccades to one of the objects in the array. Saccades were stimulus directed and rarely directed towards blank regions of space in between or away from stimuli, as in a previous study of free-viewing visual search in monkeys (Motter & Belky, 1998a). Both monkeys made their first saccade with short latencies (mean = 132 ms, SEM = 0.1 ms in Monkey Z, mean = 152 ms and SEM = 0.2 ms in Monkey R). These values are comparable to previous results from visual search tasks where monkeys were free to make multiple saccades (Motter & Belky, 1998a). Over 90% of saccade endpoints lay within 2° of the center of the nearest stimulus for both monkeys at all three set sizes, a distance comparable to the stimulus dimensions themselves (1.0° wide and 2.9° tall). 

Extrafoveal preview benefit depends on eccentricity of saccade goal

To analyze the extrafoveal preview benefit for target processing, we focused on trials in which the monkey made a first saccade towards a distractor stimulus and then foveated the target with the second saccade (26.9% of all trials in Monkey Z, and 26.3% of all trials in Monkey R). We analyzed the time from target-foveation to bar release (Figure 2B: verification time, i.e., the reaction time to bar release starting from when the monkey fixated the search target with the second saccade). We examined how the verification time after the second saccade varied depending on where the monkey was looking after the first saccade. In both monkeys, for both inverted-T and upright-T trials, there was a systematic pattern of verification time decrease as the distance of the first saccade endpoint from the search target (D_fse) decreased (Figure 2C; no-effect test: p < 0.0001). We interpret the decrease of verification time when the first saccade ends closer to the search target as being the result of an extrafoveal preview benefit, where some processing of the target occurred prior to the second saccade and aided discrimination of the target at the fovea following the second saccade. 

The verification time appears to increase with D_fse up to roughly 10° to 15° and then shows a much shallower slope (Figure 2C). For our stimulus arrangement, this encompassed approximately one to three distractors adjoining the search target, depending on set size. Corroborating this impression, a plot of the local slope (Figure 2D) showed a steady or increasing slope until about 8° (11° for inverted-T trials in Monkey Z) followed by a sharp decrease in slope. In three out of four cases, there was no significant variation of verification time for D_fse values greater than 12.5° (no-effect test performed only for D_fse values greater than 12.5°: p > 0.17); in upright-T trials in Monkey Z where there was a slight but significant late increase (no-effect test for D_fse values greater than 12.5°: p < 0.0001). The variation of verification time for D_fse values less than or equal to 12.5° remained highly significant in all cases (p < 0.0001). Finally, in Monkey Z, the verification time was higher for upright-T trials, while the reverse was true for Monkey R (cofactor test, p < 0.0001), consistent with the strategies used by each monkey to perform the task (see Methods). 

After foveating the target, Monkey R continued to foveate it until he released one of the two bars on 97.3% of trials. On the other hand, after foveating the target, Monkey Z continued to foveate the target until bar release on only 45.2% of trials; on the remaining trials, he made a saccade away from the target to a neighboring stimulus before releasing the bar. On trials in which Monkey Z's eye left the target after foveating it, the time spent fixating the target (Figure 3B: TDT) was therefore shorter than the verification time (by 138 ms ± 1.7 ms in inverted-T trials and 141 ms ± 3.5 ms in upright-T trials). However, despite this difference in absolute magnitude, the extrafoveal preview benefit measured in terms of time spent fixating the target in both monkeys was very similar in spatial dependence and magnitude to the extrafoveal preview benefit on verification time (Figure 3C; no-effect test for all four curves: p < 0.0001). 

Figure 2

 

(A) Schematic diagram of trials where monkey foveated the search target with the second saccade. Numbers alongside the search stimuli indicate that they were the endpoints of the respective saccades. (B) Schematic of time course of these trials. 1s, 2s, and 3s (1e, 2e, and 3e) represent the start (end) of the first, second, and third saccades. RT is the release of the bar and zero represents the appearance of the search stimuli (trial start). VT is the verification time and TDT the target dwell time. (C) Monkeys show an extrafoveal preview benefit during visual search: They take less time to release the bar after foveating the target (verification time) with the second saccade if their eyes were closer to the target after the first saccade. Ordinate: Verification time (ms) for second saccades that foveate the target; Abscissa: distance of first saccade endpoint from search target (D_fse in degrees). Blue and red curves are estimated means and 95% simultaneous confidence bands for time from target foveation to bar release for inverted-T and upright-T trials, respectively. Gray circles and squares show data-points from inverted-T and upright-T trials respectively. (D) The decrease in extrafoveal preview benefit with D_fse is sharp at low D_fse values and then becomes shallower above about 8° (11° for inverted-T trials in Monkey Z). Format as in C, except that ordinate plots the local estimate of the slope of the curves shown in C (ms/degree). (C), (D) Data from Monkey Z in the left column and data from Monkey R in the right column.

Figure 3

 

The extrafoveal preview benefit is also seen if target dwell times (TDTs) are plotted. Format identical to Figure 2C, except that the ordinate plots TDTs instead of verification times. TDTs are shorter than verification times for Monkey Z because this monkey often made a saccade after fixating the search target, and almost identical to verification times in Monkey R because the monkey almost never made another saccade after fixating the search target.

We tested generalized additive models as well as bivariate regression models that incorporated two more explanatory variables: the first saccade latency as well as the distance of the second saccade endpoint from the target. We systematically verified that the extrafoveal preview benefit as well as its dependence upon the distance of the first saccade endpoint from the target remained mostly unchanged, even after including these variables. The dependence of verification time upon these added variables was small and not consistent across the two monkeys. The extrafoveal preview benefit we report is not an artifact due to confounding with these variables. 

In both monkeys, first-saccades that did not foveate the target tended to be directed to stimuli closer to the search target (p < 0.0002 for all three set sizes in both monkeys, chi-squared test for deviations from a uniform distribution). This suggests that the search-target advantage in attracting the first saccade spreads to neighboring stimuli. This becomes relevant for the interpretation of the eccentricity dependence of the extrafoveal preview benefit (see Discussion). 

Extrafoveal preview benefit accrues from stimuli other than the saccade goal and can be transferred across more than one saccade

We next analyzed trials where the monkey first foveated the target on the third saccade to test whether information about target identity could be transferred over more than one saccade (Figure 4A; 5.5% of all trials in Monkey Z and 6.2% of all trials in Monkey R). We first confirmed that the standard extrafoveal preview benefit demonstrated above (Figure 2) was also present for trials in which the monkey fixated the search target with the third saccade: The verification time after the third saccade showed a similar dependence on the distance of the second saccade endpoint from the search target (Figure 4C; no-effect test: p < 0.0001). Additionally, the verification time also depended on the distance of the first saccade endpoint from the search target. In both monkeys, when the second saccade ended far away from the target (greater than 10°), the verification time after the third saccade increased with the distance of the first saccade endpoint from the search target up to about 10° (Figure 4D; no-effect test: p < 0.0001). When the second saccade ended nearer the target (less than or equal to 10°), a clear dependence upon the distance of the second saccade origin from the target was seen only in Monkey Z (no-effect test: p < 0.0001; p = 0.494 in Monkey R). The dependence of verification time after the third saccade on the distance of the first-saccade endpoint from the target indicates that information obtained during the fixation following the first saccade could be used to aid target processing in the fovea following the third saccade. Further, it also shows that information could be obtained about the search target during the first intersaccadic interval (ISI) even if it was not the goal of the impending second saccade. We note that in Figure 4D, verification times appear to be generally longer for all first-saccade endpoint distances from the target, if the second saccade ended far from (greater than 10°) rather than near (within 10°) the target. This effect is statistically significant (cofactor test; p = 0.037 in Monkey Z and p < 0.0001 in Monkey R) and results from the standard extrafoveal preview benefit discussed above (Figure 4C). 

Figure 4

 

(A), (B) Schematic diagrams of trials where monkeys foveated the search target with the third saccade. Format as in Figure 2A, B. (C) An extrafoveal preview benefit similar to that in Figure 2C is also seen when monkeys foveated the search target with the third saccade. Abscissa: Distance of second saccade endpoint from search target. Format otherwise identical to Figure 2C. (D) Extrafoveal preview benefit can carry over more than one saccade: Monkeys take less time to release the bar after foveating the target with the third saccade if their eyes were closer to the target after the first saccade. Ordinate: Verification time (ms) for targets foveated by third saccades. Abscissa: distance of first saccade endpoint from target (degrees). Blue and red curves indicate mean and 95% simultaneous confidence bands for trials where the second saccade ends within 10° of the target and trials where the second saccade ends more than 10° away from the target. Data from Monkey Z in the left column and data from Monkey R in the right column. Data from upright-T trials only for Monkey Z and pooled from upright-T and inverted-T trials for Monkey R (see Methods); trials with second saccades starting from or ending at the popout distractor omitted for both monkeys.

Extrafoveal preview benefit also aids distractor processing

We also examined the time spent fixating the distractor after a second saccade on trials in which the monkey made at least three saccades (Figure 5A; 1.4% of all trials in Monkey Z and 6.1% of all trials in Monkey R). For both monkeys, there was a dependence of distractor dwell time after the second saccade upon the distance of the first saccade endpoint from the distractor (Figure 5C; no-effect test: p < 0.0001). While the extrafoveal preview benefit that we observed using the verification time could be interpreted as resulting from transsaccadic improvements to either visual target processing or the manual bar-release process (or both), the preview effect based on distractor dwell times cannot be a result of an effect on the bar-release process. We therefore interpret it as resulting from a transsaccadically transferred extrafoveal preview benefit for visual/oculomotor processing during the distractor fixation. Target-distractor similarity produced a clear (and expected) effect on search guidance. In both monkeys, the majority of nontarget-directed saccades went towards a hard distractor (69.8% in Monkey Z and 73.8% in Monkey R). We did not find any significant difference between the extrafoveal preview benefit for hard and easy distractors (cofactor test, p = 0.6080 in Monkey Z and 0.2000 in Monkey R). Finally, we note that the occasional short ISIs visible in Figure 5 are consistent with prior reports of oculomotor behavior in humans and monkeys during search tasks (McPeek & Keller, 2001; McPeek, Skavenski, & Nakayama, 2000). 

Figure 5

 

(A), (B) Schematic diagram of trials where monkey foveated a distractor with the second saccade. Format as in Figure 2A, B. (C) Extrafoveal preview benefit also accrues for distractor stimuli: Monkeys spend less time foveating the distractor after their second saccade if the first saccade ended closer to the distractor. Ordinate: Distractor dwell time (DDT: ms). Abscissa: Distance of first saccade endpoint from distractor (degrees). Black curves show means and 95% simultaneous confidence bands. Format as in Figure 2C.

Discussion

We found an extrafoveal preview benefit in two monkeys during free-viewing visual search: the amount of time taken to release the bar after first foveating the target decreases if the target was close to the fovea on the previous fixation. This is consistent with the results from a prior study (Sheinberg & Logothetis, 2001) in monkeys, also during free-viewing visual search and indicates that processing of the foveal stimulus following the saccade can be integrated with information obtained extrafoveally about the same stimulus in the fixation preceding the saccade. Related results have previously been obtained in other visual tasks like reading and form processing in humans (e.g., De Graef & Verfaillie, 2002; Irwin, 1991; Pollatsek, Rayner, & Collins, 1984; Rayner, 1998; Schotter et al., 2012) as well as curve tracing in monkeys (Khayat et al., 2004). The extrafoveal preview benefit (reduction in verification time when the previous fixation was close to the target) decreased smoothly with eccentricity of the target from the fovea during the previous fixation, suggesting that the amount of transsaccadically available information about stimulus identity also decreases smoothly with eccentricity, up to an eccentricity of about 10° in our task. The drop-off in information with eccentricity can be plausibly related to the well-known drop in psychophysical performance with eccentricity (e.g., Geisler & Chou, 1995; Makela, Whitaker, & Rovamo, 1993; Strasburger, Rentschler, & Juttner, 2011; Virsu, Nasanen, & Osmoviita, 1987), and whose effects on search time and target-detection probability have been explored in detail in the context of visual search (Carrasco, Evert, Chang, & Katz, 1995; Carrasco, Giordano, & McElree, 2004; Carrasco, Talgar, & Cameron, 2001; Carrasco & Yeshurun, 1998; Motter & Belky, 1998a, 1998b; Motter & Holsapple, 2007; Motter & Holsapple, 2000; Motter & Simoni, 2007). A similar reduction in extrafoveal preview benefit with target eccentricity has been found in studies of reading and object identification in humans (e.g., Henderson, Pollatsek, & Rayner, 1987; Kennison & Clifton, 1995; Kliegl, Hohenstein, Yan, & McDonald, 2013; McDonald, 2006; Pollatsek et al., 1984) as well as in the prior study using free-viewing visual search in monkeys (Sheinberg & Logothetis, 2001). Interestingly, a similar pattern of reduction in performance with eccentricity to ours, where performance drops sharply at small eccentricities and then decreases more slowly at larger eccentricities, can be seen in a previous study (Virsu et al., 1987). Our work extends the large body of prior research that has focused on the spatial window for target detection and its relationship to visual search (conceptualized as the conspicuity area, or the functional visual field, or the useful field of view: Engel, 1977; Geisler & Chou, 1995; Motter & Belky, 1998a, 1998b; Motter & Holsapple, 2000; Motter & Simoni, 2007, 2008) by demonstrating a spatial window from which there is an influence of extrafoveally obtained stimulus-identity information on search decision. 

Our interpretation above of the eccentricity dependence of the extrafoveal preview benefit assumes that extrafoveal information about the foveated stimulus was accumulated during the preceding fixation. For example, for second saccades that foveated the search target, the extrafoveal preview benefit depends on the distance of the first saccade endpoint from the search target because this determines the eccentricity of the search target during the fixation following the first saccade. However, there is another alternative explanation for the eccentricity dependence that we cannot rule out, in which extrafoveal information is actually being obtained prior to the first saccade, and the amount of information obtained depends on the distance of the search target from the first saccade goal. Indeed, both explanations may be simultaneously valid. In both monkeys, first saccades that did not foveate the target tended to be directed to stimuli closer to the search target, indicating that the search-target advantage in attracting the first saccade spreads to neighboring stimuli. It is possible that the saccade goal specification is spatially diffuse, with the saccade being directed to one of the stimuli within this specification. In a closely related variant of this argument, the presaccadic attentional focus around the goal of the first saccade may extend over adjoining stimuli (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995). We emphasize that even in this scenario, information obtained extrafoveally prior to stimulus foveation is being used to aid stimulus verification after foveation. 

The extrafoveal preview benefit on verification time could reflect information specific to target orientation that allows the monkey's decision process to release the correct bar to begin before the foveating saccade. Alternatively, it could simply reflect an alerting effect of knowing that the stimulus about to be foveated is the target; this would allow the monkey to begin preparing a bar release (by either hand) in anticipation of foveating the target. Such anticipatory representations of priority have been found in the lateral intraparietal area of the monkey (Mirpour & Bisley, 2012a). In either case, it can be inferred that information about target identity is carried over transsaccadically. However, the extrafoveal preview benefit also impacts the processing of distractor stimuli in our task and this must arise from an effect on visual/oculomotor processing during the distractor fixation (as is the case in the previous evidence from reading studies) and not merely from an alerting effect on manual bar release. Our finding of an extrafoveal preview benefit and the beneficial transsaccadic transfer of saccade goal identity information during visual search is complementary to the prior literature showing that transsaccadic transfer of search target location information can aid saccadic targeting during visual search (Caspi, Beutter, & Eckstein, 2004; McPeek & Keller, 2001; McPeek et al., 2000). 

By analyzing trials where the target was foveated on the third saccade, we found that the verification time in this case depends on the distance of both the first and second saccade endpoints from the target. The increased verification time if the second saccade ended far from the target is another instance of the extrafoveal preview benefit transferred across a single saccade, as discussed above. The dependence of the verification time following the third saccade on the distance of the first saccade endpoint from the target suggests that an extrafoveal preview benefit was also obtained during the fixation after the first saccade and this was carried across the second and third saccades to impact verification time. That is, visual information can transfer and accumulate across more than one saccade in our task; we note that one study of human reading reports otherwise (McDonald, 2005). Further, information about the search target can be obtained during the fixation after the first saccade, even though the following second saccade is not directed towards the search target. Thus, extrafoveal preview benefits can be obtained even from a stimulus that is not at the saccade goal, confirming results from studies of human object processing (Gajewski & Henderson, 2005; Prime, Vesia, & Crawford, 2011) and the results from some (but not all) studies of reading (Kliegl et al., 2013; Schotter et al., 2012). We can only offer speculative explanations about why there was no dependence on the first saccade endpoint distance if the second saccade ended close to the target in Monkey R. First, it is possible that local competition between the second saccade target and the neighboring stimuli prevented information accrual from these neighboring stimuli in this monkey (Schall, Sato, Thompson, Vaughn, & Juan, 2004). However, the benefit seen in the second ISIs suggests that this may not be the case. Alternatively, target identification when the second saccade ends close to the target may be close to optimal and therefore an additional advantage from the first saccade endpoint may not be visible due to a floor effect on verification times. However, despite this difference, there was evidence from both monkeys to support the transsaccadic carry-over of stimulus information across more than one saccade. 

The demonstration of a distance-dependent extrafoveal preview benefit in monkeys during free-viewing visual search lays the ground for further physiological investigations of this process. There are at least two ways in which the extrafoveal preview benefit may be implemented in the brain (Melcher & Colby, 2008). First, the information being carried across a saccade may reside in a central buffer (presumably an area in the brain like the inferior temporal cortex [IT] or the prefrontal cortex with large receptive fields capable of holding information related to stimulus identity). This information does not have to be explicitly about visual featural identity; it may instead take the form of a decision variable regulating the decision to move the eye to the next stimulus or to release the bar, and whose value can be updated based on extrafoveally viewed stimuli and stored across saccades. Available data indicate that the response to an extrafoveally presented stimulus in area IT predicts verification time during a free-viewing search task using naturalistic scenes. Neurons responded shortly before effective targets were fixated, and the magnitude of this activity varied with both target eccentricity and verification time (Sheinberg & Logothetis, 2001). The variation in target eccentricity was controlled for in a later experiment (Mruczek & Sheinberg, 2007) and the perisaccadic activity of IT neurons continued to be inversely correlated with verification time. Interestingly, in these data, IT neurons only responded to the target (their preferred stimulus) if the saccade was directed towards the target, suggesting that a different explanation must be found for our finding that a target-preview benefit could be obtained even if the search target was not at the saccade goal and that the extrafoveal preview benefit for the search target could extend over more than one saccade. Alternatively, the extrafoveal preview benefit may be implemented in lower level form-selective visual areas with smaller receptive fields, and then be remapped across each saccade: Psychophysical evidence for such feature-selective remapping is currently controversial (Knapen, Rolfs, & Cavanagh, 2009; Knapen, Rolfs, Wexler, & Cavanagh, 2011; Melcher & Colby, 2008). These two alternatives may correspond to the distinctions drawn by previous authors between a short-lived, maskable visual analog and a longer-lived, abstract visual short-term memory (Germeys, De Graef, & Verfaillie, 2002) or that between a location-dependent and a location-independent extrafoveal preview benefit (Gajewski & Henderson, 2005). We note that both these alternatives are distinct from the mechanisms for the transsaccadic transfer of potential target location information, for which there is good psychophysical and physiological evidence (Caspi et al., 2004; McPeek & Keller, 2001, 2002; McPeek et al., 2000; Mirpour et al., 2009; Mirpour & Bisley, 2012a). Transsaccadic transfer of target-location information can be accomplished in a feature-blind priority map like area LIP or the superior colliculus based on (winner take all or similar) operations performed on transformed visual input. Transsaccadic transfer within this priority map may not contain information about the identity of the stimulus that is about to be foveated, even though it maintains a priority map of interesting locations to look at that is updated after very saccade. 

In principle, it is possible that the extrafoveal preview benefit may share mechanisms with the well-documented target preview benefit in visual search, which refers to the advantage obtained by previewing a target example immediately before the appearance of the search array (e.g., Castelhano & Heaven, 2010; Schmidt & Zelinsky, 2011). Though they are measured in different experimental contexts, both phenomena could be considered to be examples of priming where viewing a pattern improves detection and recognition of the same pattern when viewed later. However, we think it is likely that there are at least some differences between the mechanisms underlying these two phenomena. First, the target preview benefit likely operates via an improvement of the search target template that is stored in visual memory and used to guide the search process. In this case, if the extrafoveal preview benefit and the template-related target preview benefit use the same mechanism, then a potential problem seems to arise where the target-template guiding search would be corrupted by the visual information obtained during each fixation if this visual information (or a derivative of it) is transferred to the same memory store as the template. This is especially true since we show that the extrafoveal preview benefit extends to distractor stimuli. Alternatively, if the target preview benefit operates independent of the search template (which we consider less likely), then the target preview benefit should decrease in magnitude as search progresses because the target preview gets corrupted by the distractor extrafoveal previews, which is a testable prediction. Essentially, the difference here is that in the target preview benefit, the target is marked as such by the fact that it is presented alone and before the search array and therefore can be used by the searcher unambiguously, while the extrafoveal preview benefit operates over both the target and distractor stimuli in the search array and the searcher has to figure out which is which while searching. Second, the target preview benefit involves an improvement of the memorized template prior to search and is presumably stored in the activity patterns of a feature-sensitive memory area of the brain. Alternatively, the target preview benefit could act by selectively enhancing the sensory responses to the target stimuli once the search array appears (akin to feature-based attention). There is some physiological support for these ideas (e.g., Bichot et al., 2005; Chelazzi, Miller, Duncan, & Desimone, 1993). In both cases, these are spatially global effects that do not have to take saccades into account or be updated after each saccade. Template improvement in a visual memory area would be independent of spatial location and even the enhancement of target-related sensory responses would operate globally across the entire visuospatial representation in order to impact search guidance. One could, in principle, argue that the extrafoveal preview benefit could also result from a global eccentricity-weighted comparison of the entire search array to the memorized template. Then the results of this comparison are integrated across fixations and therefore, there is no need to keep track of the results of the comparison of each stimulus against the template. Such an explanation makes a strong prediction that the specific eccentricities and target similarity of the search stimuli would affect the extrafoveal preview benefit and we aim to test this in a future study. Future monkey physiological experiments could also help clarify and disambiguate the mechanisms underlying the extrafoveal and target preview benefits. We also note that essentially similar issues arise when comparing the extrafoveal preview benefit to the other context in which the term preview benefit is used in the visual search literature: namely, the advantage obtained in search when some of the distractors appear before the remaining stimuli in the search array (Dent, Allen, Braithwaite, & Humphreys, 2012). This distractor preview benefit is considered to operate via inhibition of the stimuli in the search array that share the location and/or features of the previewed distractor stimuli (Dent et al., 2012; Mavritsaki, Heinke, Humphreys, & Deco, 2006), and survives for several hundred milliseconds after the appearance of the search array (Humphreys, Stalmann, & Olivers, 2004). In addition to the two reasons mentioned above for why the extrafoveal preview benefit might differ from the target preview benefit, there is the additional difference here that the distractor preview benefit emerges from location and feature-specific inhibition of information from the search array, while the extrafoveal preview benefit represents information integration from different views of the search array separated by one or more saccades. 

At least three theories currently exist with the ambition of becoming comprehensive models of visual search: the Target Acquisition Model (TAM: Zelinsky, 2008), the Area Activation Model (AAM: Pomplun, 2007), and Guided Search 4.0 (GS4: Wolfe, 2007). Though GS4 can produce eye movements, it largely focuses on measures like reaction time and set-size slopes than on modeling scanpath patterns during free-viewing search. TAM and AAM do attempt to model scanpaths during free-viewing visual search where the eyes are free to move, but they focus on modeling fixation locations and do not attempt to reproduce plausible or empirically validated dwell times or model the factors that determine dwell time, at least in their most recently published versions. Based on our data, we can argue that when a comprehensive model that accounts for both the spatial and temporal properties of search scanpaths emerges, it will need to include mechanisms that accumulate information about the search array, integrate it across saccades, and use the accumulated information to determine stimulus dwell times. Our data add an additional factor (extrafoveal preview benefit) that governs fixation durations during free-viewing visual search. Further, we show that the extrafoveal preview benefit can extend over more than one saccade, influences both target and distractor verification processes, and can apply to stimuli other than the saccade goal. Full models of the sequencing of eye movements during visual search will therefore have to take these effects into account. 

Acknowledgments

We thank Catherine Loader, Hemant Bokil, and Partha Mitra for help with locfit and Hans Strasburger for helpful discussions. We also thank Prof. Greg Zelinsky for a meticulous review and for helpful suggestions, including the alternative explanation for our findings based on the spatial spread of the presaccadic attentional focus. We thank M. Osman and G. Asfaw for veterinary care, Y. Pavlova for expert assistance with animal care, G. Duncan for electronic and systems work, and L. Palmer for her indispensable help. Preliminary experiments were performed at the Laboratory of Sensorimotor Research, National Eye Institute, Bethesda, Maryland. The research was supported by grants to M.E.G. from the National Eye Institute (R01 EY014978-01, R24 EY015634-01, R01EY017039, P30EY019007) the Whitehall, James S. MacDonnell, W. M. Keck and the Zegar Family Foundations, and the David Mahoney Chair at Columbia University. 

Commercial relationships: none. 

Corresponding author: B. Suresh Krishna. 

Email: [e-mail]; [e-mail]. 

Address: German Primate Center, Goettingen, Germany. 

Footnotes

*  These two authors contributed equally.

Appendix

Several lines of evidence support our interpretation of Monkey Z's strategy. (A) Monkey Z's accuracy on inverted-T trials in distractor popout and nonpopout tasks was identical (about 99.8%) whether or not he fixated the target; the small differences in accuracy of less than 0.1% were not statistically significant (p ∼ 1). (B) However, Monkey Z was almost always wrong on upright-T trials if he did not fixate the target: His accuracy dropped to 9.4% and 2.3%, respectively, during the distractor popout and nonpopout task; compared to 99.1% and 99.0% respectively on trials in which he fixated the upright-T target. These differences are highly significant (p < 0.0001). (C) Monkey Z did realize the significance of the inverted T: During distractor popout and nonpopout trials in which he did fixate the target on the first saccade, he made a correct bar response faster (mean = 305 ms, SEM = 4.2 ms) during inverted-T trials than upright-T trials (mean = 341 ms, SEM = 1.6 ms; p < 0.0001, t test). The faster response after fixating the inverted-T target may have been a result of prior motor preparation as a result of the strategy biased towards making the response corresponding to the inverted-T target. In contrast, Monkey R took longer (292 ms, SEM = 2.8 ms) on average to release the bar after fixating the inverted-T target than after fixating the upright-T target (259 ms, SEM = 1.6 ms). (D) The slope of the manual reaction time versus set-size relationship in Monkey Z for trials with inverted-T targets is markedly higher than for trials with upright-T targets (13.1 ms/stimulus for trials with inverted-T targets compared to 5.2 ms/stimulus for trials with upright-T targets; p < 0.0001). This is consistent with the use of an “upright-T-absent” strategy on inverted-T trials; target-absent searches are known to have higher slopes when compared to target-present searches (e.g., Zelinsky & Sheinberg, 1997). (E) An alternative possibility is that Monkey Z performed the task covertly with very high accuracy on inverted-T trials by detecting and discriminating the inverted-T stimulus without actually fixating it. If this were so, due to the high accuracy with which the monkeys discriminated the inverted-T target, one might expect their eyes to get close to the target stimulus on inverted-T trials, even if they didn't actually fixate the inverted-T stimulus. This was not the case. The closest stimulus Monkey Z fixated on inverted-T trials was a mean of two stimuli away from the target when the set size was eight, and 2.86 and 3.7 stimuli away from the target for set sizes of 12 and 16, respectively; the medians were two, three, and four respectively. In other words, during inverted-T trials where he didn't fixate the target, Monkey Z got no closer to the target stimulus on average than to the stimulus exactly opposite the target stimulus.