A functional role for trans-saccadic luminance differences

Abstract

Abstract

Abstract:

Abstract  In typical natural environments, the visual system receives different inputs in quick succession as gaze moves around. We examined whether local trans-saccadic differences in luminance, contrast, and orientation influenced perception and target selection in the eye movement system. Observers initially fixated a peripheral position in a preview display that consisted of four patterns. They subsequently made a saccade to the center of the configuration. During the movement, two of the preview patterns were eliminated, and a small change in the luminance contrast of the remaining patterns was introduced. Observers had to make a second saccade to the test patch with the greater luminance contrast relative to the background. During the second fixation, test patterns could be in the same retinotopic location as one of the preview patterns during the initial fixation (a retinotopic match) or at a retinotopic location that was empty during the preview epoch (a retinotopic onset). We consistently found a preference to fixate retinotopic onsets over retinotopically matched patterns, but only when the patterns were defined by a luminance difference. Direct measurement of perceived luminance showed that the visual response to retinotopically matched inputs was attenuated, possibly because of retinotopic adaptation. As a consequence, the visual system responds more strongly to trans-saccadic differences in local luminance. We argue that a trans-saccadic comparison of the local luminance at the same retinotopic location is a simple way of finding high spatial frequency edge information in the visual scene. This information is important for image segmentation and interpretation.

Introduction

The visual world is sampled during brief periods of stable fixation. Fixations are interspersed with rapid movements of gaze that shift the line of sight to a different object or region in the visual image. As a result, from one fixation to the next, the image on the retina is radically different. Early cortical brain areas devoted to visual processing are retinotopically organized (Tootell, Dale, Sereno, & Malach, 1996): Neurons respond to patterns of light falling in a restricted region relative to the currently fixated location. In richly textured visual environments, neurons in these areas will cover different regions of the visual scene in quick succession as the eyes (and head for that matter) move around. From one fixation to the next, the same neuron will receive different light levels, contrasts, colors, spatial frequencies, orientations, and so on. 

It is well known that early visual mechanisms are highly sensitive to transient signals when tested under standard steady fixation protocols (Hawken, Shapley, & Grosof, 1996; Muller, Metha, Krauskopf, & Lennie, 2001). Free viewing neurophysiological investigations are rare, but some recent studies have shown that neurons in the primary visual cortex are sensitive to the difference between successive inputs. For instance, the neural response to a pattern brought into the receptive field with a saccade is attenuated when a similar pattern is present in the receptive field before the saccade (Gawne & Woods, 2003). When the presaccadic receptive field covers a patch of uniform background, the neural response can be even enhanced, compared with the standard condition in which the stimulus is flashed into the receptive field during steady fixation (Kagan, Gur, & Snodderly, 2008; MacEvoy, Hanks, & Paradiso, 2008). Eriksson, Valentiniene, and Papaioannou (2010) presented patterns in quick succession during fixation, to mimic the sequential stimulation received by neurons under free viewing conditions. The authors correlated the neural response after a stimulus transition with the previous stimulus, the current stimulus, or the difference between the two. They showed that the response correlated most strongly with the difference image shortly after the stimulus transition and only later came to represent the current stimulus. 

Both local luminance and contrast can vary extensively within a single natural scene (more than a ∼10-fold range; Mante, Frazor, Bonin, Geisler, & Carandini, 2005; Frazor & Geisler, 2006). Importantly, across the distances that are typically traversed by saccadic eye movements, both luminance and contrast show little correlation from one region to the next. Moreover, the correlation between luminance and contrast is low, and neurons in the early visual system respond to both dimensions independently (e.g., for recordings from LGN and V1, see Mante et al. [2005] and Geisler, Albrecht, & Crane [2007], respectively). The implication is that during active exploration of natural scenes, receptive fields are rapidly exposed to a great variation in luminance and contrast, and there is little relation between successive inputs that they receive across gaze shifts. The present study deals with the behavioral consequences of this flux of self-generated stimulation. In particular, we test whether trans-saccadic differences in luminance, contrast, and orientation influence perception and action. We also ask whether trans-saccadic difference signals may be exploited for a functional benefit. 

General methods

Overview of the experiments

The logic of our approach is illustrated in Figure 1. Observers were presented with a preview and a test display, separated by a saccade from the periphery (near the top or bottom of the screen) to the center. The switch from the preview to test display was instantiated once the initial saccade was under way. In the majority of the experiments, the test display consisted of two patterns that were spatiotopically stable in the sense that they had also been present in the preview display. Observers were asked to make a second saccade to the target pattern. 

Figure 1

 

Logic of the experimental paradigm. The observer initially fixates below (or above, not shown here) a configuration of four preview patterns. The eye position is shown by the solid white ellipse. The fixation point jumps to the center of the configuration. The observer follows the fixation point, and the test display is presented as soon as the eyes cross the gaze-contingent boundary (horizontal dashed line). The bottom four panels illustrate some test displays. The test consists of two patterns, both of which are spatiotopically stable in that they were also present during the preview. Test patterns may stimulate retinal locations that covered a similar pattern during the preview (coarse dashed circles) or that previously covered the background (fine dashed circles). The observer's task is to saccade to the target, which––in this example––is defined by its higher luminance. Test patterns can be either retinotopically matched or retinotopic onsets. In the “different” conditions (left stream), the trans-saccadic status of the two test patterns differ; in the “same” conditions (right stream), the trans-saccadic status is the same. Note that the luminance difference in the test displays is exaggerated for the purpose of illustration.

Figure 1 demonstrates a situation in which the eyes start toward the bottom of the screen. The assumed receptive fields of four hypothetical mechanisms are shown. Two of these mechanisms (coarse dashed circles) cover the near preview patterns; the other mechanisms (fine dashed circles) cover the uniform background. During the saccade to the center of the screen, two of the four preview patterns disappear, and the peak luminance of the remaining two patterns is altered to reveal a target (higher luminance) and nontarget (lower luminance) pattern. The test patterns are brought into the receptive field through a saccadic eye movement. 

Given the preview and test configurations, coupled with the initial and second fixation locations, a test pattern may occupy either the same (coarse dashed circles) or different (fine dashed circles) retinal position after the saccade as one of the initial preview patterns. We refer to the former as retinotopically matched and to the latter as retinotopic onsets (as distinct from “real” onsets in the world). More generally, we can think of the inputs received by a visual mechanism before and after a gaze shift as a trans-saccadic difference image. 

Four test conditions are then determined by the factorial crossing of the trans-saccadic status (matched, onset) with the two test patterns (target, nontarget). The two test displays shown on the left-hand side in Figure 1 are those in which retinotopic onsets are directly pitted against retintopically matched patterns. In the top-left test display, the target is retinotopically matched and the nontarget is a retinotopic onset. We refer to this condition as DiffMatched to reflect that the trans-saccadic status of the two patterns is different, but the target appears in a retinotopically matched location. These roles are reversed in the bottom left test display (DiffOnset). In the remaining test conditions, the trans-saccadic status of both test patterns is the same: retinotopically matched (SameMatched; top right test display in Figure 1) or retinotopic onsets (SameOnset; bottom right). These conditions are of less importance and mainly included to keep the experimental design balanced. 

In the four saccadic choice experiments, observers are asked to make a second saccade to the target pattern. If the trans-saccadic status of the patterns is irrelevant, observers will discriminate the target from nontarget with some degree of accuracy, regardless of the spatial configuration of the two test patterns. However, if the nature of previous stimulation matters, we may expect to see differences in choice behavior depending on the trans-saccadic nature of the two patterns. Across different experiments, we varied the presence and nature of the presaccadic stimulation. 

Stimuli

Stimuli were generated on a PC using an NVIDIA Quadro FX 3700 graphics card and presented on a Viewsonic G225f 21-inch monitor at a refresh rate of 85 Hz. The visual stimuli were generated in MATLAB using the Psychophysics Toolbox extensions version 3.08 (Brainard, 1997; Pelli, 1997). The graphics card output was enhanced to 14 bits using a bits++ digital video processor (Cambridge Research Systems Ltd, Rochester, UK). The range between minimum and maximum luminance in any one experiment was linearly sampled in 255 steps using a gamma-corrected look-up table. One additional gray level was used for the black calibration targets and fixation point. 

The screen was viewed binocularly in a darkened room. An adjustable chin rest and forehead support maintained a stationary head at a constant viewing distance of 57 cm. Monocular eye movements were recorded using the EyeLink 1000 system (SR Research Ltd, Kanata, Canada) at a sampling rate of 1000 Hz. Saccades were analyzed offline using the Eyelink velocity and acceleration criteria of 30°⁻¹ and 8000 s⁻². 

At the beginning of each block, observers fixated a grid of nine locations in a random sequence to calibrate the eye tracker. A trial began with the presentation of a gray background and a fixation cross (each leg of the cross measured 0.6° × 0.1°). The fixation point was placed either 8.5° below or above the center of the screen, and this position was held constant in a block of trials. Observers completed half the blocks starting from the top and the other half starting from the bottom. The order was counterbalanced across observers. When the observer accurately fixated the initial target, the experimenter initiated the trial. 

A trial started with presentation of the preview display, the nature of which was varied across experiments (see Figure 2). The preview lasted between 235 ms and 1765 ms and was approximately exponentially distributed with a mean of 565 ms, in order to counteract temporal expectancy effects (Luce, 1986). After the preview period, the initial fixation point stepped vertically to the center of the screen, which was the cue for observers to follow it with a saccade. Given the blocked nature of the initial fixation point location, the metrics of the first saccade were always known with complete certainty. Only the time at which the movement should be made was unknown. 

Figure 2

 

Sample preview and test configurations from Experiments 1 to 5. Experiments 1 to 4 were saccadic choice experiments; Experiment 5 was a psychophysical perceived luminance (contrast) experiment. For the sake of illustration, the target is always shown in the bottom location (in the choice experiments, this configuration corresponds to DiffOnset when starting below the preview). The white ellipses indicate eye position. In all five experiments, the change in eye position was triggered by stepping the fixation point to the center of the display.

The test display was introduced when a saccade was detected that crossed a boundary between the eccentric and central fixation points. The boundary ran along an invisible horizontal line placed 2° from the initial fixation point. This boundary corresponds to approximately a quarter of the required saccade amplitude, so that the test display could be presented during the saccade on the vast majority of trials (within the temporal constraints imposed by the CRT monitor). For Experiments 1 to 5, the proportion of trials in which the test display appeared before saccade offset was 90%, 87%, 86%, 83%, and 87%, respectively. The mean interval between the onset of the test patterns and the end of the saccade ranged from 12 to 14 ms across all five experiments. We did not exclude trials in which the test display appeared after saccade offset, because visual sensitivity is likely to be reduced for some time after the saccade (Matin, 1974; Burr, Morrone, & Ross, 1994; Wurtz, 2008; Ibbotson & Krekelberg, 2011). Nevertheless, we did check whether our results held with a stricter criterion. The patterns of results reported here remained the same even when selecting only those trials in which the test display appeared 10 ms or more before saccade offset. 

Where the preview was filled with patterns (as in Figure 1), these were located at the corners of an invisible square (45°, 135°, 225°, 315°, adopting the right-hand horizontal meridian as 0°), at a distance of 6° from the center of the screen. At this distance, two of the patterns were halfway between the initial and central fixation points. In the saccadic choice tasks, the test display consisted of two patterns that differed slightly in the dimension of interest (luminance or contrast). Observers were instructed to make a second saccade from the center to the target. The accuracy of this response was monitored online to give feedback on a trial-by-trial basis. When the eye position exceeded a 2° radius area around central fixation, the direction of that saccade was used to classify the observer's choice. If the direction fell within ±45° of the target pattern, the decision was classified as correct; otherwise, the saccade was scored as an error. Auditory feedback was provided to indicate the trial outcome: A low tone (500 Hz) denoted an error, whereas a higher tone (750 Hz) indicated a hit. 

The online coding of accuracy was verified offline using the endpoint of the saccade, rather than just the initial direction at 2°. Trials were included in the offline analysis if (a) the first saccade was initiated after the offset of the initial fixation point, (b) the first saccade was initiated within 2° of the initial fixation point and landed within 2° of the central fixation point, (c) the first saccade was initiated within 600 ms from offset of the initial fixation point, (d) the choice saccade was initiated within 2° of the central fixation and landed at least 2° away from central fixation, and (e) the choice saccade endpoint fell in either the target or nontarget quadrant. 

The adopted spatial criteria reflect a pragmatic balance between (a) the requirement of a precise mapping between pre- and postsaccadic retinotopic stimulation and (b) a degree of tolerance to endpoint scatter in the two major saccades so that a sufficient number of trials were obtained in each experimental condition. We have analyzed the data with a stricter 1° tolerance window on the starting point of both saccades and the endpoint of the first saccade, and the results were qualitatively similar. Note that we often refer to the choice saccade as the second movement in the sequence. We adopt this terminology for convenience only. In reality, small saccades around the central fixation point could be and were made in the interval between entering the 2° criterion region and leaving it with the choice saccade. 

Our primary measure of interest in Experiments 1 to 4 is choice accuracy, as a function of the trans-saccadic nature of the target and nontarget patterns in the test display. Stimulus parameters were chosen so that overall choice accuracy was close to threshold (i.e., about 75% correct). We also report the mean latency of the target-directed saccades, averaged across observers. All error estimates around the mean are within-subject standard errors (Bakeman & McArthur, 1996). It is critical to note that none of the effects reported in this article can be attributed to differential sensitivity to specific display locations or movement vectors. Observers initially fixated above and below the possible test locations, so the same spatial location (or movement vector) that forms a retinotopic match when starting from the top corresponds to a retinotopic onset when starting from the bottom. 

Observers

Data are reported for 40 observers (23 female; age range, 18–48 years) who took part in return for either payment or course credits. Five other participants were tested but excluded when, following initial filtering, fewer than half of their trials remained (distributed across the five experiments as follows: Experiment 1, 1; Experiment 2, 1; Experiment 3, 2; Experiment 4, 1). One further observer from Experiment 5 was excluded, because (s)he was confused about the response mapping and produced negatively sloping psychometric functions. 

The data reported for each experiment come from a different sample of eight observers. For Experiments 1 to 4, data were collected in a single session of approximately 1 hour. Experiment 5 was conducted in two 1-hour sessions, performed on different days. All observers reported normal or corrected-to-normal vision. They provided written informed consent prior to taking part and were fully debriefed upon completion of the experiment. The study was approved by the local faculty ethics committee and complied with the principles of the Declaration of Helsinki (2008 revision). 

Experiment 1

The first experiment provides a baseline against which to assess performance in the subsequent choice experiments. In the design illustrated in Figure 1 and used Experiments 2 to 4, the trans-saccadic status of a pattern co-varies with its location relative to the initial movement from the periphery to the center of the screen. That is, a retinotopic onset always involves a location that is partially opposite to the initial movement vector. A retinotopically matched pattern always involves a location that is partly in the same direction as the initial movement vector. Specifically, if 0° indicates a continuation in the direction of the previous movement and 180° indicates a complete direction reversal, matched locations correspond to an angular deviation of 45°, and onset locations correspond to a deviation of 135°. 

We might expect these location differences relative to the initial movement vector to influence choice behavior, for example, due to oculomotor inhibition of return (IOR; Posner & Cohen, 1984; Hooge & Frens, 2000; Taylor & Klein, 2000; Ludwig, Farrell, Ellis, & Gilchrist, 2009) or saccadic momentum (Anderson, Yadav, & Carpenter, 2008; Smith & Henderson, 2009). The influence of a preview, which creates differences in the trans-saccadic status of the test patterns, should therefore be assessed in the context of a baseline in which the preview is absent. This baseline was measured in this first experiment. Under these conditions, the terminology introduced to differentiate the trans-saccadic nature of the patterns does not apply. However, to facilitate comparison of the results with and without previews across different experiments, we maintain the terminology of matched and onset patterns. Across all experiments, this terminology consistently describes the location of the test patterns relative to the initial movement vector. 

Methods

During the initial saccade to the center of the screen, two Gaussian luminance patterns with a circular standard deviation of 0.5° were displayed in two of the four possible test locations. The background luminance was 25 cd m⁻². The target and nontarget pattern luminance were independently sampled from Gaussian distributions, every two video frames (Ludwig, Gilchrist, McSorley, & Baddeley, 2005). As a result, the patterns fluctuate in luminance over time. The target had a higher peak luminance (μ_T = 66 cd m⁻², σ = 10) than the nontarget (μ_N = 56 cd m⁻², σ = 10). The reason for introducing this form of external noise will become apparent later (see Experiment 2). The test display was visible for just over 1 s. 

An example of the preview and test configurations is shown in Figure 2. The two test patterns were always presented in two of the four possible pattern locations described above. For each target location, the nontarget always appeared in an adjacent position (i.e., ±90°, never diagonally across). With four target locations, this constraint results in eight possible test display configurations, only one of which is shown in the test examples of Figure 2. Each configuration was presented 12 times in a block of 96 trials. All different trial types were randomly intermixed. Prior to data collection, observers performed a practice block of 12 trials. 

Before the choice blocks, observers performed two short blocks in which the test display contained just a single pattern (μ_T = 66 cd m⁻², σ = 10). In these blocks, there were only four test configurations, and each configuration was presented 12 times to make a block of 48 trials. Observers ran one block with the initial fixation point near the bottom and another block with the initial fixation point near the top. The order of the initial fixation was counterbalanced across observers, in the same way as for the choice blocks. These single-target measurements were included to estimate the latency cost of having to make a choice but also to get a sense of the extent to which basic phenomena such as IOR and saccadic momentum were operational (both of which are frequently indexed by latency effects). 

Results and discussion

Figure 3A shows choice accuracy, averaged across eight observers. For the critical conditions, we also show the data from each individual observer (thin gray lines). It is immediately obvious that there is a large difference between these two conditions, with accuracy at chance when the target appears at a location that would require a partial return movement. Accuracy is much higher when the target appears at a location that is more congruent with the initial movement direction (the matched location), t(7) = −4.45, p < .01. 

Figure 3

 

Accuracy and latency of the choice saccade in Experiments 1 to 4. The left column indicates the proportion of correct saccades. The critical, different conditions are shown by the central two bars, along with the individual observers' data (thin gray lines). Bar labels are as follows: S-O, SameOnset; D-O, DiffOnset; D-M, DiffMatched; S-M, SameMatched. Right column shows the quartile accuracy as a function of fixation duration. The interval between arriving at the central fixation point and initiating the choice saccade is taken as the fixation duration. The 12.5, 37.5, 62.5, and 87.5 percentiles of the fixation durations are plotted along the abscissa. Choice accuracy in the corresponding four quartiles of the distribution is shown on the ordinate. These functions were computed for each individual subject and then averaged. All error bars in all panels are within-subject standard errors of the mean.

The reluctance to make backward saccades (or, alternatively, a preference for forward saccades) is consistent with IOR or saccadic momentum, despite these movements not being exact reversals and continuations of the initial movement direction. These effects are often assessed in terms of saccade latencies. Taking only the correct choice trials, the observed latency modulation was consistent with the presence of IOR or momentum. Latency was longest when the target involved a backward saccade and the nontarget appears in a forward direction [DiffOnset vs. DiffMatched: 526 ms (±7) and 495 ms (±4), respectively, t(7) = 3.09, p < .05]. The single-target latency measurements were also consistent with an IOR or momentum explanation: The mean latency to a backward target was 412 (±5) and to a forward target was 381 (±5), t(7) = 3.16, p < .05.¹ 

The spatially coarse nature of this effect (the angular deviation relative to complete continuation or reversal was 45°) may be interpreted as a manifestation of saccadic momentum, rather than IOR, which is typically thought to be more spatially specific to the return location (Anderson et al., 2008; Smith & Henderson, 2009). We recognize that we cannot distinguish between saccadic momentum and a coarsely coded, graded version of IOR on the basis of the present data. However, for the sake of brevity, we refer to the forward movement preference as a momentum effect. 

We examined the temporal dynamics of choice behavior by calculating accuracy for each quartile of the distributions of fixation duration prior to the choice saccade. Figure 3B shows how accuracy evolves over time for these two conditions. It is clear that the preference for forward saccades persists throughout the latency distribution, although the accuracy difference diminishes substantially with time. That is, it appears that the momentum effect is particularly strong for saccades that are initiated after a short fixation duration. 

As a relatively crude statistical assessment, we treated fixation duration as an errorless, categorical variable (Quartile 1–4) and ran a repeated-measures analysis of variance on the proportion correct. In this analysis, we were mainly interested in main effects of quartile and any interaction between quartile and target location (onset vs. matched). There was a main effect of quartile, F(3,21) = 4.13, p < .05, as well as an interaction with the target location, F(3,21) = 9.18, p < .001. Unsurprisingly, given the paired comparison of accuracy in the critical conditions reported above, the main effect of target location was reliable as well, F(1,7) = 20.11, p < .01. 

In the context of the experimental design, the advantage for targets in a forward movement direction cannot be attributed to specific screen locations or specific movement vectors. Forward movements may be directed up or downward, depending on the initial fixation location. It would seem unlikely that the visual response triggered by a postsaccadic test pattern would depend on the direction of the movement that preceded its onset. Therefore, we suggest that the momentum effect is best understood as a genuine response bias. We will test this issue directly in Experiment 5. 

Experiment 2

In Experiment 2, we introduced a preview, so that test patterns were always spatiotopically stable (unlike Experiment 1), and the trans-saccadic status of the test patterns varied as illustrated in Figure 1. The test displays themselves were identical to those of Experiment 1. Therefore, any change in behavior in going from Experiment 1 to 2 can be attributed only to the presence of the preview (see Figure 2). 

Methods

The spatial profile of the preview patterns was identical to the test patterns, but their average peak luminance was set in between that of the target and nontarget (μ_T = 61 cd m⁻², σ = 10). In other words, the preview contained no information at all about the upcoming test stimulus. The preview patterns were created by sampling a sufficiently large number of luminance samples for one pattern to cover the preview duration and the saccade latency after the initial “go” signal, up to a 600-ms deadline. This temporal sequence of luminance samples was then replicated and randomly shuffled independently for the other preview patterns. As soon as the eyes crossed the critical boundary during the initial saccade to the center, two patterns were removed and the mean luminance of the two remaining patterns was altered to reveal a target (Δμ = 5 cd m⁻²) and nontarget (Δμ = −5 cd m⁻²). Once again, observers performed four blocks of 96 choice trials along with two single-target blocks. 

In one sense, the “world” in this experiment is clearly not spatiotopically stable, with two of the preview patterns abruptly disappearing. However, the two test patterns were present in the image throughout the trial. Although their mean luminance changed slightly, the shift was small relative to the amount of noise that was inserted. Indeed, the critical reason for the noise manipulation was to effectively mask the luminance change in the test patterns and to promote a spatiotopically continuous percept of these two patterns. That is, the noise served to avoid that the visual system treated the two test patterns as completely “new” objects (Enns, Austen, Di Lollo, Rauschenberger, & Yantis, 2001). 

Results and discussion

Figure 3C shows the mean choice probabilities under these conditions. Overall performance accuracy was very similar to Experiment 1. However, when the trans-saccadic status of the two patterns differed, behavior was altered drastically. Observers now much more accurately selected the target when it was a retinotopic onset. They were at chance when the target appeared in the retinotopically matched location. This preference was extremely stable for individual observers, as shown by the gray lines. The difference between matched and onset targets was significant, t(7) = 10.02, p < .01. 

As in the accuracy data, the inhibitory effect on saccade latency associated with a retinotopic onset has now disappeared. This was the case for the correct choice latencies [DiffOnset vs. DiffMatched: 540 ms (±5) and 536 ms (±8), respectively, t(7) = 0.31, p = .77], as well as for the single-target latencies [onset vs. matched targets: 462 ms (±6) and 458 ms (±6), respectively, t(7) = 0.38, p = .72]. Figure 3D shows the temporal evolution of the choice preferences under these conditions. As in Experiment 1, the preference was stable throughout the latency distribution (main effect of target location), F(1, 7) = 101.30, p < .001. The interaction between quartile and target location was marginally reliable, F(3, 21) = 2.72, p = .07, but this modulation was clearly quite minor. There was no main effect of quartile, p = .35. In summary, the mere presence of a preview completely overturned the strong momentum effect observed in Experiment 1 in favor of backward movements directed to retinotopic onsets. This preference was largely independent of fixation duration. 

Experiment 3

In Experiment 2, an onset represents a luminance increase across the saccade. The term onset may be understood in different ways. One interpretation is simply as a luminance increment. At a more abstract level, the onset may refer to the existence of an object or pattern in the receptive field, regardless of whether this pattern is defined by a luminance increment, decrement, contrast, chromaticity, and so forth. Although it is this more general meaning that we have in mind when we use the term onset, it was important to test whether the observed preference for retinotopic onsets is specific to on-patterns. Therefore, in Experiment 3, the patterns were defined by luminance decrements relative to the background. 

Methods

Experiment 3 is a direct replication of Experiment 2, except that the preview and test pattern luminance were lower than the background (Figure 2). Specifically, the background luminance was set to 71 cd m⁻², and the average peak luminance values for the preview, target, and nontarget patterns were μ_P = 32, μ_T = 27, and μ_N = 37 cd m⁻². The standard deviation around these means remained σ = 10 cd m⁻². Observers performed four choice blocks only; the single-target measurements were dropped from this experiment onward. 

Note that the target pattern now actually has a lower luminance, but—as in Experiments 1 and 2—its contrast relative to the uniform background is higher. We cannot tell whether observers solve this task (and that of Experiments 1 and 2 for that matter) by judging the brightness or the brightness contrast relative to the background (Blakeslee & McCourt, 2003). Responding on the basis of brightness contrast essentially equates to responding to the pattern that “stands out” most. 

Results and discussion

Figure 3E shows the choice probabilities. Once again, there is an advantage for the onset target in the critical conditions, t(7) = 2.71, p < .05, although this choice preference is clearly reduced compared with Experiment 2. Nevertheless, the preference for onset targets was manifest in seven of eight observers. Moreover, we should point out that the null hypothesis (i.e., the absence of a difference between DiffOnset and DiffMatched) is not really appropriate in this paradigm: Baseline performance is strongly biased in favor of moving in a forward direction, which would translate into a strong choice preference for the retinotopically matched targets. It is clear that in both Experiments 2 and 3, the momentum effect has been overcome. 

The latency difference was not reliable [DiffOnset vs. DiffMatched: 505 ms (±6) and 486 ms (±10), respectively, t(7) = 1.37, p = .21]. The analysis of choice accuracy as a function of fixation duration is shown in Figure 3F. This figure reveals an interesting suggestion of a crossover pattern, or at least a preference for retinotopic onsets that takes some time to emerge. Indeed, in addition to the expected main effect of target location, F(1, 7) = 7.57, p < .05, there was a reliable interaction with quartile, F(3, 21) = 4.24, p < .05. 

Overall accuracy was higher in Experiment 3 compared with Experiments 1 and 2 (the overall proportion correct was ∼0.15 higher compared with the first two experiments). Fixation duration preceding the choice saccade was also shorter, suggesting that discrimination was overall easier despite the physically matched luminance differences across all three experiments. These findings are consistent with previous reports of greater sensitivity to decrements over increments (Patel & Jones, 1968; Bowen, Pokorny, & Smith, 1989; Lu & Sperling, 2012) and faster processing of decrements (Komban, Alonso, & Zaidi, 2011). The interaction between fixation duration and accuracy (Figure 3F) may suggest that the reduced preference for retinotopic onsets can be attributed to short latency saccades driven by the momentum effect. However, it is also clear that for the same fixation duration (e.g., 500 ms), retinotopic onsets enjoyed a much greater advantage when defined as ON patterns compared with OFF patterns. 

Experiment 4

Up to this point, it appears that the saccadic eye movement system is sensitive to trans-saccadic differences in luminance, whether they be increments or decrements. As stated in the Introduction, natural scenes vary widely in both local luminance and local contrast, and these two dimensions are not strongly related to each other (Mante et al., 2005; Frazor & Geisler, 2006). In the final choice experiment, we turn to trans-saccadic differences in luminance contrast. 

The preview and test patterns were defined by a Gabor function and had the same mean luminance as the background (see Figure 2). The use of these oriented patterns allowed us to test whether any sensitivity to trans-saccadic differences extends to differences in the internal structure (Eriksson et al., 2010). Take the sequence in Figure 2 as an example. Here the near-right pattern during initial fixation is a horizontal Gabor pattern. After the first saccade, the same retinal position now covers a vertically oriented pattern (top right in the test display). In terms of luminance and contrast, the successive inputs at this retinal location are very similar. However, it is possible that the 90° orientation difference between pre- and postsaccadic patterns is sufficiently large to elicit a strong internal response (Blakemore & Nachmias, 1971; Phillips & Wilson, 1984; Snowden & Hammett, 1992). 

In this design, the average luminance at a given retinal location (integrated over the area of a single patch) is constant, before and after the saccade. Some retinal locations receive a strong increase in local contrast; we regard this increase as a retinotopic onset in the more abstract sense given earlier (see Experiment 3). Test patterns in the retinotopically matched locations can be identical to the presaccadic input or have the orthogonal orientation. 

Methods

Considering the trans-saccadic orientation difference results in a more complicated experimental design. The preview display always contained four patterns. We imposed the constraint that two patterns were always horizontal and the other two were vertical. With this constraint, there were six possible preview configurations (going from top-left clockwise: e.g., H-H-V-V, H-V-H-V, etc.). The target pattern was defined by its higher contrast relative to the nontarget. Each of the preview configurations was independently combined with the eight test display configurations (recall: four target locations, paired with two adjacent nontarget locations). The 48 unique displays were presented twice in a block of 96 trials. Observers performed six blocks, three starting from the top fixation point and three starting from the bottom (counterbalanced between observers). 

The different preview and test configurations may be classified into seven experimental conditions (which is why we collected more data in this experiment). First we consider the conditions in which the pre- and postsaccadic orientation at matched retinal locations is the same. These are the subset of conditions that may be compared directly with Experiments 2 and 3, and we refer to these conditions using the same labels as we have used so far (DiffMatched, DiffOnset, etc.). Taking orientation differences across the saccade into account results in an additional three conditions: (a) both targets and nontargets appear in retinotopically matched locations but have an orthogonal orientation (SameMatchedOrth), (b) the target occupies a matched retinal location with a different orientation and the nontarget is a retinotopic onset (DiffMatchedOrth), and (c) the nontarget occupies a matched location with a different orientation and the target is a retinotopic onset (DiffOnsetOrth). 

The Gabor patterns had a spatial frequency of 2 cycles/deg⁻¹, and the Gaussian window had a standard deviation of 0.5°. The patterns were in sine phase so that the average luminance was the same as the background (47 cd m⁻²). The contrast of a pattern is defined as the nominal Michelson contrast (Peli, 1997): $\frac{L_{max}-L_{min}}{L_{max}+L_{min}}$ . The target had a higher mean contrast compared with the nontarget; the mean preview contrast was midway in between: μ_P = 0.4, μ_T = 0.48, and μ_N = 0.32. These mean contrast values were well above the detection threshold, and the patterns were easily visible during the preview, even in the most eccentric positions. All patterns were corrupted with temporal contrast noise with a standard deviation of σ = 0.15. All other spatial and temporal parameters were identical to the previous experiments. 

Results and discussion

Figure 3G plots the choice probabilities for the subset of the experimental conditions that corresponds to the previous experiments. Behavior in the critical conditions is clearly more mixed than in the previous experiments. In general, the direction of the choice preference is more variable and its magnitude is smaller. When averaged across observers, performance is no different for matched and onset targets, t(7) = −0.59, p = .57. The same was true for the correct choice latencies [DiffOnset vs. DiffMatched: 436 ms (±15) and 399 ms (±12), respectively, t(7) = 1.45, p = .19]. 

Figure 3H shows choice accuracy as a function of the fixation duration. As in Experiment 3, there is a hint of a crossover, with saccades following brief fixations directed in the momentum direction. However, saccades following longer fixations still do not show a clear preference for onset locations. Indeed, there were no main effects or interaction between target location and quartile (all p's > .12). 

For trans-saccadic differences in internal structure, the critical comparisons are between patterns that appear in the same retinotopic location as a preview pattern but differ in their orientation: DiffMatched versus DiffMatchedOrth and DiffOnset versus DiffOnsetOrth. If the system were sensitive to trans-saccadic orientation differences, we might expect improved performance when the target pattern orientation is orthogonal to the preview that stimulated the same retinal location: PC(DiffMatchedOrth) > PC(DiffMatched²). Likewise, performance should suffer when the nontarget has a different orientation from the preview: PC(DiffOnset) > PC(DiffOnsetOrth). As it turned out, trans-saccadic differences in internal structure had little effect on behavior, at least not in the predicted direction: M_{DiffMatchedOrth} = 0.74 (±0.03) and M_DiffMatched = 0.79 (±0.04), t(7) = 2.14, p = .07; M_DiffOnset = 0.73 (±0.06) and M_{DiffOnsetOrth} = 0.79 (±0.02), t(7) = 1.13, p = .30. 

The robust choice preference for retinotopic onsets was eliminated when the patterns had the same average luminance as the background. This finding suggests that the saccadic system is particularly sensitive to trans-saccadic differences in luminance but not in contrast and orientation. 

Experiments 1 to 4: Choice behavior relative to baseline

Although the preference for retinotopic onsets measured in Experiments 2 and 3 was significant in its own right, it is all the more impressive when taking into account the default behavior: a strong preference to move in the forward direction (i.e., toward retinotopically matched locations). Indeed, from this perspective, the absence of a clear preference for either matched or onset patterns in Experiment 4 may be taken to suggest that retinotopic onsets still had some attractive potency when the mean luminance of these patterns was the same as that of the background. It is therefore informative to attempt to quantify the observed choice preferences relative to the baseline measured without a preview in Experiment 1. 

What we would like to know, for the experiments with a preview, is how likely a particular preference for onset or matched patterns would be when behavior was, in fact, not affected by the preview at all. The statistical comparisons reported so far do not provide this likelihood because the null hypothesis is one of “no preference” rather than an overwhelming tendency to choose the pattern in the matched location. However, we can obtain a crude estimate of this likelihood in the following way. For each observer in Experiments 1 to 4, we calculated a preference score, by subtracting the proportion correct for the matched target from that of the onset target: PC(DiffOnset) – PC(DiffMatched). We calculated the mean preference and standard deviation across the sample of observers from Experiment 1. For each observer from all four experiments, we calculated a z-transformed preference score, using the sample mean and standard deviation from Experiment 1. The resulting z-scores provide an index of how likely a given observer's preference is under the distribution of Experiment 1. 

These z-transformed scores are plotted in Figure 4 for each individual observer (crosses), along with the sample means (large open squares). We can use the standard normal distribution to estimate the cumulative percentiles of these scores under the distribution of Experiment 1. A number of distinct percentile regions are shown in different shades of gray, where lighter shades correspond to scores that are increasingly incompatible with behavior seen under the baseline. Unsurprisingly, in Experiment 2, the behavior of all eight observers was extremely unlikely to be obtained under the baseline (effectively p < .01 for all observers). The same holds for the luminance decrement in Experiment 3 (p < .05 for seven of eight observers). For Experiment 4, behavior of the sample as a whole (and five of eight observers) was compatible with the baseline preference for matched patterns. However, three observers were attracted to retinotopic onsets more often than would be expected from the baseline (p <.05). It is possible that these observers responded to the local luminance within the patch (e.g., the brightest bar in the grating) or, indeed, local contrast. 

Figure 4

 

Choice preferences from Experiments 1 to 4, expressed relative to baseline behavior measured without a preview. The choice preference is calculated as PC(DiffOnset) – PC(DiffMatched). For each observer, the preference was expressed relative to the sample mean of Experiment 1 in standard deviation units derived from the same experiment. The standard normal distribution may then be used to determine the likelihood of the observed choice preferences in subsequent experiments, using the no-preview distribution. Illustrative percentile regions are shown in different shades of gray, with lighter shades indicating scores that are increasingly incompatible with the baseline. Individual observers' scores are shown by the crosses; the sample means are shown by the square symbols.

Experiment 5

We have observed clear choice preferences, in opposite directions, depending on the presence of a preview and the nature of the signals (differences in luminance versus contrast). Without a preview in Experiment 1, we found a strong preference for patterns that involve a continuation of movement in a forward direction. With a preview in Experiments 2 and 3, this preference was reversed in favor of retinotopic onsets that involve backward movements. One basic question is whether these choice preferences reflect the internal visual responses or modulations of some spatial response bias. 

In the context of Experiment 1, we argued that the momentum effect was most likely a response bias, because it seemed unlikely that the visual response to a postsaccadic pattern depends on the direction of an eye movement that came before. It is clearly desirable to support this claim with experimental evidence. In a similar vein, we can ask whether in Experiments 2 and 3 the preview—somehow—overturned this response bias in the opposite direction. 

Our hypothesis is that although the momentum effect in Experiment 1 reflects a spatial bias, the strong preference for retinotopic onsets under preview conditions stems from a differential visual response to matched and onset patterns. In particular, retinotopic onsets trigger a relatively larger internal response compared with a retinotopically matched pattern. If the target appears in the matched location and the nontarget in the onset location, the internal response difference will be smaller compared with the complementary condition. As a result, accuracy would suffer when the nontarget is a retinotopic onset but improve when the target appears in that location. This is of course the pattern of results we obtained in Experiments 2 and 3. 

A relatively greater response to retinotopic onsets may arise because (a) the postsaccadic response at a retinal location is attenuated if that location previously received a very similar input (Gawne & Woods, 2003) or (b) retinotopic onsets simply generate an enhanced visual response (Kagan et al., 2008; MacEvoy et al., 2008). An attenuated response may come about as a result of retinotopic pattern adaptation (Foley & Boynton, 1993). Alternatively, it is possible that mechanisms are driven not so much by the current receptive field content but rather by the difference between pre- and postsaccadic inputs (Gawne & Woods, 2003; Eriksson et al., 2010). This distinction is subtle, and we do not think we can distinguish between these underlying mechanisms with data presented here. 

Experiment 5 was designed to determine the loci of the observed saccadic choice preferences. We measured the perceived luminance (or brightness contrast relative to the background) of a single test pattern that appeared in a matched or onset location, with or without a preview. The observer output is now a judgment of the brightness of a single test pattern rather than a spatially directed, saccadic response (see Figure 2). As such, spatial response bias(es) can no longer influence performance. Any differences in perceived luminance across conditions are likely to reflect differences in the internal visual response. 

Methods

We reverted to the luminance increment patterns of Experiments 1 and 2, as these were the experiments in which the strongest—and opposite—choice preferences were obtained. In preview trials, four patterns were visible during the initial fixation. The patterns were absent during no-preview trials. Preview and no-preview trials were randomly intermixed. Initial fixation was either toward the top or bottom of the display, in the same, blocked manner as in all previous experiments. As in Experiment 1, without a preview the terms onset and matched patterns refer to the location of the peripheral test pattern relative to the direction of the initial saccade (i.e., backward and forward, respectively). We maintain this terminology to facilitate the direct comparison with the corresponding patterns in preview trials. 

After the foreperiod, distributed in the same way as in the previous experiments, the fixation point stepped to the center of the screen. The same gaze-contingent boundary was in place to turn the test display on. This display consisted of a single peripheral test pattern, with a peak luminance that was varied at nine levels between 48 and 90 cd m⁻². Together with the test pattern, the central saccade target was replaced by another Gaussian luminance patch that served as the standard for comparison. This patch had the same dimensions as the peripheral test, but its luminance was held constant at 61 cd m⁻². Each test luminance was presented twice at each possible test location, once with a preview and once without, to make a block of 72 trials. Observers performed eight blocks in a 1-hour session, four starting at the top/bottom fixation point (in counterbalanced order). Each observer completed two sessions on different days. 

The task for observers was to indicate whether the test pattern was brighter or dimmer than the foveal standard. Once again, we cannot tell whether this judgment is based on brightness directly or brightness contrast relative to the background (Blakeslee & McCourt, 2003). The test display was presented for only 306 ms to ensure that observers could not directly fixate the peripheral pattern (cf. single-target latencies in Experiments 1 and 2 were typically well in excess of 300 ms). They responded by pressing one of two buttons on a standard gamepad. The response was unspeeded, and no feedback was given. By analyzing the psychometric function (proportion of “brighter” responses as a function of test luminance), we can estimate the value at which both the peripheral and foveal patterns had the same perceived luminance. 

We do not expect perception to be veridical under these conditions, because the task involves a comparison between foveal and peripheral patterns. The veridicality of perception is irrelevant in the current design. Only the relative differences between the four conditions—created by crossing onset and matched test locations with the presence or absence of a preview—matter. In addition to the point of subjective equality (PSE), we examined the slopes of the psychometric functions to assess whether visual sensitivity varied between conditions. 

Results and discussion

Figure 5A shows the proportion of trials on which the test pattern was perceived as brighter than the foveal standard, as a function of its peak luminance. For the purpose of illustration, the data were pooled across all observers. The actual analyses were based on psychometric function fits for each individual observer. Note that a rightward shift in the psychometric function is interpreted as evidence for an attenuated visual response: The test pattern needs to be of higher luminance for the observer to perceive it as equal to the foveal standard. 

Figure 5

 

Psychometric functions measured in Experiment 5. (A) Pooled psychometric functions, based on all of the data from all observers. The smooth curves are maximum likelihood cumulative Gaussians, constrained to share a common lapse rate. Curves were fit for each observer individually. The luminance of the foveal standard (61 cd m⁻²) is shown by the triangle on the abscissa. (B) Mean points of subjective equality across observers. The light horizontal dotted line shows the luminance of the foveal standard. (C) Mean of the standard deviation (inverse slope) estimates across observers.

What data patterns might we expect if the choice preferences obtained in our earlier experiments stemmed from spatial biases or differential visual responses? 

•  
  If all choice preferences were due to spatial bias, and the reversal of this bias with the introduction of a preview, we would expect all four psychometric functions to lie on top of each other. That is, the design of the current experiment is such that a spatial bias simply cannot be expressed.
•  
  If the choice preference for matched locations without a preview (i.e., the momentum effect of Experiment 1) reflects differential internal visual responses to patterns in forward and backward locations, we would expect a separation between the two no-preview psychometric functions (gray lines in Figure 5A). In particular, we would expect the “no preview matched” function to lie to the left of the “no preview onset” function.
•  
  If the advantage enjoyed by retinotopic onsets is due to a differential internal response, we expect the “preview onset” function to lie to the left of the “preview matched” function.
•  
  An attenuated response to similar postsaccadic inputs would reduce response to matched patterns in the presence of a preview, relative to when the preview is absent. More specifically, we would expect the “preview matched” function to lie to the right of the “no preview matched” function.
•  
  An enhanced response to retinotopic onsets would manifest itself as an increase in brightness for retinotopic onsets, relative to the corresponding no-preview condition. However, it could be argued that both with and without a preview, the temporal pattern of stimulation provided by onset test patterns is the same. Therefore, we would expect either no difference between the “preview onset” and “no preview onset” functions or a leftward shift of the preview function.

The four psychometric functions were fit with cumulative Gaussians using maximum likelihood estimation. For each function, the mean (PSE) and standard deviation (inverse slope) were free to vary. One additional parameter was included, corresponding to a single lapse rate for all four conditions (Wichmann & Hill, 2001). Given that all conditions were randomly interleaved, we have no a priori reason to expect variation in the lapse frequency across conditions. 

Figure 5B shows the mean PSEs across all eight observers; Figure 5C shows the averaged standard deviation parameter of the psychometric functions. Together, these panels confirm the patterns that are readily apparent in the pooled psychometric functions. The statistical assessment of these patterns follows the predictions listed above. 

•  
  It is clear that not all four psychometric functions lie on top of each other, suggesting that at least some of the choice preferences identified in previous experiments are due to differences in the internal visual responses to patterns in matched or onset locations.
•  
  The two “no preview” conditions are very similar, both in terms of their location (white bars in panel B) and slopes (panel C). Planned comparisons for these two parameters did not identify any difference between these two conditions, PSE: t(7) = −1.24, p = .25; standard deviation: t(7) = 0.97, p = .36. As such, we conclude that the momentum effect of Experiment 1 is most likely a spatial response bias in favor of patterns that involve an approximate continuation in the direction of movement.
•  
  The advantage for retinotopic onsets appears to stem from a relatively larger internal visual response. The “preview matched” function was shifted to the right relative to the “preview onset” function, PSE: t(7) = 3.61, p < .01. There was no difference in the slopes of these two functions, t(7) = 1.33, p < .23.
•  
  To assess whether the response to similar postsaccadic inputs was attenuated, we compared the “preview matched” and the “no preview matched” functions. For both the location and slope parameters (left set of bars in panels B and C), this is where the most dramatic effects were found, PSE: t(7) = 5.01, p < .01; standard deviation: t(7) = 3.93, p < .01. In other words, not only did the retinal locations that received a similar input across the saccade give an attenuated response to the postsaccadic input, but the response was also less sensitive or precise.
•  
  We found no evidence in favor of an enhanced response to retinotopic onsets compared with “real” onsets. In fact, the “preview onset” function was shifted rightward from its no preview counterpart, PSE: t(7) = 4.30, p < .01 (the slope difference was not reliable, t(7) = 1.76, p = .12).

It is tempting to speculate that the rightward shift in the preview onset function reflects a spatiotopic form of adaptation (Melcher & Colby, 2008) or perhaps forward masking (Irwin, Brown, & Sun, 1988) by the preview. Reports of spatiotopic adaptation have been limited to relatively higher-level features, such as tilt, complex form (Melcher, 2005), visual motion (Melcher & Morrone, 2003), and duration (Burr, Tozzi, & Morrone, 2007), although some of these findings have been disputed (Afraz & Cavanagh, 2008; Wenderoth & Wiese, 2008; Knapen, Rolfs, & Cavanagh, 2009; Knapen, Rolfs, Wexler, & Cavanagh, 2010). One feature on which it is commonly agreed that no spatiotopic adaptation occurs is luminance contrast (Irwin, Zacks, & Brown, 1990; Melcher, 2005). In this context, we refrain from interpreting the shift of the preview onset function as an index of spatiotopic coding until the emergence of further empirical evidence.³ 

General discussion

In the natural visual world, the same retinal location will receive different inputs in quick succession as gaze shifts around through eye, head, and body movements (Mante et al., 2005; Frazor & Geisler, 2006). Little is known about how the visual system responds to this sequence of stimulation. Neurophysiological evidence suggests that the early visual system responds to trans-saccadic differences in input during free viewing (Gawne & Woods, 2003; Eriksson et al., 2010). We set out to examine the visual and oculomotor response to trans-saccadic differences in luminance, contrast, and orientation. In this section, we summarize our findings, discuss their relation with the phenomenon of predictive remapping, and ask whether responding to trans-saccadic differences could be exploited for a functional benefit. 

First, and most trivially, saccade target selection is not driven only by the stimulus information present during the fixation preceding the movement of interest. In other words, what came before and was viewed from a different position matters. This finding is most clearly demonstrated by the dramatic shift in behavior from the first to the second experiment. In Experiment 1, observers were strongly biased to move in a direction that was more similar to the preceding movement, a finding that we related to saccadic momentum (Hooge & Frens, 2000; Anderson et al., 2008; Smith & Henderson, 2009). Yet behavior was drastically different in response to exactly the same test stimuli when an uninformative preview was presented during the initial fixation. In particular, the momentum effect was completely overhauled in favor of responding to retinotopic onsets: patterns that stimulated retinal locations that previously covered a uniform patch of background (Experiments 2 and 3). This finding provides further evidence that the early visual system is not reset with every saccade (Gawne & Woods, 2003; Melcher, 2007). We have shown that the momentum effect itself is best understood as a spatial response bias, rather than the result of a differential visual response (Experiment 5). 

Second, the preference to fixate retinotopic onsets was most pronounced and consistent for trans-saccadic differences in the local luminance of pre- and postsaccadic inputs. When the mean luminance remains constant, but local contrast and/or orientation vary across the saccade, these trans-saccadic differences did not preferentially attract gaze (Experiment 4). The sensitivity to differences in luminance only, coupled with the lack of orientation tuning, suggests that the critical mechanisms may be precortical (De Valois & De Valois, 1988). 

Third, the preference to fixate retinotopic onsets is driven by a differential visual response to retinotopically matched and onset patterns. This differential response stems from an attenuated response to similar postsaccadic inputs. Attenuation may be achieved through adaptation to local luminance that remains in retinotopic coordinates and travels along with the saccade (Afraz & Cavanagh, 2008; Wenderoth & Wiese, 2008; Knapen et al., 2009, 2010). Alternatively, mechanisms may respond directly to the difference in their input across the saccade (Gawne & Woods, 2003; Eriksson et al., 2010). The difference between these two underlying mechanisms is that adaptation involves a change in the filter properties (e.g., its gain or sensitivity), whereas the difference image computation involves a change in the signal that drives the mechanism (a difference, rather than the current stimulus). Although adaptation seems like a perfectly adequate explanation, it is intriguing that it does not seem to occur for the Gabor patterns that do not introduce a difference in mean luminance. 

Regardless of the underlying mechanism, the saccadic system seems to be sensitive to differences in local luminance across gaze shifts. This finding suggests the following hypothesis. When the local luminance falling into a mechanism's receptive field is substantially different from the previous fixation, the mechanism responds. This response is passed on to neural structures involved in gaze control (Schiller, 1998), where it may influence the competition for saccade target selection. Clearly these are not the only signals that drive competitive interactions between potential saccade targets; other signals include task instructions (Ottes, Gisbergen, & Eggermont, 1987; Bichot & Schall, 1999), expectations derived from recent history (Fecteau & Munoz, 2003; Anderson & Carpenter, 2006), reward (Platt & Glimcher, 1999; Hikosaka, Nakamura, & Nakahara, 2006), and presumably many more. Nevertheless, the response of early visual mechanisms that receive different light levels across saccades may steer the system toward the locations coded by these mechanisms. 

Whether such a mechanism “sees” a local luminance difference across an eye movement will depend on its spatial integration area relative to the spatial scale of luminance modulations in the scene. For example, a small integration area relative to the spatial frequency of the grating patches used in Experiment 4 could easily signal a trans-saccadic luminance difference when positioned asymmetrically over the pattern (say, across the brightest or darkest bar). Indeed, this explanation was raised as a possible source of the individual differences observed in that experiment, with some subjects still being strongly driven by retinotopic onsets (see Figure 4). Had we used lower spatial frequency patterns, we would expect to see the reemergence of the choice preference for retinotopic onsets (of course, a Gaussian blob simply represents the lower bound of the spatial frequency continuum). 

On a related note, in our experiments, trans-saccadic differences in luminance were provided by moving receptive fields from a uniform background onto a spatially localized pattern (representing either an increase or decrease in local luminance in Experiments 2 and 3, respectively). We do not claim there is anything special about retinotopic onsets defined as such. In natural scenes, there is typically no uniform background but rather just overlapping objects, surfaces, and textures. During active exploration with gaze, many different visual mechanisms will receive different luminance levels across successive fixations. Our hypothesis is simply that larger differences trigger larger responses and that these responses may influence target selection in the oculomotor system. 

Trans-saccadic perception and predictive remapping

As a starting point of this investigation, we considered the changing inputs received by early visual mechanisms as a stable scene is actively sampled by shifting gaze. Neurophysiological recording from area V1 suggest that at this level, the visual system does not distinguish clearly between patterns that appear in the receptive field during steady fixation (i.e., “world” onsets) and patterns brought into the receptive field with a saccadic eye movement (retinotopic onsets), although there are some subtle differences in the response profiles (Richmond, Hertz, & Gawne, 1999; Gawne & Woods, 2003; Kagan et al., 2008; MacEvoy et al., 2008; Ruiz & Paradiso, 2012). It is clear that at some higher level in the system (e.g., lateral intraparietal area; Gottlieb, Kusunoki, & Goldberg, 1998), these self-generated early visual responses are treated as such, rather than as signals arising from changes in the external world. 

How perceptual spatiotopic stability may be achieved is still an open question, which is covered extensively elsewhere (for reviews, see Bridgeman, Heijden, & Velichkovsky, 1994; Irwin, 1996; Melcher & Colby, 2008; Wurtz, 2008; Cavanagh, Hunt, Afraz, & Rolfs, 2010). One signal that is often claimed to be important in maintaining perceptual stability is the anticipatory response of neurons to their future, postsaccadic receptive field content (Duhamel, Colby, & Goldberg, 1992; Umeno & Goldberg, 1997; Nakamura & Colby, 2002; Melcher, 2007; Melcher & Colby, 2008). This phenomenon is known as “predictive remapping.” The idea is that deviations from spatiotopic stability may be detected through prediction of what a mechanism should “see” after the upcoming saccade (Wurtz, 2008). 

Recently, the psychophysical evidence in favor of remapping of visual feature information (Melcher, 2005, 2007) has been called into question (Wenderoth & Wiese, 2008; Knapen et al., 2009, 2010). An alternative view is that remapping is not involved in perceptual stability and does not involve the transfer of detailed visual features (Cavanagh et al., 2010). Instead, remapping may serve to activate the mechanisms that will, after the impending saccade, code behaviorally important locations in retinotopic coordinates. Such preactivation may facilitate subsequent action toward these locations, such as a further gaze shift to analyze the visual feature information (Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011). This perspective on remapping may be relevant to the advantage associated with retinotopic onsets identified in this study. 

How could predictive remapping contribute to the advantage of retinotopic onsets over matched patterns? Suppose predictive remapping occurs only for mechanisms that will receive a substantially novel input after the saccade. In other words, only retinotopic onsets elicit anticipatory activity (this is most directly analogous to the standard paradigm to study remapping in single neurons; cf. Duhamel et al., 1992). Such preactivation may then bias the saccadic system in the direction of the locations coded by these mechanisms after the saccade. Alternatively, the postsaccadic visual response may add to the presaccadic activity to give an altogether enhanced internal response. On both accounts, retinotopic onsets would enjoy a competitive advantage in choice situations. 

The perceived luminance measurements of Experiment 5 are relevant in this regard. These data indicate that the choice preference for retinotopic onsets is best accounted for in terms of a differential visual response, rather than a response bias. However, we need not invoke the concept of predictive remapping to account for this difference. Attenuation of the response to similar inputs across the saccade is sufficient to explain the response difference between matched and onset patterns. 

Finally, note that the remapping mechanism, as formulated above, might be expected to operate regardless of whether the signals involve a change in mean luminance relative to the background. However, with the Gabor patterns of Experiment 4, no consistent preference for retinotopic onsets was obtained. Although we do not dispute the existence and functional utility of predictive remapping in general, its operation adds little explanatory power for the pattern of results reported here. 

Functional benefit

It is worth considering what a trans-saccadic difference in local luminance signifies. Imagine a uniform, blank field and a single mechanism that, to a first approximation, simply computes the mean luminance in its receptive field. As the eyes move from one point to another, this mechanism will receive exactly the same input and remains silent. Now consider an image with an abrupt luminance transition. Depending on the metrics of the saccade, the mechanism under consideration may receive a different input (e.g., from black to white), which necessarily means that it must have crossed some kind of image discontinuity. The trans-saccadic luminance difference triggers an internal response, and this response drives the eyes to fixate the spatial location coded by the mechanism. What fixation behavior does such a simple system predict? 

We can answer this question by elaborating this example with the simulation shown in Figure 6. The visual world in this example is stable and consists of a single black bar on a blank background. We simulate a one-dimensional array of mechanisms that simply register a binary luminance value (0 and 1 for black and white, respectively). As in our paradigm, the eyes start on one fixation point (left of the bar), then move to another (right of the bar). After this first saccade, there are four mechanisms that receive a different input compared with their presaccadic stimulation (shown in bold italic font). 

Figure 6

 

Saccade target selection driven by trans-saccadic differences in local mean luminance. We used a one-dimensional array of visual mechanisms that simply registered the mean luminance in their receptive field. The array was placed in a simple, stable world, consisting of a single black bar. The fovea was initialized on one fixation point to the left of the bar and then moved to a point to the right of the bar. A number of mechanisms received a different input after the first saccade (shown in bold italic font). The system now randomly chose one of the locations coded by these mechanisms as the target for the next saccade. In the actual simulation, the array size (visual field) was 51 cells, and the bar width was approximately 20% of the size of the array. The number of simulated saccades was 10,000. The frequency distribution of horizontal landing positions is shown by the histogram at the bottom. On average, fixations clustered around the edges of the bar.

For the sake of illustration, saccade target selection is now solely determined by those mechanisms that receive a different input across the saccade. That is, the target for the next movement is chosen randomly from one of the four locations coded by these mechanisms. After the second saccade, there are three mechanisms that receive a novel input, disregarding the fovea itself. Again, the system chooses one of the three locations randomly. We let this cycle continue for a sufficiently large number of saccades and show the distribution of horizontal fixation positions in the histogram at the bottom of the figure. It is clear that the system concentrates its fixations around the two edges of the bar. That is, guidance by trans-saccadic luminance differences results in fixations that are clustered around the image discontinuities. These areas are typically the informative regions in an image, as they tend to indicate object boundaries, occlusions, or internal discontinuities. 

Analysis of human free viewing data has shown that out of many image features that can be used, high spatial frequency edges are the most reliable image-based predictor of fixation locations (Baddeley & Tatler, 2006). We have proposed a simple mechanism that could direct the eyes to inspect these informative regions of the visual scene without relying on analyzing high spatial frequency information that is (a) poorly visible in the periphery and (b) becomes available later over the time scale of a fixation duration (Frazor, Albrecht, Geisler, & Crane, 2004). The trans-saccadic difference image may be thought of as a kind of spatiotemporal filter, analogous to the filters involved in detecting visual motion (Adelson & Bergen, 1985). Such mechanisms respond when they receive different inputs (luminance levels) from different regions of space at different times. In the case of trans-saccadic luminance differences, it is the saccade itself that slices space and time in discrete chunks. Finding image discontinuities is important for image segmentation and interpretation but need not involve any complex spatial analysis at the short time scales of individual fixations. The inherently sequential nature of active vision allows the system to find these informative regions with a very simple comparison of the local luminance at the same retinotopic location across eye movements. 

Acknowledgements

This work was supported by an EPSRC Advanced Research Fellowship EP/E054323/1 to CL and grant DFG GE 879/9 from the Deutsche Forschungsgemeinschaft to KRG. We thank Martin Rolfs for his helpful comments on an earlier draft of this article. 

Commercial relationships: none. 

Corresponding author: Casimir J. H. Ludwig. 

Email: [e-mail] 

Address: University of Bristol, School of Experimental Psychology, Bristol Vision Institute, Bristol, UK. 

Footnotes

¹   It is worth noting that the latencies were much longer than in previous saccadic choice tasks of this type (e.g., Beutter, Eckstein, & Stone, 2003; Caspi, Beutter, & Eckstein, 2004; Ludwig et al., 2005; Ludwig, Eckstein, & Beutter, 2007). For instance, the study by Ludwig et al. (2005) was equivalent to the current experiment, but for the initial saccade from the periphery to the center. These authors reported latencies on the order of ∼300 ms, about 200 ms shorter than observed here. However, note that even the single-target latencies were long, so the overall latency increase cannot be attributed to a more difficult decision problem (the overall latency cost of having to make a choice came to ∼111 ms). Instead, it is likely that the latency increase stems from some perisaccadic suppression of peripheral vision. That is, latency is defined with respect to test display onset. The test display appeared during the saccade. It seems reasonable to suppose that visibility was generally reduced for at least the early part of the test epoch, in both single-target and choice conditions (Matin, 1974; Burr et al., 1994; Wurtz, 2008; Johns, Crowley, Chapman, Tucker, & Hocking, 2009; Ibbotson & Krekelberg, 2011).

Footnotes

²   The same analysis on d′s instead of the raw proportions correct gave almost identical results.

Footnotes

³   The location of the psychometric function may be influenced by nonspatial response bias. Although observers may have had an overall tendency to respond “brighter” or “dimmer,” the randomized nature of the design was aimed to minimize the influence of this form of bias on the relative comparison between conditions. Nevertheless, perhaps the presence of a preview induced a tendency to respond “dimmer” by our subjects. We believe it is unlikely that such a bias would also be responsible for the additional shift in the preview matched function.

Footnotes

 This work was supported by an EPSRC Advanced Research Fellowship EP/E054323/1 to C.J.H.L.