The use of the saccade target as a visual reference when localizing flashes during saccades

Abstract

Flashes presented around the time of a saccade are often mislocalized. Such mislocalization is influenced by various factors. Here, we evaluate the role of the saccade target as a landmark when localizing flashes. The experiment was performed in a normally illuminated room to provide ample other visual references. Subjects were instructed to follow a randomly jumping target with their eyes. We flashed a black dot on the screen around the time of saccade onset. The subjects were asked to localize the black dot by touching the appropriate location on the screen. In a first experiment, the saccade target was displaced during the saccade. In a second experiment, it disappeared at different moments. Both manipulations affected the mislocalization. We conclude that our subjects' judgments are partly based on the flashed dot's position relative to the saccade target.

Introduction

Moving our eyes changes the relationship between retinal stimulation and locations in space. Normally, this does not give rise to an impression that the surrounding has moved, so apparently the shift in the retinal stimulation is anticipated. Image displacements of up to one third of a saccade's length also generally go by undetected if they occur during the saccade (e.g., Bridgeman, Hendry, & Stark, 1975). One explanation for this is that it is considered more likely that one's judgment of direction (change in eye orientation) is incorrect than that displacements would have occurred in the outside world precisely at the time of the saccade. This uncertainty about our movements can explain why visual references are used as spatial landmarks for aligning positions across saccades (Deubel, Schneider, & Bridgeman, 2002; Lappe, Awater, & Krekelberg, 2000). 

One of the most obvious visual references is the saccade target. In this paper, we study the role of the saccade target when localizing flashes that are presented around the moment of the saccade. It has been shown that if a flash occurs near the time of a saccade its location is misjudged (e.g., Bischof & Kramer, 1968; Lappe et al., 2000; Maij, Brenner, & Smeets, 2009; Matin & Pearce, 1965; Pola, 2004; Ross, Morrone, & Burr, 1997; Schlag & Schlag-Rey, 2002). It has already been shown that the pattern of peri-saccadic mislocalization depends on the visual background (Awater & Lappe, 2006; Bischof & Kramer, 1968; Honda, 1993; Morrone, Ma-Wyatt, & Ross, 2005; Ross et al., 1997) and that post-saccadic visual references play an important role in the compression of the perceived locations of flashes presented near the time of saccades toward the endpoints of the saccades (Lappe et al., 2000). The saccade target has also already been stepped repeatedly during saccades in order to induce saccadic adaptation, and to examine how such adaptation influences peri-saccadic mislocalization (Awater, Burr, Lappe, Morrone, & Goldberg, 2005; Georg & Lappe, 2009) and perceptual stability (Bahcall & Kowler, 1999). 

It has been shown that the relative positions of briefly asynchronously presented targets are judged from their retinal positions even if the eyes have moved (Brenner, Meijer, & Cornelissen, 2005) and that the position of the saccade target is similarly mislocalized as that of the flash if it does not remain visible across the saccade (Awater & Lappe, 2006). The latter finding led Awater and Lappe (2006) to propose that peri-saccadic mislocalization consists of two stages. The first, pre-saccadic stage consists of judging the position of the flash relative to the saccade target. In the second, post-saccadic stage, the relative positions are aligned to the post-saccadic scene on the basis of knowledge about the eye's orientation and visual information from references within the scene such as the saccade target. We performed two experiments to directly examine the role of the saccade target in localizing flashes near the time of saccades (in the presence of ample other visual references). 

In Experiment 1, we moved the saccade target either backward or forward during the saccade, so that its position changed but the subject did not notice this happening. We investigated how doing so influences the perceived location of flashes presented before the saccade. If subjects use the saccade target as a reference when localizing the flash, the perceived flash location will be influenced by changing the saccade target's location. In Experiment 2, we investigated whether the perceived flash location is more precise when the saccade target remains on the screen during the whole trial than when it is removed earlier. Most studies present the saccade target for only 50 ms and then remove it from the screen (e.g., Georg, Hamker, & Lappe, 2008; Lappe et al., 2000; Morrone, Ross, & Burr, 1997). If the saccade target is used as a reference for localizing the flash, we expect leaving it on longer to result in less variability (and possibly smaller systematic errors) when localizing targets flashed before the saccade. 

Methods

Subjects

We conducted two experiments in a normally illuminated room. Six subjects volunteered for each experiment (including one of the authors). Two subjects participated in both experiments. Only the author was aware of the specific conditions. All subjects had normal or corrected-to-normal vision. The study is part of a research program that was approved by the ethics committee of the Faculty of Human Movement Sciences. 

Experimental setup

Visual stimuli were presented on a touch screen (EloTouch CRT 19″, 800 × 600 pixels, 36 × 27 cm, 100 Hz) using the Psychophysics Toolbox in MATLAB (Brainard, 1997). The screen was orthogonal to the line of sight, at a distance of 60 cm, and therefore subtended 33° × 25° of visual angle. Eye movements were registered using an Eyelink II (SR Research, Mississauga, Ontario, Canada) at a sample frequency of 500 Hz using the Eyelink toolbox (Cornelissen, Peters, & Palmer, 2002). Subjects were asked to follow a 0.5 degree diameter jumping white dot (108 cd/m²) with their eyes. The dot was presented at a new position every 400 ms. It jumped in steps of 7.6 degrees across a light gray screen (100 cd/m²) and remained on the screen until the next dot appeared. Each jump displaced the dot randomly in one of eight radial directions: horizontal, vertical, and diagonal (but never choosing a direction that would bring the dot within 115 pixels from the edge of the screen). 

After a series of 3, 4, or 5 steps (random with equal probabilities), a 0.5 degree diameter black dot (7 cd/m ²) was flashed for two frames (two very short pulses with a 10-ms interval between them) at one of 5 or 2 different locations (for Experiments 1 and 2, respectively). The flash was presented along an invisible line connecting the last two positions of the white dot. The exact locations were defined with respect to the displacement. In Experiment 1, they were at 30%, 60%, 90%, 110%, or 140% of the last displacement of the white dot ( Figure 1A). During the saccade, the saccade target jumped either 20% backward or 20% forward and remained on the screen. In Experiment 2, the flash locations were at 60% and 140% of the last displacement of the white dot ( Figure 1B). The saccade target was either removed after 50 ms, removed one frame before the flash, removed during the saccade, or it remained on the screen (continuous). The trial ended when the subject indicated where he or she had perceived the flash by touching the screen. 

Figure 1

 

Schematic overview of the possible conditions within each experiment. The figures only show the time between just before the last displacement of the saccade target and shortly after the resulting saccade. Dotted lines: range of times and possible positions of the flash. Dashed lines: saccade target. Solid curve: eye orientation. Gray bar: saccade duration. (A) Experiment 1: the saccade target jumped either forward (blue) or backward (red) during the saccade. (B) Experiment 2: the saccade target was either removed after 50 ms (black), one frame before the flash (green), or during the saccade (cyan), or else it remained on the screen until the response (magenta). The green dot is an example to indicate the moment of the flash corresponding with the depicted length of the dashed green line (in this condition, the time at which the saccade target disappeared depended on the timing of the flash).

Calibration

To synchronize the eye movement recordings with the images presented on the screen, we presented two flashes at the same time. One of them was the flash that the subject had to localize. The other flash (in the lower right corner of the screen) was used to synchronize the eye movement recordings with the images presented on the screen and was not visible to the subject. We measured the moment of this second flash with a photodiode that was attached to the lower right corner of the screen. The photodiode sent a signal to the parallel port of the Eyelink computer. This signal was registered in the data file on the Eyelink computer. The temporal relationship between such a record and the record of the eye orientation at the moment of the flash was previously determined by using the photodiode to drive an infrared lamp that “blinded” one of the Eyelink's infrared cameras. Because the photodiode was placed in the lower right corner, and the flash was presented at different locations on the screen, the real timing was only known to within a few milliseconds (we did not correct for the temporal effects of variation in the position of the flash on the screen). For trials in which no signal was registered on the parallel port (due to technical failure; 27% of all trials), we estimated when the flash had occurred from the average delay (17 ms) between the record of the command to show the flash (that was also recorded on the Eyelink computer) and the record of the signal on the parallel port on trials in which there was such a signal. 

Before each session, the subject was asked to calibrate the touch screen using the standard nine-point calibration provided by EloTouch and to calibrate the Eyelink II using the standard nine-point calibration procedure of the Eyelink II. 

Procedure

In order to manipulate the saccade target's position—or make it disappear—during the saccade, we had to detect saccades rapidly online. For this, we used a displacement threshold of 0.3° (4% of the displacement of the white dot) from the gaze orientation at the moment the saccade target was displaced. For the data analysis, we used a more elaborate method to detect saccades (see below). 

Because the mislocalization only occurs around the moment of the saccade, we wanted to present flashes near the time of saccade onset on as many trials as possible. From previous experiments (Maij et al., 2009), we knew the average saccadic reaction times under similar conditions. We presented flashes within a range of 100 ms around the anticipated moment of saccade onset. The black dot was flashed on the screen for two frames at one of the possible flash locations (defined in relation to the last displacement of the white dot). 

The subjects were asked to touch the screen at the location at which they saw the black flash. If no new white dots appeared and the subject had not seen a black flash (for instance because he or she blinked), the subject indicated having missed the flash by touching the screen in one of the corners far from the location where they perceived the last white dot. In total, there were 360 trials in each session. Subjects performed between six and eight sessions for Experiment 1 and between seven and ten sessions for Experiment 2. 

Data analysis

We used the gaze of the right eye to determine various characteristics of the saccades, and the first location at which the finger touched the screen as the perceived position. For an eye movement to be considered to be a saccade, its speed had to exceed 35°/s for at least two consecutive samples (4 ms). The saccade end was determined as the first sample for which the speed was below 35°/s. We discarded trials in which the touched location differed by more than 180 pixels (7.6 degrees) in the direction of the saccade and 90 pixels perpendicular to the direction of the saccade from the actual location of the flash (this will remove trials in which the subject touched one of the corners or in which he or she accidentally touched the screen with another part of the hand). We also discarded trials if the length of the saccade differed by more than 2 degrees from the median saccade length. Furthermore, we discarded trials in which the saccadic reaction time was less than 125 ms or more than 300 ms. In the conditions in which the saccade target was displaced during the saccade ( Experiment 1) and in which the saccade target was removed during the saccade (one of the four conditions of Experiment 2), we discarded any trials in which we failed to change the image during the saccade. 

We only analyzed the mislocalization in the direction of the saccade: the component of the vector between the touched location and the true location of the flash in the direction of the last displacement of the dot. We plotted these signed errors as a function of the different moments of the flash. If the flash was presented before saccade onset (or up to 10 ms after saccade onset), we consider its timing (the first of the two frames) relative to saccade onset. If the flash was presented after the end of the saccade (or no more than 10 ms before the end of the saccade), we consider its timing relative to the saccade end. To draw a smooth curve through the data (for each condition, i.e., each flash position and saccade target manipulation), we averaged the errors for each subject and condition with weights based on a (moving) Gaussian window ( σ = 10 ms). The smooth curve was drawn as long as there were at least 5 data points within ± σ of the peak of the Gaussian. We will refer to this curve as the mislocalization curve. 

The variability of the errors around the mislocalization curve is determined in a similar way as the smooth curve through the data points. For each time sample, we calculated a standard deviation on the basis of the weighted difference (same Gaussian window) between the positions of each data point and the value of the mislocalization curve at that time sample. We then averaged these standard deviations across the time samples. 

The method that we use to quantify the mislocalization in terms of compression and shift is new. It is different from the method used by Lappe et al. (2000). We took the value of the mislocalization curves at each flash location and fit a line through these values (see Figure 2). We did so for every time sample and every subject separately. The slope of this line indicates the extent to which there was a compression of the perceived position of the flash toward the saccade target. If the perceived positions were veridical (no compression), the slope of the line would be 1. If the flashes were all perceived at the same position, the slope would be zero. Compression was therefore defined as 1 − arctan(θ), where θ is the angle derived from the fit. Assuming that any compression would be toward the saccade target (Awater et al., 2005; Lappe et al., 2000; Morrone et al., 1997; Ross et al., 1997), the shift was defined as the offset of the fit for a (hypothetical) flash at the saccade target (i.e., at 7.6° in Figure 2). Averaging across subjects provided estimates of compression and shift for each time sample. 

Figure 2

 

Values of the mislocalization curves (perceived positions) for one subject at one time of the flash (flash at saccade onset), showing how compression and shift were determined. Black dotted lines: saccade target position. Red dotted line: predicted perceived position for flash at saccade target position. Dashed diagonal line: veridical percept. Dots: values for individual flash locations. Solid line: least squares fit to the dots. Compression is defined as 1 − arctan( θ). Shift is the error in perceived position for a flash presented at the saccade target.

Statistics

All comparisons were conducted with paired t-tests (across subjects). In Experiment 1, we only had two conditions (jump forward or jump backward). We performed separate paired t-tests for every time sample to determine whether there are significant differences between the two conditions, both in the mislocalization curves (for each flash position) and in compression and shift. In Experiment 2, we had four conditions. We used the condition in which the saccade target was present for 50 ms as a baseline for similar t-tests. We compared every time sample of every curve with the corresponding time sample of the 50-ms baseline condition. 

Results

Eye movements

We obtained useful localization judgments on 34% ± 5% (mean ± standard deviation across participants) of the trials in Experiment 1 and 56% ± 5% in Experiment 2. The other trials were rejected for the reasons mentioned in the Data analysis section. Localization judgments were ignored when the saccade length was either too short or too long (14% of the trials for Experiment 1 and 18% for Experiment 2), when there was no detectable saccade near the moment of the flash (because the saccade latency was too long or too short; 19% for Experiment 1 and 9% for Experiment 2), or when the eye tracker could not identify the pupil (15% for Experiment 1 and 11% for Experiment 2). For Experiment 1, another 17% of the trials were discarded because the saccade target did not jump during the saccade. For Experiment 2, 5% of the trials were discarded because the saccade target was not removed during the saccade (whereas it should have been). Furthermore, 1% ( Experiment 1) and 2% ( Experiment 2) of the trials were removed because the screen was touched more than 180 pixels (7.6 degrees) in the direction of the saccade or 90 pixels perpendicular to the direction of the saccade from the actual location of the flash. In 760 of these 773 trials, the subject clearly touched one of the corners of the screen. In the remaining 13 trials, the subjects may have touched the screen by accident (they sometimes repeated having done so), but these may also represent extremes in mislocalization. 

Figure 3A shows the saccade lengths for one subject's individual trials when the flash was presented at various times relative to saccade onset (or saccade end). For trials in which the flash was presented nearer than the saccade target (the trials represented by blue dots), the saccade amplitude was smaller than for trials in which the flash was presented beyond the saccade target (the trials represented by red dots). A smooth line was drawn through each set of data points by averaging with weights based on a moving Gaussian window ( σ = 10 ms). This was done separately for each subject and flash condition. The mean of the six subjects' curves is shown in Figure 3B. The results show that the saccade length is influenced by the flash position if the flash occurs more than about 40 ms before saccade onset. 

Figure 3

 

Saccade amplitudes in Experiment 2. Dots: saccade amplitudes on individual trials (red for trials in which the flash was presented beyond the saccade target and blue for trials in which the flash was presented nearer than the saccade target). Amplitudes are shown as a function of the time of the flash relative to saccade onset (or saccade end). Curves: smoothed averages of the dots of the same color. Dashed lines: the two flash locations. Solid line: saccade target. Gray bar: average saccade duration. (A) A single subject's data. (B) Mean curves across 6 subjects. Transparent bands: standard error of the mean (across subjects).

Mislocalization

Experiment 1

The perceived positions in Experiment 1 ( Figure 4) show that the saccade target plays an important role when localizing flashes. When the saccade target jumped forward or backward during the saccade, the subjects' judgments were biased in that direction. If the saccade target's position is used to evaluate the amplitude of the saccade, and to correct for any discrepancy between the intended saccade amplitude and the true amplitude as judged from the saccade target's (retinal) position after the saccade, we expect the difference to be a pure shift. The average compression and shift are shown in Figure 5. 

Figure 4

 

Mislocalization curves of Experiment 1. The saccade target either jumped forward (dashed red line) or backward (dashed blue line) during the saccade. Black horizontal lines: saccade targets. Dotted lines: the five flash positions. Gray bar: average saccade duration. (A) A single subject's mislocalization pattern. The dots show errors for individual flashes. The curves are smoothed averages of the dots of the same color. (B) Mean mislocalization pattern across all six subjects. The thick sections of the red curves indicate that there was a significant difference between the two conditions.

Figure 5

 

(A) Mean compression and (B) shift of the six subjects. The saccade target either jumped forward (red) or backward (blue) during the saccade. Gray bar: saccade duration. Dashed red and blue lines: saccade target displacements. Thick sections of the red curves show the times at which there was a significant difference between the two conditions. For further details, see the Methods section and Figure 2.

We found a significant shift in response to the change in target position when the flash was presented before the saccade ( Figure 5B). The dashed colored lines represent the magnitude of the shift that would be expected if the subjects had based their judgments exclusively on the position relative to the saccade target. The actual contribution of the saccade target is approximately 30% of this magnitude. 

Just before the saccade, we found the frequently reported compression toward the saccade target (despite this being a pointing task; see Morrone et al., 2005). However, we found a small expansion for both conditions for flashes presented longer than 10 ms before the saccade, Cho and Lee (2003) found some post-saccadic expansion, but pre-saccadic expansion is unexpected, so we looked at its origin in more detail. In Figure 6, we show an example of a fit that gives rise to an expansion rather than a compression. It seems as if there may be an additional systematic difference between the perceived position for flashes that were presented closer than the saccade target and ones presented beyond the saccade target. 

Figure 6

 

Perceived positions of the flashes when they were presented 60 ms before saccade onset. Data for one subject. The saccade target jumped backward. Dashed line: veridical percept. Dotted lines: saccade target position. Red line: best fit to all points. Black lines: separate “fits” for flash locations closer and further than the saccade target.

Experiment 2

In Experiment 2, we expected to find an increase in precision (and accuracy) when the saccade target was visible for a longer time. Results for the flash location beyond the saccade target ( Figure 7) are consistent with the saccade target contributing to localization accuracy, but when the flash was presented nearer than the saccade target there was no difference between the conditions. 

Figure 7

 

Mislocalization curves of Experiment 2. The saccade target was either removed after 50 ms (black), one frame before the flash (green), or during the saccade (cyan), or else it remained on the screen until the response (magenta). Dotted lines: the two flash positions. Black horizontal lines: saccade targets. Gray bar: saccade duration. (A) A single subject's mislocalization pattern. (B) Mean mislocalization patterns across six subjects. The thick portions of each curve represent the times at which the errors were significantly different from those when the saccade target was present for 50 ms (black curve). For further details, see Figure 4.

The longer the saccade target remains visible, the more suitable it is as a visual reference. It is probably especially useful if it remains visible until after the saccade. We therefore expect the variability to depend on how long the saccade target remains visible. We compared the variability relative to the smoothed curve for the 50-ms condition with that in the other three conditions ( Figure 8). We only considered trials for which the flash was presented before saccade onset, but not more than 40 ms before saccade onset to avoid including trials in which the flash may have influenced the saccade. The variability was determined for each subject and compared across conditions with paired t-tests. When the flash was presented beyond the saccade target (140%), the variability decreased with increasing duration of the target display time. It was significantly smaller when the saccade target remained visible ( p < 0.05). In that case, the variability was not significantly different from the variability in Experiment 1 (in which the saccade target was displaced, but it remained visible; dashed line). When the flash was presented nearer than the saccade target (60%), the only significant difference was that the variability was larger in the 50-ms condition than when the saccade target was removed one frame before the flash ( p < 0.05). 

Figure 8

 

Variability in the perceived location (with respect to the mislocalization curves) for flashes presented during the last 40 ms before saccade onset. Every bar indicates the variability for one flash location and saccade target duration, with the standard error across subjects. Dashed lines: the average variability in Experiment 1 for the same flash location.

Discussion and conclusion

When the saccade target jumped forward or backward during the saccade ( Experiment 1), the subjects' judgments were biased in that direction. This implies that the subjects localized the flashes with respect to the saccade target (see Figure 4B). This is especially evident for the flashes that were presented beyond the saccade target. The differences in the perceived locations between the two conditions are smaller when the flash is presented closer than the saccade target than when it is presented beyond the saccade target (i.e., they are larger for larger retinal eccentricities). This is not the first example of differences in mislocalization for flashes presented closer than and beyond the saccade target (see Kaiser & Lappe, 2004). This difference suggests that the extent to which people rely on the saccade target depends on the retinal positions of the flash and the saccade target. Such a dependency means that when we shift the saccade target the perceived position of the flash shifts to different extents for different flash locations, which can be expected to give rise to a difference in compression between the conditions, as can indeed be observed in Figure 5A (the significant difference between the two curves). 

In Experiment 2, we found less variability in the perceived position when the saccade target remained visible than when it disappeared before the saccade, but only for the more distant flashes (right panel of Figure 8). Considering the above-mentioned dependence of the influence of the change in saccade target position on eccentricity, one would expect the influence of the saccade target duration to be most evident for the more distant flashes. However, we found no influence of saccade target duration for the nearer flash location except perhaps that removing the saccade target just before the flash increased the variability in localizing the flash (left part of Figure 8). For the more distant flashes, we found that localization was not only less variable but also more accurate for a longer display time of the saccade target ( Figure 7B; far target), which is consistent with reports by Dassonville, Schlag, and Schlag-Rey (1995) and Honda (1993). Honda (1993) showed that when there is a visible background (a map of Japan) subjects are more accurate than in total darkness. Dassonville et al. (1995) found that localization was more accurate if the saccade target was presented longer. 

We performed our experiments in a normally illuminated room. For targets flashed in the dark, only the saccade target can be used as a reference for relative position judgments, but in experiments such as ours, subjects may use many structures in the scene as references. Thus, our estimate ( Experiment 1) that subjects relied to approximately 30% on the position relative to the saccade target is only an indication of the extent to which the position relative to the saccade target can be used. If the experiments were conducted in the dark, the role of the saccade target would possibly be stronger, and with more structure on the screen it may have been weaker. 

One further important note is that subjects are able to adjust their saccade length up to 40 ms before saccade onset ( Figure 3). This can also be seen in Figure 3 of Awater and Lappe (2006). This implies that when we interpret mislocalization curves across conditions we have to consider changes to the saccade amplitude for flashes presented more than 40 ms before saccade onset. 

Our main finding is that the saccade target is used as a reference when localizing a flash that is presented near the time of a saccade (as suggested by Awater & Lappe, 2006). We estimate that the contribution of the relative positions of the flash and the saccade target to the perceived position of the flash is about 30% under these conditions, although it depended on their positions so a precise value cannot be given. 

Acknowledgments

We thank Siro Otten and Frans-Josef Halkens for their technical support. This research was supported by the Netherlands Organization for Scientific Research (NWO, ALW Grant 816-02-017). 

Commercial relationships: none. 

Corresponding author: Femke Maij. 

Email: [e-mail]. 

Address: Research Institute MOVE, Faculty of Human Movement Sciences, VU University, Van der Boechorststraat 9, 1081 BT Amsterdam, The Netherlands.