Spatial contexts can inhibit a mislocalization of visual stimuli during smooth pursuit

Abstract

The position of a flash presented during pursuit is mislocalized in the direction of the pursuit. Although this has been explained by a temporal mismatch between the slow visual processing of flash and fast efferent signals on eye positions, here we show that spatial contexts also play an important role in determining the flash position. We put various continuously lit objects (walls) between veridical and to-be-mislocalized positions of flash. Consequently, these walls significantly reduced the mislocalization of flash, preventing the flash from being mislocalized beyond the wall ( Experiment 1). When the wall was shortened or had a hole in its center, the shape of the mislocalized flash was vertically shortened as if cutoff or funneled by the wall ( Experiment 2). The wall also induced color interactions; a red wall made a green flash appear yellowish if it was in the path of mislocalization ( Experiment 3). Finally, those flash–wall interactions could be induced even when the walls were presented after the disappearance of flash ( Experiment 4). These results indicate that various features (position, shape, and color) of flash during pursuit are determined with an integration window that is spatially and temporally broad, providing a new insight for generating mechanisms of eye-movement mislocalizations.

Introduction

One important role of the visual system is to identify accurate positions of objects in visual space. It is known, however, that the position of a flash is mislocalized if it appears before or during eye movements (Ross, Morrone, Goldberg, & Burr, 2001; Schlag & Schlag-Rey, 2002), the phenomena called saccade-induced (Lappe, Awater, & Krekelberg, 2000) or pursuit-induced (Brenner, Smeets, & van den Berg, 2001; Mitrani & Dimitrov, 1982; Nijhawan, 2001) mislocalization (Figure 1A). Although precise neural mechanisms have been unclear, these mislocalizations were basically understood by assuming that the visual information from the retina is combined with signals concerning eye's position or orientation (eye position signal or EPS) (Schlag & Schlag-Rey, 2002). While there would be a long afferent delay (at least 60–100 ms) from an occurrence of a visual event (flash) to its perception in the brain, the EPS has no delay and can be obtained even before an execution of actual eye movements as an efferent copy from the oculomotor regions (Sommer & Wurtz, 2006; Thier & Ilg, 2005). Thus, if the brain matches the flash image with the EPS available at the moment of perception of the flash, the flash would be localized to a systematically displaced position in a direction of eye movement. 

Figure 1

 

Pursuit-induced mislocalization and four wall positions in Experiment 1. (A) Conventional pursuit-induced mislocalization. When observers pursue the moving target (black rectangle), the perceived position of the flash (white rectangle) is shifted in the direction of the pursuit (arrow). (B) A condition where the continuously lit object (wall) is placed between the physical and to-be-mislocalized positions of the flash. (C) Four positions of the wall in Experiment 1. L: Low, N: Near, M: Middle, F: Far. During the pursuit, the wall was presented at one of the four positions randomly determined. Relative distances between the flash and wall are shown in the unit of visual degree. The flash was always presented at the position just below the pursuit target. Although the pursuit target and walls are shown in black, they were actually white rectangles presented on the black screen. Likewise, the actual color of the flash was red (see Methods).

As predicted by this model, previous studies have found that magnitudes of pursuit-induced mislocalization can be modulated by the velocity of pursuit (Kerzel, Aivar, Ziegler, & Brenner, 2006), distance or position of the flash from the fovea (van Beers, Wolpert, & Haggard, 2001), and intensity of the flash (Mita, Hironaka, & Koike, 1950). These factors would influence the size of mislocalization by changing the “temporal” mismatch between visual (afferent) and eye position (efferent) signals (Mateeff, Bohdanecky, Hohnsbein, Ehrenstein, & Yakimoff, 1991). On the other hand, recent studies have reported that the “spatial” contexts around the flash also play a substantial role in modulating the mislocalization magnitudes (Awater & Lappe, 2006; Brenner & Cornelissen, 2000; Honda, 1993; Lappe et al., 2000; Lappe, Kuhlmann, Oerke, & Kaiser, 2006). For example, Lappe et al. (2000) showed that the saccade-induced compression of flash positions (Ross, Morrone, & Burr, 1997) was seen when visual references were present on the screen but not seen when they were absent, suggesting that the information processing for the egocentric localization of the flash (without visual references) is distinct from that for the relative localization (with visual references). The same type of the separation between egocentric and relative judgments was also found in the pursuit-induced mislocalization (Brenner & Cornelissen, 2000). 

Those studies above investigated the role of spatial contexts when they were used as visual references (Lappe et al., 2000) or a background (Honda, 1993) on the screen that provided a cue on an allocentric space of the subjects. However, another function of the spatial context is that it would signal the existence of the objects at specific positions in the visual space, regulating the spatial configuration around the flash. Based on this idea, here we tested a possibility of more radical reorganizations of the flash position in which those spatial contexts were used as obstacles in the path of the mislocalization of the flash. Specifically, we put various types of static objects (walls) between the veridical flash location and position where the flash would normally be mislocalized (Figure 1B). Unlike the brief flash that is easily mislocalized, those objects with a long duration are not subject to the mislocalization (Rotman, Brenner, & Smeets, 2005; van Beers et al., 2001). Does the visual system allow the mislocalized flash to jump beyond the wall? The classical model above suggests positively because the magnitudes of mislocalization were basically determined by the temporal mismatch between an afferent signal and EPS. Unless the wall induces some changes in the speed of visual processing of the flash or eye movements, no effect should be expected on the mislocalization magnitudes. In contrast, if the visual system determines the position of the flash considering the spatial contexts around it (as suggested by the recent studies showing the influence of spatial factors on the mislocalization), an existence of wall may impose some spatial restrictions on the mislocalization. 

Methods

Subjects

We conducted four experiments in the present study. Six, five, five, and eight subjects participated in main sessions of 1– 4, respectively. All subjects in all experiments had normal or corrected-to-normal visual acuity. Informed consent was received from each participant after the nature of the study had been explained. Approval for these experiments was obtained from California Institute of Technology IRB Committee. 

Stimuli

Stimuli were presented on a CRT monitor with a resolution of 1152 (H) × 864 (V) pixels at a refresh rate of 60 Hz. In all experiments, we used a linear motion display (Brenner et al., 2001) (Figure 1) to induce a pursuit-induced mislocalization of the flash. These stimuli were generated using Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) on Matlab (Math Works, Natick, MA). One trial began with an appearance of a white rectangular bar (a pursuit target, 0.28 × 1.39 deg, 63 cd/m²) at the left edge of the black screen (0.3 cd/m²) in the dark room. After a random interval (1–2 s), the bar began to move rightward at a speed of 33 deg/s, reaching the right edge of the screen in 833 ms. On the trajectory of that motion, a red flash (0.28 × 1.39 deg, 15 cd/m², CIE coordinates: x = 0.63, y = 0.34) was presented briefly for 1 frame at the position exactly below the moving bar. A minimum distance between a lower edge of the moving bar and an upper edge of the flash was 0.83 deg. The subjects were instructed to pursue the moving bar until the right edge of the screen, without making any eye movements to the red flash. In every trial of all experiments, their eye movements (either left or right eye) were continuously monitored at 500 Hz using the EyeLink II system (SR Research) combined with the Eyelink Toolbox (Cornelissen, Peters, & Palmer, 2002) on Matlab. The same trial was repeated again if they quitted or deflected the pursuit before reaching the right edge of the screen. 

Experiment 1

In Experiment 1, we investigated whether a white static object (wall, 0.17 × 1.39 deg, 63 cd/m ²) at various positions on the screen had a significant influence on magnitudes of mislocalization induced by the pursuit. The wall was placed at one of four positions relative to the flash ( Figure 1C): Near (N), Middle (M), Far (F), and Low (L). It appeared on the screen when the pursuit target began to move and had remained until the end of the trial. The task of the subjects was to indicate the position of the red flash they perceived, neglecting the white wall. Using a mouse pointer presented 1 s after the end of pursuit, they clicked a center of the flash perceived during the pursuit. An initial position of the mouse pointer was randomly determined for each trial. To prevent the subjects from predicting the timing of the flash, the flash was presented at variable positions along the motion trajectory (367–433 ms after motion onset of the pursuit target). One session contained 20 trials and the subjects completed four sessions after a brief practice session of 10 trials. The four positions of the wall were randomly intermixed across trials. 

Experiment 2

In the second experiment, we further investigated the flash–wall interaction by changing the rectangular wall in Experiment 1 into various shapes. While the size of the flash was identical as the previous experiment (0.28 × 1.39 deg), the wall was changed into one of six shapes in Figure 3A. Except for the control (Low) condition, these walls (63 cd/m ², 0.28 width) were placed at the position 0.42 deg to the right of the flash, the same position as the Near condition in Experiment 1. The walls in the Large- and Small-hole conditions consisted of two rectangles aligned in the vertical direction, as if the walls had “holes” in their centers. Sizes of the holes were 0.89 deg in the Large-hole and 0.39 deg the Small-hole conditions. In the Lower- and Upper-half conditions, a short rectangle (0.28 × 0.69, half length of the flash) was presented at the lower and upper positions near the flash, respectively. Finally, two triangles (0.28 width × 0.42 length) were placed in opposite directions in the Triangle condition, so that the size of the hole changed linearly along the horizontal axis. After making the pursuit till the right edge of the screen, the subjects reported the size and position of the flash they perceived by circumscribing a rectangular area on the screen using a mouse pointer. Each session contained 20 trials, and the subjects completed 3 sessions after a brief practice session of 10 trials. The trials in the six conditions (10 trials for each) were randomly intermixed. 

Control experiments for the pursuit-induced backward masking

In the five conditions (except for Low) of Experiment 2, the flash and wall of the same width (0.28 deg) were placed at the same height on the screen. When the horizontal pursuit was made over this configuration, some regions of the flash and wall should be carried into the same position on the retina in rapid succession, which raises a possibility of the pursuit-induced backward masking (White, 1976) of the flash by wall images. This masking effect may make the perception of some parts of the flash (that have the wall on their right) difficult, leading to changes in the perceived shape of the flash, regardless of the flash–wall interaction in configurations. We therefore conducted two control experiments to examine the possibility of the pursuit-induced masking in our stimulus set. In the first control experiment (Figure 4A), the subjects (N = 3) made a present/absent judgment of a brief flash backwardly masked by the wall continuously presented. In 50% of the trials, a red flash (0.28 × 1.39 deg) was presented briefly (16.7 ms) at the position 0.42 deg to the left of a white wall (0.28 × 1.39 deg, continuously presented from the beginning to end of the pursuit), so that the rightward pursuit would carry the wall image onto the same retinal position as the flash image (the flash-present trial). In the remaining 50% of the trials, only the wall was presented continuously (the flash-absent trial). The task of the subjects was to judge whether the red flash was present or not after making the pursuit till the right edge (the flash detection task). The flash-present and flash-absent trials were randomly intermixed and the subjects completed one session of 20 trials. If there was any masking effect in the main sessions of Experiment 2, the same effect should be also observed in the flash-present trial in this experiment, making the accuracy of the task far below 100%. 

Although the first control experiment can discern whether the backward masking is strong enough to impair the detection of the flash, one may argue that this task would tell nothing about an effect of the masking on the perception of the flash as a whole. Namely, the backward masking may not be strong enough to impair the detection of the flash, but may be strong enough to partly impair its visibility. We thus conducted the second control experiment ( N = 3) in which the subjects needed to identify the overall shape of the flash (the flash identification task). As shown in Figure 4B, a small red flash (0.28 width × 0.42 length) was presented either at the upper, lower, or both regions next to the wall, and the subjects were required to discriminate those three types of trials after making the pursuit. Due to the commonality of local structures (between the Upper and Both, or between the Lower and Both), they had to discriminate the shape of the flash based on its whole configuration. Three types of trials were randomly intermixed and the subjects completed one session of 21 trials. 

Experiment 3

Although colors of the flash and wall were red and white in previous experiments, they were changed into green (CIE coordinates: x = 0.27, y = 0.61) and red ( x = 0.63, y = 0.34) in Experiment 3, in order to investigate the chromatic interaction between the flash and wall. In the main condition of Experiment 3 ( Figure 6A), the red wall (0.28 × 1.39 deg) was presented 0.42 deg to the right of the green flash (0.28 × 1.39 deg), the same position as Near condition in Experiment 1 (Right-wall trial). If there are some chromatic interactions, the color of the flash (green) would be mixed with the color of the wall (red), producing a yellowish percept. However, as in Experiment 2, the horizontal pursuit would induce a successive input of the green flash and the red wall on the same retinal location. This rapid input of the green and red may produce a yellow percept (Kelly, 1983; Wisowaty, 1981), irrespective of spatial interaction between the flash and wall. To remove this ambiguity in interpretation of a yellowish percept, we set a control condition where the red wall was put in the opposite side (0.42 deg to the left) of the flash (Left-wall trial; Figure 6B). If the color interaction is elicited merely by a successive input of green and red, the yellowish percept should be reported in both main (Figure 6A) and control (Figure 6B) conditions. Trials of these two conditions were randomly intermixed within a session (20 trials) and the subjects were required to report the color of the flash in each trial. To prevent the subjects from confusing the flash with wall, they had been informed before the experiments that the veridical color of the flash was more or less greenish, so that they could easily discriminate the flash from the red wall by its color. After making a pursuit until the right edge, the subjects reported the perceived color of the flash by adjusting a color of a comparison bar presented at the center of the screen. They could change the color of the comparison bar from green (x = 0.27, y = 0.61, the original color of the flash) to yellow (x = 0.41, y = 0.50, a perfect color composition) using two buttons (one to increase the red component of the bar and another to decrease it). As in the previous experiments, there was no time limitation on the responses of the subjects. 

Experiment 4

In the final experiment, we manipulated the timing of the presentation of the wall. Although the wall was continuously presented in the whole period (833 ms) from beginning to end of the target motion in 1– 3, we divided it into three intervals: Pre, During, and Post. In the Pre condition, the wall was presented only before an onset of the flash (duration of the wall: 400 ms). Likewise, the wall appeared only during the moment of the flash (16.7 ms) in the During condition, and only after an offset of the flash in the Post condition (416.7 ms). 

In the first session of Experiment 4, we investigated an influence of the presentation timing of the wall on the shape interaction between the flash and wall (reported in Experiment 2). The wall consisted of two white rectangles (0.28 width × 0.42 length) and was located 0.42 deg to the right of the red flash (0.28 × 1.39 deg) ( Figure 7A). As in Experiment 2, the subjects were asked to report the perceived size and position of the flash by circumscribing a rectangular area using a mouse pointer. Although the walls in Experiment 2 remained visible when the subjects provided answers, they were removed from the screen in the judgment phase of Experiment 4, in order to equalize the task difficulty across three (Pre, During, and Post) conditions. The session contained 21 trials, and three types of trials (7 trials for each) were randomly intermixed. The lengths of the flash reported by the subjects were compared among those three timings. In the second session ( Figure 7C), we tested the color interaction shown in Experiment 3. All stimuli and task were identical to the main (Right-wall) condition in Experiment 3 ( Figure 6A) except that the three timings of wall presentation were randomly intermixed. The colors of the flash reported by the subjects were calculated and their x- and y-coordinates in the color space were compared across the Pre, During, and Post conditions. 

Results

Experiment 1: Positional interaction between the flash and wall

Figure 2A shows the central positions of the flash reported by the subjects in four conditions. For convenience, we set the veridical position of the flash as 0 in the ordinate. In the control (Low) condition, the flash was perceived to be far beyond the position of the wall (the dotted rectangle in Figure 2A), which was consistent with previous studies on eye-movement mislocalization (Cai, Pouget, Schlag-Rey, & Schlag, 1997; Nijhawan, 2001). In contrast, positional relationships between the flash and wall were totally reversed when the wall was placed on the same height as the flash (Near, Middle, and Far). In those conditions, reported positions of the flash were on the left of the wall, indicating that the flash was not allowed to go beyond the wall in most trials. Specifically, the flash in the Near conditions was perceived to be just next to the wall, making a partial overlap between the flash and wall regions. A one-way ANOVA of the perceived flash positions indicated a significant main effect (F(3, 20) = 7.2, p = 0.002) among the four conditions, and post hoc tests with the correction of multiple comparisons showed significant differences between Low vs. Near (p = 0.0002) and between Low vs. Middle (p = 0.007). 

Figure 2

 

Results in Experiment 1. (A) Magnitudes of pursuit-induced mislocalization (mean ± SE across the subjects) in the four conditions. Zero in the ordinate denotes the veridical (physical) position of the flash. The solid and dotted rectangles indicate the location of the wall in each condition. *p < 0.05, **p < 0.01, ***p < 0.001, post hoc tests of one-way ANOVA. (B) Eye movements averaged across the subjects. The pursuit target (dotted black line) began to move at 0 ms and reached the right edge of the screen (27. 2 deg in the ordinate) at 833 ms. The flash was presented at the random timing from 367 to 433 ms after the motion onset (shown in two vertical lines). The data in the four conditions are shown in different colors (Low: black; Near: blue; Middle: green; Far: red). (C) An enlarged illustration of B around the flash timing (350–450 ms). As shown in the error bars (SE across the subjects) in the Low and Far conditions, no differences in eye movements were observed among the four conditions. (D) A correlation diagram of the mislocalization with the flash–eye distance in one subject. Horizontal axis indicates the flash position relative to the eye position at the moment of the flash. Positive values mean that the flash appeared at the “ahead” position of the pursuit. Black and blue circles denote the Low and Near trials, respectively. (E) Distributions of the flash–eye distance (the relative flash positions) in all trials of all subjects, separately shown for the Low and Near conditions. Solid curve indicates the results of Gaussian fitting for each distribution, while the arrow shows a median (Low: 0.77 deg; Near: 0.88 deg).

Eye movements during the reduction in the magnitudes of mislocalization

Figure 2B shows eye movements of the subjects simultaneously recorded. After catch-up saccades at the beginning, eye positions were smoothly shifted along the movement of the pursuit target (dotted-black line). The gain of pursuit averaged across the four conditions at 400 ms (the mean timing of the flash) was 98.8%, indicating the successful pursuits around the moment of the flash. As shown in the error bars in Figure 2C (an enlarged illustration of Figure 2B), no significant difference in eye movements could be observed among the four conditions. Especially, eye movements in the Low (solid black line) and Near (blue) conditions were virtually identical, which made a striking contrast with the large difference in the mislocalization magnitudes between these two conditions ( Figure 2A). These results indicate that the inhibition of mislocalization cannot be attributed to the difference in eye movements across conditions. 

However, Figure 2C also indicates that there is a clear mismatch between the flash position (represented as the dotted-black line because the flash was always presented just below the pursuit target) and the actual eye position at the moment of the flash in each condition. To examine whether this flash–eye mismatch could explain some of the mislocalization effect in Figure 2A, we conducted a trial-by-trial analysis between the flash–eye mismatch and the magnitude of mislocalization. In the horizontal axis of Figure 2D, we showed the location of the “flash” relative to the eye position (at the moment of flash) in each trial and correlated it with the magnitude of mislocalization in the same trial (the vertical axis). We found that, when the gain of the pursuit was small and thus the flash was presented at the “ahead” position of the pursuit (the relative flash position >0), the mislocalization tended to be larger compared to when the flash was at the “behind” position (the relative flash position <0). Those results were consistent with many previous studies (Rotman, Brenner, & Smeets, 2004; van Beers et al., 2001), a phenomenon called “spatial expansion” (Kerzel et al., 2006). Therefore, one possible explanation for our main results in Experiment 1 (the smaller mislocalization in the Near than Low conditions) is that the Low condition might have more numbers of “ahead” trials than the Near condition. In this case, the spatial expansion would increase the overall magnitude of mislocalization in the Low trials, compared to the Near condition containing smaller numbers of the “ahead” trials. 

To examine this point, we investigated the distributions of the flash–eye mismatch in all trials and compared them between the Low and Near conditions ( Figure 2E). Although both distributions were significantly biased into the positive direction, no significant difference was observed in the shape of distribution between the two conditions (Kolmogorov–Smirnov test: χ ² = 2.97, p = 0.45). We therefore concluded that the smaller mislocalization in the Near than Low conditions could not be explained by the mismatch between the eye and flash positions. 

Experiment 2: Flash–wall interaction in shape

The previous experiment showed that the wall prevented the flash from being mislocalized beyond the wall. This raises a question of whether such a modification mechanism is limited only to position, or applicable to more general spatial features such as global configuration and shape. In the second experiment, we thus investigated the perception of the flash when the wall was partially cutoff. 

Mean regions of the flash circumscribed by the subjects were shown in Figure 3B. In the Low condition, there was no change in the perceived shape of the flash, although its position was shifted rightward by the mislocalization. On the other hand, the shapes of the flash were variously distorted in the remaining conditions. In the Small- and Large-hole conditions ( Figure 3A), mean perceived positions of the flash were between the two parts of the wall, and their vertical lengths were significantly reduced compared to the Low condition ( F(2, 12) = 10.5, p = 0.002, one-way ANOVA; Figure 3C). In the Lower- and Upper-half walls, the averaged flash positions were just above and below the wall, respectively, which resulted in vertical shifts of the center of flash ( F(2, 12) = 11.1, p = 0.002, one-way ANOVA; Figure 3D). Finally, the flash in the Triangle condition was reported in the middle of two parts of the wall, as were seen in the Small-hole and Large-hole conditions. 

Figure 3

 

Stimuli and results in Experiment 2. (A) Physical relationships of the flash and the wall in the six conditions in Experiment 2. (B) The areas that the subjects reported to perceive the flash. Mean data across 5 subjects (50 trials) were shown. Except for the control (Low) condition, the perceived areas of the flash were deformed in accordance with the shape of the wall. (C) Changes in the vertical length of the perceived flash areas (mean ± SE across the subjects) in the Low, Large-hole, and Small-hole conditions. A dotted horizontal line denotes the physical length of the flash. (D) Changes in the vertical positions of the perceived flash areas (mean ± SE across the subjects). The centers (in vertical direction) of the perceived flash areas were plotted in the Low, Lower-half, and Upper-half conditions. Zero in the ordinate denotes a center of the physical location of the flash. * p < 0.05, ** p < 0.01, *** p < 0.001, post hoc tests of one-way ANOVA.

One possible reason for those distortions might be a bias when the subjects reported the area where the flash was perceived. Because the wall in each trial was continuously presented until the end of the decision phase, the subjects, for example, might be reluctant to report that they saw the red flash where the wall was, which would result in the partial cutoffs of the flash shape in Figure 3. We thus conducted an additional experiment in which the wall was removed before the subjects provided an answer, using the two out of five subjects in the original experiment. In both subjects, the vertical length of the perceived flash area was reduced in the Large-hole and Small-hole conditions compared to the Low condition ( p < 0.05, figure not shown). Thus, the shape distortions in Figure 3 could not be ascribed to the bias in the decision phase. 

Possibility of the pursuit-induced backward masking of the flash by wall images

A common characteristic of the results in all conditions (except for Low) was that the shortening or missing of the flash selectively occurred in the regions of the flash that had neighboring walls on their right side. Although these changes in the flash shape could be interpreted as a result of the flash–wall interaction (shown in Experiment 1) that prevented the flash from going beyond the wall, there was another possibility that can explain those results: the pursuit-induced backward masking (White, 1976). In order to address this issue, we conducted two control experiments in Figure 4. In the first control experiment (the flash detection task; Figure 4A), all three subjects showed 100% accuracy. In the second control experiment (the flash identification task; Figure 4B), accuracies were 100% in the two subjects and 95.2% in the remaining one subject. These results in the two control experiments indicated that the wall image in the present study had very little effect of impairing the perception of the flash during the pursuit. The changes in the flash shape (Figure 3B) thus cannot be attributed to the pursuit-induced backward masking. 

Figure 4

 

Control experiments of Experiment 2 to examine the possibility of the pursuit-induced backward masking of the flash by the wall image. (A) The flash detection task. The flash was presented 0.42 deg to the left of the wall in 50% of the trial, while only the wall was presented in the remaining 50%. After performing the pursuit, the subjects reported whether the red flash was present or not. (B) Stimulus configurations of the second control experiment (the flash identification task) for Experiment 2. A small red flash (0.28 width × 0.42 length) was presented either at the upper, lower, or both regions next to the wall, and the subjects were required to discriminate those three while doing the pursuit. Performances in both experiments were almost perfect (see Results), indicating that the wall in the present study never inhibited the perception of the flash through the pursuit-induced backward masking.

Changes in flash shapes depending on the flash–wall distance

Figure 5A shows the data in individual 50 trials pooled over 5 subjects in the main session of Experiment 2. These analyses revealed that the changes in the length or vertical position of the flash in Figure 3 were highly dependent on the relative distance between the flash and wall. When the flash was perceived to be left of the wall, all walls had little influences on the shape of the flash. In contrast, when the positional errors were large enough to move the flash beyond the wall, the length and vertical position of the flash were strongly modulated by the configuration of the wall in each condition. Figures 5B and 5C plotted the change of the length and vertical position of the flash as a function of the relative distance between the flash and wall, which showed that these changes became prominent when the right edge of the perceived flash was beyond the left edge of the wall (flash–wall distance >0). This dependency on the flash–wall distance was obviously seen in the last condition where the wall was composed of two triangles (the Triangle condition in Figure 3A). Along with the size of hole linearly decreasing, the vertical lengths of the flash became also shorter ( Figure 5D) and showed a significant negative correlation with the relative distance from the wall ( r = −0.69, p = 0.0003, a significance test for correlation coefficients; Figure 5E). 

Figure 5

 

Distortions of shape of the flash by the wall. (A) Data in individual trials in Experiment 2. Perceived flash areas in 50 trials (5 subjects × 10 trials) are superimposed. Note that, in the four conditions except for the Low, the changes in length ( Figure 3C) and vertical position ( Figure 3D) of the flash became distinct only when the flash was perceived to go beyond the wall. (B) The changes in the flash length as a function of the relative distance between the flash and wall. Positive values in the horizontal axis mean that the right edge of the perceived flash is located beyond (to the right of) the left edge of the wall. Each point shows the vertical length of the perceived flash in each trial (blue: Low; red: Large-hole; green: Small-hole). The six horizontal lines show the mean of all data points in positive and negative regions of the horizontal axis (3 wall shapes × positive or negative). The changes in length in the Large- and Small-hole conditions were clearly seen when flash–wall distance was positive. (C) Same as panel B, but the vertical centers of the Low, Lower-half, and Upper-half conditions are shown. (D) Perceived flash areas in all trials of the Triangle condition. (E) Correlation between the vertical flash lengths and the flash–wall distances in the Triangle trials. Only the trials were included in which the flash–wall distance is 0–0.28 (when the right edge of the perceived flash was between two ends of the triangle). *** p < 0.001, significance test for correlation coefficients.

Experiment 3: Flash–wall interaction in color

Previous experiments showed the spatial interactions (position and shape) between the flash and wall during pursuits. We further investigated whether the same interaction was applicable to color dimension. Colors of the green flash reported by the subjects were shown on the CIE color space in Figure 6C. When the red wall was placed on the right side of the flash ( Figure 6A), the flash color reported by the subject was more yellowish (the yellow circle in Figure 6C, mean ± SE across the subjects, x = 0.34 ± 0.017, y = 0.55 ± 0.012) than the veridical color of the flash (the green square in Figure 6C, x = 0.27, y = 0.61). This change in color cannot be explained by the successive input of the green and red because the perceived color remained green in the Left-wall condition (the green circle in Figure 6C; x = 0.28 ± 0.005, y = 0.60 ± 0.004). The difference between the two conditions were significant both in x- and y-coordinates in the color space ( x: t(4) = 3.14, p = 0.035; y: t(4) = 3.17, p = 0.034, paired t-tests; Figure 6D). 

Figure 6

 

Stimuli and results in Experiment 3. (A) Main condition. The red wall was in the to-be-mislocalized direction (right) of the green flash. (B) Control condition. The red wall was on the left side of the green flash. (C) Results of Experiment 3. Reported colors of the flash (yellow circle: Right-wall; green circle: Left-wall) are plotted on the CIE color space. Positions of physical color of the flash (green square) and wall (red square) are also shown. (D) Comparisons of x- and y-coordinates (in the color space) of the reported flash colors in the Right- and Left-wall conditions (mean ± SE across the subjects). In the Right-wall condition, the reported color was significantly higher in the x-value but lower in the y-value than that in the Left-wall condition, indicating that the red wall placed in the direction of mislocalization made a green flash appear yellowish. Dotted horizontal lines denote the physical x- and y-values of the flash. * p < 0.05, paired t-test.

However, one remaining possibility explaining these results may be a difference in the order of color inputs between two conditions. While the green stimulus hit the retina earlier than red in the Right-wall condition, the pursuit brought the red stimulus first in the Left-wall condition. Thus, color composition in Figure 6D might be elicited by the specific order of color inputs (green then red), not by the spatial interaction of the green flash and red wall. We examined this possibility by testing another control condition in which a green wall was presented to the left of a red flash. Subjects ( N = 3) were required to report the color of the red flash (from red to yellow). If the first green input was critical to the color composition, this condition also should induce a yellowish percept. The reported color of the red flash were x = 0.60 ± 0.013, y = 0.36 ± 0.010, which was not significantly different from the original flash color ( x = 0.63, y = 0.34) ( x: t(2) = 2.06, p = 0.18; y: t(2) = 1.81, p = 0.21, paired t-tests). Thus, the changes in the reported color of the flash ( Figure 6D) cannot be attributed to the difference in the input order of red and green. 

Experiment 4: A critical time period for wall presentation

While we found the flash–wall interactions in various visual features, it remained to be elucidated which period of the presentation is important for the wall to interact with the flash. This question is of a particular interest because there were several studies indicating a relative importance of the post-trajectory as opposed to the pre-trajectory in various visual phenomenon, such as saccadic mislocalization (Lappe et al., 2000), flash-lag effect (Eagleman & Sejnowski, 2000; Krekelberg & Lappe, 1999), and auditory–visual stream/bounce effect (Watanabe & Shimojo, 2001). In Experiment 4, we compared the three presentation timings of the wall (Pre, During, and Post) in the shape and color interactions reported in the previous experiments (Session 1: shape; Session 2: color). 

As shown in Figure 7B, the shortening in the reported length of the flash (Session 1) were selectively observed in the Post condition ( F(2, 21) = 28.1, p < 0.0001, one-way ANOVA). No shape change was observed in the Pre or During condition, indicating that the flash–wall interaction does not require the wall presented in advance or even simultaneously with the flash. In the During condition, the subjects reported that the brief presentation time (16.7 ms) of the wall made it seen as another flash that was mislocalized together with the target flash, which produced no flash–wall interaction. This may be expected from the “common fate” or synchrony between the flash and the wall. On the other hand, one may argue that the During condition was just too brief as a duration to execute any effects. To address this issue, we also tested the Pre and During condition ( N = 3) in which the wall was continuously presented from a motion onset of the pursuit target to an offset of the flash, but no effect on the length of the flash was observed. Consistently, the data in the second (color) session also showed that the change in the perceived color of the flash was seen only in the Post condition both in x- and y-coordinates ( Figure 7D; x: F(2, 21) = 32.9, p < 0.0001; y: F(2, 21) = 33.1, p < 0.0001, one-way ANOVA). These results indicated that presenting the wall after the flash had a crucial effect in inducing the flash–wall interaction in shape and color dimensions. 

Figure 7

 

An effect of the presentation timing of the wall on the distortion and color change of the flash. (A) Session 1 in Experiment 4 (the shape session). The white wall with a hole (size: 0.56 deg) in its center was presented 0.42 deg to the right of the flash in one of three periods: before the onset of the flash (Pre), from onset to offset of the flash (During), and after the offset of the flash (Post). (B) Results in Session 1. Vertical lengths of the reported flash areas (mean ± SE across the subjects). We also tested the Pre and During condition in subset of the subjects (see Results). (C) Session 2 in Experiment 4 (the color session). The red wall (0.28 × 1.39 deg) was presented 0.42 deg to the right of the green flash either in the Pre, During, or Post period. (D) Results in Session 2. The x- and y-values of the reported color of the flash were compared (mean ± SE across the subjects). In both Sessions 1 and 2, the changes in length and color of the flash were selectively elicited in the Post condition, indicating that the presentation of the wall after the flash is critical for distortion and color change of the flash by the wall. *** p < 0.001, post hoc tests of one-way ANOVA.

Discussion

In the present study, we reported that the localization errors of the flash by eye movements could be significantly reduced by placing a wall in a direction of mislocalization. A unique aspect in our results compared to the previous studies investigating the effect of the spatial contexts (Awater & Lappe, 2006; Brenner & Cornelissen, 2000; Honda, 1993; Lappe et al., 2000, 2006) was that we showed the positional selectivity of the wall (spatial context) to inhibit the mislocalization. The mislocalization of the flash was reduced only when the wall was put between veridical and to-be-mislocalized positions of the flash and not observed when the wall was placed elsewhere (the Low condition in Experiment 1). Moreover, and clearly beyond the implications of the earlier studies, subsequent experiments showed that this flash–wall interaction was so strong that the shape and color of the flash was also changed depending on the wall nearby. Those results cannot be explained by the classical model on the eye-movement mislocalization that simply assumes a temporal mismatch between the visual processing of the flash and EPS and further indicate a contribution of the spatial contexts (especially after an offset of the flash) to determine the stimulus features during eye movements. 

In a broader context, our findings suggest that a simple sequential model of the visual information processing, in which the position and shape of the object are determined first independent of the global context, cannot be hold. Instead, various attributes including the location would interact with each other until being settled down in a consistent interpretation of the scene or events. 

An inhibition of positional errors by the wall

Although our data in Experiment 1 showed an inhibitory effect of the wall on the pursuit-induced mislocalization, there may be some other possibilities that can explain these results. First, those reductions in the mislocalization might result from using wall as a spatial marker or visual reference in the dark environment. Compared to previous studies with no visual reference (Brenner et al., 2001; Kerzel et al., 2006), the wall in our study could work as a spatial marker that provided a hint for the flash position, which might reduce the mislocalization considerably (Brenner et al., 2001). However, it should be noted that our inhibition effect was selectively observed when the wall was placed at the same height as the flash (the Near, Middle, and Far conditions). The large magnitude of mislocalization in the Low condition indicates that a power of the wall as a spatial marker was not strong enough to inhibit the positional error. Thus, a crucial factor for the inhibition of the mislocalization would be a position, not an existence, of the wall. 

The second possibility is that the wall might change the speed of visual processing of the flash. Although there was no difference between eye movements among four conditions ( Figures 2B and 2C), it was possible that the wall presented near the flash altered the perceived luminance of the flash, changing its processing speed in the brain (especially when a wall-flash distance was short). According to the classical temporal delay account (see Introduction), this would affect the magnitude of the temporal mismatch between the afferent and efferent signals, resulting in the difference in the mislocalization among the four conditions. However, in our Experiment 1, the physical luminance of the wall (63 cd/m ²) was far higher than that of the flash (15 cd/m ²). This contrast in luminance between the wall and flash would make the flash appear darker (rather than brighter) than actual, leading to a longer afferent delay of the flash from the its presentation to perception (Mansfield, 1973; Purushothaman, Patel, Bedell, & Ogmen, 1998). The magnitudes of mislocalization are thus predicted to be larger when the wall was placed near the flash (e.g., the Near condition), which was not the case as shown in Figure 2A. 

Position of the wall during eye movements

One characteristic of the present study is that the task was an allocentric, not egocentric, localization of the flash. Since the wall was continuously lit until the end of the trial, the subjects would indicate the position or shape of the flash based on the relative spatial relationships between these two. This may raise another concern for an interpretation of Figure 2A; all changes in the perceived position of the flash might be actually induced by the changes in the perceived position of the wall, not the flash. However, this concern would not be applicable to the present results for several reasons. First, previous studies have reported that there was no pursuit-induced mislocalization of the visual objects when the duration of those objects was long enough (>200 ms) (Rotman et al., 2005; van Beers et al., 2001). Given the long duration of the wall in the present study (more than 800 ms in Experiment 1), it is unlikely that the positions of the walls were shifted by the eye movements. Second, our results in Experiment 1 do not support the view above. Since the horizontal positions of the walls were identical between the Near and Low conditions (Figure 1C), their horizontal positions at the moment of the flash should be also the same. Nevertheless, the flash was perceived to the left of the wall in one condition (Near), whereas it was to the far right in another condition (Low). Thus, our data cannot be explained by the shift of wall position by eye movements and indicate that the relative position of the flash (not wall) was changed among the four conditions. 

Sensory-level interaction between the flash and wall

Another issue in interpreting our data of Experiment 1 is whether the reduction in the mislocalization involved the actual changes in visual representations of the flash–wall configurations or just reflected the results of the cognitive- or memory-based inference on the flash positions. Namely, it remains unclear only from Figure 2A whether the flash–wall interaction occurred in the sensory-level or the later cognitive or decision stages in the brain. This point is of particular importance since the subjects in our experiment gave their response well after the event had occurred, meaning that they had enough time to reconstruct the image based on their memory. In that case, the suppression of the mislocalization in Experiment 1 would not reflect the direct change of the visual percepts, but show the results of such cognitive re-interpretation for the rapid event done in hindsight. Such an interpretation is possible but unlikely when one examines the results in Experiment 3. 

In Experiment 3, it was shown that the wall placed near the flash directly changed the color of the flash. Importantly, the subjects had been explicitly informed before the experiments that the veridical color of the flash was more or less green. Nevertheless, the perceived color of the flash was shifted to the yellow when the flash–wall interaction was induced (Right-wall trials; Figure 6A), but not in the reversed order (Left-wall trials; Figure 6B). Those results indicate that the flash and wall images dynamically interacted as two visual representations in the brain, suggesting that our flash–wall interaction occurred in the sensory-level of the neural processing, rather than as a result of cognitive repositioning or reconstruction. 

What are the underlying mechanisms of this color composition? We presume it would be induced as a result of a positional overlap of the flash with the wall image. As shown in Experiment 1 ( Figure 2), putting the wall at the Near position substantially shortened the flash–wall distance represented in the brain, making a collision or spatial overlap between these two in some trials. The yellowish percept thus would be made as a result of the color composition after processing the positional relationship of the flash and wall (although combining opponent colors begins in the retina, previous studies indicated that the color composition could occur also in the cortical level; Hecht, 1928; Hurvich & Jameson, 1951). This concept that the positional processing precedes the color processing is consistent with a previous study using the flash-lag effect (Nijhawan, 1997). 

Integration window of the flash–wall interactions

The data in 1– 3 demonstrated the interactions between the flash and wall in various features of the visual stimuli (position, shape, and color), indicating that our visual system determines features of the flash by taking the spatial contexts (a presence of the wall) into account. Furthermore, the results in Experiment 4 showed that those interactions could be induced without a temporal overlap between the flash and wall, meaning that the various flash–wall interactions resulted from an integration of the information over time. Importantly, the critical time period for the presentation of the wall was found to be after the disappearance of the flash. This indicates that the temporal integration above was made by combining the flash with the information of the wall in the post-flash period. In other words, the final percept of the flash was determined postdictively, considering the wall information after the flash. 

This postdictive characteristic has been found in many psychological phenomenon, such as color phi (Kolers & von Grünau, 1976), and flash-lag illusion (Eagleman & Sejnowski, 2000). A common implication of those effects is that the visual system makes a final (conscious) percept of an event by consulting the ongoing input from the near future of that event. We thus presume that the present flash–wall interactions also could be interpreted by this postdictive framework. In the flash–wall positional interaction in Experiment 1, the signals of the wall was provided into the subjects' brain just after they saw the flash. Those successive inputs in a brief interval would enable the wall to interact with the flash in the same temporal integration window. Consequently, the visual system could determine the final position of the flash after considering the presence of the wall. If the wall with a high luminance was placed at the Near position, this information would prevent the visual system from localizing the flash beyond the wall, while this would not occur when the wall was at the Low position (note, however, that we claim this postdictive process to occur at an early sensory-perceptual level, not at a later cognitive level, as described in the previous section). The shape change of the flash in Experiment 2 could be interpreted in the same way; the distortions of the flash would result from the integration or binding between the shape (edges) of the wall and the presence of the flash (as was seen in the asynchronous feature binding (Cai & Schlag, 2001), although the binding took place within the feature in the present study). The postdictive characteristics revealed in Experiment 4 thus would provide an important insight why the spatial contexts could interact with the mislocalization process induced by the pursuit. 

Implication for the previous model of eye-movement mislocalizations

As described in the Introduction section, the positional error induced by eye movements has been ascribed basically to the temporal mismatch between the slow afferent signal and the fast EPS. It remains unclear, however, why the afferent processing of the flash is so delayed compared the extra-retinal signal. Although one reason is, of course, the time required for the retinal processing (Schlag & Schlag-Rey, 1995), several studies suggested other possibilities. For example, Brenner et al. (2001) estimated the magnitude of the temporal mismatch at about 100 ms, which was somewhat longer than the retinal delay (about ∼40 ms; Schlag & Schlag-Rey, 2002). With this respect, the present results showed that the visual features (position, shape, and even color) of the flash during pursuit were determined after integrating the wide range of information surrounding the flash, suggesting an involvement of massive and inclusive neural computation in the brain before a stable conscious percept emerges. Especially, the postdictive characteristic in Experiment 4 indicates an existence of the long temporal integration window after the offset of the flash, which might become a direct reason for the slowness of the afferent processing. Thus, in the context of the previous framework for the mislocalization, our data suggest that the neural processing in the brain also play a substantial role in producing the temporal mismatch of the afferent and efferent signals, as well as the delay in the retinal processing. 

Acknowledgments

We thank Dr. Romi Nijhawan for his helpful comments on the manuscript and Dr. Junghyun Park for the fruitful discussion. This work was supported by grants from the Japan Society for the Promotion of Science for Young Scientists to Y.N. and ERATO project, JST. 

Commercial relationships: none. 

Corresponding author: Yasuki Noguchi. 

Email: [e-mail]. 

Address: Division of Biology, California Institute of Technology, 1200 E. California Blvd., M/C 139-74, Pasadena, CA 91125, USA.