Continuous perception of motion and shape across saccadic eye movements

Abstract

Although our naïve experience of visual perception is that it is smooth and coherent, the actual input from the retina involves brief and discrete fixations separated by saccadic eye movements. This raises the question of whether our impression of stable and continuous vision is merely an illusion. To test this, we examined whether motion perception can “bridge” a saccade in a two-frame apparent motion display in which the two frames were separated by a saccade. We found that transformational apparent motion, in which an object is seen to change shape and even move in three dimensions during the motion trajectory, continues across saccades. Moreover, participants preferred an interpretation of motion in spatial, rather than retinal, coordinates. The strength of the motion percept depended on the temporal delay between the two motion frames and was sufficient to give rise to a motion-from-shape aftereffect, even when the motion was defined by a second-order shape cue (“phantom transformational apparent motion”). These findings suggest that motion and shape information are integrated across saccades into a single, coherent percept of a moving object.

Introduction

The fact that we typically make several saccadic eye movements every second means that the position of objects on the retina is constantly changing. Thus, one of the fundamental questions of vision science is how we keep track of the location of objects across saccades (for review, see Bays & Husain, 2007; Melcher & Colby, 2008; Wurtz, 2008). However, a perhaps more basic question is how our naive perception of a smooth and continuous visual flow is built out of a series of relatively brief visual snapshots that are separated by abrupt jumps, like in a poorly filmed home movie. This problem is made even more clear by the fact that the new input to the eyes in each fixation must travel through the visual system before it reaches awareness, necessitating around 120–200 ms (Genetti, Khateb, Heinzer, Michel, & Pegna, 2009; Liu, Agam, Madsen, & Kreiman, 2009; Thorpe, Fize, & Marlot, 1996), and that visual input is partially suppressed while a saccade is performed (Burr, Morrone, & Ross, 1994). The issue of achieving stable perception based on discrete and discontinuous input is particularly troublesome in the case of visual motion. While the brain is extremely efficient in integrating motion cues over time and space over a period of seconds (Burr & Santoro, 2001; Neri, Morrone, & Burr, 1998), motion detectors are typically assumed to operate in retinal coordinates (although see Ong, Hooshvar, Zhang, & Bisley, 2009). Unless motion for the same object is integrated across saccades (Melcher & Morrone, 2003), then this impressive ability to integrate motion over time would be essentially useless. 

There are essentially three main ideas about how visual stability is maintained (for review, see Melcher & Colby, 2008). The first is that our impression of smooth perception is essentially an illusion (Dennett, 1992). Failures to detect changes in the position of an object across a saccade (Bridgeman, Hendry, & Stark, 1975), for example, argue against detailed information being maintained across saccades (Bridgeman, Van der Heijden, & Velichkovsky, 1994). In the case of motion perception, this theory would predict that motion processing begins anew with each fixation, since any matching of object location across the saccade would be based solely on memory (Irwin, 1991). 

A second idea is that our impression of visual stability comes from cross-saccadic priming, in which our post-saccadic perception is influenced by what was previously seen. A clear example comes from studies of reading, in which information about the word to the right of fixation (the “parafoveal preview”) primes us to quickly read the word after the saccade (Rayner, 1998, 2009). Similar results, in which post-saccadic perception is influenced by what was seen before the saccade, have been reported for color perception (Wittenberg, Bremmer, & Wachtler, 2008), time perception (Burr, Tozzi, & Morrone, 2007), motion perception (Melcher & Morrone, 2003), object recognition (Van Eccelpoel, Germeys, De Graef, & Verfaillie, 2008), and face perception (Melcher, 2005; van Boxtel, Alais, & van Ee, 2008). Such cross-saccadic priming might contribute to the subjective impression that the world is stable, since the post-saccadic stimulus would be processed quickly and efficiently (Khayat, Spekreijse, & Roelfsema, 2004a, 2004b). However, this theory still maintains the idea that perception is essentially discrete and tied to individual fixations. 

The third, and most radical, proposal is that conscious perception fuses information from before and after the saccade into a single, coherent percept. This idea agrees with the common, naïve impression of an unbroken stream of visual consciousness—although, of course, our intuitions could simply be wrong. In fact, early attempts to demonstrate the “fusion” of dot patterns across a saccade were without success (Bridgeman & Mayer, 1983; Irwin, Yantis, & Jonides, 1983). Likewise, the finding that changing the case of all letters in a word (McConkie & Zola, 1979) had little effect on reading behavior suggests that abstract codes are used in integrating information across saccades in reading. 

More recently, however, the idea of trans-saccadic perception has been revived based on two types of evidence. The first is the discovery of dynamic receptive fields (RFs), which change their sensitivity around the time of eye movements (Duhamel, Colby, & Goldberg, 1992; Melcher & Colby, 2008; Wurtz, 2008). This “remapping” involves both prediction (before the saccade, the neuron responds to a stimulus in its future RF) and a type of memory trace updating mechanism in which a neuron continues to respond, after the saccade, to the stimulus in its old RF. One important implication of these findings is that the neural activity bridges the saccade, rather than showing discrete and discontinuous firing patterns. The second type of evidence comes from changes in visual perception, such as peri-saccadic mislocalization, which have been reported around the time of saccades (Matin & Pearce, 1965; Ross, Morrone, Goldberg, & Burr, 2001). These findings suggest that the brain anticipates the saccade and uses this information to update spatial information and match it across saccades. However, most experiments have looked at localization of briefly flashed stimuli in laboratory settings. It is less clear how everyday perception, in which objects rarely appear and disappear during saccades, would be influenced by dynamic receptive fields. 

We directly tested the predictions of this third, trans-saccadic perception hypothesis by studying apparent motion. One interesting perceptual property of apparent motion is that our visual system “fills in” the entire motion path, rather than seeing two discrete events (Kolers, 1972; Morgan, 1976). A dot flashed in two different locations, given the right timing parameters, is seen to move through the entire trajectory between point A and point B. This property of apparent motion makes it the perfect test of the hypothesis that perception bridges the saccade. In fact, Rock and Ebenholtz (1962) had already reported a version of trans-saccadic apparent motion many years ago. In their experiment, observers were asked to synchronize left/right eye movements with two alternating flashing lights visible through two vertical slits. In this way, the illuminated vertical lines would be presented at the fovea after each eye movement. The displacement of the vertical line directly followed the size and direction of the saccade. Although the retinal position of the flash was constant, participants reported seeing motion in external (in their terms, “phenomenal”) space. 

In line with this observation, it has been shown that participants are able to detect changes in the position of a moving object across a saccade (Gysen, De Graef, & Verfaillie, 2002). In contrast to the Rock and Ebenholtz studies, recently replicated by Szinte and Cavanagh (2009), studies of change detection for moving objects measured the ability to notice changes in the expected position of the stimulus rather than to perceive smooth trans-saccadic motion. Thus, we adapted the Rock and Ebenholtz technique to study the perception of a coherent motion sequence across saccades. 

In a new set of experiments, we built upon the Rock and Ebenholtz finding in four ways. First, we added motion orthogonal to the direction of the saccade in order to disentangle motion caused by the saccade from motion of the stimulus. This also resulted in the two stimuli being shown in different visual hemifields (and thus to different cerebral hemispheres), providing a greater challenge for mechanisms of trans-saccadic perception. Second, we varied the temporal delay between the two flashes in order to provide a more fine-tuned measure of motion perception. Third, and most importantly, we used “transformational apparent motion” (TAM), in which two differently shaped stimuli are perceived, when shown in an apparent motion display, as smoothly changing shape over time (Tse, Cavanagh, & Nakayama, 1998; Tse, 2006; Tse & Caplovitz, 2006; Tse & Logothetis, 2002). Finally, we varied the amount of shape information in the stimulus to provide an estimate of the reliability of TAM perception judgments and used this stimulus to measure a transformational apparent motion aftereffect. 

The overall aim of these four experiments was to test whether motion can carry shape information—in addition to spatial location—across the saccade. If perception essentially begins anew with each fixation, then there should be little or no impression of smooth motion across saccades in the TAM condition. The ability to integrate the two stimuli into continuous shape-based motion, however, would provide strong evidence that pre- and post-saccadic information are combined into a single, trans-saccadic perceptual event. 

Experiment 1: Percept of a motion event occurring across the saccade

The aim of the first experiment was to measure the smooth and continuous perception of motion across a saccadic eye movement. Following the example of Tse and Logothetis (2002), we presented two stimuli, which differed in location or shape, separated by a blank delay of varying duration (Figure 1). We expected that the perception of smooth apparent motion would decrease for longer blank delays (ISI) between the first and second stimuli. In addition, we investigated the influence of saccades on transformational apparent motion. While Tse and Logothetis (2002) had matched both the retinal and spatial locations of the two stimuli, we investigated whether spatially matching the external location of stimuli across the saccade, despite a change in retinal coordinates, was sufficient to support the perception of object-based motion.

Figure 1

 

(a) Example of the stimuli adopted in Experiment 1. Click on the links to see demonstration movies for T-Bar (Movie 1), Necker cube (Movie 2), single dot (Movie 3), and flipping square (Movie 4) stimuli. During the experiment, the direction of motion was varied randomly on each trial. (b) Trial procedure for Experiment 1 with time course and degree of visual angle of the screen on the x- and y-axes, respectively. Black lines represent the eye fixating the screen, and gray rectangles represent the duration of the displays. Note that on saccade trials the first and second stimuli were always shown in different retinal positions, separated by the saccade. (c) Trial procedure and stimuli adopted in the Necker cube control experiment in order to bias the type of shape transformation (see Methods section). Depending on the inducer, the shape appeared to either move forward or backward in depth.

 

Varying the blank delay between the two stimulus frames was also important in order to allow sufficient time to make a saccade during this blank period between the first and second frames. We cued the subject to make the saccade while the first motion frame was still visible, so that the last 100 ms of the presentation of the first frame was spent preparing for the saccade (trials in which saccade onset was less than 100 ms were excluded from analysis). Then, there was a blank delay, with no stimulus, during which participants moved their eyes to the new fixation position. As a result of the saccade, the two different stimuli were always shown in opposite visual hemifields. The minimum blank delay duration was chosen to be 100 ms for three main reasons. First, this allowed enough time for subjects to make a saccade on the majority of trials, even for the shortest blank delay duration (saccade onset less than 200 ms). Second, this brief ISI gave a strong impression of motion without a saccade. Finally, a blank delay of at least 100 ms was necessary to avoid the suppression of trans-saccadic displacement of the stimulus (Deubel, Bridgeman, & Schneider, 2004). 

Methods

Observers

Six observers participated in the experiment (two authors and four participants naïve to the aims of the experiment). Informed consent was obtained for all participants and all subjects reported normal or corrected-to-normal vision. 

Stimuli and apparatus

Stimuli were presented on a PC using Matlab software and presented on a gamma-corrected Iiyama CRT 1900 monitor running at 85 Hz (resolution: 1280 × 1024, short persistence phosphors). Figure 1a shows the four different types of stimuli used during the experiment; each configuration consisted of 2 frames in which the shape or vertical position of the stimulus was changed. Both shapes were modified from studies by Tse et al. (Tse et al., 1998; Tse, 2006; Tse & Caplovitz, 2006; Tse & Logothetis, 2002). The T-bar stimulus (Movie 1) subtended 8.8 × 4.5 deg/visual angle whereas the short and expanded Necker cube (Movie 2) subtended 2.9 × 3.3 and 2.9 × 4.4 deg/visual angle, respectively. For these stimuli, background was set to white (CIE coordinates: x = 0.28; y = 0.30; luminance: 80 cd/m²), and the stimuli were black (CIE: x = 0.35; y = 0.37; luminance: 0.25 cd/m²). The fixation point consisted of a red (CIE: x = 0.56; y = 0.33; luminance: 70 cd/m²) circle that subtended 0.4 degree of visual angle. 

The other two types of stimuli used in this experiment (Movies 3 and 4) were a single black disk (1.4 deg/visual angle diameter) that could shift its vertical position from the first to the second frame by 4 deg/visual angle toward up or down (randomized across trials) and a flipping square apparent motion sequence whose first frame consisted of a black wireframe rectangle (2 deg/visual angle side) and the second frame consisted of the same square, shifted vertically by 2 deg/visual angle, with one side missing (Figure 1a). The typical percept for this sequence is that of a square that flips in the third dimension until it reached its final position depicted on frame 2, as described by Rock (1997). For this second group of stimuli (disk and flipping square), the screen background was set to gray (CIE coordinates: x = 0.28; y = 0.31; luminance: 8.8 cd/m²). 

For the control experiment, the stimuli were presented on a gray background. The Necker cube subtended 3.1 and 6 deg/visual angle for the contracted and the expanded versions, respectively. Color used were CIE: x = 0.20; y = 0.32; luminance: 58.6 cd/m², CIE: x = 0.27; y = 0.31; luminance: 4.8 cd/m², and CIE: x = 0.26; y = 0.34; luminance: 9.8 cd/m². The fixation point consisted of a black circle that subtended 0.4 degree of visual angle presented either to the left or the right of the stimuli (4.5 deg/visual angle). 

Observers sat in a dimly lit room and viewed the screen binocularly at a distance of 57 cm, with their heads stabilized by a chin rest. Right eye position was monitored using an EyeLink 1000 Desktop Mount (SR Research, Ontario, Canada) sampling at 500 Hz. Eye position was recorded for each trial and saved for offline analysis. 

Procedure

Prior to the experiment, subjects were presented with practice trials showing examples of both smooth apparent motion (ISI between the two stimuli of ∼105 ms) and non-motion (ISI of ∼1200 s). During training, participants were shown the “no saccade” condition, which involved maintaining gaze on the fixation point throughout the trial. In the main experiment, all three viewing conditions (“no saccade,” “saccade,” and “retinal control”) were presented in separate, interleaved blocks in randomized order. The four different stimulus types were divided into two blocks, with two different types of stimuli presented in random order within each block. One type of blocks contained the T-bar and Necker cube stimuli, while the other blocks contained the moving disk and flipping cube apparent motion stimuli. 

Each trial began with the participant looking at the fixation point and then pressing a button when ready. For the first group of stimuli (the T-Bar and the Necker cube), the first stimulus was presented for 400 ms, followed by a variable blank that could vary between 105 ms (9 frames), 210 ms (18 frames), and 400 ms (34 frames) and then the second stimulus display for a further 400 ms. In the “no saccade” condition, participants viewed a fixation point to the left or right of the display and stimuli were shown at the center of the screen. 

In the “saccade” condition (Figure 1b), the fixation point was displaced during the trial to the other side of the screen, requiring a 10-degree saccade. This saccade cue occurred ∼300 ms into the trial (25 frames), when the first stimulus was still visible, leaving participants ∼210 ms (18 frames) to ∼500 ms (43 frames) to move their eyes, depending on the condition. 

Trials in which participants executed the saccade before the first stimulus disappeared (saccadic latency < 105 ms), as well as trials in which saccades were too short (amplitude < 8.5 degrees) or too slow (such that the saccade was not started by the onset of the second motion frame) were excluded from further analysis (mean saccade latency was 184 ms, mean saccade amplitude was 9.8° visual angle). Note that in the saccade condition, the spatial coordinates of the two stimuli on the screen were matched, but they were always shown in different retinal coordinates. In total, 27% of trials on the saccade condition were excluded based on eye movements for these blocks of trials (30% for ISI = 105 ms, 23% for ISI = 210 ms, and 28% for ISI = 400 ms, see Figure 2).

Figure 2

 

Timing of the stimuli with respect to saccade onset for (a)Experiment 1 and (b) Control experiment. Red bars represent the time of stimulus 1 offset (always before saccade onset), whereas blue bars represent stimuli 2 onsets (always after the saccade). The black filled rectangle along the horizontal represents the mean saccade duration.

 

In the other blocks (with the moving disk and flipping square apparent motion stimuli), the procedure was identical except that four different ISIs were tested (105, 210, 600, and 1200 ms) and subjects were requested to perform a 14 deg/visual angle saccade. Again, trials in which participants executed the saccade before the first stimulus disappeared (saccadic latency < 105 ms), as well as trials in which saccades were too short (amplitude < 12.5 degrees) or too slow, such that participants failed to make a saccade by the time of the second motion frame, were excluded from further analysis (mean saccade latency was 182 ms, mean saccade amplitude was 13.4 deg/visual angle). In total, 22% of trials on the saccade condition were excluded based on eye movements (50% for ISI = 105 ms, 26% for ISI = 210 ms, 19% for ISI = 600 ms, and 12% for ISI = 1200 ms, see Figure 2). 

The third condition provided a control for the saccade condition by replicating the same retinal stimulation but without the intervening saccade. The participant maintained fixation at the center of the screen. The first stimulus was shown on one side of the screen (distance of 5 degrees) while the second stimulus was shown on the opposite side of the screen. This condition tested the possible role of large receptive fields (radius of 10 degrees or more), which might integrate the two motion stimuli despite different retinal locations. Such large-range spatial pooling might have been a potential confound for the saccade condition, thus necessitating this control condition. In each of the three conditions, observers were presented with a total of 20 trials for each ISI level for the T-bar and the Necker cube (120 trials overall) and 10 trials for each ISI level for the disk and the flipping square stimuli (80 stimuli overall). 

The motion stimuli were oriented along the vertical axis of the screen in the “saccade” and “no saccade” conditions (Figure 1). After each trial, subjects were requested to report whether they perceived motion in the vertical axis or, instead, perceived the appearance/disappearance of the stimuli. Subjects gave their responses by pressing a button on the keypad. For the T-Bar and the Necker cube trials, subjects could report one of the following three choices: no motion, moving up, or moving down. These choices were included because it has been shown that the T-bar stimuli have a preferential direction of perceived motion that goes from the single point to the bar itself (Tse & Caplovitz, 2006), whereas the direction of the perceived motion for the Necker cube depends on the subjective interpretation of the first display orientation (Tse & Logothetis, 2002). 

For the blocks of trials showing the black disk and flipping square apparent motion sequences, subjects could report one of the following three choices: no motion, translating, or flipping. Thus, it was possible to explicitly test whether subjects perceived a shape change between the two frames in the transformational motion sequence. 

In the case of the Necker cube, the direction of perceived shape change (forward or backward in depth) depended on the initial interpretation of the ambiguous shape. To directly test the perceived shape transformation, we ran an additional control condition (Figure 1c), which biased the perceived orientation of the Necker cube by presenting a solid cube (Tse & Logothetis, 2002). Participants were presented with 2 different conditions (“saccade” and “no saccade”), in interleaved blocks (48 trials for each block, 96 trials for each condition). Each trial began with the participant looking at the fixation point and then pressing a button when ready. The first stimulus of the Necker cube TAM (stimulus 1) sequence was presented for 400 ms (see Figure 1), followed by a solid biasing shape (inducer) remaining on the screen for a variable time (1000 ms, 1200 ms, or 1400 ms), in order to prevent participants from anticipating the saccade. In the “saccade” condition, stimulus 1 was shown again for 6 flips (∼70 ms) together with a 9 deg/visual angle displacement of the fixation point on the other side of the screen. Participants were asked to perform a saccade toward the displaced fixation point. After an ISI of 250 ms, stimulus 2 of the transformational apparent motion sequence was presented for ∼100 ms (9 frames). Thus, participants had approximately 320 ms to shift their gaze to the new fixation position. Subjects were requested to report the direction of the perceived motion (“forward” or “backward” in depth, see Figure 1) or to report that no motion was perceived. The “no saccade” condition was identical except that the fixation point remained on the initial position. To ensure that subjects correctly performed the eye movement and maintained fixation as requested, we implemented a gaze contingent display that checked eye position online. If a saccade was not performed correctly (the saccade did not occur during the 250-ms blank ISI between stimulus 1 and stimulus 2 or did not land inside an area of 3 deg/visual angle around the target fixation point) or if participants did not maintain fixation as requested in “no saccade” trials, then the trial was repeated at the end of the block. During offline analysis, an additional 12% of the trials were excluded due to loss of data acquisition due to eye blinking and other factors. 

Since the procedures for the Necker cube and the T-Bar, the black disk and the flipping square, and the control condition (with the Necker cube) were all run in separate blocks with different parameters, data were analyzed separately for each of these three types of trials. 

Results

In the “no saccade” condition, the proportion of trials in which subjects reported coherent motion decreased as a function of the delay (ISI) duration between the two stimuli (Figure 3). In sharp contrast, participants did not report seeing coherent vertical motion in the control condition (Figure 3, diamonds). The main finding was that performance in the saccade condition (Figure 3, triangles) was similar to that found in no saccade trials (Figure 3, squares) indicating that transformational apparent motion occurred across saccadic eye movements, in non-retinal coordinates.

Figure 3

 

Experiment 1 results, different stimuli are plotted on different graphs ((a) Necker cube, (b) T-bar, (c) single black dot, (d) flipping square); vertical bars represent SEM. Note that the x-scale differs in (a) and (b) from (c) and (d), see text.

 

For the Necker cur and the T-Bar stimuli, we performed a repeated measures analysis of variance (ANOVA) on subject proportion of perceived movement (“up” and “down” responses were pooled together, given the bistability of the Necker cube). We also report post hoc comparisons reaching significance after Bonferroni correction. A 3 (ISI) × 3 (viewing condition) × 2 (stimuli) repeated measures ANOVA showed a main effect of ISI, F(2,10) = 18.366, p < 0.001, η ² = 0.79, a main effect of viewing condition, F(2,10) = 18.386, p < 0.001, η ² = 0.78, and a significant ISI × viewing condition interaction F(4,20) = 8.249, p < 0.001, η ² = 0.63. Stimulus type did not influence the proportion of perceived movement, F(1,5) = 0.435, p > 0.05, ns. Bonferroni-corrected comparisons failed to reveal differences between the no saccade and saccade conditions for ISIs of 105 ms (p > 0.4, ns), 210 ms (p > 0.5, ns), and 400 ms (p > 0.3, ns). 

Similar results were found with the moving disk and flipping square apparent motion stimuli. To look specifically at the change in shape in the flipping square trials, only “flipping” responses were examined (no motion and translation responses were pooled together). Again, there were main effects of ISI and viewing condition (F(3,15) = 33.137, p < 0.001, η ² = 0.86 and F(2,10) = 23.108, p < 0.001, η ² = 0.82, respectively) and a significant interaction between the two, F(6,30) = 11.998, p < 0.001, η ² = 0.70. Likewise, in this case, Bonferroni-corrected comparisons did not reveal any differences between no saccade and saccade conditions at any of the ISI durations. 

The results for the control condition with the “biased” Necker cube followed the same trend (Figure 4). The reported direction of motion was effectively biased by the inducer in both the saccade (t(6) = 3.566, p < 0.001) and no saccade (t(6) = 4.290, p < 0.001) trials. There was no difference between saccade and no saccade conditions (t(6) < 1, ns).

Figure 4

 

Proportion of trials in which the perceived motion was biased by the inducer. The left bars show average results for saccade and no saccade trials. Error bars represent 1 SE. Individual subject results are also shown (small vertical bars to the right of the figure).

 

Discussion

The main finding of the first experiment was that TAM and apparent motion perception continues across saccades. Participants reported a compelling percept of object transformation in both the no saccade and trans-saccadic motion condition. This finding suggests that the smooth perception of motion in non-retinal coordinates reported with location-defined apparent motion (Cavanagh & Szinte, 2009; Rock & Ebenholtz, 1962; Szinte & Cavanagh, 2009) occurs also with a more complex, shape-defined TAM. 

Previous studies have reported that the visual system performs poorly in detecting intrasaccadic displacements of stimuli (Bridgeman et al., 1975). At a first sight, our results might seem to be in conflict with these reports, but there are considerable differences between our paradigm and the classic saccadic suppression of displacement (SSD) paradigm. First of all, in the SSD paradigm the saccadic target is displaced, while in our method the stimuli were presented in the center of the screen, not as the saccade target. Second, in our case, the displacement of the stimulus was considerably larger, around 4 deg/visual angle, than the usual displacement of ∼1 deg/visual angle adopted in the classical paradigm. Third, and perhaps most importantly, we included a blank delay between the vertical displacement so that it did not occur surreptitiously during the saccade (and during saccadic suppression). It has been shown (Deubel, Schneider, & Bridgeman, 1996) that blanking the target considerably lowers the threshold for detection of saccadic displacements of target stimuli. Thus, it is perhaps not surprising that observers in our experiments could easily detect the shift between successive frames of the apparent motion sequence. What was more striking was the finding that subjects perceived coherent, vertical motion at the center of the screen even though the two stimuli were presented in two completely different retinal positions. 

Experiment 2: Comparing retinotopic versus spatiotopic motion

In the first experiment, subjects were able to integrate two stimuli in different retinal positions into a coherent motion perception in spatiotopic coordinates. However, the spatiotopic percept was by far the simplest interpretation of the display. Thus, it is not clear whether the spatiotopic preference would still hold also when both spatiotopic and retinotopic interpretations of an apparent motion display could be possible. To test the preference for spatial or retinal coordinates in trans-saccadic motion, we included two different post-saccadic stimuli, one at the retinal location and one in the spatially matched location on the screen (Movie 5). 

Methods

Observers

Six observers participated in this experiment, two authors and four naïve observers, two of whom participated also on the first experiment. All observers had normal or corrected-to-normal vision. Informed consent was obtained for all participants. 

Stimuli and apparatus

Each stimuli comprised two frames: the first display consisted of a single bar (1 × 5 deg visual angle) with two possible orientations (90° or 45°), while in the second display two bars were presented, one in the same spatiotopic position of the screen and the other one in the same retinotopic coordinates after the saccade (see Figure 5a). The orientation of the bars in the second display depended on the bar orientation in the first display. If the bar in display one was at 90°, the orientations of the bars in the second display were 45° and 135°, while if the bar in display one was at 45°, the orientations were 0° and 90°. Background was set to white (CIE coordinates: x = 0.28; y = 0.30; luminance: 80 cd/m²); stimuli color was set to black (CIE: x = 0.35; y = 0.37; luminance: 0.25 cd/m²). The fixation point was a red (CIE: x = 0.56; y = 0.33; luminance: 70 cd/m²) circle that subtended 0.4 degree of visual angle.

Figure 5

 

Experiment 2 stimuli and results: (a) stimuli adopted and typical percept, click on the link to see demonstration movie:Movie 5. (b) Mean proportion of spatiotopic response for six subjects; vertical bar represents 1SE. Individual subject results are also shown (thin white bars). (c) The distribution of saccade landing positions with respect to the target (presented at 10° of visual angle); dotted line represents mean landing position. Any under- or overshoot of the saccade would have resulted in a spatial mismatch between the two motion frames, both in retinotopic and spatiotopic coordinates.

 

Procedure

Experiment 2 was similar to the previous experiment, but only the “saccade condition” was tested. On each trial, observers were instructed to perform a 10-degree saccade (following the change in fixation point position) between the presentations of the first and second displays, with the direction of the saccade randomized between trials. The ISI between displays was fixed (∼130 ms, 11 frames), so the subject had ∼230 ms (20 frames) to perform the eye movement. After each trial, subjects were requested to report the direction of perceived rotation (rotated to the left/right or toward up/down) by pressing key 1 or key 2 on the keypad. Unlike the first experiment, the participants had to choose (or guess) one direction or the other and could not report “no motion.” The orientation of the first and second bars and the rotation direction for retinotopic and spatiotopic coordinates were counterbalanced across trials. The direction of motion with respect to the direction of the saccade was randomized across trials, in order to take account of the tendency to see motion in the same direction as the horizontal saccade (although any tendency to see motion as congruent with the saccade direction would tend to mask a preference for retinal or spatiotopic motion). The experiment was run in a single block of 120 trials. Trials in which participants failed to make the saccade during the blank delay, as well as trials in which saccades were too short (amplitude < 8.5 degrees), were excluded from further analysis (mean saccade latency 175 ms, mean saccade amplitude 9.8 deg/visual angle). In total, 22% of trials were discarded. 

Results

Participants reported motion consistent with the spatiotopic, rather than the retinotopic, interpretation on a majority of trials (Figure 5b). None of the participants preferred the retinotopic interpretation of motion. There was a clear preference across participants toward the spatiotopic position, which differed from 50% (no preference), t(5) = 8.982, p < 0.001. 

Of course, some saccades fell slightly short of the new fixation target or overshot the target (Figure 5c). Such saccade errors would lead to imperfect alignment of the stimuli, in both retinotopic and spatiotopic coordinates. However, participants clearly preferred the interpretation of motion, which was defined by the spatial matching of the two bars, rather than the retinal position of the bars. 

Experiment 3: Perception of a second-order “phantom TAM”

In order to provide a finer measure of motion perception, we build upon the Tse's idea of using second-order shape stimuli to build TAM configurations (http://www.dartmouth.edu/~petertse/, second-order transformational apparent motion demo). This “phantom TAM” display gives an impression of motion without any motion energy (Movie 6b). One advantage of this type of stimulus is that is makes it possible to modulate the amount of shape information presented on the TAM displays. Previous reports (Tse & Caplovitz, 2006) show that TAM perception relies on mechanisms that parse and match the images across successive fixations giving the impression of motion. By modulating the amount of shape information present in the first frame of the motion configuration, we expected to modulate the resulting TAM, thus providing a finer measure of the motion percept. 

In addition, this stimulus provided an additional control to eliminate any hypothetical role of monitor persistence, since stimuli in this case are defined by a transient polarity inversion of pixels inside a predetermined area (see Methods section) that lasts less than a frame (Bridgeman, 1998). Of course, in our study the measured persistence of the CRT monitor was well below the frame rate, but the use of the new phantom TAM would allow the stimulus to be more widely used, in the future, in LCD and other types of displays. 

Methods

Observers

Five observers participated in the experiment (one author and four participants who were naïve to the aims of the experiment). All subjects reported normal or corrected-to-normal vision. 

Stimuli and design

The basic design of the stimulus was that of a second-order apparent motion display, similar to the T-bar studied in Experiment 1 (but without the T on top), that involved a small region appearing to expand upward or downward into a larger region (Figure 6). In order to create this “phantom TAM” illusion, the region was defined by a change in the polarity of the dots. The rectangular stimulus (oriented vertically) was made up of random dots, which contained two different virtual regions: one at the top of the rectangle and one at the bottom (Figure 6). The grayscale random-dot stimulus, which subtended 6.0 × 3.0 visual degrees, was centered on the screen and filled with dots (192 columns × 96 rows of square dots), randomly generated on each trial (mean luminance 9.4 cd/m²). Each square had random brightness between 0 and 220 (grayscale RGB values) and subtended 0.028 visual degree. This central stimulus was surrounded by a grayscale border subtending 7.2 × 3.8 visual degrees, centered on the screen, and filled with dots (230 × 122 squares). Each square in the background had random brightness between 0 and 150 (grayscale RGB values, CIE: x = 0.27; y = 0.32; luminance: 0.15 cd/m² and CIE: x = 0.28; y = 0.30; luminance: 13.8 cd/m², respectively). The overall impression given by the stimulus was of a gray rectangle oriented vertically, surrounded by a darker gray border.

Figure 6

 

Experiment 3 stimuli and results: (a) The two different arrangements implemented to obtain motion impression toward up or down, click on the link to see demonstration movies for low-density (Movie 6a) and high-density (Movie 6b) conditions. (b) Mean proportions of perceived motion across conditions.

 

In order to create a second-order region, within the rectangle, all dots inside one of two possible areas (at the top or bottom) of the central random-dot rectangle of the random-dot stimuli abruptly inverted the polarity following the rule: 

$\mathrm{b}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}{\mathrm{s}}_{\mathrm{n}\mathrm{e}\mathrm{w}}=(\mathrm{b}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}{\mathrm{s}}_{\mathrm{o}\mathrm{l}\mathrm{d}}-1)*-1.$

(1)

As a result, the area of dots in which polarity was reversed was perceived as a uniform shape that instantaneously appeared among the random-dot stimuli (see Movie 6b). In Figure 6, this polarity-changing region is illustrated by a red rectangle (depicted with 1 in the figure). In order to ensure that the participant was fully able to perceive the region defined by the change in polarity, this polarity change was presented repeatedly three times at 2 Hz. Together, these three polarity changes over a total time period of 1500 ms gave the impression that there was a region, at either the top or bottom of the rectangle, which stood out as different from the rest. After the initial 1500-ms time period, in which three polarity changes occurred for the same virtual region at the top or bottom of the rectangle, the second part of the TAM display was shown. Specifically, both the polarities of the first area (depicted with 1 in the figure) and the second area (depicted with 2 in the figure) were changed, such that a vertical bar comprising areas 1 and 2 was perceived on the screen. 

In order to vary the strength of the second-order shape information in the first part of the phantom TAM sequence, the proportion of the dots in frame 1 (see Figure 6) that changed polarity was varied among 6 different levels: 3%, 5%, 10%, 25%, 50%, and 95%. This allowed for measuring a psychometric curve showing proportions of perceived motion as a function of the shape coherence (proportion of dots changing polarity). Subjects responded by choosing one among the following two choices: no motion or motion. 

Procedure

Participants started each trial by directing their gaze to a fixation point positioned 5 deg/visual angle on the left or right of the screen. After a button press, the trial started and the random-dot stimulus appeared on the center of the screen. Region 1 was repeatedly presented on the screen, one repetition every 43 frames (∼500 ms), then Region 2 was presented for 43 frames (Figure 6). After each trial, subjects were asked to report whether they perceive the second frame rapidly expanding (motion response) or just the flash of the bar on the screen (no motion response), by pressing button 1 or 2 on the keypad. 

Before each block, subjects were informed about whether they would be cued to make a saccade within each trial. In the “fixation” condition, the fixation point maintained its position throughout trial duration, while in the “saccade” condition, the fixation point shifted its position, after ∼150 ms (13 frames), from the third repetition of the polarity change in Region 1 to a position 10 deg/visual angle away on the opposite side of the screen, and then frame 2 was presented. In this way, subjects had about 350 ms (30 frames) to move their eyes. Trials in which participants made a saccade before the fixation cross changed position or executed the saccade after the presentation of the second frame of apparent motion sequence were excluded from further analysis. Likewise, all trials in which saccades were too short (amplitude < 8.5 degrees) were excluded. Overall, the mean saccade latency of the saccades in the good trials was 186 ms and the mean saccade amplitude was 10.7 degrees of visual angle. Overall, 17% of trials were discarded. 

The experimental session consisted of 8 blocks of 36 trials each. The fixation and saccade conditions were run in interleaved blocks. Starting block was randomized across subjects. In addition, there was an initial training period, before running the main experiment, in which participants were familiarized with both the saccade task and the second-order TAM stimulus. To practice making the cued saccades, subjects sat in a dimly illuminated room and were presented with a single fixation point (black circle, 0.32 × 0.32 visual degree) shifted 5 degrees of visual angle to the left or the right of the stimuli (randomized across blocks). In each practice trial, fixation point shifted its position from left to right (or vice versa) for 10 times with a frequency of ∼2 Hz (every 43 flips) and the task was to follow the fixation point. The training consisted of five blocks of 10 trials each. 

In the second-order TAM familiarization procedure, participants were shown a repeated presentation of the motion stimulus. On each trial, subjects maintained their gaze on a fixation point positioned 5 deg/visual angle to the left or right of the screen center. After they pressed a button, the random-dot rectangle stimulus appeared on the center of the screen, followed by the two-frame TAM sequence. The sequence was shown 10 times, at a frequency of 2 Hz, with the direction of motion constant within each trial (but randomized across trials). Subjects were asked to report whether they perceived the configuration as moving and in which direction. One subject who was unable to consistently perceive the TAM direction was excluded from the main experimental session. 

Results

The transformational apparent motion was clearly perceived by subjects. As expected, the perception of vertical motion was strongest when the percentage of dots in the first region (at the top or bottom of the rectangle) was highest (Figure 6b). A repeated measures analysis of variance (ANOVA) revealed a main effect of shape information (proportion of dots changing polarity), F(5,20) = 32.294, p < 0.001, η ² = 0.88. Neither the viewing condition (saccade versus fixation) nor the interaction between shape information and viewing condition yielded significant results (F(1,4) < 1, p > 0.8, F(5,20) < 2, p > 0.2, respectively). Bonferroni-corrected pair-wise comparisons failed to show any significant differences between saccade and fixation conditions. 

Experiment 4: Measuring a second-order TAM aftereffect

Although participants reported seeing motion in the TAM displays, a nagging question in apparent motion studies is whether what subjects perceive is “really motion” (i.e., a visual signal that taps into neural motion detectors) or rather a more abstract interpretation (Anstis, 1980). One generally accepted indicator of “real motion” is the motion aftereffect (MAE), which is thought to be caused by the adaptation of motion-tuned detectors. There is some evidence that apparent motion can evoke an MAE: a prolonged adaptation phase with stroboscopic apparent motion stimuli can bias the perceived direction of a subsequent flickering test (von Grünau, 1986). However, there are no existing reports showing MAE following TAM adaptation. We tested to see whether the second-order phantom TAM was capable of evoking a motion aftereffect. The use of the random-dot stimulus allowed us to build an MAE probe whose structure closely resembled the adapting stimuli, a general rule of thumb adopted to elicit visual aftereffects (Thompson, 1994). Specifically, we used a dynamic probe, considered more sensitive than a static probe, and easier to judge by subjects when measuring MAE (Thompson, 1994). The importance of using a dynamic test probe was also suggested by parallels between second-order motion and TAM stimuli (Tse & Logothetis, 2002), since it is well established that following second-order motion adaptation an MAE can be elicited only when tested with a dynamic test (Nishida, Ashida, & Sato, 1994; Nishida & Sato, 1995). 

Methods

Observers

Four observers participated in the experiment (one author and three participants naïve to the aims of the experiment). All subjects reported normal or corrected-to-normal vision. 

Stimuli

The testing conditions were similar to the previous experiment, except that the stimulus and procedure had to be adapted to create, and then measure, the MAE. As in the previous experiment, the phantom TAM stimulus was made up of random dots (Movie 7). In this case, however, there were four different virtual regions defined within the random-dot rectangle. One complete adaptation cycle consisted of 4 different frames (one frame every 500 ms), each of which was visible by the instantaneous polarity inversion of the squares composing the area (Figure 7a). Four frames (resulting in two bars) were used instead of two to minimize the chances of perceiving Illusory Rebound Motion (IRB), which has been reported with random-dot stimuli (Hsieh, Caplovitz, & Tse, 2005), such as those implemented here. There were two different adaptation directions, toward up or toward down, and the arrangement of the four areas defining the stimuli changed accordingly to the direction (see Figure 7a).

Figure 7

 

Experiment 4 stimuli and results: (a) Depiction of the TAM sequence for upward and downward motions. A demonstration of the stimulus is shown in Movie 7. (b) Psychophysical curves showing the transformational motion aftereffect, in which the percentage of “up” responses is plotted against the coherence of the test stimulus. Negative values correspond to downward motion, while positive values indicate upward movement. Data are shown for four participants. Each data point shows the average from 24 trials. Psychometric curves were obtained using PsychFun package (Kuss et al., 2005) running under the R environment for statistical computing. Error bars show approximate Bayesian confidence intervals for the estimate of the mean.

 

As can be seen in Movie 8, which shows a modified version of the adapter in which only the two tall bars are shown without any TAM, the appearance of the virtual regions, defined by the reversal of polarity, provides a much more compelling percept than any local motion cues given by repeatedly showing random dots. This movie also makes clear the difference between the phantom TAM, which gives a strong motion percept, and the mere appearance of second-order bars. 

The MAE probe (“test”) consisted on a first-order motion stimulus with the same size as the inner rectangle where the phantom TAM had been shown. For high percentages of coherence, motion direction could easily be detected whereas, with low motion signal, the test was perceived as a dynamic visual noise pattern. In each test probe, a given percentage of squares moved toward up or down throughout the test duration (2000 ms) at 0.88 deg/s, with a frequency of ∼8.5 Hz (every 10 frames with 85-Hz refresh rate). Each square that made up the MAE probe stimulus subtended 0.028 visual degree and was initially assigned a random brightness between 0 and 220 (grayscale RGB values, CIE: x = 0.27; y = 0.32; luminance: 0.15 cd/m² and CIE: x = 0.28; y = 0.30; luminance: 48 cd/m², respectively), which was kept constant throughout test duration. The remaining points of the test stimuli were redrawn at each refresh with a random brightness between 0 and 220 (grayscale RGB values). The percentage of coherent motion on each trial was randomly chosen among five different levels (−50%, −25%, 0%, 25%, and 50%, positive and negative numbers represent movement toward up and down, respectively). 

Procedure

Each subjects participated in 3 separate sessions on three non-consecutive days, each of which comprised 8 blocks of 15 trials each. During the first session, the test stimuli were presented without prior adaptation to provide a baseline condition. Each trial started with the presentation of a single fixation point (black circle, 0.32 × 0.32 visual degree) shifted 5 degrees of visual angle to the left or the right of the stimuli (randomized across blocks). Subjects pressed a button to start each trial, were instructed to pay attention to the adapter, and then after the presentation of the test (MAE probe) were cued to report the direction of perceived motion of the test probe by pressing button 1 (up) or 2 (down) on the keypad. In the two remaining experimental sessions, the adapting stimuli were presented prior to the test. The eye tracker was calibrated using a 5-point calibration sequence before every block on each session. 

In the adaptation blocks, a top-up procedure was implemented to measure the strength of the MAE, with the first trial using a long adaptation period (40 s) and then a 6-s adaptation period in the subsequent trials. The adaptation direction in the first session was randomized between subjects. The same adaptation direction was kept constant throughout the session. After the disappearance of the adapter, there was a blank gray screen with the fixation point for 105 ms (9 frames). Then, the test pattern was presented for 2 s. The test probe began with the exact same pattern of dots as had been presented in the final frame of the adapter, in order to avoid any local first-order motion. Then, starting with that pattern of random dots, the first-order motion was added to the dynamic test probe (see Stimuli section, above, for details). After the test presentation, subjects were given a two-alternative forced-choice direction discrimination test for the test probe. Then, the next adaptation sequence started 500 ms after the response was collected. 

Data analysis

The percentage of “up” responses was computed for each combination of test coherence and adaptation condition (baseline, adapt up, and adapt down). Data were analyzed for each subject to derive three psychometric curves for each participant, one for each adaptation condition. Psychometric curves were obtained using an approximate Bayesian inference method (Kuss, Jakel, & Wichmann, 2005) fitting a logistic function to our data. As a lapse rate prior, we used a beta distribution with α = 2 and β = 50. For the location parameter, we chose a Gaussian distribution with mean = 0 and SD = 0.7, whereas as prior for the width parameter, we choose a gamma distribution with α = 5 and β = 10. Acceptance rates for 2000 MCMC generated parameters across subjects ranged from 68% to 81%. 

Results

A strong MAE was found for each of the observers, as shown by the shift of the psychometric curves in the opposite direction from the adapter motion (Figure 7b). When presented with upward TAM adaptation, subjects were biased to respond “down” (mean = 11%), whereas the opposite tendency was found for down adaptation (mean = −17%). The estimated width parameter of the psychophysical curve was constant between adapting conditions (mean = 0.41 and 0.43, adapt up and down, respectively). A slightly higher value was found for the baseline condition (mean = 0.50). In the baseline condition, there was a slight tendency to report the motion moving upward in 3 subjects out of 4 (see Figure 7b). 

General discussion

The main finding of these experiments was that both apparent motion and transformational apparent motion were perceived as a coherent event across saccades. This result provides further evidence that our conscious perception of the world continues across saccade rather than always beginning anew with each fixation (for review, see Melcher, in press; Melcher & Colby, 2008). In the case of the stimuli used in this experiment, the perception of motion bridged the saccade and the participants reported seeing a smooth, continuous path of motion even though the retinal input, by itself, was entirely inconsistent with this interpretation. The finding that TAM, and not just apparent motion, could be perceived across saccades suggests that some three-dimensional shape information is retained across separate glances. 

In the second experiment, spatiotopic and retinotopic motion interpretations were pitted against each other. We found that participants strongly preferred the motion in spatial coordinates. Initially, this might seem like a surprising finding, since it suggests that their natural inclination was to ignore the retinally defined motion signals from low-level detectors that are usually thought to be the foundation of motion perception. In real life, however, it would be important to be able to accurately distinguish between the motion trajectories of separate objects, and link the correct motion to its respective object, even when we move our eyes. This point is illustrated in Figure 8, which illustrates the problem at the heart of trans-saccadic motion perception. As shown in the figure, we do not spontaneously integrate a two-frame transformational apparent motion display in retinal coordinates across a saccade (as we are not able to integrate a 2-frame random-dot kinetogram, see Shiori & Cavanagh, 1989).

Figure 8

 

Illustration of the importance of ignoring irrelevant retinally defined apparent motion in everyday life. If one shifts the point of gaze between the two red fixation points, the percept is of two different objects rather than of one object in transformational apparent motion. In contrast, a clear motion impression arises if the two stimuli are presented spatiotopically across eye movements (Movie 1).

 

While the perception of motion during saccades has received a great deal of study, the mechanisms by which we correctly match moving objects across saccades, while at the same time avoiding to incorrectly perceive irrelevant retinally defined apparent motion, remain less explored. One idea might be that high-level motion, such as second-order or attention-based motion (Cavanagh, Holcombe, & Chou, 2008), involves transforming retinal coordinates into head-centered, object-centered, or world-centered coordinates via gain fields (d'Avossa et al., 2007). A second idea might be that trans-saccadic apparent motion derives from the spatial updating of an object “pointer” (Melcher & Colby, 2008), which is fed back into the motion computation. There is an interesting parallel between trans-saccadic motion perception and TAM perception. In the case of TAM, the percept is defined by higher order shape information rather than low-level motion detectors. Similarly, trans-saccadic motion perception would seem to require updating of object location in spatial maps, likely those beyond V1 (Melcher & Colby, 2008). 

Our results suggest that the spatial maps that are involved in matching the object in the TAM display are spatially updated across saccades, perhaps via the dynamic remapping of receptive fields. It is important to note that many of the stimuli we tested did not involve a change in spatial location, but only in the stimulus shape. Thus, while updating the spatial locus of attention across the saccade is undoubtedly an important aspect of visual stability (Cavanagh, Hunt, Afraz, & Rolfs, 2010), the remapping of “attentional pointers” cannot, by itself, account for the present results. Given that TAM cannot be accounted for a shift in attentional gradients (Tse & Cavanagh, 1995), the present results are more in line with predictions of object pointers, which include links to visual features such as shape (Melcher, 2009; Melcher & Colby, 2008), than attentional pointers (Cavanagh et al., 2010). 

The final two experiments allowed us to directly measure the role of shape information in supporting trans-saccadic motion perception. The second-order phantom TAM showed that polarity-change-defined regions were sufficient to support trans-saccadic motion perception. In addition, this type of stimulus allowed us to measure an MAE, which provides further evidence that participants were seeing “real” motion. Different types of motion aftereffects has been described throughout the years (Mather, Pavan, Campana, & Casco, 2008), suggesting that more than one neuronal population contribute to the arising of MAE. This new MAE may have interesting implications for the current debate about motion aftereffects, in particular regarding the relationship between TAM and second-order motion stimuli. 

Perception of TAM requires a tight interplay between shape information and the extraction of motion signal in order to fill in the features along the path of perceived motion (Tse & Logothetis, 2002). Motion detection and shape matching have to run in parallel, computing the motion trajectory according to the outcome of shape processing (Tse & Caplovitz, 2006). A functional neuroimaging study of TAM suggested that a number of brain regions were involved, including hMT and LOC (Tse, 2006). Interestingly, these areas have been implicated in the use of extra-retinal coordinates in the processing of motion (d'Avossa et al., 2007) and shape (McKyton & Zohary, 2007). A critical question for future research is how visual areas that process information in retinal coordinates can support the non-retinotopic perception of shape and motion shown here. 

In conclusion, our results suggest that our naïve impression of a “stream of consciousness” in visual perception is not an illusion. Our findings suggest that visual stability involves the integration of information about motion and shape across saccades into a single, coherent percept. The units of time underlying event perception, the “moments” of awareness, can bridge saccades and are not strictly tied to individual fixations. Our results provide further evidence that object and spatial information are matched across saccades, perhaps based on specific objects rather than remapping the entire scene (Melcher & Colby, 2008; Prime, Tsotsos, Keith, & Crawford, 2007). In addition, our findings suggest that trans-saccadic TAM, and in particular the second-order TAM, could be a useful tool to measure the object-based matching of features, such as shape, across saccades. 

Supplementary Materials

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Acknowledgments

This work has been realized also thanks to the support from the Provincia Autonoma di Trento and the Fondazione Cassa di Risparmio di Trento e Rovereto. D.M. was supported by COFIN 2007. 

Commercial relationships: none. 

Corresponding author: Alessio Fracasso. 

Email: [e-mail]. 

Address: Palazzo Fedrigotti-corso Bettini 31, 38068 Rovereto (TN), Italy.