Dynamic, object-based remapping of visual features in trans-saccadic perception

Abstract

Saccadic eye movements can dramatically change the location in which an object is projected onto the retina. One mechanism that might potentially underlie the perception of stable objects, despite the occurrence of saccades, is the “remapping” of receptive fields around the time of saccadic eye movements. Here we examined two possible models of trans-saccadic remapping of visual features: (1) spatiotopic coordinates that remain constant across saccades or (2) an object-based remapping in retinal coordinates. We used form adaptation to test “object” and “space” based predictions for an adapter that changed spatial and/or retinal location due to eye movements, object motion or manual displacement using a computer mouse. The predictability and speed of the object motion was also manipulated. The main finding was that maximum transfer of the form aftereffect in retinal coordinates occurred when there was a saccade and when the object motion was attended and predictable. A small transfer was also found when observers moved the object across the screen using a computer mouse. The overall pattern of results is consistent with the theory of object-based remapping for salient stimuli. Thus, the active updating of the location and features of attended objects may play a role in perceptual stability.

Introduction

Humans make frequent saccadic eye movements in order to bring items of interest to the fovea, where visual acuity is best. Each saccadic eye movement creates a spatial and temporal disruption in the flow of visual information. A shift in gaze displaces the retinal image so that a new set of feature detectors become active in processing that object. This trans-saccadic displacement of the image on the retina suggests that the object is, in a sense, re-perceived all over again after each gaze shift. A similar problem occurs when the object itself moves on the retina, shifting its image out of the receptive fields of the neurons that first processed its visual features. This challenge to visual cognition is illustrated, in a humorous way, by cartoons in which an object moves so fast that its features (such as color or pattern) get left behind. In everyday life, however, mis-binding of features and objects in perception is rare unless the object is moving extremely fast or under conditions of crowding or uncertainty (Arnold, 2005; Blaser, Pylyshyn, & Holcombe, 2000; Holcombe & Cavanagh, 2001). 

A fundamental question in vision is how the perception of an object can continue across the spatial and temporal interruptions caused by blinks, gaze and body movements and object movement. It is critical to keep track of the location of objects in order to guide actions (Heide et al., 2001; Land & Hayhoe, 2001; Medendorp, Golz, Vilis, & Crawford, 2003; Vaziri, Diedrichsen, & Shadmehr, 2006). In addition to object location, there is also evidence that information about object properties is combined across glances (Melcher, 2005, 2007; Melcher & Morrone, 2003). One powerful demonstration of dynamic updating in the visual system comes from a recent neuroimaging study in which observers made a horizontal saccadic eye movement from one side of a stimulus to the other (Merriam, Genovese, & Colby, 2007). Although the visual stimulus was extinguished prior to saccadic onset, the activation pattern was “remapped” to visual processing areas in the other cortical hemisphere in anticipation of the saccade. This finding of cross-hemispheric remapping in humans confirms previous studies showing remapping in humans in the parietal cortex (Merriam & Colby, 2005) and dynamic shifts in receptive fields in the frontal eye fields (Sommer & Wurtz, 2006; Umeno & Goldberg, 1997), posterior parietal cortex (Duhamel, Colby, & Goldberg, 1992; Kusunoki & Goldberg, 2003) and visual cortex (Nakamura & Colby, 2002; Tolias et al., 2001) in non-human primates. 

A current challenge, then, is to develop a model of trans-saccadic perception that can account for both the physiological evidence for remapping and the pattern of psychophysical results in which visual perception continues over time and across saccades. The experiments in this paper were designed to test the psychophysical predictions of two possible neurophysiological mechanisms that might be involved in trans-saccadic perception of objects: (1) spatiotopic receptive fields and (2) transient remapping of object locations in saliency maps. 

Spatiotopic receptive fields and trans-saccadic perception

In the first stages of visual processing, the response of neurons is determined by the location of the stimulus in retinal coordinates. A critical question is how soon, and to what extent, visual processing is able to “free itself” from retinal coordinates. Other coordinate systems, independent of the eye, are used in the brain to encode the location in space of objects with respect to the body, parts of the body or even external space (Colby & Goldberg, 1999; Duhamel, Bremmer, BenHamed, & Graf, 1997; Galletti, Battaglini, & Fattori, 1995; Graziano, Yap, & Gross, 1994; Snyder, 2000; Snyder, Grieve, Brotchie, & Andersen, 1998). Evidence for spatiotopic receptive fields in visual processing has been found in a recent neuroimaging study of human MT+ (d'Avossa et al., 2007). In addition, the influence of gaze position throughout visual cortex (Galletti & Battaglini, 1989; Shenoy, Bradley, & Andersen, 1999; Trotter & Celebrini, 1999) has been taken as evidence that there may be a strategy by which there is a gradual transformation from retinotopic to gaze-independent receptive fields (Pouget, Fisher, & Sejnowski, 1993; Zipser & Andersen, 1988). 

Spatiotopic receptive fields that combine information over time would seem to be an ideal mechanism to support trans-saccadic integration, as well as to guide behavior. In the case of temporal integration of a weak motion signal, for example, a network of spatiotopic neurons in motion processing areas would be uninfluenced by intervening saccades that might occur during the temporal integration process. Thus, a motion processing network that includes both retinal (V1) and more eye-independent (MT/MST) receptive fields could, in theory, show spatiotopic motion integration across saccades (Melcher & Morrone, 2007). 

The advantages in using a space-based reference frame to combine predictive and invariant information across glances is clear. But there are also reasons to question whether “spatiotopic” perception requires that neurons themselves encode sensory information in a coordinate system based on external space. First, there are numerous studies suggesting that the brain uses a network of interconnected maps, each coded in sensory or effector coordinates, rather than a single map (for review, see Snyder, 2000). Thus an action, such as grasping, could be programmed based on hand-centered coordinates, even while the visual areas guiding that action continue to process the world in retinal coordinates. A second challenge for the spatiotopy hypothesis is the fact that objects can also move, changing their coordinates in external space. A number of studies have shown that visual processing can integrate information along the pathway of a moving object (Nishida, 2004; Nishida, Watanabe, Kuriki, & Tokimoto, 2007; Shimozaki, Eckstein, & Thomas, 1999; Watanabe & Nishida, 2007). Thus, at least in some cases, visual perception appears to follow the moving object rather than stable external space. When the object itself moves in the world, receptive fields linked to head-centered or world-centered coordinates might be of little use in maintaining perceptual stability for that object, yet people seem able to actively track objects with fixed head and gaze using only shifts of attention (Blaser et al., 2001). 

Finally, there are suggestions that perception around the time of saccades is based on a reference object, not on Cartesian space (Deubel, Bridgeman, & Schneider, 1998; Lappe, Awater, & Krekelberg, 2000). Peri-saccadic localization shows compression of the location of a flashed stimulus, with the object typically localized along the path of the saccade and/or toward the saccadic target. One striking example comes from a recent study in which three colored bars that were flashed immediately before saccadic onset were viewed as all superimposed on the same saccade target (Lappe, Kuhlman, Oerke, & Kaiser, 2006). This finding, along with other studies of peri-saccadic mislocalization, suggest that a reference object, rather than external coordinate space, is used to keep track of objects across saccades. 

Saliency map theories of trans-saccadic perception

A second hypothesis for visual stability across saccades is that the networks of maps that encode object location are updated across saccadic eye movement (for review, see Colby & Goldberg, 1999). This “remapping” and re-alignment of the maps to account for saccades involves dynamic changes in the receptive fields of retinotopic neurons. Numerous studies have found predictive remapping, in which a neuron responds before the saccade to a stimulus that will be in the receptive field after the eye movement occurs (Duhamel et al., 1992; Kusunoki & Goldberg, 2003). Perception also appears to anticipate the consequences of saccades (Hafed & Krauzlis, 2006; Melcher, 2007; Wexler, 2005; Wexler, Panerai, Lamouret, & Droulez, 2001). At the same time, the receptive fields of other neurons shift or contract (Kubischik, 2002; Tolias et al., 2001). Other neurons may respond to both the future and current receptive field. Finally, there are neurons that respond even after the saccade to stimuli that had been present in the pre-saccadic receptive field (for review, see Colby & Goldberg, 1999). That complex pattern might reflect that activity of a network of neurons working together to remap the location of objects across saccades. 

The dynamic nature of receptive fields is particularly evident in the “saliency maps” in posterior parietal cortex (PPC) (Colby & Goldberg, 1999; Gottlieb, 2007). In addition to a role in eye movements, attention and remapping, the lateral intraparietal sulcus (LIP) of the PPC is connected to other maps that encode object location in different coordinate systems (Colby & Goldberg, 1999), and is directed connected to visual areas such as V4, TE and TEO (Baizer, Ungerleider, & Desimone, 1991; Webster, Bachevalier, & Ungerleider, 1994) and to brain regions involved in spatial memory including the parahippocampal gyrus (Suzuki & Amaral, 1994). Thus, LIP is thought to play a role in linking space, perception and action (Gottlieb, 2007). Evidence for object-based encoding in the PPC also comes from studies of object-based attention (Serences, Schwarzbach, Courtney, Golay, & Yantis, 2004) and from object-based neglect in parietal patients (Driver & Halligan, 1991; Hillis & Caramazza, 1995). Although the saliency maps appear to respond to the overall task relevance of objects, these maps are not thought to be “object-centered”, since the objects must be located in a larger coordinate frame (such as eye, hand or external space). Object-based encoding has been shown in the supplementary eye fields (Olson, 2003; Olson & Gettner, 1995) which are strongly connected to LIP. 

Remapping has typically been considered in terms of an updating of spatial coordinates. However, this study investigates the hypothesis that the saliency maps involved in remapping might encode the location of “perceptual objects”, not simply “space”. In other words, the spatial maps in the posterior parietal cortex might play a role in object perception, not just spatial localization. The possible role of PPC maps in object processing has been suggested previously in the context of feature-binding (Serences & Yantis, 2006; Treisman, 1996; Walther & Koch, 2006). Given the importance of binding across temporal and spatial disruptions, and given that similar brain regions have been implicated in both feature-binding and remapping, then we can hypothesize that dynamic remapping of these saliency maps might play a role in trans-saccadic perception. There is evidence that maps in PPC show flexible, task-specific sensitivity to “ventral stream” properties such as shape, orientation and color (Durand et al., 2007; Freedman & Assad, 2006; Mitchell & Cusack, 2007; Sabes, Breznen, & Andersen, 2002; Sereno & Maunsell, 1998; Sereno, Trinath, Augath, & Logothetis, 2002; Shikata, Tanaka, Nakamura, Taira, & Sakata, 1996; Toth & Assad, 2002). Thus, saliency maps might be regarded as a convergence point (or, perhaps, pointer) at which information across multiple sensory modalities is integrated into a single coherent entity. If so, then it would be much simpler to update the spatial location of a one or two salient objects—and thus update the pointer to the object's properties—than to completely remap the entire perceptual space. 

In on-line perception, an object is inextricably linked to a particular spatial location: a perceived object must be perceived as somewhere. Objects must follow the rule of spatiotemporal continuity (Spelke, 1990), which is a principle that could be critical both for object constancy and in guiding eye movements. Given the importance of the object as a basic cognitive category that organizes our perception of the world (Feldman, 2003; Kant, 1781; Mareschal, 2000; Piaget, 1977; Spelke, 1990), it is critical to bind our perceptions into objects. At present, however, the role of objects in trans-saccadic perception remains unclear. 

Purpose of the present study

This study investigates whether it is “space” or rather “objects” that are remapped across saccades. We used the tilt adaptation aftereffect (TAE) to measure changes in orientation processing (Blakemore & Campbell, 1969; Gibson & Radner, 1937) in order to provide a perceptual correlate of remapping. The TAE is spatially limited, so that it normally occurs only when the adapter and test stimuli are shown in the same retinal location. One exception to this rule is when an object remains stationary and the eye moves to a new fixation position (Melcher, 2005, 2007). On such trials, the retinal location of the adapter and test differ, but the spatial location is matched, resulting in a “spatiotopic” aftereffect. In the present study, however, the object and space were dissociated by moving the stimulus in space after the adaptation period and before the test stimulus was presented. Moving the object itself should, in theory, also lead to an updating of the object location in internal saliency maps, raising the question of whether the TAE stays at the old location in space or is updated to the new object location. 

The aim of the first experiment was to determine whether trans-saccadic remapping occurs for moving stimuli and, if so, whether this remapping depended mainly on stimulus characteristics or on the interpretation of the stimulus in terms of a moving object. We found that an object that translated across the screen during the saccade served as a cue to remap the aftereffect to the new location of that object. The second experiment investigated the roles of expectation and selective attention. Transfer of the aftereffect to a new retinal position was influenced by whether or not the translating object was attended as well as by the subject's expectations about the impending location of the test stimulus. The third experiment study tested the limits in remapping for objects that move quickly and unpredictably. It was found that an object that moved faster than the speed that supports smooth pursuit (without catch-up saccades) led to the elimination of the retinal transfer of the TAE. 

Experiment 1

Methods

Subjects

Seven adults with normal vision participated in the experiment. All observers gave informed written consent. 

Apparatus

The experiment was run on a PC, using the Psychophysics toolbox (Brainard, 1997; Pelli, 1997) and MATLAB (Mathworks, Inc.). The experimental stimuli were displayed on a Sony F520 monitor at 100 Hz refresh rate. The display was viewed from a distance of 60 cm. Responses in each trial were entered via a USB keyboard. 

The fixation position of the right eye was monitored using the Viewpoint eye tracker (Arrington Research Ltd.). Eye position was calibrated before each session and checked in between each block of trials. During the experimental blocks, eye position was recorded for offline analysis. On blocks of trials requiring maintained fixation on the central point, trials were excluded when gaze deviated by more than 1° from fixation. On trials with saccades cued by a moving fixation point, trials were excluded in which the participant failed to make a saccade to the correct position at the right time. Overall, less than 2% of the trials were excluded from data analysis. 

Stimuli

The stimulus was a gray disk (5% higher luminance than the background gray) that contained a superimposed Gabor patch (Gaussian windowed grating with spatial frequency of 9 cycles/degree), which was either invisible (zero contrast against the gray disk) or progressively made visible (60% of full contrast) during the adaptation and test period. The disk subtended 2° of visual angle on the display when viewed from 60 cm. 

The adapting stimulus was tilted by 20° to the left (counterclockwise) or right (clockwise) of vertical. Half of the trials contained an adapting stimulus tilted to the right, while the other trials used a leftward tilted adapter. The test grating was oriented vertically with one of seven orientations: −4°, −2°, −1°, 0°, 1°, 2°, or 4°. In each experimental condition, each test orientation was presented 10 times. Both adapter and test were shown superimposed on the gray disk. 

Procedure

In the main experimental condition, observers began each trial by fixating a dot that was displaced 4° above or below the center of the screen ( Figure 1). After a brief delay (100 ms), the light gray disk appeared at a position from 4° to 6° (randomized) to the left or right of fixation. After a delay of 200 ms, the adapter appeared in the center of the disk, ramped up in contrast over a period of 100 ms. After 3 s, the contrast of the grating was ramped back down over a period of 100 ms, so that it was no longer visible against the gray disk. At this point, the disk moved 10° to the other side of the screen over a period of 780 ms. During the time in which the disk was moving across the screen (200 ms after it had starting moving), the fixation point was displaced vertically by 8° to the other side of the vertical center of the screen. This jump indicated to the observer to make a vertical saccade to the new fixation position. 

Figure 1

 

Illustration of the procedure used in the main condition of Experiment 1. Starting from the top left, the main order of events is shown. The initial fixation point was displaced vertically by 4°, above or below the center of the display screen. A light gray disk was shown displaced horizontally from center (4° to 6°, randomly varied across trials). After an adaptation period of 3 s, the disk moved horizontally by 10° to a new position on the opposite side of the screen. During this movement by the disk, the fixation point was moved to a new position, triggering a vertical saccadic eye movement. After the eye and the disk were both stationary at their final locations, a test stimulus was shown at the location of the disk.

After the disk had come to a halt for 100 ms, the test stimulus was presented for 50 ms at either the final location of the moving disk (50% of the trials) or at the original adaptation location. Then the disk returned to gray and remained on the screen for 100 ms before disappearing. After the test stimulus disappeared, a written prompt on the screen was presented to ask participants to press a key to indicate whether the stimulus had been tilted to the left or to the right. Thus, in the main experimental condition, both the retinal and spatial location of the stimulus moved from one side of fixation to the other during the trial. The position of the peripheral stimulus (right or left of fixation), the distance from fixation (4° or 6°) and the location of the initial fixation position (above or below the midline) were randomized across trials. 

In addition to the main experimental condition, there were two other types of trials, run in separate blocks. In the baseline condition, the disk disappeared during the post-adaptation delay and the test stimulus was shown (against the gray disk) at either the original adapter position (50% of trials) or at a position 10° away. This condition replicated the main experimental condition but with a blank interval between adaptation and test, rather than a moving disk. Observers maintained fixation on a central fixation point and there was no saccade during the trial. On 50% of trials the test stimulus was shown in the same location as the adapter, while on the other half of the trials it was displayed on the other side of the fixation, separated from the adapter by 10°. These trials allowed for the calculation of the maximum and minimum TAE values. The minimum was critical to make sure that the 10° separation between adapter and test was sufficient to eliminate the TAE. The maximum TAE served as a baseline for characterizing performance in the other conditions. All of the conditions were run together in each session to avoid order effects. This was made possible by interleaving the blocks from the different conditions in the same session, rather than running all of the blocks of one condition separately. The order of the blocks and conditions was counterbalanced across subjects. 

These control conditions also investigated whether the interpretation of the moving disk as a single “object” was important for trans-saccadic perception. The exact definition of an “object” is somewhat controversial, but the spatiotemporal continuity of a single entity is widely considered to be essential to object status (Spelke, 1990). This rule was violated in two ways, either by having a single disk “jump” across the screen or by having a second disk appear and move across the screen while the original disk remained stationary. In addition to breaking the rules of object-ness, these two conditions are interesting controls because the moving (or jumping) disk would still attract exogenous attention. Thus, they serve as further controls to confirm that attracting attention is not, by itself, sufficient to cause remapping of the TAE (Melcher, 2007). 

In the “object-ness” control condition, run in separate blocks of trials, one disk (the “object”) remained at the same location after the adaptation period without moving, while a second disk moved to the other side of the screen. Participants were instructed that the object in these blocks of trials remained in the same location, and that the moving disk was irrelevant. Unlike in the other conditions, the test stimulus was not presented in each location on an equal number of trials. Instead, the test stimulus was presented at the location of the adapter on 88.5% of the trials, and at the final position of the moving disk (on the other side of fixation) on the other trials. In all other ways, including the timing of the saccade and test stimulus, this condition was identical to the main experimental condition. 

Prior to running the experiment, all participants were trained on the orientation discrimination task. This involved a practice session of 20 trials in which the test stimulus was presented for 50 ms at a position 4° to 6° to the left or right of fixation. In the main experiment, trials were run in blocks of 70 trials. Each block was self-paced, with the trial initiated by pressing a key. 

Data analysis

For each condition; the number of trials responding “left” for each test orientation was converted to a proportion and then a psychometric curve was fit to the data. The data for −20° and 20° tilt adapters was plot separately, creating two psychophysical curves. Data was fit with a sigmoid Boltzmann function (Non Linear Least Squares Fitter, Origin 8 software, OriginLab USA). All functions fit the data significantly ( p < 0.05). The magnitude of tilt aftereffect (TAE) was calculated as the distance between the 50% point of the curve for rightward and leftward tilted adapters. 

Performance on the baseline condition was used to estimate a “maximum tilt aftereffect” for each participant. This value varied across participants, ranging from 2.85° to 4.25°. Based upon this maximum value, the “proportion of full TAE” was calculated for each observer and each condition (Melcher, 2005, 2007). For example, a TAE of 2° in a particular condition, given a maximum TAE of 4° for that observer, would be recorded as a proportion of 0.5 out of a possible 1.0. The proportion of full TAE was used for all between-subjects statistical analyses in Experiment 1. 

In the object-ness control, the tilt adaptation aftereffect at the ignored location was calculated by comparing the proportion of “left” responses for the two adapters for a stimulus shown with a particular tilt (Shepherd, 2001), in this case 0°. Any difference for the two adapter orientations was quantified and used to determine whether there was a significant TAE. The tilt adaptation at the attended object location in the object-ness control was calculated both as above (to determine the proportion of full TAE), but also by comparing performance at 0° tilt for the two adapter orientations, allowing comparison with the ignored location. 

Results

On the baseline trials without the moving disk a strong TAE was found at the adapter position, as expected (mean = 3.18°). No aftereffect was found at the location on the opposite side of the screen (mean proportion of full TAE = 0.02, standard error = 0.05), confirming the limited spatial extent of the classic tilt aftereffect (Gibson & Radner, 1937). 

In the main experimental condition, in which the disk containing the adapting stimulus moved across the screen during the delay period, a portion of the tilt aftereffect was transferred to the new spatial location across the saccade ( Figure 2). Thus, remapping of the TAE does not appear to be limited to matched spatial location as would have been predicted by the spatiotopic coordinate theory. 

Figure 2

 

Proportion of trials in which the test stimulus was perceived as tilted counterclockwise (“left”) as a function of the orientation. Separate curves indicate performance with −20° and 20° adapters (see Methods). Data is shown for the six subjects in the trans-saccadic moving disk condition.

On average, the object-based TAE was 50.6% of the full adaptation aftereffect ( Figure 3), similar to the value found for a stationary object across saccadic eye movements (Melcher, 2005, 2007). Thus, there is no suggestion that the additional factor of the disk moving during the saccade changed the remapping mechanism or had a large impact on the magnitude of remapping. At the original location of the adapter, a small TAE was also found (6.3%), although it was significantly smaller than the TAE at the final resting location of the moving disk (t = 5.056, p = 0.002). This pattern of results suggests that the transfer of the TAE across the screen was paired with a concomitant reduction of the TAE at the original adapter position. 

Figure 3

 

Proportion of full tilt aftereffect for trials in which the test stimulus was shown in either the adapter position or in the final resting location of the moving disk. Adapter and test position were separated by 10° of visual angle on the display. Error bars show standard error.

Critically, the transfer of the TAE across the screen depended on the observer's interpretation of the moving disk. In the control condition in which the observers were instructed to ignore the moving disk—and to instead consider the object as stationary—the TAE at the final location of the moving disk was not significantly different from zero ( t = 0.085, p = 0.935). When the moving disk was ignored in trials in which the object was perceived as stationary, the entire TAE was maintained at the original adapter location (97.7%) as would be expected given the retinotopic nature of the TAE. On the contrary, when the observer interpreted the disk as a moving object, there was a greater TAE at the final position of the object than at the original adapter position ( Figure 3) ( F(1,6) = 10.17, p = 0.019). 

Discussion

Consistent with the object-based remapping hypothesis, the TAE was transferred to the final location of the moving target. This suggests that object identity is important for remapping, more than the spatial location of features in external coordinates. This transfer of the TAE also appeared to be specific to the adapter object and not simply a result of a shift in attention. The appearance of a moving distractor disk, which might have attracted attention, was not sufficient to remap the TAE to the other side of the screen to the distractor location. 

A possible criticism of the first experiment is that expectations about where the stimulus would be presented might have led subjects to bias their attention to one location or another in the moving disk trials. One can discount the extreme hypothesis that attention was the only factor in the pattern of results. Dividing attention evenly between two equally likely target locations in the baseline condition was not sufficient to transfer the TAE to a new location, or divide it between the two locations. Instead, the TAE stayed at the adapter location when there was no object motion. This confirms the finding that an attentional cue is not sufficient to cause remapping of the TAE (Melcher, 2007). In practice, if remapping followed every shift of attention—even between different objects—then it would cease to be a useful mechanism for perceptual stability. If attention played a role, it was most likely in the object-ness condition, in which observers ignored the moving distractor disk. However, even in this condition subjects were aware that the test stimulus could be presented at the final location of the moving distractor. Thus, some attention was allocated to that location. 

While selective attention may not be sufficient to cause remapping without any saccade, it might nonetheless play an important role by limiting what is salient. The results of the first experiment support the hypothesis that remapping of visual features involves salient objects, as opposed to the entire visuo-spatial field. Thus, object-based attention to a moving object might determine whether or not the visual feature is bound to that object across saccades. The potential roles of object-based attention (object tracking) and expectation (test location) were investigated more fully in a second experiment. 

Experiment 2

Methods

Subjects

The author and three naïve subjects participated in the experiment. All participants had normal vision or wore contact lenses. Informed consent was obtained for all observers. 

Apparatus

The experiment was run using MATLAB (Mathworks, Inc.) and the Psychophysics toolbox (Brainard, 1997; Pelli, 1997) on a PC computer. The experimental stimuli were displayed on a Mitsubishi Diamond Pro 2070 monitor at 75 Hz refresh rate. The display was viewed from a distance of 80 cm. 

The fixation position of the right eye was monitored using the Eyelink 1000 video-based tracker. Eye position was calibrated at the beginning of each session and fixation correction was run before each block of trials. On blocks of trials requiring maintained fixation on the central point, trials were excluded when gaze deviated by more than 1° from fixation during the trial. In blocks in which saccades were cued by a moving fixation point, trials were excluded in which the participant failed to make a saccade to the correct position at the right time. Overall, less than 1% of the trials were excluded from data analysis based on off-line analysis after data collected had finished. 

In addition to using a USB keyboard to give the response, in the second condition of this experiment participants used an optical USB mouse to move a stimulus across the screen. 

Stimuli

The stimuli were identical to the first experiment. 

Procedure

There were two main conditions in this experiment. The first measured the transfer of the TAE when the object moved across the screen (as in Experiment 1) while in the second condition the disk was moved manually by the observer using a computer mouse. In addition, the attention to the moving disk was manipulated along with the probability that the test stimulus would be shown at the final position of the moving stimulus. The order of conditions was changed across each participant. In the first condition, there were two disks on the screen, only one of which contained the adapting stimulus ( Figure 4). The disks were displaced vertically from the midline by 3°. In each block of trials, only one of the disks, either the adapter or the distractor, moved across the screen. The general order of events was similar to that of the main experimental condition in the first experiment. The main difference was that the moving disk briefly changed contrast for four frames during the final 1/3 of its movement (the exact timing was randomized across trials). Observers were required to report whether this 5% change in contrast was an increment or decrement at the end of each trial. Observers were told, before each block of trials, which type of disk would move and which would be stationary, but were instructed always to report the contrast change in the moving disk. 

Figure 4

 

Order of events in a saccade trial in which the adapting stimulus moves across the screen. Participants reported the orientation of the test stimulus and the increment or decrement in the contrast of the moving disk. The test was shown in either the final location of the moving adapter disk or at the location of the distractor. In separate blocks of trials, the distractor disk, rather than the adapter, moved across the screen, with the test presented at either the final location of the moving distractor or at the original distractor disk.

In separate blocks (fixation trials), the fixation point remained at the center of the screen throughout all of the trials. This condition tested whether the features of attended objects might be remapped, along the path of the object motion, even without a saccade. This hypothesis was motivated by the reasoning that the updating of the spatial location of salient objects in the world would be of advantage even when the eye is stationary. Otherwise, a saccade or grasping movement to an object that had changed location while the eye was stationary would be inaccurate. If the updating of objects in saliency maps is an important part of the “remapping network”, then this would predict that remapping of the TAE occurs for a moving disk even when the eye is stationary. 

In the second condition, observers used an optical computer mouse to move the disk across the screen. At the beginning of each trial, a red dot was displayed at a location displaced horizontally (4° to 6° to the left or right) and vertically (2° to 4° up or down) from fixation. Observers moved an arrow (cursor) toward that position and then, while maintaining fixation on the central point, clicked on the dot to begin the trial. The arrow (cursor) was replaced by a gray disk, on which the adapter was ramped up to 60% contrast over a period of 100 ms. After the 3 s adaptation period, the adapter faded and red dot was displayed on the other side of the screen, 10° away in the same vertical position. Observers had been trained (40 trials) to move the disk to the red dot without moving the eyes and within the 800 ms interval between adapter and test. After 800 ms, the test was shown on the gray disk that was being controlled by the mouse. The initial and final positions of the mouse were recorded to ensure that the observer had moved the disk correctly. Trials in which the observer failed to move the disk across the vertical midline were eliminated from analysis (an average of only 2 trials per observer). In one block of trials, the test stimulus was always shown at the final position of the moving mouse, while in the other two blocks it was shown 50% of trials at the final position and 50% of trials at the adapter position. Observers were told, before each block, which were the possible test locations. 

Results

In the first condition, the degree of transfer of the TAE to the final position of the moving disk depended on whether the adapter or the distractor had moved across the screen. When the adapter moved, a cross-saccadic TAE was found at the final position of the disk ( Figure 5), replicating the results of Experiment 1. 

Figure 5

 

Tilt aftereffect perceived at the final position of the moving adapter for each observer in Experiment 2. Each curve shows the proportion of “left” responses as a function of the test orientation. The distance between the two curves indicates an adaptation aftereffect due to the different adapters, which were tilted clockwise or counterclockwise by 20°.

In the moving adapter trials ( Figure 6, top panel), the average TAE was larger at the endpoint of the moving disk than at the distractor disk location ( t = 2.232, p < 0.05). The TAE at the location of the moving disk was significantly greater than zero ( t = 4.303, p < 0.01), while the TAE at the distractor location was not ( t = 1.717, N.S.). On trials in which observers maintained fixation ( Figure 6, leftmost bars), there was a small but significant TAE perceived at the final location of the moving adapter stimulus ( t = 3.496, p < 0.02). This finding of a “remapped” TAE even without a saccade is consistent with previous reports—many of which did not measure eye movements—that at least some features are bound to moving objects (Blaser et al., 2000; Nishida et al., 2007; Shimozaki et al., 1999; Watanabe & Nishida, 2007). 

Figure 6

 

Proportion of full TAE for tests shown at the adapter and distractor position as a function of whether or not a saccade occurred during the trial. The top panel shows performance when the adapter disk moved across the screen during the delay between the adaptation period and the test. The bottom panel shows TAE when the distractor disk moved across the display. Data from four subjects, with error bars showing standard error of the mean.

When the distractor moved, however, the full TAE stayed at the adapter position on fixation trials ( Figure 6, bottom panel). This replicated the finding of the object-ness control in Experiment 1, which suggested that paying attention to the moving stimulus was not sufficient to explain the spatial transfer of the TAE. When observers made a saccade ( Figure 6, rightmost bars), there was a significant trans-saccadic TAE at the original adapter position as expected ( t = 4.721, p < .005). Unlike in Experiment 1, there was a small TAE perceived also at the final position of the moving distractor ( t = 3.388, p < 0.05). This TAE was smaller than at the original adapter location ( t = 4.01, p < 0.01). One possible explanation is that the saccade resulted in spatial uncertainty about the original position of the adapter, leading this remembered location (where there was no longer any visible stimulus) to be incorrectly mapped onto the distractor location (where there was a visible reference). Alternatively, the saccade might have led to an expansion in the effective receptive field, perhaps if trans-saccadic perception requires relatively large receptive fields. In any case, although the TAE at the distractor location is statistically significant, it is a much smaller effect than the object-based TAE found across saccades in the other conditions. 

Performance was above 90% for each observer in correctly judging the increment or decrement of the contrast of the moving disk, across each condition. In addition, the fact that the test stimulus was shown at the final location of the moving disk on 50% of trials always made the final position of the disk a salient, task-relevant location. Unlike the object-ness control in Experiment 1, participants were always paying attention to the moving disk, even when the distractor rather than the adapter was moving. Yet the overall pattern of results was similar: the TAE was displaced across the screen only when the adapter disk moved. 

In the second condition, observers moved the disk across the screen using a computer mouse. The magnitude of the perceived TAE at the final location of the moving disk depended on the predictability of the test location ( Figure 7). The TAE was largest in blocks of trials in which the test stimulus was always shown at the final location of the moving disk ( Figure 7, top row). The transfer of the TAE was strongly diminished, however, when the test was not predictable ( Figure 7, middle row). 

Figure 7

 

Tilt aftereffect on trials in which the observer moved the disk across the screen using a computer mouse. Data is shown for one representative subject in the three main conditions of predictability and test location. The top row shows performance on trials in which the test was always shown at the final location of the moving disk. The bottom two rows show tilt judgments when the location of the test was not predictable on each trial.

Thus, the predictability of the test location was an important factor in the transfer of the TAE. When the location of the test stimulus was not predictable, the TAE was found mainly at the original adapter location ( Figure 8), rather than at the final position of the moving disk ( t = 5.161, p < 0.02). The lack of predictability resulted in a significant decrease in the TAE at the final ending position of the moving mouse ( t = 3.621, p < 0.05). However, the small TAE at the found at the final location of the moving disk was greater than zero for each observer ( t = 2.983, p = 0.058). 

Figure 8

 

Average tilt aftereffect for four subjects as a function of test location and predictability. Blocks of trials in which the test was always shown at the final location of the moving disk are shown by the leftmost bars. The central and rightmost bars show performance when the location of the test was unpredictable for any given trial. In these blocks of trials, the TAE was much larger at the original adapter location (rightmost bar).

Discussion

As in the first experiment, it was found that the TAE was transferred across saccades to the final location of the moving disk across saccades. However, a smaller transfer of the TAE was also found without saccades, as long as the test stimulus was reliably presented at the final location of the disk. Overall, both saccade condition and predictability of the test location were found to be important for object-based perception of the TAE. 

It is interesting to note that the TAE was also perceived, in some trials, at the final position of the moving distractor. One possible explanation is that the saccade resulted in spatial uncertainty about the original position of the adapter, leading this remembered location (where there was no longer any visible stimulus) to be incorrectly mapped onto the distractor location (where there was a visible reference). Alternatively, the saccade might have led to an expansion in the effective receptive field. Thus, it might be interest to test whether trans-saccadic perception requires relatively large receptive fields. In any case, although the TAE at the distractor location was statistically significant, it was a much smaller effect than the object-based TAE found across saccades in the other conditions. 

Experiment 3

In the first two experiments, the major source of uncertainty was the location of the test stimulus, which might fall in either the right or left visual field. While this manipulation was useful for measuring the role of selective attention and expectations on the transfer of the TAE, such uncertainty would be unlikely in the real world. Objects do not typically lose their features across saccades and features do not move about without objects. Thus, the visual system should expect that the same object, particularly if it is a real three-dimensional object in the world, will keep its properties even when it is invisible due to a blank delay (saccade, blink) or by occlusion. This ability to perceive stable objects over time is the foundation of object constancy. 

A much more likely source of uncertainty in binding features to real-world objects would come from the ability to predict the path of a fast, erratically moving object. The movement of objects on the retina can often be predicted when it is caused by self-movement such as eye, head, or full-body movements. This raises the question of whether remapping might be limited to relatively predictable movements. To test the limits of remapping, the speed and predictability of the path of the moving disk was varied in a new experiment. 

Methods

Subjects

Three participants from the first experiment also took part in the third experiment. All observers gave informed written consent. 

Materials

The materials were identical to the first experiment. 

Procedure

All observers were trained with the task prior to running the experiment. There were two experimental conditions. In the first condition, the procedure was identical to the first experiment except that the speed and trajectory of the moving disk was varied across trials (described below). In the second condition (“no saccade”), run in separate blocks of trials, observers were instructed to maintain gaze on a fixation point that remained stationary throughout the entire trial while the disk moved smoothly across the screen. This condition tested whether or not the TAE would be transferred with the moving object even without an intervening saccadic eye movement. 

The speed of the moving disk was varied across trials, so that it moved horizontally across the screen in 800 ms (12.5 deg/s), 400 ms (25 deg/s), or 200 ms (50 deg/s). The maximum speed tested (50 deg/s) was well above the rate at which humans can perfectly track an unpredictable object using smooth pursuit eye movements (Pola & Wyatt, 1991). At such speeds, the eye tends to lag behind the object and there is frequent use of “catch-up” saccades to keep the moving object near the fovea. 

To ensure adequate time to make the saccade and still see the test stimulus clearly, the delay after the disk came to rest before the test was presented was varied: 100 ms (12.5 deg/s trials), 200 ms (25 deg/s trials), or 300 ms (50 deg/s trials). Thus, there was always at least 400 ms between the cue to make the vertical saccade and the display of the test stimulus. Thus the attentional cue to the test location was similar across all conditions. 

In addition to the three different speeds, the trajectory of the object was varied in the 400 ms and 200 ms trials. Specifically, a vertical component was added to the speed of the disk. The magnitude of the vertical component varied across trials and between the first and second part of the movement itself, such that the disk might move upward first and then downward or not at all during the second half of its movement. On each trial, at the beginning of the disk movement, the direction and magnitude of movement for the first half of the movement was randomly chosen. There were five possible vertical speeds: 0°/s, 6.2°/s, −6.2°/s, 12.5°/s, or −12.5°/s. The vertical speed, however, was not completely constant: on each frame, there was a random chance (80%) that the disk would move vertically and a 20% chance that the disk would stay in the same vertical position. The direction and magnitude of vertical movement of the disk was again chosen randomly for the second half of the movement sequence. The total vertical displacement of the disk varied across trials. At minimum, with different directions for the first and second parts of the movement sequence, the final disk location was near the vertical center of the screen. On 400 ms trials, with maximum and consistent vertical displacement, the disk location at the end of the movement was displaced by about ±4° from the vertical center of the screen. 

The slow trials (12.5 deg/s) were run in separate blocks from the other two speeds and were identical to the experimental condition of the first experiment. The trials with faster speeds were run together. The test stimulus was always shown at the final location of the moving disk in the blocks with the fast moving disk. Trials, in which gaze deviated from the fixation point by more than 3°, were excluded in order to ensure that the adapter and test stimuli were not shown in similar retinal locations (5.2% of trials). 

Results

The speed and predictability of the moving disk influenced the magnitude of the transfer of the TAE across saccades ( Figure 9, open circles). An increase in both the speed and unpredictability of the disk movement led to a reduction in the TAE found at the final location of the disk ( F(1,2) = 167.18, p < 0.001). A similar trend was found on trials in which the observers were not cued to make any vertical saccade ( Figure 9, filled squares). Overall, the magnitude of the TAE that was transferred to the new target location was lower for the no-saccade trials ( F(1,2) = 52.39, p < 0.02). 

Figure 9

 

Proportion of full TAE as a function of the speed of the moving disk. Circles show performance on blocks of trials with vertical saccades, while squares show proportion of TAE when participants were instructed not to make a saccade. While the motion path was predictable for the slowest speed (leftmost symbols), the path was not predictable for the two faster speeds.

The spatiotopic TAE was also influenced by whether or not the observer was cued to make a vertical saccade. On blocks with the slow, predictable motion path, some trials contained test stimuli presented at the original adapter position, rather than at the final location of the moving disk. In the fixation (no-saccade) trials, there was a larger TAE at the matched spatial location than at the endpoint of the moving disk, while the pattern was reversed in the saccade trials ( Figure 10). The interaction between saccade instruction (saccade or no-saccade) and test location (spatial or object-based) was significant ( F(1,2) = 97.00, p = 0.01). 

Figure 10

 

The interaction between saccade condition and test location. Average proportion of full TAE and standard error are shown for saccade trials (circles) and fixation trials (squares) for both the final object position 10° from the original adapter position (rightmost symbols) and the original adapter position (leftmost symbols).

Discussion

The results show that object-based remapping of the TAE depends strongly on the speed and predictability of the moving stimulus. The largest TAE was found at the final location of a slowly and predictably moving disk on saccade trials. At the extreme values of speed (50 deg/s) and unpredictability, no TAE was found at the final resting place of the disk. Although this suggests a strict limit on trans-saccadic perception with dynamic scenes, these limits may be viewed as reasonable since the oculomotor system has difficulty maintaining smooth pursuit of a target object which moves at high speed (Pola & Wyatt, 1991). One could argue that if an object is moving fast and unpredictably and, in addition, the eye is not trying to track that object, then such an object would not be a good candidate for trans-saccadic perception. 

It is not possible, based on the experimental design, to distinguish whether the increased speed or the predictability was more important. In comparing the 25 deg/s and 50 deg/s trials, which both contained unpredictable paths, it is clear that the greater speed led to a complete elimination of the TAE. What is clear is that the TAE was not transferred to the final location of the moving disk in the fast velocity condition, in which the path was unpredictable, despite the fact that the location of the test stimulus was itself predictable based on the fact that the disk was stationary at that final location for 300 ms prior to the test presentation. 

General discussion

The main finding of these experiments is that the trans-saccadic TAE followed the moving object. The first experiment demonstrated the basic finding that TAE was transferred to the final location of the moving target. In the second experiment, it was found that the transfer of the TAE was largest for attended objects across saccades. Even on trials without a saccade, a smaller TAE was found at the final location of the moving object, but only when the location of the test stimulus was predictable. This pattern of results suggests that the three key events—adaptation, object movement and the presentation of the test—must be linked to the same perceptual object. The third experiment demonstrated that the transfer of the TAE was only found for relatively slow and predictable motion paths for the object. At higher speeds with unpredictable motion, the TAE was not transferred to the final position of the object, despite the fact that the final position of the object was perceived equally well in all conditions. 

Overall, these results provide some initial support for the theory that trans-saccadic perception involves the spatial updating of an object “pointer” that is linked to visual feature information. Our findings are consistent with the reference object theory of trans-saccadic localization (Deubel et al., 1998), which states the identity of salient objects is matched across saccades in order to maintain perceptual stability. The current results extend this theory, however, to include the visual features of salient objects, not just the spatial location. These findings suggest that the occurrence of a saccadic eye movement creates the need to link the attended pre-saccadic object with a post-saccadic object. Across all three experiments, the maximum transfer of the TAE occurred when observers made a saccade and the adapter was linked to an attended object. This finding demonstrates the importance of the saccade itself in evoking a matching mechanism that “searches” for possible correspondences. In practice, it seems likely that this updating mechanism would be limited to a few salient objects, since there is no need to update the location of unattended items. 

Attention played an important role in the transfer of the adaptation aftereffect, particularly in actively tracking the moving object. No adaptation aftereffect was transferred to the final position of an ignored object, and little or no TAE was found at the ending point of a moving distracter object. Our interpretation of these findings is that object-based attention is a necessary condition for the trans-saccadic perception of moving stimuli. Some further evidence for the role of attention in selecting objects for remapping comes from a recent study that manipulated spatial attention and the number of distractor objects (Melcher, 2008). Thus, the role of selective attention appears to be in determining which objects are salient, and then only salient objects are remapped. Thus, in a complex natural scene with dozens of objects this implies that the visual features of most objects would not be remapped across saccades. 

In Experiments 2 and 3, a small TAE was transferred to a new position even without a saccade. This “non-saccadic remapping” was found when the observer manually moved the object with a computer mouse and, on some trials, when the path of the object was consistent and predictable. It is critical to note, however, that this non-saccadic transfer of the TAE only occurred for slow, predictable motion in which the test stimulus was always presented at the final position of the moving object. In contrast, the trans-saccadic TAE was found even when the test position was not predictable. This pattern of results suggests the object-based remapping can occur in different two conditions: (1) due to a saccade, in which case large shifts in retinal position are fast but predictable, or (2) due to object or observer motion in which the change in retinal position of the object is relatively slow and predictable. Both situations may involve the same mechanism of updating object pointers in spatial maps. 

What are the implications of these experiments, conducted with gray disks on a flat screen, for perception of real world objects? The first implication is that attention and expectation are likely to determine which objects are salient and, thus, which objects are “candidates” for trans-saccadic perception. Second, these results demonstrate that the perception of moving objects continues even across saccades. Thus, our findings predict that it should be possible to find evidence for the accrual of information about a moving object across saccades. Third, we found that objects that move too fast and unpredictably appear to “lose” their visual features, either because the spatial location of the object is not updatable or because the visual features themselves are no longer perceived clearly. At the speeds tested here, which exceeded those at which objects can be smoothly tracked, perception of visual features (such as the spatial frequency grating) is compromised. As stated in the Introduction, there is a naïve idea—illustrated by cartoons in which objects moved so fast that the feature details such as spots or stripes get left behind—that rapidly moving objects can lose their features. The results of Experiment 3 can be viewed as support for this idea: a moving object can in fact lose its stripes. 

Conclusions

Perceptual continuity across saccades is a tremendous feat—and a fundamental mystery—of visual perception. The current results provide further evidence for one mechanism that might be involved in maintaining perceptual stability: remapping of object location and features across saccades. In addition, the current results provide new evidence that dynamic objects which change spatial position can also show trans-saccadic remapping. Such resilience to changes in the location of objects might be useful for displacements of the objects on the retina due to self-movement and also for keeping track of moving objects across saccades. 

The present findings may help to shed some light on the apparent contradiction between the strict limits of visual memory and the richness of perceptual experience. The remapping mechanism explored here depends upon prediction, rather than storage (Melcher, 2007). Thus, remapping would be less computationally expensive than an internal, memory-based “Cartesian theater” (Dennett & Kinsbourne, 1992) that integrates visual information. Remapping also has the advantage of not requiring a constant comparison of current input with a stored memory of the environment—a task at which we seem to be particularly poor (O'Regan & Noë, 2001). Given that the world itself is its own best model (Hayhoe & Ballard, 2005; O'Regan & Noë, 2001), a simple and elegant solution to feature binding across saccades may be to allow visual processing of the external world to carry on for the same object despite changes in its retinal position. 

Acknowledgments

The author thanks two anonymous reviewers for their helpful comments and suggestions. 

Commercial relationships: none. 

Corresponding author: Dr. David Melcher. 

Email: [e-mail]. 

Address: Center for Mind/Brain Studies and Department of Cognitive Sciences, University of Trento, Corso Bettini 31, Rovereto 38068, Italy.