Coding of identity-diagnostic information in transsaccadic object perception

Abstract

In four experiments, we examined the hypothesis that a presaccadic extrafoveal preview of an object normally affects subsequent postsaccadic foveal processing of the object. On each trial, viewers inspected an array of three objects and were instructed to remember one object characteristic (in-depth orientation, image-plane orientation, color, or semantic category). During the saccade to one of the objects, an intrasaccadic change in the in-depth orientation or the color could occur and its effect on gaze duration on the object was analyzed. When participants were instructed to remember the objects' depth orientation, gaze durations increased after an intrasaccadic depth rotation but not after a color change, demonstrating task dependence. Color information was only integrated when it was task relevant (i.e., when it had to be remembered). When the task required access to stored object models (categorizing the object or deciding whether it was upright or inverted), an intrasaccadic depth rotation again prolonged gaze durations, even though there was no explicit instruction to remember the objects' depth orientation. This suggests that orientation-dependent object models are accessed during object perception across saccades and that transsaccadic integration serves to expedite object identification through the integration of presaccadic and postsaccadic object-diagnostic information.

Introduction

Visual sensitivity in humans is highly heterogeneous across the visual field, with acuity dropping off sharply as objects are projected further into the retinal periphery (Anstis, 1974). Therefore, depending on the task, human observers make a saccadic eye movement on average two to four times each second, in order to bring new information into the high-acuity fovea centralis of the retina (e.g., Hallett, 1986; Henderson & Hollingworth, 1998; Rayner, 1998; Tatler, Gilchrist, & Land, 2005). With each saccade, the retinal projection of the world drastically changes, which stands in sharp contrast with the visual phenomenology of a spatially stable and temporally seamless world. The question arises how different glimpses of an object, separated both in space and time (i.e., the presaccadic and the postsaccadic view), are integrated. One view, which is often implicitly assumed, is that higher level processing of visual information is confined to the foveal object and that processing of peripheral information only plays a role for selecting the next saccade target (on the basis of low-level processing of local image features like luminance gradients or contour density). Under this view, there is no need for high-level transsaccadic information integration. Instead, high-level perception proceeds strictly serially and starts anew with every new fixation. Under an alternative hypothesis, the visual system already starts pre-processing the object that is the target of the upcoming saccade at a relatively high level of processing, even before the object is foveated. One of the raisons d'être of this processing scheme could be that transsaccadic object perception expedites object identification. While a single fixation may in principle be sufficient to identify a foveal object, the preferred modus operandi in the exploration of real-world environments could be to increase speed and reliability of object identification by transsaccadic integration of foveal and/or extrafoveal evidence from multiple fixations (De Graef & Verfaillie, 2002; Germeys, De Graef, Van Eccelpoel, & Verfaillie, 2007; Verfaillie, De Graef, Germeys, Gysen, & Van Eccelpoel, 2001). 

Evidence for the latter hypothesis is provided by the so-called preview benefit effect. In the basic paradigm, originally introduced by Pollatsek, Rayner, and Collins (1984) and later developed further by Henderson and colleagues (Henderson, 1994; Henderson & Anes, 1994; Henderson, Pollatsek, & Rayner, 1987; Pollatsek, Rayner, & Henderson, 1990; also see Germeys, De Graef, & Verfaillie, 2002), participants are presented with a line drawing of one (or two) object(s) presented in peripheral vision and are instructed to make a saccade to (one of) the object(s) and then name it as quickly as possible (or remember its name, like in Germeys et al., 2002). On some trials, the to-be-named object is replaced by another object during the subject's saccadic eye movement. The basic finding is that object identification is facilitated (as reflected in shorter naming latencies or shorter gaze durations) when the object can be previewed before it is foveated (i.e., when the object is not changed during the observer's saccade), in comparison to the condition in which presaccadic and postsaccadic objects have different identities. 

In sum, experiments documenting preview benefits provide evidence for the hypothesis that high-level visual processing is not confined to the foveal object, but that the human visual system already starts processing peripheral objects (and the saccade target is the most prominent candidate), possibly to expedite object identification. However, in most preview benefit experiments reported until now, participants were explicitly instructed to identify the saccade target object, in order either to name it (e.g., Henderson et al., 1987) or to remember its name (Germeys et al., 2002). Maybe it is only under these conditions that preview benefits show up. The general purpose of the present series of experiments is to examine the benefit of a peripheral preview (or the cost in the absence of a useful preview) when participants are engaged in another task than object naming (also see Demeyer, De Graef, Wagemans, & Verfaillie, 2008). More specifically, we investigate whether transsaccadic integration of object information depends on the task relevance of the previewed information. 

To investigate the role of task relevance in transsaccadic object perception, but now in tasks that do not necessarily require subjects to name the saccade target, we developed a new paradigm. On each trial, subjects were instructed to fixate a sequence of three objects and remember one particular aspect of the objects (e.g., their depth orientation). At the end of the trial, memory for one of the three objects was probed. During the saccade to the second object, a change occurred in a visual attribute of the saccade target object (e.g., the second object was rotated in depth) on a proportion of the trials. The possible impact of the intrasaccadic change ¹ was examined by comparing gaze duration on that object on trials in which an intrasaccadic change occurred with gaze duration on trials without change. The underlying rationale is the hypothesis that, if an object feature is not processed presaccadically, gaze duration should not be affected by an intrasaccadic change in that particular feature. Conversely, if a particular object feature is already processed before the object is actually fixated and that feature is changed during the saccade to the object, gaze duration should increase. To examine the task-dependent nature of transsaccadic integration, we independently manipulated the to-be-remembered object feature and the feature that could be changed intrasaccadically. The changed feature could be task relevant (e.g., rotating the object in depth in a task in which participants have to remember the object's depth orientation) or task irrelevant (e.g., changing the color of the object in a task in which participants have to remember the object's depth orientation). 

Three aspects of this paradigm deserve particular attention. First, if an object feature was changed intrasaccadically, the change was always in the second object of the series of three objects that had to be fixated (we will refer to this object as the critical object). The rationale for this methodology was the following. The paradigm used in most previous studies reporting preview benefits differs in a number of potentially important ways from the object-to-object scanning behavior that is typically observed in everyday vision (e.g., De Graef, Christiaens, & d'Ydewalle, 1990; Henderson & Hollingworth, 1999). First, in the classical preview paradigm (e.g., Henderson et al., 1987; but not in Germeys et al., 2002), subjects are fixating an empty location (or a small fixation cross) before saccading to the target object. During real-life scene exploration, however, empty spaces are rarely fixated and observers tend to foveate informative regions (Antes, 1974; Mackworth & Morandi, 1967), implying that saccades are generally initiated from a foveal object. The resulting foveal load during the preview of a peripheral object may very well affect the nature and size of the preview benefit (Henderson & Ferreira, 1990). In our paradigm, while subjects were previewing the critical object, they were fixating the first object (and the task required encoding of that object, at least of the to-be-remembered feature), effectively creating a foveal load. Second, in the typical preview paradigm, the sudden appearance of the target object(s) in the periphery functions as a signal to launch the eyes. However, in real-world scenes, objects very rarely pop up in peripheral vision. This may create a very different interplay of attentional and oculomotor dynamics that could affect possible preview benefits. In our paradigm, when subjects were fixating the first object, the critical object was already available in the periphery and attention was not drawn exogenously to the object. Third, in earlier preview paradigms, once participants were fixating the target object, they did not have to pre-process a subsequent saccade target object. Again, this stands in contrast to more ecologically valid situations. In our paradigm, while participants were fixating the critical object, they already had the opportunity to preview the third object (which was task relevant, because the third object was one of the three objects that could be probed at the end of the trial). In sum, by having subjects fixate a series of three objects and by examining the effect of an intrasaccadic change on the gaze duration on the second object, we tried to reach a better approximation of transsaccadic object perception as achieved during real-world scene exploration. 

Second, because we wanted to investigate the effect of having a useful preview or not in a task that did not require subjects to explicitly identify the object, we could not use naming latency as the dependent variable. Instead, gaze duration on the critical object (the sum of all consecutive fixations of the first visit to the object) was measured in a task in which a specific object feature had to be encoded and remembered. Gaze duration is generally assumed to reflect ongoing cognitive processing, with gaze duration increasing as the difficulty of processing an object increases (e.g., De Graef et al., 1990; Rayner, 1998; Sanders & van Duren, 1998). This is an implicit measure of the effect of intrasaccadic stimulus changes, in contrast to paradigms in which participants are instructed to detect intrasaccadic changes explicitly. There is a large body of research on transsaccadic information integration that is based on explicit detection of intrasaccadic changes (e.g., Gysen, De Graef, & Verfaillie, 2002; Henderson & Hollingworth, 2003a, 2003b; McConkie & Currie, 1996; Rensink, 2002). However, the use of explicit measures has been criticized for underestimating the amount of information that can be retained across saccades (e.g., Hayhoe, Bensinger, & Ballard, 1998; Hayhoe, Karn, Magnuson, & Mruczek, 2001) or other stimulus discontinuities (e.g., Fernandez-Duque & Thornton, 2003). 

Third, we were particularly interested in the effect of an intrasaccadic change in one specific object characteristic, namely its orientation in depth. This has to do with our general working hypothesis that transsaccadic object perception expedites object identification. Under the assumption that object identification involves the activation of viewpoint-dependent object models (e.g., Bülthoff & Edelman, 1992; Palmeri & Gauthier, 2004; Tarr, 1995; Tarr & Pinker, 1989; Verfaillie, 1993), it can be predicted that an intrasaccadic depth rotation should lengthen the duration of the gaze on the object, at least in tasks in which subjects access stored object models and indeed already attempt to do that before they actually fixate the object. Studies in which participants explicitly detected different types of intrasaccadic changes (Verfaillie, 1997; Verfaillie & De Graef, 2000; Verfaillie, De Troy, & Van Rensbergen, 1994) already documented that, in principle, an object's depth orientation can be maintained accurately across saccades (in contrast to the object's image-plane position, which probably does not play a role in accessing viewpoint-dependent, but presumably location-independent stored object models). One of the primary questions investigated in the present study is whether information concerning an object's depth orientation is integrated automatically across saccades (i.e., even when subjects are not required to identify objects or when the task does not involve the detection of intrasaccadic depth rotations). 

Experiment 1

On each trial, participants fixated three objects that were displayed equidistantly on an imaginary circle on the screen. They were instructed to remember a particular feature of each object after having foveated it. During the saccade from the first to the second object, a characteristic of the second object was changed intrasaccadically in three quarters of the trials. Participants were not informed about this manipulation. At the end of a trial, subjects had to make a judgement about the to-be-remembered feature of one of the three objects previously seen (by manually pressing one of two response buttons). Across experiments, the to-be-remembered object feature and the feature that could be changed intrasaccadically were manipulated independently. In Experiment 1, participants had to remember the depth orientation of the objects. During the saccade to the critical object, the saccade target object could change in one of three ways: Either it was rotated in depth, or its color changed, or both the depth orientation and the color changed. In a fourth, baseline, condition, there was no intrasaccadic change. 

Predictions were given as follows. First, in the conditions in which the to-be-remembered object characteristic and the characteristic that could be changed intrasaccadically were the same (in this experiment, the object's depth orientation), we expected that an intrasaccadic change would slow postsaccadic processing (as reflected in increased gaze duration), in comparison to the condition in which the presaccadic and postsaccadic views of the object were identical. Second, if transsaccadic integration is strictly restricted to the information that is relevant to the task, then gaze durations on the critical object after a task-irrelevant intrasaccadic color change should not be longer than in the no-change baseline condition and gaze durations after a combined (task-relevant) orientation and (task-irrelevant) color change should equal gaze durations in the condition in which only the orientation was changed. 

Methods

Participants

Eight students (3 male, 5 female, with the age ranging between 18 and 24 years) from the University of Leuven participated in the experiment. They were naive regarding the hypotheses under investigation. All reported either normal vision or vision corrected to normal with tightly fitting, soft lenses. To participate in the experiment, it was required that the standard deviation of calibration measurements in subjects with corrected vision did not deviate from that for subjects with normal vision. This was the case for all participants, including those in Experiments 2, 3, and 4. Each participant was paid 15 Euro for taking part in two sessions. 

Apparatus

The stimuli were displayed in 60-Hz non-interlaced mode on a Sony GDM-W900 Trinitron Color Graphic Display with a 756 × 486 resolution, which subtended 18° × 11.5°. Eye movements were recorded with a Generation 5.5 dual-Purkinje-image eye tracker (Crane, 1994; Crane & Steele, 1985), with a spatial accuracy of 1 min of arc and a 1000-Hz sampling rate. The tracker was interfaced with a PC that stored every sample of the left eye's position and, for each sample, made an on-line decision about the eye state: fixation, saccade, blink, or signal loss. Eye state and position were fed into a second PC, in control of stimulus presentation. The device took about 4 ms to detect a saccade. Display changes were accomplished within 16.7 ms because they could be initiated at any moment during the screen refresh cycle (the display-change command did not wait for the vertical blank). Thus, a complete saccade-contingent display change was typically achieved in 21 ms. 

Stimuli

Twenty objects were rendered with 3D Studio Max (Autodesk, 1997), a 3D-modeling and animation application program. The package allows the user to build a 3D model of an object and apply diverse materials (defined by colors, transparency, texture, reflection, etc.) to selected parts of the model. The 3D model can then be placed in a 3D environment, and different lights and spots can be added to the scene. By rotating the object in depth, images of the object viewed from different vantage points are made. Some of the objects were taken from public databases on the Internet, others were taken from the object database constructed by Verfaillie and Boutsen (1995). Half of the objects were vehicles, the other half were animals (see 1 for a complete list). For each object, two views were created (see Figure 1 for an example). The initial in-depth orientation from which the two other views were derived was chosen in such a way that objects were facing the viewer (the 0° view). From that orientation, the objects were rotated around the vertical axis over an angle of 20° to the right and over an angle of 70° to the left. All objects were equally tinted in one of four different colors (yellow, orange, green, or blue) while preserving inner detail. In a single trial, all objects had different colors and they were presented on a grayish background. The objects were scaled in such a way that they each fitted in a window of 2.4 by 2.4 degrees of visual angle. 

Figure 1

 

Stimulus example: (A) Top view of an object placed in an imaginary sphere, illustrating the two viewing angles under which the rendered objects were saved, (B) the rightward orientation, and (C) the leftward orientation.

Procedure

Participants were seated at a distance of 150 cm from the screen with their head stabilized by a headrest and a bite bar with dental impression compound. Following a calibration phase for nine points along the diagonals of the stimulus field, the participants received a block of practice trials after which the experiment started. 

Participants initiated each trial (see Figure 2) by fixating a central cross. After detection of a steady fixation of 100 ms, a second cross indicating the future fixation position and two small circles were displayed at equal distances (7.3 degrees of visual angle) on an imaginary circle centered on the middle of the screen. The exact position of the second cross, the small circles, and hence, the to-be-displayed objects (positions A and B) differed from trial to trial: The coordinates for one object were generated at random (yet restricted to positions on the circle) and the coordinates for the other objects were calculated accordingly. A line segment connected the central fixation cross and the second cross toward which the participants had to make a saccade. During this saccade, the two circles were replaced by two objects (positions A and B) and a line segment again indicated the next saccade target. Upon making a saccade to the first object (position A), the peripheral cross was replaced by a third object (position C) and a line pointing at the second object (position B) indicated the following saccade target. Upon saccading to the second object (position B), the first object (position A) was replaced by a circle and a line was now connecting the second (position B) and third objects (position C), indicating the target of the following saccade. It was at this point that in three quarters of the experimental trials an intrasaccadic change could occur in the second object (a depth rotation in the cow in Figure 2). In one out of every four trials, the in-depth orientation of the second object was changed. In another quarter of the trials, the color of the second object was changed. In yet another quarter of the trials, both the color and the in-depth orientation were changed. In the remaining trials, nothing changed in the second object. During the saccade from object 2 (position B) to object 3 (position C), the second object was replaced by a circle. After having fixated the third object, participants had to make a saccade to the central cross. During this saccade, the third object was also replaced by a circle. After a fixation of 100 ms on the central cross, the response display was shown. It consisted of a right or left pointing arrow, centered at one of the positions previously occupied by an object, and of circles at the two other object positions. By pressing one of two buttons, participants then indicated whether the direction in which the arrow pointed corresponded to the direction/orientation of the object previously seen at that position. Without informing the participants about a possible difference between the pre- and postsaccadic views of the displayed objects, it was stressed that the in-depth orientation of the objects should be retained as seen foveally. More specifically, participants were instructed to try to remember the orientation (i.e., is the object facing to the right or to the left?) of each object before looking at the next object. At the end of a trial, feedback was provided by a word (“Right” or “Wrong”) on the screen. If the participants' eye did not land within prespecified landing regions surrounding the objects, the trial was stopped and marked for recycling and participants had to repeat it at the end of a block. It should be noted that, because all changes were made in the course of a saccade, there were no visible onsets throughout a trial. 

Figure 2

 

Course of a trial for Experiments 1 to 4. The ellipse indicates the position of the eye. After fixation of the central cross in display 1, display 2 was shown. All subsequent changes from display x to display x + 1 were made during the saccades toward the positions indicated by the line segments. The crucial intrasaccadic change (in this example, a change in the in-depth orientation of the second object) occurred during the transition from display 4 to display 5. Like the response display (display 8), the type of change differed from experiment to experiment. In Experiment 1, the participants had to indicate whether the arrow pointed in the same direction as the object previously seen at that position (the horse).

Design

Every subject participated in one practice block of 20 trials and 4 experimental blocks of 90 trials, which were carried out in two experimental sessions that each lasted approximately 1 h. An initial 320 experimental trials were produced by the within-subject factorial combination of 4 types of change (orientation and color change, orientation change only, color change only vs. no change), twenty stimuli that served as the critical object, and 4 different starting configurations (possible combinations of three objects in one of two possible starting depth orientations). Note that a maximum of 8 different starting configurations was possible, with 4 randomly assigned to one set of 10 critical objects and the other 4 assigned to the remaining set of 10 critical objects. Within a trial, all objects belonged to the same category (either vehicles or animals). In 40 additional trials, the intrasaccadic in-depth orientation change was not restricted to the second object, but changes could also occur either in the first, or in the third or in all three objects. These trials served as filler trials and were not included in the analysis. The position of the response item (at the position of the first, the second, or the third object) and the response alternative (same orientation or different orientation) were counterbalanced across conditions. The trials were presented in a pseudo-random order, generated independently for each participant. 

Results and discussion

During data collection, 7.6% of the trials were recycled because participants did not follow the required scanpath. The proportion of recycled trials did not vary as a function of the presence or type of intrasaccadic change ( p = 0.32). The collected data were reduced as follows: 0.8% of the data were excluded from analysis based on an on-line criterion following signal loss or inappropriate saccades to the specified positions. Outliers were defined as those trials in which gaze duration on the critical object deviated more than 3 standard deviations from the mean per participant, which further reduced the data set with 1.8% of the trials. Finally, only trials that had been answered to correctly were included in the analysis and this was the case in 92% of the remaining trials. An analysis of the percentage correct of the trials in which the orientation of the critical object had to be judged showed no effect of the intrasaccadic depth rotation ( F < 1). 

The remaining data were subjected to a within-subjects analysis of variance (ANOVA) with the type of change (orientation change only, color change only, orientation and color change, or no change in the critical object) as the independent variable and gaze duration on the critical object as the dependent variable. In this experiment and in all the following experiments, first fixation durations and number of first-gaze fixations on the critical object were analyzed as well. Because patterns of results were always in the same direction as for the gaze duration, but not equally reliable statistically, we restrict our report to gaze durations. Means are shown in Figure 3. There was a significant main effect of the type of change, F(3, 21) = 13.49, MSE = 106088, p < 0.001. A priori contrasts showed that this effect was due to task-dependent reactions to the type of change. Gaze durations in the orientation and the orientation plus color change condition were significantly longer than gaze durations in the no-change condition, F(1, 7) = 11.59, MSe = 134554, p < 0.02; F(1, 7) = 10.35, MSE = 173192, p < 0.02, and the color condition, F(1, 7) = 14.90, MSE = 165429, p < 0.007; F(1, 7) = 13.79, MSE = 181703, p < 0.008. For the other conditions, gaze durations did not differ significantly from each other (orientation vs. orientation plus color: F < 1; color vs. no change: p = 0.12). Because gaze durations on the critical object could be affected by the time spent in presaccadically previewing the object, the gaze duration on the first object was included as a covariate in the analysis of gaze durations on the second object. This analysis again showed a main effect of the type of intrasaccadic change [ F(3, 21) = 14.48, MSE = 93640, p < 0.0001]. 

Figure 3

 

Mean gaze durations (including 95% confidence intervals) on the critical object for Experiment 1 as a function of the type of intrasaccadic change.

The main conclusions to be drawn from the data in Experiment 1 are the following. First, when human observers try to remember the depth orientation of a series of objects, processing is easier when the object that is about to be fixated can be previewed. This is evidenced by the observation that the task takes longer, as reflected in prolonged gaze durations, when the presaccadic, peripheral depth orientation of the critical object differs from its depth orientation in the postsaccadic, foveal view (in comparison to a condition without intrasaccadic depth rotation). Note that participants were still able to perform the task, even when there was an intrasaccadic change in the task-relevant feature. This is evidenced by the observation that participants did not make more errors after an intrasaccadic change in the critical object's depth orientation than in the trials without change. The main finding is that it took them longer to encode the object's depth orientation when the postsaccadic depth orientation differed from the presaccadic depth orientation, suggesting that there was indeed presaccadic coding of the task-relevant information. However, this does not imply that the depth orientation of objects is automatically and mandatorily integrated across saccades (i.e., also in tasks that do not require participants to process the objects' depth orientation). This issue will be addressed in Experiment 2. 

Second, an intrasaccadic change in the color of the critical object did not prolong gaze durations on the critical object (in comparison to a control condition without intrasaccadic change). Note that this observation invalidates an alternative explanation for the finding that an intrasaccadic depth rotation did result in longer gaze durations. As discussed in the previous paragraph, we interpret the observation of shorter gaze durations in the absence of an intrasaccadic depth rotation as evidence for facilitation. However, there is an alternative interpretation, namely in terms of inhibition in the presence of an intrasaccadic change. Indeed, one could claim that the mere fact that “something” changed during the saccade to the critical object (irrespective of the type of change and even affecting lower levels of processing than pre-processing of an object's depth orientation) disrupted processing, resulting in longer gaze duration in the conditions with an intrasaccadic depth rotation. The absence of an effect of the color change (and the observation that a combined change in depth orientation and color had the same effect as a change in the depth orientation only) suggests that this is not a sufficient condition for prolonged gaze durations. We will come back to this issue in the General discussion section. 

An intrasaccadic color change did not affect gaze durations, but it is possible that the color change was less salient than the orientation change. Indeed, the density of color-sensitive cones drops off rapidly as eccentricity increases: At an eccentricity of only 1.75 degrees, the number of cones already decreases to about 50% of their number in the fovea (Curcio et al., 1991). Although it has been shown (Naïli, Despretz, & Boucart, 2006) that, even within those constraints, color vision is still possible at eccentricities up to 60 degrees, we decided to address this issue more directly in Experiment 2. 

Experiment 2

The aim of Experiment 2 was twofold. First, we investigated whether the depth orientation of real-world objects is always integrated transsaccadically or only when it is task relevant. The former can indeed be expected if one of the functions of integration across saccades is to expedite object identification and if identification requires access to a viewpoint-dependent object lexicon. To this end, the subjects' task was changed: While the depth orientation and/or the color of the critical object still could be changed intrasaccadically, participants now had to remember the color of the three objects, instead of their depth orientation. If gaze durations increase after a (now task-irrelevant) intrasaccadic depth rotation, this would imply that the depth orientation of the saccade target is automatically processed during the presaccadic fixation. Second, Experiment 2 allowed us to examine an alternative hypothesis for the finding in Experiment 1 that color changes did not result in longer gaze durations: Maybe the color of a peripheral object is not picked up presaccadically. If that is indeed the case, intrasaccadic color changes in the critical object should not disrupt processing, even if color is the task-relevant object feature. 

Methods

Participants, apparatus, stimuli, and procedure

Eight students (3 male, 5 female) drawn from the same pool as in Experiment 1 participated. None of them had taken part in the previous experiment. They all reported normal or corrected-to-normal vision and were paid 15 Euro for participating in two sessions. Apparatus and stimuli were the same as in Experiment 1. The procedure was also the same except for the task and the response display. As in Experiment 1, either the in-depth orientation, or the color, or both the color and the orientation of the critical object were changed or no change occurred during the saccade to the critical object. Each of the four events occurred in one fourth of the trials. Instead of registering the in-depth orientation of the three objects, the participants' task was to register the color of the objects. As a consequence, the response display consisted of a colored disk at the position of one of the objects and two circles at the other positions ( Figure 4). Participants had to indicate whether the color of the disk matched the color of the object previously seen at that position. When the color of an object was changed, it was changed to the fourth color that had not been used for one of the other objects in the display. 

Figure 4

 

Part of the trial course in Experiment 2 (panels 4, 5, and 8 correspond to the respective panels in Figure 2). Panel 5 shows the condition in which an intrasaccadic in-depth orientation change in the second to-be-fixated object (the cow) occurred during the transition from display 4 to display 5. Display 8 shows the response display for Experiment 2. Participants had to indicate whether the color of the disk matched the color of the object previously seen at that position (the horse).

Results and discussion

The proportion of recycled trials was 10.4% and did not vary as a function of the presence or type of intrasaccadic change ( p = 0.08). After recycling, 3.6% of the data were excluded on-line due to signal loss or inaccurate saccades and 1.3% were discarded because gaze duration on the critical object deviated more than 3 standard deviations from the mean per participant. Again only trials that had been answered to correctly were included (88% of the remaining trials). An analysis of the percentage correct of trials in which the color of the critical object had to be judged showed no effect of an intrasaccadic change ( F < 1). 

Mean gaze durations on the critical object are shown in Figure 5. An ANOVA indicated that the type of change affected gaze durations object in significantly different ways, F(3, 21) = 8.68, MSE = 68003, p < 0.0006. A priori contrasts showed that gaze durations were influenced by the task relevance of the manipulations. Task-relevant changes in the critical object (color and color plus orientation change) resulted in longer gaze durations on the critical object than in the no-change baseline condition [ F(1,7) = 23.64, MSE = 42620, p < 0.002; F(1, 7) = 13.89, MSe = 86822, p < 0.008] and the condition with an orientation change [ F(1, 7) = 5.84, MSE = 110421, p < 0.05; F(1, 7) = 7.19, MSE = 103928, p < 0.03]. The gaze durations in the other conditions did not differ from each other (color vs. orientation plus color: F < 1; orientation vs. no change: p = 0.25). As in Experiment 1, we only observed longer gaze durations in the conditions where the changed feature coincided with the to-be-reported object feature. Inclusion of the gaze duration on the first object as a covariate in the analysis of the gaze duration on the critical object slightly changed the size of the effect of the intrasaccadic changes, F(3, 21) = 10.13, MSE = 52658, p < 0.0003. 

Figure 5

 

Mean gaze durations (including 95% confidence intervals) on the critical object for Experiment 2 as a function of the type of intrasaccadic change.

The following conclusions can be drawn. First, in a task in which subjects had to remember the color of the objects, an intrasaccadic color change prolonged gaze durations, indicating that the color of the objects can be processed selectively across saccadic eye movements. Second, an intrasaccadic depth rotation did not affect gaze durations, suggesting that the in-depth orientation of an object is not automatically and mandatorily a part of the integrated collection of features that is carried across saccades. There is evidence that, when asked to do so, observers are able to remember multiple features of an object, both within fixations (Luck & Vogel, 1997) and across saccades (Carlson, Covell, & Warapius, 2001; also see Irwin & Andrews, 1996). Experiment 2 suggests that, at least for transsaccadic integration, this might be the case only when subjects are explicitly instructed to. Wheeler and Treisman (2002) also concluded that, although multiple bound features can be maintained, and this at no extra cost (in line with the findings of Luck & Vogel, 1997), binding integrated collections of features does not occur automatically, but only when it is required by the task. 

In retrospect, it is not surprising that in a task in which participants were instructed only to remember the color of stimulus objects we did not observe an effect of an intrasaccadic depth rotation of the critical object. There is indeed firm evidence that color and form are processed in separate cortical pathways (e.g., Corbetta, Miezin, Dobmeyer, Shulman, & Petersen, 1990; Dow, 2002; Gulyás & Roland, 1991; Livingstone & Hubel, 1988; Posner & Raichle, 1994; Quinlan & Humphreys, 1987; Smid, Jakob, & Heinze, 1997; Ts'o, Roe, & Gilbert, 2001; Zeki et al., 1991). This suggests that it is possible that observers indeed are able to attend to color information exclusively, without any processing of shape-related information (or at least not to such a degree that it would be reflected in gaze duration). 

To test the hypothesis that transsaccadic integration expedites object identification, it would therefore be useful to devise a task in which participants are not explicitly instructed to remember the objects' depth orientation but, at the same time, require them to process the shape of the objects and access a stored object lexicon. This was the purpose of Experiment 3. 

Experiment 3

The results of Experiments 1 and 2 suggest that object information is only integrated transsaccadically when the explicit task of the observers is to register that particular type of information: The depth orientation and the color of the saccade target are encoded from the periphery and integrated across saccades, but only when the information is task relevant, i.e., when observers are instructed to remember either the depth orientation or the color, respectively. However, task relevance can be defined more broadly than is implied in the explicit coding of a particular aspect of an object. Boucart and Humphreys (1992, 1997, Boucart, Humphreys, & Lorenceau, 1995; Boucart et al., 2000), for example, provided evidence that semantics are accessed (suggesting that object models stored in an LTM object lexicon are activated) whenever judgements have to be made about global shape features of an object, even when semantics do not have to be accessed to perform the task. This implies that, whereas color probably can be attended to selectively (as supported by Experiment 2), it is still possible that, whenever a task requires activation of stored object knowledge, the depth orientation of the saccade target will be integrated transsaccadically, even when subjects are not required to explicitly remember the objects' depth orientation. In Experiment 3, we presented upright and inverted objects and asked participants to judge the image-plane orientation of one of the objects, while the in-depth orientation of the critical object was changed intrasaccadically in half of the trials. Under the assumption that a viewpoint-dependent object lexicon has to be accessed in order to judge the image-plane orientation of an object, we expected gaze durations to be longer when the in-depth orientation was changed intrasaccadically, even without instruction to remember the objects' depth orientation. 

The predictions for upright and upside-down presented objects differ. In the case of upright objects, we assume that access to an object lexicon can be successful and therefore expect prolonged gaze durations after an intrasaccadic depth rotation of the saccade target. In the case of an inverted object, however, the absence of an effect of intrasaccadic depth rotations can be predicted. The reason is that all objects in our stimulus set were mono-oriented (i.e., they have a privileged upright orientation relative to a gravitationally defined axis, Verfaillie, 1992) and it has been shown that access to stored object models is more difficult for inverted mono-oriented objects than for upright versions (e.g., Jolicoeur, 1985, 1990; Lawson & Jolicoeur, 2003; Logothetis, Pauls, & Poggio, 1995). We can even speculate that, given the poor quality of the presaccadic view, observers will be unable to start identifying the inverted objects before they foveate them. Likewise, Boucart and Humphreys did not observe any semantic effects when objects were presented upside down, suggesting that the orientation-dependent object lexicon was not accessed for processing inverted objects. More specifically, in Experiment 3, the working hypothesis is that, to decide whether an object is presented upright or upside down, the visual system tries to access the object lexicon. If access is successful, it is decided that the object is upright. If access to the object lexicon is not possible, it is decided that the object is inverted. Note that a somewhat similar mechanism has been proposed in models of visual word recognition to account for reaction times in lexical decision tasks: These models assume that, if a certain processing duration has elapsed and lexical access has not been successful (in the case of a nonword), subjects decide that the string of letters is a nonword (e.g., Coltheart, Rastle, Perry, Langdon, & Ziegler, 2001; Grainger & Jacobs, 1996). In sum, in the case of upright objects, we predict that postsaccadic processing of the saccade target will be slowed when the depth orientation is changed intrasaccadically, because an orientation-dependent object model was activated. Because we assume that the orientation-dependent object lexicon will not be accessed for inverted objects, we do not expect an effect of an intrasaccadic orientation change for inverted objects. 

Methods

Participants

Nine new, naive, students (7 female, 2 male) from the University of Leuven with normal or corrected-to-normal vision participated in the experiment. Each participant was paid 30 Euro for participating in 4 sessions. 

Procedure

The course of a trial (see Figure 6) was analogous to the conditions in Experiments 1 and 2 in which the in-depth orientation of the second object could change but the stimulus and the task differed. In the present experiment, objects could be presented upright or upside down. In half of the experimental trials, the critical object could be rotated in depth during the observer's saccade to the object. In the other half of the trials, the object's depth orientation was not changed. At the end of a trial, participants had to indicate whether the orientation of an arrow matched the image-plane orientation of the object previously shown at the location of the arrow. Each participant was administered 720 trials resulting from the factorial combination of 2 types of change (depth rotation vs. no change), 8 different start configurations (all possible combinations of three objects in two start orientations), 2 image-plane orientations of the critical object (upright vs. inverted), and 20 stimuli that served as the critical object, in addition to 80 trials that were irrelevant filler trials. Participants thus performed a practice block of 20 trials and 8 experimental blocks of 90 trials. The total procedure lasted approximately 4 sessions of 1 h. 

Figure 6

 

Part of the trial course in Experiment 3 (panels 4, 5, and 8 correspond to the respective panels in Figure 2). Panel 5 shows the condition in which an intrasaccadic change occurred in the depth orientation of the critical object (the cow) during the transition from display 4 to display 5. Display 8 shows the response display for Experiment 3. Participants had to indicate whether the image-plane orientation of the arrow corresponded to the image-plane orientation of the object (the horse) previously seen at that location.

Results and discussion

During data collection, 10.1% of the trials were recycled because participants did not follow the required scanpath. The proportion of recycled trials did not vary as a function of the presence of intrasaccadic change or the image-plane orientation of the critical object (all p > 0.22). The data set was reduced by 2.6% of the trials following signal loss or inappropriate saccades and by 1.8% because the gaze duration on the second object deviated more than 3 standard deviations from the mean per participant. Only the trials that had been answered correctly were included in the analysis (91%). An analysis of the percentage correct of the trials in which the orientation of the critical object had to be judged showed no effect of the intrasaccadic depth rotation ( F < 1). 

Means for the gaze duration on the second object are shown in Figure 7. An ANOVA with the change variable and the critical object's orientation in the image plane as within-subject variables and gaze duration on the critical object as the dependent variable revealed a marginally significant interaction between the change variable and the orientation in the image plane, F(1, 8) = 3.73, MSE = 13142, p = 0.09. Because we had different predictions for the upright and the inverted objects, we ran two separate analyses for upright and inverted critical objects. For the inverted objects, gaze duration on the critical object was not influenced by the intrasaccadic in-depth orientation change ( F < 1, mean gaze duration of 409 ms when the depth orientation was changed vs. 408 ms without change). Inclusion of the gaze duration on the first object as a covariate did not reveal an effect either ( F < 1). When the critical object was upright, we did observe a significant effect of an intrasaccadic depth rotation on the gaze duration on that object, F(1, 8) = 6.17, MSE = 18457, p < 0.04 ( M = 403 ms with change vs. M = 389 ms without change). Inclusion of the gaze duration on the first object as a covariate in the analysis of the gaze duration on the second object did not change the observed effect, F(1, 8) = 7.98, MSE = 18886, p < 0.03. 

Figure 7

 

Mean gaze durations (including 95% confidence intervals) on the critical object for Experiment 3 as a function of the change variable and the image-plane orientation of the changed object.

From the results of the current experiment, it can be inferred that, when participants are asked to perform a task that requires access to stored object models, the objects, if presented upright, are encoded and integrated across saccades as objects in a particular in-depth orientation. As predicted, for inverted objects, no effects of an intrasaccadic orientation change were observed (either because of the higher difficulty/inability of accessing the in-depth orientation of (extrafoveal) inverted objects and/or because observers decided that an object was inverted because they did not succeed in accessing the object lexicon). These results thus provide strong evidence for the hypothesis that transsaccadic integration subserves subsequent object identification by means of the activation of orientation-dependent representations in a stored object lexicon. Explicit orientation coding was not required by the task and the intrasaccadic change of the critical object's in-depth orientation did not alter correct responses to the task. Nevertheless, differential gaze duration for the upright objects following change and no-change trials showed that the in-depth orientation was integrated across saccades. 

Note that, in contrast to Experiments 1 and 2, only one type of intrasaccadic change could occur in Experiment 3 (and Experiment 4). Moreover, whereas in Experiments 1 and 2, a change occurred in the critical object in three quarters of the trials, the critical object was changed in half of the trials in Experiment 3 (and Experiment 4). Although there is no a priori reason to expect that this would have an effect, we ran four separate experiments (with four independent, naive groups of participants). In one experiment, participants had to remember the objects' depth orientation and the critical object's depth orientation was changed intrasaccadically in half of the trials. In another experiment, participants again had to remember the objects' depth orientation, but the critical object's color was changed in half of the trials. In a third experiment, participants had to remember the objects' color and the critical object's color was changed in half of the trials. In a final experiment, participants again had to remember the objects' color but the depth orientation was changed in half of the trials. We replicated the findings of Experiments 1 and 2: An intrasaccadic change in a feature (color or depth orientation) of the critical object only had an effect when the object feature was task relevant. 

Experiment 4

Experiment 3 showed that the in-depth orientation of an object can be integrated transsaccadically without explicit instruction to code the in-depth orientation. What seems to be important is the relevance of a particular object feature to accomplish the task at hand. As outlined above, we assume that an orientation-dependent model is activated before object recognition is completed. To further explore this issue of indirect task relevance, we set up another experiment in which participants were not explicitly instructed to encode the objects' depth orientation, but which required access to stored object knowledge. More specifically, participants had to categorize objects in one of two categories (animals or vehicles). In half of the trials the in-depth orientation of the critical object was changed during the saccade toward the object. If the object lexicon has to be accessed in order to be able to perform the categorization task and if objects are represented in an orientation-dependent way in the object lexicon, we expected that a mismatch between the pre- and postsaccadic in-depth orientation of the critical object would influence categorization of that object and delay postsaccadic processing of the object. 

Methods

Eight students (5 female, 3 male), who did not participate in one of the previous experiments and were all naive to the experimental setup and the hypotheses under investigation, took part in the experiment. Each subject was paid 15 Euro for participating in two sessions. All objects were presented upright. The in-depth orientation of the second object was changed intrasaccadically in half of the trials but in the present experiment participants had to categorize the objects ( Figure 8), rather than remembering its depth or image-plane orientation. Accordingly, the response display consisted of a word (animal or vehicle) and two circles. Participants had to indicate whether the word in the response display corresponded to the category of the object previously seen at that position. 

Figure 8

 

Part of the trial course in Experiment 4 (panels 4, 5, and 8 correspond to the respective panels in Figure 2). Panel 5 shows the condition in which an intrasaccadic change occurred in the depth orientation of the critical object (the cow) during the transition from display 4 to display 5. Display 8 shows the response display for Experiment 4. Participants had to indicate whether the object previously seen at the position of the letter string (the horse) belonged to that category.

Results and discussion

During data collection, 11.9% of the trials were recycled because participants did not follow the required scanpath. The proportion of recycled trials did not vary as a function of the presence of intrasaccadic change ( F < 1). The data were reduced and analyzed in the same way as in the previous experiments. The data set was reduced by 1.4% on the basis of an on-line criterion. The exclusion of outliers (3 standard deviations above or below the mean per participant) further reduced the data set with 1.4%. Only correctly answered trials were included in the analysis (94% of the remaining trials). An analysis of the percentage correct of the trials in which the orientation of the second object had to be judged showed no effect of the intrasaccadic in-depth orientation change ( p = 0.15). 

Means for the gaze duration on the second object are shown in Figure 9. Gaze durations on the second object were longer ( F(1, 7) = 15.54, MSE = 29487, p < 0.006) when an intrasaccadic change occurred ( M = 400 ms) than when no change occurred ( M = 371 ms). The inclusion of the gaze duration on the first object as a covariate in the analysis of the gaze duration on the second object strengthened the effect of the intrasaccadic change, F(1, 7) = 29.46, MSE = 13141, p < 0.001. This result and the results from the previous experiment provide the most compelling evidence for viewer-centered transsaccadic representations subserving object recognition. Although participants were not explicitly asked to register the in-depth orientation of the objects, and although the intrasaccadic change of the in-depth orientation did not alter the required response, the data showed that the in-depth orientation was nevertheless retained across saccades. Accessing semantics necessarily involved the activation of orientation-dependent object representations. 

Figure 9

 

Mean gaze durations (including 95% confidence intervals) on the critical object for Experiment 4 as a function of the change variable.

General discussion

In four experiments, we examined the selective nature of transsaccadic object integration. Participants had to fixate three 3D-rendered objects positioned on an imaginary circle in an indicated order and were instructed to process one particular aspect of each object for future judgement. During the saccade from the first to the second object, either the to-be-reported aspect or another aspect of the second object could change. Gaze duration on that object—which is assumed to be a reflection of the amount of time needed to process an object—was analyzed as a function of the type of intrasaccadic change and the type of task. 

Two main conclusions concerning the task-dependent nature of transsaccadic integration are supported by the data. First, transsaccadic integration can be selective and this selectivity is determined by task requirements. In Experiments 1 and 2, either the depth orientation, or the color, or both depth orientation and color could be changed intrasaccadically, while only one of the two types of object aspects had to be reported (the objects' depth orientation in Experiment 1 and the color in Experiment 2). Only a change in the to-be-reported information resulted in longer gaze durations. An intrasaccadic change in a task-irrelevant object characteristic did not prolong gaze durations. This suggests that shape information (involved in registering an object's depth orientation) and color information can be attended to, remembered, and integrated across saccadic eye movements selectively. 

Second, this selectivity in integrating information transsaccadically was not restricted to cases where the changed object information had to be reported explicitly. Experiments 3 and 4 documented that the diagnosticity of the changed information to solve the task was an important determinant of whether information was integrated across saccades or not. In Experiment 3, participants had to judge whether an object was presented upright or inverted. When the object was upright, gaze durations were longer when the in-depth orientation of that object was changed than when no change occurred. When the objects were inverted, gaze durations were not higher following a change than without change. In Experiment 4, participants had to categorize the objects either as a vehicle or as an animal. In order to perform this task, access to stored object knowledge presumably was necessary. Because the in-depth orientation of the objects was integrated across saccades, it can be inferred that it is instrumental in categorizing an object. In line with our working hypothesis that transsaccadic memory serves to expedite object identification through the integration of presaccadic and postsaccadic object-diagnostic information, we thus found an effect of intrasaccadically changing an object feature on gaze duration even when that feature was only indirectly relevant to task performance. 

That task relevance can affect information acquisition, storage, and use across saccadic eye movements has been shown before (e.g., Ballard, Hayhoe, & Pelz, 1995; Hayhoe et al., 1998). A case in point is the recent work by Droll and colleagues (Droll, Hayhoe, Triesch, & Sullivan, 2005; also see Droll & Hayhoe, 2007). In a virtual-reality environment (originally developed by Triesch, Ballard, Hayhoe, & Sullivan, 2003), participants performed a brick sorting task in which brick features (e.g., color or width) could change intrasaccadically on 10% of the trials. Explicit change detection was poor in general, but changes to features that were relevant to the sorting task were detected reliably more often than changes to features that were task irrelevant. 

Our findings extend and complement this finding in two important ways. First, while change detection in the studies by Droll et al. (2005) reflects the contents of visual short-term memory (i.e., what is remembered from an object that was fixated prior to the feature change), our studies tap into transsaccadic information integration (i.e., what is remembered from a presaccadic, peripheral preview and integrated with a postsaccadic, foveal view of the same object). Second, in the study of Droll and colleagues, participants were instructed to detect intrasaccadic object changes explicitly.² In our study, participants were never required to detect changes explicitly and hence integrate pre- and postsaccadic information. The instructions stressed that the information that had to be remembered concerned the object as seen foveally. Therefore, in principle, participants could base their judgement exclusively on the postsaccadic stimulus without transsaccadic integration of any information at all. Nevertheless, we observed that gaze duration increased following an intrasaccadic change in a task-relevant feature, even when the feature only had indirect task relevance (i.e., when the feature was instrumental to performing the task at hand, as illustrated by the transsaccadic sensitivity to in-depth object orientation changes when judging semantic category membership of objects). 

Taken together, our findings indicate that prior to a saccade to a given object, extrafoveal object features are routinely processed when they are task relevant and are stored and subsequently compared during postsaccadic foveation of the object. When pre- and postsaccadic data are identical, postsaccadic object gaze durations are reliably shorter. We interpret this effect as a preview benefit, that is, a presaccadic facilitation of postsaccadic processing in the absence of an intrasaccadic change (and a cost when a useful presaccadic preview was not available). 

Admittedly, one could argue for an alternative interpretation, according to which intrasaccadic change adds processing demands in comparison to the conditions without intrasaccadic changes, with the former condition confusing subjects in one way or another. At present, there are two reasons why we are hesitant to accept this alternative interpretation of our observations. First, our finding that changes in task-irrelevant object features did not prolong gaze durations (e.g., changes in the critical object's depth orientation did not result in longer gaze durations when subjects had to remember the color of the objects) suggests that there was no general inhibitory mechanism at work, which interrupted processing whenever a discrepancy between pre- and postsaccadic views was detected. Of course, this argument against an overall inhibitory mechanism does not rule out the possible operation of a feature-specific inhibitory process, which only kicks in when feature changes occur in the attended (task-relevant) feature dimension. 

A second reason why our interpretation is inclined toward facilitation rather than inhibition is that in classical transsaccadic preview paradigms, in which subjects had to name an object (e.g., Pollatsek et al., 1984), observers were always faster and/or more accurate to name the postsaccadic object that could be previewed (i.e., there was no intrasaccadic change) in comparison to conditions in which there was no presaccadic object at all. This supports the hypothesis of facilitation in the absence of an intrasaccadic change over the hypothesis of inhibition in the presence of an intrasaccadic change. 

In sum, we conclude that task-relevant information about objects is acquired extrafoveally, stored transsaccadically, and subsequently integrated with postsaccadic foveal information in order to expedite the perceptual task that is currently performed by the observer. Furthermore, under conditions that require a shape judgment, transsaccadic object perception will lead to the automatic, presaccadic activation of representations in a stored object lexicon. Our results favor theories that propose that objects in such a lexicon are stored in a viewpoint-dependent format. 

Appendix A

Stimuli of Experiments 1 to 4

Horse	Fish	Car	Airplane
Cow	Chicken	Bus	Helicopter
Cat	Bee	Truck	Sleigh
Moose	Rabbit	Jeep	Boat
Camel	Penguin	Van	Motorbike

Acknowledgments

This research was supported by the Leuven University through Grant GOA 98/01, by the Flemish government through FWO Grants G.0583.05 and 7.00050.05, and by the European Community through GazeCom Project IST-C-033816. 

Commercial relationships: none. 

Corresponding author: Karl Verfaillie. 

Email: [email protected]. 

Address: Department of Psychology, University of Leuven, Tiensestraat 102, B-3000 Leuven, Belgium. 

Footnotes

Footnotes

1  An intrasaccadic change occurs during the observer's eye movement. Because the intake of information during saccades is drastically reduced, the transient that normally accompanies a stimulus change is invisible.

Footnotes

2  We should mention that in a fraction of the trials in which participants failed to report the change explicitly, their hand movements and fixations to the brick were longer when a change was made to the brick.