Breaking object correspondence across saccades impairs object recognition: The role of color and luminance

Abstract

Rapid saccadic eye movements bring the foveal region of the eye's retina onto objects for high-acuity vision. Saccades change the location and resolution of objects' retinal images. To perceive objects as visually stable across saccades, correspondence between the objects before and after the saccade must be established. We have previously shown that breaking object correspondence across the saccade causes a decrement in object recognition (Poth, Herwig, & Schneider, 2015). Color and luminance can establish object correspondence, but it is unknown how these surface features contribute to transsaccadic visual processing. Here, we investigated whether changing the surface features color-and-luminance and color alone across saccades impairs postsaccadic object recognition. Participants made saccades to peripheral objects, which either maintained or changed their surface features across the saccade. After the saccade, participants briefly viewed a letter within the saccade target object (terminated by a pattern mask). Postsaccadic object recognition was assessed as participants' accuracy in reporting the letter. Experiment A used the colors green and red with different luminances as surface features, Experiment B blue and yellow with approximately the same luminances. Changing the surface features across the saccade deteriorated postsaccadic object recognition in both experiments. These findings reveal a link between object recognition and object correspondence relying on the surface features colors and luminance, which is currently not addressed in theories of transsaccadic perception. We interpret the findings within a recent theory ascribing this link to visual attention (Schneider, 2013).

Introduction

Human vision is based on a reiterating cycle of saccadic eye movements and intervals of relatively stable eye position, the so-called fixations. Saccades shift the eye rapidly, directing its foveal high-acuity region at potentially interesting parts of the environment. Fixations provide clear visual snapshots of objects, snapshots that are not corrupted by the suppression of information uptake or by the motion blur, which occur during saccades (e.g., Krock & Moore, 2014; Wurtz, 2008). However, snapshot-like sampling also poses a problem for perception and action. Perceiving objects as continuously present across saccades and to act based on this perception require that the snapshots of objects from successive fixations are linked (e.g., Higgins & Rayner, 2015; Schneider, 2013). This linkage is complicated by the fact that every saccade displaces an object's image on the retina and changes its resolution (e.g., Herwig & Schneider, 2014; Wurtz, 2008, 2015). How, then, does the visual system achieve coherent representations of external objects across saccades? 

Current theories propose that coherent transsaccadic object representations depend on a test for object correspondence across saccades (Hollingworth, Richard, & Luck, 2008; Tas, Moore, & Hollingworth, 2012). This means the visual system tests whether postsaccadic and presaccadic object representations likely stem from the same external objects. If the test for object correspondence is positive (i.e., object correspondence is established), presaccadic object representations are updated with postsaccadic information (Demeyer, De Graef, Wagemans, & Verfaillie, 2009; Henderson & Anes, 1994), leaving only one postsaccadic representation (Tas et al., 2012; see also Schneider, 2013). Having only one object representation may entail visual stability, the perception of a continuous and stable visual world despite the saccade-induced changes of retinal images (for reviews on visual stability, see Bridgeman, Van der Heijden, & Velichkovsky, 1994; Mathôt & Theeuwes, 2011; Wurtz, 2008; Wurtz, Joiner, & Berman, 2011). However, this also means that presaccadic and postsaccadic objects cannot be compared, which explains why object displacements (Bridgeman, Hendry, & Stark, 1975) and changes of visual object features (Deubel, Schneider, & Bridgeman, 2002; Weiß, Schneider, & Herwig, 2015) are hard to perceive when they occur during saccades. In contrast, if the test for object correspondence is negative (i.e., object correspondence is broken), presaccadic and postsaccadic objects are assumed to be represented separately (Deubel, Schneider, & Bridgeman, 1996; Schneider, 2013; Tas et al., 2012). This diminishes the perception of visual stability but helps to discriminate intrasaccadic object changes, presumably because the two representations can be compared (Deubel & Schneider, 1994; Deubel et al., 2002; Deubel et al., 1996; Tas et al., 2012; Weiß et al., 2015). 

Transsaccadic object correspondence not only is important for visual stability but also has recently been shown to affect object recognition (Poth et al., 2015; see also Schneider, 2013). In this study, participants made saccades to a peripheral object. After the saccade, a letter was shown in this object and terminated by a pattern mask. Correspondence between the presaccadic and the postsaccadic object was broken with two different manipulations: first, by introducing a blank screen after eye landing and before onset of the postsaccadic object (see Deubel & Schneider, 1994; Deubel et al., 1996) and, second, by a large change of the luminance and the contrast polarity of the object during the saccade (see Tas et al., 2012). In both cases, recognition of the postsaccadic letter was deteriorated. This shows that breaking transsaccadic object correspondence impairs postsaccadic object recognition. Two explanations of this effect rely on the idea that the presaccadic and postsaccadic object are represented separately if object correspondence is broken. First, the theory of Task-dRiven visual Attention and working Memory (TRAM; Schneider, 2013) proposes that broken object correspondence (object continuity) across fixations results in two different object representations. Limited attentional resources must be split between the two representations, cutting the resources available to each individual object representation. This loss of attentional resources per object explains the deteriorated recognition of the postsaccadic object. Second, the creation of a separate postsaccadic representation in addition to the presaccadic one may delay (or hinder) processing of the postsaccadic object. Because the postsaccadic object was terminated by a mask, this delay would have become manifest in deteriorated recognition of the object (Poth et al., 2015). 

Postsaccadic object recognition depends on transsaccadic object correspondence (Poth et al., 2015), but the mechanisms underlying this effect remain elusive. To shed light on these mechanisms, it is important to clarify which object features contribute to the test for object correspondence. Two classes of features are distinguished in the literature on object correspondence across occlusion (Hollingworth & Franconeri, 2009) and movement (Kahneman, Treisman, & Gibbs, 1992; Mitroff & Alvarez, 2007): spatiotemporal and surface features. Classical theories proposed that object correspondence was established solely (Kahneman et al., 1992) or primarily (Flombaum, Scholl, & Santos, 2009; Scholl, 2007) on the basis of spatiotemporal features. In stark contrast, however, more recent research revealed that object correspondence across occlusion can also be established based on surface features (such as color and/or luminance), even when it conflicts with the spatiotemporal feature location (Hollingworth & Franconeri, 2009). Along the same lines, object correspondence across saccades seems to rely on both spatiotemporal (Demeyer, De Graef, Wagemans, & Verfaillie, 2010; Deubel, Bridgeman, & Schneider, 1998; Deubel et al., 1996; Deubel et al., 2002) and surface features (Tas et al., 2012). As explained above, there is first evidence (Poth et al., 2015) that postsaccadic object recognition is deteriorated both when transsaccadic object correspondence is broken by blanking, which is a violation of spatiotemporal correspondence, and by introducing large changes of luminance and contrast polarity, which is a strong violation of surface feature correspondence. Critically, however, it remains to be clarified whether this holds also for surface features other than luminance and contrast polarity and less intense feature changes. 

The surface feature of color is generally considered vital for human vision (e.g., Gegenfurtner & Kiper, 2003; Moutoussis, 2015), but it is unknown whether color is used for establishing object correspondence across saccades. Changing the apparent color of an object is a common manipulation to study how surface features contribute to object correspondence across occlusion (Hollingworth & Franconeri, 2009) and movement (Kahneman et al., 1992; Mitroff & Alvarez, 2007). However, such changes of apparent color may coincide with changes in luminance and contrast polarity (e.g., Mitroff & Alvarez, 2007), as these surface features are usually not distinguished from color. Therefore, the role of color for object correspondence across occlusion and movement remains unclear. The role of color may even be less clear for object correspondence across saccades. On the one hand, the color of a given object is represented with much lower quality in the visual periphery than in the fovea (e.g., Hibino, 1992; Johnson, 1986; Livingstone & Hubel, 1987; Nagy & Wolf, 1993). Thus, if color was used to establish transsaccadic object correspondence, the natural differences between an object's peripheral presaccadic and its foveal postsaccadic color could erroneously break object correspondence. This would impair postsaccadic object recognition (Poth et al., 2015). One may therefore hypothesize that transsaccadic color changes are ignored. On the other hand, there is evidence that at least large changes in apparent color, which may include changes in luminance, can break transsaccadic object correspondence and perceived visual stability (Tas, 2015; cf. Hollingworth et al., 2008, for evidence from corrective saccades). 

Here, we investigated whether breaking object correspondence across the saccade by changing the surface feature of color impairs postsaccadic object recognition. To retain the link to previous studies of object correspondence, we examined the effects of changes in apparent color (color, luminance, and contrast polarity) on transsaccadic object correspondence (Experiment A). In addition, we examined the effects of changes between approximately equiluminant colors on transsaccadic object correspondence (Experiment B). Both experiments employed the experimental paradigm by Poth et al. (2015; Experiment 2). Participants made saccades to a peripheral object, a letter was shown in this object after eye landing, and the letter presentation was terminated by a pattern mask. Participants' task was to report the identity of the postsaccadic letter. Transsaccadic object correspondence was manipulated in two conditions: The surface features of the object either stayed the same across the saccade (no-change condition) or they were changed during the saccade (change condition). Experiment A used the opponent colors green and red as surface features, each coinciding with a different physical luminance and contrast polarity. Experiment B used the colors blue and yellow with approximately the same luminance and contrast polarity. If breaking transsaccadic object correspondence by changing these surface features impairs postsaccadic object recognition, then letter report performance should be lower in the change compared with the no-change conditions of both experiments. 

Method

Participants

Ten participants took part in Experiment A. They were between 20 and 30 years old (MD = 25 years), five were male, and five female. Ten different participants performed Experiment B. Their ages ranged from 21 to 26 years (MD = 23.5 years), and two were male and eight were female. All participants of both experiments reported normal color vision and normal or corrected-to-normal (contact lenses) visual acuity. All participants were paid and gave written informed consent before participation, and the experiments were approved by Bielefeld University's ethics committee. 

Apparatus and stimuli

Participants performed the experiments in a dimly lit room. They viewed the 19-inch CRT screen (G90fB, ViewSonic, Brea, CA) from a distance of 71 cm while their head position was fixed by forehead and chin rests. The screen had a resolution of 1,024 × 768 pixels (at physical dimensions of 36 × 27 cm) and a refresh rate of 100 Hz, and it was controlled by a GeForce GT 640 (NVIDIA, Santa Clara, CA) graphics card. A video-based desktop-mounted eye tracker sampled participants' right eyes at 1000 Hz (Eyelink 1000, SR Research, Mississauga, Ontario, Canada). The eye tracker was calibrated using a nine-point grid procedure. Calibration was performed in the beginning of the experiment, after training trials, after a pause in about the middle of the experiment (and after participants had made 50 fixation or saccade errors in total). Saccades were detected online using velocity and acceleration thresholds of 35° (degrees of visual angle) × s⁻¹ and 9500° × s⁻². Responses were collected using a standard QWERTZ computer keyboard. The experiment was programmed using the Psychophysics toolbox (3.0.12; Brainard, 1997; Kleiner, Brainard, & Pelli, 2007; Pelli, 1997) and Eyelink toolbox (3.0.12; Cornelissen, Peters, & Palmer, 2002) extensions for MATLAB (R2014b; The MathWorks, Natick, MA). 

Color and luminance were measured using an X-Rite i1 Pro spectrophotometer (Munich, Germany), and measurements are provided as CIE Lxy coordinates. A black (L = 0.228 cd/m², x = 0.290, y = 0.286) square (0.1° × 0.1°) was used as central fixation stimulus. In Experiment A, saccade target objects were green (L = 90.871 cd/m², x = 0.279, y = 0.591) and red (L = 30.664 cd/m²; x = 0.599, y = 0.327) ellipses (0.65° × 1.05°). Note that the objects differed in luminance and contrast polarity. In Experiment B, saccade target objects had approximately the same luminance, and they were blue (L = 37.113 cd/m²; x = 0.194; y = 0.143) and yellow (L = 36.887 cd/m²; x = 0.415; y = 0.479). Letters (ABDEFGHJKLMNOPRSTVXZ; 0.32° × 0.4°) and special characters (%#§&; 0.4° × 0.4°) were written in Arial font and matched the gray background (L = 47.687 cd/m²; x = 0.283, y = 0.291) in both experiments. In each experiment, 99 pattern masks were algorithmically produced for each individual participant and for both colors. A large number of masks was used to minimize adaptation to them. The masks consisted of colored squares (2° × 2°), filled with nine black letters that were drawn randomly without replacement from the set of letters. The nine letters were mirror reversed and upside down and overlapped partially, and all letters together covered an area of about 1° × 1° within a square. 

Procedure and design

Figure 1a illustrates the experimental paradigm, which is based on the paradigm by Poth et al. (2015). The participant pressed the space bar to start a trial. A central fixation stimulus was shown, and the participant fixated it for a random interval between 500 and 1000 ms. Next, an elliptic saccade target object appeared 8° horizontally from screen center. Whether the object appeared to the left or right of screen center was randomized across trials, whereby each side occurred equally often for each of the postsaccadic surface features and each condition. The object contained an irrelevant special character, which was randomly drawn from the set of special characters, and it was presented until the participant made a saccade to it. Figure 1b illustrates the surface features and experimental conditions of both experiments. In Experiment A, the presaccadic object was either green or red (coincident with different luminances and contrast polarities), and in Experiment B, it was either blue or yellow (with approximately the same luminances and contrast polarities). These surface features were randomized across trials, each occurring equally often in each condition. The postsaccadic object contained a letter that was randomly drawn from the set of letters¹ and was shown during the saccade (on the next screen refresh after detection of saccade onset). The presaccadic and the postsaccadic object had the same surface features in the no-change condition and different surface features in the change condition. Thus, in the change condition of Experiment A, green objects changed into red ones and vice versa. In the change condition of Experiment B, blue objects changed into yellow ones and vice versa. After the postsaccadic object, a pattern mask of the same surface feature was presented. This mask was shown two or three screen refreshes (frames) after the online detection of saccade end (and its registration by the experimental software), so that the postsaccadic object was visible after the saccade end detection for 31 ms on average (SD = 3 ms). The mask was randomly drawn from the set of produced masks and was shown for 300 ms. It was followed by a blank screen, and participants reported the identity of the letter using the keyboard. There was no time limit for the report. The next trial could be started after an intertrial interval of 100 ms. Participants did not receive any instructions regarding the surface features or changes. 

Figure 1

 

(a) Schematic illustration of the experimental paradigm. Participants fixated a central fixation stimulus, which was followed by an elliptic saccade target object, displaying one of two surface features (green and red with different luminances in Experiment A, blue and yellow with approximately the same luminance in Experiment B). This object appeared in the periphery and contained an irrelevant special character. Participants made a saccade to the object. In the no-change condition, the object displayed the same surface feature after the saccade. In the change condition, the postsaccadic object displayed different surface features than the presaccadic one. In both conditions, the postsaccadic object contained a letter and was terminated by a pattern mask. Participants reported the identity of the letter. Special characters, letters, and the background were gray (here drawn in black and white for better visibility). (b) Employed surface features and experimental conditions of Experiment A and Experiment B.

Participants performed 64 trials of each condition in randomized order. Trials were aborted and repeated on a randomly selected subsequent trial if observers did not fixate the central fixation cross or missed the saccade target object by more than 2.5°. In Experiment A, 24.3% of the trials were repeated, in Experiment B 31.11%. Participants performed 32 training trials before each experiment. 

Results and discussion

Trials were excluded from analysis if saccade latency (the time from onset of the saccade target object until saccade onset detection) was less than 100 ms (anticipatory saccades) or greater than 400 ms. Two trials were excluded from Experiment A and four trials from Experiment B. Table 1 provides descriptive statistics of all dependent variables in both experiments. 

Table 1

 

Means of letter report performance, saccade latency, and saccade landing errors across participants for Experiment A and B. Values are provided for cells formed by the two conditions (no-change and change) and the two postsaccadic surface features (green and red in Experiment A, and blue and yellow in Experiment B). Standard deviations are in parentheses.

 

Letter report performance in Experiment A

Letter report performance was assessed as the proportion of correctly reported letters for each individual participant. Because of the truncated range proportions take, all analyses were also performed on acrsine-squareroot-transformed proportions in addition to original proportions. Both sets of analyses yielded consistent results, and therefore, only analyses of original proportions are reported. 

Figure 2a depicts the mean proportion of correctly reported letters across participants of Experiment A. The effects of condition and postsaccadic surface features on letter report performance were analyzed using a 2 × 2 (no-change vs. change × green vs. red) repeated-measures analysis of variance (ANOVA; with type III sums of squares and Display Formula as effect size; Bakeman, 2005). The ANOVA revealed a significant main effect of condition, F(1, 9) = 15.607, p = 0.003, Display Formula = 0.181. Performance was higher in the no-change (M = 0.90, SD = 0.13) compared with the change condition (M = 0.73, SD = 0.19). Thus, breaking transsaccadic object correspondence by changing the combination of the surface features color, luminance, and contrast polarity impaired recognition of the postsaccadic letter. As argued previously (Poth et al., 2015), one might suspect that the change of the special character in the presaccadic object into the letter in the postsaccadic object also broke transsaccadic object correspondence. It is important, however, that even if this were the case, the present results would still demonstrate an effect of breaking object correspondence by changing color, luminance, and contrast polarity in addition to the possible effect of changing the presaccadic special character.  

Figure 2

 

Letter report performance. Depicted are mean proportions of correct letter reports in the two conditions (change vs. no change) and for both postsaccadic surface features (green and red in Experiment A, blue and yellow in Experiment B). Error-bars indicate ±1 standard error of the mean for within-subjects designs (Loftus & Masson, 1994); the dashed line indicates chance level. (a) Experiment A. (b) Experiment B.

There was also a significant main effect of postsaccadic surface features F(1, 9) = 12.751, p = 0.006, Display Formula = 0.227, showing that performance was higher for green (M = 0.91, SD = 10.10) than for red (M = 0.72, SD = 0.22) postsaccadic objects. Postsaccadic object color, luminance, and contrast polarity may have affected the visibility and hence recognition of the postsaccadic letter (as has been shown for luminance contrast by Petersen & Andersen, 2012). The interaction between the two factors was significant as well, F(1, 9) = 9.895, p = 0.012, Display Formula = 0.028. Specifically, the difference between performance in the no-change compared with the change condition was smaller for the green postsaccadic objects (M = 0.11, SD = 0.14) compared with the red ones (M = 0.23, SD = 0.16).  

Saccade latencies and landing errors in Experiment A

Saccade latencies were assessed as each participant's mean interval (in ms) between the onset of the saccade target object and detection of the saccade. Table 1 provides descriptive statistics for saccade latencies in the two conditions in conjunction with the postsaccadic surface features. Saccade latency was affected neither by condition, F(1, 9) = 0.067, p = 0.801, Display Formula < 0.001, nor by postsaccadic surface features, F(1, 9) = 0.214, p = 0.655, Display Formula < 0.001. However, there was a disordinal interaction between the two factors, F(1, 9) = 21.152, p = 0.001, Display Formula = 0.036. Note that this interaction effect corresponds to a main effect of presaccadic surface features if the presaccadic rather than the postsaccadic surface features entered the ANOVA as second factor besides condition. Saccade latencies were shorter when presaccadic objects were green (M = 148 ms; SD = 11 ms) rather than red (M = 153 ms; SD = 12 ms). This may indicate that green saccade target objects were perceptually more salient, which implies they were easier to detect and localize than red ones, leading to faster saccades for the former compared with the latter.  

Saccade landing errors were assessed as each participant's mean Euclidian distance (in °) between saccade landing sites and saccade target objects (i.e., the center coordinates of these objects). Neither condition, F(1, 9) = 0.362, p = 0.562, Display Formula = 0.003, nor postsaccadic surface features, F(1, 9) = 1.399, p = 0.267, Display Formula = 0.007, nor the interaction between the two factors, F(1, 9) = 2.346, p = 0.160, Display Formula = 0.015, had significant effects on saccade landing errors.  

Letter report performance in Experiment B

As for Experiment A, letter report performance was analyzed based on the original and the arcsine-squareroot-transformed proportions of correct reports. Only the analyses of the original proportions are reported because both sets of analyses delivered consistent results. 

Figure 2b depicts the mean proportion of correctly reported letters across participants of Experiment B. The ANOVA showed a significant main effect of condition, F(1, 9) = 13.514, p = 0.005, Display Formula = 0.031. Thereby, performance was higher in the no-change (M = 0.76; SD = 0.26) than in the change condition (M = 0.67; SD = 0.28). This indicates that breaking transsaccadic object correspondence by changing between the approximately equiluminant object colors impaired recognition of the postsaccadic letter.  

There was also a significant main effect of postsaccadic color, F(1, 9) = 7.902, p = 0.020, Display Formula = 0.023, whereby letter report performance was higher for blue (M = 0.75; SD = 0.25) than for yellow postsaccadic objects (M = 0.67; SD = 0.29). This may suggest that the postsaccadic color affected the visibility and recognition of the letter. The interaction between condition and postsaccadic color was not significant, F(1, 9) = 0.013, p = 0.912, Display Formula < 0.001.  

Saccade latencies and landing errors in Experiment B

Saccade latency was unaffected by condition, F(1, 9) = 0.079, p = 0.785, Display Formula < 0.001, and postsaccadic color, F(1, 9) = 0.026, p = 0.876, Display Formula < 0.001. However, there was an interaction between these two factors, F(1, 9) = 45.035, p < 0.001, Display Formula = 0.067. This interaction effect corresponds to a main effect of presaccadic color if this was included in the ANOVA instead of the postsaccadic color. Saccades were faster when presaccadic objects were yellow (M = 159 ms; SD = 15 ms) rather than blue (M = 167 ms; SD = 17 ms). This effect may be due to a higher perceptual saliency of the yellow compared with the blue objects, which may have sped up the detection and localization of saccade target objects (see Experiment A).  

Saccade landing errors were neither affected by condition, F(1, 9) = 0.377, p = 0.554, Display Formula = 0.001, nor by postsaccadic color, F(1, 9) < 0.001, p = 0.987, Display Formula < 0.001, nor by the interaction of the two factors F(1, 9) = 0.949, p = 0.355, Display Formula = 0.003.  

General discussion

We tested the hypothesis that breaking object correspondence across the saccade by changing surface features impairs postsaccadic object recognition. The results of two experiments support this hypothesis. Experiment A revealed that transsaccadic changes between the combined surface features color, luminance, and contrast polarity deteriorate postsaccadic object recognition. Experiment B yielded similar findings for colors of approximately the same physical luminance and the same contrast polarity. Together, the results indicate that postsaccadic object recognition depends on mechanisms establishing transsaccadic object correspondence on the basis of these surface features. Although it has been shown before that luminance and contrast polarity contribute to transsaccadic object correspondence, the effects of color with approximate equiluminance may be surprising. That is, the presaccadic peripheral and the postsaccadic foveal retinal images of an object provide color information of substantially different quality (e.g., Hibino, 1992; Johnson, 1986; Nagy & Wolf, 1993; and possibly perceived luminance, Livingstone & Hubel, 1987). Therefore, color might not be an ideal feature for establishing object correspondence across the saccade. Nevertheless, color seems to be used for this purpose in concert with luminance and contrast polarity, together paving the way for object recognition. 

Hitherto, the dependency of postsaccadic object recognition on transsaccadic object correspondence has only been studied using two correspondence-breaking manipulations: blanking and changing the contrast polarity of achromatic objects, which coincided with large luminance changes (Poth et al., 2015). The present findings extend these results to chromatic objects. In Experiment A, transsaccadic object correspondence was broken by changing between colors with different physical luminances and contrast polarities. Experiment B replicated the results using colored objects of about the same physical luminance. The perceived luminance of a given color differs at different retinal locations (and this might depend on individual participants; Livingstone & Hubel, 1987), so that saccades might always imply a change of an object's perceived luminance. Consequently, transsaccadic changes of object color may change perceived luminance, even for physically equiluminant objects. This means that perceived luminance could still have contributed to the effect of Experiment B. Hence, this effect may either be due to changes of the object's chromaticity and/or the associated luminance changes. In either way, this demonstrates that the changes are not ignored when transsaccadic object correspondence is determined, so that they affect postsaccadic object recognition. The present findings indicate that postsaccadic object recognition depends on mechanisms of transsaccadic object correspondence that use information from surface features in general or at least from the surface features of contrast polarity and luminance and of color, whereby the latter might inherently include a contribution of luminance. As such, these findings conflict with the view that transsaccadic object correspondence relies exclusively on spatiotemporal features (Kahneman et al., 1992; with respect to effects on postsaccadic object recognition). In sum, our experiments demonstrated that postsaccadic object recognition is deteriorated when transsaccadic object correspondence is broken by changes of spatiotemporal features (blanking) and of surface features such as contrast polarity and luminance (Poth et al., 2015), combined color, luminance, and contrast polarity (Experiment A) and of color (Experiment B). 

Presaccadic and postsaccadic objects are assumed to be represented as separate entities if transsaccadic object correspondence is broken (e.g., Tas et al., 2012; see also Deubel et al., 1996; Schneider, 2013). Consequently, the deteriorated object recognition can be interpreted in at least two ways, which need not be mutually exclusive. First, creating a separate representation for the postsaccadic object may delay or hinder processing of this object. Recognition of this object would then be deteriorated, especially if the object is terminated by a mask (Poth et al., 2015). Second, having separate representations of the presaccadic and the postsaccadic object may introduce attentional competition between them (Schneider, 2013). That is, limited attentional resources are split between the two representations. Fewer resources are available for processing each individual object representation, which then deteriorates object recognition (Bundesen, 1990; Desimone & Duncan, 1995). We assessed postsaccadic object recognition as performance in reporting a letter, which participants viewed after the saccade in the saccade target object. The letter should have been processed as part of the surface of this object, so that letter report performance reflects recognition of this object (cf. Henderson & Anes, 1994; Kahneman et al., 1992; Mitroff & Alvarez, 2007). However, it might be possible that the letter has been processed as a separate object. Letter report performance would then reflect recognition of a newly appearing object at the spatial location of the saccade target object rather than recognition of this object itself. This would still be in line with the two explanations, following the assumption of competitive object recognition (e.g., Bundesen, Habekost, & Kyllingsbæk, 2005; Desimone & Duncan, 1995). Either the creation of a representation for the letter at this very location would be delayed or it would suffer from greater attentional competition if transsaccadic object correspondence was broken and led to two rather than one representation of the saccade target object. 

The present findings argue that the surface features of combined color, luminance, and contrast polarity, as well as color alone, are used by object correspondence mechanisms, which track objects across saccade-induced shifts of retinal images. Moreover, they argue that these object correspondence mechanisms affect mechanisms of object recognition. Based on TRAM (Schneider, 2013), we suggest that this may be due to an interface between the two sets of mechanisms, which is provided by visual attention (see also Poth et al., 2015). We elaborate this hypothesis in the following. 

Which mechanism tracks objects across the saccade-induced changes of retinal images? The tracking of object locations across saccades may be accomplished by retinotopically organized brain areas (as, e.g., monkeys' lateral intraparietal area, Duhamel, Colby, & Goldberg, 1992; superior colliculus, Walker, Fitzgibbon, & Goldberg, 1995; and frontal eye fields, Umeno & Goldberg, 1997). Shortly before a saccade, neurons in these areas respond to stimuli at the locations where their receptive fields (the retinal regions from which they receive information) will be after the saccade. This has been interpreted as a predictive remapping of the neurons' receptive fields to these locations (Duhamel et al., 1992; but see Zirnsak, Steinmetz, Noudoost, Xu, & Moore, 2014). The necessary information about the amplitude and direction of the saccade seems to come from a corollary discharge (efference copy) of the motor signals eliciting the saccade (Sommer & Wurtz, 2006). When the receptive field of a neuron is predictively remapped, the neuron responds to a particular object before the saccade. The following saccade-induced shift of the neuron's receptive field makes the neuron respond to the same object again after the saccade. An additional process comparing the presaccadic and postsaccadic activity of such neurons might then allow one to infer the presence of an object before and after the saccade. Therefore, such a comparison has been hypothesized to underlie the perception of visual stability of object locations across saccades (Cavanaugh, Berman, Joiner, & Wurtz, 2016; Duhamel et al., 1992; Wurtz et al., 2011). The comparison may be part of the neuronal implementation of the test for transsaccadic object correspondence, the test that is assumed to govern visual stability (Poth et al., 2015; Tas et al., 2012). One problem remains, however. The comparison provides information about whether an object is present at a given location before and after the saccade. It does not provide information about the (surface) features of the object (e.g., Cavanagh, Hunt, Afraz, & Rolfs, 2010). Surface features clearly contribute to visual stability (Tas, 2015; Tas et al., 2012), which argues that the test for transsaccadic object correspondence cannot be accomplished based on the described comparison alone. A potential solution to this problem is provided by TRAM (Schneider, 2013). 

TRAM proposes a mechanism that tests for object correspondence (object continuity) across interruptions of visual input in between fixations (as due to the suppression of input during saccades, e.g., Krock & Moore, 2014) and across changes of visual objects within a fixation. Critically, this test for object correspondence allows us to take into account the (surface) features of objects. Correspondence between objects of successive fixations should be tested on the basis of the objects' attentional weights (Schneider, 2013). The concept of attentional weight comes from Bundesen's (1990) theory of visual attention. The attentional weight of an object indicates its current relevance in a spatially organized fashion. It is computed as the sum of the sensory evidences that the object has certain features, whereby the sensory evidence for each feature is weighted by the current relevance of this feature. According to TRAM, the attentional weight that an object will have after the saccade should be predicted before the saccade is executed. This counteracts the changes of attentional weights due to predictable changes of sensory evidence, for example, due to saccade-induced shifts of retinal images of objects, which change their resolution (cf. Herwig & Schneider, 2014). After the saccade, the predicted attentional weight is compared with the current attentional weight of the object. The test for object correspondence is positive if the two agree and negative if they disagree. Evidence that prediction affects attentional weights has been provided recently (Poth, Petersen, Bundesen, & Schneider, 2014). In this study, participants increased the attentional weight of an object that was monitored for a luminance change in order to compensate for a low predicted (expected) salience of this change. 

Attentional weights are proposed to be implemented in priority maps (Bundesen et al., 2005): spatially organized (retinotopic) brain areas whose neurons seem to code for the relevance and physical salience of objects (Bisley & Goldberg, 2010; Fecteau & Munoz, 2006; Zelinsky & Bisley, 2015). Interestingly, the brain areas supposed to contain priority maps are among the ones whose neurons seem to predictively remap their receptive fields before saccades (e.g., monkeys' lateral intraparietal area, Duhamel et al., 1992; superior colliculus, Walker et al., 1995; frontal eye fields, Umeno & Goldberg, 1997; note that some extrastriate areas show remapping as well, Nakamura & Colby, 2002). Extending TRAM (Schneider, 2013), we therefore hypothesize that predictive remapping contributes to the prediction of attentional weights. 

Now that we have sketched a mechanism using attentional weights to test for transsaccadic object correspondence, we can ask how transsaccadic object correspondence is linked to the object recognition system. An answer to this question may reside in the attentional weights as well. In fact, attentional weights have originally been introduced to explain how relevant objects are selected for object recognition at the expense of irrelevant ones (Bundesen, 1990). To be recognized, an object has to be processed with enough processing resources, which might mean by enough neurons (Bundesen et al., 2005; to eventually enter visual working memory where recognition is complete and report possible). Attentional weights control the allocation of processing resources to objects; each object is assumed to receive processing resources in proportion to its attentional weight relative to the summed attentional weights of all objects in the visual field. More neurons are allocated to relevant than irrelevant objects by virtue of a gating mechanism: Gates in between the lower and higher cortical levels of the ventral object recognition pathway are opened and closed so that the receptive fields of neurons are dynamically remapped to locations of relevant objects (Bundesen et al., 2005; Desimone & Duncan, 1995). Importantly, which gates are opened and which are closed is determined by the attentional weights from spatially organized priority maps (Bundesen et al., 2005). 

Taken together, following TRAM (Schneider, 2013; see also Poth et al., 2015), transsaccadic object correspondence and object recognition should be linked by attentional weights. In this view, postsaccadic object recognition may be impaired because of broken transsaccadic object correspondence for (at least) two reasons. First, a negative test for object correspondence means that an object's predicted attentional weight and its actual postsaccadic attentional weight mismatch. Therefore, after the saccade, there are two discrepant attentional weights present. Postsaccadic object recognition may be impaired because the postsaccadic object is allocated neuronal processing resources according to its attentional weight divided by the sum of the two present attentional weights. In contrast, if the test for object correspondence is positive, this means that the object's predicted and postsaccadic attentional weight match. In this case, there is only one postsaccadic attentional weight. Hence, the postsaccadic object is allocated all available neuronal processing resources (i.e., according to its attentional weight divided by only itself). Compared with the situation of a negative test for object correspondence, the object receives more processing resources, which consequently improves object recognition. In addition, the state of the gates in the ventral object recognition pathway that is dictated by the predicted postsaccadic attentional weight is then the same as the one dictated by the actual postsaccadic attentional weight. This may provide a basis for transsaccadic updating processes (cf. Demeyer et al., 2009; Henderson & Anes, 1994; and transsaccadic integration, Ganmor, Landy, & Simoncelli, 2015; Herwig, 2015; Wolf & Schütz, 2015; but see Wittenberg, Bremmer, & Wachtler, 2008). The lower- or mid-level (surface) features of the postsaccadic object may be routed to presaccadically created object representations through a consistent state of gates (cf. Poth et al., 2015). As a consequence, representations from the presaccadic and postsaccadic retinal images of objects are combined within a common postsaccadic object representation. This leads to the second reason why breaking transsaccadic object correspondence affects postsaccadic object recognition. We suggest that combining presaccadic and postsaccadic representations may provide computational savings: The processing of the object in question that started before the saccade can be continued after the saccade. In contrast, if object correspondence is broken and updating is blocked, then processing of the postsaccadic object might have to start completely anew. As a consequence, object recognition might be delayed, leading to performance decrements (especially when postsaccadic objects are terminated by masks; Poth et al., 2015). 

In sum, extending TRAM (Schneider, 2013), we hypothesize that the dynamic remapping of receptive fields in the ventral stream for object recognition (for reviews, see Bundesen et al., 2005; Desimone & Duncan, 1995) and the predictive remapping of receptive fields in dorsal and frontal areas (for a review, see Wurtz et al., 2011) is accomplished by common attentional weights. In this vein, attentional weights allow us to keep track of objects across saccades, they establish correspondence between presaccadic and postsaccadic objects, and they distribute neuronal resources across these objects for object recognition. 

Conclusion

The present study investigated whether breaking object correspondence across the saccade by changing the surface features of combined color, luminance, and contrast polarity and the surface feature of color impair postsaccadic object recognition. The findings from two experiments indicate that this is the case. As such, they provide new evidence for an interface between mechanisms of transsaccadic object correspondence relying on surface features and mechanisms of object recognition. Based on the TRAM theory (Schneider, 2013), we propose that this interface is provided by visual attention. 

Acknowledgments

This research was supported by the Cluster of Excellence Cognitive Interaction Technology CITEC (EXC 277) at Bielefeld University, which is funded by the German Research Foundation (DFG). We acknowledge support for the Article Processing Charge by the Deutsche Forschungsgemeinschaft and the Open Access Publication Fund of Bielefeld University. 

Commercial relationships: none. 

Corresponding author: Christian H. Poth. 

Email: [e-mail]. 

Address: Neuro-cognitive Psychology, Department of Psychology, Bielefeld University, Bielefeld, Germany. 

Footnotes

¹  Letters and special characters were drawn randomly with a unique random sequence per participant. χ² tests indicated that the presentation frequencies of specific letters (Experiment A: χ²[57] = 65.578, p = 0.204; Experiment B: χ²[57] = 56.006, p = 0.512) and of the combination of special characters and letters (Experiment A: χ²[237] = 238.86, p = 0.454; Experiment B: χ²[237] = 215.84, p = 0.834) did not significantly depend on the cells formed by the two experimental conditions (no-change and change) and the two postsaccadic surface features (green and red in Experiment A and blue and yellow in Experiment B).