The maintenance and disambiguation of object representations depend upon feature contrast within and between objects

Abstract

The brain processes many aspects of the visual world separately and in parallel, yet we perceive a unified world populated by objects. In order to create such a “bound” percept, the visual system must construct object-centered representations out of separate features and then maintain the representations across changes in space and time. Here, we examine the role of features themselves in maintaining and disambiguating the representations of the objects to which they belong. In three experiments, we measure how the perceived motion of two objects traversing ambiguous trajectories is affected by the contrast between the features and surrounding fields, by the contrast between features, and by changes to orientation of texture within objects. We report that the maintenance and disambiguation of object representations depend on the contrast of the features relative to their surrounds and on the extent of feature differences between the two objects. These feature dependencies indicate that object representation relies on relative response to many stimulus dimensions.

Introduction

The brain processes many aspects of the visual world separately and in parallel, yet we perceive a unified world, populated by coherent objects (James, 1890; Treisman, 1996). The implication is that the visual system creates our object-centric perceptual world by binding together the output of separate processes (which presumably compute individual visual features such as orientation, texture, color, and motion direction). Two fundamental questions underlying this “binding problem” (Treisman, 1996) can be summarized as follows: (1) How and under what conditions does the brain combine (or fail to combine) separate features into a unified object representation? (2) How are object representations maintained over time and space? Here, we describe a series of experiments designed to investigate these questions by examining the spatiotemporal conditions under which object representations are created and maintained or created and disrupted. Specifically, we investigate the role features play in maintaining the representations of the objects to which they belong. 

Our investigations take advantage of a new variant of a “bouncing–streaming” paradigm (Kanizsa, 1969; Metzger, 1934; Michotte, 1946/1963) that has been used to study properties of motion perception as well as object representations (Bertenthal, Banton, & Bradbury, 1993; Feldman & Tremoulet, 2006; Mitroff, Scholl, & Wynn, 2005; Sekuler & Sekuler, 1999; Watanabe & Shimojo, 1998). We show two objects (in this case, rectangles) that start on opposite ends of the screen and move toward and past each other, traveling from one side of the screen to the other. If the objects are identical, then the stimulus is ambiguous: The objects could be perceived either as passing from one side of the screen to the other (streaming: passing through each other) or as moving toward the middle and returning to the side from which they started (bouncing). If the velocity of the two rectangles is selected to remove occlusion cues, the percept will most commonly be that of bouncing, despite the stimulus being consistent with both percepts (Movie 1). 

 

Movie 1

 

Black rectangles. The image contains two black rectangles, one that moves from right to left and back again, and another that moves from left to right and back again. The display is physically ambiguous: Each rectangle could be interpreted as passing from one side of the screen to the other (i.e., the perception of streaming) or as bouncing off the other rectangle and returning to its point of origin (i.e., the perception of bouncing). When the rectangles are both black and they do not overlap at the intersection, observers perceive the rectangles as bouncing off of each other.

However, if the rectangles are not identical, their distinctive features may provide additional information to disambiguate the motion paths and potentially bias the percept to that of streaming. 

For instance, if one rectangle is red and the other is green, then the colors of the rectangles will, in fact, move from one side of the screen to the other in a manner consistent with streaming. 

In this paper, we examine if, and under what conditions, non-identical rectangles appear to bounce off of each other. A bounce percept would signal that each rectangle had transferred its individualized features to the other rectangle at the point of collision and would reveal dissociable processing of information about the objects' trajectories and information about their defining features. The experiments place two established theories of object representation in opposition. One theory is that the visual system generates reasonable hypotheses about the external world by combining information in the visual scene with prior knowledge about the behavior of objects (Albert & Hoffman, 2000; Anstis & Ramachandran, 1987; Feldman & Tremoulet, 2006; Gregory, 1980, 1997; Hsieh, Caplovitz, & Tse, 2005; Moore, Stephens, & Hein, 2010; von Helmholtz, 1866/1925/1962). According to this view, changing the colors of the rectangles should always lead to the perception of “streaming” instead of “bouncing”—after all, the information representing the colors of the rectangles is readily available, and a reasonable motion trajectory (streaming) can be inferred without needing to invoke an unlikely spontaneous swap of colors. An alternative theory is that features and objects are bound together based on spatiotemporal continuity; in this view, features (such as color) play only a limited (if any) role in the maintenance of object representations (Kahneman, Treisman, & Gibbs, 1992; Mitroff & Alvarez, 2007; Pylyshyn, 1989; for review, see Flombaum, Scholl, & Santos, 2009). According to this hypothesis, the representation of an object at any moment corresponds to whatever object representation is spatially closest to it at the next or previous moment. Once the correspondence has been formed, the features present at the current location (which may or may not match those previously bound to the object) become attributed to the persisting object representation. In this manuscript, we will refer to these two competing theories as the “features are important” and “spatiotemporal continuity” hypotheses, respectively. 

Conflicting support for each of these theories has been derived using a number of stimulus paradigms. One of the paradigms involves measuring the object-specific preview benefit (OSPB), an implicit behavioral measure of object maintenance (Kahneman et al., 1992). In a typical OSPB experiment, an observer views two objects, each of which is identified by a distinct letter that appears briefly on the screen near the object at the beginning of a trial (the object specific preview); the objects are then moved to new locations, and a letter is subsequently flashed near one of the two objects. The observer is typically faster to respond if the letter corresponds to the original preview letter for that object (the benefit). The OSPB is typically thought to arise because of the congruent mapping of the letter with a continuously maintained representation of the object it identifies. 

One set of recent experiments demonstrated that there was no OSPB when the spatiotemporal relationship between two objects was made ambiguous (Mitroff & Alvarez, 2007). This finding was true even when the features of the two objects were different, suggesting that features contribute little or nothing to the maintenance of object representation. On the other hand, recent experiments using a similar paradigm have shown that abrupt changes in the features of an object established by spatiotemporal continuity can abolish the OSPB (Moore et al., 2010). Furthermore, this study found that under certain circumstances an OSPB can be observed on the basis of feature information alone (Moore et al., 2010). These results suggest that features do indeed contribute to the maintenance of object representations. 

Similarly, evidence for and against a spatiotemporal continuity hypothesis can be found in the motion literature. For instance, studies conducted with ambiguous motion displays have shown spatiotemporal proximity (Burt & Sperling, 1981; Navon, 1976), rather than features, to be dominant factors in determining motion perception. However, studies conducted with apparent motion displays have found that features can also contribute to resolving object correspondences (Cavanagh, Arguin, & von Grunau, 1989; Green, 1986, 1989; Watson, 1986), particularly in the case where spatiotemporal correspondences between alternate percepts are equated (Burt & Sperling, 1981; Green & Odom, 1986; Kolers & Pomerantz, 1971; Mack, Klein, Hill, & Palumbo, 1989; Shechter, Hochstein, & Hillman, 1988). Likewise, studies examining the tunnel effect (Burke, 1952; Michotte et al., 1964/1991), in which objects change features while passing behind an occluder, have also demonstrated a primary role for spatiotemporal continuity in determining object correspondences (Burke, 1952; Flombaum, Kundey, Santos, & Scholl, 2004). However, psychophysical work using a modified version of the tunnel effect found feature contributions to the maintenance of object representations behind occluders (Feldman & Tremoulet, 2006). Specifically, when two objects in a bouncing–streaming paradigm passed each other behind an occluder, the perceived object correspondences were biased toward those objects that shared similar features (Feldman & Tremoulet, 2006). For such a straightforward question and wealth of existing data, these conflicting findings indicate that much remains to be known regarding the specific role features play in object maintenance. 

To better understand our motivation for the current set of experiments, we encourage readers to view Movie 2 while maintaining central fixation. In this demonstration, the two rectangles are red and green, as described in the example above. It is easy to observe that changing the color of the rectangles does not automatically produce the perception of streaming. Instead, the two non-identical rectangles appear to bounce off of each other in a manner that is consistent with the simultaneous exchange of features (i.e., colors). ¹ This counterintuitive percept is consistent with the hypothesis that features do not contribute toward maintaining an object's identity. However, the contributions made by the features may simply be insufficient to alter the perceptual outcome in such configurations. It is possible that the visual system may resolve the ambiguity of object identity by combining visual information that includes features such as color (and not just spatiotemporal continuity). In this view, sources of visual information contribute more or less to the final percept depending on their relative strengths (Feldman & Tremoulet, 2006). For instance, the relative strength of the feature cues may be too weak to overcome the strong perceptual bias toward bouncing imposed by sources of information such as spatiotemporal continuity or the lack of occlusion. 

 

Movie 2

 

Bounces with red and green rectangles. Same as Movie 1, except that one rectangle is red and the other is green. When the contrast of the bars relative to the background is high, as is the case here, observers primarily report that the rectangles appear to bounce off of each other.

A general criticism of existing theories of object maintenance is that they do not adequately identify the underlying neural mechanisms that mediate feature contributions and/or spatiotemporal continuity. In this paper, we raise and test the hypothesis that the maintenance and disambiguation of object representations is mediated in part by the weighting of separate motion processing systems that operate on the spatiotemporal correspondence of a wide range of stimulus features, including luminance, color, texture, and even the allocation of attention (e.g., Cavanagh, 1992; Lu & Sperling, 1995, 2001). In three experiments, we apply the version of the bouncing–streaming paradigm described above to systematically increase or decrease the relative strength of the feature differences between the two rectangles. We use the frequency with which observers perceived streaming or bouncing as a means of determining whether and how much features contribute to maintaining object identity. As shown in Movie 2, we find that spatiotemporal continuity can drive the perceptual outcome even for conditions in which the features of the two objects are different, i.e., in this instance, bouncing is perceived. However, in certain circumstances, namely, when the feature differences between the two objects are of greater significance, then the perceptual outcome is driven by feature correspondence rather than spatiotemporal continuity. The results reported here support the hypothesis that object representations are maintained and disambiguated through the processing of multiple motion systems that form spatiotemporal correspondences within specific feature domains. The ultimately perceived object correspondence is derived from the relative weighting and strength of activation of each of these motion systems. 

General methods

Participants

Each participant had normal or corrected-to-normal vision. Prior to the experiments, all participants included in this study gave written informed consent, according to the guidelines of the Department of Psychology and the Internal Review Board of Princeton University. Participants received $8 for each of the experimental sessions they completed. Prior to the experiments, each participant was shown versions of the stimuli corresponding to strong collision and streaming percepts to ensure that they understood the nature of the task they were about to complete. Five people naive to the specific aims of the study participated in each of the three experiments described below for a total of 15 participants. 

Stimulus presentation

All stimuli were generated using MATLAB and the Psychophysics Toolbox (Brainard, 1997). Stimuli were presented on a Sony Trinitron Multiscan G500 with a screen resolution of 1280 × 1024 and with a 90-Hz refresh rate. Participants viewed the stimuli in a darkened room with a viewing distance of 61 cm. Luminance values were measured using a Minolta LS-100 luminance meter. 

Experiment 1

Methods and results

The effect of background luminance on perceived bouncing

As can be observed in Movie 2, when the rectangles differ in color, the percept is often that of collision. Because the luminance contrast of the two bars relative to the background is high (as is the case in Movie 1, where the two bars are identical), we hypothesize that the relative contribution of the color information may be low and thus insufficient to alter the perceptual outcome. Here, we manipulate the brightness of the background, which effectively changes the contrast of the two bars relative to the background, thereby changing the relative contribution of the color information. This approach allows us to manipulate differences between the two bars without explicitly changing the bars themselves. 

Procedure

In each of a total of 140 trials, observers were presented with a pair of rectangular bars, one red (21.7 cd/m²) and one green (44.8 cd/m²), each 2.4° × 0.6° visual angle in size (height × width). As illustrated in Figure 1, the bars began from a position of 5.6° visual angle along the horizontal axis and displaced downward 1.5° from a centrally located fixation point. The bars were vertically displaced so that the fixation point would not lie along the motion trajectories and, thus, could not be used to provide an additional cue, or lack thereof, to occlusion. In each trial, the bars moved with a sinusoidal velocity, starting with a velocity of 0°/s, accelerating to a maximum velocity of 24.0°/s visual angle in the center of the display, and then decelerating back to 0°/s as they crossed to the other side of the display. We controlled the velocity of the bars by varying the spatial step size between successive positional updates. The positions of the bars were updated approximately every 50 ms. Importantly, the speeds and positions were carefully chosen so that at the critical moment where the two bars passed each other, their displacements were such that they precisely exchanged positions, in a single frame refresh, on the screen with an intervening gap, equal to the width of a single bar: 0.6° visual angle (see exchange frames, Figure 1). There were 20 trials each of seven gray background luminance conditions (0.3, 0.8, 1.4, 2.3, 4.1, 6.9, and 8.9 cd/m²), which were presented in a pseudorandom order. The sides on which the red and green bars started were randomly determined on every trial. Observers were required to indicate whether they perceived the bars to stream past each other or bounce off of each other by pressing one of two buttons on a keyboard. 

Figure 1

 

The bouncing–streaming paradigm. Two rectangles travel in opposite directions, from one side of the visual field to the other. The velocity of each rectangle is chosen such that at the critical point of intersection, labeled intersect frames 1 and 2, they exactly swap positions. The stimulus is consistent with two distinct perceptual outcomes: (left) the two rectangles can appear to stream past each other, each maintaining its feature identity; or (right) the two rectangles can appear to bounce off of each other, co-occurring with a spontaneous exchange of their feature identity.

Results

For each subject, we computed the percentage of trials in which bounces were reported for each of the seven background luminance conditions. The data shown in Figure 2 indicate a systematic decrease in the number of reported bounces as the luminance of the background increased. This was confirmed by a repeated measures ANOVA, which revealed a significant main effect of background luminance (F(6,24) = 9.614, p < 0.001, η _p ² = 0.706). A follow-up polynomial contrast computed using log-scaled background luminance values revealed a significant linear relationship between background luminance and perceived collisions (F(1,4) = 14.949, p = 0.019, η _p ² = 0.789). In summary, increasing the background luminance increased the frequency of the streaming percept. This main effect of background luminance can clearly be observed by viewing Movie 3, in which the contrast of the bars relative to the background has been decreased and streaming is most commonly perceived. 

Figure 2

 

Results of Experiment 1. As the luminance of the background was increased, the likelihood of perceiving a bounce decreased. Error bars represent the standard error of the mean. Background luminances are plotted on a log scale.

 

Movie 3

 

Bounces with red and green rectangles: reduced contrast. Same as Movie 2, except the background luminance is gray (i.e., a luminance level intermediate to the luminance of the red and green bars). In this case, observers primarily report the perception of streaming.

Discussion

Movie 2 showed that rectangles of two different colors can appear to bounce of each other even though the trajectory of the colors are consistent with streaming. However, in this first experiment, we showed that the perception of bouncing depends on the luminance of the background relative to the rectangles: The perception can be switched from bouncing to streaming without making any changes to the two rectangles themselves. The features matter hypothesis provides one explanation for why this is. Specifically, at low luminance contrast (when the background is brighter), the contribution of the feature differences between the two rectangles to the maintenance of their representations is increased relative to the contribution of luminance contrast with the background. This observation is also consistent with the idea that changes in the luminance of the background changes the relative strength of activation between motion systems that operate on luminance and those that operate on color. InExperiments 2 and 3, we will continue to use the bouncing–streaming paradigm to examine more specifically whether feature information itself influences the maintenance of an object's representation over changes in space and time. 

Experiment 2

Methods and results

The interaction between background and relative bar contrast

In this experiment, we parametrically manipulate the luminance of the rectangles and of the background to create a range of rectangle/background contrasts. The primary goal of this experiment is to dissociate the roles of luminance contrast and feature differences in maintaining object representation. If the object representation were determined solely by luminance contrast relative to the background, then we would expect that higher contrasts would lead to a greater probability of seeing the bouncing percept. If, however, features do contribute to the maintenance of an object's representation, then we would expect that the greater the feature differences between the two rectangles (i.e., contrast of one rectangle to the other), the more likely the rectangles would be perceived to stream. 

Procedure

The procedures used in this experiment were the same as those used in Experiment 1. To simplify the parametric dimensions of the experiment, the rectangles and background were both achromatic (as compared to the red and green rectangles used in Experiment 1). The luminance values of the two rectangles were selected from the following list of five pairings, ranging from very similar to very different: 27.6 and 32.2, 21.5 and 45.4, 11.4 and 62.7, 5.3 and 75.5, and 2.49 and 90.8 cd/m². In addition, for each luminance pairing, seven different background luminance values were presented: 0.4, 1.7, 4.1, 7.7, 15.5, 26.8, and 31.2 cd/m². Table 1 illustrates the Michelson contrast values for the bars in each of the background luminance conditions. Each rectangle pair condition was paired with every background condition in separate trials, for a total of 35 conditions. The sides on which the two rectangles were presented were randomly determined on every trial. Observers were required to indicate whether they perceived the rectangles to stream past each other or collide with each other by pressing one of two buttons. In each experimental session, 10 trials of each condition were presented in a pseudorandom order. Each subject participated in two experimental sessions, across which the data presented here were combined. 

Table 1

 

The Michelson contrast values are shown for each pair of bars at each of the seven background luminance levels tested in Experiment 2.

Background (cd/m2)	Pair 1		Pair 2		Pair 3		Pair 4		Pair 5
	Bar 1	Bar 2	Bar 1	Bar 2	Bar 1	Bar 2	Bar 1	Bar 2	Bar 1	Bar 2
0.4	0.971	0.975	0.963	0.983	0.932	0.987	0.860	0.989	0.723	0.991
1.7	0.884	0.900	0.853	0.928	0.740	0.947	0.514	0.956	0.189	0.963
4.1	0.741	0.774	0.680	0.834	0.471	0.877	0.128	0.897	−0.244	0.914
7.7	0.564	0.614	0.473	0.710	0.194	0.781	−0.185	0.815	−0.511	0.844
15.5	0.281	0.350	0.162	0.491	−0.152	0.604	−0.490	0.659	−0.723	0.708
26.8	0.015	0.092	−0.110	0.258	−0.403	0.401	−0.670	0.476	−0.830	0.544
31.2	−0.061	0.015	−0.184	0.185	−0.465	0.335	−0.701	0.415	−0.852	0.489

Results

For each subject, we computed the percentage of trials in which collisions were reported for each of the 35 experimental conditions. As shown in Figure 3, the data are plotted as a function of background luminance for each of the rectangle–contrast pairings. A 2-way, 5 × 7 repeated measures ANOVA revealed significant main effects of both rectangle–luminance pairing (F(4,16) = 82.261, p ≪ 0.001, η _p ² = 0.954) and background luminance (F(6,24) = 6.941, p ≪ 0.001, η _p ² = 0.634), as well as a significant interaction between bar luminance and background luminance (F(24,96) = 13.816, p ≪ 0.001, η _p ² = 0.775). 

Figure 3

 

Results of Experiment 2. Each panel illustrates the likelihood of perceiving a bounce as a function of increasing background luminance for each of the five rectangle–contrast pairings. As relative luminance of the two rectangles increased, the likelihood of perceiving a bounce decreased (compare left panels to right panels). The likelihood of perceiving a bounce was also contingent upon the background luminance (center and two left panels). Error bars represent the standard error of the mean. Background luminances are plotted on a log scale.

As can be seen by comparing the left panels of Figure 3 to the right panels, as the relative contrast of the two rectangles increased, the rate of feature exchange decreased. Similarly, as can be seen in three of the five panels of Figure 3, as the background luminance increased, the rate of feature exchange decreased. The interaction between the effects of the two parameters can be observed in three ways. First, when the difference between the two rectangles was substantial (i.e., 5.3 and 75.5; 2.49 and 90.8 cd/m²), feature exchange was seldom reported. Second, feature exchange was also seldom reported when the luminance of the two rectangles had opposite contrast polarities relative to the background (i.e., the luminance of the background was higher than the darker rectangle and lower than the lighter rectangle). Lastly, feature exchange was seldom reported when the luminance of one of the bars was similar to the luminance of the background (e.g., when one of the bars had a luminance of 27.6 cd/m², and the background had a luminance of 26.8 cd/m²). 

Discussion

From this experiment, three main observations can be made about the importance of luminance contrast as a feature for disambiguating object representations. The first is that, as was the case in Experiment 1, reducing the contrast of the rectangles relative to the background significantly reduced the likelihood of the bouncing percept. Importantly, this was true even when the two rectangles had very similar luminance values to each other (i.e., 27.6 and 32.2 cd/m² on a 26.8 cd/m² background). This observation is consistent with the hypothesis that at low luminance contrast, the relative contrast differences between the two rectangles are increased and thus contribute more in determining the perceptual outcome. The second observation is that the sign of the contrast (i.e., lighter or darker than the background) strongly influences whether bouncing or streaming will be perceived. A bounce was rarely perceived when the signs of the contrast of the two rectangles were different. This was true even if the absolute contrast of each rectangle relative to the background was high (i.e., 2.49 and 90.8 cd/m² on a 31.2 cd/m² background). The third observation is that when the contrast signs of the two rectangles are the same (i.e., both rectangles are brighter or darker than the background), increasing the relative luminance contrast between the two rectangles increased the likelihood of the streaming percept. 

All three observations lend themselves to the interpretation that features give useful information regarding the objects' paths. In each case, the degree to which the two rectangles differ in feature space (i.e., relative to each other) influences the degree to which the features themselves contribute to the bouncing or streaming percept. However, in the case of the third observation, another non-mutually exclusive interpretation is possible. Namely, as the feature distance of the two rectangles increases, the contrast of one of the rectangles relative to the background also decreased. It is therefore possible that the resultant increase of the streaming percept arises because the visual system separates the rectangles based on different responses to high and low contrasts. Such an interpretation is consistent, for instance, with contrast influences on the relative strength of first- and second-order motion processing (Ledgeway & Smith, 1994, 1995; Lu & Sperling, 1995, 2001; Morgan & Chubb, 1999). 

Experiment 3

Methods and results

Effects of distance in feature space

Experiment 2 confounded features and contrast: Changing the luminance of one of the bars also changes the contrast of that bar relative to the background. In Experiment 3, we empirically address this issue by examining stimuli that differ in the orientation of the internal texture and/or color. Because the stimuli used here keep the same mean luminance and contrast relative to the background, the division between the objects should not be due to processes (such as first-order and second-order motion mechanisms) that respond differentially to differing levels of contrast. 

Another goal of the experiment is to determine if systematic changes in multiple features lead to systematic influences on the likelihood of the streaming–bouncing percepts. If feature information were used to maintain object representations, then we would expect bounces to be less frequent when the two rectangles differ more in their features. However, if the features were not contributing (i.e., the results of Experiment 2 are being driven by low contrast of one rectangle relative to the background versus the high contrast of the other rectangle), then we would expect that systematic changes in the features would have no impact on the rate of seeing a bounce. 

Procedure

Experiment 3 followed the same basic protocol as Experiment 1, except for the following differences. Instead of each rectangle being of uniform color, the rectangles consisted of an oriented sinusoidal texture (see bottom of Figure 4). On every trial, the texture orientation of one of the rectangles was 45°, while the texture orientation of the other was selected from the following list: 55°, 60°, 65°, 70°, 75°, 80°, 85°, and 90°. In addition, on each trial, the two rectangles could be either different colors (red or green) or the same color (both red or both green). Prior to the start of the experiment, the luminance of the green rectangle was adjusted using a minimal motion technique (Anstis & Cavanagh, 1983), to be subjectively equiluminant to the red rectangle. On average, the physical luminance of the red rectangle was 21.7 cd/m², and the physical luminance of the green rectangle was 19.75 cd/m². The screen background was black (0.33 cd/m²) and did not change across the trials. Therefore, there were 16 total conditions. Over the course of the experiment, 20 trials of each condition were presented in pseudorandom order. As in the other experiments, the sides on which the two rectangles were presented were randomized on every trial. Observers were required to indicate whether they perceived the rectangles to stream past each other or bounce off each other (feature exchange) by pressing one of two buttons on a keyboard. 

Figure 4

 

Results and stimuli of Experiment 3. In Experiment 3, we varied the orientation of sine-wave textures between the two rectangles. On each trial, one of the rectangles (the reference rectangle) always had an orientation texture of 45°, and the other rectangle (the test rectangle) had a texture orientation ranging from 55° to 90°. In addition, the two rectangles were either the same color or different colors. The likelihood of perceiving a bounce is plotted as a function of the texture orientation of the test rectangle. Increasing the angle of separation between the reference and test rectangles decreased the likelihood of perceiving a bounce. Bounces were more likely to be perceived when the two rectangles were the same color rather than different colors. Error bars represent the standard error of the mean.

Results

The percentage of trials in which a bounce was reported was computed for each condition. As shown in the top of Figure 4, increasing the angle difference between the two rectangles decreased the likelihood of perceiving a bounce. This result was true both when the rectangles were the same color and when they were different colors; this result is supported by a 2 × 8 repeated measures ANOVA, which showed a main effect of orientation (F(7,28) = 6.532, p ≪ 0.001, η _p ² = 0.62) and a significant linear relationship (revealed by a follow-up polynomial contrast) between orientation and the rate of reported bouncing (F(1,4) = 9.264, p < 0.038, η _p ² = 0.696). Furthermore, a main effect of color was observed (F(1,4) = 11.775, p < 0.026, η _p ² = 0.746). Specifically, a bounce was more common when the two rectangles were the same color than when they were different colors. There was no significant interaction between color and orientation on the likelihood of reporting a bounce (F(7,28) = 1,517, p > 0.20, η _p ² = 0.275). In summary, bounces were reported less frequently as the distance in feature space between the two rectangles increased. This was true within the dimensions of color and texture orientation. 

Discussion

The orientation of the rectangles had a clear and systematic effect on the likelihood of perceiving a bounce: When the rectangles were similar in orientation, they frequently appeared to bounce, but when the orientations differed, the rectangles frequently appeared to stream. The results support the hypothesis that feature information does indeed contribute to the maintenance and disambiguation of object representations. The results are particularly important because, unlike the changes to the stimuli in Experiment 2, the features in this experiment are not tied to changes in the luminance contrast relative to the background. 

Discussion

We employed a variant of the bouncing–streaming paradigm to investigate the role features play in the maintenance and disambiguation of object representations. The results of our experiments reveal the tenuous relationship between the spatiotemporal maintenance of object representations and the visual features that define the object. On the one hand, we report that two rectangles can be perceived to bounce off of each other even when they have different features (Movie 2). Consistent with object file theory (Kahneman et al., 1992), this indicates that object representations are maintained in part by using non-feature-specific information (i.e., spatiotemporal continuity). We summarize this theoretical framework with the expression “Determine the object's direction of motion, and the features will follow”: that is, if two identical rectangles look as if they are bouncing off each other instead of passing through each other, then changing the color of one of the rectangles will not change the perception of bounce. 

Object file theory (Kahneman et al., 1992; see also Pylyshyn, 1989) has long provided a framework for understanding the nature of object representations. The central idea is that visual information that co-occurs in space and time gets bound together into an “object file”—a visual representation of a distinct object to which the co-occurring visual information belongs. In essence, the object file represents the object and contains within it the feature information associated with that object. According to this theory, once established, the object file is indexed or tracked via spatiotemporal information (Kahneman et al., 1992). Thus, if an object is moving through the visual field, so long as there is a sufficient proximal match of the object's location from one moment to the next, the object file will be maintained. This spatiotemporal correspondence would hold even if the feature information present at the updated location were different from the original contents of the object file (i.e., the features do not contribute to the maintenance of the object file; Kahneman et al., 1992). 

On the other hand, we have empirically demonstrated that this theoretical framework appears to have limitations. When the features of the rectangles differ substantially or when the rectangles are different in contrast relative to the backgrounds, the rectangles were more likely to be seen as streaming. This aspect of the results is, therefore, consistent with recent psychophysical findings (Feldman & Tremoulet, 2006; Moore et al., 2010) that suggest that feature-specific and contextual information is used in maintaining an object's representation. Moreover, the greater the differences between the features of two objects, the more the features contribute when a perceptual ambiguity arises. Bayesian-based hypotheses could, in principle, account for switches between bouncing and streaming as a function of feature difference. Feldman and Tremoulet (2006) proposed that the maintenance of object representations is based on the most plausible feature correspondence given an observer's subjective expectations about how objects are likely to change over time. For example, due to changes in illumination, the reflected color or luminance of an object may be expected to subtly change over time, whereas dramatic, spontaneous shape or identity changes may be less likely to occur. According to this view, spatiotemporal continuity would represent only one, albeit a heavily weighted one, of many possible Bayesian priors. 

The fact that a bounce can ever be perceived between two distinct rectangles, however, argues against the strongest form of Feldman and Tremoulet’s Bayesian hypothesis. Unlike Feldman and Tremoulet's paradigm, the stimuli used here explicitly showed the point of intersection. Here, when bars with different features are seen to bounce, the features themselves must become unbound from their original objects and rebound to their new ones. This spontaneous, “right-before-your-eyes” exchange of features is not something likely to be encountered in everyday viewing. Therefore, while it is possible to interpret our data in terms of Bayesian priors, the examples of different-feature bouncing that we report seem unlikely and unexpected, particularly in light of a highly plausible feature correspondence that does not involve such dynamic unbinding and rebinding (Movie 2). 

In fact, we find that bouncing percepts can occur between objects that seem so dissimilar that it belies any reasonable expectations about the way objects change. Figure 5 (which illustrates Movie 4) shows that bouncing percepts can occur when the rectangles differ along multiple feature dimensions, including color, luminance, and texture. In this example, the textures of the rectangles are made from images of stones, seeds, or oceans. The horizontal positions of the rectangles shown in Movie 3 are determined by sine-wave functions that differ from each other only in their temporal phase. The rectangles, therefore, move from left to right across the screen in a smooth, continuous path; however, at some intersections, the rectangles appear to bounce off of each other and reverse direction, co-occurring with an unlikely spontaneous exchange of features. The bouncing percept is even more compelling in this instance because the rectangles do not appear to bounce off of each other when they overlap significantly, thus giving the display “a ball in a pinball machine” appearance. This demonstration is similar in nature to the “hopping” man, described by Kanizsa (1969), and illustrates a perceptual outcome consistent with a highly unexpected and seldom experienced situation. These stimuli clearly demonstrate that even in a situation where a perceptual outcome wholly consistent with our past experience is possible (i.e., a person walking or two rectangles passing by each other), the perceptual outcome can be different, corresponding to a less experienced situation (Kanizsa, 1969). 

Figure 5

 

Bounces can be observed between objects with very distinct textures. Bounces can be perceived even when the rectangles differ on multiple feature dimensions, such as color, luminance, and texture.

 

Movie 4

 

Bounces can be observed between objects with very distinct textures. Bounces can also occur when the rectangles are complex in texture. In this demonstration, the textures of the rectangles are made from images of stones, seeds, or oceans. The horizontal positions of the rectangles are determined by sine-wave functions that differ from each other only in their temporal phase. The rectangles, therefore, move from left to right across the screen in a smooth continuous path; however, at some intersections, the rectangles appear to bounce off of each other and reverse direction. The rectangles do not appear to bounce off of each other when they overlap; bouncing appears to occur only at some intersections, thus giving the rectangles the appearance of “a ball in a pinball machine.”

One parsimonious hypothesis that reconciles these observations with the Bayesian model proposed by Feldman and Tremoulet (2006) is that prior knowledge is instantiated at the level of feature processing and not at the level of the object identity. For example, while the rectangular stimuli used in Movies 2 and 3 may be dramatically different at the level of object identity (i.e., they differ along many feature dimensions), they may still be similar at the level of each individual feature (i.e., luminance contrast, size, overall shape). The disambiguation of the objects in these stimuli shows little dependence on knowledge of the objects themselves. In contrast, distinct differences at the level of features do seem to contribute to the disambiguation. For instance, in Experiment 3, the orientation of the internal portion of the rectangles determines the probability of the streaming or bouncing percepts (a similar example can be generated with lengths of the rectangles—when the rectangles are close to the same size, bounces are more likely, but when the size of the rectangles differs by a large amount, streaming is observed more often). To use Feldman and Tremoulet's example, we predict that a bird and a box could be perceived to bounce if certain underlying features, such as luminance, size, and contrast, were not too dissimilar. 

Such a proposed hypothesis, however, leads to questions concerning what constitutes a feature. The results presented here suggest that contextual information plays a distinct role in determining the nature of an object's features. For example, we find that the likelihood of streaming versus bouncing is determined not by the overall luminance of an object but rather by its contrast relative to the background as well as by its contrast relative to the other object. This dissociation of absolute luminance versus relative contrast information is consistent with observations drawn from “contrast asynchrony” displays (Shapiro, 2008; Shapiro, Charles, & Shear-Heyman, 2005; Shapiro et al., 2004). The basic contrast asynchrony display consists of two physically identical disks sitting side by side and modulating between light and dark; one disk is on a black background, and the other is on a white background. In this configuration, the luminance of the disks modulates in phase, but the contrast of each disk relative to its surround modulates in antiphase. Under certain conditions, observers will report the paradoxical perception that the disks modulate in antiphase but get light and dark at the same time. The implication is that the visual system responds separately to luminance and luminance contrast. In the context of the present study, the contrast asynchrony paradigm suggests that the contrast between an object and its background in this experiment is a “feature” in and of itself and not just a contextual cue that modifies the object. Therefore, when we change the relationship between the bars and the background, we are not simply changing the salience of the feature. We are manipulating a feature in itself. 

Complexity at the levels of features also creates complexity for the idea of spatiotemporal continuity, the theories for which are surprisingly vague if not circular about the answer to the question—the spatiotemporal continuity of what? Models based on the idea of an object file (Flombaum et al., 2009; Kahneman et al., 1992) assert that the spatiotemporal continuity of the object representation itself determines spatiotemporal object correspondence (namely, the object at time B corresponds to the most proximal object at time A). Our data suggest that this is likely not the case. In our displays, the spatiotemporally continuous object correspondences are always consistent with the bouncing percept, something that, depending on the stimulus, is not always perceived. Rather than operating at the level of the object, we propose that spatiotemporal continuity is being implemented at the level of the feature (however such a notion is defined). Importantly, as our displays make evident, different features may have different spatiotemporal correspondences. For example, in Experiment 3, when the two rectangles are the same color, the spatiotemporal continuity of surface texture (orientation) is consistent with streaming, whereas the spatiotemporal continuity of color is consistent with bouncing. The resultant object representations appear to reflect the relative weighting and stimulus strengths along the different feature dimensions (similar in principle to Feldman & Tremoulet, 2006). 

How might such relative weightings be neuronally instantiated? One hypothesis is that they are instantiated through the processing of motion information. It is widely held that motion perception is mediated by multiple processes that operate on different sources of visual information (for example, Cavanagh & Anstis, 1991; Dobkins & Albright, 1993a, 1993b, 1994, 1995; Ledgeway & Smith, 1994, 1995; Lu & Sperling, 1995, 2001). Mechanisms that operate on spatiotemporal changes of luminance (1st-order motion energy), mediated in large part by the magnocellular LGN–V1–MT (Born & Bradley, 2005), are generally considered the primary motion system. The general primacy of this motion system can account not only for the effects of luminance and luminance contrast observed in Experiment 2 but also for many previous findings cited as support for object-level spatiotemporal continuity hypotheses. For example, in Experiment 2, we find that the sign of contrast (lighter or darker than the background) greatly influences whether a streaming or bouncing percept will be observed. Specifically, when the two rectangles are of opposite signs, bouncing percepts are seldom if ever reported. In these cases, the motion signals derived from the 1st-order motion system are wholly consistent with the streaming percept. 

In addition to the primary luminance-defined motion system, there are secondary motion systems that operate, for example, on spatiotemporal changes of color, texture, or even the spatial allocation of attention (Cavanagh, 1992; Cavanagh & Anstis, 1991; Dobkins & Albright, 1993a, 1993b, 1994, 1995; Ledgeway & Smith, 1994, 1995; Lu & Sperling, 1995, 2001). The seemingly small contributions of features to the maintenance of object representations may simply reflect the relatively lesser strength of these motion systems in comparison to the system that processes luminance. Going back to the results of Experiment 2 when the rectangles had opposite contrasts, whereas the 1st-order motion processing system will signal a streaming percept, the signals derived from the 2nd-order motion system that can operate on rectified contrast (Lu & Sperling, 1995, 2001) is wholly consistent with the bouncing percept. The fact that bounces are seldom reported in these conditions likely reflects the relative strengths of these two motion processing systems. 

Our displays reveal that changing the stimuli to bias the relative strengths of the signals these motion systems produce significantly alters the perceptual outcome. The resultant object correspondences are consistent with the spatiotemporal continuity of the feature that produces the strongest or least ambiguous motion signal. For instance, the perception of streaming in Experiment 2 that arises when one rectangle is of low contrast relative to the background and the other is of high contrast relative to the background is consistent with motion energy models that suggest that 1st- and 2nd-order motion responses differ at the level of contrast (Lu & Sperling, 1995, 2001). Similarly, increasing the luminance of the background in Experiment 1 decreases the relative strength of 1st-order motion signals, biasing the percept to be mediated by the spatiotemporal correspondence of color, thereby leading to a streaming percept. 

Intriguingly, motion mechanisms that allow for the tracking of attention provide a flexible, top-down means for reweighting the various sources of information that contribute to object maintenance. If, for example, in Movie 2 the observer attends to the color of one of the rectangles as opposed to attending in the middle of the screen, it is likely that he or she will perceive streaming rather than bouncing. Conceivably, this attentive tracking could allow for a variety of sources of information to contribute to the maintenance of an object representation, from texture (Ben-Shahar, Scholl, & Zucker, 2007) to potentially high-level semantic (attend to the car) or even object identity-specific (attend to the red car located over there) sources of information. 

Conclusion

Here, we sought to address the following question: how is the representation of an object maintained over space and time? Using an ambiguous motion paradigm, we find that in addition to spatiotemporal correspondence, objects are maintained in part by using information about the features that define them. By demonstrating perceptual outcomes highly inconsistent with “likely” physical world situations, our observations put constraints upon Bayesian models of object correspondence and suggest that any such probabilistic correspondences are being derived at early stages of feature processing likely embodied within the motion processing system and not at the level of object identity. 

Acknowledgments

We would like to thank Sabine Kastner for her support of this project and Sherri Geller for her helpful comments during the preparation of this manuscript. This project was supported by a grant from the National Eye Institute (R15EY021008) to AGS. 

Commercial relationship: none. 

Corresponding author: Gideon P. Caplovitz. 

Email: [e-mail]. 

Address: Department of Psychology, University of Nevada Reno, 1664 N. Virginia Street, Reno, NV 89557, USA. 

Footnote

Footnotes

1  Kanizsa (1969) describes a similar phenomenon, in which the legs of a person swing back and forth. At certain oscillation speeds, the legs can appear to bounce off each other. It is observed that placing a white disk on the “shoe” at the end of one of the legs will not influence the percept: The two legs still appear to bounce off each other. The white disk thus appears to jump from the “shoe” of one leg to that of the other. In this case, it is not clear whether the disk should be considered a “feature” since the disk could be perceptually interpreted as a separate object.