Conflicting motion information impairs multiple object tracking

Abstract

People can keep track of target objects as they move among identical distractors using only spatiotemporal information. We investigated whether or not participants use motion information during the moment-to-moment tracking of objects by adding motion to the texture of moving objects. The texture either remained static or moved relative to the object's direction of motion, either in the same direction, the opposite direction, or orthogonal to each object's trajectory. Results showed that, compared to the static texture condition, tracking performance was worse when the texture moved in the opposite direction of the object and better when the texture moved in the same direction as the object. Our results support the conclusion that motion information is used during the moment-to-moment tracking of objects. Motion information may either affect a representation of position or be used to periodically predict the future location of targets.

Introduction

The ability to track multiple objects is extremely useful if we are driving a car through a busy intersection or watching a sporting event, such as basketball. What information are we using to keep track of objects as they move? Knowing the positions of the objects is crucial, but knowing something about their motion may also be useful. The speed and direction of a moving object can be used to predict its future location over short periods of time. To date, research on the human ability to track objects has not provided definitive evidence for the use of motion information during tracking. Here, we address the question of whether or not people use motion information to keep track of multiple moving objects and investigate how the use of motion can inform us about the tracking process. 

Most of the research investigating attentive tracking has used the multiple object tracking (MOT) task (Cavanagh & Alvarez, 2005; Pylyshyn & Storm, 1988; Scholl, 2009), though some recent work examining the use of motion information did not (Fencsik, Klieger, & Horowitz, 2007; Horowitz, Birnkrant, Fencsik, Tran, & Wolfe, 2006; Keane & Pylyshyn, 2006). Instead, this research used the target recovery (TR) task, which is similar to the MOT task in the following ways. At the beginning of a trial, several identical objects appear and a few are cued as targets. After the cues are removed, the objects begin moving independently around the display. Because the objects are identical, participants only have position and motion information available to update the representation of the target objects. At the end of the trial, the participant is asked to indicate which objects were targets. Different from the MOT task, the TR task includes a moment during the object motion in which the objects simultaneously disappear for a brief period. The targets must be recovered after the blank period in order to continue tracking. 

Using the target recovery task, previous work has investigated whether or not motion information is available during tracking. Keane and Pylyshyn (2006) concluded that people do not use motion information to veridically extrapolate the positions of moving objects during the blank, because target recovery performance was better when objects did not move during the blank than when they did move. In contrast, Fencsik et al. (2007) concluded that some motion information is used because target recovery performance was worse when objects did not move before the blank, for a tracking load of two targets but not four. Taken together, these results suggest that motion information may be available during tracking and can aid target recovery by reducing uncertainty for a limited number of targets. Iordanescu, Grabowecky, and Suzuki (2009) provided further evidence to this point by asking participants to track three targets and then select the location of a single target after the objects disappeared at the end of the trial. Participants tended to select the location slightly ahead in the target's trajectory, indicating that they used the target's motion to respond. Although Iordanescu et al. did not manipulate tracking load, Howard and Holcombe (2008) used a similar paradigm and found target localization errors increased with tracking up to seven targets. In sum, all the evidence published thus far that suggests motion information is available during tracking has relied on paradigms where the targets disappear. Although these studies suggest that motion information is available during the tracking period, they do not address whether the information is actually used for the moment-to-moment tracking process of visible objects. 

Measuring the perception of targets after they disappear to examine the types of information used to track targets relies on the assumption that recovering the location of targets that are no longer visible uses the same mechanism as tracking visible targets. Previous work has shown that the visual system may use different mechanisms for processing invisible and visible objects, which opens the possibility that the tracking mechanism may be different for visible than for invisible objects. Research on the oculomotor system has shown that eye movements can be vastly different for invisible objects than for visible objects (Barnes, 2008; Becker & Fuchs, 1985; Ilg, 2008; Steinbach, 1976). Smooth pursuit eye movements are only consistent when visual motion is present or implied by the stimulus. After a pursuit target disappears, the velocity of smooth pursuit eye movements rapidly decreases to 60% of the target velocity (Becker & Fuchs, 1985). In addition, people misperceive the location at which a moving object disappears by extrapolating its motion, a phenomenon called representational momentum (Freyd & Finke, 1984; Hubbard, 2005; Thornton & Hubbard, 2002). In fact, Iordanescu et al. (2009) suggested that their result showing motion influences on target localization after tracking could be explained by representational momentum. It is unclear whether representational momentum occurs for objects that do not disappear. Studies of perceived position of visible stimuli in motion seem to suggest that there are many factors that determine the interaction between location and motion information (Kerzel & Gegenfurtner, 2004; Post, Welch, & Whitney, 2008; Whitney, 2006). Thus, the existing evidence of motion influence on target localization after disappearance may or may not address the question of what information is used for tracking visible objects. 

The question remains, therefore, whether people use the motion information available during the moment-to-moment tracking of visible objects. To investigate this, we asked participants to track textured objects and varied the motion of the texture independently of the motion of the objects. The texture of the objects was either static or moved relative to the object motion. If people only use location information to track multiple objects, as suggested by some previous work (Keane & Pylyshyn, 2006), tracking accuracy should be the same in all texture conditions. However, if people use local motion to track objects, tracking accuracy should vary across conditions. We assume that the motion of the texture will be integrated with the motion of the object's edges producing an estimate of local motion (Lorenceau, 1996; Mingolla, Todd, & Norman, 1992; Qian, Andersen, & Adelson, 1994; Weiss, Simoncelli, & Adelson, 2002). This is the same type of local averaging that explains induced motion (Brosgole, 1968; Duncker, 1929; Johnston, Benton, & McOwan, 1999; Rock, Auster, Schiffman, & Wheeler, 1980). It has also been suggested that the coherent percept of a moving plaid occurs through combining the motion signals of the two individual gratings that make up the plaid (Adelson & Movshon, 1982; Burke, Alais, & Wenderoth, 1994; Derrington & Suero, 1991). Consequently, we predict that texture moving in a different direction than the objects will produce a local estimate of motion that does not match the true object motion and will lead to tracking errors. Furthermore, if people use the direction of motion to predict the future location of targets, tracking accuracy should decrease as direction of the texture motion deviates further from the target's motion. 

Experiment 1

In this experiment, we examined whether or not people use motion information while tracking objects. In order to do this, we asked participants to track textured objects as they moved on a textured background (Flombaum, Scholl, & Pylyshyn, 2008). Because these objects were defined by motion, motion information is critical to the perception of the object and so may be important in the tracking task. To manipulate the motion information available during tracking, the texture of each object either remained static or moved relative to the object's motion (Figure 1). If people use only location information to track objects, tracking accuracy should not be sensitive to the texture motion. If people are using motion to track objects, tracking accuracy will differ across texture conditions. 

Figure 1

 

Illustration of the texture conditions in Experiment 1. The arrows under the square indicate the direction of the object. The object always moved at 1.1°/s. Arrows inside the square indicate the direction of the texture motion. The value inside the square gives the texture speed relative to the object. The speed of the texture motion was adjusted across conditions so that the speed of the texture relative to the background was always 2.2°/s, except for the static condition. The static condition did not have texture motion, so the speed of the texture relative to the background was 1.1°/s.

Methods

Participants

Nineteen participants from Vanderbilt University participated in this experiment (8 males; ages 18–30) following the procedures defined for the protection of human participants by Vanderbilt University and the APA Code (2002). 

Apparatus

The stimuli were generated and presented on an eMac with a built-in 17-in. CRT monitor with a refresh rate of 89 Hz. Participants sat approximately 55 cm from the monitor. Stimuli were produced by Matlab 7.5.0 for OS X version 10.5.5 and the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) version 3.0.8. 

Stimuli

Ten squares filled with random-dot texture were presented in a white frame filled with random-dot texture ( Figure 2). The random-dot texture of the background was regenerated at the beginning of each trial and remained stationary throughout the trial. The frame was approximately 16° × 16° of visual angle (hereinafter °) and each square was approximately 1° × 1°. Green frames approximately 2° × 2° were used to designate targets. The initial positions of the squares were randomly chosen with the constraint that squares did not overlap with each other or the frame's border. Squares traveled in linear paths at approximately 1.1°/s and were permitted to overlap one another during the tracking period. Squares bounced off the frame's borders by reflecting with an added amount of random jitter, between 6 and 11 angular degrees. The direction of motion of the texture within each square varied across trials. The texture either remained static or moved relative to the square's direction of motion ( Figure 1). The texture of each square moved either in the same direction, the opposite direction, or orthogonal to the square's trajectory. Across conditions, the speed of the textures varied relative to the square's motion (1.1°/s, 3.3°/s, 2.2°/s, respectively) so that the texture speed was approximately 2.2°/s relative to the background for all conditions, except the static condition. 

Figure 2

 

Illustration of the multiple object tracking task used in Experiments 1 and 2. Participants were presented with a set of objects and a subset was cued as targets. The cues were removed and the participants tracked the targets as they moved independently around the display. Borders during tracking were motion-defined. White outlines and arrows are shown for illustration only. At the end of the tracking period, the objects stopped moving, the black borders appeared and the participants attempted to select the targets.

Procedure

Each trial began with the presentation of 10 squares with black borders ( Figure 2). The cues, green frames, were presented for 2 s to designate either 3 or 4 squares as targets. The cues and black borders were removed as soon as the squares started to move. A tone was presented to designate the beginning of the tracking interval. The squares moved for 5.6 s. At the end of the trial, the black borders reappeared and the squares remained stationary while participants used the mouse to select the targets. After each selection, the selected square turned blue and a tone was presented giving performance feedback to the participant. After 2 s, the next trial started. Feedback was also shown to participants at the end of each block as their average percent correct for the block. For the practice trials, participants completed one trial of each condition for a total of 8 trials. For the experimental trials, each condition was presented twice in a block and participants completed five blocks for a total of 10 repetitions per condition, or 80 trials. 

Results and discussion

Tracking accuracy was computed as the mean proportion of correctly tracked targets. Analysis of variance (ANOVA) was performed on tracking accuracy with texture condition and tracking load as within-subject variables. Tracking accuracy was reliably higher when tracking 3 targets than when tracking 4 targets, F(1, 18) = 8.98, p = 0.008, η _p ² = 0.333. Note that chance performance is different for different number of targets tracked, higher for 3 targets than for 4 targets (Hulleman, 2005). The interaction between texture condition and tracking load was not significant, F(3, 54) = 1.25, ns, η_p² = 0.065 ( Supplemental Figure 1), so we will focus on the results averaged over tracking load. Although a significant interaction was expected, the null result is difficult to interpret as the experiment may not have used a wide enough range of tracking loads to provide adequate power for a significant interaction. 

Tracking accuracy is plotted against texture condition in Figure 3 (dark gray bars). If participants did not use motion information to track targets, tracking accuracy should have been identical across texture conditions. The results, however, showed that texture motion influenced tracking accuracy, F(3, 54) = 93.11, p = 0.0001, η _p ² = 0.838. A family-wise correction (Bonferroni) with α = 0.0125 was used to correct for the 4 multiple comparisons in this experiment. Compared to Static textures (mean tracking accuracy M = 0.84), tracking accuracy did not differ significantly when the texture moved in the Same direction as the target ( M = 82; t(18) = 1.76, ns, d = 0.034) but was worse when the texture moved in the Opposite direction ( M = 0.60; t(18) = 12.79, p = 0.0001, d = 0.369). The influence of texture motion on tracking suggests that motion is used during multiple object tracking. This is the first evidence demonstrating that the moment-to-moment tracking process uses motion information. It extends previous work that showed motion information can be used to reduce uncertainty in target recovery (Fencsik et al., 2007) and influences target localization after disappearance (Iordanescu et al., 2009) by showing that motion information is not just available but is used for multiple object tracking. 

Figure 3

 

The mean proportion of targets correctly identified for each texture condition in Experiment 1 (dark gray bars) and the same conditions replicated in Experiment 2 (light gray bars). Error bars represent the standard error. The bottom-most arrows indicate the direction of the object and arrows inside the boxes indicate texture direction. Numbers in the squares indicate the speed of the texture motion.

These results also suggest that motion may be used during multiple object tracking to predict future locations of targets. If this is the case, we would expect tracking accuracy to decrease as the direction of texture motion deviates further from the object's direction of motion. Tracking accuracy was better in the Same condition ( M = 0.82) than the Orthogonal condition ( M = 0.74; t(18) = −4.93, p = 0.0001, d = 0.131) and better in the Orthogonal condition than in the Opposite condition ( M = 0.60; t(18) = 8.48, p = 0.0001, d = 0.215). However, if local motion is used to predict target locations we would expect performance to be best in the Static condition because the texture motion is fully consistent with the object motion. When the texture moves in the same direction as the object, performance should be worse than in the Static condition because the local motion would lead to a prediction that is further along the object's trajectory. Our results, however, indicate that performance was not better in the Static condition than in the Same condition. It is possible that static textures do not make a good baseline, because, unlike the other conditions, the texture does not move. Moving texture may make these texture-defined objects more visible and therefore easier to track (Burr & Ross, 1982). 

Experiment 2

Results from Experiment 1 suggested that motion information is used during the tracking of targets. However, the speed of the texture was adjusted for each texture condition to maintain a constant texture speed relative to the background. Consequently, the texture conditions in Experiment 1 differed not only by texture direction but also by texture speed in relation to the object. The Same condition had the slowest texture speed and the Opposite condition had the fastest texture speed ( Figure 1). It is possible that the slower texture motion in the Same condition produced objects that were easier to track than the faster texture motion in the Opposite condition, resulting in the observed differences in tracking accuracy. To determine whether differences in texture speed can account for our results, Experiment 2 manipulated the direction of the texture while holding the speed of the texture constant. 

Another finding in Experiment 1 was that tracking accuracy was the same in the Static and Same conditions, contrary to the expectation that the Static condition would produce better performance. These results could have occurred because the lack of texture motion in the Static condition made objects more difficult to detect than the moving textures in the other conditions. To address this concern, we introduced a condition in Experiment 2 with flickering textures. Because the texture is changing in the Flickering condition, it may make the objects more visible than the Static texture. Finally, we investigated whether tracking accuracy decreases as a function of texture direction, by adding a condition where the texture moved 120° from the object's direction of motion. If texture direction influences tracking accuracy monotonically, performance in the 120° condition should be better than performance in the Opposite (180°) condition but worse than performance in the Orthogonal (90°) condition. 

Methods

Participants

Sixteen Vanderbilt University undergraduates participated (3 males; ages 18–30) and were recruited as in Experiment 1. 

Apparatus and stimuli

The apparatus used in this experiment was the same as in the previous experiment. The stimuli used in this experiment were the same as those used in the previous experiment except 3 targets were tested in all trials and new texture conditions were added. We tested 8 texture conditions: the same four texture conditions as described in the previous experiment (Same, Opposite, Orthogonal, and Static) with the addition of four new conditions (Same-2×, Opposite-2×, 120°, and Flicker). In the Same-2× condition, the texture moved in the same direction as the object but at 2.2°/s, twice the object's speed. In the Opposite-2× condition, the texture moved at 2.2°/s, twice the object's speed, in the opposite direction of the object. In the 120° condition, the texture moved at 2.2°/s, twice the object's speed, 120° from the square's trajectory so that the texture moved orthogonal relative to the background. Finally, in the Flicker condition, the texture was randomly regenerated on every frame so that the texture did not have any coherent motion. 

Procedure

The same procedure was used as in the previous experiment with the exception that every trial contained 3 targets. For the practice trials, participants completed one trial of each condition for a total of 8 trials. For the experimental trials, each condition was presented twice in a block and participants completed five blocks for a total of 10 repetitions per condition, or 80 trials. 

Results and discussion

The average proportion of correctly tracked targets is plotted in Figure 3 (light gray bars) for the conditions that replicated Experiment 1. Analysis of variance (ANOVA) was performed on tracking accuracy with these texture conditions as a within-subject variable. Texture condition had a significant effect on tracking accuracy, F(7, 105) = 29.95, p = 0.0001, η _p ² = 0.666. A family-wise correction with α = 0.025 was used to correct for 2 multiple comparisons for the following comparisons. As in Experiment 1, tracking accuracy compared to Static ( M = 0.84) was not different from performance when the texture moved in the Same direction as the target ( M = 0.82; t(15) < 1, ns, d = 0.030) and was worse when the texture moved in the Opposite direction ( M = 0.53; t(15) = 7.32, p = 0.0001, d = 0.453). These results replicate our finding in Experiment 1 that motion is used in the moment-to-moment tracking of multiple objects. 

The same patterns of results were found when texture speed was 2.2°/s for all conditions. A family-wise correction with α = 0.01 was used to correct for 5 multiple comparisons for the following conditions. Compared to Static ( M = 0.84) tracking accuracy was not different from performance in the Same-2× condition ( M = 0.81; t(15) = 1.7, ns, d = 0.053) and was worse in the Opposite-2× condition ( M = 0.56; t(15) = 8.48, p = 0.0001, d = 0.471). Furthermore, texture speed did not influence tracking accuracy ( Figure 4). Targets in the Same condition ( M = 0.82) were tracked as well as targets in the Same-2× condition ( M = 0.81; t(15) < 1, ns, d = 0.017). Similarly, targets in the Opposite condition ( M = 0.53) were tracked as well as targets in the Opposite-2× condition ( M = 0.56; t(15) = −1.1, ns, d = 0.041). However, tracking accuracy was higher for the textures moving in the same direction ( M = 0.81) than the opposite direction ( M = 0.56; t(15) = 8.97, p = 0.0001, d = 0.369) as the object, even when the speed of the textures remained constant relative to the object. These results suggest that the texture direction but not the texture speed may be used to track targets. 

Figure 4

 

The mean proportion correct plotted for the Flicker, Same, Same-2×, Opposite-2×, and Opposite conditions in Experiment 2. The data from the Same and Opposite conditions are repeated from Figure 3 (light gray bars). Error bars are the standard error. The bottom-most arrows indicate the direction of the object and arrows inside the boxes indicate texture direction. Numbers in the squares indicate the speed of the texture motion.

If motion is used to predict future locations of targets, predictions based on motion of textures moving in the same direction but faster than the object would be erroneous. To determine whether or not same texture motion impaired tracking, we compared flickering textures to textures moving in the same direction as the object. A family-wise correction with α = 0.0167 was used to correct for 3 multiple comparisons for the following conditions. Poorer tracking performance for the Flickering texture ( M = 0.75, data not shown) compared to the Static texture ( M = 0.84; t(15) = −2.50, ns, d = 0.102) ruled out the possibility that making textures dynamic would improve tracking accuracy. Compared to Flicker ( M = 0.75), tracking accuracy was not statistically different in the Same condition ( M = 0.82; t(15) = −2.30, ns, d = 0.087) or the Same-2× condition ( M = 0.81; t(15) = −1.77, ns, d = 0.079). Therefore, it is unlikely that texture moving in the same direction impairs performance. This result is also consistent with the conclusion that texture speed is not used to predict the future location of targets. 

To better describe how texture direction influenced tracking accuracy, we manipulated texture direction across four conditions while maintaining a constant texture speed (Same-2×, Orthogonal, 120°, and Opposite-2×). Tracking accuracy decreased as the texture direction deviated further from the object's direction, even when the speed of the textures remained constant relative to the object, F(3, 45) = 33.5, p = .0001, η _p ² = 0.690 ( Figure 5). This suggests that differences in tracking accuracy are monotonically related to the direction of the texture motion. This result may be due to a process by which texture direction is used to predict the future locations of targets. When the texture moves in the same direction as the target, the predicted location will be along the object's trajectory so the prediction is accurate. When the texture moves in the opposite direction, however, the prediction is further back in the object's trajectory and thus less accurate. We will elaborate on this idea in the General discussion section. 

Figure 5

 

The mean proportion correct for the Same-2×, Orthogonal, 120°, and Opposite-2× conditions in Experiment 2. The data from the Same-2× and Opposite-2× conditions are repeated from Figure 4. Error bars are the standard error. The bottom-most arrows indicate the direction of the object and arrows inside the boxes indicate texture direction. Numbers in the squares indicate the speed of the texture motion.

Experiment 3

Experiments 1 and 2 demonstrated that motion was used for tracking targets that were motion-defined. It is possible that motion is only used to track objects when motion is crucial to the perception of the objects. In order to determine whether our findings would generalize, we asked participants to track 3D-rendered textured spheres rolling across a solid floor plane. These objects were not motion-defined, as the average luminance of the spheres was brighter than the floor plane. Just as in the previous experiments, we manipulated the motion of the texture on the balls. From our previous findings, we expected that tracking accuracy would be higher when the texture moved in the same direction as the ball and worse when the texture moved in the opposite direction of the ball. 

Methods

Participants

Twenty-four participants from Vanderbilt University participated in this experiment (6 males; ages 18–23) and were recruited as in Experiment 1. 

Apparatus

Participants sat 60 cm from the display. A chinrest was used to control viewing distance. The stimuli were generated on a Dell Precision T3400 with an Intel Core 2 Duo 3.15-GHz processor with 3.25-GB RAM using Windows XP. Stimuli were presented on a Dell 1909W monitor with a resolution of 1440 × 900 pixels and a refresh rate of 60 Hz using an NVIDIA Quadro NVS 290 graphics card. Stimuli were generated using software written in Python (using the 3-D graphics software package Blender, www.blender.org). 

Stimuli

Stimuli were 12 white balls with a wavy black line texture ( Figure 6). The balls were presented on a gray rectangular floor depicted at an angle of 20° in the x– y plane on a blue background. The front and back edges of the floor were 22.2° and 37.7° of visual angle, respectively, and the left and right edges were 13.3° of visual angle. The balls had a diameter that ranged from 1.1 to 1.9° depending on their location on the floor so that they appeared to move in a 3-D space. The initial positions of the balls were randomized with the constraint that they appeared on the floor. Balls traveled on linear paths and were permitted to overlap throughout the trial. The speed of the balls varied across trials and was 4, 5, or 6°/s. Balls bounced off the floor's edges by reflecting the ball's direction of motion. The motion of the texture on the balls varied across trials and either remained static or moved relative to the ball's direction of motion. The textures moved either in the same direction or the opposite direction of the ball's trajectory. Texture speed was varied across conditions so that it either moved at the same speed as the ball or twice the speed of the ball. Note that when textures moved in the same direction as the ball at the same speed, the balls appeared to be rolling normally, and when textures moved in the opposite direction, balls appeared to roll with backspin. When textures were static, the balls appeared to float and move without rolling. 

Figure 6

 

Illustration of the cue period in the 3-D rendered multiple object tracking task used in Experiment 3. Red balls indicate targets. During the tracking period, all balls were black and white.

Procedure

Each trial began with the appearance of 12 balls. Three balls were designated as targets by flashing red 5 times over the course of 2 s and then remaining red for 1.5 s. After the targets returned to their original color, all the balls moved for 8 s. At the end of the trial, the balls remained stationary while participants used the mouse to select the targets. After each selection, the selected ball turned red. At the end of the trial, participants were shown in words how many targets they had successfully identified (e.g., “2 out of 3”). Participants pressed the space bar to start the next trial. Each condition was presented once during the practice session for a total of 15 trials. Each condition was presented 10 times for a total of 150 experimental trials. 

Results and discussion

Analysis of variance (ANOVA) was performed on tracking accuracy with texture condition and ball speed as within-subject variables. Texture condition had a significant effect on tracking accuracy, F(4, 92) = 10.18, p = 0.0001, η _p ² = 0.307, as did ball speed, F(2, 46) = 43.28, p = 0.0001, η _p ² = 0.653. However, the interaction between texture condition and ball speed was not significant, F(8, 184) < 1, ns, η _p ² = 0.026 ( Supplemental Figure 2), so we will focus on the results averaged over ball speed ( Figure 7). 

Figure 7

 

The mean proportion correct plotted for each texture condition in Experiment 3. Error bars are the standard error. The bottom-most arrows indicate the direction of the object and arrows inside the circles indicate texture direction. Numbers in the circles indicate the speed of the texture motion relative to the speed of the object, such that 1× is the same speed as the object and 2× is twice the object speed.

A family-wise correction with α = 0.0125 was used to correct for 4 multiple comparisons for the following comparisons. As in Experiment 2, compared to Static ( M = 0.80) tracking accuracy was not different from the performance in the Same condition ( M = 0.81; t(23) < 1, ns, d = 0.018) and was worse in the Opposite condition ( M = 0.73; t(23) = 4.83, p = 0.0001, d = 0.151) when the texture moved at twice the speed of the ball. Similarly, compared to Static ( M = 0.80) tracking accuracy was not different from performance when the texture moved in the Same direction as the target ( M = 0.81; t(23) < 1, ns, d = 0.015) and was worse when texture moved in the Opposite direction ( M = 0.77; t(23) = 2.84, p = 0.009, d = 0.059) when the texture moved at the same speed as the ball. 

As in Experiment 2, texture speed did not influence tracking accuracy, though there was a non-significant trend in the Opposite conditions. A family-wise correction with α = 0.025 was used to correct for 2 multiple comparisons for the following comparisons. Targets in the Same condition ( M = 0.81) were tracked as well as targets in the Same-2× condition ( M = 0.81; t(23) < 1, ns, d = 0.004). Although it did not reach significance when corrected for multiple comparisons, the speed effect for the Opposite conditions was marginal, t(23) = 2.33, p = 0.029, d = 0.082). Targets tended to be tracked better in the Opposite condition when textures moved the same as the object speed ( M = 0.77) than targets in the Opposite-2× condition when textures moved twice the object speed ( M = 0.73). 

This result is slightly different than Experiment 2 where texture speed produced a negligible difference between conditions. It is not clear at this time why this is the case as Experiment 2 differed in many ways. In Experiment 2, for example, the speed of the objects was constant across trials. Participants may have relied on speed expectations built across trials to predict the speed of objects in subsequent trials. In Experiment 3, however, the ball speed varied across trials so participants may not have developed as reliable an expectation of ball speed as in Experiment 2 in which the object speed was constant. Alternatively, it is also possible that motion in depth is more greatly affected by texture speed whereas perception of the planar motion used in Experiment 2 may use a different mechanism (Blakemore & Snowden, 1999; Geesaman & Qian, 1996). It is also possible that the difference in results is a consequence of using luminance-defined objects instead of motion-defined objects. The mechanism for localizing motion-defined objects may be less precise than the mechanism for localizing luminance-defined objects (Banton & Levi, 1993; Regan, 1989). Perception of second-order motion, as with moving motion-defined objects, is impaired relative to luminance-defined objects in the minimum duration and maximum speed paradigms (Derrington, Badcock, & Henning, 1993; Derrington & Cox, 1998; Seiffert & Cavanagh, 1999). Sluggish perception of the motion-defined objects in Experiment 2 may have precluded any effect of speed on tracking accuracy. Regardless, results of Experiment 3 replicated the main effects found in Experiment 2 that tracking accuracy was higher for textures moving in the same direction as the ball than for textures moving in the opposite direction of the ball (F(1, 23) = 30.68, p = 0.0001, η_p² = 0.572). These results provide further support that differences in tracking accuracy are due to differences in the direction of texture motion. 

General discussion

We investigated whether people use motion information during multiple object tracking by manipulating the motion of the texture on moving objects. We found that tracking accuracy was better when textures moved in the same direction as the object than when textures moved in the opposite direction of the object for both motion-defined squares ( Experiments 1 and 2) and textured balls moving in depth ( Experiment 3). Furthermore, we found that tracking accuracy declined as texture direction deviated further from the object's direction. Texture speed did not affect tracking reliably. We conclude that motion information is used during the moment-to-moment tracking of multiple objects. 

Motion information used for prediction in tracking

Our findings are consistent with and extend previous work testing for the use of motion information in multiple object tracking. Keane and Pylyshyn (2006) concluded that motion was not used to predict future locations of targets. Fencsik et al. (2007) extended the findings of Keane and Pylyshyn (2006) and concluded that although motion information may not be used to predict future target locations it can be used to reduce uncertainty when recovering two targets. However, both of these investigations used the target recovery paradigm, in which a blank period occurred during the tracking interval and participants had to recover the targets after the blank. The use of motion information may have been for recovery rather than for the tracking process itself. The sudden disappearance of the objects may serve as a cue to store motion information in memory for use in recovering targets (Horowitz et al., 2006). Similarly, two other recent investigations have documented that the motion of tracked objects influences the objects' perceived location after they disappear (Howard & Holcombe, 2008; Iordanescu et al., 2009). These observations could reflect the well-known phenomenon of representational momentum (Freyd & Finke, 1984; Hubbard, 2005; Iordanescu et al., 2009). In our experiments, however, we have examined the influence of motion during multiple object tracking while objects are continuously visible. Our evidence that motion information affects tracking is not confounded by the disappearance of objects and thus reflects the moment-to-moment mechanism of tracking. 

Our results not only demonstrate that motion information is used for object tracking but also gives insights into how the tracking mechanism uses motion. Tracking performance showed a steady decline as the difference between the texture direction and the object direction increased. This result can be accounted for by a tracking mechanism that integrates the local motion about a target over discrete time intervals to predict possible future locations of the target. For example, a target with a texture moving in the opposite direction would be predicted to move to a location further back than its true trajectory because the object motion would be countered by the texture motion to produce an inaccurate estimate. The same predictive mechanism tracking a target whose texture is moving in the same direction as the object, however, will lead to more accurate predictions, because the local information from the texture is consistent with the object motion. Orthogonal texture motion could produce intermediate accuracy if the predictive mechanisms allowed for a zone of possible target locations that included those of the predicted direction and surrounding regions. This additional flexibility would allow the mechanism to keep track of targets that change direction, such as the objects in these experiments did when they reached the edges of the display. A mechanism that estimates target position by integrating motion information over space and time would be best at tracking objects in the natural environment when texture motion is consistent with object motion. 

There are, however, alternate explanations of our data. One explanation is that tracking is more difficult when the texture motion is inconsistent with the object motion because the visual system is confused by these conflicting motion pairings and finds these objects more difficult to track. This could occur if the texture motion and the object motion are not combined at an early stage, but instead the texture motion and object motion are sustained as two separate signals that relate to the tracked object. As a result, the tracking process receives conflicting motion information that results in decreased tracking accuracy. Another possibility is that the texture motion alters the visibility of the objects being tracked by degrading the objects' borders. This account could explain why the difference between tracking accuracy in the Same and Opposite conditions was smaller in Experiment 3 than Experiment 1, because the luminance-defined edges in Experiment 3 were likely to be more visible than the texture-defined edges in Experiment 1. However, future research will need to examine how texture motion influences object visibility to determine whether our prediction explanation or the visibility explanation should be preferred. 

The idea that motion information is linearly integrated over space and time to create local estimates of direction is well supported by previous work (Lorenceau, 1996; Mingolla et al., 1992; Morrone, Burr, & Vaina, 1995; Qian et al., 1994; van Doorn & Koenderink, 1984; Watamaniuk & Sekuler, 1992; Weiss et al., 2002; Yang & Blake, 1994). Integration within motion channels or motion detectors' receptive fields improves perception by improving the signal-to-noise ratio, thereby strengthening the representation of motion (Anderson & Burr, 1989; Burr, 1981; Tadin & Lappin, 2005; Tadin, Lappin, Gilroy, & Blake, 2003). Similarly, motion integration may also improve motion estimates for predictions in tracking attended targets. Previous work has shown the motion of a single moving dot can be integrated over segmented regions of space, as through occlusions, even in the presence of moving distractors (Verghese, Watamaniuk, McKee, & Grzywacz, 1999; Watamaniuk & McKee, 1995). Some evidence indicates that this integrated motion can be used to predict future motion or locations of stimuli (Watamaniuk, 2005). It is possible that the motion information benefit is instead due to a static representation of the trajectory (motion smear), but some researchers argue that the motion system integrates motion smear as part of the motion signal (Burr, 1980). Other research has demonstrated that visual attention modulates motion integration over a patch of moving dots such that distractors that are separated in space can be completely ignored, while distractors separated only by time may not be (Burr, Baldassi, Morrone, & Verghese, 2009; Burr & Santoro, 2001; Melcher, Crespi, Bruno, & Morrone, 2004). As such, motion of targets may be separable from the motion of distractors during multiple object tracking through directed visual attention. 

For integrated motion information to affect tracking, there must be a discrete time interval over which integration occurs and repeats periodically. All of the previous work on motion integration used detection or discrimination tasks that require one response over a set time interval, such as a trial, for which the motion information is relevant. For motion information to be useful to tracking, however, it must be integrated over shorter time scales than a trial, as the average motion of any one target across a trial is roughly equivalent to distractors and irrelevant to the target identification task. Motion integration over short time scales is beneficial because it would increase accuracy of predicted changes in target location over discrete time intervals. Computer vision algorithms for tracking objects, even as basic as the Kalman filter, use estimates of motion to facilitate tracking (Bar-Shalom, Li, & Kirubarajan, 2001; Grewal & Angus, 1993; Kalman, 1960). To briefly summarize the methods, a noisy estimate of the location of a target is improved across frames by adding in an estimate of velocity and acceleration. Recent advances in computer vision trackers, called multiple frame assignments, have included a longer integration period over which information is held for the purpose of resolving ambiguities and potentially improving past decisions to fit better with current information (Poore & Gadaleta, 2006; Poore, Lu, & Suchomel, 2001). Human object tracking could take advantage of the same benefits by using local motion information integrated over short time scales to better predict future target locations. In fact, local motion estimates are only useful to a process that has a prediction component. Only if position information is modified in the relevant time scale is motion information relevant. That being the case, the current result demonstrating that motion information is used in tracking also indicates that the tracking process calculates information periodically within discrete time intervals. 

An unresolved question is the duration of the time interval over which motion direction is integrated and predictions are made in multiple object tracking. The answer is difficult to estimate because of the number of factors that likely affect the process. Research indicates that motion integration occurs over two different time scales for two different processes. Integration based on simple contrast summation can only occur over short time scales of 100 to 500 ms, while a higher level integration of motion direction can extend to several seconds (Burr, 1981; Burr & Santoro, 2001; Morrone et al., 1995; Watamaniuk & Sekuler, 1992). If we assume that the high-level integration is more likely to be used in tracking, because of its relationship to visual attention, then the integration time could be anything less than several seconds. The optimal integration time would depend on the consistency of object speed and direction and the probability of future changes. While a longer integration time would provide a more reliable estimate for objects that keep the same direction for extended periods, a shorter time scale would be more beneficial to track an object that changes direction often. However, even a consistently moving object would be tracked better with a shorter integration time at the moment that a change in direction was imminent, such as close to the edge of the box. A better strategy would be to flexibly vary the integration time depending on the distance of the object to an edge of the box. Future experiments are needed to measure the integration time used for motion direction to affect tracking and to determine whether or not integration can be flexibly modulated. 

Implications for theories of multiple object tracking

Previous work has provided frameworks to understand the tracking process that must be modified to account for the current results. The two most prominent theories to date are visual index theory and multifocal attention theory. Pylyshyn proposed that a mental representation of location, called a visual index, allows people to continuously reference the particular spatial location of a moving target (also called FINST theory, Pylyshyn, 2001, 2006; Pylyshyn & Storm, 1988). Several visual indexes are perpetuated in parallel to track multiple targets. These visual indexes are pre-attentive and do not contain any information about the target's features. The visual index serves as a reference to a particular spatial location, upon which visual attention and action can be rapidly deployed. In order to account for our finding that motion is used during tracking, this theory must be modified to include direction information. Rather than just a representation of location, a visual index would have to contain direction, perhaps a visual vector. The visual vector would be updated throughout the tracking interval, reflecting both the position and direction of motion of target objects. This modification in some ways violates the spirit of the FINST theory (Pylyshyn, 2001) because visual indexes were proposed to contain no information about object properties but serve as a location reference, like an address. In addition, the idea of an integration period over which motion is combined is also a deviation from the original theory that describes visual indexes as automatically updating with changes in object locations. 

As an alternative to the visual index theory, the multifocal attention theory proposes that objects are tracked with multiple independent foci of attention that allow us to attend to multiple locations simultaneously (Cavanagh & Alvarez, 2005). A control process shifts the foci of attention to keep them centered on the targets as they move. Everything about the target is processed by the focus of attention, such as location, color, and motion. The current results suggest that the motion information from the targets biases the control process used to predict future target locations and guide the foci of attention. A higher level control process might also be capable of flexibly determining the optimal integration time for the most useful motion information and apply it to shifts of attention. If the control process were a limited resource that could not shift all the attentional foci at the same time or took more time to calculate multiple shifts in parallel, then predictions over discrete time intervals would be beneficial. With these additional specifications, the multifocal attention theory of object tracking fits well with the current results. 

The most recent model of object tracking is also consistent with the conclusions that motion information affects a prediction of target location. Oksama and Hyönä (2008) have fully specified a model that tracks objects with unique identities. One stage of the model is a serial control system that selects an object for visual attention that best matches the stored memory representation of a target location. The model also describes that the efficacy of tracking depends on object speed because serial analysis of targets takes time that can lead to a discrepancy between the stored and actual locations of targets (Oksama & Hyönä, 2004, 2008). It also includes a stage that checks identity information in memory with the identity of the attended object to appropriately select targets that have unique identities. To modify this model to include direction information, one could propose that motion direction is part of the identity information that is stored in memory and benefits tracking with a better match to candidate targets. However, such a modification would not account for the current results because tracking was impaired by an inconsistency between the object motion and texture motion, not an inconsistency of the motion across time. For example, storing a target as “an object moving left with texture moving right” would provide a perfect memory match to the next instance of target selection most of the time. Alternatively, one could modify the model so that the control process that selects objects for attention is modulated by motion direction such that the selection is biased toward the direction that the target was last moving. This modification is similar to that proposed for the multifocal attention account of object tracking. Both models, then, are capable of incorporating the current results by adding periodic predictions of motion direction to a control process. 

So far, we have discussed these model modifications in terms of motion direction, without considering speed. Our experiments did not demonstrate a reliable effect of texture speed. Although it is difficult to make conclusions from null results, one could take the insensitivity to speed to suggest that the only motion information used for tracking is direction. This is counterintuitive for the models of object tracking, because predicting a possible future location of a target would benefit from an estimate of speed. A control process that had information about how far to move would be more effective than one that simply had information about direction. It seems more likely that speed is part of the prediction process and our experiments were not sensitive enough to detect its effects. These experiments do not allow us to make definitive conclusions about speed and decide whether or not models of object tracking should include speed estimates. 

Conclusions

Tracking multiple objects as they move randomly among distractors is a challenging task that requires maintenance of information as objects move. These experiments have demonstrated that tracking objects with moving textures is impaired when the texture motion conflicts with the object motion. Results held not only for texture-defined objects, but 3-D rendered, textured spheres rolling on a plane. Motion information affects the moment-to-moment tracking process. We have proposed that these results are most consistent with a tracking process that makes periodic predictions of target locations based on integration of motion over discrete time intervals. Future work is needed to determine how people weight motion and location information and over what time scales integration occurs. 

Supplementary Materials

Supplementary Figure 1 - Supplementary Figure 1 

Supplementary Figure 2 - Supplementary Figure 2 

Acknowledgments

This work was supported by P30EY008126 Core Grant in Vision Research to the Vanderbilt Vision Research Center. We thank Nicole Jardine for assistance with data collection. 

Commercial relationships: none. 

Corresponding author: Rebecca St.Clair. 

Email: [e-mail]. 

Address: Department of Psychology, Vanderbilt University, Wilson Hall, 111 21st Ave South, Nashville, TN 37240, USA.