Severe loss of positional information when detecting deviations in multiple trajectories

Abstract

Human observers can simultaneously track up to five targets in motion (Z. W. Pylyshyn & R. W. Storm, 1988). We examined the precision for detecting deviations in linear trajectories by measuring deviation thresholds as a function of the number of trajectories (T). When all trajectories in the stimulus undergo the same deviation, thresholds are uninfluenced by T for T <= 10. When only one of the trajectories undergoes a deviation, thresholds rise steeply as T is increased [e.g., 3.3° (T = 1), 12.3° (T = 2), 32.9° (T = 4) for one observer]; observers are unable to simultaneously process more than one trajectory in our threshold-measuring paradigm. When the deviating trajectory is cued (e.g., using a different color), varying T has little influence on deviation threshold. The use of a different color for each trajectory does not facilitate deviation detection. Our current data suggest that for deviations that have low discriminability (i.e., close to threshold) the number of trajectories that can be monitored effectively is close to one. In contrast, when the stimuli containing highly discriminable (i.e., substantially suprathreshold) deviations are used, as many as three or four trajectories can be simultaneously monitored (S. P. Tripathy, 2003). Our results highlight a severe loss of positional information when attempting to track multiple objects, particularly in a threshold paradigm.

Introduction

Many of our daily activities involve tracking objects and detecting deviations in the trajectories of moving objects. Even the simple act of crossing a street can involve tracking many cars simultaneously and detecting deviations in their paths. In sporting activities, the ability to accurately determine the trajectory of a swerving football, or the direction and height of a spinning tennis ball while simultaneously observing the movements of the opponent(s), can make the difference between winning and losing. For an air-traffic controller staring at an information-rich screen, failure to detect a deviation in the trajectory of one of the dots on the screen can result in tragic consequences. Given the fundamental importance of detecting deviations in tracked objects in our everyday lives, it is important to ask how sensitive we are at detecting such deviations and, in particular, how this sensitivity changes as the number of trajectories presented is increased. Somewhat surprisingly, these issues have received little attention in the literature. When detecting a deviation in a single trajectory (target trajectory) in the presence of additional undeviating trajectories (distracter trajectories), there are at least two factors that could potentially limit the ability of the subject to perform this task. First, it is possible that the distracters may interfere with the target trajectory so that even if the subject knows in advance which trajectory is to deviate, the judgment could be affected because of the presence of the distractors. Second, in cases where the subject is unaware which trajectory is to undergo deviation, the presence of the distractor trajectories may have an effect on the allocation of the attentional resources or resources of working memory required to perform the task. Thus, an examination of the effects of increasing the number of distracter trajectories on the sensitivity for detecting deviations in the target trajectory may help us to understand some of the strategies used by the visual system for tracking multiple trajectories. Although the ability to track multiple linear trajectories has not been examined, previous work has indicated that up to five items in motion can be simultaneously tracked. Pylyshyn and Storm (1988) employed a stimulus consisting of 10 identical crosses moving in random directions. In an initial stationary phase lasting 10 s, a subset of 1 to 5 of the crosses were flashed to identify them as “target” items. Following this was an animation phase during which all items started to move in random directions and observers were required to track the target items for 7 to 15 s. During the animation phase, white solid squares were flashed at random intervals over the moving items, and the observers were required to respond to flashes on the target items as quickly as possible, ignoring flashes on nontarget/distracter items. The results from these experiments showed that observers could reliably track up to 5 items, with target-flashes correctly identified on about 85% of the trials. Subsequent experiments by Yantis (1992) suggested that observers group the many target elements into one coherent nonrigid virtual object and attend to its deformations, rather than attentively tracking the target elements individually in parallel. However, more recent experiments found reaction time to events on distractor items that lay within the region encompassed by the target items to be similar to reaction time to events on distractor items outside this region, arguing against the hypothesis that attention is directed at a single virtual object when tracking multiple items (Sears & Pylyshyn, 2000). 

Although Pylyshyn and Storm (1988) showed that multiple moving targets could be simultaneously tracked, the detection of a deviation in a linear trajectory requires memory for the previous direction of motion and an ability to compare (consciously or unconsciously) current direction of motion with the direction in memory. In the Pylyshyn and Storm experiments, at any instant of time, observers were required to know only the current positions of target items, and the previous positions of the target items, beyond the information required to update the target positions, were largely irrelevant. Our primary goal in this study was to examine the extent to which multiple-object tracking (MOT) is possible when the histories of the object-paths are more relevant to the task performed by the observer. We measured thresholds for detecting deviations in linear trajectories while varying the number of trajectories; detecting deviations in trajectories requires integrating information over substantial portions of the trajectories. Our choice of task was motivated by previous findings that when tracking multiple objects, spatio-temporal properties, such as location and direction of motion, are more reliably coded than featural properties, such as color and shape (Scholl, 2001, p. 23). The first experiment examined the changes in deviation thresholds for a single trajectory when we changed stimulus parameters. The next two experiments measured how deviation thresholds changed with increasing number of trajectories. Thresholds remained relatively unchanged with the number of trajectories when all the trajectories underwent the same deviation (Experiment 2), but dramatically increased when only one of the trajectories deviated (Experiment 3). To ensure that this increase did not result from an inappropriate choice of target speed, Experiment 4 verified the effect of the number of trajectories on deviation thresholds for a range of speeds. To investigate the role of attention, Experiment 5 measured deviation thresholds when the target trajectory was cued using two separate techniques. Experiment 6 measured deviation thresholds when different colors identified the different trajectories. Our main finding is that when a threshold paradigm is employed, deviation thresholds increase rapidly when the number of trajectories is increased, suggesting that observers can accurately process only one trajectory in this paradigm. However, when deviations are much larger than threshold, observers can simultaneously process as many as three to four trajectories very accurately, or a larger number of trajectories with lower accuracy (Tripathy, 2003). 

General methods

Apparatus

The stimuli were generated on a Gateway 2000 computer and displayed on a Vivitron 1776 monitor. The monitor had a frequency of 60 Hz, yielding a frame duration of 16.67 ms. The portion of the monitor screen that was used for displaying the stimulus was 798×574 pixels. The viewing distance was 1.29 m. At this distance, each pixel subtended 1′ in the horizontal and vertical directions. Chin and forehead rests were used to minimize head movements during the experiment. Experiments 1, 2, and 3 were conducted with the room lit dimly by a 60-W angle-poise lamp directed away from the observer and facing a black screen. Subsequent experiments were conducted with the room lit normally using two fluorescent lights. The brighter lighting ensured that observers could not utilize the persistence of the trajectory trails on the screen for making their responses. 

Stimulus and procedure

The stimulus consisted of one or more dots undergoing apparent motion, each dot moving along either a linear or a bilinear (i.e., having two linear segments) trajectory. 

First, consider a stimulus consisting of a single dot/trajectory, a schematic of which is shown in Figure 1 (upper panel). The dot/trajectory always moved from left to right. In Experiments 1, 2, and 3, the trajectory (before deviation) was, on average, oriented at 45°, with a random jitter added to the orientation. In later experiments, the trajectory was oriented, on average, horizontally with a large random jitter added to the orientation. The amount of jitter added is described in each experiment. Two vertical markers indicated the mid-line of the screen. When the moving dot was aligned with the vertical markers, the dot was exactly halfway along its trajectory. The observers were aware that any deviation in the trajectory occurred when the dot was exactly halfway along its trajectory (i.e., all deviations occurred at the vertical mid-line of the screen). The deviations were either clockwise or counter-clockwise (CCW). Following each trial, the observers reported the direction of the perceived deviation, clock-wise or CCW, by pressing appropriate keys on the computer keyboard. All deviations were measured as angles in the plane of the monitor, with the convention that CCW deviations were considered to be positive (in Figure 1, the deviation shown is approximately −10°). The average speed of the trajectory was controlled by varying the dot displacement between frames. The length of the trajectory was varied by varying the number of frames for which the stimulus was presented, or by varying the displacement between frames. Noise introduced into the stimulus on account of screen pixellation is investigated in the Discussion of Experiment 1. 

Figure 1

 

General methods. Upper panel: Schematic of the stimulus used in Experiment 1. The stimulus was a single dot moving along a bilinear trajectory, with a deviation at the mid-point of the trajectory. The horizontal position of the deviation was indicated by the two markers. The deviation was either clockwise (as in figure) or counter-clockwise. In Experiment 1, the trajectory was always from lower left to upper right. The starting orientation of the trajectory had a small random jitter added to it on each trial; the vertical position of the deviation was similarly jittered between trials. Lower panel: estimation of threshold deviation from the raw data. A cumulative normal function was fitted to the percentage of counter-clockwise responses data plotted against the trajectory deviation. Positive values of deviations correspond to counter-clockwise deviations. Threshold was estimated at d’ = 1. The data shown are for observer ST when the stimuli had 17 frames and the dot-speed was 16°/s (see lower panel of Figure 2, black triangle at the corresponding speed). See text for additional details.

When the stimulus consisted of more than one moving dot, all dots traveled with the same average speed and reached the mid-line of the screen at the same instant. In some experiments, all of the trajectories underwent deviation at the mid-line, whereas in others only one trajectory underwent deviation. During an experiment, the observers were aware whether all trajectories would deviate, or only one trajectory would deviate. Observers reported the perceived direction of deviation (clockwise or CCW) on each trial. The random jitter that was added to the orientation of each trajectory ensured that the trajectories were not parallel. Thus, deviation from parallelism could not be used as a cue for the direction of deviation. Further, the orientation of the trajectory after deviation, by itself, could not indicate the direction of deviation; the orientations before and after deviation would have to be compared to determine direction of deviation. When more than one trajectory was present on a trial, the different trajectories were permitted to intersect. However, the starting points and orientations of the trajectories were selected so that when the trajectories reached the mid-line of the monitor (where the deviation[s] occurred), they were separated from each other by an average separation (± a smaller random jitter). At the mid-line, the average inter-trajectory separation on a trial was 10′, 40′, or 90′ in the different experiments; the average separations and the jitter are described with the individual experiments. The separation at the instant of deviation(s) ensured that intersections among the trajectories did not mask their deviation(s). 

The background luminance was 0.1 cd/m² when the room was dimly lit (Experiments 1–3) and 5.3 cd/m² when the room was lit with standard fluorescent lighting (Experiments 4–6). In the experiments that did not involve colored stimuli (i.e., Experiments 1–4, part of Experiment 5) dot luminance was 69.9 cd/m², as measured from a large rectangular region on the screen, having the same intensity as the stimulus dots and presented for an extended duration. All dot luminances reported here and in the rest of this work were measured with the room in the dimly lit condition described previously. For the experiments involving different colored dots/trajectories, the chromaticity coordinates and luminances are as described in Experiment 5. Dot size was 3′×3′ for Experiments 1–3 and 5′×5′ for the remaining experiments. Other stimulus parameters are described with the individual experiments. 

Chin and forehead rests were used; observers were not instructed to fixate any particular point on the screen. Instead, during the practice sessions, observers were encouraged to experiment with performing the task both with moving their eyes to pursue the trajectories and without moving their eyes to determine the eye-movement strategy that yielded the best performance. Once they had decided on a strategy, they were encouraged to use the same strategy during the collection of the data. We were interested in their best performance for detecting deviations, irrespective of eye movements. 

Performance for detecting deviations was measured using a threshold paradigm. Within an experimental block, the only parameter that was varied was the angle of deviation of the trajectory/trajectories. A method of constant stimuli (MOCS) was used, with each block containing nine uniformly spaced levels of deviation: four clockwise, four CCW, and one without any deviation. The spacing of the deviations was such that the observer’s response covered a large portion of the permitted range of the psychometric function (see Analysis). Following each trial, appropriate beeps from the computer provided feedback as to the correctness of the observer’s response. On trials where the target trajectory did not deviate, the computer signaled an incorrect response with a probability of 0.5. Within each block there were 20 trials at each level of deviation. Four blocks were run for each condition, yielding a total of 720 trials. From the observer’s responses, thresholds were estimated as described in the next section. 

The authors (ST and BB) acted as observers. Three other observers (DB, SS, and SN) also participated. Only SN was naïve with regard to the purpose of the experiment. Observers BB, DB, SS, and ST had normal or corrected-to-normal vision and performed the experiment binocularly. Observer SN, who was amblyopic in his left eye, performed the experiment under monocular conditions with his right eye. At the time of starting the experiments, BB and ST were experienced psychophysical observers, whereas the others had little experience as observers in psychophysical experiments. 

Analysis

The observer’s responses for each experimental condition were plotted as shown on the lower panel of Figure 1, with the angle of deviation plotted on the abscissa and with the percentage of trials on which the observer reported the deviation to be CCW for each level of deviation plotted along the ordinate. A cumulative normal function was fitted to the data, with the upper and lower asymptotes of the fit being fixed at 100% and 0%, respectively. The SD of the normal fit was taken to be the empirical deviation threshold. In traditional psychophysics, for the two-response classification task used here, under the assumptions of a cumulative normal response distribution and constant variance noise, this would correspond to a discriminability of 1.0 (Macmillan & Creelman, 1991, pp. 212–218). This can be seen in the lower panel of Figure 1 as the deviation yielding a d′ = 1.0, compared to the null stimulus, after correcting for response bias. In the absence of bias, a positive (negative) deviation of this size would correspond to a “counter-clockwise” response on 84% (16%) of the trials. Our data, when plotted along linear coordinates, were well described by cumulative normal functions (e.g., lower panel of Figure 1), which would correspond to a linear function, if the ordinate were plotted using z-coordinates. 

Experiment 1: Detecting deviations in single trajectories

The goals of this experiment were (i) to determine how well deviations can be detected in single-trajectory stimuli, and (ii) to establish a set of parameters for which performance was close to optimal, and which could be used for studying stimuli with multiple trajectories. The parameters of interest were the number of frames for which the stimuli were presented and the average speed of the moving dots. 

Stimulus and methods

The stimulus consisted of a single trajectory on each trial. The initial portion of the trajectory was oriented at 45° to the horizontal on average, with a uniform random jitter of up to ±3.5° added to the orientation (i.e., the orientation could vary between 41.5° and 48.5°). In one experimental condition, the average dot displacement between frames was kept fixed at 20′ (corresponding to a speed of 20°/s), and the number of frames was varied between 3 and 39 (corresponding to trajectory lengths varying between 0.67° and 12.67°, measured from the center of the first dot to the center of the last dot, along the trajectory). In another experimental condition, the number of frames was kept fixed at 9 for observer DB and 17 for ST, and the displacement between frames was varied from 1′ to 64′ (corresponding to velocities between 1 and 64°/s) for DB and from 1′ to 45′ for ST (corresponding to velocities of 1 to 45°/s). 

Some of the longest trajectories used in this experiment were clipped (i.e., the two ends of the trajectory extended beyond the margins of the monitor). In the first experimental condition, when the number of frames was 39, the number of frames actually displayed on a trial was between 35 and 39, depending on the amount of jitter. In the second experimental condition, for ST (number of frames = 17) when the speed was 45°/s, the number of frames actually presented was 15 to 17. The data actually obtained for these clipped-trajectory conditions were not critical for this experiment or for subsequent experiments, but represent the limits of our display device and are presented for completeness. 

Results

The upper panel of Figure 2 shows the results when the dot speed was constant while the number of frames was varied. Data are shown for three observers. For observers ST and SN, thresholds generally decreased with an increase in the number of frames, leveling off when the number of frames exceeded 10. Observer DB showed a decrease in thresholds and then an increase with a minimum occurring for 9 frames. We do not know why DB’s results are different from the other two, but we suspect that it reflects loss of concentration for longer duration stimuli. Thresholds were low for 9-frame stimuli for DB and for ST and SN when the stimuli had 17 or more frames. These parameters were used for testing in the second experimental condition. 

Figure 2

 

Thresholds for detecting deviations in Experiment 1. Upper panel: Thresholds for three observers when the speed was fixed at 20°/s and the number of frames was varied. Error bars in this and subsequent plots of threshold deviations indicate ±1 SE. The lowest thresholds were obtained when the number of frames was 9 for DB and was about 17 or greater for ST and SN. Lower panel: Thresholds for two observers as the speed of the dot was varied. The number of frames was fixed at 9 for DB and 17 for ST. Note that the ordinate in the lower panel has a different scale from that in the upper panel. Thresholds were lowest when dot velocities exceeded 20°/s.

The lower panel of Figure 2 shows the results obtained when the number of frames was held fixed (at 9 for DB and 17 for ST) and the speed was varied. Thresholds were high for speeds less than 20°/s and leveled off at about 2° for greater speeds. 

Discussion

Several factors could potentially have contributed to the elevated thresholds seen in the lower panel of Figure 2:

•  
  Because the number of frames was fixed and the speed of the dots was manipulated by varying the displacement per frame, slower velocities implied shorter trajectories. Perhaps longer trajectories are required for detecting small deviations.
•  
  Faster velocities might be more conducive for detecting small deviations.
•  
  Pixellation of the screen can produce stimulus noise that can elevate thresholds (discussed below).

However, our main concern in this experiment is not the conditions yielding elevated thresholds but the conditions yielding low thresholds for use in subsequent experiments. Thus, conditions yielding elevated thresholds were not studied further. Thresholds are low when the dot speed exceeds 20°/s. For the next two experiments, the number of frames was fixed at 9 for DB and 17 for ST, and dot speed was fixed at 32°/s. 

The coordinates of the dots constituting the various trajectories were calculated as real numbers. However, in plotting the dots on the computer screen, we used discrete coordinates. This added noise to the stimulus. We used simulations to estimate the amount of noise in the stimulus. 

The following steps were used to determine, for a set of fixed stimulus parameters, the amount of error in the deviation in the trajectory on account of pixellation:

•  
  Within a block we fixed the number of frames of the stimulus, the desired displacement per frame (i.e., speed), and the desired deviation in the trajectory, using values for these parameters within the ranges that would have been used for these in Experiment 1. Eighty trajectories, with orientation jitter and vertical jitter added as in the actual experiment, were generated for each set of fixed parameters (the same number of trials that would have been used for each data point in the lower panel in Figure 1). The dots on each of these 80 trajectories would not fall along perfect straight lines on account of pixellation of the screen.
•  
  For each of the 80 trajectories in (i), we fitted straight lines to the left and right halves of the trajectories. The difference in the orientations of the two trajectories was taken as the actual deviation in the trajectory. The signed difference between the actual deviation and the desired deviation was the error in the deviation for one trajectory. The mean and SD of the deviation errors were calculated over the 80 trials.

These errors would add variability in the horizontal direction, along the “Deviation” axis, for the data shown in the lower panel of Figure 1. Stimulus uncertainties from pixellation would have contributed to some of the variability in the thresholds shown in Figure 2. Their actual contribution to the error estimates of the thresholds is difficult to estimate. However, evaluation of these errors can indicate the stimulus parameters that will provide less stimulus noise. 

The upper panel of Figure 3 shows mean errors in the deviation and their SDs as a function of the desired deviation. The speed of the dots was fixed at 20°/s as used for the data in the upper panel in Figure 2. The number of frames varied between 3 and 31, covering a large portion of the range used in Figure 2 (upper panel). The SDs of errors did not vary with the size of the deviation. Increasing the number of frames dramatically reduced the SDs of the errors. At this speed, errors were small when the number of frames was 9 or more. Subsequent experiments used at least 9 frames. 

Figure 3

 

Errors in the angle of deviation in the stimulus introduced by the discrete nature of monitor pixels. Upper panel: Mean errors in the plotted deviation of the trajectories and their associated SDs for four different numbers of frames, out of the nine used in the upper panel of Figure 2. The abscissa represents desired or intended deviations. The data, apart from those for three frames, have been offset horizontally for clarity. There is no systematic bias in the errors except when the number of frames was three. The SDs are large when each trial lasts three frames, but they decrease rapidly with increasing numbers of frames. Lower panel: Mean errors and SDs for different angles of deviation and for different dot velocities (i.e., displacement sizes) covering a large subrange of the target velocities used in Experiment 1. The data, apart from those for speed of 1°/s, have been offset horizontally for clarity. Large biases were observed for large deviations when the speed was 1°/s or 2°/s. The biases and magnitudes of errors dropped as the dot speed increased further.

For the simulation results shown in the lower panel of Figure 3, the number of frames was fixed at 17. The speed was varied between 1°/s and 42°/s. As expected, errors were smaller as the speed was increased, with errors being small for speeds of 8°/s or more. In Experiments 2 and 3, the speed used was 32°/s. 

Experiment 2: Detecting deviations in multiple trajectories, with all trajectories deviating

The previous experiment showed that for most observers thresholds for detecting deviations in a single trajectory were low if the number of frames was 17 or more (though observer DB seems to be an exception) and the speed of the dot was at least 20°/s. We were interested in knowing how well observers could detect deviations in trajectories if the stimulus consisted of more than one trajectory. In this experiment, all the trajectories on a trial underwent an identical deviation. One might anticipate that the observer’s mental representations of the trajectories might interfere with one another, and thresholds for detecting deviation might rise as a result of increasing the number of trajectories. Another possibility is that observers might have difficulty solving the correspondence problem (i.e., identifying which dot belongs to which trajectory) when there are multiple trajectories, particularly with some of the trajectories intersecting each other. In either of these cases, thresholds should increase systematically with the number of trajectories. A second possible outcome is that observers could pool information across the different trajectories and lower their deviation thresholds. A third possibility, of course, is that deviation thresholds are unaffected by the addition of more trajectories. 

Stimulus and procedure

The stimuli consisted of one or more trajectories, which, prior to deviating, were oriented about 45° with the horizontal. A random jitter of up to ±3.5° was added to the orientation to ensure that the trajectories were not parallel, either before or after deviation. The number of trajectories was 1, 2, 3, 4, 6, 8, or 10. Within a block all the stimuli had the same number of trajectories on each trial. Between blocks the number of trajectories was varied. On the frame in which the dots were aligned at the mid-line, the dot spacing in the vertical direction was 10′, with an additional jitter of up to ±10′ in the vertical direction (i.e., the dots were permitted to overlap). At this point the trajectories deviated, with all trajectories deviating by the same angle (upper panel of Figure 4 shows a schematic of a stimulus with three trajectories). This angle of deviation was varied between trials using a MOCS. Other stimulus details and observer’s responses were as discussed in General Methods. Observers DB and ST participated in this experiment. 

Figure 4

 

Tracking multiple trajectories when they all undergo the same deviation. Upper panel: Schematic of typical stimulus used in Experiment 2. For the stimulus shown (not drawn to scale), the number of trajectories was three, and the deviation of each trajectory in the figure is approximately −10°. All trajectories moved from lower left to upper right, with each having a fixed deviation occurring in line with the markers. The stimulus consisted of 9 frames for DB and 17 for ST. The dots moved at 32°/s. The jitter in the orientations of the trajectories has been exaggerated. Lower panel: Deviation thresholds as a function of the number of trajectories for observer DB (open red circles) and observer ST (open black triangles), when the trajectories had an orientation jitter of ±3.5°. Straight-line fits to the data for each observer are shown, and their calculated slopes are indicated. Thresholds were relatively unaffected by changes in the number of trajectories over the range measured. Also shown are DB’s thresholds when the trajectories had an orientation jitter of ±32° (solid red circles).

Results

The lower panel of Figure 4 plots the deviation thresholds for two observers as a function of the number of trajectories. When the number of trajectories was 1, deviation thresholds were 2.3° and 1.9° for DB and ST, respectively; when the number of trajectories was 10, the respective thresholds were 3.8° and 2.8°. The figure shows best-fitting straight lines to each observer’s data. Over the number of trajectories tested, incrementing the number of trajectories by 1 resulted in an elevation of deviation threshold by 0.16° on average for DB and by 0.07° on average for ST, as estimated from the slopes of the best-fitting lines. Also shown are DB’s data (solid red circles) when the orientations were jittered by ±32°, instead of the ±3.5° jitter used for the rest of the data in the figure. The motivation for these data is explained in the Discussion section of Experiment 3. 

Discussion

Over the range of stimulus parameters tested, changing the number of trajectories had little influence on the deviation thresholds measured. There was no evidence of facilitation between the different trajectories. If there was any interference between the trajectories, or if the observers had any difficulty solving the correspondence problem, the influence of either factor on thresholds was minimal, as evidenced by the flatness of the fits in the lower panel of Figure 4. Thresholds were low when there were 10 or fewer trajectories. The good performance suggests that the observers were accurately processing the deviations in one or more trajectories, but cannot tell us whether the observers were capable of processing more than one trajectory accurately. The observers may have processed a single trajectory on each trial, ignoring the remaining trajectories presented. However, this experiment suggests that the many trajectories present on each trial did not interfere with one another and did not compromise the observers’ ability to solve the correspondence problem (Experiment 5 further confirms these observations). Furthermore, we failed to find any evidence for pooling. Thresholds did not drop when the number of trajectories was increased; rather, they increased slightly. However, absence of evidence is not evidence of absence, and this issue is discussed further in General Discussion. In the next experiment, we modified the stimulus so that the observers were required to process simultaneously all the trajectories presented, or as many as they could. 

Experiment 3: Detecting deviations in multiple trajectories, with only one trajectory deviating

In this experiment, one or more trajectories were presented, but only one trajectory underwent a deviation. When there were multiple trajectories, the observer was unaware, beforehand, as to which trajectory would be the deviating one, and would have to process all of them simultaneously to carry out the task successfully. However, it is not evident that observers have this ability to process many trajectories. If observers can process only one trajectory accurately, then increasing the number of trajectories to two or more should result in a big increase in deviation thresholds. If observers can effectively process (say) four trajectories simultaneously, then as we increase the number of trajectories, we might expect small changes in threshold when the number of trajectories is between one and four, and thereafter a steep increase in threshold. A measure of the change in deviation thresholds as we vary the number of trajectories will convey information regarding the number of trajectories that can be processed simultaneously. 

Stimulus and procedure

The stimulus was identical to that in Experiment 2, except that the trajectory of one of the dots on a trial deviated at the monitor mid-line, whereas the remaining dots continued along undeviated trajectories. Prior to the deviation, all trajectories had a mean orientation of 45°, with a jitter of ±32°, which was larger than the ±3.5° used in Experiment 2. The large thresholds obtained in this experiment (see Results below) necessitated the increase of the orientation jitter, so that the deviating trajectory did not stand out from the other trajectories. Observers had to report the direction of deviation of the one deviating trajectory, ignoring the other trajectories. In all other respects, the stimuli and procedures were identical to Experiment 2 and the same two observers DB and ST participated in this experiment. The upper panel of Figure 5 shows a schematic of the stimulus used when the number of trajectories was three. A comparison of the upper panels of Figures 4 and 5 emphasizes the difference in the stimuli used in Experiments 2 and 3. 

Figure 5

 

Tracking multiple trajectories when only one of them undergoes a deviation. Upper panel: Schematic of typical stimulus used in Experiment 3. For the stimulus shown, the number of trajectories was three. In this experiment, all trajectories moved from lower left to upper right, with only one of the trajectories undergoing a deviation occurring in line with the markers; the other trajectories proceeded without deviation. The deviation shown is −10°. The stimulus consisted of 9 frames for DB and 17 for ST. The dots moved at 32°/s. Lower panel: Deviation thresholds as a function of the number of trajectories for two observers. Also shown are straight-line fits to the data for each observer, and their calculated slopes. Thresholds could be measured for only up to four trajectories. As the number of trajectories was increased, thresholds increased far more rapidly than thresholds obtained when all the trajectories deviated.

When designing our experiment, we had expected that as we increased the number of trajectories, thresholds would rise more slowly than they actually did (see Results below). The largest deviation permitted in our paradigm was 32°. Larger deviations would have resulted in the deviating trajectory standing out from among the nondeviating trajectories, or overwriting the upper mid-line marker, or terminating outside the upper right quadrant where the nondeviating trajectories terminated. Consequently, we could measure thresholds only when the number of trajectories was four or less. Even for four trajectories, the responses did not cover the full range of the psychometric function, the estimated threshold being larger than the largest deviation used in the experiment. 

Results

The lower panel of Figure 5 plots the results of this experiment in a format identical to that of the previous one for the same two observers. It was not possible with our experimental paradigm to measure thresholds reliably for more than four trajectories (see Stimulus and Procedure above). Regression lines were fitted to the data for each observer. DB’s deviation thresholds rose from 3.3° for one trajectory to 32.9° for four trajectories. The corresponding rise in threshold for ST was from 3.0° to 38.4°. On average, incrementing the number of trajectories by 1 resulted in an elevation of threshold by 10.0° for DB and 12.4° for ST, as estimated from the best-fitting lines to the data. 

Discussion

Over the range of stimulus parameters tested, changing the number of trajectories even by one had a large effect on thresholds. Changing the number of trajectories from one to two resulted in an increase in thresholds from 3.3° to 12.3° for DB and from 3.0° to 10.1° for ST. The increase of thresholds by a factor of 3.8 and 3.4 for DB and ST, respectively, as a consequence of increasing the number of trajectories from one to two clearly shows that observers were unable to effectively process more than one trajectory for deviations. This inability may reflect attentional limitations (i.e., an inability to adequately attend to all the trajectories presented) or limitations of working memory (i.e., an inability to store adequate information about the individual trajectories). In either case, the visual system is unable to process more than one trajectory effectively. 

When there was only one trajectory in the stimulus, the stimulus in this experiment was identical to that in Experiment 2, except for the greater uncertainty in the orientation of the trajectories. Increasing the orientation jitter from ±3.5° to ±32° resulted in an increase in thresholds from 2.3° to 3.3° for DB and from 1.9° to 3.0° for ST (Figures 4 and 5). An increase in orientation uncertainty results in an increase in deviation threshold. 

Could the elevated thresholds observed in Experiment 3 be a consequence of the increased orientation uncertainty in this experiment? Observer DB repeated Experiment 2 (i.e., with all trajectories deviating) for 6, 8, and 10 trajectories, with the jitter in the orientation increased to ±32°. The results are included in the lower panel of Figure 4 (solid red circles). Increasing stimulus uncertainty produced an increase in thresholds; for 10 trajectories, thresholds for ±32° orientation jitter were 6.4°, compared to 3.8° for ±3.5° jitter. However, these increases were not of the order seen in the lower panel of Figure 5 where DB’s threshold was 32.9° with just 4 trajectories. 

The elevated thresholds in Figure 5 cannot be explained by interference between the mental rep-resentations of trajectories, or by the correspondence problem, or by the increased orientation jitter used in the experiment. If there was interference between the different trajectories, then such interference should have also occurred in Experiment 2, when the stimulus was almost identical (also see Experiment 5). For the set of parameters chosen, observers were unable to effectively process more than one trajectory simultaneously. 

Experiment 4: Detecting deviations in multiple trajectories — effect of speed

Pylyshyn and Storm (1988) showed that observers could track up to five objects simultaneously. However, in Experiment 3, we found that observers’ deviation thresholds were severely elevated when attempting to track more than one trajectory at a time. The difference in the results could reflect differences in the nature of the observer’s tasks in the two studies. Alternatively, the choice of stimulus parameters in Experiment 3 might have been such as to make simultaneous processing difficult. In this experiment, we aimed to optimize parameters to facilitate simultaneous processing of the trajectories. 

Stimulus and procedure

The stimulus was similar to that used in Experiment 3, with the following differences:

•  
  The experiment was repeated at 2, 4, 8, and 16°/s, instead of the 32°/s used previously. In the Pylyshyn and Storm (1988) experiments, the speed of the dots varied between 1.25 and 9.4°/s. Perhaps the speed that we used was too high to permit simultaneous processing. Experiment 1 suggested that velocities in excess of 20°/s were conducive for obtaining low deviation thresholds. However, this might be true only for stimuli with one trajectory. Speeds that are optimal for single trajectory stimuli might not be optimal for stimuli with many trajectories.
•  
  In the Pylyshyn and Storm (1988) experiments, the crosses were initially stationary for 10s, with the target items flashing, before all the dots started moving. Yantis (1992) suggested that observers grouped the target elements into one coherent virtual object toward which their attention was directed. In our Experiment 3, the dots appeared abruptly and moved immediately along their designated trajectories. Perhaps the inability of observers to process more than one trajectory simultaneously reflects their inability to group the dots into one coherent virtual object in the time available. To permit potential grouping mechanisms to operate, our trajectories were modified so that on every trial, the dots first appeared stationary and started to move only after the observer pressed the appropriate key on the keyboard.
•  
  The orientation of the trajectories was modified to have a mean of 0° (horizontal) and a uniform jitter of ±80°. It was hoped that this modification would permit a wider range of deviations, thus permitting us to measure thresholds for stimuli consisting of five or more trajectories. The maximum deviation used in the experiment was 76°.
•  
  In Experiments 2 and 3, the spacing between the trajectories at the point of deviation was small (10′ with a jitter of ±10′). We increased the spacing between the trajectories to 90′ with a jitter of ±5′, to ensure that the different trajectories did not interfere with one other at the instant of deviation.
•  
  In the current and subsequent experiments, dot size was 5′ × 5′ to ensure that any increase in thresholds, if observed, was not a consequence of the visibility of the individual dots.
•  
  In Experiment 1, when the dot speed was varied, the trajectory length was proportionately varied. In Experiments 2 and 3, the speed and the number of frames were fixed for each observer, as was the length of the trajectories. In the current experiment, the trajectory length was kept fixed at close to 200′ by reducing the number of frames (101, 51, 25, and 13) as the displacement/frame (101, 51, 25, and 13) as the displacement/frame (2′, 4′, 8′, and 16′, respectively), i.e., speed (approximately 2, 4, 8, and 16°/s, respectively), was increased. Figure 3 (lower panel) suggests that 17-frame stimuli can be noisy for a speed of 4°/s or less. For the 101-frame and 51-frame stimuli used in this experiment, the noise is substantially less than that shown in Figure 3, even at the slower velocities used.

The upper panel of Figure 6 shows a schematic of the stimulus with four trajectories. One of the trajectories is shown deviating by approximately −10°. Only observer ST participated in this experiment. It was anticipated that the above modifications would facilitate parallel tracking of the trajectories and would yield deviation thresholds similar to those in Figure 4. 

Figure 6

 

Effect of dot speed on thresholds for tracking multiple trajectories. Upper panel: Schematic of typical stimulus used in Experiment 4, with four trajectories. All trajectories moved from left to right, with only one of the trajectories undergoing a deviation. Unlike previous experiments, the starting point for the trajectories could be in either of the quadrants on the left (see text for other differences). Lower panel: Observer ST’s thresholds as a function of the number of trajectories for four different speeds are represented by the four left-most curves. The two right-most curves represent ST’s thresholds for two different speeds, under the experimental conditions used in Experiment 3; the data along these curves have been offset by 1 (i.e., the abscissa has been incremented by 1), for clarity. Irrespective of speed, thresholds increased rapidly with the number of trajectories. In fact, thresholds were higher in the modified paradigm (four left-most curves) than they were using the paradigm in Experiment 3 (two right-most curves), presumably because of the greater uncertainty in the orientations of the trajectories in the former.

For the purposes of comparison, data were also obtained using the experimental conditions described in Experiment 3, but with the speeds changed. For these data the numbers of frames were 129 and 33, with the corresponding displacements/frame being 4′ and 16′, yielding speeds of 4°/s and 16°/s. These parameters ensured that the length of each trajectory was 512′ long, from the center of the first dot to the center of the last dot, measured along the trajectory (the same length as that used in Experiment 3). 

Results

We had hoped that by increasing the jitter we would be able to increase the largest permitted deviation of the target trajectory and hence measure deviation thresholds even when the number of trajectories was five or more. However, the added stimulus uncertainty contributed to an increase in the measured thresholds. Consequently, we were again unable to measure deviation thresholds for stimuli containing more than four trajectories. 

The four left-most curves in the lower panel of Figure 6 show deviation thresholds as a function of the number of trajectories for ST for the four different speeds. Data are shown for up to four trajectories for each speed, with the exception of speed of 16°/s for which thresholds could only be obtained for up to two trajectories. Regardless of the speed, thresholds increased very steeply with the number of trajectories. In fact, thresholds increased more steeply than they did in the previous experiment, probably due to the increased orientation jitter of the trajectories. This increase in steepness can be seen by comparing with the two right-most curves, representing thresholds for speeds of 4°/s and 16°/s, under conditions described in Experiment 3 (for clarity, these data have been offset along the abscissa by 1 unit). 

Under the modified experimental conditions, when there were four trajectories, the responses did not cover the full range of the psychometric function. The speeds of the dots did not qualitatively affect deviation thresholds for velocities between 2°/s and 8°/s. When the dot speed was increased to 16°/s, thresholds were elevated further and the task was difficult to perform when there were three or more trajectories. Hence, Figure 6 shows only two data points for this speed. ST’s thresholds for one trajectory were elevated compared to those shown in Figures 4 and 5 and in the right-most two curves of Figure 6 because of the increased stimulus uncertainty; the orientation jitter was ±80° here compared to ±3.5° in Figure 4 and ±32° in Figure 5 and for the two right-most curves in Figure 6. 

Discussion

The goal of this experiment was to refine the stimulus parameters with the aim of facilitating simultaneous processing of more than one trajectory. However, regardless of the choice of parameters and the modification of the methods to improve performance, thresholds were again found to increase rapidly as the number of trajectories was increased. Observers are unable to process two or more trajectories simultaneously with high efficiency when looking for deviations. 

The modified conditions involved several changes, some of which may have been detrimental to performance (such as the increased orientation jitter). Even when all the conditions were similar to those in Experiment 3, and speed was the only parameter varied, severe set-size effects were still observed at the two speeds. When thresholds were plotted against number of trajectories on log-log scales, the log-log slopes ranged from 1.21 (for 4°/s speed, under conditions of Experiment 3) to 1.66 (for 8°/s speed, under the modified conditions). The largest log-log slope was 1.99 for 16°/s speed under the modified conditions, but this was based on only 2 data points. Regardless of our choice of stimulus parameters, increasing the number of trajectories from one to four resulted in thresholds being elevated by a log unit or more. 

Our experiments in this study used an apparent motion paradigm, primarily because of the limitations of our display device. It is plausible that had real motion been used, observers might have been able to track the deviations more accurately. The relevant question is whether we expect any qualitative difference from the current results if real motion had been used. The results in the lower panel of Figure 6 suggest that we should not expect such qualitative differences. The slower the speed of the dots, the closer the motion was to real motion. For the smallest velocity of 2°/s, the displacement between frames was 2′, which was smaller than the size of the dots themselves. At this velocity, the percept was one of smooth and continuous motion, and yet no qualitative difference was found between the set-size effects seen for this velocity and those observed at higher velocities. We anticipate that even if the stimulus dots had undergone real motion, large set-size effects would still exist; any difference would be quantitative and not qualitative. The next experiment looks at the effect of cueing on deviation thresholds. 

Experiment 5: Detecting deviations in multiple trajectories — effect of cueing

The results of Experiment 2 suggested that any interference between trajectories was very small, in spite of trajectories being permitted to intersect. However, an alternative explanation exists for the absence of an effect of the number of trajectories on threshold (Figure 4, lower panel). Good performance in Experiment 2 requires that the deviation in any one trajectory be detected reliably, say one of the “end” trajectories. There could be interference between the trajectories, but if (say) the uppermost trajectory, being flanked on only one side by other trajectories, was not subject to much interference from these, then the observer could still perform well by processing the uppermost trajectory. (Note that the word “uppermost” is loosely used here. Because the different trajectories could intersect, different trajectories could be uppermost at different instants of time.) On the other hand, this strategy will not work well if the uppermost trajectory is not the one that deviates on most trials, as was the case in Experiment 3. In Experiment 2, the stimulus permits the observers to pick the trajectory they will process, and perhaps the observers learn to process the trajectories that experience the least interference from the other trajectories. This could explain why thresholds were low in Experiment 2, and high in Experiments 3 and 4. Although this is an unlikely explanation, further experiments were conducted to rule it out. 

One can determine if there is interference between the trajectories by cueing the deviating trajectory in the stimuli used in Experiments 3 or 4 (i.e., when only one of the trajectories deviates). If there is interference between the trajectories, then thresholds for detecting the deviation should rise as the number of trajectories is increased, in spite of the observer knowing beforehand which trajectory will deviate. On the other hand, if there is no interference between the different trajectories, then deviation thresholds should be relatively unaffected as the number of trajectories increases. This would suggest that the elevated thresholds in Experiments 3 and 4 are a consequence of processing the wrong trajectory with a high probability on trials that had more than one trajectory. In this experiment, two different cueing methods were used to evaluate inter-trajectory interference. 

Stimulus and procedure

The stimuli and procedures were identical to those used in Experiment 4, with the differences described below. 

In the first paradigm, the colors of the dots cued the deviating trajectory. The deviating dot was always red [14.4 cd/m², CIE coordinates (0.58, 0.36)], whereas the colors of the nondeviating dots were randomly selected from green [41.4 cd/m², (0.30, 0.57)], blue [8.8 cd/m², (0.16, 0.08)], or white [56.9 cd/m², (0.28, 0.30)]. Observers were aware that the deviating dot was always the red one. 

In the second paradigm, all the dots were white. At the start of each trial, one stationary dot was presented; this was the dot that would deviate during the animated phase. When the observer pressed the appropriate key on the keyboard, the other dots appeared and all the dots moved across the screen as before. The difference between this stimulus and that in Experiment 4 that here during the initial stationary phase only one deviating dot was presented whereas in the previous experiment, during the stationary phase, as many dots were presented as there were trajectories. 

Data were collected for 1, 4, and 10 trajectories using each of the above paradigms. The number of frames was 51 (50 for the nondeviating dots in the second paradigm), dot speed was 4 °/s, and trajectory separation was 40′ with a jitter of ±5′. The reduced trajectory spacing was necessary in order to accommodate all 10 trajectories in the space between the markers. The observers were BB and ST. 

Results

The upper panel of Figure 7 shows deviation thresholds as a function of the number of trajectories for observers BB and ST when dot color was used as a cue. Also shown are the best-fitting straight lines to each observer’s data. When the number of trajectories was 10, deviation thresholds for BB and ST were 6.3° and 7.1°, respectively. On average, incrementing the number of trajectories by 1 resulted in an increase in thresholds by 0.15° and 0.32° for BB and ST, respectively, as estimated from the best-fitting lines to the data. 

Figure 7

 

Effect of cueing on thresholds for tracking multiple trajectories. Upper panel: Thresholds for two observers as a function of the number of trajectories, with only one of these trajectories deviating. The target trajectory was cued using color. The target trajectory was a moving red dot and the other trajectories were moving green, blue, or white dots. Data for ST are slightly offset in the horizontal direction for clarity. Lower panel: Same as upper panel, but the cueing strategy was different from that used to obtain the data in the upper panel (see text).

The lower panel of Figure 7 shows similar data collected with the second cueing paradigm for the same observers. For stimuli with 10 trajectories, deviation thresholds for BB and ST were 5.6° and 9.5°, respectively. On average, incrementing the number of trajectories by 1 resulted in an increase in thresholds by 0.03° and 0.54° for BB and ST, respectively, as estimated from the fits to the data. 

Discussion

When the deviating trajectory was cued using a different color, observers experienced little additional difficulty in detecting deviations when the number of trajectories was increased from 1 up to 10. One might be tempted to conclude that the negligible increase in threshold with increase in number of trajectories implies that there was little interference between the trajectories in previous experiments. However, an alternative possibility is that the deviating dot, having a different color from the other dots, “pops out” from among the other dots (Treisman & Gelade, 1980; Treisman & Souther, 1985; Treisman & Gormican, 1988), resulting in little interference from the other dots/trajectories on the deviating trajectory. This could account for the low thresholds observed in this experiment, even when there were 10 trajectories. 

In the second cueing paradigm, all the dots were white and had the same luminance; the deviating dot/trajectory could not have “popped out” from the other trajectories. However, even with this cueing paradigm, we find that the deviation thresholds are relatively unaffected by the number of trajectories. 

The thresholds seen in Figure 7 are qualitatively similar to those in the lower panel of Figure 4. However, the thresholds in Figure 7 are noticeably higher than those in Figure 4. This difference, as discussed previously, presumably results from the increased orientation jitter used in this experiment (±80°) compared to Experiment 2 (±3.5°). 

In summary, this experiment shows the following:

•  
  When presented with many trajectories, of which only one deviates, observers are quite sensitive to detecting deviations in this trajectory, provided they are cued beforehand as to which trajectory is to deviate.
•  
  The nondeviating trajectories do not interfere with the observers’ abilities to detect deviations in the target trajectory. The information present in the deviating trajectory is not compromised by the presence of additional nondeviating trajectories.
•  
  The elevation of thresholds seen in Experiment 3 is not due to interference between the trajectories (confirming similar suggestions made in the Discussion of Experiment 2), but an inability to effectively process the information available when two or more trajectories are presented simultaneously in this paradigm.

Experiment 6: Detecting deviations in multiple trajectories — effect of color

In Experiment 4, when we tried to optimize stimulus parameters to obtain the lowest deviation thresholds, we did not vary the colors of the dots; all dots in the stimulus of Experiment 4 were white. In Experiment 5 (first paradigm), we varied the colors of the dots, but the deviating dot was always red. An interesting question is, “How does performance change with increasing number of trajectories if each trajectory on a trial has a unique dot color?” Could color help to solve the correspondence problem and facilitate the simultaneous processing of several trajectories? The role of color in multiple object tracking is controversial. Earlier experiments, using a color-change detection paradigm, suggested that featural properties, such as color, are not reliably coded by the visual system during multiple object tracking (Scholl, Pylyshyn, & Franconeri, 1999; Scholl, 2001). However, more recent experiments suggest that color changes are encoded on a high proportion of the trials (Bahrami, 2003). In the more recent experiments, the stimuli consisted of four target trajectories and four distractor trajectories, with one of the target trajectories undergoing a change of color. For these stimuli, observers tracked the target trajectories and identified the changes in color on more than 75% of the trials (see Bahrami, 2003; Figure 3, upper panel: “No mud condition,” when the change occurred on the target trajectories; chance performance was 12.5% in their experiment). Because the experiments in Bahrami (2003) show that color transients are encoded on a relatively high proportion of trials, it is plausible that the available color information may facilitate deviation detection in our experiments by helping to separate the identities of the different trajectories, particularly when two trajectories intersect, which they do very frequently (see General Discussion). However, there is a need for caution when extrapolating results from previous MOT paradigms to the current paradigm. Previous MOT paradigms suggest a special role for spatiotemporal parameters, such as direction of motion; however, direction of motion is not efficiently processed in our threshold paradigm when there are two or more trajectories. (The above distinction between spatiotemporal properties and featural properties may not be universally recognized. For example, Treue & Trujillo, 1999, refer to location and direction of motion as features.) Similarly, there may be differences in the way color is used in our threshold-detecting paradigm compared to previous paradigms. To study the contribution of color in the current paradigm, we randomly varied the colors of the deviating dot and any nondeviating dot(s) on each trial (unlike Experiment 5, first paradigm, where the deviating dot was always red). 

Stimulus and procedure

The stimulus was identical to that in Experiment 4, with the difference being that the dots now had color. When there was only one trajectory, the dot could be white, red, green, or blue; the color was fixed within a block and varied between blocks. When there were two trajectories, they consisted of white and red dots. The third and fourth trajectories, when present, consisted of green and blue dots, respectively. The luminances and chromaticity coordinates of the colors used are as specified in Experiment 5. 

The upper panel of Figure 8 shows the schematic of a stimulus with four trajectories. The number of frames was 25, dot speed was 8°/s, and trajectory separation was 90′ with a jitter of ±5′. No data were collected for five or more trajectories because the observers had difficulty performing the task when there were four trajectories. Observers BB, SS, and ST participated in the experiment. 

Figure 8

 

Effect of color on thresholds for tracking multiple trajectories. Upper panel: Schematic of typical stimulus used in Experiment 6. For the stimulus shown, the number of trajectories was four, and dots from different trajectories had different colors. Only one of the trajectories deviated; the observer was unaware of the color of the deviating trajectory. Lower panel: Thresholds as a function of the number of trajectories for three observers. For number of trajectories = 1, the threshold shown for ST was the average of the thresholds obtained for each of the four target colors; for BB and SS the thresholds shown were for white targets. Also shown are ST’s thresholds for each target color (red triangles), when the number of trajectories = 1; an offset has been added to the abscissa for clarity, with the four triangles representing, from left to right, deviation thresholds for white, red, green, and blue targets respectively. Straight lines were fitted to the data for each observer and their slopes are indicated. The use of color did not facilitate the tracking of the moving dots.

Results

The lower panel of Figure 8 shows deviation thresholds for BB (green diamonds), SS (blue inverted triangles), and ST (black upright triangles) for one to four trajectories. For BB and SS, the threshold for one trajectory was obtained for a white, moving dot. ST’s threshold for one trajectory was estimated with each of the four colors. These four thresholds are shown as red triangles in the figure, with the abscissa offset by different amounts for each color to aid readability. Deviation thresholds were not significantly different for single trajectory stimuli of different colors. The threshold shown for ST for one trajectory (black triangle at number of trajectories = 1) has been obtained by pooling all the data for the four color conditions and represents the average threshold for a single trajectory of different colors. The average deviation threshold is comparable to the four deviation thresholds obtained for trajectories of the four different colors. 

Straight lines were fit to the data for each observer in Figure 8. Deviation thresholds increased sharply as the number of trajectories was increased. On average, incrementing the number of trajectories by 1 resulted in an increase in thresholds by 18.5°, 20.5, and 16.6° for BB, SS, and ST, respectively (their corresponding log-log slopes were 1.74, 2.07, and 1.86). These slopes were comparable to those seen in Figure 6 and higher than those in Figure 5. For the data shown in Figure 5, the corresponding slope for ST was 12.4°/trajectory, that for DB was 10.0°/trajectory, and BB and SS did not participate in Experiment 2. The higher thresholds and steeper slopes obtained in this experiment presumably result from the increased orientation jitter (±80°) used. 

Discussion

The assignment of unique colors to trajectories made little difference to deviation thresholds. Thresholds still rose steeply with increase in the number of trajectories, even when the number of trajectories increased from one to two. In the current paradigm, observers were unable to process more than one trajectory accurately, even when the different trajectories were uniquely colored. 

General discussion

When measuring deviation thresholds only one trajectory can be processed accurately

The question addressed in the present series of experiments concerns the extent to which the sensitivity for detecting a deviation in a linear trajectory is influenced by the number of trajectories presented simultaneously. We expected that the rate at which thresholds increase when the number of trajectories increases could inform us of the number of trajectories that can simultaneously be processed by human vision. When all trajectories presented deviate in a similar manner, deviation thresholds are only slightly influenced by the number of trajectories (Experiment 2). Even with 10 trajectories in the stimulus, thresholds are not substantially higher than with only one trajectory. These low thresholds could result from observers processing one trajectory randomly selected, a subset of the trajectories selected randomly, or all of the trajectories in parallel. Although we cannot determine which of the three strategies the observer uses, it is clear that the information content in at least some of the trajectories is not compromised by other trajectories present in the neighborhood. However, it is still possible that the information in some of the trajectories might be compromised; thresholds could still be low if the observers learned to process those trajectories that were not compromised (say the “uppermost” or the “lowermost” trajectory). 

When only one out of several trajectories present has a deviation, thresholds are severely elevated by the presence of the additional trajectories (Experiment 3). Increasing the number of trajectories from one to two elevates thresholds by a factor of 3 or 4. Even when the experimental paradigm was modified to facilitate the parallel processing of several trajectories and the stimulus parameters were optimized, thresholds were still elevated by the presence of other nondeviating trajectories (Experiment 4). One possibility is that the observers are unable to process more than one trajectory accurately, and the elevated thresholds obtained when several trajectories are present result from the observers processing, with a high probability, a trajectory that does not deviate. A second possibility is that the available resources are distributed among the available trajectories, and the resources available per trajectory do not permit an accurate detection of deviations, even when there are only two trajectories available. Another, though less likely, possibility is that inter-trajectory interference occurs for some trajectories, but the thresholds in Experiment 2 are not affected because the observer learns to track one or more of the trajectories that are not compromised. 

The last of the above possibilities is ruled out by the cueing experiments (Experiment 5). Because thresholds were low when the observers were cued to the only deviating trajectory on each trial, even when there were as many as 10 trajectories, the information in the cued trajectory must have been available to the observer and was not compromised by the presence of the other trajectories. This is supported by the results of Experiment 6 in which each of the different trajectories was uniquely colored but no cueing was employed. The use of different colors may help to solve the correspondence problem (i.e., to determine which dot belongs to which trajectory). However, even when the color information for simplifying the correspondence problem is made available, thresholds are not lowered. This suggests that observers have no difficulty assigning dots to trajectories in the first place, and the correspondence problem does not limit the ability of observers to detect deviations. Deviation thresholds are low provided the observers know which trajectory will deviate. Deviation thresholds are high if the observers are unaware as to which trajectory will deviate, even if the trajectories have different colors and even when there are as few as two trajectories. We believe that the limits to deviation-detecting performance when processing multiple trajectories are most likely to be attentional, though limitations of working memory could also have contributed to the poor performance. Observers are unable to accurately process more than one trajectory at a time, when using a threshold paradigm, even when the stimulus parameters have been optimized. When more distractor trajectories are presented, this increases the proportion of the observer’s resources that are directed at trajectories that do not deviate and decreases the resources available for processing the deviating trajectory, thus resulting in elevated deviation thresholds. When the deviating trajectory is cued, this trajectory is allocated adequate resources and thresholds become independent of the number of trajectories (Experiment 5, Figure 7). The role of cueing could be to ensure that the target trajectory is assigned adequate attentional resources, or the target trajectory is assigned adequate working memory, or information relevant to the target trajectory is more efficiently coded in working memory, or all of the above. Questions remain as to whether the critical limiting resource is attention or working memory, and the circumstances under which one or the other resource becomes critical. 

A possible strategy that observers could have used in the current threshold paradigm is to track one randomly selected trajectory, ignoring the remaining trajectories. Could this strategy explain the steep increase in thresholds with increase in the number of trajectories in Experiments 3, 4, and 6? A closer look at the psychometric functions obtained (not shown) in these experiments convincingly showed that this cannot be the case. If observers followed the strategy of tracking only one trajectory when (say) two trajectories were presented, then the deviating trajectory would be ignored on 50% of the trials. If observers guessed the direction of deviation on these trials, then they would be incorrect on 25% of the trials. The resulting psychometric functions would no longer asymptote at 0 and 100% (see lower panel of Figure 1), but at 12.5% and 87.5%. Similar calculations for three trajectories yield asymptotes at 16.67% and 83.33%. For no deviation should the proportion of CCW responses fall outside these asymptotes. However, in our experimental results, for stimuli with 1-3 trajectories, when the appropriate range of deviations was used, we consistently found the psychometric function extending from 0% to 100%. The performance of the observers was better than it would have been had they employed the strategy of tracking one trajectory only. It would appear that for stimuli with more than one trajectory, observers were tracking more than one trajectory, but not very efficiently. 

When detecting thresholds in Experiments 1–6, we varied the magnitude and sign of the angle of deviation between trials, sometimes over a wide range, particularly when there were three or four trajectories. Under these circumstances, it is plausible that on individual trials observers processed fewer trajectories when the deviations were small and a greater number of trajectories when the deviations were large (Tripathy, 2003). If that is the case, then the constant variance assumption made when fitting cumulative normal functions (see Analysis under General Methods) might no longer hold, and thresholds might not correspond to a d’ = 1. It might be more appropriate to think of our thresholds as empirically defined, specified at one SD from the subjective null deviation, rather than at a fixed d’ = 1. This difference is only of theoretical importance. In practical terms, our data were well fit by cumulative normal functions when there were three trajectories or less in the stimulus. When there were four trajectories, the data were noisier, and frequently only partial psychometric functions could be obtained, as discussed previously. However, the drop in performance with increases in the number of trajectories is very evident from the psychometric functions obtained (not shown), which are independent of any definitions of threshold. 

When the deviations are suprathreshold, as many as three or four trajectories can be accurately processed simultaneously

The experiments presented here collectively suggest that observers can effectively process only one single trajectory. In these experiments, the paradigm used involved determining thresholds for detecting deviations. Consequently, the deviations of the target trajectories were comparable to the threshold deviations. Tripathy (2003) addressed the issue of whether a greater number of trajectories could be processed, if the deviations in the trajectories are substantially suprathreshold. For an angle of deviation of 76°, Tripathy found that the data obtained, when the number of trajectories was varied from one to 10, were consistent with observers simultaneously processing three or four trajectories reliably, a result that is reasonably consistent with the findings of Pylyshyn and Storm (1988). Our current finding, that efficiency drops when tracking more then one trajectory for deviations, only applies to our threshold paradigm. When the deviations are highly discriminable, a greater number of trajectories can be effectively tracked (Tripathy, 2003). 

Most of the multiple-tracking studies support the idea that there is an upper limit to the number of items (four or five) that can be tracked simultaneously. Davis, Welch, Holmes, and Shepherd (2001) propose that this limit is imposed by the complexity of the stimulus used; when they increased the number of items while keeping the information content of their stimulus fixed, they found that observers were able to attend to as many as six items without loss of efficiency. Our results suggest that if the event attended is easily discriminable (i.e., if the event is easy to detect, such as when the deviation is substantially suprathreshold), then the number of items that can be monitored is increased. Thus, either complexity of the stimulus or its discriminability could influence the number of items attended or tracked. 

Pylyshyn has proposed a “visual indexing” mechanism that can be used to track a limited number of objects (Pylyshyn, 1989, 1994, 2000, 2001). These indices are assigned to various items in the visual field, and once assigned, they move with the item. Their function is similar to pointers in computer data structures, permitting attention to access the indexed items directly and assisting in feature binding. If observers are using such an indexing mechanism for tracking in our threshold paradigm, indexing appears not to have been as effective as in previous MOT paradigms. It is possible that these visual indices point, at any instant, to the “current positions” of objects and do not maintain a record of previous positions. In our threshold paradigm, this could explain the increase in threshold when the number of trajectories increases from one to two. 

Comparison with previous paradigms used inmultiple object tracking

Previous approaches to measuring performance with multiple object tracking include studying either reaction times to events or determining the proportion of trials on which the events were detected (e.g., Pylyshyn & Storm, 1988; Sears & Pylyshyn, 2000; Scholl, 2001; Bahrami, 2003). The current study focuses on how having multiple objects/trajectories in the stimulus influences the precision for making spatial judgments involving these trajectories. In what ways is our paradigm different from previous MOT paradigms? This section addresses this question. 

Before comparing our paradigm to previous ones, we need to understand that tracking can involve several different levels. What one intuitively refers to when using the word “tracking” is quite different from the tracking measured in our experiments or in experiments using the Pylyshyn paradigm. Consider this from the perspective of four tennis players warming up before a doubles game, each player hitting his/her shots to the player diagonally opposite. Returning the ball to the opposite court requires each player to track the ball, among other things. Tracking the ball has several components, such as recognizing which is the ball to be hit (there are two in this situation), recognizing the spin on the ball and following the deviation in the trajectory after the ball has landed, knowing when and where the trajectories of ball and racket intersect, etc. The experimental paradigms used for studying “tracking” only measure subsets of the tracking activities implied by the intuitive use of the word. It is illustrative to map the task in the experimental paradigms to the tasks that the tennis players (discussed above) must perform to track the ball. Identifying which ball to hit, out of the two on the court, corresponds loosely to knowing whether a dot/cross is a target in the Pylyshyn paradigm. Detecting the deviation in the trajectory, after the spinning ball has landed, corresponds loosely to detecting a deviation in our experimental paradigm. The two paradigms measure very different components of what we intuitively refer to as tracking and may involve very different mechanisms. 

There are several other notable differences between the two paradigms:

1.  
   In the Pylyshyn and Storm (1988) experiments, subjects were required to track the targets for several seconds, whereas in ours the stimuli were typically presented for much shorter durations.
2.  
   In the earlier experiments, the targets and distractors moved along complex trajectories, whereas we used simple linear trajectories with uniform speed.
3.  
   The observers in Pylyshyn’s experiments were required to track the targets for the entire duration of the stimulus. In our experiments, observers could choose when to attend to or track the trajectories; one strategy could be to attend/track the dots only when they were close to the point of deviation. At the very least, three samples of position per trajectory have to be registered and remembered, with at least one sample each on the left and right half-trajectories, to reliably detect a deviation.
4.  
   Pylyshyn and Storm’s paradigm requires that the visual system know the positions of the tracked items at the current instant. There was no requirement for it to know where the items were, say 1 s previously (i.e., the visual system only needed to know enough of the history of the tracked items to be able to update their current positions). On the other hand, in our study, knowing the current position of the dots provides no information as to the direction of the deviation. To determine the direction of the deviation, it is critical to know the previous positions of the dots over a time window and this might involve working memory to a greater extent compared to previous paradigms.

Despite the above differences, the two approaches to studying tracking complement each other. The traditional MOT experiments (e.g., Pylyshyn & Storm, 1988) give us information regarding the individuality of the tracked items; however, they do not give us information regarding the shape of the trajectories of the tracked items. Our experiments require monitoring deviations in linear trajectories, giving us a very primitive measure of the shape of trajectories. Tracking deviations in quasi-linear trajectories is something we do all the time in everyday life, from sporting activities that involve hitting, kicking, or passing a ball to tracking cars changing lanes while driving. In other real-world situations, such as controlling air traffic at a busy airport or fighter pilots in a dogfight, the deviations in the trajectories of tracked items are as important as the individualities of the tracked items. Further studies using our paradigm could improve our understanding of tracking and allocation of attentional resources and working memory in such real-world situations. 

Is the current paradigm a tracking paradigm?

Several studies have attempted to distinguish between (1) the ability to attend to (and track) an object, and (2) the ability to encode properties of that object into working memory, as a result of attending to it or tracking it. Subjects can track and attend to objects, yet fail to encode properties of the tracked objects, or to detect changes to these objects (Bahrami, 2003; Scholl, 2001). Our paradigm involves detecting deviations in the trajectories of moving dots. Is this a tracking paradigm, or a change-detection paradigm? To answer this question, we need to ask, “What is the minimum amount of information that the visual system needs to detect a deviation?” At the very least, the visual system must sample, at three different instants of time, the positions of the dot that corresponds to the deviating trajectory. If more than three samples are processed, the accuracy with which deviations can be detected will be improved. However, regardless of the number of samples used, the dot that corresponds to the deviating trajectory must have been tracked on at least three occasions. Further, Scholl (2001) suggests that in the MOT paradigm, spatiotemporal properties, such as location and direction of motion, are reliably encoded during the tracking process, whereas featural properties, such as color and shape, are not reliably encoded. The Pylyshyn experiments track the changes in the first of the above two spatiotemporal properties, namely location; our experiments track changes in the other relevant spatiotemporal property, namely direction of motion. Given that both paradigms involve coding of spatiotemporal properties, we believe that both paradigms measure spatiotemporal tracking, and both paradigms measure spatiotemporal change detection. However, it is possible that because our paradigm involves keeping track of the history of the dots over longer durations, it may be more dependent on working memory than previous MOT paradigms. 

Relevance to other studies of attention

Pylyshyn and Storm (1988) have suggested that MOT involves a preattentive system. Most other researchers, however, consider MOT as a paradigm for attentional selection and pursuit (Scholl, 2001). Although the contribution of attention to the processing of tasks such as those examined in the current investigation remains controversial, it is useful to speculate on the implications our findings may have for the theories of attention. 

Space-based theories of visual attention fall into two broad categories, unitary and distributed. Among the unitary models/metaphors for spatial attention, the most popular has been the Spotlight Metaphor. The Spotlight Metaphor compares attention to a spotlight with its “beam” covering an area of space; objects within the “beam” of the spotlight are more effectively processed than those outside (Posner, 1978; Posner, Snyder, & Davidson, 1980). The spotlight of attention was found to be variable in area, depending on the task to be performed (Jonides, 1983; LaBerge, 1983). Eriksen and Yeh (1985) proposed that attention resembled a zoom lens more closely than a spotlight; the power of the lens was inversely related to the area over which attention was spread. That proposal was more directly tested and confirmed in Eriksen and St. James (1986). Subsequent studies showed that not only is the area of the spotlight variable, its shape can be dynamically varied as well (Pan & Eriksen, 1993; Juola, Bouwhuis, Cooper, & Warner, 1991; Eimer, 1999). For example, attention can be directed to an annular region of space, while excluding the region enclosed by the attended region (Egly & Homa, 1984; Eimer, 1999). However, other behavioral and electrophysiological studies suggest that these results have limited generality and attention cannot be arbitrarily allocated across the visual field (Posner et al., 1980; Eriksen & Yeh, 1985; Kiefer & Siple, 1987; McCormick & Klein, 1990; Heinze et al., 1994; McCormick, Klein, & Johnston, 1998). In addition, Eriksen and Yeh (1985) found that if observers were asked to attend simultaneously to a primary location and a secondary location, the time they took to respond to a target at one of these locations was consistent with the observers first directing their attention to the primary location, and then, if necessary, shifting their attention to the secondary location, suggesting that observers were able to attend to only one location at a time. Moving the focus of attention from one location to another has been suggested to be an analogue process (Shulman, Remington, & McLean, 1979). This process is believed to involve three separate phases: the disengaging of attention from the first location, the moving of attention to the second location, and then the reengaging of attention at this location; and these three phases are believed to be controlled by three different anatomical regions of the brain (Posner & Petersen, 1990). 

In contrast to the above unitary beam approach taken by classical attention models, a few studies have suggested that the focus of attention can be distributed over several noncontiguous locations. Shaw and Shaw (1977) proposed that humans in a search task have a fixed total cognitive capacity that they can distribute optimally over space. Castiello and Umilta (1992) suggested that the focus of attention could be split. However, both studies have been criticized for the interpretations of their results (see McCormick et al., 1998). Hahn and Kramer (1998) hypothesized that distractors with abrupt onsets might have reoriented attention in previous studies that did not find evidence for the splitting of attention between noncontiguous locations. They tested this hypothesis using two targets and two distractors on the circumference of a circle, with the separation between the targets being a quarter of the circumference and the distractors positioned between the targets. They looked for distractor compatibility effects when performing a same-different task on the two targets. These effects were measured in two conditions, the abrupt-onset condition with targets and distractors being presented briefly against a blank background and a non-onset condition with targets and distractors constructed by removing appropriate segments from figure-of-eight pre-masks presented at the relevant locations. Distractor compatibility effects were found in the abrupt-onset condition and not in the non-onset condition, suggesting that attention was split between the two target locations in the latter condition. Awh and Pashler (2000) showed that the focus of attention could be split even when the stimuli were presented with an abrupt onset. They used a 5 × 5 stimulus array, consisting of two target numerals and 23 distractor letters. Cues, separated by one element, identified the two target locations with an 80% probability. On invalid trials, one of the target numerals was presented in the array location that was between the two cued trials. Performance for identifying the targets was better at each of the two cued locations on the valid trials than at the intermediate location on the invalid trials, suggesting that the focus of attention was split between the two cued locations. The above two studies used briefly presented stimuli. A more recent study, using steady-state visual-evoked potentials, showed that the focus of attention can be split between separated locations over longer periods of time (Muller, Malinowski, Gruber, & Hillyard, 2003). 

What do these models predict in relation to the present experiments? If focal attention is unitary, then whether the attentional system chooses to process one out of the several trajectories presented, or sequentially scan the different trajectories for processing, or zoom out so that all trajectories are within its “beam,” the prediction is that thresholds would rise steeply as the number of trajectories increases, in cases where only one of the trajectories undergoes deviation. The previous section rules out the possibility that only one of the trajectories presented is processed. Either a scanning or a zooming out hypothesis would predict thresholds that are qualitatively consistent with our current findings. 

The prediction for a distributed model of focal attention would depend on the demands of the task used. Under low-load conditions, we would expect performance to drop relatively slowly for a small number of trajectories, but once the capacity of the system is exceeded, performance should fall steeply. If, however, even when there is only a single trajectory, the task is very demanding and the system capacity is approached, then one should expect that any increase in the number of trajectories would result in a steep drop in performance. In our current paradigm, the trajectory deviations were adjusted to be close to the observers’ deviation thresholds. Under these circumstances, it would be appropriate to classify the task as a high-load task, and expect performance to drop (thresholds to increase) rapidly as the number of trajectories increased from one to two. The high-load predictions of the distributed model of focal attention are consistent with our experimental findings in the threshold paradigm. Whereas our data are consistent with either a single spotlight that scans or zooms out, or a distributed model of focal attention under conditions of heavy load, the experiments of Sears and Pylyshyn (2000) suggest that a single spotlight would be inappropriate for this task (see our Introduction). 

Several studies suggest that attention might be object-based, rather than space-based (i.e., that attention can only be directed at an object, or a collection of objects) (e.g., Duncan, 1984; Egly, Driver, & Rafal, 1994). For our experiments, the simplest predicted outcomes on the basis of object-based theories of attention are fundamentally similar to those of space-based distributed theories discussed previously. Our experiments, therefore, cannot distinguish between attention being distributed between several spatial locations or several independent objects. However, other studies suggest that performance on tasks such as those employed in the current investigation is consistent with an object-based theory of attention (Sears & Pylyshyn, 2000; Scholl, 2001). 

Recent neurophysiological studies provide potential explanations for the elevation of deviation thresholds when there are two or more trajectories. Direction-selective neurons in primate cortical area MT respond more vigorously to motion in their preferred direction when the primate is attending to motion in the neuron’s preferred direction, compared to when attention is directed to motion in the neuron’s null direction (Treue & Maunsell, 1996; Treue & Trujillo, 1999). Attending to a direction of motion enhances the gain of neurons tuned to motion in that direction. In our Experiment 4, when there was only one trajectory in the stimulus, there was only one direction of motion in the stimulus, before the deviation occurred, and attention could presumably be directed to this one direction. This would have enhanced the sensitivity of the associated direction-selective neurons and lowered the deviation thresholds. However, when there are three or four trajectories, each of which could be a potential target, and the orientations of these trajectories are spread over 160° (a jitter of ±80°), it would no longer be feasible to preferentially enhance the responses of the underlying direction-selective neurons in any particular direction, and this would result in low sensitivity in the underlying neurons. This could explain why thresholds increased rapidly when the number of trajectories was increased in Experiments 3, 4, and 6. This would also explain why deviation thresholds increased as the angle of jitter was increased from ±32° in Experiment 3 to ±80° in Experiment 4 (compare the right-most curves in Figure 6 with the curves on the left having the same speeds). In Experiment 2, in which all the trajectories underwent deviation, attention was presumably directed to a randomly selected trajectory and neurons tuned to its direction of motion had their sensitivity enhanced, yielding low deviation thresholds. 

Relevance to signal detection theory

Recent studies suggest that signal detection theory (SDT) can explain some attentional effects in visual search (e.g., Palmer, Ames, & Lindsay, 1993; Palmer, 1994, 1998; Eckstein, 1998; Palmer, Verghese, & Pavel, 2000; Verghese, 2001). Traditional approaches to visual search (e.g., Treisman & Gelade, 1980) invoke a two-stage process to explain the range of search performance observed experimentally. The first stage is preattentive, has unlimited capacity, and operates in parallel at all locations of the visual field, whereas the second is a limited-capacity serial stage that focuses attention on items or groups of items sequentially. The first stage is believed to account for performance in search tasks that do not yield set-size effects (i.e., a reduction in accuracy or an increase in search time as the number of items is increased) (e.g., Palmer, 1998), whereas tasks such as conjunction-search that do show set-size effects, are believed to involve processing by the second stage. SDT provides an alternate approach to analyzing set-size effects in visual search. 

It is informative to discuss whether an unlimited-capacity model could explain our set-size effects, and the implications of our data for limited-capacity models. To apply SDT to our deviation detection task, we need to make some assumptions regarding the detectors used to perform this task. We assume that there exists a hypothetical set of “deviation detectors” with their receptive fields tiling the stimulus plane. This assumption permits us to map our deviation detection task onto the orientation detection task used by Verghese (2001). That study suggested that set-size effects could readily be explained by a single stage, unlimited-capacity model, without the limited-capacity second stage proposed by Treisman and Gelade (1980). Could such an approach explain our data? 

We used computer simulations to address the above question. In principle, our simulations were similar to the simulations in Verghese (2001). The details of the simulations are briefly outlined below. Further details and a more complete set of simulation results are available at http://www.brad.ac.uk/acad/lifesci/optometry/research/projects/ObjectTracking.htm. 

The starting point for the simulations were ST’s thresholds for a single trajectory when all trajectories deviated (from Experiment 2, ST’s leftmost data in Figure 4, lower panel) and when one trajectory deviated (from Experiment 6, ST’s leftmost data in Figure 8, lower panel). Because ST had participated in all experiments, it was appropriate to use his data for the simulations. ST’s data from Experiment 6 were selected because this was the single trajectory condition for which maximum data had been collected (ST repeated the experiment for white, red, green, and blue dots, and because the thresholds were similar, the four sets of data were combined). The large number of trials (2880) ensured a smooth psychometric function and a reliable estimate of threshold. Because our thresholds are specified at 1 SD of the underlying psychometric function, the thresholds are automatically the SDs to be used in the simulations (1.87° — all deviating; 3.89° — one deviating). 

The internal representation of each trajectory deviation was assumed to be a noisy version of the external deviation, with the SD of the noise determined from the single trajectory data above. When simulating a trial of the all-trajectories-deviating case with (say) three trajectories and a deviation of (say) −3°, a random number generator produced three numbers with a underlying mean of −3° and a underlying SD of 1.87°. The simulated response was CCW if the number with the largest absolute value was positive, and clockwise otherwise. The simulated trial was repeated 10,000 times, each time with a different set of random numbers, but with the same underlying statistics, yielding the proportion of CCW responses for a −3° deviation. Repeating the simulations for different values of deviation (positive and negative) yielded a simulated “psychometric” function for the model, similar to that shown in Figure 1. The threshold for the model, for (say) three trajectories, can be determined from this function as discussed in Analysis in General Methods. 

By repeating the entire set of simulations with different numbers of trajectories, we obtained the set of data shown in Figure 9 (upper panel, green filled symbols). For the case where only one trajectory deviates, the procedure was similar, except the SD provided to the random number generator was 3.89° and for (say) three trajectories, and a deviation of (say) −3°, one random number was generated with a underlying mean of (say) −3° and two random numbers were generated with a mean of 0°. By repeating the simulations for different magnitudes of deviation and for different numbers of trajectories, we obtained the data shown in Figure 9 (upper panel, green open symbols). 

Figure 9

 

Comparison of simulation results for an unlimited-capacity signal detection theory (SDT) model with experimental data. Upper panel: Model predictions for thresholds as a function of the number of trajectories when all trajectories deviate (filled green triangles) and when only one trajectory deviates (open green triangles). The dashed line represents a fall off from the one trajectory threshold for subject ST by a factor equivalent to the square-root of the number of trajectories presented for the case when all trajectories deviate. The continuous line represents an increase by the same factor for the case where only one of the trajectories deviates. Also shown are ST’s experimental thresholds for all trajectories deviating (black filled triangles) and for one trajectory deviating (black open triangle). Lower panel: Relative efficiency of observer ST’s performance, with respect to the model performance, when all trajectories deviate (filled triangles), and when one trajectory deviates (open triangles). See text for further details.

In the upper panel of Figure 9, green filled triangles show the simulated thresholds when all of the trajectories deviate. The simulations suggest that the thresholds should decrease from the single trajectory value by approximately the square root of the number of trajectories (dashed line in Figure 9 — calculated from the single trajectory threshold and anchored to the leftmost point, which corresponds to the threshold for one trajectory). Green open triangles represent simulated thresholds when only one of the trajectories deviates. In this condition, thresholds increase from their single trajectory value by a factor close to, but less than, the square root of the number of trajectories (continuous line — again calculated from the single trajectory threshold and anchored to the leftmost point). Also shown in the figure are observer ST’s data from Figure 4 (filled black triangles in Figure 9, upper panel) and from Figure 8 (open black triangles in Figure 9). A logarithmic scale has been used for the ordinate to facilitate comparison between human and simulated thresholds at the lower and upper ends of the scale. The unlimited capacity SDT approach to modeling fails to capture human performance, either in the all-trajectories-deviating case or in the one-trajectory-deviating case. In all cases, when there was more than one trajectory in the stimulus, human thresholds were much higher than the simulated thresholds. 

One way to quantify the difference in performance between the human observer and the simulated observer is to measure the relative efficiency of one with respect to the other (Swets, 1988, pp 132). Absolute efficiency is typically calculated using an ideal observer as a reference and is quantitatively given by the square of the ratio of observed to ideal d’s. Alternatively, absolute efficiency can be calculated from the square of the ratio of ideal to observed thresholds, where both are specified at a fixed value of d’. In our current task, the ideal observer’s performance was far superior to the human observer. Therefore, we used relative efficiencies to quantify the fall off in performance with increase in the number of trajectories; our relative efficiencies were calculated with reference to the simulated thresholds, instead of ideal thresholds. In particular, because our thresholds in the upper panel of Figure 9 are specified at d’ = 1 (see note of caution earlier in the discussion), relative efficiencies were calculated by squaring the ratio of simulated and observed threshold in each condition. A relative efficiency of 1.0 would suggest that the human observer processes all the available trajectories with the same efficiency as when processing a single trajectory. A low relative efficiency would suggest greater loss of information when processing more than one trajectory. 

The lower panel of Figure 9 shows observer ST’s relative efficiency on a log-scale as a function of the number of trajectories. The relative efficiency drops rapidly with more trajectories, particularly when only one of the trajectories deviates (open triangles). The steep drop in relative efficiency highlights the severe loss of positional information when tracking multiple trajectories for deviations. Note that if absolute efficiency had been plotted instead of relative efficiency, using a log ordinate as used here, the shape of the efficiency curves would be the same as shown here, but the curves would be shifted down. Regardless of whether absolute or relative efficiencies are plotted, the human observer’s ability to detect a deviation in a trajectory is severely compromised when there are other nondeviating trajectories presented simultaneously. 

Swets (1988, p. 132–133) suggests possible reasons why the efficiency of a human observer might drop below that of an ideal observer: noise in the decision process, noise inherent in the sensory system, and faulty memory. Which, if any, of these three factors could account for the loss of efficiency as the number of trajectories is increased? 

Noise in the decision process could result from several factors, such as the conscious or unconscious use of a strategy, to preferentially process some trajectories over others, or to preferentially process parts of some trajectories over others (e.g., the portions of trajectories close to the point of deviation), particularly under conditions of high attentional load. This would significantly reduce the efficiency, if the critical information is contained in the part of the stimulus that is not preferentially processed. As demonstrated by the cueing results (Experiment 5, Figure 7), observers are capable of preferentially attending to some trajectories over others, and are more likely to use sophisticated strategies when the attentional load is high. 

Noise inherent in the sensory system is unlikely to explain the difference in thresholds seen between the one-trajectory-deviating and the all-trajectories-deviating conditions. Because the stimuli in the two cases are very similar, one would expect sensory noise in the two conditions to be very similar. 

Faulty memory of the stimulus could also have contributed to the loss of relative efficiency with increasing number of trajectories. The memory could be faulty because parts of the stimulus were not attended to and hence not coded, or they were attended and coded, but were overwritten as subsequent portions of the trajectories were presented. Recent experiments suggest that at the instant of decision regarding the direction of trajectory deviation, observers have poor recollection of the early parts of the trajectories (Narasimhan, Tripathy, & Barrett, 2004). In these experiments, the distractor trajectories disappeared at the mid-line, leaving only the target trajectory to continue on the right side of the monitor. Deviation thresholds still showed large set-size effects, implying that observers could not recall the early part of the target trajectory (also see Sekuler, Siddiqui, Goyal, & Rajan, 2003). 

The simulations using the unlimited capacity SDT approach are interesting from the theoretical point of view. However, they fail to capture human performance, as seen in Figure 9 and in more elaborate simulations (http://www.brad.ac.uk/acad/lifesci/optometry/research/projects/ObjectTracking.htm). The unlimited capacity model does not take into account the strategy employed by the human observer. The model remembers many trajectories with the same accuracy that it remembers one trajectory, unlike the human observers who have poor recollection of the early parts of trajectories when there are many trajectories. A limited capacity SDT model that takes into account strategy and memory capacity might be able to explain human performance. However, this requires understanding the role of memory and the strategies that humans use in performing the current task. This is the goal of our current research. 

Effects of practice

Measures obtained using traditional MOT paradigms generally yield highly variable results (Scholl, personal communications, 2003). In our experiments, we found that performance in the presence of feedback was remarkably consistent across observers, once they were adequately practiced. Thresholds for the different observers were also highly consistent with one another (with the exception of subject DB in Experiment 1), as was the number of trajectories that could be processed. For example, SS’s data in Figure 8 suggest that she was unable to process more than three trajectories, but with a few months practice with this paradigm, her deviation threshold for four trajectories was comparable to the others (new data not shown). Our approach of providing practice over several months ensures that the limit to tracking performance is achieved. Under highly practiced conditions, most observers seem to asymptote to very similar limits. However, with inexperienced psychophysical observers, as in the case of SS, it can take several months of daily practice on tracking tasks to reach this asymptotic level of performance. It is interesting to speculate as to what are the changes that occur during the training phase. As suggested in the previous section, the role of strategy and the role of memory are crucial for good performance. With practice, observers might be refining their tracking strategy, or they might be improving their capacity for recall of the trajectories. 

Relevance to other studies of motion perception

The current study is devoted to measurement of thresholds for detecting changes in the directions of multiple moving objects. Sekuler, Sekuler, and Sekuler (1990) performed a similar experiment using a stimulus that consisted of a single object that moved in a particular initial direction for a specified time period after which it underwent a change in direction. Sekuler et al. (1990) measured reaction times for detecting that the deviation had occurred, as opposed to the accuracy measure employed by us. They found that when there was uncertainty about the initial direction of the object’s motion or the duration of the initial motion phase was brief, reaction times were about 50-ms longer than when the initial direction was held fixed. However, the adverse effect of uncertainty concerning the initial direction of motion disappeared when the initial motion phase duration lasted for 500 ms. A similar effect of uncertainty in initial directions of our trajectories can be seen in our experiments, comparing deviation thresholds for the one trajectory condition across experiments. Orientation jitters were ±3.5, ±32, and ±80° for Experiments 2, 3, and 4, respectively, and the corresponding deviation thresholds for the one trajectory condition for observer ST were 1.9, 3.0, and 6.8° (Figures 4, 5, and 6). The duration of the initial segment of our stimulus was typically much less than 500 ms; and, therefore, there is a noticeable effect of orientation uncertainty in our data. Thus, the effects of uncertainty in the initial direction of a single trajectory, measured in terms of deviation thresholds, mirror the findings that Sekuler et al. (1990) obtained in the reaction time domain. 

Hohnsbein and Mateef (1998) used a threshold duration technique for measuring the ability of the visual system to detect changes in speed and direction of motion for a random dot pattern. In their Experiment 2, the dots moved in a fixed direction for an initial duration, changed to a different direction for an intermediate duration, and then returned to the original direction of motion. They measured the threshold duration for which the dots had to move in a different direction for observers to reliably detect the direction change. Their threshold durations were observed to decrease systematically with increasing angle of deviation. In their study, there were more dots in a stimulus than in ours. However, all of the dots underwent the same deviation, similar to our Experiment 2. As a result, the involvement of tracking cannot be inferred from this experiment, because observers could have tracked a single dot, a subset of dots, or all of the dots. However, a hybrid paradigm, using our multiple-trajectory stimulus with one trajectory deviating and their threshold duration measure, could enlighten us as to the time taken to detect the deviation when there are multiple trajectories. 

The perceived shapes of trajectories

During this study, we observed that the perceived shape of the deviating trajectory could be very different from the bilinear trajectory actually presented. When there are several trajectories, of which one undergoes deviation, quite frequently there is a substantial time delay between when the deviation occurs and when it is detected. This time delay results in the misperception of the location of the deviation, an overestimation of the initial angle of the deviation, and an underestimation of the final angle of the trajectory. The resulting trajectory is perceived to be curvilinear rather then bilinear. These distortions are qualitatively described in Tripathy and Barrett (2003) (also see Brown & Voth, 1937). In our previous study, the distortions were more apparent when the number of distractors was increased. With a greater number of trajectories in the stimulus, less attentional resources would be directed to the target trajectory, resulting in greater latency for detecting the target deviation, and hence greater distortions in the trajectory shape. 

Pursuit eye movements have long been known to distort the shapes of trajectories of moving spots of light (e.g., the pendular whiplash illusion; Dodge, 1904, 1910; Carr, 1907; Ford, 1910; Mack, 1986). The distortions in our trajectories persist even when eye movements are eliminated. Tripathy, Barrett, and Narasimhan (2004) used a stimulus similar to our previous one but with the deviation occurring within the observer’s (monocular) physiological blind spot. A piece of colored paper on the computer screen within the observer’s blind spot provided feedback as to eye movements. Even on trials on which the colored paper was not seen, gross distortions in the perceived shape of the deviating trajectories were still evident, ruling out eye movements from being a primary cause of the distortions. The shapes of these distorted trajectories are not consistent with the simple smoothing of trajectories either. 

Summary

With the threshold-measuring paradigm for measuring spatial precision in the presence of multiple trajectories that we have introduced in this study, observers are unable to accurately process more than one trajectory at a time. Thresholds rise steeply if observers are required to track two or more trajectories simultaneously for deviations. When the discriminability of the tracked event is increased by making the angle of deviation much larger than threshold, as many as 3-4 trajectories can be effectively processed simultaneously (Tripathy, 2003). Our results highlight a severe loss of positional information when attempting to track multiple objects. 

Acknowledgments

We thank David Bundy, Syed Nadeemullah, and Sathyasri Narasimhan for their participation in this study and for discussions and comments. Observer DB’s data formed the basis of David Bundy’s undergraduate project “Localising Non-Uniform Speed Trajectories” submitted to the Department of Optometry, University of Bradford, June 2002. Experiments 1–4 were presented at the Vision Sciences Annual Meeting, Sarasota, Florida, May 2003 (Tripathy, 2003). We are grateful to Dr. Brian Scholl and an anonymous reviewer for their helpful criticisms. We thank Professors Harold Bedell, Tony Norcia, Haluk Ogmen, Suzanne McKee and Robert Sekuler for suggestions, comments, and discussions. Professors Haluk Ogmen and Tony Norcia independently proposed the main experiment on color (Experiment 6). 

Commercial relationships: none. 

Corresponding author: Srimant P. Tripathy. 

Email: . 

Address: Department of Optometry, University of Bradford, Richmond Road, Bradford BD7 1DP, UK.