Perceptual asynchrony between color and motion with a single direction change

Abstract

When a stimulus repeatedly and rapidly changes color (e.g., between red and green) and motion direction (e.g., upwards and downwards) with the same frequency, it was found that observers were most likely to pair colors and motion directions when the direction changes lead the color changes by approximately 80 ms. This is the color–motion asynchrony illusion. According to the differential processing time model, the illusion is explained because the neural activity leading to the perceptual experience of motion requires more time than that of color. Alternatively, the time marker model attributes the misbinding to a failure in matching different sorts of changes at rapid alternations. Here, running counter to the time marker model, we demonstrate that the illusion can arise with a single direction change. Using this simplified version of the illusion we also show that, although some form of visual masking takes place between colors, the measured asynchrony genuinely reflects processing time differences.

Introduction

What is the relationship between the timing of neural activity and the subjective time course of events represented by that activity? This question has recently been discussed in the context of the color–motion asynchrony illusion (Arnold, 2005; Arnold & Clifford, 2002; Arnold, Clifford, & Wenderoth, 2001; Bedell, Chung, Ogmen, & Patel, 2003; Johnston & Nishida, 2001; Moradi & Shimojo, 2004; Moutoussis & Zeki, 1997; Nishida & Johnston, 2002). This striking illusion occurs when a stimulus repeatedly and rapidly changes color (e.g., between red and green) and motion direction (between two opposite directions) with the same frequency. In order to reliably bind one direction of motion with one color, direction changes must occur about 80 ms earlier than color changes (Moutoussis & Zeki, 1997). 

According to the differential processing time explanation (Arnold & Clifford, 2002; Bedell et al., 2003; Moutoussis & Zeki, 1997), the illusion occurs because different attributes of a visual stimulus are processed in relatively separate cortical areas (Livingstone & Hubel, 1988). It is proposed that the subjective time course of changes in a visual attribute is related to the timing of neural activity of the areas that process this attribute. The illusion is thus explained because the neural activity leading to the perceptual experience of motion requires more time than that of color. Alternatively, it has been argued that the perceived time of occurrence may not correlate directly with neural processing time and would be, instead, the result of an interpretative process of the brain (Dennett & Kinsbourne, 1992; Eagleman & Sejnowski, 2000; Johnston & Nishida, 2001; Krekelberg, 2003; Walsh, 2002). 

A hallmark of this illusion is that it seems to require a repetitive display in which the time interval between two successive changes is very short (∼300 ms) (Arnold, 2005; Arnold & Clifford, 2002; Arnold et al., 2001; Bedell et al., 2003; Clifford, Spehar, & Pearson, 2004; Moutoussis & Zeki, 1997; Nishida & Johnston, 2002). Nishida and Johnston (2002) specifically addressed this question. They asked observers to perform a synchronous judgment task; that is, they asked observers to make a yes–no judgment about whether or not the two attribute oscillations were perfectly in phase while varying the relative phase between color and motion changes. They found the color–motion asynchrony illusion when the time between two successive changes was 250 ms, which replicated previous results. However, they showed by increasing the time between two successive changes that the perceived asynchrony decreased until it disappeared when the time interval was 2000 ms. Furthermore, the order of a single change in motion direction and a single change in color was reliably perceived (Bedell et al., 2003; Nishida & Johnston, 2002). These findings posed a challenge to simple versions of the differential processing time explanation that predict a motion delay regardless of the frequency of changes. Nishida and Johnston proposed the time marker explanation as an alternative (Nishida & Johnston, 2002). This explanation for the illusion claims that the misbinding is due to a failure in matching the neural representations (time markers) of the different sort of changes. They argue that at the high temporal frequencies that characterize the illusion, the second-order changes in position (motion changes) are difficult to detect. Consequently, the first-order changes in color are matched with the first-order changes in position (motion direction) resulting in perceived asynchronies. 

However, a recent version of the differential processing time explanation (Bedell et al., 2003) suggests that the critical factor for the appearance of the illusion is the type of judgment that observers perform rather than the mismatching between time markers at high frequencies. The model proposes that judgments of correspondence between attributes (e.g., which direction is the red stimulus moving) imply the use of sustained information for which the differences in processing time between color and motion are significant, whereas temporal order judgments (e.g., did the color change occur after or before the direction change) involve transient information for which the differences in processing time are not significant. Accordingly, the illusion will appear only when a correspondence task will be performed, but not for a temporal order judgment task. 

However, as previously mentioned, Nishida and Johnston (2002) required observers to perform a synchronous judgment task. This task is more similar to a temporal order judgment task than to a correspondence task. Therefore, the differential processing time model of Bedell et al. (2003) would not predict an asynchrony. Nishida and Johnston, however, did observe the typical asynchrony for high frequencies. How can this apparent contradictory result be explained? Bedell et al. proposed that observers may have changed the task depending on the temporal frequency. That is, although observers were asked to perform a synchronous judgment task, they may have instead performed a correspondence task for high frequencies. 

In the present study, we show that, in opposition to the time marker explanation, a single change in motion direction demonstrates the perceived asynchrony that characterizes the illusion. Using this simplified display we show that, although it does not contribute to the measured asynchrony, some form of visual masking takes place between colors. Furthermore, we also show that motion signals briefly displayed after a direction reversal are not perceived, revealing what we think is the origin of the illusion. 

General methods

The stimuli were displayed on a 21-in. CRT monitor (Sony Trinitron GM 520) at a refresh rate of 100 Hz viewed binocularly from a distance of 50 cm in a dimly lit room. It consisted of 200 dots (size: 0.11° × 0.11°) randomly distributed in a circular aperture with a diameter of 8° of visual angle on a dark background. All dots moved coherently at 6°/s. The dots could be red (luminance: 17 Cd/m ²; chromaticity: 0.62, 0.34) or green (luminance: 17 Cd/m ²; chromaticity: 0.28, 0.61). Three observers participated in the experiment, the first author and two observers who were naïve as to the purposes of the study. Observers reported normal, or corrected to normal, visual acuity and color vision. Observers were instructed to maintain fixation on a cross, presented at the center of the aperture. A demonstration of the stimuli can be found at http://www.ub.edu/pbasic/visualperception/joan/en/demos.html. 

Experiment 1

According to the time marker explanation (Nishida & Johnston, 2002), the use of a repetitive display is a necessary condition for the illusion to appear. However, according to the differential processing time explanation suggested by Bedell et al. (2003), the critical factor is the type of task: performing a correspondence task would lead to perceptual asynchrony whereas temporal order judgments would not. Bedell et al., however, used a repetitive display and so it is not entirely clear whether the correspondence task requires repetitive changes. Here, we test both explanations by performing two types of tasks (a correspondence task and a temporal order judgment task) using a variation of the typical cycle displays (Arnold, 2005; Arnold & Clifford, 2002; Arnold et al., 2001; Bedell et al., 2003; Clifford et al., 2004; Moutoussis & Zeki, 1997; Nishida & Johnston, 2002) in which only a single change in motion direction was presented. 

Methods

Correspondence task

We studied the perceptual binding of color around a single direction change when observers performed a correspondence task. Three experimental conditions were randomized within each session:

•  
  180° direction change: The stimulus (diagrammatic representation of the temporal relationship between color and motion changes in Figure 1A) consisted of 200 random dots that were displayed for 300 ms moving either upward or downward (at random in each trial). After this interval, the dots abruptly reversed direction maintaining it up to the end of the trial. The color of the dots was red during an interval of 300 ms close to the direction change. Before this red interval the dots were green, and after that the dots were green again for 300 ms, after which the dots disappeared. Consequently, the display consisted of one direction change and two color changes. The relative timing between the red interval and the direction change was varied from trial to trial (see below).
•  
  90° direction change: The stimulus (diagrammatic representation in Figure 1B) was the same as described above except that the direction change was 90° either to the right or to the left (at random in each trial).
•  
  Without last color interval: The stimulus (diagrammatic representation in Figure 1D) was the same of the 180° direction change condition except that the dots disappeared after the red interval. That is, the last green interval was not displayed.

Figure 1

 

Diagrammatic representation of the temporal relationship between color and motion for each condition and results in form of individual psychometric functions in Experiment 1. (A) A single change in motion direction was sufficient to obtain the asynchrony between color and motion. The distribution of solid circles shows the proportions of trials in which the second direction of motion was reported for red dots as a function of the relative timing between the direction change and the center of the red interval. (B) A direction change of 90° strongly reduced the asynchrony (distribution of diamonds). (C) When a temporal order judgment was made between the direction change and the first change in color no asynchrony appeared (unfilled circles). (D) Removing the last green interval in the first condition resulted in the same asynchrony. The horizontal segments around the PSE correspond to 95% confidence intervals of the mean of the cumulative Gaussian function obtained with bootstrap method.

Observers performed a motion correspondence task; that is, they made judgments about the predominant direction of motion (the first or the second direction of motion) when the dots were red (two forced-choice judgment). The relative timing between the direction change and the red interval varied from trial to trial. We assigned a relative timing of zero in the situation where the red interval was centered on the direction change. Hence, the dots were red during 150 ms before and 150 ms after the direction change. We denoted positive values of the relative timing for those values in which the direction change occurred before the point of time that corresponded to the center of the red interval and negative values for those in which the direction change occurred after that point. Eight relative timings ranging from −100 to 250 ms in increments of 50 ms were used. 

Each observer performed two different sessions: one in which the target color was red (as described above) and another in which the colors were reversed and the target color was green. The order was counterbalanced across subjects. For the sake of clarity, hereafter we will refer to red as the target color. During a session, each relative timing was sampled 20 times according to the method of constant stimuli. 

Temporal order judgment task

The stimuli were identical to the 180° condition. The only difference was the task performed by the observers. In this case, whether the direction change occurred before or after the first color change was to be reported (diagrammatic representation in Figure 1C). For this task, we assigned a relative timing of zero to the situation in which the color and the motion change were physically synchronous. Positive and negative values corresponded to the situations in which the color change followed or preceded, respectively, the direction change. Eight relative timings ranged from −150 to 200 ms in steps of 50 ms were used. The sessions were administered exactly as in the correspondence task, as were the relative timing values. Subjects performed the task in different orders. 

Data analysis

For the correspondence task, the set of data of each participant provided a distribution of the proportion of trials in which the second direction was paired with red color as a function of the relative timing between the direction change and the center of the red interval. For the temporal order judgment task, the distribution denotes the proportion of trials in which the first color change is reported as occurring after the direction change. We fitted cumulative Gaussian to derive the point of subjective equality (the mean of the distribution), which served as a measure of the perceptual asynchrony. If the pairing was veridical, the distribution would be centered on 0. Positive values would correspond to apparent motion delays (motion–color asynchrony illusion) and negative values to apparent color delays. In order to obtain the 95% confidence intervals of the parameters of the cumulative Gaussian functions, we used the parametric bootstrap method (Efron & Tibshirani, 1993) as conducted by Kanai, Sheth, and Shimojo (2004). When conclusions could not be drawn by merely looking at the overlap between two confidence intervals, parametric bootstrap and Monte Carlo simulations were used to compare two given psychometric curves by testing the null hypothesis that the observed difference between points of subjective equality PSEs (or the two slopes) is not different than zero. To accomplish this, we used the same procedure as that implemented in PFCMP (Wichmann & Hill, 2001a, 2001b); however, we computed a bootstrap p value independently for each parameter (López-Moliner & Linares, 2006) instead of a combined (PSE and slope) one as was carried out in PFCMP. 

Results and discussion

Figure 1 (solid circles) shows the proportion of trials in which the second direction of motion was reported for red dots (correspondence task) as a function of the relative timing between the 180° direction change and the center of the red interval (individual results). If the pairing was veridical, the distribution of results would be centered on a relative timing of 0. However, the distribution is centered around 73 ms (mean across observers) reflecting the typical delay in the perception of motion. Actually, when the red interval was physically centered on the direction change (relative timing of zero), observers bound the second direction of motion with red in only 5% of the trials. 

One important property of the illusion is that it depends on the angle between the two directions of motion (Arnold & Clifford, 2002; Bedell et al., 2003; Clifford et al., 2004), with a direction change of 180° producing the maximum asynchrony. Thus, if the angle decreases, so must the illusory effect. Precisely, this result is the main current evidence contrary to the time marker model because a direction change is a second-order change independently of the specific directions of motion. In order to make sure that the measured shift reflected a perceived asynchrony not confounded with response biases in favor of the first direction, we also measured the magnitude of the illusion for an angle of 90° between directions (distribution of solid diamonds in Figure 1). As expected, the asynchrony decreased significantly (p < .001). The average value of the asynchrony for this case was 26 ms. 

We have thus shown that when a correspondence task is performed, against the time marker explanation, a single direction reversal suffices to obtain a similar asynchrony between color and motion as those reported elsewhere using repetitive displays (Arnold, 2005; Arnold & Clifford, 2002; Arnold et al. 2001; Bedell et al., 2003; Clifford et al., 2004; Moutoussis & Zeki, 1997; Nishida & Johnston, 2002). 

For the temporal order judgment task, in agreement with previously reported results (Bedell et al., 2003; Nishida and Johnston, 2002), we found that observers judged the temporal order very accurately (distribution of unfilled circles in Figure 1, notice that in this case the relative timing is between the direction change and the first color change and the ordinate represents “direction change occurring first”). This shows that the critical factor is the task; hence, the obtained patterns were not an artifact of the new stimulus we have used. 

Repetitive stimulation has been considered a necessary condition for the color–motion asynchrony illusion to occur (see for example the review of visual illusions of Eagleman, 2001). Explaining this has proven to be a major obstacle to the differential processing time explanation because there is no reason why the latency for perceiving a change of motion direction, for example, should depend on whether this change appears in isolation or is embedded in other direction changes. Here, however, we solve this problem by showing that a single direction change is sufficient for motion to be systematically bound to a color that appears later in time. 

According to the time marker explanation, the illusion arises because direction changes are difficult to detect when displayed at high frequencies. In the present investigation, although only used a single directional change, an attempt was made to maintain the similarity with previous studies reporting the illusion. Accordingly, we showed the first direction of motion for a very short time (300 ms). It could be argued that our results can still be accommodated by the time marker explanation considering that when motion is displayed for such a short interval of time, the detection of the direction change is impaired as in the case of several direction changes displayed at high frequencies. However, our finding showing accurate temporal order judgments demonstrates that the direction change could be correctly detected. Furthermore, it could also be suggested that the detection of the direction change is only impaired when a correspondence task is performed. But in this case, as the task involves the use of sustained information, it is not clear how the time marker explanation, which is formulated for temporal order judgments between events, can be applied. 

Therefore, our results run counter to the time marker explanation (Nishida & Johnston, 2002) and are consistent with the differential processing time explanation of Bedell et al. (2003) for which the critical factor to obtain a perceptual asynchrony consists in performing a correspondence task between attributes. Interestingly, a similar dependence on task was found using the attributes of color and orientation (Clifford, Arnold, & Pearson, 2003). However, the question still remains as to why Nishida and Johnston (2002) find the illusion for rapid alternations even when they asked observers to perform a synchrony task (that is similar to a temporal order judgment task). We think, as Bedell et al. proposed, that for rapid alternations subjects might perform a correspondence task instead of a synchrony task. As subjects were explicitly asked to make synchrony judgments, we suggest that this hypothetical switch they made to a correspondence task might be due to the impossibility of performing a synchrony task under conditions of rapid alternations. We think that this may somehow be related to attentional limitations. 

In order to maintain consistency with typical repetitive displays, we presented 300 ms of green color following the color target interval (red). However, according to the differential processing time explanation the asynchrony should occur regardless of the presence of this last color interval. To further ensure that the asynchrony that we measured is not caused by an influence of the last green interval over the red interval following the direction change, we eliminated the last green interval ( Figure 1D). Indeed, the distribution corresponding to this manipulation (solid triangles in Figure 1) completely overlaps with the one having green as the last color. Hence, consistent with the differential processing time explanation, the perceived asynchrony does not depend on the presence of the last color interval. 

However, upon questioning the observers, all of them reported experiencing a different percept when the last green interval was not present (compare the two demos shown in the Web page): they observed the red color moving in the two directions of motion more frequently. In contrast, they saw red moving in the second direction of motion much less often when the last green color interval was present. As observers were required to bind red to the predominant direction, this misperception of red color in the second direction would not affect the measured asynchrony. So as to explore this possible influence of the last green interval in the perceived color after the direction change, we carried out Experiment 2. 

Experiment 2

In Experiment 1 we showed that, inconsistent with the time marker explanation, the perceptual asynchrony between color and motion can be observed when a correspondence task is performed on a single direction change. We also showed that the interval of color that followed the color on which the task was performed did not contribute to the measured asynchrony. However, informal reports of observers suggested that the color of the stimulus presented after the color target had an effect on the visibility of the color target. In Experiment 2, we explored this issue using a similar stimulus, but changing the task: observers were asked to report the number of directions they saw for the target color when the last color interval was present or absent. We hypothesize that if the last green color interval makes the perception of the red color difficult after the direction change, then the red color moving in two directions would be reported less frequently with respect to the situation where the last green color interval was absent. 

Methods

The stimuli were very similar to that of Experiment 1 except that red is always presented 150 ms before the direction change and the amount of red after the direction change was varied on a trial-to-trial basis (see Figure 2). In this experiment, observers had to report whether they saw one or two directions of motion for red while varying the presence ( Figures 2B and 2D) or absence ( Figures 2A and 2C) of the last green color interval. We used two angles for the direction change: 180° ( Figures 2A and 2B) and 90° ( Figures 2C and 2D). Within one session, each relative timing value was sampled on 20 occasions according to the method of constant stimuli. Each observer performed two sessions with the color target being different in each one. 

Figure 2

 

Diagrammatic representation of the temporal relationship between color and motion for each condition and data pooled over all participants in Experiment 2. Observers had to decide whether red color was perceived in one or two directions of motion while the relative timing between the direction change and the offset of red color was manipulated. (A) Direction change of 180° without last green interval. (B) Direction change of 180° with last green interval. (C) Direction change of 90° without last green interval. (D) Direction change of 90° without last green interval.

Results and discussion

Figure 2 shows the proportion of two directions reported for red while varying the amount of red presented in the second direction (data pooled over all participants). The mean of the distribution indicates how long red must be shown along the second direction for observers to report two directions for red color in 50% of trials. 

When the last green color interval was absent we found a mean of 55 ms for the 180° direction change (distribution of triangles in Figure 2) and a mean of 19 ms for the 90° direction change (distribution of diamonds in Figure 2). These results show that when the dots move along the second direction for very short times, this direction of motion is not perceived. Interestingly, the mean is significantly reduced for the direction change of 90° with respect to the 180° condition ( p < .001) and in a magnitude very similar to that reported in Experiment 1. This suggests that the perceptual delay in motion perception, as measured in Experiment 1, reflects that motion signal needs integration time to reach awareness (Burr & Santoro, 2001) and that this time depends on the former direction of motion. As the maximum asynchrony is obtained when reversing the direction, we think, as stated elsewhere (Arnold & Clifford, 2002; Bedell et al., 2003; Clifford et al., 2004), that mechanisms of opponency in MT underlie this perceptual asynchrony (Snowden, Treue, Erickson, & Andersen, 1991). 

When the last green interval was present, the red dots had to be shown moving along the second direction for a slightly longer time (mean of 73 ms) to be equally perceived moving in one or two directions in comparison to when green was absent ( p < .001 for the 180° condition distribution of squares in Figure 2). For the 90° condition (distribution of circles in Figure 2), this difference was not significant. In agreement with an effect of the last green interval, one would expect larger differences in the PSE (e.g., curve shifted further to the right when the green is present). We think that the lack of such an effect might have been caused by a response bias that shifts the curve to the left. The fact that subjects responded in several trials “two directions” for the relative timing of 0 for which red only is presented in one direction may reflect this bias. In any case, it is clear that the last green interval has an effect on the visibility of the red interval, which is shown by the significant variation of the slopes for the two angle conditions. The standard deviation increased from 25 to 97 ms for the 180° condition and from 11 to 64 ms for the 90° condition presumably reflecting a higher difficulty of the task. 

Our findings share similarities with previous results published by Moradi and Shimojo (2004). They also reported a misbinding of color and motion within a single event. In one condition of their Experiment 5, observers were asked to report the color of a briefly moving surface that became perceptually segregated. During motion, the dots inside this surface were gray and the color switched back to green when they stopped. Subjects reported the color following motion offset more often than gray, which was the actual color during motion. Moradi and Shimojo regarded this result as the onset of a new surface triggering the analysis of the properties (including color) of the surface. They suggested that these properties are computed during a temporal window of 50–150 ms following the onset of the surface. Remarkably, the gray color was not perceived even when presented for 120 ms. As the temporal window of analysis lasts for 150 ms at the most, this implies that the color is not treated uniformly during the time window of analysis (Arnold, 2005) but it is temporally weighted distinctly favoring color information available later in time. 

We think that another, and maybe simpler, explanation of the effect of the last color interval could be backward masking (Bachman, 1994). According to it, the last green color interval would mask red color in the last part of the red color interval. This account does not require any direction change to work. Thus, regardless of the direction change, a fixed quantity of red color would be masked. Although, it may seem contradictory, we think this is consistent with the fact that the difference (p < .001) between the two direction changes (90° and 180°) has the same trend as when the last color interval is absent: the last green color interval could mask a physically longer interval of red presented after a direction change of 180° simply because the first part of motion is not perceived in this case (last interval absent condition). 

In summary, taking into account the results of Experiments 1 and 2, we believe that in a typical cycle display with several alternations, the perceived asynchrony genuinely reflects differential processing latencies between color and motion: perceptual experience of motion is delayed when motion in the opposite direction is previously displayed. However, we show that some form of visual masking also contributes to the percept making the perception of color after direction changes more difficult and facilitating the pairing of colors with directions presented before direction changes. 

Conclusions

The color–motion asynchrony illusion has been used to study the temporal perception of events and the binding of visual attributes. The asynchrony is typically measured using repetitive displays in which the relative phase between motion direction changes and color changes is varied using the entire range of the cycle. One advantage of such displays is that any response bias should be eliminated. Here, we did not follow this tradition because we were interested in the binding between motion and color when only a single change was displayed. Although this might have created some response bias, it cannot account for the whole perceptual asynchrony we found as shown by the dependency on the angle. 

In Experiment 1, using a single direction change display, we showed that the perceived asynchrony cannot be explained according to the time marker explanation. However, the results were perfectly compatible with the differential processing time explanation for which the differences in processing time between attributes result in perceptual differences. Importantly, the asynchrony arose when a correspondence task was made, but not for temporal order judgments. This means that the perceptual delay of motion is not a fixed quantity but depends on the mechanisms engaged in the task. Normally, the delay in motion perception is associated with mechanisms of opponency of MT. The results of Experiment 2 without a last color interval following the color target support this view: motion needs integration time below a threshold to be perceived, and this integration time depends on the direction of motion before the change being maximum when the directions are totally opposed. 

Although in Experiment 1, we showed, using a typical task of pairing, that the color interval that follows the color target interval has no effect on the measured asynchrony, the results of Experiment 2 with a color interval following the color target showed that this last color interval had a perceptual effect of masking. 

Acknowledgments

We thank Alex Holcombe, Hans Supèr, Scott Sinnett, and two anonymous reviewers for their comments and suggestions. This research was supported in part by Grant TINS2004-04363-C03-02 from the Ministerio de Educacion y Ciencia of the Spanish government. 

Commercial relationships: none. 

Corresponding author: Daniel Linares. 

Email: [e-mail]. 

Address: Hospital de Sant Joan de Deu, Edifici Docent, C/Santa Rosa, 39-57, 08950 Esplugues, Barcelona, Spain.