Veridical perception of moving colors by trajectory integration of input signals

Abstract

For rapid alternations of two colors (e.g., red and green), human observers see the mixed color (yellow). This chromatic flicker fusion has been considered to reflect neural integration of color signals presented successively at the same retinal location. If so, the retinal alternation rate should be a critical fusion parameter. However, here we show that temporal alternations of two colors on the retina are perceptually segregated more veridically when they are presented as moving patterns rather than as stationary alternations at the same rate. This finding is consistent with the hypothesis that the visual system integrates color signals along the motion trajectory, in addition to at the same retinal location, for reducing motion blur and seeing veridical colors of moving objects. This hypothesis is further supported by a covariation of perceived motion direction and perceived color in a multipath motion display.

Introduction

When two colors (e.g., red and green) are presented with rapid alternations, human observers see the mixed color (yellow). This is because the visual system integrates sensory signals over time, a property that is considered to improve the signal-to-noise ratio of neural color signals. Previous studies (De Lange Dzn, 1958; Uchikawa & Ikeda, 1986; Wisowaty, 1981; Wyszecki & Stiles, 1982) have only considered temporal integration of color signals presented at the same retinal location. Longer temporal integration time for color perception than for luminance perception is indicated, for instance, by relatively low critical fusion frequency (at which rapidly alternating stimuli are perceptually fused into a stable pattern) for color perception (De Lange Dzn, 1958; Wisowaty, 1981). A sluggish integration of color signals on the retinal coordinates should lead to severe image degradation when the retinal color distribution changes rapidly as a consequence of either object motion or the observer's eye movements. This is comparable to the motion blur that photographers often encounter when taking pictures with a long exposure time. Nevertheless, we do not normally perceive such color blurs. Why is this? 

Here we show that alternations of different colors at the same retinal location are more clearly segregated (i.e., less mixed) in moving patterns than in stationary patterns flickering at the same alternation rate. This phenomenon, “motion-induced color segregation,” demonstrates that the magnitude of retinal fusion of drifting colors is not as strong as simply expected from flicker fusion. The motion-induced color segregation occurs regardless of the color modulation axis. It cannot be ascribed to eye movements or to a change in total intensity. This finding indicates the existence of a special neural mechanism in human visual system that actively suppresses temporal blurs of moving colors. 

It is suggested that the visual system suppresses subjective motion blur of luminance-defined patterns (Burr, 1980). A proposed mechanism for this motion deblurring is neural integration of visual signals along the motion trajectory (Burr, 1981; Burr & Ross, 1986; Burr & Ross, 2004). We recently found a phenomenon that suggests that the visual system also integrates color signals along the motion trajectory. Moving bars whose colors alternates between two colors are perceived as the mixed color to human observers although the two colors are not superimposed on the retina (Nishida, Watanabe, Kuriki, & Tokimoto, 2007). Theoretically, trajectory integration of color signals can help us see veridical colors of moving objects, in which colors are kept along motion trajectories, by reducing motion blur while maintaining a high signal-to-noise ratio. We consider that the present finding of motion-induced segregation of colors, which are kept along the motion trajectory while alternated at the same retinal location, is the evidence supporting this theoretical prediction. The contribution of motion to color perception is further supported by an experiment reported in the second part of this paper—using a multipath motion stimulus, we observed a change in the perceived color in accordance with a change in perceived motion direction in a way predicted by the trajectory integration theory. 

General methods

Observers

Observers in each experiment were two of the authors and two to five volunteers. Most of the volunteers were unaware of the purpose of the experiments. All the observers had normal or corrected-to-normal vision. 

Apparatus

Stimuli were displayed on a GDM-F520 CRT monitor (Sony), with a refresh rate of 160 Hz, driven by a VSG2/5 visual stimulus generator (Cambridge Research Systems Ltd.) installed in a Precision 350 workstation (Dell). The spatial resolution of the monitor was 800 × 600 pixels, with each pixel subtending 1.5 min at the viewing distance of 113 cm. The observer viewed the monitor while sitting in a dimly illuminated room with his or her head fixed on a chin rest. 

Color specification

A linear scale of color mixture was made by modulating the intensities of red (CIE, 1931; x = 0.625, y = 0.341) and green ( x = 0.290, y = 0.606) phosphors. Two colors of a given red–green pair were aI _red + bI _green and bI _red + aI _green, where I _x is full contrast intensity of color x, a > b, and a + b = 1. The chromatic contrast, defined as a − b, was 100%, 75%, 50%, 25%, and 0% for M (mixture magnitude) = 1, 2, 3, 4, and 5, respectively. This gives rise to a pair of red and green for M = 1 (perfect segregation) and a pair of identical yellows for M = 5 (perfect mixture). Under the maxluminance condition, each phosphor was driven at maximum intensities of the monitor at which point M = 1 ( I _red = 28 cd/m ² and I _green = 84 cd/m ²). Under the equiluminance condition, I _red = 28 cd/m ² and I _green was defined individually by means of flicker photometry (Wyszecki & Stiles, 1982), or by a minimum motion method (Anstis & Cavanagh, 1983). The chromaticity of yellow varied depending on the luminance setting of I_green. For S-axis modulation, a similar scale of color mixture was made between (x = 0.409, y = 0.492) and (x = 0.233, y = 0.123), 30 cd/m². 

Motion-induced color segregation

Phenomenon

The basic phenomenon of motion-induced color segregation is as follows. In a stimulus consisting of red and green bars separated by constant intervals ( Figure 1A), the bars move at a constant speed without changing color ( Figure 1B). Bright bars are presented on the dark background for producing strong luminance-based motion (Cavanagh & Mather, 1989). Given fixation of eyes, red and green stimulation alternates at each retinal location at a rate dependent on the interbar interval. Compare this color-keeping motion stimulus with a flicker pattern, in which retinally local color changes in the motion pattern are viewed through stationary slits (Figure 1C, see also Movie 1). The two stimuli should seem to have the same colors, if the apparent color mixture is solely determined by retinal temporal integration because the colors alternate at the same rate on the retina. However, under certain temporal conditions, whereas only a fluctuation between yellowish colors is seen in the flicker pattern, moving red and green bars are clearly visible in the motion pattern. 

Figure 1

 

Stimuli used to test motion-induced color segregation. (A) Spatial configuration and (B) space–time plot of the color-keeping motion stimulus. The color integrated over time at each retinal position (yellow) is shown at the bottom, and the colors integrated along the motion trajectory (green and red) are shown at the right. (C) Space–time plot of the control flicker stimulus, whose spatial configuration is the same as panel A.

 

Movie 1

 

This is not a demonstration but an explanatory slow movie of the stimuli for motion-induced color segregation. A moving pattern ( Figure 1B) is shown above, and a flickering pattern ( Figure 1C) is shown below. They have the same local color flicker (alternation) rate. The expected effect is stronger color segregation in the moving pattern than in the flickering pattern. In this pattern, the interbar interval was as three times wide as the bar width, which is the same as described in Figure 1. Demonstration of this effect requires fast frame rates (at least 60 Hz, 160 Hz in our experiments). Demonstrations for Mac OS X and Windows developed using an Open-GL library are available on a pubic database, Visiome Platform.

This phenomenon suggests that motion information makes the magnitude of color fusion less than expected from that of nonmoving flicker. Specifically, it can be explained by a hypothesis that the color signals are integrated along a motion trajectory, suggested by the color mixing phenomenon of the moving color-alternating bars (Nishida et al., 2007). According to this hypothesis, colors should be veridically segregated for the motion case because the color does not change along the trajectory of moving bars (diagonal direction in the space–time plot), whereas motion should be mixed for the flicker case because the color alternates along the trajectory of stationary bars (the vertical direction in the space–time plot). 

We called this phenomenon “motion-induced color segregation” simply because alternations of different colors at the same retinal location are phenomenologically more segregated in motion stimuli. We do not assume such a complex process that different colors are once integrated at the same retinal location by a peripheral mechanism and then segregated again by a subsequent motion-sensitive mechanism, which would be impossible in the present case (see Discussion). Our conjecture is that integration of alternating colors along stationary trajectory may enhance color fusion for flicker stimuli. This is the same mechanism underlying color fusion along color-alternating trajectories in “motion-induced color mixing” (Nishida et al., 2007). The two apparently opposite effects (motion-induced color mixing and segregation) are complementary functions of the same mechanism. 

Rating

We verified the observation of motion-induced segregation in a rating experiment. 

Methods

A (color-keeping) motion stimulus consisted of two horizontal arrays (20° in width) of vertical bright bars (each subtending 6 min in width, 1.0° in height) presented on a dark background 1.0° above and below fixation ( Figures 1A and 1B). In each array, red and green bars (when M = 1 of the red–green condition) spatially alternated with a constant interval. The bars shifted every 6.25 ms by one bar width keeping color constant, in opposite directions in the upper and lower arrays. The color alternation rate at each retinal location increased from 6.7 to 40 Hz (for S-axis stimuli, from 4 to 26.7 Hz) as the interbar interval decreased from 66 to 6 min (from 114 to 12 min). Initial bar positions were randomized. The control flicker stimulus ( Figure 1C) was made from the motion stimulus by removing bars to be presented in the interbar areas of a given frame. In each trial, a test stimulus was presented for 1000 ms, with 300-ms pre- and postmasks of stimulus onset/offset. The mask pattern was a uniform yellow presented in the area of the bar arrays. 

We asked observers to evaluate the magnitude of subjective color mixing by means of a 5-point rating scale (with referencing to physical samples of various chromatic contrasts) for the motion and flicker stimuli while systematically changing the local alternation rate. In each trial, a stimulus presentation was followed by a presentation of a static color sample. The sample consisted of five pairs of colored bars showing five mixture magnitudes, and the observer had to choose the sample number closest to the impression of perceived colors. Alternatively, the color sample was bicolor bar arrays presented with one of five mixture magnitudes, and the observer varied the mixture magnitude to match sample colors to the perceived test colors. We confirmed that the results obtained with this “matching” procedure were not significantly different from those obtained with the “rating” procedure. The observer could request replays of the stimulus sequence. 

To check the reliability of the rating, we showed the stimulus with three magnitudes of color mixture, each corresponding to the reference sample colors for rating 1 (perfect segregation, equivalent to 100% chromatic contrast), 3 (moderate mixture, 50%), and 5 (perfect mixture, 0%). Because these colors were shown in random order, the observers could not tell whether a given stimulus that appeared yellow to them was indeed yellow or a mixture of red and green. 

Between trials within a single session, the stimulus type, the bar interval, the color mixture magnitude, and the motion direction were randomly varied. 

Results

As the local alternation rate was increased (by reducing the interbar interval), the perceived colors gradually changed from the veridical perception of the physical colors presented (indicated by an orange dotted line in each panel) to complete yellow, which ensures the accuracy and the consistency of the observers' rating. The increase of the mixture rating with increasing alternation rate for the flicker pattern can be accounted for by the known temporal summation properties of the color system. The critical finding is that in comparison to the flicker pattern, the color mixture rating was lower (thus color segregation was stronger) for the motion pattern for nearly all the alternation rates examined. 

Similar results were obtained regardless of whether the red and green phosphors were driven at the maximum intensities of the monitor (maxluminance, Figure 2A) or the colors were subjectively equiluminant ( Figure 2B), whereas the mixture rating was slightly higher in the latter case. Note that the background was dark in either case (see the Control experiments section for the effect of equiluminant background). Spatial scale is not a critical factor. A strong motion-induced color segregation was observed even when the bar width was increased from 6 to 18 min ( Figure 2C). Color axis is neither a critical factor. Although the red–green color axis we used was close to the L–M axis in the standard cone contrast color space (Derrington, Krauskopf, & Lennie, 1984; MacLeod & Boynton, 1979), a similar effect was observed for the S-axis stimuli (Figure 2D). 

Figure 2

 

Rating of color mixture as a function of the frequency of local color alternation. (A) Red–green maxluminance condition. (B) Red–green equiluminance condition. (C) Red–green equiluminance bar width ×3 (18 min). (D) S-cone modulation condition. Each data point shows the average for seven (A, B), five (C), or four (D) observers (six judgments for each). The results suggest that apparent color mixture is weaker for the motion stimulus than for the flicker stimulus.

Chromatic contrast detection

Although the rating method was good to directly evaluate motion-induced changes in subjective color appearance, it could introduce some ambiguity into quantitative evaluation of the magnitude of the effect. To overcome this limitation, we also evaluated the effect of motion on color perception in terms of a more objective performance measure. Specifically, we measured the chromatic contrast detection threshold; that is, the contrast below which a color alternation was perceptually indistinguishable from the mixed color. We expected that the stronger the magnitude of perceptual color segregation was, the lower would become the contrast required for correct judgment. 

Methods

We presented a color-keeping motion (or a control flicker stimulus) of a given chromatic contrast (target), together with a nontarget stimulus that was identical to the target except that the chromatic contrast was 0%. The background was dark. The observers had to make a two-alternative forced choice judgment about which stimulus was the target. In a single trial, a target of a given chromatic contrast and a nontarget of 0% contrast were simultaneously presented, one above and the other below fixation. The observer had to keep fixating at the central point during stimulus presentation and then indicated the target position by pressing one of two buttons. The color condition was red–green equiluminance. Unless otherwise noted, other stimulus parameters were the same as those used for the rating experiments. Chromatic contrast was varied within a block, whereas stimulus type (color-keeping motion or flicker) and the alternation rate (6.67 or 13.3 Hz) were varied between blocks. Alternatively, the alternation rate was varied within a block whereas the chromatic contrast was kept at 50%. 

A preliminary test indicated that at very high temporal frequencies, the drifting motion of color-keeping stimulus was visible for high-contrast stimuli but not for 0% contrast stimuli due to contribution of the chromatic contrast (and residual luminance contrast) to motion disambiguation. To avoid the difference in motion strength as a discrimination cue, we added luminance (yellow) noise to one of the color components. Specifically, when the luminance noise was added to say target red (randomly chosen for each trial), target red = ( a + L _t) I _red + ( b + L _t) I _green; target green = bI _red + aI _green; nontarget = 0.5 I _red + 0.5 I _green; or (0.5 + L _n) I _red + (0.5 + L _n) I _green, where L _t and L _n are luminance noise amplitudes randomly and independently determined between 0.25 and 0.5. As a result, the maximum chromatic contrast ( a − b) was 50%. With the addition of luminance noise of large and random magnitudes, the observer saw motion for both the target and the nontarget, and the apparent motion strength was not correlated with stimulus chromatic contrasts. 

Results

Either at 6.67- or 13.3-Hz alternation, the threshold chromatic contrast obtained under the motion condition was only about one-third of that obtained under the control flicker condition ( Figure 3A), suggesting that the color-keeping motion facilitated color segregation in agreement with the rating results. We also varied the alternation frequency while remaining the chromatic contrast at 50% ( Figure 3B). The result showed that the threshold temporal frequency was considerably higher for the motion condition than for the flicker condition (24.2 vs. 13.8 Hz). 

Figure 3

 

Motion-induced color segregation indicated by differences in the threshold of discriminating stimuli with and without chromatic contrast. The stimulus was a color-keeping motion (open blue square) or a control flicker stimulus (filled red circle). Proportion correct of chromatic contrast detection as a function of the target chromatic contrast (A) or as a function of alternation temporal frequency (B). Each data point shows the average for five observers (40 judgments for each). The smooth curves are best-fit logistic functions, and the arrows indicate the 75% correct points. That the proportion correct is slightly higher than the chance level even at the highest temporal frequency in panel b could be ascribed to occasional involuntary eye movements that we could not completely exclude.

Control experiments

This section describes the results of control experiments that we carried out to address several concerns about our interpretation that motion-induced color segregation is a result of trajectory integration of color signals. 

Effects of eye movements

Firstly, one might suspect possible artifacts due to eye movements. If the observer's eyes track the stimulus movement, the conventional retinal color integration mechanism could account for our findings. However, our observers were asked to fixate the center of the display and to simultaneously judge the colors of two arrays of bars that moved in opposite directions. Subjectively, color appearance was similar for the two arrays and was stable over the period of stimulus presentation. We further tested the possibility of eye movement artifacts in two additional experiments. 

In one experiment, we measured eye movements while the observers performed the same tasks as in the rating experiment. Eye movements were monitored by an infrared eye tracker (Iota Orbit 8) with the sampling rate of 1 kHz. The results suggest that motion-induced color segregation does occur without tracking eye movements ( Figure 4A). The observers sometime made small involuntary eye movements, but their occurrence did not correlate with the magnitude of color mixing. 

Figure 4

 

Results of experiments testing the effects of eye movements. (A) Separately for the three stimulus conditions, the horizontal position of the left eye measured in each trial is plotted as a function of time, with the color of the line indicating the rating of color mixing in the trial. The stimulus presentation started at time = 0. Diagonal lines indicate the trajectories of stimulus movements. The results of one observer, but similar results were obtained for the second observer. The data do not indicate such large eye movements as tracking the stimulus motion (eye movements parallel to the diagonal lines) nor indicate significant differences between the stimulus conditions. (B) Results of a psychophysical test for eye movement artifacts, in which motion and flicker stimuli were simultaneously presented. Each point shows the average for five observers (six judgments for each). Motion-induced color segregation remained to be observed.

In another experiment, we psychophysically evaluated the contribution of eye movement. In each trial, motion and flicker stimuli were simultaneously presented, one above and the other below fixation. The observer rated the magnitude of color mixture first for the upper pattern, then for the lower one, based on visual memory. The positions of moving and flicker stimuli were exchanged between sessions. Because the pattern of eye movements should be always the same for the two stimuli, if colors are segregated in the motion stimulus because of any form of eye movements, colors should be similarly segregated in the simultaneously presented flicker stimulus. However, the magnitude of motion-induced color segregation was as strong as that observed in the main experiment ( Figure 4B). 

Effects of spatiotemporal chromatic contrast energy

Stronger color segregation for the motion stimulus might be due to the increase in total stimulus energy relative to the flicker stimulus. We tested this possibility in a subsidiary experiment using stimuli that had the same total spatiotemporal energy as the motion stimulus. Observers rated the magnitude of color mixture for three types of flicker stimuli that had the same total spatiotemporal energy as the motion stimulus. Stimulus color was equiluminance red–green, and the width of the total stimulus area was reduced to 6°. In data collection, all stimuli, including the original motion and flicker stimuli, were randomly presented in the same session. 

In the first condition ( Figure 5A), the test was a continuous flicker stimulus. It was identical to the original flicker stimulus except that each color flash was continuously presented until the onset of the next color. We found that the color mixture rating was just as high for the flicker stimulus. Note that the continuous flicker stimulus has strong motion energy in the vertical direction (in the space–time plot), which is expected to strongly support temporal integration of different color signals at the same retinal location. 

Figure 5

 

Results of an experiment testing the effect of spatiotemporal chromatic contrast energy. Apparent color mixture was measured with test stimuli that have the same total spatiotemporal energy as the motion stimulus. (A) The test was a continuous flicker stimulus. (B) The test was a counterphase red–green square wave grating. (C) The test was a uniform field flicker. The results indicate that it is neither the increase of total stimulus energy nor the increase in apparent stimulus size that makes the motion stimulus appear less color-mixed than the flicker stimulus. Each point shows the average for five observers (six judgments for each).

In the second condition ( Figure 5B), the test stimulus was a counterphase red–green square wave grating, which was made from the original flicker stimulus by extending the width of each bar to fill the interbar gap. The mixture rating was higher than that for the motion stimulus, although only slightly. The reason for the significant reduction in the mixture rating in comparison with the flicker stimulus could be that the counterphase grating is an ambiguous motion stimulus, having substantial motion energy in diagonal directions that supports segregation of different color signals, but little motion energy in the vertical direction that supports color mixing (see next section for a related discussion about the continuous flicker condition obtained with an equiluminant background). 

In the third condition ( Figure 5C), the test was a uniform field flicker stimulus, consisting of flickering two squares presented above and below the fixation. The mixture rating was lower than that for the original bar flicker stimulus, but still significantly higher than that for the motion stimulus. It is well known for achromatic flicker that the larger the test stimulus area, the higher the critical flicker fusion frequency (Harvey, 1970). A similar effect was reported also for chromatic fusion (Vul & MacLeod, 2006). Some effect of spatial scale on critical fusion frequency may result from motion-based signal integration along stationary trajectory because a slim trajectory will be advantageous for signal integration. 

Taken together, the results of the subsidiary experiment indicate that it is not the reduction of total stimulus energy but integration along the apparent motion trajectory that makes the flicker stimulus appear more color-mixed than the motion stimulus in motion-induced color segregation. 

Effects of background luminance

We presented bright color stimuli on a dark background to optimally drive motion mechanisms that prefer achromatic signals, as well as to minimize the effect of spatiotemporal integration with color signals between the bars and the background (background mixing effect). To relate our findings with other color studies, however, one might want to know what happens when the background is replaced by a more conventional equiluminant yellow ( Figure 6A). We therefore measured the magnitude of color mixture for motion, flicker, and continuous flicker stimuli with an equiluminant background ( Figure 6B) and the original dark background ( Figure 6C) using the same procedure and observers. 

Figure 6

 

Results of an experiment testing the effects of equiluminant background. (A) Space–time plot of three types of stimuli (motion, flicker, and continuous flicker). (B) The rating results obtained with the equiluminant background. (C) The control results obtained with the dark background.

A rating experiment was conducted with equiluminant yellow and dark backgrounds in separate sessions. Motion, flicker, and continuous flicker stimuli made of equiluminant red–green bars at three color mixture levels were randomly presented from trial to trial. To reduce a display artifact visible at vertical chromatic borders, we rotated the orientation of the stimulus by 90°. 

Even with the equiluminant background, the color mixture rating was significantly higher for the flicker stimuli than for motion stimuli. With regard to these two stimuli, overall ratings were higher for the equiluminant background than for the dark background, which could be ascribed to the background mixing effect. Flicker bars were hard to detect even at lower temporal frequencies due to insufficient chromatic energy. On the other hand, moving color bars were still clearly visible for a wide range of temporal frequency. This might at least partially result from color signal integration along motion trajectory, where motion information was carried by chromatic signals, residual achromatic signals, or both. 

However, only from the results obtained with the equiluminant background, we cannot conclude that stronger color segregation for motion stimuli is caused by the integration of color signals along the motion trajectory. This is because apparent color segregation was even stronger for nonmoving continuous flicker stimuli whose total chromatic energy was identical to the motion stimuli. This is in contrast to the results obtained with the dark background, where color mixing for the continuous flicker stimuli was as strong as that for the flicker stimuli. 

Weak color mixing (stronger color segregation) observed with the continuous flicker stimuli presented on the equiluminant background can be ascribed to an increase in local chromatic contrast energy and a decrease in background mixing effect in comparison with the motion stimuli. Yet the notion of trajectory integration of color signals is useful to account for the difference between the two background conditions. With the dark background, steady luminance increments clearly define stationary bar trajectories. Along them, the visual system can integrate red and green signals. This color mixing is strong enough to overcome the effect of an increase in local chromatic energy. With the equiluminant background, on the other hand, the steady luminance increments of bars are almost excluded. In addition, along the stationary color-alternating trajectory (but not along the color-keeping trajectory of the motion stimuli), a color transition reverses the sign of chromatic contrast, and possibly the sign of residual luminance contrast. Therefore, motion signals supporting the stationary trajectories must be very weak, if they exit at all. As a result, our theory predicts that red and green signals are not strongly integrated with the equiluminant background. 

In summary, using equiluminant background, a standard procedure to isolate chromatic mechanism, makes the effect of motion on color perception less visible. 

Effects of chromatic aberration

In our previous study on motion-induced color mixing (Nishida et al., 2007), we used an aperture viewing condition to exclude contributions of optical blur arising form chromatic aberration (Flitcroft, 1989). Using a similar viewing condition, we show here that chromatic aberration had little effect on motion-induced color segregation. 

We had observers monocularly view the stimuli through a small artificial pupil (2 mm in diameter), and we confined stimulus presentation in the central 3° × 3° area. We found that this procedure did not affect the magnitude of motion-induced color segregation both in rating experiment ( Figures 7A and 7B) and in chromatic contrast detection ( Figure 7C), which indicates that chromatic aberration plays no significant role in motion-induced color segregation. 

Figure 7

 

Results of experiments testing the effects of optical blur arising from chromatic aberration. For reduction of this factor, observers monocularly viewed the stimulus (presented only in the central 3° × 3° area) through a small artificial pupil (2 mm in diameter) placed in front of their dominant eyes. (A) The results of a rating experiment for RG maxluminance stimuli. (B) The results of a rating experiment for S-cone modulation stimuli. (C) The chromatic contrast detection performance obtained with the aperture viewing and RG equiluminance stimuli.

Covariation of motion and color perception

Our interpretation of motion-induced color segregation is that although retinal color alternation is the same between the motion and the flicker stimuli, perceived motion trajectory is different, and thus the way of color integration is different. However, one might doubt the existence of causal relationship between the change in color perception and the change in motion perception. In addition, one might be reluctant to consider stationary flicker to be equivalent with other motion trajectories. To clear these concerns, we made a more critical test of the notion of trajectory color integration. We used a multipath motion display in which color was alternated at every jump along one path but kept the same along another path. We asked the observers to rate the perceived motion direction and color mixture while we manipulated the perceived motion direction by changing the jump size of the color-keeping path and luminance condition. The question was whether color perception changed in correlation with the direction change. Our hypothesis predicts that colors should be preserved when motion perception prefers the color-keeping path whereas mixed when motion perception prefers the color-alternating path. 

Methods

The methods were identical to those used in the experiments described in previous sections except for the following. 

Stimulus

The spatiotemporal structure of the multipath stimulus is illustrated in Figure 8A. The spatial configuration was similar to the stimuli in Figure 1A. The stimulus consisted of two arrays of bright bars, each subtending 9 min in width, 1.0° in height, presented on the dark background. The bar arrays were presented 2.0° above and below fixation. In each array, red and green bars spatially alternated ( Figure 8A). The center-to-center separation between adjacent bars was 9 min × (path–length ratio + 1). The bars shifted by one bar width in opposite directions in the upper and lower arrays. As a result, the displacement size of color-alternating path was always 9 min, whereas that of color-keeping path was 9 min × (path–length ratio). Each stimulus frame was presented for 6.25 ms with a blank interstimulus interval. The stimulus-onset asynchrony (SOA) was 52 ms. 

Figure 8

 

Covariation of motion and color perceptions. (A) Space–time plots of multipath displays in which integration of color signals along a rightward color-alternating path results in color mixing, whereas integration along a leftward color-keeping path results in color segregation. When the path–length ratio of the color-keeping path was 1 (left), the color-keeping path predominates in motion perception. When the path–length ratio was 4 (right), the color-alternating path predominates. (B and C) The results of double rating experiment. The ratings of motion direction (first row) and color mixture (second row) are plotted as functions of the path–length ratio. The scale of subjective color mixture was the same as in the other experiment (absolute scale, B) or relative within each luminance condition (C). Each data point shows the average for four observers (six judgments for each). In the third row, the two ratings are plotted against each other. The observed covariation of motion and color perceptions is consistent with the notion of integration of color signals along motion trajectories.

A preliminary test indicated that with the shortest SOA, the observer often saw unexpected motions corresponding to low-speed paths in which bars of a given frame jumped to nearby bars presented two or more frames ahead (Burt & Sperling, 1981). On the other hand, longer SOAs made the perceived motion path perceptually multistable. This is an interesting situation in which the covariation of motion and color perceptions could be tested without changing stimulus parameters (Shim & Cavanagh, 2004), but longer SOAs also made the magnitude of color mixing too small to reliably measure. We therefore used an intermediate value of SOA. Each observer made consistent direction ratings for a given path–length ratio across trials. 

Procedure

In a trial, two arrays of a multipath stimulus moving in opposite directions were presented for 1000 ms. The observer first rated the perceived motion direction using a 5-point scale (−2 = the upper array moved unambiguously leftward; −1 = dominantly leftward; 0 = totally ambiguous; +1 = dominantly rightward; +2 = unambiguously rightward) and then made a 5-point scale rating about the perceived color mixture. In each trial (of the absolute rating condition), a stimulus presentation was followed by a presentation of a static color sample. The sample consisted of five pairs of colored bars showing five mixture magnitudes, and the observer had to choose the sample number closest to the impression of perceived colors. The observer could request replays of the stimulus sequence. The path–length ratio was randomly varied across trials. 

Four conditions were carried out in separate sessions: two conditions of luminance setting (maxluminance or equiluminance) combined with two conditions of color rating (absolute or relative). For the absolute rating, we presented the color sample, as described above. For the relative rating, without showing the sample, we asked the observer to fully use the five scales for the stimuli presented in that session and normalized the obtained responses (excluding those of the first block) into the range between 0 and 1 for each condition and each observer. The magnitude of the physical color mixture was 1, 3, or 5 for the absolute rating condition, whereas the magnitude of the physical color mixture was only 1 for the relative rating condition. 

Results

The direction judgments ( Figure 8B, top row) obtained under the maxluminance condition (open squares) indicate that when the path–length ratio of the color-keeping path to the color-alternating path was 1, bicolor stimuli appeared to move dominantly in the direction of the color-keeping path because the motion system uses luminance and color as correspondence cues (Cavanagh & Anstis, 1991; Nishida & Takeuchi, 1990). With an increase in the path–length ratio, the perceived motion gradually changed toward the direction of the color-alternating path because the motion system prefers shorter motion paths (Burt & Sperling, 1981; Ullman, 1979). In accordance with this change in perceived direction, the color rating, which indicated no mixture for the path–length ratio of 1, gradually increased with the path–length ratio (middle row). The covariation of the motion and color ratings (bottom row) is consistent with the expected change in the way color signals are integrated (see also Movies 2A and 2B). 

 

Movie 2A

 

Movie 2B

 

Movie 2. Movie demonstration of covariation of perceived motion direction and perceived color (Figure 8). These two movies have different path–length ratios. (Left panel) Path–length ratio is 1. Expected dominant motion path is the color-keeping path, and expected color perception is color segregation. (Right panel) Path–length ratio is 10. Expected dominant motion path is color-alternating path, and expected color perception is color mixing.

In comparison with the maxluminance condition, the motion direction rating obtained under the equiluminance condition (filled circles in Figure 8B) shifted as a whole in favor of the color-alternating path due to minimization of the luminance cue. In accordance with this change, the color mixture rating also increased. As a result, the patterns of covariation for the two luminance-conditions were very similar. 

Because the color alternation rate along the color-alternating path (10 Hz) was not fast enough to induce perfect color mixing, the change in the mixture rating was relatively small when we used the absolute scale that assigned one for no mixture and five for perfect mixture. We therefore replicated a part of the experiment (physical mixture = 1) by asking the observers to assign the full rating scale to the range of the color mixtures seen in the session (relative rating). The results show the same pattern ( Figure 8C). The small discrepancy between the luminance conditions in the bottom panel can be ascribed to the mismatch of the smallest magnitude of color mixture. 

Discussion

Temporal limit of color perception

The motion-induced color segregation demonstrates that color alternations at the same rate are perceived more veridically in moving patterns than in nonmoving flickering patterns. If temporal color mixing is exclusively determined by peripheral motion-insensitive mechanism, it would be impossible to see color segregation in motion stimuli when the two colors alternate at the frequency beyond the temporal limit indicated by flicker stimuli that have the same pattern of local color alternation. The color signals must be segregated in early stages at least up to the temporal limit indicated by the motion stimuli. The difference in color mixing between motion and flicker stimuli should be ascribed to a subsequent process, where we propose that color signals are integrated along the motion trajectory. 

Physiological findings (Gur & Snodderly, 1997; Lee, Martin, & Valberg, 1989), as well as a psychophysical finding (Vul & MacLeod, 2006), indicate that color-selective neural mechanisms in early visual processing can carry chromatic temporal modulations well above the perceptual fusion limit. Considering the efficiency of neural coding, one would naturally wonder why the brain has to encode invisible high-frequency signals if the subsequent processing simply filters them out. This inconsistency may be explained, at least partially, by our hypothesis that the temporal limit of chromatic perception is determined by a higher level process that flexibly integrates color signals along the trajectory indicated by motion signals. 

Our findings suggest that the standard method of using chromatic flicker fusion may underestimate the upper temporal resolution of color perception. In particular, when small flickering elements are presented on the dark background (e.g., Wisowaty, 1981), color integration along a stationary trajectory will enhance color fusion. On the other hand, when we presented continuous flicker stimuli on equiluminant background, the temporal color segregation limit is as high as those for motion stimuli (Figure 6). We suspect this is because motion energy along stationary trajectory is very week. Wisowaty (1981) suggested that the increase in chromatic fusion frequency by addition of a bright (including equiluminant) surround should be ascribed to a luminance artifact. However, we did not measure flicker detection thresholds, but we used a rating of color mixture, which is similar to the perception of chromatic alternation (PCA) criterion that Wisowaty used to avoid the luminance artifact. 

Vul and MacLeod (2006) found induction of a weak but significant amount of orientation-contingent color aftereffect (McCollough effect) even when adaptation colors alternated at rates (25 and 50 Hz) higher than the chromatic fusion limit. Although the present findings indicate that chromatic fusion may underestimate the temporal limit of conscious color perception, whether the best temporal resolution of conscious color perception can be as high as the limit of the adaptation effect remains an open question. Although our data (Figure 3) showed that chromatic contrast detection for 40 Hz motion stimulus was significantly above chance, we cannot exclude contribution of involuntary eye movements to this effect. 

Related studies

In comparison with a large body of literature on the effects of color on motion perception (e.g., Cavanagh & Anstis, 1991; Croner & Albright, 1997; Dobkins & Albright, 2004; Kool & De Valois, 1992; Krauskopf & Farell, 1990; Lu, Lesmes, & Sperling, 1999; Takeuchi, De Valois, & Hardy, 2003), only a small number of studies have made on the effect of motion on color perception (Chen & Cicerone, 2002; Cicerone, Hoffman, Gowdy, & Kim, 1995; Hepler, 1968; Moller & Hurlbert, 1997; Monnier & Shevell, 2004; Nijhawan, 1997; Wallach, 1935). Among these studies, as far as we notice, there were two reports of motion-induced color segregation. Wallach (1935) found perceptual color decomposition occurring when motion transparency is seen for orthogonal gratings of different colorings. Nijhawan (1997) found decomposition of yellow into red and green when motion induces an apparent shift between physically superimposed stimuli (flash-lag effect). In either case, the visual system decomposes a physically mixed color into two component colors. Undoing color mixing, which sounds logically impossible, is accomplished by the aid of spatiotemporal contexts that indicate the component colors. On the other hand, in our case, if red and green signals from our color-keeping motion stimulus were completely mixed in earlier processes, the subsequent process would only see a simple achromatic pattern with no contextual cues, thus being unable to recover original colors. The color-segregation mechanism indicated by our phenomenon, which requires segregation of color signals in early processes, is different from those indicated by previous findings (Nijhawan, 1997; Wallach, 1935). 

Neural correlates

Given that visual motion detectors do not exist before the visual cortex in humans, our findings suggest a significant contribution from cortical processing to color mixing and segregation. As we suggested before (Nishida et al., 2007), a possible mechanism underlying motion-based color processing is local spatiotemporal integration of color information by neurons sensitive to both motion direction and color (Burr & Ross, 1986; Burr & Ross, 2004; Gegenfurtner, Kiper, & Fenstemaker, 1996; Tamura, Sato, Katsuyama, Hata, & Tsumoto, 1996). Alternatively, the visual system may implement motion-based chromatic integration as a more global interaction between motion and color processing subsystems. Specifically, feedback signals from motion analysis in the luminance-sensitive pathway might modulate the pattern of spatiotemporal integration of neurons in the pathway representing color, by way of dynamically changing connection weights implemented by shunting inhibition (Blomfield, 1974), neural synchrony (Engel, Fries, & Singer, 2001), or other gating mechanisms. In relation to this hypothesis, it is intriguing that neurons in monkey V4, an area often associated with color processing, are narrowly tuned to motion speed (Cheng, Hasegawa, Saleem, & Tanaka, 1994). Although not many monkey V4 neurons show direction-selective response (Ferrera, Rudolph, & Maunsell, 1994), a motion adaptation effect is direction selective even for non-direction-selective V4 neurons (Tolias, Keliris, Smirnakis, & Logothetis, 2005; Tolias, Smirnakis, Augath, Trinath, & Logothetis, 2001). Human imaging studies suggest that motion adaptation effect in V4 is direction selective as in the case of monkey (Huk, Ress, & Heeger, 2001; Nishida, Sasaki, Murakami, Watanabe, & Tootell, 2003). These findings may imply that V4 neurons receive balanced inputs from motion-sensitive mechanisms tuned to different directions in MT/V5 (Tolias et al., 2005), and that these connections are responsible for spatiotemporal integration of chromatic signals in the stimulus motion direction. 

Functional implications

The integration of color signals along the motion trajectory is a useful mechanism for seeing colors in dynamic scenes. Given that most objects do not change color as they move, this mechanism can improve the signal-to-noise ratio by excluding rapid fluctuations of color signals over the motion trajectory, which potentially reflect spatial inhomogeneity in the sensitivity of photosensors or subsequent neural mechanisms or reflect spatial variations in illumination irrelevant to object color per se. Motion-based integration has also been suggested for luminance-defined spatial pattern signals (Burr, 1981; Burr & Ross, 1986; Burr & Ross, 2004) to account for motion deblurring (Burr, 1980) and for the perception of a shape moving behind multiple slits (spatiotemporal pattern interpolation; Burr, 1979; Nishida, 2004). 

There has been a debate as to whether motion deblurring is a result of blur removal by some active process involving motion-based signal integration or just a change in subjective appearance because the visual system is unable to perform the discrimination necessary to decide the sharpness of the moving object (e.g., Anderson, Van-Essen, & Gallant, 1990; Burr & Morgan, 1997; Ramachandran, Rao, & Vidyasagar, 1974). Although the latter hypothesis remains as a general explanation of perceptual sharpening, changes in objective perceptual performance in color perception (chromatic contrast detection experiment of our study), as well as those in pattern perception (Nishida, 2004), indicates the operation of active motion deblurring processes at least under some specific circumstances. 

We conjecture that motion may generally play a role in guiding how patterns and colors are spatiotemporally bound to object representations. Our findings have at least functional if not mechanistic relevance to such phenomena as object-specific feature integration mediated by apparent motion (Kahneman, Treisman, & Gibbs, 1992; Oğmen, Otto, & Herzog, 2006; Otto, Oğmen, & Herzog, 2006; Shimozaki, Eckstein, & Thomas, 1999), object updating in apparent motion (Moore & Enns, 2004), temporal misbinding of features in visual masking (Enns, 2002), and integration of sensory information through a moving window of attention (Cavanagh & Holcombe, 2005). Our hypothesis is also conceptually related to the consideration of the visual persistence as active integration, not passive continuation, of visual signals (Dixon & Di Lollo, 1994). 

A functional drawback of our hypothetical mechanism is the difficulty of localizing moving features at the correct space–time position. We speculate that the mislocalizations of moving stimuli and features in the direction of motion (Cai & Schlag, 2001; De Valois & De Valois, 1991; Nijhawan, 1994) may be explained as epiphenomena of the integration of sensory signals along motion trajectories. 

Acknowledgments

We thank I. Kuriki, M. Terao, I. Murakami, I. Motoyoshi, T. Tokimoto, A. Johnston, D. MacLeod, and S. Tachi for their help, comments, and support. Demonstrations of the phenomena reported in this paper are available on a pubic database, Visiome Platform. 

Commercial relationships: none. 

Corresponding author: Junji Watanabe. 

Email: [email protected]. 

Address: 3-1 Morinosato Wakamiya, Atsugi, Kanagawa 243-0198, Japan.