Quantifying the effect of natural and arbitrary sensorimotor contingencies on chromatic judgments

Abstract

When a body movement systematically co-occurs with a stimulus, a change in perception to compensate for this correlation may occur. In Experiment 1 (similar to A. Bompas & J. K. O'Regan, 2006), we induced a correlation between leftward eye saccades and a red stimulus, and rightward eye saccades and a green stimulus. In a subsequent test phase, observers compared the color of two stimuli after leftward and rightward saccades. The major result was that stimuli tended to look greener after a leftward saccade and redder after a rightward saccade (A. Bompas & J. K. O'Regan, 2006). Measured here in meaningful units for the first time, the shift in the point of subjective equality was ∼d′ = 0.4, a remarkably large effect for only 40 minutes of eye movement/color exposure. A control experiment ruled out a simple reduction in initial bias as the cause of the effect. In Experiment 2, we hypothesized that the blue of the sky might cause an initial bias to judge spots seen with upward gaze as “yellower”; this expectation was not met, but the basic effect was replicated and extended to other chromaticities and eye movement directions. Experiment 3 substituted listening to tones for the eye movements (a sensorisensory correlation) to explore differences between sensorimotor adaptation and sensory integration; neither effect was found for our task. Sensorimotor adaptation can be a remarkably powerful influence on perception; because it operates in a compensatory direction, it may oppose the effects of sensory integration, depending upon task demands.

Introduction

A sensorimotor contingency is the systematic co-occurrence of a body movement with a sensory stimulus. Over time, a change in sensation, motor performance, or both may occur to compensate for this contingency, a process known as sensorimotor adaptation. 

Most evidence for sensorimotor adaptation has come from studying adaptation to artificial spatial distortions (e.g., Gibson, 1966; Harris, 1965; Held & Hein, 1963; Von Helmholtz, 1962; Stratton, 1897). Recently, Bompas and O'Regan (2006) provided experimental evidence for a similar type of learning but with colored patches on a monitor rather than spatial distortions caused by wedge prisms. The role of a sensorimotor mechanism in color perception, they argue, may have been first discovered by Kohler (1951), but he interpreted this effect as the result of adaptation to a contingency between color and eye position, rather than a movement of the eyes. 

Bompas and O'Regan (2006) had subjects judge the relative color of two spots (“greener” or “redder”), one coincident with a leftward saccade and one with a rightward saccade, both before and after a learning phase in which observers were introduced to a contingency between direction of eye movement and color. A shift in color judgments was measured after the learning phase, providing evidence that observers altered their point of subjective equality (PSE) after being exposed to the contingency. In the learning phase, leftward eye movements were paired with red spots, rightward ones with green, and the observers later judged spots appearing on the left as relatively greener and spots appearing on the right as relatively redder than they had done in the pre-learning control phase. In Experiment 1, we seek to replicate and extend the basic finding of Bompas and O'Regan (2006), introducing a contingency between eye movements and the appearance of red or green spots. Bompas and O'Regan reported the magnitude of their effect as a distance in the CIE color space (x, y). This is not a very useful color difference metric (as admitted by the authors), since equal distances have little perceptual meaning, and it is therefore difficult to judge the real-world (as opposed to statistical) significance of the effect. In the present study the effect is measured in units of perceptual sensitivity and in cone contrast units. In order to make the calculation in cone contrast units, a well-specified background field is needed (Bompas & O'Regan used none). The addition of an approximately equiluminant background will also rule out a large luminance modulation in the learning phase being necessary for sensorimotor adaptation. These stimulus changes permit us to specify the magnitude of the basic effect in more meaningful units. 

Experiment 2 is similar to Experiment 1, but we change the directions of the eye movements and the colors used. We use upward and downward eye movements and blue/yellow rather than red/green hues. Critically, in addition to focusing on the difference between the pre- and post-learning experimental phases, we also closely examine the pre-learning stage per se. We might expect, if all sensorimotor contingencies lead to adaptation, to measure an initial bias due to the adaptation over the observers' lifetime experience of looking up and seeing the blue sky. In addition to this initial bias, we also expect that we will find a change as a result of the learning phase, in the same vein as in Experiment 1. 

One important point about the study of Bompas and O'Regan (2006) has to do with the direction of the sensorimotor learning effect. The shift in PSE that they produced was in the same direction as typical adaptation to color that does not involve eye movements. Just as in chromatic adaptation, where being steadily exposed to a green field of light (for example) over time reduces the effective “greenness” of the light (Gibson & Radner, 1937; Helson & Michels, 1948; Jameson & Hurvich, 1972; Judd, 1940; many others), the effect detailed by Bompas and O'Regan was that subjects responded as if a spot looked the opposite of the hue that was paired with the eye movement in the learning phase. This complementary direction of adaptation has also been seen with blur (Webster, 1999), orientation (Greenlee, Magnussen, & Nordby, 1988), motion (Anstis, Verstraten, Frans, & Mather, 1998), and specific facial characteristics (Webster & MacLin, 1999), though not as part of a sensorimotor contingency. Following Held and Hein (1963), we refer to this direction of sensorimotor effect as “compensatory”. 

Bompas and O'Regan rule out two alternative explanations before concluding that sensorimotor adaptation is the cause of the color shift they observe: that the effect is retinal adaptation, and that the effect is contingent upon eye positions rather than eye movements. A possible third alternate explanation is that the learning phase altered observers' expectancies. The direction of the effect reported by Bompas and O'Regan (2006) makes this explanation seem unlikely: if two events are coincidental, then one event would signal both, not the opposite of the second event. Sensory integration, for example, works in the opposite direction as sensorimotor adaptation. Ernst (2007) argues that after subjects were exposed to a distribution of objects in the laboratory in which luminance and tactile stiffness were positively correlated, a high luminance or a high stiffness signaled the combination of the two. 

Thus, in general, sensory integration and sensorimotor adaptation work in opposite directions: when two sensory signals co-occur frequently they may be integrated, but when a body movement frequently co-occurs with a sensory signal the sensation is diminished. However, there are some likely counterexamples, sensorimotor contingencies that should result in integration rather than compensation. For example, depth parallax, or the relationship between the distance from the head to a visual target and the speed at which its image moves across the retina as the head moves from side to side, can be used as a depth cue (Rogers & Graham, 1979) and it would seem counterproductive to have sensory compensation reducing the perception of parallax. If there were perfect compensation, parallax could not be used as a depth cue. Integration, on the other hand, would be useful in this case. We explore these issues in Experiment 3, by substituting two different tones, for eye movements in an experiment with similar task demands as in the study of Bompas and O'Regan. If we see compensation when there are tones and no eye movements (a sensorisensory contingency rather than a sensorimotor one), this would suggest the effect described by Bompas and O'Regan is not due to sensorimotor learning. However, if we see integration or no effect, then we will conclude that the pairing of a colored spot and a body movement was fundamentally different from pairing the colored spot and another sensory signal for this experimental task. 

Methods

Apparatus

The display was a CRT (Philips 202P7) with resolution of 1152 by 870 pixels and a vertical refresh rate of 75 Hz. The display was gamma corrected in software to ensure a linear relationship between nominal and actual screen increments independently for the red, green, and blue guns. A chin rest at a viewing distance of 30 cm stabilized the observer's head. 

The monitor was spectroradiometrically calibrated using a PhotoResearch PR650 spectroradiometer. The luminance of the ever-present gray background was 50 cd/m ², and the CIE chromaticity was (0.301, 0.315). Initial measurements indicated that the two sides of the monitor, where the spots would appear in Experiment 1, had almost identical chromaticities as the center; this conclusion proved to be incorrect, as will be described in the Results and discussion section. All experiments were conducted in an otherwise dark room. 

Observers

A total of 47 observers participated in these experiments. Most observers were volunteer undergraduates drawn from Psychology classes and given partial class credit for participating. These participants were naive to the purposes of the experiment and were not experienced psychophysical observers. No naive observer participated in more than one experiment. Fifteen observers participated in Experiment 1A, 10 in Experiment 1B, 7 in Experiment 2, and 15 in Experiment 3. A few of these observers were experienced, some (but not all) of whom were not naive to the purposes of the experiment. The pattern of results was the same for naive and experienced observers. 

General procedure

In all experiments, there were three phases. The initial phase was the pre-learning baseline phase, in which relative color judgments are made. The learning phase followed, in which a contingency was introduced. The post-learning phase was the final phase and was identical to the pre-learning phase. Instructions emphasized making eye movements in Experiments 1 and 2, but the eyes were not tracked. However, the incidental tasks used in the learning phase were designed to be extremely difficult if the appropriate movements were not made. Performance on the incidental task was 85% correct or better in all cases in these experiments. Procedural details of the individual experiments follow. 

Experiment 1

Visual stimuli

All stimuli were created using MATLAB with the psychophysics toolbox extension (Brainard, 1997). The visual stimuli used for the initial and final testing stages were 5° diameter circular spots. In the learning phase, half the stimuli were 5° disks and half were ellipses of eccentricity 0.52. In all cases, stimuli were flashed for 400 ms. 

The chromaticity of the spots was determined by the vector sum of two color directions in cone contrast space. Every spot contained a constant component, referred to here as a “pedestal.” To the pedestal, a test consisting of a variable amount of chromatic contrast was added. The constant pedestal consisted of equal increments in L and M cone contrast ( Table 1), {ΔL/L, ΔM/M, ΔS/S} = {0.38, 0.38, 0.00}. The added test was a variable amount of approximately equiluminant red–green contrast, produced by a variation along the L–M contrast direction (the same direction, but of lesser magnitude, as the chromatic vector listed in Table 1), where L and M cone quantal catches vary in opposite direction (at constant sum). The resultant stimulus was a yellow disk tinged with small amounts of red or green. S cones were never modulated in this experiment. 

Table 1

 

Sensorimotor contingencies introduced in all experiments, including cone contrast vectors.

Experiment	Learning phase	Chromatic vector	Pedestal vector	Spot Size
1A	Red-Left, Green-Right	{0.2, −0.2,0} and {−0.2, 0.2, 0}	{0.38, 0.38, 0} (Yellow)	5°
1B	Green-Left, Red-Right	{0.2, −0.2,0} and {−0.2, 0.2, 0}	{0.38, 0.38, 0} (Yellow)	5°
2	Blue-Up, Yellow-Down	{−0.07, −0.01, 0.3} and {0.07, 0.01, −0.3}	{0.27, 0.27, 0.27} (White)	5°
3	Red-Low, Green-High	{0.2, −0.2,0} and {−0.2, 0.2, 0}	{0.27, 0.27, 0.27} (White)	8°

In the learning phase, in which the observers experienced the sensorimotor contingency, both circular and oval spots were used, as in Bompas and O'Regan (2006). Chromaticities were approximately equiluminant red and green, near the maximal contrast that could be produced by our monitor; the yellow pedestal component was not present. Table 1 gives the cone contrast vectors {ΔL/L, ΔM/M, ΔS/S} of these spots. 

Procedure

Observers completed three stages. The initial stage measures the baseline for chromatic judgments while making leftward and rightward saccades. The observer was first shown an arrow pointing either right or left. If this arrow pointed left, the observer looked left as (or just before) the first spot appeared on the screen, then looked right as the second spot appeared on the screen. Figure 1 (top) shows the time course of a trial in this phase. 

Figure 1

 

The time course of trials in Experiment 1A. (Top) Pre- and post-learning phase trials. The arrow in the first panel could point to the left or the right and is an indicator of the order of left and right spot presentations. The observer sees a neutral spot, then a slightly greenish spot. The observers' task is to make a relative color judgment of the spots (here, “greener”). (Bottom) Learning phase trial. The observers' task is to make a relative shape judgment (here, “different”).

The left spot was centered on a point 8° left of the center of the screen, and the right spot was centered on a point 8° right of the center of screen. Observers were told where the spots would appear and were further instructed to anticipate the appearance of each spot so that they completed each saccade just as the spot appeared, so that spot would be foveated. After seeing both spots, the observer made a judgment about their relative color. The observer always judged whether the second spot was redder or greener than the first spot. Both spots needed to be seen in order to make this judgment. Twelve observers pressed “2” to indicate that the second spot was redder and “5” to indicate that the second spot was greener. Thirteen pressed “2” to indicate that the second spot was greener and “5” to indicate that the second spot was redder. These two buttons are vertically arranged on the numeric keypad used to collect responses. 

The contrast of the chromatic component of one of the two spots was chosen randomly from a list, and the chromatic contrast of the other spot was set to zero. Thus, one spot was comprised only of the neutral pedestal, while the other spot was the pedestal tinged with color to be reddish, greenish, or also neutral if the chromatic contrast was zero. 

The pre-learning phase consisted of two blocks of trials with 100 trials each. Because the test contrast of the non-neutral spot was chosen randomly, the number of trials with a given test contrast varied slightly. 

In the learning phase the time course of the trial was identical, but the stimuli and the observer's task were different. Figure 1 (bottom) shows the time course of a trial in this phase. There were two variations (A and B) of Experiment 1, defined by the contingency introduced in this learning phase. In Experiment 1A, the spot appearing on the left was always red and the spot appearing on the right was always green, as shown in Figure 1. In Experiment 1B, the colors were reversed, pairing left with green and right with red, both of high chromatic contrast ( Table 1). Each spot was randomly determined to be a circle or an oval, with 0.5 probability of each. The task was to report whether the two spots had the same or different shapes. This incidental task forced observers to look at both spots on each trial and gave them experience with the color/eye movement contingency. Observers were instructed to make eye movements as in the pre-learning test phase, determined by the arrow appearing at the beginning of each trial. If both spots were circles or both were ovals, they were instructed to press “2,” if one spot was a circle and one an oval, they were to press “5.” There were four blocks of trials in this stage with 100 trials each block, for a total of 400 trials. Feedback was given after each trial to inform the observer whether the response was correct or not. 

The post-learning test phase was identical to the pre-learning baseline phase, in order to look for differences caused by the introduction of the sensorimotor contingency. 

Data analysis

Data analysis procedures described here apply to all experiments. Performances on the pre- and post-learning phases were calculated in units of sensitivity change. First, trials were separated into 4 types, based on the direction of the initial eye movement (or tone frequency for Experiment 3) and which spot had a zero test contrast, creating a two by two matrix with four trial types. The proportions of “2nd spot redder” responses (“2nd spot bluer” for Experiment 2) were calculated for each test contrast for each trial type. Eight Gaussian cumulative distributions were fitted to these data for each observer, one for each trial type in both the pre- and post-learning phases. Two parameters, the location parameter μ and (inverse) slope parameter σ, were extracted from each fit. Examination of the fitted curves showed that the slopes across conditions were very similar, so a single pooled standard deviation was calculated for each observer as the root mean square of the eight fitted σ parameters. The ratio μ/ σ _pooled, referred to here as D, was calculated for each of the trial types, pre- and post-learning. D is an estimate of d′ if the distribution of contrast effects is Gaussian and of constant variance; see, e.g., Treisman and Watts (1966). We have no evidence to test this assumption, hence we refer to our metric as D rather than the more conventional d′. Because the observer was responding based on the temporal order of the spots and the analyses here were concerned with which spots are coincident with which eye movement or tone, half of the D values were multiplied by negative one, in order to make all D values have a positive value for results in the expected direction for the pairing of [left + red] and [right + green]. For example, if the predicted result following a [right + red]-to-[left + neutral] saccade was μ > 0, then the [left + red]-to-[right + neutral] trial type had a prediction of μ < 0 and the D values were reversed in sign for this second type. The four trial types showed no systematic differences, so they were pooled for subsequent analysis. 

These sign conventions were applied to the values of D in both the pre- and post-learning phases; the signs of the changes in D across phases were not altered. Any change in D in the positive direction from the pre-learning phase to the post-learning phase indicates compensation. 

Results and discussion

Figure 2 shows two representative sample psychometric functions for a single observer (AV). These come from a single trial type: the neutral spot was on the left, and the trial began with a leftward saccade, as shown in Figure 1 (top). For all the trials represented here, the red–green contrast on the abscissa specifies the chromatic component of the second (right) spot, in units of cone contrast vector length (Eskew, McLellan, & Giulianini, 1999). The ordinate is the proportion of “2nd redder” responses. As the chromaticity of the second spot gets more positive, the proportion on the ordinate increases. For the trial type where the second spot is neutral and the first spot has the colored test component, the opposite relationship would be seen (the curves would be flipped horizontally). 

Figure 2

 

A pair of psychometric curves for one observer (AV) for 1 of the 4 trial types in Experiment 1A. Pre-learning results (gray) and post-learning results (black) show a shift in PSE as a result of the learning phase. Values of μ and σ are displayed graphically, and the calculation of D is shown.

The dark diamonds and line are the data and curve fit from the post-learning phase, and the lighter spots and line are from the pre-learning phase. Because the learning phase paired left with red and right with green (Experiment 1A), we expect spots to be judged greener when making leftward saccades and redder when making rightward saccades (a compensatory effect; see Introduction section). For the trial type depicted in Figure 2, the first spot is neutral, the second spot has some amount of red–green contrast, and we expect a larger proportion of “2nd redder” responses in the post-learning phase than in the pre-learning phase. The first saccade, to the neutral spot, should cause that spot to be judged slightly greener, and the second saccade, to the right, should cause that spot to be judged slightly redder in comparison to the pre-learning phase. Figure 1 shows an example of this trial type with a negative (green) test contrast; the leftward shift of the post-learning curve in Figure 2 is an example of the change in PSE. 

Figure 3 shows a scatter plot of all the data from Experiment 1A, with the pre-learning D along the abscissa and the post-learning D on the ordinate. The dashed line of slope 1 denotes where the two D values are equal, and the region above this line is where the post-learning D is larger than the pre-learning D, meaning that color judgments were altered in the predicted (compensatory) direction. The arrow indicates the D values corresponding to the two curves shown in Figure 2. 

Figure 3

 

The effect of the learning phase on relative color judgments in Experiment 1A, showing D for the pre-learning and post-learning phases, where to the degree that the points lie above the line, post-learning D is greater than pre-learning D, indicating a compensatory change following the learning phase. Each subject is represented by four points, representing the four different trial types (see Data analysis section). The large black symbol is the centroid computed across all observers and trial types; error bars are standard errors. The point labeled “Figure 2” is the D values derived from the curves shown in Figure 2. One point is not shown (−2.8, −2.3) but is included in the analysis.

This shift in PSE indicates that, for example, when the observer made a [left + green]-to-[right + neutral] saccade the neutral yellow spot looked redder compared to the pre-learning phase. The mean Δ D (post-learning D minus pre-learning D) for all observers was 0.39, a small but statistically significant difference from zero ( t = 5.43, p < 0.05). In cone contrast units, the mean shift in PSE is 0.004 (vector length in the ΔL/L, ΔM/M plane). For comparison, this is roughly twice the threshold red–green contrast for small spots for highly practiced observers (Eskew et al., 1999). 

Thus, on average, the exposure in the learning phase caused observers to adapt to both red and green spots, contingent on saccade direction, and in a compensatory direction. However, the values of D in the pre-learning phase (projection of the points onto the horizontal axis in Figure 3) raise a possible flaw in the experiment: the mean D in Experiment 1A in the pre-learning phase was −0.25, significantly less than zero ( t = −2.51, p < 0.05), indicating that the observers tended to refer to the spots on the left as redder and those on the right as greener before the learning phase. The initial bias can be seen in the pre-learning psychometric curve in Figure 2 (gray), which has a PSE that is shifted to the right. The reason for this initial bias was initially unclear (see below), but whatever its cause, its direction raised a disturbing possibility: that the effect of the second phase was not due to sensorimotor learning at all, but simply that practice at the task made the initial bias diminish. A change from an initial bias of D = −0.25 to a post-learning D ≈ 0 would produce an apparent sensorimotor effect (a difference score Δ D > 0) that could in fact be just an elimination of initial bias. 

Bompas and O'Regan (2006) made a prediction that their observers would have no initial bias, but they did not report the initial bias or confirm that their prediction was upheld, only reporting the difference scores. To test the possibility that our effect (as well as theirs) was merely a reduction with practice in initial bias, we performed the following control experiment. In Experiment 1B the color pairings in the learning phase were reversed; leftward saccades were paired with green and rightward saccades were paired with red. Now, a sensorimotor learning effect should cause the values of D to become more negative, because the colors in the learning phase were reversed but the values of D were calculated in the same way as in Experiment 1A. However, reduction of initial bias would cause the value of D to become more positive (tending towards zero), just as in Experiment 1A. 

Figure 4 shows the pre- and post-learning D values for all observers in Experiment 1B. The mean Δ D in Experiment 1B was −0.40 ( t = −2.97, p < 0.05) meaning that when the pairings of movement and color were reversed, the change in color judgments was also reversed. The initial bias seen in Experiment 1A was also seen here, which is expected because the procedures before the learning phase were identical. However, here the initial bias is not reduced: the manipulation in the learning phase caused observers to alter their color judgments such that the post-learning phase Ds are even farther from zero and in the same direction as the initial bias. These control results strongly indicate that, whatever the cause of the initial bias (see below), the effect of the learning phase is not merely a reduction in that bias, but rather a sensorimotor learning effect. 

Figure 4

 

The effect of the learning phase on relative color judgments in Experiment 1B (symbols and line as in Figure 3). Reversing the positions of the colors in the learning phase reverses the direction of the effect (points lying below the line), as expected if the effect is due to sensorimotor compensation. Three points are not shown (2.2, 0.0; −1.2, −2.2; −1.0, −2.0) but are included in the analysis.

The initial “left-redder/right-greener” bias found in Experiments 1A and 1B is not a simple response bias. The observers' judgments depend upon the temporal order of the two spots, so that any simple response bias can only lead to a bias of calling the first or second spot redder. An observer who replied “2nd redder” on most trials in the pre-learning test phase would have positive PSEs in two of the four trial types, and negative PSEs for the other two, and the two effects would cancel, resulting in a pre-learning D of zero. The negative D values we found mean that the observers pressed one response button more frequently on a left-to-right trial and the other button more frequently on a right-to-left trial. 

One possible explanation for the bias is that the nominal chromatically neutral stimulus on the left of the screen is not exactly the same as the neutral stimulus on the right of the screen, due to a non-uniformity in the monitor. Our initial measurements (see Methods section), as well as casual observation of the screen, had suggested that there was no substantial chromatic non-uniformity. After obtaining the initial bias in Experiment 1, however, we very carefully recalibrated the background field in the two locations (left and right) where the spots appeared in Experiments 1A and 1B. CIE coordinates were (0.297 ± 0.000007, 0.313 ± 0.000016) on the left and (0.297 ± 0.000009, 0.314 ± 0.000042) on the right (average of 100 measurements per side), showing that despite the initial calibration, the background was very slightly “redder” on the left and “greener” on the right. After converting these chromaticities to CIELuv space (Wyszecki & Stiles, 1982), the difference between the two sides was 5.40, which is likely suprathreshold. This chromaticity difference is not apparent under normal viewing conditions but can be seen (barely) with the aid of a reduction screen used to block all but the two regions on the monitor where the tests appeared. 

Although the chromaticity difference is tiny, it was apparently large enough to cause observers to have a non-zero PSE during the pre-learning test phase. However, the control experiment (1B) shows that the difference between the pre-learning and post-learning stages is not just a reduction in this initial bias but a genuine effect of the learning phase. 

Experiment 2

Visual stimuli

The visual stimuli used for Experiment 2 differed from Experiment 1 in three ways: location, the colors of the test components, and the makeup of the pedestals. 

Spots in all three phases were located 5° up or down from the center of the screen, rather than 8° left or right. The test components added to the pedestal were along the “sky blue” direction. This color direction was created by simulating the effect of Rayleigh scattering by applying Rayleigh's equation to the spectrum of the monitor's gray background (see 1 for further details). Because the test components used here varied along a blue–yellow dimension, a white neutral pedestal was used instead of a yellow one. 

Procedure

The procedure used in Experiment 2 differed from Experiment 1 in only two ways. Observers made upward and downward saccades (as instructed by the now upward or downward pointing arrow) rather than leftward or rightward. The observers now judged if the second spot was bluer or yellower than the first spot, and responses were collected using the horizontally arranged “1” and “2” buttons on the numeric keypad (“1” for “second spot yellower” and “2” for “second spot bluer”). 

Results and discussion

Because of the initial non-zero PSE observed in Experiment 1 and the small non-uniformity in the monitor that apparently caused it, it is difficult to interpret the initial bias in isolation. Our prediction was that observers would have a predisposition to judge spots as yellowish when making upward saccades, based on the pre-existing contingency between upward saccades and the blue sky. Our prediction based on this sensorimotor contingency and the effect shown in Experiment 1 was that these pre-learning D values should be positive. Figure 5 shows the pre- and post-learning values of D. 

Figure 5

 

The effect of the learning phase on relative color judgments, showing D for the pre-learning and post-learning phases. The area above the diagonal line, where the majority of points lie, shows where D post is greater than D pre, indicating a compensatory change following the learning phase. Two points are not shown (−2.1, −0.3; −2.0, −0.2) but are included in the analysis.

Looking only at the pre-learning values of D (projection onto the horizontal axis) reveals the opposite result: upward saccades caused the spot to be judged as bluer rather than more yellow. These initial values of D are significantly different from zero ( t = 3.1, p < 0.05). 

A monitor non-uniformity cannot as readily explain the observers' behavior in this case, however. Observers made judgments as if the top part of the screen was bluer than the bottom, but in actuality the top was slightly yellower. Very careful measurements of the monitor reveals there is a difference in the v* dimension in CIELuv space of −2.43, indicating that the top is “yellower,” but that this difference should be below threshold for most observers. There was a difference in the u* dimension of 4.80, meaning that the top was slightly redder than the bottom to a visible degree. This difference does not straightforwardly explain the pre-learning phase bias of the observers because the blue–yellow difference is sub-threshold and in the wrong direction, and the red–green difference has no direct bearing on the blue–yellow judgments that observers were asked to make. 

The correlation between looking upward and seeing blue, in the Boston area, may simply be too low to produce sensorimotor adaptation, but this would not explain the small initial bias in the opposite direction. It is possible that a natural contingency exists that pairs upward eye movements with “yellow,” as might be suggested by the observed initial data. Tungsten lights are yellowish and are often above the head. It is also possible that a long-term association between the concepts of “up” and “blue” had more impact on observers' judgments than the sensorimotor contingency had. Whatever the cause of this bias, it is clear that the natural contingency between making upward saccades and seeing the blue sky did not result in the same kind of adjustment seen in Experiment 1. 

The contingency that we introduced in the learning phase did result in the same kind of compensation seen in Experiment 1. Figure 5 shows the pre- and post-learning phase values of D for all subjects with the blue and yellow spots and vertical eye movements. The mean Δ D was 0.55, significantly different from zero ( t = 3.04, p < 0.05). In cone contrast units, the mean shift in PSE is 0.028 (vector length in three-dimensional cone contrast space), a suprathreshold value for R/G or S cone detection for experienced observers (Eskew, 2008; Eskew et al., 1999). 

In addition to replicating the basic effect, Experiment 2 provides two novel findings. One is that this kind of sensorimotor learning is not exclusive to the red–green color direction but can be seen with blue and yellow hues. The stimuli used in Experiment 2 were not arranged along the maximally effective color direction for a blue or yellow detection mechanism, as the reds and greens in Experiment 1 were, so it is also apparent (and unsurprising) that such maximally effective chromaticities are not required. The second novel finding is that compensation can be measured when eye movements are in the up–down rather than left–right directions. 

Experiment 3

Visual stimuli

The visual stimuli used for the pre- and post-learning phases of Experiment 3 differed from Experiment 1 in three ways: the location, the makeup of the pedestals, and the size of the spots. 

Spots in all three phases were located at the center of the screen, rather than to either side, and a white pedestal was used identical to the one in Experiment 2. Spots in Experiment 3 were larger (10° diameter) than in Experiments 1 and 2, because we were less restricted due to the spots not being presented near the monitor's edges. In the learning phase, 10° diameter spots (red and green) were used, with four fifths of them being identical in contrast to the spots used in Experiment 1 and one fifth of them being half that contrast. 

Auditory stimuli

The auditory stimuli used for Experiment 3 were clearly audible square-wave tones of low (nominally 300 Hz, 66 dB) and high (nominally 2000 Hz, 71 dB) frequencies, lasting for 400 ms. 

Procedure

The pre- and post-learning phases for Experiment 3 were similar to Experiments 1 and 2, with the following differences. Rather than pairing spots with eye movements, each spot was presented with one of the two tones. The judgments (“2nd redder” or “2nd greener”) were identical. No eye movements were required in this experiment, as all spots were located in the center of the screen. Figure 6 (top) shows the time course of a trial. 

Figure 6

 

The time course of trials in Experiment 3. (Top) Pre- and post-learning phase trials. The observer sees a neutral spot with a low-frequency tone (represented by the icon and wave), then a slightly reddish spot with a high-frequency tone. The observers' task is to make a relative color judgment of the spots (here, “redder”). (Bottom) Learning phase trial. The observer sees 5 spots, red always with the high-frequency tone and green with the low-frequency tone. The observers' task is to report the hue of the low contrast spot (here, “red”).

The learning phase, as in Experiments 1 and 2, provided the observer with many samples of the relevant pairing (in this case the tone with the colored spot). In each trial, 5 spots were presented one after another in the center of the screen, each for 400 ms with 500 ms between presentations ( Figure 6, bottom). Each spot was randomly determined to be red or green with equal probability, and all red spots were presented with the higher frequency tone and all green spots were presented with the lower frequency tone. Four of the spots were maximum contrast (very red or very green), but one spot, chosen randomly, was half the contrast of the others (still far above threshold). The observer's task was to indicate the hue of the low contrast spot, pressing “7” for red and “8” for green. This incidental task was designed to have the observers attend to the color of each spot and be exposed to the tone–color pairings. There were 20 trials per block, with 10 blocks in this phase, for a total of 1000 pairings of spot and tone (approximately 500 per color). Feedback was given after each trial. 

Results and discussion

No eye movements were required in Experiment 3; instead, the learning phase exposed observers to a sensorisensory correlation between color and tone frequency. Figure 7 shows the pre- and post-values of D. The initial PSE was not different from zero, nor was there any effect of the learning phase. The mean Δ D was 0.03, not significantly different from zero ( t = 0.56, p > 0.05). 

Figure 7

 

The effect of the learning phase on relative color judgments for Experiment 3. The centroid is represented by the black circle; standard error bars are too small to be seen.

One possibility that was not borne out was the potential for sensory integration. Bresciani, Dammeier, and Ernst (2006) found that sensory cues from different modalities can be automatically integrated. Observers were presented with brief flashes (visual stimuli) and taps (tactile stimuli), and estimated the number of either taps or flashes (without feedback, as in our experiments; Ernst, personal communication). When, for example, observers were estimating flashes the visual stimuli were deemed task relevant, and the taps were task irrelevant (just as the tones were task irrelevant in our experiment). Observers' responses were biased in the direction of the number of task-irrelevant taps or flashes. This was taken to be evidence for automatic integration, because the task-irrelevant stimuli influenced the judgments even though they should have been ignored. 

In our study, it could have been that color and tone were integrated after the observers were exposed to the pairings in the learning phase. If this were the case, then the high-frequency tone would have signaled the integrated experience of both the high-frequency tone and the red spot, causing the spot to be judged more reddish. This prediction is in the opposite direction of the sensorimotor compensation measured in Experiments 1 and 2. 

We did not obtain integration, or in fact any effect of the learning phase on observers' responses in this experiment. However, even if subjects in the experiment of Bresciani et al. (2006) were using only the task-irrelevant information, their estimates would be very close to correct, as the number of irrelevant stimuli were within ±1 of the number of relevant stimuli. It would make more sense, perhaps, to say that their nominally irrelevant stimuli were actually relevant but noisy. Perhaps, therefore, the sensory integration observed by Bresciani et al. was not actually automatic but depended upon the utility of combining the sources of information. In the present experiment, the task in the learning phase was visually easy (perhaps meaning that there was less incentive to recruit more information) and separated in time from the testing phase. 

It is clear that under the right circumstances information from different sensory modalities can be perceptually integrated (e.g., Ernst, 2007; Rock & Victor, 1964; Shams, Kamitani, & Shimojo, 2000), but that did not occur here. Making the tones more relevant to the task might have encouraged sensory integration, but more to the point, compensation did not occur. The tones in this experiment were formally as relevant to the task as the eye movements in Experiments 1 and 2 were so there is no reason to believe that a change in task parameters would result in the compensation seen in Experiments 1 and 2. Our results suggest there may be fundamental differences between sensorisensory and sensorimotor adaptation. 

Conclusion

Magnitude of the effect

The mean effect, measured as Δ D, across Experiments 1A, 1B, and 2 was 0.42. In cone contrast units, the change was 0.011. Understanding the context of the cause of this effect, namely a 40-minute session learning an arbitrary sensorimotor contingency, pitted against a lifetime of learning that there is no systematic relationship between eye movements and color perception (at least for Experiments 1A and 1B), shows the power of sensorimotor adaptation. While effect sizes of this magnitude are often thought of as “small”, a large effect based on a 40-minute learning phase would be unreasonable and certainly unexpected. 

Pre-learning bias and monitor inhomogeneity

The difference in color between the left and right sides of the experimental display, where the stimuli in Experiments 1A and 1B were presented, apparently caused the initial PSEs to appear negative. Similar effects, generally unmeasured, must contribute to the results of many studies in which visual stimuli are presented in different locations on displays. Experiment 3, in which all stimuli were presented in the center of the screen did not produce any bias in the pre-learning PSEs, confirming our interpretation that the monitor's non-uniformity did in fact cause the non-zero initial PSEs in Experiments 1A and 1B. 

In Experiment 1, the chromaticity difference in the two locations was so small that the display appeared uniform to both the eye and initial measurements from our spectroradiometer, and yet it had a measurable effect upon the relative color judgments made in the pre-learning phases. Direct visual observation, and even routine measurements with a precise spectroradiometer, were not sufficient to detect the inhomogeneity; very careful colorimetric procedures were required. The use of a reduction screen to compare the two screen locations in isolation might have alerted us to the issue prior to running Experiment 1A (and we commend that tool to other researchers), but correcting such a tiny difference in the absolute chromaticity on the screen and being confident in that correction would have been technically very difficult indeed; instead, we rely upon the control Experiment 1B to demonstrate that the measured sensorimotor adaptation is not an artifact of the monitor non-uniformity. 

The same negative initial bias exists, and to the same degree, in the data from Experiments 1A and 1B. The manipulation in Experiment 1A caused the negative D value to become positive, whereas reversing the hues in Experiment 1B showed a shift of the same magnitude but in the opposite direction, becoming more negative. In other words, the learning phase pushed the PSEs in the two expected directions from the same starting bias, just as predicted from sensorimotor adaptation, and not as would be predicted from reductions in bias due to practice or to other explanations that would be independent of the color–movement pairing chosen in the learning phase. 

General discussion

When a body movement systematically co-occurs with a sensation, adaptation to this contingency may occur. Adaptation is characterized by an adjustment to a sustained or frequently presented stimulus, making the stimulation more “neutral” in order to maintain relative sensitivity. The adapting stimulus is often treated as a unitary event, but there are many examples that involve adaptation to a contingent color stimulus such as the McCollough (1965) effect. Like the McCollough effect, the present research highlights the ability to either rapidly switch between two states of adaptation (in the time it takes to make a saccade) or to maintain two concurrent states of adaptation. This is important because it allows for more refined adaptation—adaptation to a conjunction of events or stimuli rather than a single stimulus, allowing for more targeted adaptations or adaptation to more complex environments. 

Appendix A

The reason that the sky looks blue is Rayleigh scattering (Nassau, 2001). Light from the sun is scattered upon reaching the Earth's atmosphere, with a strong wavelength dependence. This dependence can be approximated by λ⁻⁴, meaning that short wavelengths are scattered more than long wavelengths. To make a sky blue stimulus, one can begin with the spectrum of the gray background and apply Rayleigh scattering to it, in order to produce the same cone contrast effect that the scattering would, relative to our monitor background. Just matching the blue of the stimulus to the blue of the sky without attending the background spectrum would not produce the same cone contrast effect as the sky actually does. 

The white background spectrum is first cross-correlated with the L, M, and S cone fundamentals. These determine the effect of the background on the photoreceptors. We next multiply the spectrum of our gray background, by λ ⁻⁴ to simulate “sky blue”. The same cross-correlations are performed for the Rayleigh spectrum (background * λ ⁻⁴). Six integrals are taken, one for each cone class for each spectrum. In this case, the L, M, and S values for the background were 1.0693, 0.93922, and 0.71879. For the “Rayleigh scattered” spectrum, they are 0.83790, 0.85481, and 1.2371. These units are arbitrary and could be scaled in any way. The non-arbitrary cone contrasts are defined as the difference in the cone effect divided by the background's cone effect. The value of the L cone component of the cone contrast vector is (0.83790 − 1.0693)/1.0693. This yields −0.21642. The three-dimensional vector (L, M, and S) is [−0.21642 −0.089870 0.72103]. These values were used to determine the desired [R G B] values used for the sky blue stimulus. 

Acknowledgments

We thank Adam Reeves, Rachel Theodore, and Scott Gabree for their comments and suggestions, and Jeremy Jameson and Olga Meleshkevich for serving as observers. 

Commercial relationships: none. 

Corresponding author: David P. Richters. 

Email: [e-mail]. 

Address: 125 NI Northeastern University, Boston, MA 02115, USA.