Switching between visuomotor mappings: Learning absolute mappings or relative shifts

Abstract

Abstract

Abstract:

Abstract  Adaptation to specific visuomotor mappings becomes faster when switching back and forth between them. What is learned when repeatedly switching between the visuomotor mappings: the absolute mappings or the relative shift between the mappings? To test this, we trained participants in a rapid pointing task using a unique color cue as context for each mapping between pointing location and visual feedback. After extensive training, participants adapted to a new mapping using a neutral contextual cue. For catch trials (a change in cue and no visual feedback) different adaptation performances are predicted depending on how the mappings are encoded. When encoding an absolute mapping for each cue, participants would fall back to the mapping associated with the cue irrespective of the state they are currently in. In contrast, when a shift in mapping is encoded for the cue, pointing performance will shift relative to the current mapping by an amount equal to the difference between the previously learned mappings. Results indicate that the contextual cues signal absolute visuomotor mappings rather than relative shifts between mappings.

Introduction

In daily life we frequently switch between visuomotor mappings without a moment's thought. To give a very crude example, when we participate in traffic on the streets, there are similar constraints and visual inputs regardless of whether we are driving a motorcycle or a car. But, depending on the vehicle we need to apply completely different mappings from visual input to motor behavior in order to avoid accidents, etc. Yet, we never run into the problem of accidentally applying the visuomotor mapping associated with a motorcycle when we are driving a car or vice versa. So the visuomotor system has learned to switch between motor mappings as we switch vehicles. There is even evidence that seeing the vehicle may already allocate the appropriate motor mapping, as can be argued from priming effects on object recognition when showing objects that have a similar way of interaction relative to showing objects that only look similar (Helbig, Graf, & Kiefer, 2006). That is, seeing the vehicle may serve as a cue for activating the corresponding motor mapping after learning to drive the vehicle. 

Similarly, as most people who wear glasses may have experienced, when first putting on a new pair of glasses we tend to get dizzy, despite being able to see better. This dizziness is due to the fact that the glasses lead to geometric distortions in the retinal image, leading to changes in multi-sensory and sensorimotor interactions that we are not yet used to when first donning the glasses. However, after some time the dizziness goes away as you adapt to these distortions, i.e., the sensorimotor interactions become normal again. And after just a few days of putting the glasses on and off every day, adapting to glasses-on and glasses-off conditions becomes immediate, i.e., we do not feel dizzy at all when putting on the glasses or taking them off and sensorimotor interactions remain normal throughout. This means we have learned both glasses-on and glasses-off mappings and can switch automatically between them. 

The purpose of this study is to investigate the learning process that underlies the establishment and switching of multiple distinct visuomotor mappings. Particularly we ask what information is actually being stored with respect to the mappings. As a first approach we use a pointing task, similar to the tasks used in prism adaptation studies. From the prism adaptation literature, it is known that different visuomotor mappings can simultaneously be learned and maintained (e.g., Martin, Keating, Goodkin, Bastian, & Thach, 1996; McGonigle & Flook, 1978; Welch, Bridgeman, Anand, & Browman, 1993). However, it is not clear what is actually learned with regard to the mappings that enable us to switch between them. In principle there are two possibilities. The first possibility is that each individual mapping is individually stored in an absolute sense independent of other recently experienced mappings. This would mean that we would be able to switch to those previously learned mappings regardless of what the current mapping is, thus even if the system is significantly perturbed from its normal behavior by, for instance, having adapted to an entirely new mapping just beforehand. To relate this to the car example: you would be able to retrieve the car mapping independent of whether you've just travelled by motorcycle, by using roller blades, or any other means of transport. The second possibility is that what has actually been learned are not the individual mappings themselves, but only the ability to shift behavior by the amount consistent with the relative shift between the trained mappings. That is, we might have learned to go from motorcycle to car and vice versa, but not from roller blades to car or from roller blades to motorcycle. For a pointing task, learning the relationship between the mappings would mean that when cued, behavior is dependent on what the mapping was before the cue was presented. That is, if after training the system is perturbed from normal behavior by adapting to an entirely new mapping, the learned shift would be applied relative to this new current mapping rather than retrieving a specific absolute mapping. 

From the car/motorcycle example above it seems unlikely that we should code the difference (i.e., the relative shift) between those mappings rather than the mappings themselves. But the car and motorcycle mappings are also different in quite complex ways, each vehicle coming with its own set of special skills that have to be learned before being able to drive it. In contrast, when adapting to a new mapping in a pointing task we do not have to relearn or readopt the skill of “how to point” but just have to adjust the “where to point” with respect to the sensory input. In this case there is no reason to store absolute mappings per se, and learning a relative shift between two mappings would mean that you only have to learn one shift compared to two separate mappings. Moreover, our sensory systems are especially adept at teasing apart relative differences compared to determining absolute coordinates. For instance, for distance perception of sound sources it has been shown that we are much better at judging the distance between sound sources than at judging the absolute distance of a single source (e.g., Coleman, 1962). Similarly, visual motion in depth can only be perceived relative to a reference point or surface (see e.g., Erkelens & Collewijn, 1985; Regan, Erkelens, & Collewijn, 1986). Also the coupling of visual lateral motion and perceived depth is based on relative depth order rather than absolute depth (Sohn & Seiffert, 2006). Thus, since our sensory systems specialize in determining relative relationships it would make sense if the sensorimotor interactions would also be coded in relative shifts rather than absolute coordinates. 

Previous studies on the storage of multiple mappings so far only looked at the learning stage itself and for instance investigated how switching between two trained mappings became more efficient with training, rather than investigating what information has actually been learned (e.g., Kravitz & Yaffe, 1974; McGonigle & Flook, 1978). In those experiments both storage of the individual absolute mappings and learning the relative relationships between the mappings would lead to the same predictions for learning to switch. In the current study we will try to tease apart the absolute mapping and relative shift hypotheses. We do this by having participants, after they've learned two separate mappings, adapt to a new visuomotor mapping before cuing one of the two previously learned mappings. This will reveal whether, upon contextual cuing, they apply a shift relative to the current mapping or whether they retrieve the absolute mapping. 

In order to cue the separate mappings it is useful to have contextual cues that are not directly behaviorally relevant for the pointing task itself. We decided to pair each of the two trained mappings with a color cue by presenting the target objects in different colors during training. Participants were not informed about these cues or their meaning. The role of previously irrelevant cues in visuomotor adaptation has been investigated before. Most studies that involve cues for visuomotor adaptation, investigated whether the cue can elicit cue-contingent aftereffects of adaptation after only a short amount of training. Significant cue-contingent aftereffects have indeed been found for simply wearing the prism-glasses themselves, i.e., differential aftereffects were found dependent on whether the glasses were being worn or had been taken away (Kravitz, 1972; Kravitz & Yaffe, 1974; Welch, 1971); for auditory tones (Kravitz & Yaffe, 1972); head posture (Seidler, Bloomberg, & Stelmach, 2001); gaze direction (Hay & Pick, 1966; Pick, Hay, & Martin, 1969) and target color (Donderi, Jolicoeur, Berg, & Grimes, 1985). Such aftereffects generally are very quickly obtained but also relatively short-lived, so from these studies it is not directly clear what this will mean for repeated adaptation to the mappings. There is, however, also strong evidence that cues can become sufficient to switch between mappings after more extensive training. For instance, Martin et al. (1996) found that after a two-week period of training with and without prism glasses, simply the act of putting on or taking off the glasses was a contextual cue for the participants to adopt (or shift to) the associated visuomotor mapping. 

The study of Martin et al. (1996) also provides a first insight into whether absolute mappings or relative shifts are being stored. In that study Martin et al. (1996), unbeknownst to the participants, reversed the prisms in the glasses at the end of training, effectively reversing the required mapping for these manipulated glasses, keeping the contextual cue, i.e., the glasses themselves present. They found that the error that participants made when first donning these manipulated glasses was twice as large as could be expected from the current prism shift alone, indicating that, indeed, the participants shifted behavior to the mapping that was previously associated to wearing the glasses during the training phase. Furthermore, after prolonged adaptation to the manipulated glasses with the reversed prisms, i.e., participants learned a new mapping for the known glasses-on context, Martin et al. (1996) found significant and about equal aftereffects for both the glasses-on context as well as for the glasses-off context. This suggests that the two mappings cannot adapt independently but are always coupled by the same relative shift. However, in that study, reversing the prisms in the glasses means that the known glasses-on context suddenly has a new additional mapping coupled with it. Logically, this could be interpreted as the environment in general having changed, thus going beyond the scope of the context, rather than just the conditions for the separate glasses-on context having changed. If so, the change in mapping is treated as an additional disturbance independent of context and therefore, the observed change in behavior, i.e., the aftereffects occurring for both contexts, do not necessarily relate to how the context-specific information has been stored. The advantage of using color cues is that we can easily add new colors for adapting to new mappings after the training has been completed. In this way the correspondence between the trained mappings and their contextual color cues will remain intact when adapting to a new mapping after training. Switching to one of the trained contexts from such a new context will then be informative as to what information has been stored with respect to each trained context individually (i.e., whether the context represents an absolute mapping or a relative shift in behavior regardless of the previous context). 

Learning the mappings

Before we can investigate what information (relative shift or absolute mappings) is stored when learning two visuomotor mappings, our participants needed to learn the two separate mappings. For this purpose participants were trained on five consecutive days on which they repeatedly switched between two separate mappings, each with its own color context. To do this, we used the setup in Figure 1 to be able to manipulate the visuomotor mappings and the contexts independently. The two trained mappings were separated by 6.4°. We will refer to the context associated with a 6.4° offset mapping as Context 1 and to the context associated to the 0° mapping as Context 2. Note, however, that the contexts and their associations with specific mappings were counterbalanced across tests and participants, which leads to quite complex experimental details. See the Methods section for the full details on the methods used in this study. 

Figure 1

 

A) Experimental setup. Targets were presented on a large back projection screen and participants pointed to the corresponding positions on a graphics tablet. Head movements were restricted by a chin-rest and participants could not see their own arm. B) Time course of a single trial. Participants started the trial by tapping at the start location after which a target was shown. The trial ended when the participant again landed on the tablet and visual feedback about the tapped location was shown immediately. C) Use of cued mappings. Two mappings are being trained. One for which the feedback is veridical (top) and one for which the visual feedback is horizontally displaced (middle). Each of these two mappings comes with its own color context. After training we expect people to adopt the 6.4° mapping whenever the corresponding context is presented (bottom) and vice versa. D) Schematic of training and test conditions. On training days participants were required to repeatedly switch between the 6.4° and 0° mapping each of which was associated with its own contextual cues. For the last two mapping switches for each training session catch trials (i.e., no feedback was provided) were included to test the progress of learning. On Day 5, test blocks were included in which the participant adapted to a new 12.8° mapping using a neutral context. After adapting to this new mapping, catch trials with either the neutral context itself, Context 1 or Context 2 were interleaved to test what information with regard to the trained mappings had been stored.

In order to track the learning we used catch trials, i.e., trials that directly followed a context change but for which no visual feedback was provided. These catch trials were presented towards the end of each training session (see Figure 1D). If the participant learned the context-associated mappings, presenting them with, for example, Context 1 should result in their behavior switching to the 6.4° mapping. Similarly, presenting participants with Context 2 should lead them to switch to the 0° mapping. Figure 2 shows the results for these catch trials. Figure 2A shows the average mapping participants adopted on catch trials for the two separate contexts and expresses the learning results in terms of absolute mappings. Figure 2B shows the behavioral shift in mapping as a result of the context change versus the five consecutive training days. The behavioral shift in mapping was obtained by taking the difference in behavior between catch trials and the directly preceding three trials. In this way Figure 2B shows the learning effect in terms of relative behavioral shift. The results show that with the change in context, behavior immediately shifts towards the associated mapping and that the extent of the shift grows across days. This shows that learning of the contextual cues and the associated mappings did occur. That is, no behaviorally relevant error signal (i.e., through presentation of the pointing position feedback) is needed to adopt the learned mapping, but simply showing the right context suffices after training. The learning rate is however relatively low and even on Day 5 the learning is not yet fully complete for at least some of our participants, as indicated by the shift in behavior not covering the complete difference of 6.4°. 

Figure 2

 

Training results. A) The adopted mapping upon being cued with the corresponding context on catch trials. B) Extent of the applied relative shift upon a context change averaged across contexts. Error bars represent standard errors across participants. The results indicate that learning of the context-mapping associations occurred.

Absolute mapping or relative shift: Adding a new context mapping pair

The above results show that the training was effective for learning, both the mappings and the association with the corresponding context. But what information has exactly been stored with respect to the individual contexts when switching between the mappings: the absolute mapping between target location and pointing behavior or the relative shift between the experienced mappings? To answer this question, participants adapted to a new 12.8°¹ visuomotor mapping using a neutral context that had not been used in the training, before catch trials with the trained contexts were introduced. Even though the learning was not complete for some of our participants, the shift in behavior relative to this newly introduced neutral context mapping on these catch trials should be informative of whether absolute mappings or relative shifts have been learned in association with the individual contexts. If the participants learned absolute mappings for the corresponding contexts, behavior should shift toward the mapping associated with the particular context that is being probed. If participants learned relative shifts, the shift associated with the probed context should be applied relative to the newly adapted mapping (see also Figure 3A). In short, in this way the representation of the learned mappings associated with the contexts can be sampled. 

Figure 3

 

On the last day of the experiment participants adapted to a new visuomotor mapping before they were exposed to catch trials (no visual feedback) involving different contextual cues. A) Predictions for catch trials results depending on how the mappings are encoded: i.e., in terms of absolute mapping or relative shifts. B) Average participants' behavior during catch trials in terms of visuomotor mapping. Error bars represent standard deviations across participants. Results indicate that the contextual cues more likely signal absolute visuomotor mappings rather than relative shifts.

The predictions for our two distinct absolute mapping and relative shift hypotheses are shown in more detail in Figure 3A. If the mappings are stored in absolute coordinates, the absolute mappings should be retrieved for the catch trials when the corresponding context cue is provided. That is, behavior for Context 1 catch trials should correspond to the 6.4° mapping since this was the mapping associated with that context during training (Figure 3A, left panel). Similarly for Context 2 catch trials behavior should switch to a 0° mapping. However, if the cue-mapping associations have been learned in a relative sense, the context change should signal that behavior needs to be shifted by a certain amount relative to the current mapping. In this case the previously learned relative shift of 6.4° should now be applied relative to the our newly introduced mapping of 12.8° leading to the predicted 19.2° mapping for Context 1 and 6.4° mapping for Context 2 (Figure 3A, right panel). Note, that for Context 1 the absolute mapping and relative shift predictions are in opposite directions and thus the results for this condition will be most informative about the mode of storage of the mappings. 

Figure 3B shows the results across participants. The bars show pointing behavior on catch trials for Context 1 (C1) and Context 2 (C2) as well as for the Neutral context (N), which was used to adapt to the new 12.8° mapping. Error bars show the standard deviations across participants. What is evident from the results is that for both Context 1 as well as Context 2 the behavior during the catch trials shifts toward the associated absolute mappings rather than applying a relative shift compared to the new neutral mapping. This is particularly evident for the Context 1 catch trials for which the two predictions are in opposite directions and the resulting behavioral shift clearly favors the absolute mapping hypothesis. However, the associated absolute mappings are also not completely reached, which is most evident for the Context 2 results for which there is still a relatively large gap between the results and the absolute mapping prediction for that context. This could be due to the fact that, as indicated above, the learning was not complete for some of our participants, which allows a substantial influence of the recently experienced mapping. It is likely that participants applied a weighting strategy between the recently experienced neutral context mapping of 12.8°, for which the task-relevant visual feedback would naturally have reinforced its reliability, and the absolute mapping indicated by the trained context cues. 

Absolute mapping or relative shift: Changing the associated mapping for Context 2

Our results indicate that visuomotor mappings are stored in an absolute sense rather than as a shift between mappings. However, to test our predictions the context changed from a neutral context to one of the trained contexts, either Context 1 or Context 2. This leaves unaddressed the possibility that when switching between Context 1 and Context 2 directly, as was the case during the training, a different switching strategy, i.e., relative shift, is being used. Indeed, if participants did learn the relative relationship between the specific pair of Context 1 and Context 2 this information would be rather useless when switching between either one of these contexts and any third context as in our test conditions presented above. That is, a direct relative coupling between Contexts 1 and 2 would be uninformative for the relationship between Context 1 and the Neutral context as well as for the relationship between Context 2 and the Neutral context. In this light, it would also not be very surprising that we did not find any evidence for storage of relative shifts with respect to either Context 1 or Context 2 when measured in isolation. In short, it could be that a relative shift component is more prominent when switching directly between the two trained contexts, rather than when the effects of context are measured relative to a third neutral context. 

If indeed the system has learned the relative shift between the pair of Contexts 1 and 2, changing the associated mapping for one of the contexts should also affect the associated mapping for the other, since they would be coupled through the constant relative relationship. To test this possibility, it makes sense to adapt one of the trained contexts, e.g., Context 2, to a new mapping and test what happens when switching to the other context, Context 1, using catch trials. If only absolute mappings are being stored with respect to the contexts, the results for the unchanged Context 1 should not be any different from the previously trained absolute mapping for this context. If, however, the relative relationship between Context 1 and Context 2 has been learned, the mapping applied for the nonadapted Context 1 should be shifted to the same extent as the directly adapted Context 2. 

As mentioned above, we chose to change the associated mapping for Context 2 in this part of the experiment. More specifically, participants adapted to the 6.4° mapping while being cued by Context 2 that had until now been associated with the 0° mapping. After adaptation to this changed Context 2 we again introduced catch trials for the different contexts and in particular for Context 1. If the specific switch between Contexts 1 and 2 is associated with a relative shift in mapping, the result for the unchanged Context 1 should now be relative to the new mapping for Context 2, i.e., 6.4° added to the new 6.4° mapping for Context 2, rather than the absolute mapping associated with Context 1. As a control condition we added catch trials for a neutral context to see the change in behavior as a result of context change per se (i.e., without the new context after the change having a clear mapping associated with it). 

For a better comparison, we repeated the same test using a neutral context to adapt to the 6.4° mapping. In this case neither the mapping associated with Context 1 nor the mapping associated with Context 2 should be affected. Particularly the difference in results for Context 1 between these two test conditions (either adaptation occurring using a Neutral context or changing Context 2 through adaptation) should be informative of whether a relative shift is only encoded for the particular context change from Context 2 to Context 1. If the results for Context 1 are the same in both conditions, then clearly the particular context before the context change is not of importance. If a relative shift is specific for switches between Context 1 and Context 2, a relative shift should occur relative to the changed mapping for Context 2 but not when the adaptation to the 6.4° mapping occurred using the Neutral context. 

The results of this test if relative information is used when switching between Context 1 and 2 directly are displayed in Figure 4. Figure 4A shows the results when participants adapted to the 6.4° mapping using the known Context 2 that had previously been associated with 0°. Figure 4B shows the results when a neutral context was used for the same adaptation to 6.4° as a comparison. The bars show the average results for the catch trials for the different contexts that were interleaved after the initial adaptation. If relative information is specific for switches between Context 1 and Context 2 and vice versa, we expect the results for the unchanged Context 1 (C1 in Figure 4) to depend on which context is used for the adaptation. That is, we would expect the result for C1 to be the trained shift away from the changed Context 2 (C2) in Figure 4A but not with respect to the adapted Neutral context (N) in Figure 4B since the neutral context was not part of the training and no relative relationship with respect to this context could have been learned. However, the figure shows that particularly for the results of Context 1 it does not matter whether a neutral or a known context was used for the initial adaptation. That is, in both graphs the results for the Context 1 catch trials end up at the absolute mapping prediction for that context. In fact, the only significant difference is the difference between the results for Context 2 catch trials when previously adapting using a neutral target color (C2 in Figure 4B) and the other results (p < 0.01). This difference simply represents the difference in Context 2 associated mapping for each test (adapted to 6.4° in Figure 4A and unchanged 0° mapping in Figure 4B). 

Figure 4

 

Results changing the mapping for Context 2. Participants adapted to a 6.4° mapping using either the familiar Context 2 (A) or a neutral context (B) cue in different blocks before exposure to cued catch trials. Error bars represent standard deviations across participants. The mapping for Context 1 is not affected by adapting to a new mapping beforehand using a different context, regardless of whether the adapted context is a known or a neutral one. C) The relative difference between Context 1 and Context 2 catch trials results in both A and B conditions as well as the difference as observed during training. Error bars represent standard deviations across participants. The difference observed when the mapping for Context 2 is changed is much less than expected based on the training results.

To demonstrate more clearly what these results mean in terms of the possibility of relative shifts being learned, Figure 4C displays the relative difference between Context 1 and Context 2 catch trial results for both conditions as well as the expected difference based on the catch trials during the training blocks on the same day. Note that when the mapping for Context 2 was changed during initial adaptation the difference between Context 1 and Context 2 catch trials (first bar) is significantly smaller than can be expected based on the training results (last bar; p < 0.05), which is inconsistent with relative shifts being learned. Instead, when the neutral context was used for initial adaptation (middle bar), leaving associations for the individual contexts Context 1 and Context 2 intact, the difference between Context 1 and Context 2 did not differ from the difference observed during training (p > 0.05). 

Together these results indicate that even when directly switching between the trained contexts there is no evidence that the relative relationship between those contexts is being used. Thus, rather than learning information regarding the pair of trained contexts, the independent contexts themselves are cues to adopt the corresponding behavior. Furthermore, these results reconfirm that it is the absolute mappings that have been learned and not the relative shift between the mappings. 

Discussion

Here we investigated the learning and usage of multiple (cued) visuomotor mappings. We found that participants were able to learn and maintain two visuomotor mappings at the same time, replicating previous results (e.g., Martin et al., 1996; McGonigle & Flook, 1978; Welch et al., 1993). Moreover, we found that after training with color context cues indicating the separate required mappings, presenting the color context is sufficient to switch visuomotor mapping. This means that after training a switch in mapping can occur without receiving any behaviorally relevant feedback about the pointing movement itself, but by simply presenting the learned color contexts. The main purpose of this study, however, was to determine what exactly is learned when training occurs on two separate mappings. Are the mappings separately stored in an absolute sense or do we learn the relative shift between the two mappings? We found that upon being presented with one of the trained contexts after adaptation to an entirely new mapping, the participants' behavior shifted towards the corresponding absolute mapping rather than relatively with respect to the current mapping. Thus, our results indicate that participants learn absolute mappings rather than the relative shift between the mappings. 

Whether associative learning between the contexts and mappings occurs automatically or whether conscious awareness of the role of the context cues is needed for associative learning to occur is not completely clear from this experiment. Upon debriefing, all participants indicated they had become aware of the role of the context cues. Therefore, we can neither rule out nor confirm that conscious awareness of the cues is a necessary condition for associative learning to occur in this case. However, this does not necessarily mean that the mode of storing the associated mappings is possibly based on conscious strategies as well. Here, it is important to note that on test conditions, specifically sampling the mode of storage of the associated mappings, all participants adopted the same behavior. If storing the mappings in terms of absolute mappings or relative shifts were a matter of conscious choice, we would have expected more variation. Thus, this indicates that storing absolute mappings in association to the context cues is the default. 

Does it make sense that the separate visuomotor mappings are stored independently in an absolute sense rather than their relative relationships? This is perhaps not so clear when only two mappings are considered. In the two mappings case coding the relative relationship only requires one shift to be stored versus two separate mappings when the mappings are stored independently. However, already with only three mappings this benefit is lost and the number of relative shifts needed to be learned and stored rapidly increases with the number of mappings N by N(N − 1)/2. Whereas, if the system stores the mappings independently, the storage capacity needed increases only linearly with the number of mappings. 

However, our results also provide evidence that the mappings are not completely independent of the current mapping. That is, at least there is an influence of the current state of the mapping when the participants are cued to go to either of the trained mappings. This can be seen in Figure 3B where there is a relatively large gap between the results and the absolute mapping prediction for each context. Thus, it seems that participants do not abandon the current mapping completely upon being presented with a different context indicating a different mapping but rather end up somewhere in-between the current and the cued mappings, weighing the two different sources of information. 

Note that this is very similar to when we are presented with behaviorally relevant visual feedback. For instance in normal circumstances, when there is a mismatch between our estimate of the mapping and the mapping that is actually currently required, we can see the error between target and pointing locations that tell us that we need to adjust our aim. However, when we are presented with such an error, we do not adjust behavior to the full extent of the error but instead weigh that information against our current estimate of the mapping and end up somewhere in between on the next movement (e.g., Baddeley, Ingram, & Miall, 2003; Burge, Ernst, & Banks, 2008; Körding & Wolpert, 2004; Korenberg & Ghahramani, 2002). In other words the information provided by the feedback, which has uncertainty, is weighed against the prior estimate of the mapping required that also has uncertainty. In this experiment, it was not the visual feedback but the contextual cues that carried the information that a different behavioral mapping needed to be adopted. The mappings associated with the contextual cues also have uncertainty just like the current mapping and thus the resulting behavior is likely to end up somewhere in-between, depending on the amount of uncertainty in each. Note, however, that there is a fundamental difference between the visual error feedback and the color cues in how they indicate the required mapping. The error signal presented by the visual feedback in essence is a relative signal. That is, it signals how to shift behavior relative to the current mapping in order to be more accurate and in that way is directly behaviorally relevant. In contrast, our contextual cues appear to indicate a particular mapping in an absolute sense and need to be learned in order to become associated with a mapping. It is therefore, very likely that the contextual cues are easily overruled by visual feedback when both are present. This can be seen in the results when we changed the associated mapping for Context 2, in which case participants easily adapted to this new mapping for Context 2, indeed showing that the visual feedback has a high weight in comparison to the absolute mapping cued by that context. 

Conclusion

Here we have shown that participants can learn the associations between color context cues and visuomotor mapping in a target-pointing task. We found that, rather than storing the relative relationship between the two mappings, the mappings associated with the context cues are stored independently in absolute coordinates. On the other hand, upon being presented with the context cue, behavior generally does not completely shift to the associated mapping, but is also influenced by the immediately preceding mapping. We suggest the system weighs the evidence presented by the contextual cues for their associated absolute mapping against the evidence of the current mapping in a similar manner as would normally be the case for visual feedback about the pointing error. 

Methods

Apparatus

Stimuli were displayed on a large back-projection screen (220 cm × 176 cm; JVC DLA-C15E, JVC Deutschland GmbH, Bad Vilbel, Germany) in an otherwise dark room. Participants were seated behind a custom-made rack (see Figure 1A). The pointing behavior of the participants was recorded using a graphics tablet (WACOM Intuos 3 A3-wide; active area 48.8 cm × 30.5 cm and a grip pen; Wacom Europe GmbH, Krefeld, Germany) placed on the first level of the rack. A second level draped in black cloth prevented the participants from seeing their own arm or the graphics tablet. The head movements of the participants were restricted by a chin-rest and the viewing distance was 53.5 cm. The visual stimuli were implemented in C using OpenGL libraries. Distances on the screen were mapped one-to-one to distances on the tablet meaning that for the target display only a small portion of the screen was actually being used. 

Stimulus, task, and trial procedure

Participants used their preferred hand for pointing. They were told to hold the grip pen in a full grasp to ensure that the posture of the hand relative to the tablet would be more or less constant across days. Individual trials were initiated by tapping with the pen within the start zone, which was a semicircular area (radius of 25 mm) centered on the lower edge of the tablet (see Figure 1B). A small bump on the graphics tablet haptically indicated this starting position. Participants were told to position their nonpreferred hand on this bump as a guidance to blindly find this starting position more easily with the preferred hand used for pointing. Before trial start, a white horizontal line on the screen provided a visual reference for the vertical position of the starting position. This horizontal line was displayed along the whole width of the screen so as not to provide a visual reference for the horizontal direction. 

Upon trial initiation a target (disk of radius 0.75°) was displayed for 0.5 sec after which both the target and the horizontal white line disappeared. The participant's task was to try to tap on the corresponding location on the graphics tablet as accurately and as quickly as possible. When the movement was complete (i.e., the participant landed on the tablet) feedback about the tapped location was displayed in form of a Gaussian blob (standard deviation of 3.0°) at the corresponding visual location on the screen (Figure 1B). By manipulating the horizontal position of this feedback we introduced different visuomotor mappings between visual target location and the required motor response. That is, for visuomotor-adaptation conditions we manipulated the position of the visual feedback by horizontally displacing it from the actual tap locations (see Figure 1C). 

Since adaptation in the horizontal and vertical directions have been shown to be independent of each other (Burge et al., 2008), we decided to simplify our design by studying adaptation along the horizontal dimension only. This means we chose to manipulate both the target and feedback location only in the horizontal direction. Target locations were chosen randomly from within a horizontal range of 16.0° visual angle centered at 0 (straight ahead). The vertical position was always 15.7° above the horizontal white line. All visual displays had an intermediate gray background color. 

Training

The first part of the experiment was to train participants on context-color-contingent visuomotor mappings. Training took place in five sessions (1 hour each) on five consecutive days. Participants were not informed about the purpose of the context color. 

The procedure was as follows. To familiarize the participants every day with the pointing task, each session started with a short block of 50 trials with veridical feedback (i.e., corresponding to the 0° mapping) and a red context (on the first day this block consisted of 200 trials). The data of this block was not used for analysis. 

After a 2-minute break, five experimental training blocks followed. Each block consisted of three stages (see Figure 1D): a) a random number of trials (between 20 to 40 to prevent counting) for which the context color was red and the mapping 0° (i.e., visual feedback was not shifted); b) 40 trials for which the context color was cyan and the position of the feedback was such that the mapping was displaced by 6.4° visual angle (half the participants were trained with a positive 6.4° mapping and the other half with a negative 6.4° mapping and for the analysis we combined both groups by mirroring their response at 0); c) 100 trials for which the context color was again red and the mapping 0°. 

To track the progress of learning the color-cue for the associated mapping the fifth and last block in each training session included catch trials. These catch trials were the first three trials following a color change (from red to cyan or from cyan to red) and no visual feedback was provided for these trials (Figure 1D). If subject followed the cue in the catch trials this means they have successfully learned the cue-mapping contingency. After each block in the session the participants were forced to take a 5-minute break to prevent fatigue. 

The color context for each trial consisted of several different elements. First the pointing target itself had the color of the current context. Second, to allow early registration of the color context, the context was already provided before the trial actually started by printing the question “ready?” on the screen in the color corresponding to the next trial. Also whenever the target was about to have a different color in the next trial compared to the previous trial, the message “hit space bar” would appear on the screen. This message was also written in the same color as the target would be in the next trial (as before, participants were not informed about this color contingency). Following this message, participants had to press a button on the graphics tablet to continue with the next trial in the training block as usual. In this way the color cue would be presented well before the participants would make their first movement for the cued mapping. 

Absolute mapping or relative shift: Adding a new context mapping pair

On Day 5, after training, we tested whether this mapping change was encoded absolute or relative. These tests were performed in block 3 and block 5 of the session. Participants first adapted to a new mapping with a yellow contextual color before catch trials with the learned color contexts were introduced. This new mapping was here learned the first time, so there was no association to one of the old overly-trained color cue-mapping contingencies. To prevent interference from the learned context-mapping associations a novel yellow context color was used for the adaptation to this new mapping. In one of two test blocks participants adapted to a 12.8° mapping in the other to a −6.4° mapping. To adapt to the new mapping, each test block started with a random number of trials (between 70 and 90) in which the context color was yellow and with feedback corresponding to the new mapping. To examine in what way the previously learned mappings are used, short catch trials periods (of five trials each) were introduced after the initial adaptation phase. For these catch trials no feedback was provided and the target color switched to one of the previously learned color contexts (red or cyan). The change in behavior on these catch trials is informative of the learned cue-mapping associations (absolute or relative). As a control to examine if behavior changes for catch trials regardless of the context color we also included periods of yellow context color catch trials (i.e., for these catch trial periods the feedback was simply removed for five trials without changing the color context). For each color context in this design (i.e., cyan, red, and, as a control, also yellow) there were two such catch trial periods of five trials each. To ensure that all catch trial periods started more or less from the same initial mapping the catch trial periods were interleaved with a random number of trials (between 15 and 25) in which the context color was yellow and feedback corresponded to the initial mapping (either 12.8° or −6.4°) for that block (top-up adaptation). At the end of each test block participants adapted back to the 0° mapping in 50 trials. 

The test using the 12.8° and −6.4° neutral context mappings can be considered mirror conditions with respect to the previously trained mappings. For simplicity reasons, and to control for asymmetries in the egocentric motor space, the results of the two test blocks are combined in Figure 3 by mirroring the results of the −6.4° in the appropriate way before averaging. Note that this means that, e.g., for Context 1, the results for the cyan context from the 12.8° test are combined with the results from the red context from the −6.4° test. 

Absolute mapping or relative shift: Changing the associated mapping for Context 2

To test whether a context switch or just the context itself is important for learning the mappings, the training and testing procedure was extended after the fifth day for five of our participants. Days 6 and 8 were normal training sessions in the exact same manner as on Days 1 to 4. Day 7 included the same tests as on Day 5 and we will not discuss the results here since they were quite similar as the results of Day 5. Day 9 was again a test day, following the same overall scheme of training and test blocks as in Day 5, but this time different tests were performed in the third and fifth block. 

If participants learn to switch behavior in response to a switch in context rather than simply in response to the context itself, different results can be expected if a new mapping is learned using a known context versus when a neutral context is used. To investigate the potential role of the context switch, in the test blocks on Day 9 the role of the neutral cue to adapt to a new mapping versus using one of the known cues was assessed. To make the adaptation phase less conspicuous, especially for the known context condition, we gradually increased the mapping up to 6.4° during the initial adaption in the test blocks. Kagerer, Contreras-Vidal, and Stelmach (1997) and Michel, Pisella, Prablanc, Rode, and Rossetti (2007) used a similar gradual shift paradigm for rotated feedback conditions and found that changes in mapping were both less conspicuous and led to bigger aftereffects, thus were more fully adapted to. The adaptation in smaller steps thus also means that we can more safely use the smaller initial adaptation amplitude. The gradual increase up to the 6.4° mapping was achieved by ramping up the mapping in six small steps of, on average, 1.07° per 10 trials. A random number of trials (between 0 and 20) were added once the amplitude of 6.4° was reached. In one block the yellow context color was used, in the other the trained red context color, previously corresponding to the 0° mapping. After this initial adaptation period, periods of five catch trials (again two times for each of the red, cyan and yellow target colors) were included in the exact same manner as for the test blocks on Day 5, using the initial adaptation context color (yellow or red respectively) for the interleaved top-up adaptation trials. At the end of the test blocks participants de-adapted to the 0° mapping (also ramped down in small steps of on average 1.07°/10 trials). 

The reasons for having a smaller initial adaptation amplitude (6.4° vs. 12.8° on Day 5) are twofold: first the test blocks were already relatively long compared to the training blocks and to adapt to a 12.8° mapping using these smaller steps would extend the test blocks even beyond the number of trials we think reasonable without a break. Second, some of the results we found up until Day 5 suggests that there is tendency towards a 0° mapping and also to keep changes in pointing behavior relatively small compared to the change needed to be fully accurate. So if there is a relative shift component to the cue we will be more likely to find it if we start out with a mapping that is closer to the 0° mapping. An initial mapping of 6.4° should still be sufficient to conclude between the absolute mapping (6.4°) versus relative shift (12.8°) hypotheses for the cyan color context. 

Participants

Eight participants, ages ranging from 20 to 30, gave informed consent and participated in the experiment. All participants had normal or corrected-to-normal vision and were right-handed. To motivate participants to improve their performance in terms of accuracy and speed we provided a score and a high-score list. The score was calculated based on accuracy and movement completion time. It was presented only at the end of a session so it did not provide any feedback on a given trial or about the cue-mapping contingency. After the experiment was complete participants were debriefed using a questionnaire to investigate to what extent they were aware of the role of the context cues. 

The results for one participant were removed because it turned out he did not adapt very well to the different mappings and thus could not perform the task. 

Acknowledgments

This research was supported by the Human Frontier Science Program, EU Grant 27141 “ImmerSence” and EU grant 248587 “THE.” The experiments were conducted at the Max Planck Institute for Biological Cybernetics in Tübingen, Germany. 

Commercial relationships: none. 

Corresponding author: Loes C.J. van Dam. 

Email: [e-mail] 

Address: Department of Cognitive Neuroscience, Universität Bielefeld, Bielefeld, Germany. 

Footnotes

¹   The side of the neutral mapping relative to the training range was counterbalanced in separate tests. That is, in a separate test participants adapted to a −6.4° neutral context mapping. The results are combined by mirroring the appropriate data. See Methods section for full experimental details.