Predictability is necessary for closed-loop visual feedback delay adaptation

Abstract

Abstract

Abstract:

Abstract  In case of delayed visual feedback during visuomotor tasks, like in some sluggish computer games, humans can modulate their behavior to compensate for the delay. However, opinions on the nature of this compensation diverge. Some studies suggest that humans adapt to feedback delays with lasting changes in motor behavior (aftereffects) and a recalibration of time perception. Other studies have shown little or no evidence for such semipermanent recalibration in the temporal domain. We hypothesize that predictability of the reference signal (target to be tracked) is necessary for semipermanent delay adaptation. To test this hypothesis, we trained participants with a 200 ms visual feedback delay in a visually guided manual tracking task, varying the predictability of the reference signal between conditions, but keeping reference motion and feedback delay constant. In Experiment 1, we focused on motor behavior. Only training in the predictable condition brings about all of the adaptive changes and aftereffects expected from delay adaptation. In Experiment 2, we used a synchronization task to investigate perceived simultaneity (perceptuomotor learning). Supporting the hypothesis, participants recalibrated subjective visuomotor simultaneity only when trained in the predictable condition. Such a shift in perceived simultaneity was also observed in Experiment 3, using an interval estimation task. These results show that delay adaptation in motor control can modulate the perceived temporal alignment of vision and kinesthetically sensed movement. The coadaptation of motor prediction and target prediction (reference extrapolation) seems necessary for such genuine delay adaptation. This offers an explanation for divergent results in the literature.

Introduction

It has been known since the 19th century that humans can adapt to spatial perturbations of visual feedback, for instance, when wearing prism glasses that displace the visual field (von Helmholtz, 1867). This kind of adaptation alters both perception and behavior in a semipermanent way. When the perturbation is removed after adaptation, a participant will keep up the compensatory behavior for a short period of time, leading to negative aftereffects, generalization of the adaptive strategies to nonadapted situations, and to corresponding perceptual biases (cf. Bedford, 1999; Welch, 1978). There remains, however, considerable controversy about whether and to what extent such adaptation is also possible for temporal distortions of visuomotor mappings, such as increased visual feedback delays. This controversy is driven by conflicting results currently present in the literature (cf. section “Previous studies on delay adapation” [p. 4]). 

Here, we test the hypothesis that a predictable reference signal (e.g., the target to be tracked in a manual tracking task) is a necessary factor for semipermanent delay adaptation. With a predictable reference signal, we find evidence for recalibration (aftereffects) to 200 ms feedback delays in all investigated measures: 

• (a)   
  Spatial error (tracking performance).
• (b)   
  Temporal error (anticipatory movements).
• (c)   
  Spectral power in the range of the reference motion, which indicates alterations in the use of feedback.
• (d)   
  Shifts in the point of subjective simultaneity, which indicates temporal recalibration on the level of perceptual experience.

The fact that there are aftereffects in all measures employed leads us to the conclusion that, after predictable delay adaptation, participants recalibrate different delay compensation mechanisms to the feedback delay: 

• (a)   
  Motor prediction (i.e., the delay-compensated estimation of hand position from visual feedback and movement history).
• (b)   
  Reference extrapolation (i.e., anticipation of the target movement).
• (c)   
  Perceptuomotor delay compensation (i.e., recalibration of the perceived temporal alignment between kinesthetically sensed movement and visual feedback).

If there is an unpredictable reference signal, participants do not exhibit this wide range of aftereffects, despite being exposed to the same feedback delay. Particularly, there is no recalibration of perceived simultaneity between visual and movement events, and no overanticipatory aftereffect in temporal error. To get a better understanding of the variables we report, we first briefly review some of the known effects of visual feedback delays on visually guided manual behavior and how they can be compensated. 

Visual feedback delays in visuomotor behavior and perception

There are naturally occurring latencies between the sensation of visual events and a possible motor reaction, which depend on the task (Poulton, 1974) but are often estimated to be around 150 ms (Miall, Weir, Wolpert, & Stein, 1993) or 180 ms (Poulton, 1974). Real-time interactions with the environment, such as catching a ball in flight, therefore require delay compensation. By the time a currently issued motor command takes effect, both the hand and the ball will have moved from the place where they are currently seen. These discrepancies can be predicted and compensated if visuomotor delays are known, as illustrated in Figure 1. 

Figure 1

 

Schematic of visuomotor prediction with different internal delay estimates Δt in the unpredictable (A) and predictable (B) manual tracking tasks used in this study. Top: short internal delay estimate Δt₁ (pretest), bottom: long delay estimate Δt₂ (after adaptation). A: The internal delay estimate determines how much of the motor history (leftward movement, represented by dotted line) is taken into consideration to estimate the current hand position x′ (orange circles). This leads to differences in the estimated error signal e′ used for correction (orange bars). B: In the predictable task, the estimated error signal e′ is additionally influenced by the target prediction (reference extrapolation r′ from predictable target path, gray empty circles), which is not available in the unpredictable condition. The reference estimate r′ is also dependent on the internal delay estimate Δt.

There are (at least) three different mechanisms involved in compensating for visuomotor delays. 

Motor prediction helps us to make better use of delayed sensory feedback. The current state of the hand (e.g., its horizontal position x) can be estimated from delayed visual feedback (x_vis) and the history of motor commands (Figure 1A). That is, the expected sensory change due to self-generated movement during the estimated feedback delay time Δt can be combined with the visual feedback to compute a delay-corrected estimate the hand position x′. This leads to a better error estimate e′. Forward models that internally simulate the effects of motor outputs are possible mechanisms for motor prediction. For instance, Miall et al. (1993) proposed that the human cerebellum might implement a Smith predictor, i.e., a forward model to estimate the effect of an issued motor command from internal simulation, combined with a separate circuit for feedback delay compensation. With this or a different motor prediction mechanism, the hand position estimate x′ depends on an accurate estimate of the visuomotor delay Δt. 

Furthermore, also the future position and movement of the reference r can be extrapolated for better control if a task is predictable, which can be seen as a form of visual motion extrapolation (gray empty circles in Figure 1B). The estimate r′ depends on the internal Δt parameter, as this indicates the amount by which such motion extrapolation is performed. We refer to the prediction of target movement here as reference extrapolation. Successful reference extrapolation improves feedback control, as e′ depends on r′ (Figure 1B horizontal pale red lines), as well as open loop motor planning, as also the predicted movement direction of r′ depends on Δt (black arrows above r′_Δt1 and r′_Δt2 in Figure 1B). 

Finally, despite constantly compensating for sensory feedback delays, we usually have no conscious awareness of their existence. During interaction with the environment, the kinesthetic sensation of movement feels aligned with visual feedback. This implies that also our perceptuomotor system has mechanisms to compensate for delays between, at the very least, efferent signals, vision, proprioception, and often also touch on the experiential level. We refer to this as perceptuomotor delay compensation. 

Confronted with additional visuomotor delays, all these compensation mechanisms fail at first (in Figure 1, this corresponds to a situation where Δt₂ is the actual delay and Δt₁ is the estimated delay). In manual tracking, this leads to the following behavioral effects: 

• –   
  Participants misestimate the amount by which the visual input x_vis and the actual hand position x differ (wrong Δt in motor prediction). This can cause repeated oscillatory overshooting of the reference because participants feel they have not yet reached the reference when they have already surpassed it (x′_Δt1 vs. x′_Δt2 in Figure 1A; cf. also Miall et al., 1993; Steinbach et al., 2012). This destabilization of control manifests as an increase in the spectral power of hand movement compared to the reference motion (e.g., Foulkes & Miall, 2000). It also causes spatial errors (bad tracking performance) and temporal errors (laggy reaction to reference motion). These measures are investigated in Experiment 1.
• –   
  A temporal miscalibration of reference extrapolation implies that reactions to predictable changes of the reference r are too slow (r′_Δt1 vs. r′_Δt2 in Figure 1B), which will also result in temporal errors and, to a lesser extent, in spatial errors. Miscalibrated reference extrapolation alone (e.g., during open-loop tracking) cannot lead to spectral changes. Reference extrapolation aftereffects are investigated in Experiments 1 and 2.
• –   
  Lastly, a miscalibration of perceptuomotor delay compensation means that we are perceptually aware of temporal discrepancies between vision and kinesthetically sensed movement. The point of subjective simultaneity (PSS) gives an indication of the temporal alignment of vision with respect to kinesthetic (efferent, proprioceptive, tactile) cues. This measure is investigated in Experiments 2 and 3.

When behavior is spatially perturbed, for example by using a prismatic displacement, behavioral performance and perceptual experience can usually be adapted with practice. This is often thought of as a recalibration of parameters in internal models. For instance, adaptation to spatial perturbations can be modeled as a Kalman filter (e.g., Burge, Ernst, & Banks, 2008). If humans can adapt similarly to feedback delays, the Δt parameter should be recalibrated in all three compensation mechanisms (motor prediction, reference extrapolation, and perceptuomotor delay compensation) and all of the described perturbations should be at least partially neutralized. Additionally, if delay adaptation is semipermanent, negative aftereffects are expected in a nondelayed post-test (switch from Δt₂ to Δt₁ in Figure 1). That is, an increase in spatial errors, a decrease in spectral power (undershooting of r), and overanticipatory temporal error should be observed as a result of the recalibration. 

Previous studies on delay adaptation

The existing literature on delay adaptation is marked by controversies about the levels at which delay adaptation takes place, about whether adaptation on one level transfers to another, and about whether such changes are or can be semipermanent. 

Concerning perceptuomotor recalibration, numerous psychophysical studies have shown that humans recalibrate their perception of visuomotor simultaneity to partially compensate for changed visuomotor lags (e.g., Heron, Hanson, & Whitaker, 2009; Keetels & Vroomen, 2012; Rohde & Ernst, 2013; Rohde, Greiner, & Ernst, 2014; Stetson, Cui, Montague, & Eagleman, 2006; Sugano, Keetels, & Vroomen, 2010; Sugano, Keetels, & Vroomen, 2012). These paradigms usually do not involve continuous feedback or motor tasks with a performance criterion. Instead, exposure to a discrepancy is discrete and serves no motor goal; for instance, a visual flash may follow a button press with a fixed lag. These studies suggest that humans can use the statistics of how action and perception relate in time to recalibrate time perception. 

Compared to the numerous psychophysical demonstrations of temporal recalibration, evidence from visually guided motor control in favor of delay adaptation is sparse. Early studies (e.g., Ferrell, 1964; Smith & Smith, 1962) mostly documented the disruptive effects of visual feedback delays but reported no recalibration effects. In these studies, participants partially neutralized the destabilizing effects of a visual feedback delay but did not show the characteristic behavioral changes and aftereffects (cf. previous section). For instance, Ferrell (1964) observed that participants tended to employ one of two different strategies to stabilize behavior after the introduction of a delay in a remote manipulation task (gripping and moving of objects). The first strategy was a “move and wait” strategy, where the visual inputs were ignored during execution of fast movements to the target; a corrective movement was then performed once the visual feedback had caught up. The second compensatory strategy was to slow down movements (i.e., a decrease in control gain corresponding to slower response to feedback) until the destabilizing effects of the visual feedback delay disappeared (Ferrell, 1964). Neither of these compensatory strategies indicates adaptation of Δt in motor prediction, such as in a forward model, and both are suboptimal (i.e., motor performance after adaptation does not approximate initial levels). Smith and Smith (1962) similarly report only partial compensation of visual feedback delays in a number of tasks (e.g., drawing the outline of shapes) and no adaptation aftereffects. Poulton (1974) interpreted delay adaptation in sine wave tracking mostly as a process of reference extrapolation (only open-loop control was recalibrated). He also observed that corrective movements occur at a lower frequency, which is due to the fact that feedback delay Δt defines a lower bound for reaction times to unpredictable events. This limits the frequency at which fast corrective movements can be performed in the closed loop. It is important to note that recalibration of internal models (Δt in motor prediction and reference extrapolation) cannot compensate this effect of increased Δt and that the feedback delays can thus never be fully neutralized. This, however, is also the case for the naturally occurring latencies of approximately 150 ms. Vercher and Gauthier (1992) reported that participants could partially compensate additional feedback delays in a predictable sine wave tracking task. However, participants struggled to manage the feedback delay appropriately if movement direction was inverted (spatial error/overshooting). This suggests that only reference extrapolation, not the motor prediction, was recalibrated. In two more recent studies (Foulkes & Miall, 2000; Miall & Jackson, 2006) participants were trained with visual feedback delays in a manual tracking task. The authors observed adaptation aftereffects, which were mostly inconsistent with their predictions for recalibration of a Smith predictor. Participants compensated for the feedback delay by slowing down their movements, like in Ferrell's (1964) study. This sluggish tracking behavior carried over to the post-test, but there was no anticipatory behavior (in terms of temporal error). The authors concluded that no recalibration of internal delay parameters occurred. Held, Efstathiou, and Greene (1966) also showed that visuomotor delays interfere with spatial adaptation to prismatic displacements, an effect that increases with delay length (Held & Durlach, 1991). Tanaka, Homma, and Imamizu (2011) and Honda, Hirashima, and Nozaki (2012) tested in two recently published studies whether training with delays might alleviate the disruptive effect of visual delays on prism adaptation. The studies, despite their similarities, came to opposite conclusions: Tanaka et al. (2011) found that temporal adaptation had no effect on the rate of visuomotor adaptation to displacements, which speaks against delay adaptation. By contrast, Honda et al. (2012) reported that delay adaptation accelerated adaptation to a spatial perturbation. To date it is still unclear why these studies led to different results. However, the use of discrete or continuous feedback is mentioned as one possible explanation (cf. Honda et al., 2012). 

Yet, there are a number of other studies (Botzer & Karniel, 2013; Cunningham, Billock, & Tsou, 2001a; Cunningham, Chatziastros, von der Heyde, & Bülthoff, 2001b; de la Malla, López-Moliner, & Brenner, 2012; Kennedy, Buehner, & Rushton, 2009; Morice, Siegler, Bardy, & Warren, 2007) that reported delay adaptation aftereffects more consistent with semipermanent recalibration of internal delay parameters and transfer across domains. Specifically, Cunningham et al. (2001a) mentioned anecdotally that participants, after delay adaptation in a visually guided obstacle avoidance task, spontaneously reported a recalibration of perceived simultaneity: In the post-test, they felt that the cursor they controlled moved even before they moved their hand. Botzer and Karniel (2013), de la Malla et al. (2012), Kennedy et al. (2009), and Morice et al. (2007) also observed behavioral aftereffects of moving too early (inverse lags) in the post-test. However, in most of these studies, only some of the characteristics listed in the previous section were reported, which makes it difficult to assess how general delay adaptation was and specifically whether only reference extrapolation was recalibrated, or also motor prediction, and in how far these behavioral changes involved a recalibration of time perception. 

It is still an open question why results on delay adaptation in motor control are so divergent. Cunningham et al. (2001a) proposed that the time pressure in their task might be necessary to trigger delay adaptation (continuous and fast reference motion). However, Foulkes and Miall (2000) and Miall and Jackson (2006) also used a continuously moving reference signal in their unpredictable tracking paradigm and found no evidence for delay adaptation. We therefore propose that predictability of the reference motion might be another factor in delay adaptation (cf. Rohde, 2010). Recalibration in motor prediction and time perception might only be possible in combination with recalibration in reference extrapolation as a catalyzing process (Figure 1B). 

In order to put this hypothesis to empirical test, we trained participants in predictable and unpredictable variants of a continuous manual-tracking task with a 200 ms visual feedback delay. In Experiment 1 (motor behavior), we compared the aftereffects in motor behavior (spatial error, temporal error, band power) between the predictable and unpredictable conditions. We also tested for transfer of adaptation between the conditions. Participants revealed all characteristics of delay adaptation only when trained in the predictable condition, in agreement with the hypothesis. In Experiment 2 (synchronization), we tested whether such delay adaptation also involves recalibration of perceived simultaneity, using a synchronization task without visual feedback. We observed a PSS shift to partially compensate for the visual feedback delay only after training in the predictable condition. In Experiment 3: Interval estimation, we test for a similar PSS shift in an interval estimation task (participants had to judge the relative timing of a visual flash and a kinesthetically sensed motor event). The results concur with Experiment 2. 

Experiment 1: Motor behavior

Participants were trained in a manual-tracking task with a 200 ms visual feedback delay, with either predictable or unpredictable reference motion. Our hypothesis was that participants exhibit the full catalog of delay adaptation aftereffects only when they are trained in the predictable condition, where a coadaptation of motor prediction and reference extrapolation is possible. In the unpredictable condition, participants receive the same information about visuomotor delays, but do not recalibrate. 

Methods

Participants

Ten healthy adult volunteers participated in the study (age range 18–42, seven female, all right handed as by self-report). They received a small monetary compensation (6 €/hr) for their participation. All participants were naïve to the purpose of the experiment and signed an informed consent form. The experiment was conducted in agreement with the ethics standards laid out in the Declaration of Helsinki and was approved by the ethics committee of the Department of Medicine of the University of Tübingen (Germany). 

Apparatus

Participants were seated in an office chair in front of a table (Figure 2). They rested their chin on a chin rest at 1-m viewing distance from a large back-projection screen (220 cm × 176 cm; Eyevis Gesellschaft für Projektions-und Großbildtechnik mbH, Reutlingen, Germany) in an otherwise dark room. They could control the horizontal position of a cursor disk projected onto the screen by moving a pen left and right on a graphics tablet (WACOM Intuos 3 A3-wide; active area 48.8 cm × 30.5 cm and a grip pen; Wacom Europe GmbH, Krefeld, Germany). The tablet and their hand were occluded from vision (see Figure 2). The visuomotor task (section Procedure) was implemented using the psychophysics toolbox for Matlab (Kleiner, Brainard, & Pelli, 2007). The stylus motion was scaled so that 1 cm of movement on the graphics tablet corresponded to ∼4.76 cm of movement on the screen. The end-to-end latencies of the device were 60 ms (measured with the method described in Di Luca, 2010). To mask possible auditory cues, such as the scratching of the pen on the tablet, participants were played pink noise over headphones. 

Figure 2

 

Schematic of the experimental setup.

Task

Participants performed a manual-tracking task (Figures 3A and 3B), adapted from the unpredictable tracking task used in Foulkes and Miall (2000). A black visual reference dot of 3.3° visual angle on a 50% gray background moved left and right at participants' eye height in an unpredictable fashion (five nonharmonic sine waves overlaid with random initial offset in phase; frequencies: 0.09, 0.165, 0.195, 0.375, and 0.495 Hz). The most lateral position possible was at 24.4° visual angle, but most of the time it was much more central. Participants controlled the lateral position of a white cursor dot of 1.7° visual angle. 

Figure 3

 

Task and procedure for Experiment 1 on motor behavior. A: Tracking in the unpredictable condition. B: Tracking in the predictable condition. Tasks A and B were training tasks for Experiments 1 and 2. Task B was a training task for Experiment 3. In Experiment 1, Tasks A and B were also test tasks. C: Procedure Experiment 1 (motor behavior).

The task was to track the motion of the reference using a stylus on a graphics tablet with the dominant hand. Throughout the experiment participants could start a trial by pressing a button on the graphics tablet with the nondominant hand. For the first 2 s of each trial the cursor dot flashed blue and white (to signal “invincibility”). Afterwards, the cursor dot was red when it was outside the black reference dot (negative feedback) and white if it was inside. Consistent with these color changes, participants received points for time spent exactly on the path. This feedback encouraged the continuous use of feedback, as staying just relatively close to the reference does not gain them any points or a white cursor. The score indicating their tracking performance (percentage of time the white dot was inside the black dot) was provided as feedback after each trial. To further motivate participants they were presented with a high score list of made-up competitors after every third trial. Depending on their performance they moved up and down in the top half of this list. 

In the no-delay condition, participants performed the task with the imperceptible 60 ms system delay. In the delay condition, an additional 200 ms artificial feedback delay was injected between stylus movement and the display of the feedback cursor. This delay was clearly noticeable by participants and destabilized behavior (Figure 4). Still, it has been shown in previous research to be small enough for delay adaptation (cf. Cunningham et al., 2001b). 

Figure 4

 

Example tracking behavior (position across time) for different conditions and training phases for an example participant. Left to right: (a) at the end of pretest; (b) when the delay is introduced; (c) at the end of the training; (d) during the last post-test trial; and (e) during a post-test trial of the nonadapted condition (transfer). Top: U_training; bottom: P_training. Animations of these trials (i.e., the display as seen by the participant) can be found in the supplementary material.

In the unpredictable conditions (U_training and U_test), participants only saw the random movement of the reference dot (Figure 3A). In the predictable conditions (P_training and P_test), participants also saw the upcoming and past movement of the reference dot in a vertical window that scrolled down as the task proceeded (Figure 3B). This window corresponded to 2 s of temporal information (1 s into the future and 1 s into the past) and was displayed in 25% gray. To rule out that the temporal offset in the delay condition was interpreted as a fixed spatial offset, the vertical scaling of this 2 s window was varied between trials (randomly between 25° and 67.2° visual angle), i.e., the path was compressed or expanded in the vertical dimension without changing the temporal information provided. This window was fixed within a given trial. Note that even the unpredictable condition is not fully unpredictable (due to continuous changes in position, velocity, and acceleration of the reference), just as the predictable condition is not fully predictable (due to sensory and motor noise). Importantly, the reference predictability varies between the two conditions. 

Procedure

We used a 2 × 2 × 2 within-participant design, comparing pretest and post-test performance across two training and two test conditions (Figure 3C). Factor 1 is test phase with levels pre and post. Factor 2 is training condition with levels predictable (P_training) and unpredictable (U_training). Factor 3 is test condition with levels predictable (P_test) and unpredictable (U_test). The P_training and U_training session were conducted on different days. The order of the training sessions was counterbalanced across participants. 

Each experimental session started with a pretest block, in which participants' baseline performance in both the predictable and unpredictable conditions was assessed (12 × 60 s, alternating P_test and U_test, starting with P_test; Figure 3C). Afterwards, participants took a 5 min break. Participants were trained with the additional 200 ms feedback delay in the second block for 17 × 60 s in either the unpredictable condition (U_training) or the predictable condition (P_training). During the post-test phase, which followed without a break, participants were tested using shorter trials (10 s) without delay both in the predictable (P_test) and unpredictable (U_test) conditions. The post-test trials were shorter because from pilot experiments we expected a very rapid readaptation. The removal of the delay was announced in writing on the screen before the trial to make sure that participants' aftereffects are not influenced by cognitive strategies (e.g., due to false beliefs about visuomotor mappings). Participants were exposed to two top-up adaptation trials (60 s) with delay between each two post-test trials. Altogether, participants were tested in six post-test trials, alternating P_test and U_test trials, starting with P_test (Figure 3C). 

Analysis

Throughout the experiment, the position of the stylus on the graphics tablet, the cursor dot, and the position of the reference were recorded at 60 Hz. For preprocessing, participants' movement trajectories (stylus motion/cursor dot) were low-pass filtered with a 10 Hz Butterworth filter. The following variables of interest were then analyzed. 

Temporal error:

The cross-correlation between the reference path and the path of the feedback cursor was computed for lags up to ± 500 ms. The lag (in ms) at which the cross-correlation was maximal was used as a measure of temporal error (negative lag: cursor moves before the reference; positive lag: cursor lags behind the reference). 

Spatial error:

The tracking error was calculated as the mean squared deviation of the visual cursor from the center of the reference dot in square centimeters. 

Spectral power:

The power spectrum for participants' hand movement was computed for the frequency band 0.45–0.6 Hz. This frequency band contains the fastest of the sine wave frequencies that make up the reference motion (0.495 Hz; cf. Task). This sine wave component is responsible for most of the noticeable turning points of the reference motion (see example trajectories in Figure 4). The movement power in this band thus captures systematic overshooting (more power than reference motion) or undershooting (less power than the reference motion) at turning points. By contrast, spectral power for higher frequencies corresponds to corrective movements, which we did not investigate here. Power corresponding to the lower path frequencies tends to be contaminated stronger by motor noise. The path segments were first filtered with a Hanning window. The spectral power was then computed with a fast Fourier transform for frequency bands of width 0.117 Hz (Matlab function fft; log 10 of resulting values was analyzed). Values within the band 0.45–0.6 Hz were averaged. The same was done for the reference motion. The difference of these two values indicates a miscalibrated use of feedback (positive: overshooting; negative: undershooting). Note that a compensatory strategy of ignoring the feedback and merely recalibrating reference extrapolation (open-loop tracking) cannot explain changes in this variable. 

For each combination of training and test conditions (U_training-U_test, P_training-P_test, U_training-P_test, and P_training-U_test) three post-test trials of 10 s duration were available (Figure 3C). These were matched to the last three pretest trials of each test condition for the comparison between pretest and post-test (the first three pretest trials were discarded, as participants were still getting accustomed to the tasks). To eliminate possible confounds due to trial length, only the first 10 s of these pretest trials were used. The first 2 s of all pretest and post-test trials were removed, as the reference started in a random position and participants first had to approach it (invincibility period; cf. Task). The median values from the remaining 3 × 8 s recordings per phase (pre vs. post), training condition (U_training vs. P_training), and test condition (U_test vs. P_test) were used to test statistical significance of differences in these measures using a three-way repeated-measures analysis of variance (ANOVA; Matlab script RMAOV33 by Antonio Trujillo-Ortiz) with factors test phase (pre vs. post), training condition (P_training vs. U_training), and test condition (P_test vs. U_test). 

Results and discussion

Example behaviors for the different phases and conditions are depicted in Figure 4 (animations of these examples can be found in the supplemental material). The additional feedback delay destabilized behavior (overshooting) in both training conditions. Participants learned to control this over time. However, only in the predictable condition, participants also recalibrated anticipatory behavior, which caused them to move too early in the post-test. The following sections describe these results in terms of the measures outlined above. 

Temporal error

Figure 5A depicts the time course of the temporal error for the different training sessions of the congruent P_training-P_test and U_training-U_test conditions. In addition, a comparison to the transfer conditions P_training-U_test and U_training-P_test is shown in Figure 5B. 

Figure 5

 

Temporal error. A: Temporal error in the training tasks over time (blue: unpredictable; green: predictable); population average and SEM (standard error of the mean). To improve readability, the order of top-up adaptation trials and post-test trials (cf. Figure 3C) was rearranged and transfer trials (pretest and post-test measurements of nonadapted condition) are not displayed. B: Differences between pretest and post-test for all training and test conditions; population average and SEM. There is a significant interaction between the factors phase and training condition, F(1, 9) = 5.8, p = 0.039, as well as significant main effects of the factors training condition, F(1, 9) = 7.0, p = 0.027, and test task, F(1, 9) = 13.8, p = 0.005; full ANOVA results in the Appendix, Table 1.

Even prior to adaptation, the cross-correlation lag is lower (closer to zero) in the predictable condition (pretest). This means that there is more anticipation in the stylus movement and thus less temporal error (Figure 5A).¹ When the delay is introduced, the cursor dot initially lags behind much more in both conditions (high value in Figure 5A middle). The different training conditions then have opposing effects on the temporal error. The temporal error is reduced in the predictable condition (green line going down), consistent with a recalibration of anticipatory mechanisms. Interestingly, the opposite occurs in the unpredictable training condition: as adaptation proceeds, the cursor dot lags more and more behind the reference (blue line going up). This result in U_training has been observed previously by Foulkes and Miall (2000). One possible explanation for this effect is that the delay is neutralized by a decrease in control gain (i.e., reacting slower to the error signal), which is a possible strategy to stabilize a control system destabilized by feedback delays (e.g., Foulkes & Miall, 2000). However, a growing lag is not consistent with genuine delay adaptation. 

The directional difference in temporal error between conditions carries over to the post-test (aftereffect, Figure 5B) and even transfers to the nonadapted condition, albeit to a lesser extent. This transfer shows that differences between training conditions cannot just be due to task context. Different adaptive strategies were acquired in the two training conditions even if the underlying perturbation was the same. This result supports our hypothesis that predictable reference motion is necessary for delay adaptation in compensatory mechanisms. 

Spatial error

Figure 6 shows the time course (A) and adaptation aftereffects (B) of the spatial tracking error (same arrangement of conditions as in Figure 5). On average, the error is a bit lower in the predictable condition already in the pretest. When the delay is introduced, the tracking error increases substantially in both conditions. It is then gradually reduced in the P_training condition. By contrast, in the U_training condition, spatial error levels early on at a comparably high level after some improvement in the first trial. Note that the gradually increasing lag of the cursor behind the reference in the U_training condition appears to have no positive or negative effects on the spatial error (cf. Figure 5A). The comparison of pretest and post-test reveals a negative aftereffect, i.e., reduced performance, for all combinations of training and test conditions (Figure 6B). 

Figure 6

 

Spatial error. A: Performance in the training tasks over time (blue: unpredictable; green: predictable); population average and SEM. To improve readability, the order of top-up adaptation trials and post-test trials (cf. Figure 3C) was rearranged and transfer trials (pretest and post-test measurements of nonadapted condition) are not displayed. B: Differences between pretest and post-test for all training and test conditions; population average and SEM. There is a significant main effect of the factor: phase F(1, 9) = 18.8, p = 0.002, and a significant three-way interaction between phase, training condition, and test condition, F(1, 9) = 8.0, p = 0.020; full ANOVA results in the Appendix, Table 2.

This result is consistent with our hypothesis and with the literature, in which performance reduction aftereffects in terms of spatial errors were observed for both unpredictable training tasks (Foulkes & Miall, 2000; Miall & Jackson, 2006) and predictable training tasks (Cunningham et al., 2001a; Cunningham et al., 2001b; Kennedy et al., 2009; Morice et al., 2007). However, these performance aftereffects seem to result from different compensatory strategies for the predictable and unpredictable training conditions. 

Spectral power

Figure 7A shows how the introduction of the delay destabilizes control (high spectral power due to overshooting of r) in both conditions. Furthermore, it shows how participants learn to stabilize these oscillations in both conditions (spectral power decreases to initial levels by the end of training). Thus, participants trained in the predictable condition make stable use of feedback during tracking (Figure 7) at comparable levels of temporal error as in the pretest (Figure 5). Even though the decrease in spectral power observed after unpredictable training is also some form of altered use of feedback, the increase in temporal error (Figure 5) suggests that this is not temporal recalibration. The combination of high temporal error and stabilized control with a feedback delay instead suggests a recalibration (decrease) of control gain. 

Figure 7

 

Spectral power. A: Spectral power in the training tasks over time (blue: unpredictable; green: predictable); population average and SEM. To improve readability, the order of top-up adaptation trials and post-test trials (cf. Figure 3C) was rearranged and transfer trials (pretest and post-test measurements of nonadapted condition) are not displayed. B: Differences in spectral power between pretest and post-test for all training and test conditions; population average and SEM. There are very significant main effects of the factors: phase F(1, 9) = 173.9, p = 0, and test condition F(1, 9) = 54.5, p = 0. There is also a significant two-way interaction between the factors: test phase and test condition, F(1, 9) = 8.0, p = 0.020, and a three-way interaction between phase, training condition, and test condition, F(1, 9) = 11.8, p = 0.007; full ANOVA results in the Appendix, Table 3.

There are again marked adaptation aftereffects that also transfer to the nonadapted condition (Figure 7B). In both conditions, participants stay too close to the midline during the post-test. This is due to turning too early in the predictable training condition (overanticipation, Figure 5 and example Figure 4). In the unpredictable training condition, this is likely due to a low control gain (sluggish reaction to the reference, Figure 5). 

Taken together, the results from Experiment 1 on motor behavior suggest that humans have different strategies to stabilize behavior after the introduction of a feedback delay. If the reference moves unpredictably, participants act in ways consistent with a decrease in control gain (sluggish tracking): Stability is restored (Figure 7) at the expense of a growing temporal error (Figure 5) with lasting high levels of spatial error (Figure 6). On the other hand, participants show all characteristics of delay compensation recalibration if the reference is predictable: decrease of temporal (Figure 5) and spatial (Figure 6) error, as well as stabilization of feedback control (Figure 7). All these adaptive changes lead to negative aftereffects in the post-test and transfer to the nonadapted task, which shows semipermanency and context independence of the adaptation effects. In short, these results confirm our initial hypothesis that coadaptation of reference extrapolation and motor prediction can make delay adaptation possible. 

Note that a transfer of recalibrated reference extrapolation to the unpredictable condition is possible because the continuous changes in reference velocity and position make some reference extrapolation possible. Evidently, this predictability is insufficient for recalibration but sufficient to cause negative aftereffects in temporal error and spectral power. 

Experiment 2: Synchronization

In Experiment 2 we tested whether predictability of the reference during delay adaptation also modulates perceived simultaneity. For this we used a synchronization task. Sugano et al. (2012) used a tapping synchronization task to quantify sensorimotor recalibration of perceived simultaneity: Humans had to adjust rhythmic tapping movements such that these felt simultaneous with a rhythmic visual or auditory signal. The temporal offset between the sensory stimuli and the tapping movement was used as an estimate for the PSS (cf. also Aschersleben & Prinz, 1997; Kennedy et al., 2009). Compared to other methods, such as temporal order judgments or simultaneity judgments, a synchronization task has the advantage that an estimate of the PSS is very fast and can be extracted individually from each trial. Thus, a PSS shift can be measured in relatively few trials and, compared to other psychophysical methods, synchronization serves to measure the time course of relatively short lasting recalibration effects. 

Methods

Participants

Ten healthy adult volunteers participated in the study (age range 18–24, four female, all right handed as by self-report). They received a small monetary compensation for their participation (6 €/hr). All participants were naïve to the purpose of the experiment. None of them had participated in Experiment 1, but three had participated in a pilot version of the experiment. All signed an informed consent form. The experiment was conducted in agreement with the ethics standards laid out in the Declaration of Helsinki and was approved by the ethics committee of the Department of Medicine of the University of Tübingen (Germany). 

Apparatus

The apparatus was the same as in Experiment 1 on motor behavior. 

Task

The predictable and unpredictable training tasks were the same as in Experiment 1. There were some minor changes applied with respect to the dimensions of the stimulus and the movement (in Experiment 1, the task covered a larger field of view and movement than necessary). All dimensions defining the task were decreased to approximately half (reference dot: 1.7°; white cursor dot: 0.8°; maximum lateral reference position: 12.8°; vertical prediction window: between 12.6° and 36.7°) and 25% gray bars were added as terminators on the vertical prediction window to increase the impression of a scrolling path in the P_training condition (these bars were also present in the U_training condition to ensure comparability). 

The test task (pre and post) in Experiment 2 was a synchronization task (Figure 8A). A black dot (same as the reference dot in the tracking task) oscillated at one of three target frequencies: 0.6, 0.8, or 1 Hz (5.2° left and right) in the horizontal dimension (sinusoidal movement). The target frequency was varied randomly to avoid that participants get habituated to a certain pattern of behavior, without paying attention to the visual input. Participants were instructed to synchronize left and right movement of the stylus with the movement of the dot such that the reference motion and kinesthetically sensed hand movement felt synchronous to them (there were no constraints imposed on width or horizontal offset of these movements). They received no visual feedback about the movement of the stylus. 

Figure 8

 

Task and procedure for Experiment 2 on synchronization. A: Sinusoidal movement of the reference without visual feedback was displayed. Participants had to synchronize their occluded stylus motion with the visual signal. B: Procedure Experiment 2.

Procedure

Using the same training procedure as in Experiment 1, participants were trained in the unpredictable (U_training) and the predictable (P_training) training condition on different days. The order of the training conditions was counter-balanced across participants. One session (Figure 8B) consisted of a familiarization block, where participants first had to perform the synchronization task for 15 trials for 15 s each (i.e., five for each of the three target frequencies) followed by the tracking task without delay either in the predictable or the unpredictable conditions (the same as their training condition; 10 × 60 s). After the familiarization to synchronization and tracking tasks, participants performed again 15 trials (15 s each) of the synchronization task. This was taken as the pretest measure. Following a 5 min break, 17 × 60 s of tracking with the additional 200 ms visual feedback delay was performed (counterbalancing the U_training and P_training starting session across participants). Afterwards, participants' perception of simultaneity was tested with another 15 synchronization trials for 15 s each (post-test). The order of the target frequencies was identical for the two test phases within a session. It was randomized across sessions and participants.² 

Analysis

The preprocessing of trajectories was the same as in Experiment 1. The only dependent variable of interest for the synchronization trials was the cross-correlation lag that was used as a measure of the PSS in this experiment. Trials in which the maximum cross-correlation coefficient was smaller than 0.5 were discarded and treated as missing values (three trials in total). 

We used a three-way repeated measures ANOVA (Matlab script RMAOV33 by Antonio Trujillo-Ortiz) to compare the PSS between conditions. The factors were: test phase (pre vs. post), training condition (U_training vs. P_training), and trial number (2–14; Trials 1 and 15 were omitted from analysis because they contained missing values). 

Results and discussion

To check for comparability of the results between Experiments 1 and 2, we also analyzed the tracking performance in Experiment 2. The effect of delay adaptation on performance in the tracking task (temporal error, spatial error, spectral power) was equivalent to Experiment 1 on motor behavior (results in Appendix). 

Figure 9 shows how during the first five trials after delay adaptation in the predictable condition participants need to move their hand 31 ± 8 ms (mean and SEM) earlier than in the pretest to perceive the hand motion as simultaneous. This confirms the hypothesis of a PSS shift only after predictable training. This difference between conditions decreases with trial number. 

Figure 9

 

PSS aftereffects in the synchronization in the unpredictable (blue) and the predictable (green) training conditions across test trials, population mean of difference between pretest and post-test (error bars: SEM across participants). The aftereffect is stronger in the predictable than in the unpredictable condition at the beginning of the post-test: There is a significant three-way interaction between trial number, test phase, and training condition: F(12, 108) = 1.9, p = 0.048. There are also significant main effects of test phase, as post-tests show more anticipation; F(1, 9) = 5.7, p = 0.014, of training condition, as P anticipates more than U; F(1, 9) = 9.2, p = 0.041, and of trial number, as there is a trend to anticipate more each trial; F(12, 108) = 2.1, p = 0.025. This trend is also reflected in a significant anticorrelation of PSS with trial number within a test phase: Pearson's r = −0.41, p ≪ 0.001, pooled across test phases, training conditions, and participants. Furthermore, there is a significant interaction between test phase and trial number, F(12, 108) = 1.9, p = 0.048; indicating that the downward slope of PSS within a test phase is stronger in the pretests than in the post-tests. Full ANOVA results in the Appendix, Table 4.

At first glance it appears that even without visual feedback, the recalibration effect decays over the course of the post-test (Figure 9, green line converges to zero). However, the results are not conclusive in this respect. There are order effects that impact on the slope of PSS across a test phase, independent of training condition (interaction test phase and trial number, see statistical results in caption for Figure 9). This makes it difficult to draw firm conclusions on the time course of readaptation. The main result that a shift in PSS between phases and training conditions depends on the predictability of the reference signal, however, is not compromised by these order effects. 

Note that this kind of perceptuomotor recalibration involves not just a readjustment of visual, proprioceptive, and motor (efference copy) signals, but also mere motor learning as a possible part of the recalibration process. It is therefore an open question to what extent this recalibration of perceived relative timing should be called perceptual learning (cf. Discussion). 

Another possible concern could arise from the possibility that participants may not have realized that their task was not tracking anymore. This is unlikely because the PSS shifts occur only for the predictable training condition and because participants often used very different movement amplitudes during synchronization (Figure 8A, left). This is confirmative evidence that participants understood the task switch, which renders this possibility unlikely. 

This result shows that delay adaptation in a predictable motor control task can transfer to a task measuring subjective relative timing. However, it does not show whether this adaptation also affects properties of time perception other than the PSS, such as perceptual precision or the perception of longer intervals. 

Experiment 3: Interval estimation

In Experiment 2, we observed that delay adaptation in a predictable motor control task caused a shift in the PSS. However, it is unclear how the recalibration would generalize and affect visuomotor time perception over a larger range of temporal discrepancies. Own previous work has shown that, in a discrete training task (button press and flash), visuomotor temporal recalibration of perceived simultaneity is temporally asymmetrical: The window of perceived simultaneity is widened one-directionally on the side of movement-lead temporal discrepancies only (Rohde et al., 2014). Such asymmetries in generalized temporal recalibration cannot be measured using a synchronization or a binary temporal order judgment task (Roach, Heron, Whitaker, & McGraw, 2011; Rohde et al., 2014). 

In Experiment 3, we therefore investigated the generalization of temporal recalibration using an interval estimation task, where participants had to judge the relative timing between a motor event (turning point in a left-right movement) and a visual flash event. Our focus was on shifts of the decision boundaries between simultaneous responses and turn first/flash first decisions, respectively. As this task involves no reference signal during movement, it also generates strong evidence of a transfer to perception. 

Methods

Participants

Fifteen healthy adult volunteers participated in the study (age range 21–25, 10 female, all right handed except one as by self-report). Five participants had either participated in a pilot experiment for this study or in Experiments 1 or 2. Participants received a small monetary compensation for their participation (6 €/hr). All participants were naïve to the purpose of the experiment. All signed an informed consent form. The experiment was conducted in agreement with the ethics standards laid out in the Declaration of Helsinki and was approved by the ethics committee of the Department of Medicine of the University of Tübingen, Germany. 

Apparatus

The apparatus was the same as in Experiment 1 on motor behavior. 

Task

The training task (predictable tracking) was the same as in the predictable condition of Experiment 2. The test task (pre and post) was an interval estimation task (Figure 10A; cf. also Rohde et al., 2014). Participants first observed movement of a black dot (same as the reference dot in the tracking task) that they later had to reproduce. The dot performed one cycle of a sinusoidal movement (starting at phase −π/2) in the horizontal dimension at one of the reference frequencies of Experiment 2 picked randomly: 0.6, 0.8, or 1 Hz (5.2° visual angle from left to right and back). Participants were instructed to reproduce the movement after the reference dot had vanished. They received no visual feedback about the movement of the stylus. 

Figure 10

 

Task and procedure for Experiment 3 on interval estimation. A: The interval estimation consists of four phases: reference presentation, reproduction with flash around turning point, estimation of the interval between turning point, and flash and feedback about reproduction performance. B: This 3-hr experiment was split into five blocks: familiarization (to learn the interval estimation task with feedback) and two blocks each for pretest (tracking with no delay) and post-test (delay adapted) interval estimation.

The participants' turning point was predicted in real time in order to time the presentation of visual stimuli probabilistically around the turning point (cf. Rohde & Ernst, 2013; Rohde et al., 2014; Stetson et al., 2006). The movement was kept similar to the movements performed during tracking, as it is not obvious whether recalibration also generalizes to very different movements such as button presses. The visual flash was a white disk of the same size as the cursor during tracking. It was flashed for one frame at the center of the screen. The temporal discrepancies between flash and estimated turning point were drawn uniformly from the interval [−150, 150] ms. The flash presentation was timed relative to the estimated turning point (half cycle). Prediction noise implies that the actual presentation of discrepancies was drawn from the range [−275, 275] ms. A running average of movement reproduction performance (deviation from expected turning point) was recorded and subtracted from future stimulus presentations (to correct for individual biases and ensure that flash presentation was balanced around the participants' turning point). 

After the reproduction of the movement was finished, participants had to judge whether the flash occurred before or after their turning point in the reproduced trajectory. They could use the stylus to move a point on a scale corresponding to temporal discrepancies between −300 and 300 ms in steps of 50 ms. Negative values indicate that the flash occurred before the turning point. These responses can be either used as magnitude estimation (metric information) or as ternary temporal order responses (vision first, simultaneous, movement first; Rohde et al., 2014; Ulrich, 1987), if the metric information is discarded and only the sign of the response is used (emphasis here is on this latter interpretation). 

The graphical representation of the scale ranged ± 15.7° left and right from the midline. During the initial familiarization block, participants received feedback to indicate task performance. There was a green dot that indicated the correct discrepancy after the response was submitted. Providing feedback helped the learning and the calibration of the scale (cf. Procedure). 

Procedure

The experiment was conducted in one 3-hr session consisting of five experimental blocks (Figure 10B). The first familiarization block started with a 30 s trial of the predictable tracking task without delay. Afterwards, the participants were introduced in stages to the more difficult interval estimation task (Figure 10A). They first practiced observing and reproducing the one-cycle movement of the reference. Then they practiced verbally judging the temporal order of turning point and flash. Finally, they performed the interval estimation adjusting the dot on the scale. Participants were explicitly instructed to execute the motor movement without interference from the flash. That is, they were told they should not try and wait for or catch up with the flash. For the rest of the first block (100 trials), participants performed the interval estimation task with feedback in order to learn the temporal interpretation of the response scale.³ 

Blocks 2 and 3 served as pretest blocks, Blocks 4 and 5 as post-test blocks (Figure 10B). At the beginning of each block participants performed manual tracking (Blocks 2 + 3: 5 × 60 s without delay; Block 4: 15 × 60 s with delay; Block 5: 5 × 60 s with delay). Afterwards, they switched 25 times between interval estimation trials and 30-s top-up tracking trials (Figure 10B). Thus, at the end of the experiment there were 50 interval estimation responses per participant and phase (pre and post). 

Analysis

Two participants had to be excluded from the analysis because they consistently moved before the reference motion had terminated and thus caused an imbalanced presentation of visual stimuli (wrong prediction of turning point). For the remaining 13 participants interval-estimation trials where discarded if either the turning point had been wrongly identified by the prediction algorithm (e.g., due to small unconscious movement of the participant), or if the flash and the identified turning point were more than 275 ms apart. In total 69 trials had to be discarded (5.3%). Between 43 and 50 responses per participant and condition remained. 

The metric information contained in the responses is only used for visualization (Figure 11B), as the emphasis of this experiment was to test for asymmetrical recalibration of temporal order decisions. For the statistical analysis, the responses were reinterpreted as ternary (i.e., three alternative: flash first, simultaneous, turn first) temporal order judgments (Rohde et al., 2014; Ulrich, 1987). That is, only the direction of the perceived interval response was used. As proposed by Ulrich (1987), the results were analyzed twice (once counting simultaneous responses as flash first and once counting them as turn first responses) to determine the decision boundaries flanking the window of perceived simultaneity (left decision boundary: flash first vs. turn first/simultaneous; right decision boundary: flash first/simultaneous vs. turn first). Subtracting the cumulative probability distributions for the two binary decision functions, the window of simultaneity can be modeled (cf. also Cravo, Claessens, & Baldo, 2011; Rohde et al., 2014; Yarrow, Jahn, Durant, & Arnold, 2011). This subtraction results in a window function (temporal window of simultaneity) that resembles a Gaussian bell shape for some parameters (e.g., Figure 11A). This procedure and analysis is required to detect a possible asymmetrical widening of the window of perceived simultaneity (Rohde et al., 2014). The PSS can also be estimated with such a model as the average of the two thresholds delimiting the window of simultaneity (Yarrow et al., 2011). 

Figure 11

 

Results for Experiment 3 on perceived relative timing of visual and movement events. A: The window of perceived simultaneity before (dark green) and after (pale green) predictable delay adaptation. Note that these are not Gaussian functions, but instead the difference between two cumulative Gaussian functions corresponding to the two binary decision boundaries: vision first versus simultaneous/turn first (left) and vision first/simultaneous versus turn first (right) (cf. two-criterion model, cf. Cravo et al., 2011; Rohde et al., 2014; Yarrow et al., 2011). Due to small precision differences (left JND: 151 ms; right JND: 159 ms), the PSS (i.e., midpoint between left and right criterion that delimit the shaded areas) is slightly offset from the peak of the probability function for simultaneous replies. B: Median and interquartile range (IQR) of participants' interval estimates (binned in 50 ms bins and pooled across participants). These are presented for illustration only.

To estimate the two threshold functions flanking the window of perceived simultaneity, generalized linear mixed models (GLMMs) with a probit link function (Moscatelli, Mezzetti, & Lacquaniti, 2012) were fitted to the responses to test for differences in PSS and just noticeable differences (JND; half the distance between the estimated 25% and 75% cuts for responses turn first) between conditions (pre, post). A GLMM allows the estimation of PSS and JND on the population level and individual level simultaneously. Thus, it is especially suited for data sets where individual data are sparse as it is the case here. This analysis was performed using R and the MERpsychophysics toolbox developed by Moscatelli et al. (2012). Given the twofold analysis (i.e., the two obtained decision boundaries marking “earlier than simultaneous” and “later than simultaneous”), the resulting p values were corrected with the Bonferroni-Holmes method. 

Results and discussion

The effect of delay adaptation on performance in the tracking task (temporal error, spatial error, spectral power) was equivalent to Experiment 1 on motor behavior (results in Appendix). 

Figure 11A depicts the windows of perceived simultaneity (difference between flanking cumulative Gaussian functions) before adaptation (dark green) and after adaptation (pale green). In agreement with the results reported in Rohde et al. (2014), significant differences in perceived relative timing only occurred on the movement-lead side of the range of discrepancies (differences between curves only on the right side in Figure 11A). The cumulative Gaussian function flanking the window on the movement-lead side shifts significantly by 53 ms (p = 0.008; cf. Appendix, Tables 5, 6, and 7, for a complete report of the statistical results. By contrast, the cumulative Gaussian function flanking the window of perceived simultaneity on the left side (where vision leads) shifts nonsignificantly (p = 0.386) by just 23 ms in the direction of the feedback delay (to the right). The midpoint of the window of simultaneity (PSS) shifts by 38 ms, which is a similar value as found in Experiment 2. The value is also similar to the temporal error aftereffect found in Experiment 1 (Figure 5B). 

The raw interval estimation responses pooled across participants before (dark green) and after (light green) delay adaptation are depicted in Figure 11B. The one-sided widening of the window of simultaneity can be seen as the pale green plateau at the horizontal zero line in Figure 11B. These results on the estimated magnitude are provided for completeness only; no statistical tests were performed on the magnitude information. 

These results show that delay adaptation in motor control leads to the same one-sided modulation of the window of perceived simultaneity that characterizes visuomotor recalibration in psychophysical tasks (Rohde et al., 2014). It also shows that the PSS is recalibrated, supporting the findings of Experiment 2. 

Experiment 3 was performed using only the predictable condition. Given the clear results from Experiments 1 and 2, it seems very unlikely that unpredictable delay adaptation might have caused perceptual recalibration in this task. In principle, order (due to fatigue or continued task learning) could be a confounding factor. However, tracking performance tends to converge after approximately 10 minutes whereas this experiment lasted for 3 hrs. Additionally, sufficient breaks were granted, such that there is no evidence for fatigue in tracking performance (Appendix Figure 13). Furthermore, it is not clear by which mechanism order could produce such a specific result; recalibration only in one half of the range of stimuli, which is a pattern specific to visuomotor recalibration of perceived simultaneity (Rohde et al., 2014), and a PSS shift that matches the effect size in the other two experiments. In addition, there were no changes in perceptual precision indicating that perceptual discrimination performance did not deteriorate due to fatigue or improve due to task-learning during the experiment (statistical results on JND in the Appendix, Table 5). 

General discussion

We investigated the role of reference signal predictability on visual feedback delay adaptation in three experiments. In Experiment 1 on motor behavior, we found that participants revealed strong signs of delay adaptation (in terms of temporal error, spatial error, and spectral power of tracking) only if the task environment was predictable. This recalibration persisted after the delay was removed, i.e., we observed negative aftereffects. Furthermore, recalibration transferred from the predictable training to the unpredictable test condition. We interpret the fact that adaptation and aftereffects were observed for all three measures as evidence that delay parameters in both motor prediction mechanisms and reference extrapolation mechanisms were recalibrated. When participants were trained in the unpredictable condition, they also learned to stabilize their visually guided tracking behavior (spectral power, Figure 7). However, they did so at the cost of increased temporal error lagging, behind more and more as adaptation proceeded. This behavior is more compatible with explanations in terms of a decreased feedback control gain. 

In Experiment 2 we investigated whether the recalibration effects also transferred to perceptual experience, using a synchronization task. Participants had to adjust their hand movement such that it felt synchronous with a visual stimulus. Perceptuomotor recalibration in this task is only observed after predictable training. This effect was also replicated in Experiment 3, where we investigated whether delay adaptation in predictable tracking transfers to a visuomotor interval estimation task. We observe a one-directional widening of the window of perceived simultaneity for movement-lead stimuli only (cf. Rohde et al., 2014). The size of the PSS shifts in Experiments 2 and 3 (31 and 38 ms) and the temporal aftereffect in Experiment 1 (anticipatory temporal error: 30 ms) are all of a similar magnitude. This suggests that the same processes may underlie the adaptive shifts in behavior and perception. 

Taken together, these results demonstrate that delay adaptation in motor control can transfer to tasks measuring perceived temporal alignment of vision and kinesthetically sensed movement. Secondly, they show that such adaptation is not merely driven by the available information on visuomotor temporal discrepancies, but requires a predictable reference signal. Thus, they offer an explanation for some of the divergent results present in the current literature. In the following these points are discussed in more detail. 

The results from Experiments 2 and 3 on perceptuomotor recalibration link delay adaptation in motor control with existing psychophysics research in recalibration of perceived visuomotor simultaneity. Temporal recalibration of PSS after exposure to visuomotor delays between discrete events (e.g., button press and visual flash) has been reported many times (Heron et al., 2009; Keetels & Vroomen, 2012; Rohde & Ernst, 2013; Rohde et al., 2014; Stetson et al., 2006; Sugano et al., 2010; Sugano et al., 2012). This kind of recalibration appears to occur more or less inevitably. In motor control, by contrast, semipermanent adaptation to feedback delays is a much more elusive phenomenon, to the point that the possibility of such adaptation or a transfer to perceived relative timing has been questioned (e.g., Smith & Smith, 1962). Our results show that this kind of transfer occurs at least in some situations. This result also backs Cunningham et al.'s (2001a) anecdotal report of an experienced loss of the sense of agency after delay adaptation, caused by the experienced temporal inversion of cause (hand movement) and effect (cursor movement) during the post-test. During debriefing, several of our participants reported a perturbed sense of agency for the post-test of Experiment 1 (predictable training) after the delay they had adapted to was removed. However, others simply felt that the cursor was moving very fast or that “something weird” caused their poor tracking behavior during the post-test. 

It should be noted that the measured changes in perceived relative timing between visual and movement events might be driven by perceptual learning (visuoproprioceptive, visuotactile), by motor learning (changes in efferent processing), or by both. The experiments presented here do not aim to distinguish the relative contribution of the two. One might thus argue that this form of temporal recalibration is not perceptual learning in a strict sense of altered sensory processing (this is the case for most studies on visuomotor temporal recalibration). Therefore, we refer to this temporal recalibration here as perceptuomotor learning to avoid misunderstandings. Further research, e.g., investigating transfer to a passive visuoproprioceptive or visuotactile time perception task, can help to assess the relative contribution of these two components. 

The results also show that delay parameters in visuomotor control are not recalibrated merely based on the available temporal information about movement and visual feedback. Usually, state estimation models and forward models of visuomotor adaptation, such as Kalman filter models for spatial recalibration (e.g., Burge et al., 2008), assume that updating of internal model parameters relies on an automatic statistical analysis of mismatches between estimated state x′ and actual state (revealed later by visual feedback x_vis). A recent neural model similarly proposes task-independent processing of visuomotor delays for perceptual recalibration (Cai, Stetson, & Eagleman, 2012). Such a mechanism would not be able to explain the results we report here. Participants are exposed to the same delay between hand and cursor movement in the predictable and unpredictable training conditions. They also continually use this delayed feedback information for tracking in both conditions (as evident from aftereffects in spectral power, Figure 7B, which occur in both conditions and can only be caused by the altered use of feedback). Still, only in the predictable training condition, participants seem to recalibrate their motor prediction as well as their reference extrapolation and time perception in the predicted fashion. 

There are several possible mechanisms that could explain the difference between the predictable and the unpredictable training condition. The simplest possible explanation would be that delay adaptation in perception is directly linked to an adaptation of reference extrapolation. Such reference extrapolation is only possible in the predictable condition, not in the unpredictable condition. However, this seems unlikely. Anticipatory motor planning is a very common requirement in motor behavior, for example when dealing with inertial lags (driving a heavy car, canoeing, or flying a kite). Such activities do not usually involve changes in time perception. Furthermore, participants trained in the predictable condition show a range of delay adaptation effects that do not occur after unpredictable training. This suggests that they adapt their motor prediction as well as their reference extrapolation. We therefore favor another possibility: Reference predictability may be necessary as a catalyst for delay adaptation of motor prediction. As illustrated in Figure 1B, reference extrapolation r′ can improve the error estimate e′. This might help the adaptation mechanisms to identify the nature of the perturbation. Thus, we propose that semipermanent adaptation to feedback delays with transfer to time perception may require the temporal coadaptation of a reference extrapolation mechanisms (r′) and motor prediction (x′). 

A third noteworthy result from this study concerns the relative success of sluggish tracking as a compensatory strategy for the delay in unpredictable tracking. In terms of spectral power of movement, this strategy is equally successful in stabilizing control as genuine delay adaptation (Figure 7). This could explain at least some of the divergent results in the literature. In a situation where some spatial or temporal errors are admissible (e.g., Ferrell, 1964; Smith & Smith, 1962), or where reference extrapolation is impossible (e.g., Foulkes & Miall, 2000; Miall & Jackson, 2006), decrease in control gain may be a simpler and a more suitable strategy (as also argued by Cunningham et al., 2001a). In other tasks, mere adaptation of reference extrapolation, i.e., ignoring the delayed feedback may also be a viable strategy. We propose that delay adaptation with aftereffects in perceived relative timing only occurs in scenarios where the coadaptation of the use of feedback and reference extrapolation is both necessary and possible. This is the case in the predictable tracking task tested here, as well as for example in the tasks employed by Cunningham et al. (2001a) and Cunningham et al. (2001b). Delay adaptation would then be a rather rare and elusive adaptation effect, which would explain the lack of consistent previous evidence for this phenomenon. 

In conclusion, our results show that humans can adapt to feedback delays in stronger forms than just the instrumental extrapolation of reference motion or by a decrease of the control gain. Specifically, participants adapt motor prediction as well as the perceived temporal alignment of vision and kinesthetically sensed movement. This recalibration, however, only occurs after training in a predictable training task. 

Supplementary Materials

mov S1 

mov S2 

mov S3 

mov S4 

mov S5 

mov S6 

mov S7 

mov S8 

mov S9 

mov S10 

Acknowledgments

M. R. was funded by the DFG Grant RO 4147/1-1 ELAPS: Embodied Latency Adaptation and the Perception of Simultaneity and the EU FP7/2007-2013 project n8 248587 THE-the Hand Embodied. M.O.E was supported by EU FP7/2007-2013 projects n° 601165 WEARHAP and n° 248587 THE-The Hand Embodied. We would like to thank Massimiliano Di Luca for helpful initial discussion. We acknowledge support for the Article Processing Charge by the Deutsche Forschungsgemeinschaft (DFG) and the Open Access Publication Funds of Bielefeld University Library. 

Commercial relationships: none. 

Corresponding author: Marieke Rohde. 

Email: [e-mail]. 

Address: Department of Cognitive Neuroscience, University of Bielefeld, Bielefeld, Germany. 

Footnotes

¹  It is important to remember that participants were exposed to both tasks in the pretest (Figure 3C), i.e., these differences cannot be due to differences in training.

Footnotes

²  A mistake in the seeding of the random number generator led to the repeated generation of the same sequence of target frequencies for some experimental sessions. However, given that the main results are differential (pre vs. post, comparing the same sequence), this is not expected to bias the main result.

Footnotes

³  Due to a programming mistake, a constant was added to the cursor position during the first block only. This means that participants could sometimes not reach all locations on the response scale. However, this only occurred in the first block and did not impact the important comparison of pre- and post-test.

Appendix

Table 1

 

ANOVA results: Temporal error, Experiment 1.

Three-way repeated-measures ANOVA with factors:
Test phase	2 levels: pre, post
Training condition	2 levels: predictable, unpredictable
Test condition	2 levels: predictable, unpredictable
Test phase	F(1, 9) = 0.0	p = 0.864
Training condition	F(1, 9) = 7.0	p = 0.027
Test condition	F(1, 9) = 13.8	p = 0.005
Test Phase × Training Condition	F(1, 9) = 5.8	p = 0.039
Test Phase × Test Condition	F(1, 9) = 2.2	p = 0.169
Training Condition × Test Condition	F(1, 9) = 1.1	p = 0.329
Test Phase × Training Condition × Test Condition	F(1, 9) = 0.0	p = 0.924

Table 2

 

ANOVA results: Spatial error, Experiment 1.

Three-way repeated-measures ANOVA with factors:
Test phase	2 levels: pre, post
Training condition	2 levels: predictable, unpredictable
Test condition	2 levels: predictable, unpredictable
Test phase	F(1, 9) = 18.8	p = 0.002
Training condition	F(1, 9) = 2.2	p = 0.166
Test condition	F(1, 9) = 3.5	p = 0.095
Test Phase × Training Condition	F(1, 9) = 3.8	p = 0.084
Test Phase × Test Condition	F(1, 9) = 2.0	p = 0.187
Training Condition × Test Condition	F(1, 9) = 4.2	p = 0.072
Test Phase × Training Condition × Test Condition	F(1, 9) = 8.0	p = 0.020

Table 3

 

ANOVA results: Spectral power, Experiment 1.

Three-way repeated-measures ANOVA with factors:
Test phase	2 levels: pre, post
Training condition	2 levels: predictable, unpredictable
Test condition	2 levels: predictable, unpredictable
Test phase	F(1, 9) = 173.9	p = 0.000
Training condition	F(1, 9) = 0.6	p = 0.459
Test condition	F(1, 9) = 54.5	p = 0.000
Test Phase × Training Condition	F(1, 9) = 0.02	p = 0.905
Test Phase × Test Condition	F(1, 9) = 8.0	p = 0.020
Training Condition × Test Condition	F(1, 9) = 2.2	p = 0.175
Test Phase × Training Condition × Test Condition	F(1, 9) = 11.8	p = 0.007

Table 4

 

ANOVA results: Temporal error, Experiment 2.

Three-way repeated-measures ANOVA with factors:
Test phase	2 levels: pre, post
Training condition	2 levels: predictable, unpredictable
Trial number	13 levels: Trials 2–13 within each test block (Trials 1 and 15 were deleted for analysis due to one missing value each)
Test phase	F(1, 9) = 5.7	p = 0.014
Training condition	F(1, 9) = 9.2	p = 0.041
Trial number	F(12, 108) = 2.1	p = 0.025
Test Phase × Training Condition	F(1, 9) = 2.3	p = 0.163
Test Phase × Trial Number	F(12, 108) = 1.9	p = 0.048
Training Condition × Trial Number	F(12, 108) = 1.6	p = 0.117
Test Phase × Training Condition × Trial Number	F(12, 108) = 1.9	p = 0.048

Figure 12

 

Tracking behavior Experiment 2. (A) Temporal error (mean and SEM) (B) Spatial error (mean and SEM). The different scales compared to Experiment 1 is due to the rescaling of the tracking task (cf. Methods Experiment 2; this leads to smaller spatial errors) (C) Spectral power (mean and SEM). Longer trials allow a finer resolution in spectral power, so analysis in a slightly different frequency band is displayed.

Figure 13

 

Tracking behavior Experiment 3. (A) Temporal error (mean and SEM) (B) Spatial error (median and IQR, used because of several outlier trials). The different scales compared to Experiment 1 is due to the rescaling of the tracking task (cf. Methods Experiment 2; this leads to smaller spatial errors) (C) Spectral power (mean and SEM). Longer trials allow a finer resolution in spectral power, so analysis in a slightly different frequency band is displayed.

Table 5

 

GLMM results Experiment 3, model evaluation. Notes: The Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) both favor models with just one JND parameter per subject (i.e., there are no differences in precision between pretest and post-test).

	Model evaluation
	Left window limit			Right window limit
	Df	AIC	BIC	Df	AIC	BIC
1 JND	6	1375.8	1421.2	6	1308.6	1354.0
2 JNDs	7	1377.5	1430.5	7	1310.5	1363.5

Table 6

 

GLMM results Experiment 3, parameter estimates. Notes: CI = confidence interval.

	Parameter estimates (population)
	Left window limit		Right window limit
	Estimate	CI	Estimate	CI
JND (ms)	151	[120, 183]	159	[122, 196]
PSSpre (ms)	−20	[−67, 28]	61	[6, 116]
PSSpost (ms)	3	[−43, 48]	114	[62, 167]

Table 7

 

GLMM results Experiment 3, fixed effect statistics. Notes: After Bonferroni-Holmes correction for repeated comparisons, the p-values for test phase are: p = 0.386 (left window limit) and p = 0.008 (right window limit).

	Fixed effects
	Estimate	Std. error	z value	Pr(>|z|)
Left window limit
(Intercept)	0.08785	0.10438	0.842	0.400
SOA	4.45455	0.47474	9.383	<2e-16 ***
Test phase	−0.10022	0.07699	−1.302	0.193
Right window limit
(Intercept)	−0.25746	0.13190	−1.952	0.05095
SOA	4.24438	0.50768	8.360	<2e-16 ***
Test phase	−0.22852	0.07903	−2.892	0.00383 **