Learning to imitate novel motion sequences

Abstract

Many imitative behaviors entail complex sequences of component actions that must be recalled and performed in the proper order. It is well known that imitation of complex actions tends to improve with repeated opportunities to observe and execute the target behavior. But what actually makes this practice-based improvement possible? To address this question, we had subjects view and then reproduce sequences of connected, randomly directed motions of a disc. Even a single repetition of a motion sequence substantially reduced errors in reproduction. Improvement seemed to follow a power law, with accuracy in reproducing each motion segment improving by an amount proportional to the current error for that segment. Analysis of the pauses separating a reproduction's segments suggests that with learning, multiple segments in memory are grouped into more compact representations. To test overt performance's contribution to repetition-based improvement, we compared subjects' performance when they reproduced the stimulus trajectory after each repetition to when they did so only once, after the final repetition. Performance was similar following the final repetition in both conditions, indicating that seeing the model, without actual imitation, was sufficient for learning—even in the absence of an explicit error signal. In another experiment, subjects viewed three presentations of each model, with the second presentation given in forward (start to end) or backward (end to start) order. Performance was significantly better when all three presentations were in the same, consistent order, suggesting that repetition reinforced some temporal aspects of a trajectory as it was being learned, and not merely a better representation of the static shape traced by the motion of the disc. These results provide a first look into explicit learning of sequential, nonverbal material, which is central to many tasks of daily life.

Introduction

Humans acquire many important skills by observing and then imitating the actions of others. Imitative behaviors, which often involve complex sequences of component actions that must be recalled and performed in order, tend to improve with repeated opportunities to observe and practice the target behavior. To understand the factors that support repetition-based improvement in imitation, we used simple, random motion sequences as stimuli and corresponding hand and arm movements as responses (Agam, Bullock, & Sekuler, 2005). This task affords several advantages, including one that is crucial to a quantitative analysis of learning: Quality of imitation can be measured and expressed as a continuous variable, rather than just a binary (pass–fail) one. 

Serial recall—the storage and recall of information in specified order—has attracted much interest among a wide range of researchers. In a typical serial-recall study, subjects are presented with a sequence of items, such as words, syllables, or spatial locations. Then, following some retention interval, they attempt to repeat that sequence in its original order. In most studies, each list is presented and recalled only once, before a new, novel list is presented on the next trial. Of course, daily, a person is afforded multiple opportunities to encode most sequences of items, such as digits in a telephone number, and these repetitions of the stimulus sequence promote more robust recall. The effect of repetition on serial learning has recently received some attention, after decades of neglect (Addis & Kahana, 2004; Johnson, 1991; Klein, Addis, & Kahana, 2005). Still, to date, all studies of repetition-based changes in serial recall have used verbal stimuli, for example, words or letters, which engage just one type of serial memory. 

We adapted tools that have been useful in studying serial learning of verbal material and applied those tools to the domain of imitative behaviors. Previously, in two studies of imitation (Agam et al., 2005; Sekuler, Siddiqui, Goyal, & Rajan, 2003), the stimuli were sequences of directions defined by a disc that moved through a quasi-random series of directions. In those studies, the responses were imitative movements of hand and arm that subjects made while attempting to reproduce what they had seen. After a single presentation of one of these quasi-random motion sequences, the magnitude of error in imitating successive directed motions produced a serial-position curve in which imitation began at a relatively high degree of accuracy, then decreased through the remaining items, until finally showing some improvement in accuracy on the last item. This characteristic form of serial-position curve has been observed numerous times, most frequently with serial recall of verbal material (e.g., Crowder, 1970; Drewnowski & Murdock, 1980). Agam et al. (2005) interpreted the characteristic shape of their serial-position curves, particularly the one-item recency, as reflecting the operation of a competitive queuing process (Page & Norris, 1998; Rhodes, Bullock, Verwey, Averbeck, & Page, 2004). This interpretation is related to Lashley's well-known theory of sequencing by means of parallel response activation (Lashley, 1951) and was supported by several ancillary results. Most notable among these results was the prevalence of near-neighbor transposition errors in subjects' imitations. 

Regardless of the mechanism responsible for the serial-position curve for imitation of novel motion sequences, it is valuable to understand how that curve might change as subjects' performance improves with repeated practice. We therefore carried out five experiments using quasi-random motion sequences as stimuli, gauging changes in performance over multiple presentations of the sequences. The experiments reported here were meant to address a number of questions. Using sequences with two different numbers of directed motions, we asked in Experiments 1 and 2 whether learning curves produced by imitation of motion sequences resembled learning curves for sequences of verbal items. As verbal and visual working memory are thought to comprise distinct memory systems (e.g., Baddeley, 2003), it is theoretically important to know whether similar principles govern learning in the two domains. In addition, we probed the timing of subjects' imitations, including patterns of interresponse times. These provided valuable information on what subjects were actually learning with multiple opportunities to observe and imitate. 

The mechanisms that underlie rehearsal of nonverbal material are quite poorly understood (Prinz, 2006). Some researchers have suggested that the motor system supports the encoding of spatial information in working memory by sustaining a motor plan (based on the previously seen visual material) throughout the retention interval (Awh & Jonides, 2001; Chieffi, Allport, & Woodin, 1999; Postle, 2006; Postle & D'Esposito, 2003; Schneider, 1999). Possibly, this sustained motor plan could be achieved using covert motor activity, a category that includes motor imagery and mental simulation (Jeannerod, 2001). Covert motor activity engages neural circuits that overlap those that support overt motor activity (Gerardin et al., 2000; Porro et al., 1996) and can facilitate subsequent, overt motor activity (Gemignani et al., 2004; Lacourse, Orr, Cramer, & Cohen, 2005). Moreover, results from simple motor-timing tasks (Badets, Blandin, & Shea, 2006; Meegan, Aslin, & Jacobs, 2000) suggest that learning can take place without explicit performance but based on observation alone. This led us to ask, in Experiments 3 and 4, whether overt motor imitation, that is, the actual performance of the imitative behavior, was necessary for learning sequences of imitative behaviors. 

In Experiment 5, we sought to clarify what was being learned as subjects performed the imitation task multiple times. The motion sequences used as stimuli here and in Agam et al. (2005) comprised directed components that were linked head to toe. Although the moving disc that defined each motion component left no trail and was visible only at each of its successive locations, the overall trajectory did define a random shape that subjects could have knitted together in their minds' eyes. We wanted to determine whether with repeated presentation of the same movement sequence, subjects constructed and refined a template of the disc's trajectory and then used that remembered, static shape as the basis for their imitation or whether the learned representation of the movement sequence was dynamic, incorporating the directional and temporal relationships between its subcomponents. 

Experiments 1 and 2

As mentioned earlier, our first two experiments were designed to evaluate the feasibility of studying learning in imitation of sequences of nonverbal items. Previously, we (Agam et al., 2005) found that errors in imitation following a single presentation of a motion sequence produced serial-position curves qualitatively similar to ones seen in serial-recall experiments using verbal material (e.g., Crowder, 1970; Drewnowski & Murdock, 1980). We were interested in probing further parallels between verbal serial recall and imitation of motion sequences by examining performance following multiple presentations of the same stimulus model. As shown in Figure 5 of Agam et al. (2005), there is a considerable increase in difficulty of imitation as models increase in length from five to six segments. To produce a more comprehensive analysis of imitation learning, we used both of these model lengths in our experiments. 

Methods

A total of 24 subjects took part in these two experiments. Twelve subjects (ages 18–28, 7 female) participated in Experiment 1, and another 12 (ages 20–29, 8 female) participated in Experiment 2. All subjects (in all the experiments reported here) gave their informed consent, and all procedures were approved by the Institutional Review Board of Brandeis University. 

Each quasi-random motion stimulus was generated by the steady movement of a white disc (1° visual angle in diameter) against a black background on a computer screen, which subjects viewed from a distance of 57 cm. The disc moved along a series of five ( Experiment 1) or six ( Experiment 2) connected, straight segments, each with a length of 1.5° of visual angle. Each segment took 525 ms to complete, followed by a 225-ms pause between segments. The motion segments' directions were randomized under two constraints. First, to minimize verbal encoding resulting from the shape implied by the disc's motions being similar to an object, for example, “square” or “star” (Sekuler et al., 2003), segments could not intersect one another or even approach one another closer than half the length of a segment; second, the angle between consecutive segments had to lie between 30° and 150°. After completing the series of motion segments, the white disc disappeared, and a 3.75-s retention interval ensued. At the end of the retention interval, a colored disc appeared on the screen, prompting subjects to begin their imitation. Each subject performed 96 multipresentation trials in Experiment 1 and 76 trials in Experiment 2. 

Subjects reproduced the movement of the disc by drawing with a stylus on a graphic tablet positioned directly in front of their preferred hand. To begin an imitation, they touched the tablet with the stylus and started drawing. The colored disc's location on the screen was yoked with a 1:1 aspect ratio to the position of the stylus on the tablet (relative to the stylus' starting location), moving along with the stylus as the subjects were drawing. After finishing the imitation, subjects lifted the stylus from the tablet. The path traveled by the stylus was saved after each imitation for off-line analysis. Subjects were instructed to reproduce the motion trajectory as accurately as possible but were not informed about the metrics that would be used for analysis. They were asked to reproduce the appropriate number of segments and to try drawing straight segments to facilitate automatic segmentation. 

As we were interested in learning over repeated presentations, each model was presented multiple times. In Experiment 1, each five-segment model was presented four consecutive times, whereas Experiment 2 consisted of five successive presentations of six-segment models. After every presentation, subjects tried to reproduce the motion sequence. Following each imitation attempt, the experiment paused, until the subject touched the tablet to continue. The color of the disc representing the stylus movement differed between successive reproductions of the same model. After completing the final imitation, subjects were shown all their attempted imitations, color-coded and superimposed on a static image of the entire path that had been traversed by the stimulus disc. Note that this was the first time that subjects would actually see the entire path all at once; previously, the path was available only by knitting together the individual, time-sampled disc positions that were shown on the display. For a video clip showing an entire trial in Experiment 1, see Figure 1. 

Figure 1

 

Video clip showing an entire trial from Experiment 1. The subject viewed the same stimulus model (the motion of the white disc) four times. After each presentation, the subject tried to reproduce the motion of the disc by drawing with a stylus on a graphic tablet. The motion of the stylus is seen here as a colored disc moving on the display. At the end of four sets of viewing and imitation, the subject received feedback in the form of the static image of the entire path that had been traversed by the stimulus disc, superimposed on the four color-coded imitation attempts.

 

To score the accuracy of each imitation, an automatic segmentation algorithm used temporal and spatial criteria to break down each drawn path into individual segments. The algorithm searched for points where the stylus stopped on the tablet or where it changed its direction of motion to a sufficient degree. Segments were then extrapolated by connecting the chosen breakpoints using straight lines (see Agam et al., 2005, for details of algorithm). For inclusion in the analysis, we required the number of segments in the reproduction to match that of the stimulus in at least all but one of the imitations in a given trial. Otherwise, that entire trial was excluded from analysis. For each segment identified by the algorithm, the error in reproduction was defined by the absolute difference in orientation between that segment and the corresponding segment in the stimulus. The smaller the orientation difference, the more accurately a segment had been reproduced. 

Results

Figure 2 shows the results of Experiments 1 and 2. From Panels A and B, which describe mean orientation errors across all segments, it is clear that significant learning occurred after as little as one repetition of the stimulus ( p < .001 in each experiment, paired t test). Performance continued to improve after the second trial, but gains in learning diminished with each trial thereafter. In fact, in Experiment 1, the fourth presentation did not yield significant improvement over what was seen following the third reproduction ( p > .35). Note that substantial improvement in accuracy with successive reproductions was accompanied by only very modest changes in the number of reproductions containing the correct number of identified segments, that is, five segments (in Experiment 1) or six segments (in Experiment 2). This result is important because trials were excluded from error analysis if more than one reproduction during that trial had the wrong number of segments. In Experiment 1, the proportion of excluded trials went from 26% to 24% from the first reproduction to the fourth; in Experiment 2, the proportion of excluded trials went from 31% to 26%. The similarity of values for the two experiments suggest that repetition-based improvement in accuracy of reproduction is not much mediated by changes in the proportion of reproductions that contain the appropriate number of segments. 

Figure 2

 

Experiments 1 and 2: Learning across repeated presentations of the same model. (A and B) Mean orientation error across all segments as a function of presentation order for five- and six-segment models. (C and D) Serial-position curves. Plots show the orientation error for each individual segment. Each color corresponds to a different presentation of the stimulus model. Error bars are within-subject SEMs (Cousineau, 2005; Loftus & Masson, 1994) for each presentation.

Qualitatively, the learning curves in Figures 2A and 2B resemble ones reported previously for verbal material (Klein et al., 2005). However, one must exercise caution when comparing learning curves across studies. Later stages of learning, in particular, are extremely difficult to compare, as, in many cases, there is some upper limit on performance, where recall is perfect or near perfect. That outcome is especially likely when the underlying measure of success on any trial is binary and performance is summarized in terms of proportion correct. In our paradigm, this kind of limit is not an issue; our error metric for each segment is a continuous variable (orientation error), as opposed to a strictly correct or incorrect response. On the other hand, when a trajectory is reproduced by drawing on a tablet, even a very good memory for the trajectory is likely to result in some orientation errors, which arise from imperfect transfer to the motor system. The asymptotic convergence we observed in Figures 2A and 2B, then, may actually reflect “perfect” recall, tempered by limitations on subjects' drawing ability. At present, we cannot determine the source of the limitation reflected by the asymptotic convergence of the error curves. Additional extensive experiments would be required to find out whether it is related to the visual system, the motor system, or both. 

Next, we employed a finer grained analysis, looking at the errors associated with individual segments in each reproduction. Figures 2C and 2D show the serial-position curves, superimposed, for all presentations in each experiment. The first presentation, when examined alone, resembles our previous results with a similar task (Agam et al. 2005), with the most notable features being a strong primacy effect and a smaller, one-item recency effect. When considering how serial-position curves change with learning, we find a trend that has been seen previously with verbal material: a gradual flattening of the serial-position curve. As implied by such flattening, the middle, more error-prone segments benefit more from the repetitions than do the more error-resistant, first and final segments. However, although the bowed shape of the serial-position curve is less obvious in late rather that in early presentations, serial position remained a significant factor after the fourth imitation (p < .001, ANOVA) in Experiment 1 and after the fifth presentation in Experiment 2 (p < .015). The primacy effect, here tested as the difference between Segments 1 and 2, remained significant even after the final repetitions in both Experiments 1 (p < .0035) and 2 (p < .03). We should note that these effects are not the product of some artificial limit on performance, that is, a so-called ceiling or basement effect. 

The serial-position curves, then, retain some degree of bowing as learning proceeds and the error decreases. Improvement for each segment is roughly proportional to that segment's error on the preceding reproduction. This proportionality is consistent with the Hunter–McCrary law, which states that the serial-position curve remains invariant under a wide variety of circumstances related to serial learning (Brown, Neath, & Chater, 2002; Jensen, 1962; Lewandowsky & Murdock, 1989; McCrary & Hunter, 1953; Murdock, 1960). When expressed as a proportion of total error across the entire list, serial-position curves should be invariant across stages of learning. Although our error data do not lend themselves to such a proportional error analysis as easily as verbal (or any binary) data would, we generated an analogous analysis. For this analysis, we divided the orientation error at each serial position by the sum of errors across the entire model. Figure 3 shows proportional serial-position curves for both experiments. The principle outlined by McCrary and Hunter (1953) holds for our data. This is particularly true for results from Experiment 1, where all four curves, one for each reproduction of a model, are essentially identical (Figure 3A). In Experiment 2, the distribution of orientation errors does eventually level out, defying the Hunter–McCrary law, but does so only late into the learning process. The curves for the first and second presentations, where most of the learning was obtained, are striking in their congruence with one another (see Figure 2B). 

Figure 3

 

Serial-position curves representing orientation error as a fraction of total error across all segments. (A) Experiment 1; (B) Experiment 2. Each color corresponds to a different presentation of the stimulus model.

In a learning paradigm like ours, it is important to make sure that learning is model specific and does not reflect general improvement in the task with practice. Figures 4A and 4B show learning curves similar to Figures 2A and 2B but with trial binned into five groups, according to their chronological order. Both panels clearly show that the improvement in imitation accuracy happens within trials and that generalized, practice-based learning is tiny. In fact, there were no statistically significant differences in performance between the five groups for presentations at any position ( p > .2, Experiment 1; p > .85, Experiment 2; one-way ANOVA with factor quintile). 

Figure 4

 

Learning curves for groups of trials divided according to their chronological order. The first group represents the initial fifth of the trials, and the last group represents the final fifth of all trials. Ordinate values are mean error across all segments in stimulus models. Each set of bars represents a different group of trials, and bar colors reflect the order of presentation, from left to right. (A) Experiment 1; (B) Experiment 2. Error bars are within-subject SEMs for each presentation position.

We selected absolute orientation error as our primary measure of performance. However, other measures could be used to gauge the accuracy of imitation. A previous publication (Agam et al., 2005) reported detailed comparisons between various measures of error, such as the relative orientation between adjacent segments and error expressed in Cartesian (x–y) coordinates. As measures of error are correlated with one another, no one measure reflects just a single attribute of imitations. Our choice of absolute orientation differences was influenced by the striking similarity between the serial-position curves produced with this measure and those seen with other serial-recall tasks involving verbal material (Crowder, 1970; Drewnowski, 1980; Drewnowski & Murdock, 1980; Haberlandt, Lawrence, Krohn, Bower, & Thomas, 2005), spatial locations (Farrand, Parmentier, & Jones, 2001), and pictures (Avons, 1998). Errors in relative orientations did not produce the typical bow-shaped curves and were much higher than errors in absolute orientation, as would be expected from a correlated, but not explicitly encoded, variable. 

Figure 5 shows two additional analyses based on 2-D spatial criteria: location errors and rigid rotation errors. The first analysis focused on errors in the locations of reproduced segment endpoints: The dependent variable was the Euclidean distance between the middle points of segments in the stimulus and their corresponding reproduced segments ( Figures 5A and 5B). As noted previously (Agam et al., 2005), the relative flattening of the curves toward the end of the imitation suggests some late-course correction, possibly based on memory of the trajectory's overall shape or endpoint. Here, mean location errors seem to decrease following the second presentation (p < .001, both experiments, ANOVA) but not any further; hence, matching of the trajectory's endpoints seems unlikely to be the main factor underlying learning. This one-time improvement, however, suggests the interesting possibility of a two-stage learning process: First, subjects learn the general “gist” of the stimulus, for example, its overall orientation, and then later master the finer configurational details. We return to this point later, in the Discussion section. 

Figure 5

 

Alternative measures of error. (A and B) Error in spatial location, defined as the x–y Euclidean distance between the midpoints of corresponding stimulus and imitation segments. (C and D) Error in the orientation of the entire model, defined as the mean orientation of all five (Experiment 1, Panel C) or six (Experiment 2, Panel D) segments. (E and F) Absolute orientation errors, calculated the same way as in Figures 2C and 2D but with orientation differences (Panels C and D) removed on each trial prior to error calculation. Each color in Panels A, B, E, and F corresponds to a different presentation of the stimulus model. Error bars are within-subject SEMs.

Another possible source of error could be an imperfect alignment between the coordinate system used by the subject and the one on which stimuli were displayed. For example, it might be that models undergo some rigid rotation during transfer from visual memory to series of motor output, even if the stimulus model itself were recalled perfectly. To examine this type of error, we calculated, for each stimulus model and imitation, the mean orientations of all segments. This mean represents the “global” orientation of the entire trajectory. We then computed the rotation error as the absolute difference between the mean segment orientation in the imitation and in the stimulus. As can be seen in Figures 5C and 5D, this orientation difference decreases with repeated presentations but by no means to a degree that could explain the data in Figure 2. More direct evidence for the weak contribution of rotation errors is shown in Figures 5E and 5F, which, like Figure 2, describe absolute orientation errors but after correcting for whole-model rotations by subtracting the mean segment orientation from the orientation of each individual segment. In other words, the imitation has been superimposed on top of the stimulus model as closely as possible. The near identity of the curves, with and without this correction, suggests that nearly all of the imitation's inaccuracy arises from impaired reproduction of the trajectory's fine structure and not from shifts in coordinate systems. 

In many domains of cognitive science, time-based, chronometric measures have proven to be valuable supplements to other non-time-based measures. To support chronometric analysis of performance in our task, we examined two features of subjects' reproductions at various stages of learning: First, we measured the time subjects took to reproduce what they had seen; second, we computed the duration of pauses between successive reproduced segments. Figure 6 shows the mean time needed to complete an entire reproduction in Experiment 1 (Panel A) and in Experiment 2 (Panel B), for each of the successive reproductions. Total reproduction time was defined as the interval between the initial recorded stylus movement and the moment at which the stylus was lifted from the tablet, signaling the reproduction's end. Generally, subjects slowed down as learning progressed ( p < .001 in both experiments, ANOVA), interestingly converging to the approximate duration of the stimulus trajectory. This result is somewhat counterintuitive, as one might expect faster output after subjects have mastered the motion sequence. Rather, it suggests, especially given the convergence on a value close to the actual duration of the stimulus, that subjects have stored some dynamic representation of the motion segments and not simply a set of directions. 

Figure 6

 

Timing of imitation in Experiments 1 and 2. (A and B) Total time taken to reproduce entire models as a function of presentation order. Dashed lines represent the actual duration of stimulus presentation (750 ms per segment). (C and D) Pauses between reproducing individual segments. Note that each data point corresponds to the pause prior to the segment at the serial position for which it is drawn; the pause durations plotted against Serial Position 1 represent the lag between touching the tablet and the onset of stylus motion. Each color represents a different presentation of the stimulus model. Error bars are within-subject SEMs for each presentation.

Figures 6C and 6D show the mean duration of pauses between successive segments in reproductions. Separate curves are shown for each reproduction, from first to last, in both experiments. Pauses between segments are defined as the time taken to move a threshold distance (0.5 cm) away from a segment's starting point, as determined by the segmentation algorithm. Note that the first data point in each curve is not reaction time per se but the time between the stylus touching the tablet and the onset of movement. In both experiments, plots of intersegment pauses against serial position are strongly bowed, reaching maximum values at segments near the middle of the reproduction sequence. Paradoxically, although imitation generally slows down as performance improves, the longest pauses occur just prior to those middle segments, which are least accurately reproduced. Moreover, this trend becomes more pronounced as learning progresses. In both experiments, only in the final reproduction do pauses grow noticeably shorter. 

Recently, in a single-exposure, verbal serial-recall task, Haberlandt et al. (2005) reported similar serial-position effects for duration of items and pauses between items. These authors suggested that the bowing of the pause curves, with longest pauses toward the middle of the series, reflected grouping (“chunking”) of several words together, which created longer pauses between group boundaries. Applying the same approach to grouping of motion segments provides a compelling explanation for the results with intersegment pauses: Imitation accuracy improves because subjects group multiple individual segments into larger units, and the longer pauses represent the boundary between those “chunked” units, most likely to be situated near the middle portions of the serial-position curves. 

Experiment 3

Experiments 1 and 2 established a reliable procedure for measuring learning of novel motion sequences. They showed that as little as one repetition sufficed to produce substantial improvement in imitation performance and provided some clues to the basis of the improvement. In those experiments, after each stimulus presentation, subjects imitated what they had seen. But was overt imitation actually necessary for learning to take place? Might not a similar degree of learning have taken place if the subject merely viewed the stimulus multiple times, without trying to reproduce it between successive presentations? To assess learning in the absence of overt imitations, we compared (1) performance from a single actual imitation after viewing of multiple presentations and (2) performance following an equivalent number of presentations, each followed by an explicit imitation, as featured in Experiments 1 and 2. 

Methods

Fourteen subjects, none of whom had participated in Experiments 1 and 2, were tested here. Stimuli were identical to those in Experiment 1, with stimulus models comprising five motion segments. Here, however, subjects did not attempt to reproduce the stimulus trajectory after every presentation: Once the stimulus disc had disappeared, rather than reproduce the model, subjects were asked to touch the tablet to continue, which initiated the next presentation of the stimulus. While viewing the stimulus, subjects were instructed to keep the stylus motionless, at a fixed location at the corner of the tablet. This constrained possible movements of the subject's hand, which held the stylus, minimizing the opportunity for movement-based rehearsal involving full-scale movements. Only after the fourth and final stimulus presentation did subjects try to reproduce the trajectories they had seen. They were then shown their (one) imitation attempt superimposed on the stimulus trajectory, which provided subjects their sole opportunity to receive explicit feedback about the accuracy of their performance. Each subject was tested on a total of 96 trials. 

Results

Figure 7A shows the serial-position curve from Experiment 3 superimposed on the curve from the fourth presentation in Experiment 1 with similar stimuli. Note that the degree of learning appears to be equivalent between experiments: There is no significant difference in mean error between the two conditions ( p > .75, t test). This result suggests that visual perception alone mediates the learning process. Before settling on such a conclusion, though, several caveats need to be considered. First, subjects in Experiment 3 were tested only after the fourth presentation and had possibly (judging by the results of Experiment 1) reached their maximum ceiling performance at that point, which could have obscured differences between the two conditions. Additionally, the comparison of Experiments 1 and 3 is confounded by differences in timing within each trial. In particular, different intervals were allowed to elapse between stimulus presentations in the two experiments, as no imitation intervened between adjacent presentations in Experiment 3. Potentially, the shorter interstimulus intervals in Experiment 3 might have aided retention of the memory of what had been seen, giving a spurious advantage to the lone reproduction on trials in Experiment 3. The next experiment was meant to eliminate that confounding difference in timing, to support a cleaner assay of overt reproduction's contribution to learning. 

Figure 7

 

Learning with and without overt imitation. (A) Experiment 3. Serial-position curves following the fourth presentation, with ( Experiment 1) and without ( Experiment 3) imitation after every presentation, using a between-subject design. (B) Experiment 4. Two presentations, with a balanced time interval between them, using a within-subject design. Insets show mean orientation error across all five segments. Error bars are within-subject SEMs for each serial-position curve and standard error in insets.

Experiment 4

The results of Experiment 3 opened the possibility that visual input alone, without overt motor imitation, sufficed for learning of complex motion sequences. As mentioned before, though, this conclusion was compromised by the fact that compared with Experiment 1, the interval between successive presentations was shorter in Experiment 3, as stimulus presentations were not delayed by the time taken to produce an imitation. Thus, the comparable learning seen in Experiments 1 and 3 could have come from the longer delay between successive presentations in Experiment 1, which could have negated any benefit from an actual, overt imitation. Additionally, as noted above, a ceiling effect could have masked differences between the two experiments. 

Using a within-subject design, Experiment 4 equated the intervals between consecutive stimulus presentations, separating successive presentations in the absence of imitations by the same time that was required for the actual average imitation. Additionally, to avoid ceiling effects, we tested subjects after just two, rather than four, presentations of the same five-segment stimulus model. 

Methods

Fourteen new, experimentally naive subjects viewed two presentations of each five-segment model under two experimental conditions: imitation after each of the two presentations ( imitation condition) and imitation only after the second presentation ( observation condition). In the latter condition, the delay separating the two stimulus presentations was controlled so as to match the time each subject took to imitate the model (a running average was computed for each subject throughout the session). This way, the two presentations of each model were separated by a similar amount of time in both conditions. On imitation trials, the retention interval after the first presentation was followed by the appearance of a red disc, indicating that an imitation was expected. On observation trials, the display remained blank. In both conditions, the second presentation was followed by the appearance of a green disc, again prompting subjects to begin their imitation. Subjects performed 44 trials in each condition. For both conditions, explicit feedback, in the form of a superimposed display of the imitation and the model, was presented only after the second presentation of a model. This meant that any improvement seen from the first to the second test with the imitation condition would have had to occur in the absence of explicit feedback. 

Results

Figure 7B shows the serial-position curves for both imitation attempts in the imitation condition and for the sole imitation in the observation condition. The figure shows strong improvement from the first to the second reproduction in the imitation condition ( p < .001, paired t test), confirming what had been seen in the preceding experiments. Additionally, the one reproduction in the observation condition was significantly more accurate than the first reproduction in the imitation condition ( p < .02, paired t test) but did not differ significantly from the second reproduction in the imitation condition ( p > .29, paired t test). Thus, the improvement afforded by the additional reproduction performed in the imitation condition is no better than the advantage of simply viewing the model twice in the observation condition. This equivalence suggests that the bulk of learning in the imitation condition does not require actual, overt reproduction per se. However interesting, though, this result stops short of identifying the factor (or factors) that makes the improvement possible. The next experiment was meant to investigate one possible source of improvement: better representation of the static shape outlined by the motion of the disc. 

Experiment 5

This experiment explored the nature of the mental representations subjects were using when they performed the imitation task. Specifically, we asked: What was being learned during multiple presentations? Obviously, whatever had been learned could be learned without an explicit teaching signal; that is, subjects did not receive feedback after every presentation. One possibility is that successive presentations of the same model produced an enhanced sense of the trajectory's global shape. In other words, repetitions could have fine-tuned the representation of some static shape in subjects' memory, reminiscent of the demonstration of Haber and Hershenson (1965) that repeated brief exposures promoted the ability to see clearly some word that on first presentation had been seen poorly. If repeated presentations of a moving stimulus model worked in a similar fashion, sharpening subjects' mental template of the model's static shape, the serial order of segment presentation should be of little consequence, as long as the shape outlined by the disc trajectory was preserved across presentations. We tested this hypothesis directly by presenting the model in reverse order—from end to start—so as to preserve the outlined shape, while reversing the direction and order of the motion segments. 

Methods

Seven subjects, none of whom had participated in the previous experiments, viewed three presentations of each six-segment stimulus model. Two experimental conditions, each comprising 66 repeated sequences, were randomly interleaved. In the forward condition, a model's trajectory was presented three times in the same order; that is, on each presentation, the same directed segment was consistently first and another directed segment was consistently last. In the reversed condition, the second presentation of the stimulus model was presented with segments in reverse order: The disc moved from the point where the first trajectory ended backward along one segment after another, ending at the point where the first trajectory began. The first and third presentations were carried out in the regular, forward order. In both conditions, subjects attempted to reproduce the trajectory only once, following the third and final presentation. They were instructed to always perform their imitation in the order seen last, that is, the “forward” order, and received feedback on their performance following their single imitation attempt, after the third presentation. 

Results

Figure 8 shows mean orientation errors across segments (Panel A) and the serial-position curves (Panel B) for the two experimental conditions. Clearly, performance in the reversed condition is inferior to that in the forward condition ( p < .01, paired t test). This difference suggests that subjects were not merely building an improved static template of the disc trajectory's shape. Rather, learning of motion sequences may depend upon a more complex representation, one in which temporal relations among components play an important role. 

Figure 8

 

Results of Experiment 5. Curves show orientation error plotted against segment serial position in the forward and reversed conditions. Inset shows mean orientation error across all segments. Error bars are within-subject SEMs.

Discussion

In a series of five experiments, subjects viewed and then attempted to reproduce sequences of motions, allowing us to examine improvement in accuracy of reproduction with massed, multiple opportunities to view particular sequences. Experiments 1 and 2 showed that as little as one repetition of a stimulus model was enough to drive significant improvements in the accuracy with which motions in a sequence were reproduced. Experiments 3 and 4 addressed the role of motor rehearsal, comparing reproduction accuracy after repeated presentation, with and without overt imitation between consecutive presentations. The results of these experiments suggest that at least in our testing environment, actual overt imitation is not necessary for learning to occur. Contrasting improvements in performance produced when motion sequences were presented either in a consistent order or in variable order, Experiment 5 showed that the temporal organization of the sequence was incorporated into the sequence's representation in memory. 

Before discussing the implications of these experiments, it is worthwhile to distinguish the task used in our experiments from other motor tasks that focus on learning by repetition. Among the best known of those tasks is the serial reaction time (SRT) task, in which a long sequence of simple motor responses is learned, often implicitly (see Clegg, DiGirolamo, & Keele, 1998, for review). Unlike the task employed here, the SRT task does not involve working memory; each movement is executed immediately after it is cued, and the movements to be learned are typically directed to fixed, visible target locations. Using a somewhat different approach to repetition-based learning, Hebb (1961) showed that serial recall of a list of words improves when the list is repeated every few trials, which are interspersed among other lists, without the subject's explicit knowledge of the repetition. Subsequently, implicit serial learning was demonstrated a number of times with this same approach (Cunningham, Healy, & Williams, 1984; McKelvie, 1987; Page, Cumming, Norris, Hitch, & McNeil, 2006; Schwartz & Bryden, 1971; Sechler & Watkins, 1991; Stadler, 1993). However, it is clear that consciously learning, by explicit multiple presentations and recalls of the same sequence, comprises a task that is essentially different from the one conceived by Hebb, where list repetitions are separated by at least two intervening trials. Given how common such explicit learning by massed repetition is in everyday life, this mode of learning has attracted surprisingly little attention in recent years (Addis & Kahana, 2004) and none at all in the nonverbal domain, including learning by imitation. 

The learning curves from our experiments share several features with curves from previous studies using verbal test materials. For example, in Agam et al. (2005), the serial-position curve produced by one imitation of a (five- or six-segment) model exhibits a strong primacy effect and a smaller, one-item recency effect, both typical outcomes of immediate serial recall (see Figures 2C and 2D). As in studies with verbal stimuli (e.g., Klein et al., 2005), the experiments here show that a single repetition suffices to induce a significant reduction in error. An important result not shown previously with nonverbal stimuli is the invariance of serial-position curves over successive stages of learning, particularly during the earlier presentations. We found that when error is expressed as a fraction of total error in the whole imitation, the serial-position curves are highly similar to one another, despite substantial improvement in overall accuracy. This invariance of the serial-position curve is known as the Hunter–McCrary law (Brown et al., 2002; Lewandowsky & Murdock, 1989; McCrary & Hunter, 1953). The similarities among serial-position curves corresponding to different presentations in Experiments 1 and 2 suggest that learning of motion sequences follows a power law. In other words, improvement in reproduction accuracy from imitation n to imitation n + 1 is proportional to the existing error for each segment in imitation n. Although our results are compatible with several alternative mechanisms, learning's invariance over serial position does rule out “chaining” accounts (Addis & Kahana, 2004; Lewandowsky & Murdock, 1989) in which improvement is “anchored,” leading to a sharpening of the initial segment's (or segments') representation and, then, as practice progresses, to a sharpening of the representation of later segments. 

Two lines of evidence indicate that subjects were using a representation that was not strictly shape based but one that incorporated an essential temporal dimension (Freyd, 1987; Stone, 1998; Wallis & Bülthoff, 1999, 2001): First, in Experiments 1 and 2, the time it took subjects to complete the imitation gradually converged to the duration of model presentation. This convergence suggests that subjects had encoded temporal aspects of the stimulus. It should be noted, though, that the slowing down of the imitation might also reflect subjects' growing confidence in their memory of what they had seen, in which case the match between imitation and stimulus duration could be coincidental. A second indication of the representation's dynamic character is the fact that reversing motion direction in the second of three presentations retarded learning in Experiment 5; learning was most effective as when models were presented repeatedly with segments in the same order and movement along each segment in a consistent direction. The importance of consistency of order and directions suggests that the representational basis for the imitation was not simply a static, shape-like mental image but a representation replete with distinct spatiotemporal properties. 

An analysis of pauses between the reproduction of individual segment suggests that the learning process involves grouping of multiple segments into a more compact representation (Haberlandt et al., 2005; McLean & Gregg, 1967). This assumption is supported by the disparities among segment durations at various serial positions: The slowest segments to complete, on average, were the middle ones, where group boundaries were most likely to occur. Moreover, this pattern intensified with each repetition, suggesting the formation of memory “chunks,” but subsided during the final imitation, possibly indicating the attainment of a more stable representation in memory. One could offer another alternative explanation, which links the higher error in late segments (and subjects' awareness that they are making larger errors) to the time it takes to plan or to produce a segment. Such an account is unlikely, however, for two reasons: First, the more accurate subjects become (with stimulus repetition), the more pronounced the patterns of intersegment pauses become, and second, the peaks in the pause curves are not in the “right” serial position to be explained as a consequence of higher error or uncertainty, in which case the slowest segments to be drawn would be the next-to-final and final segments. 

One striking result is the weak effect that overt motor imitation exerts on the rate of learning. When subjects only viewed the models, without trying to reproduce them after each presentation, the loss in imitation accuracy following the final repetition was, if anything, modest. This near equivalence of performance with and without opportunity for overt imitation suggests that improvement was based in subjects' memory and not in some enhanced ability to transfer a visual representation of a trajectory into a motor plan. The fact that overt motor rehearsal is not, under our experimental conditions, necessary for learning to occur could mean one of three things: (1) Stimulus trajectories were not encoded using motor representations, and the visual representation was transformed to a motor plan only at the time when imitation was required; (2) internal representations used some motor substrate that was not controlled for, such as eye movements, which are thought to contribute to spatial memory processes (Lawrence, Myerson, Oonk, & Abrams, 2001; Pearson & Sahraie, 2003; Postle, Idzikowski, Sala, Logie, & Baddeley, 2006; Theeuwes, Olivers, & Chizk, 2005); or (3) subjects did rely on hand movements as a rehearsal mechanism but used motor simulation rather than actual hand movements. As motor simulation, that is, imagined motor actions, recruits many of the same neural mechanisms that support real movements (Gerardin et al., 2000; Jeannerod, 2001; Jeannerod & Frak, 1999; Lotze et al., 1999; Porro et al., 1996), it is entirely possible that simulation could provide a rehearsal mechanism for motor-based working memory. 

Additional experiments will be required to distinguish among the three possibilities outlined in the preceding paragraph. Studies that use eye tracking and functional neuroimaging of areas such as the frontal eye fields (Balan & Ferrera, 2003; Curtis, Rao, & D'Esposito, 2004) could settle the question of eye-movement-based memory. Of course, the final alternative mentioned above, motor simulation, is much more difficult to monitor. Most likely, a test of motor simulation's role in learning would require a carefully calibrated, secondary motor task designed to interfere with motor simulation (Fortin, Duchet, & Rousseau, 1996; Stevens, 2005). The nature of possible motor representations in memory could be addressed by changing the effector in the interference task, for example, between the preferred hand (which executes the imitation) and the nonpreferred hand. 

Much research is still needed to better understand the mechanisms that govern learning by repetition, including what is arguably the most intriguing aspect of the results described here: Learning was able to take place in the absence of an explicit correction signal (i.e., information provided by the experimental environment, as opposed to internally generated knowledge stored in working memory). Subjects did not receive feedback on their performance after every imitation; thus, they could not compare their memory representations to the “correct” answer and, then, exploit the discrepancy between the two as an error signal. That such learning occurred in the absence of explicit feedback is consistent with the notion that some or all learning takes place at a perceptual level. Although explicit feedback does facilitate perceptual learning under some conditions, there is consensus that robust perceptual learning can occur without such a teaching signal (see, e.g., Herzog & Fahle, 1997; Seitz & Watanabe, 2005). 

In a separate study, meant to probe the interaction between working memory and perception, we measured scalp EEG signals while subjects viewed, for later imitation, five-segment stimuli like the ones used in the experiments presented here. Decreases in time- and frequency-based EEG markers of visual attention, namely, event-related potential (ERP) amplitude and power in high-frequency oscillations, suggested that later segments are processed less effectively than earlier ones (Agam & Sekuler, 2007). We interpreted these results as reflecting competition for attentional resources between working memory (whose load increases with each additional segment) and visual perception. This finding, which was based on just a single presentation of each model, could be exploited to probe changes that occur with learning over repeated presentations. If learning were indeed perception based, as we have suggested above, then perceptual encoding of successive motion segments during learning could be facilitated by past exposure. As a result of this hypothesized facilitation, subsequent encoding of motion segments would consume fewer attentional resources, which could then be directed toward maintaining representations of previous motion segments in working memory. This account would be most plausible if subjects initially remembered the general directions in which successive segments moved (the model's “gist”) and, thus, reduced the possible range of orientations that could be expected for each presented segment. This hypothesis, in which reduction of uncertainty and enhanced predictability play a key role (Ball & Sekuler, 1981; Sekuler & Ball, 1977), could be tested by examining repetition-based learning along with coordinate recordings of EEG and ERPs. It makes two basic predictions: Generally, markers of perceptual processing, that is, ERP amplitude and high-frequency power, should be reduced after the initial presentation, as the model becomes more familiar and its movements become more predictable (Poggio & Bizzi, 2004; Schubotz & von Cramon, 2002, 2004). Additionally, the serial-position dependence of these markers (Agam & Sekuler, 2007) should decrease as learning progresses and as the model-imposed memory load diminishes. 

Acknowledgments

This study was supported by NSF Grant SBE-0354378 and NIH Grant R01MH068404. H. G. and B. J. G. were responsible for data collection and preliminary analysis of Experiments 1 and 2, respectively. We thank Dan Bullock for encouragement and advice and Trisha Hussoin for assistance with Experiment 2. 

Commercial relationships: none. 

Corresponding author: Yigal Agam. 

Email: [e-mail]. 

Address: 415 South St., Waltham, MA 02453, USA.