Recruitment of a novel cue for active control depends on control dynamics

Abstract

We investigated how the visual-motor system recruits a novel visual feedback cue for a manual control task. We presented conditions in which an arbitrary cue (color) was coupled with task-relevant feedback (position or velocity), and measured the effect of the novel cue on performance. Participants used a joystick to keep a moving horizontal line centered on a display under velocity or acceleration control dynamics. Participants normally rely primarily on line position feedback for velocity control and line velocity feedback for acceleration control. The novel color cue was coupled with either line position (becoming red as it deviates from center) or line velocity (becoming red as it moves faster). For velocity control, performance error was smaller and response gain was larger when the novel color cue was coupled with line position than when it was coupled with line velocity. Conversely, for acceleration control, performance was better when color was coupled with line velocity than with line position. Our findings show that the visual-motor system can recruit a novel arbitrary cue to improve active control performance, but the effectiveness of the novel cue depends on its relationship to the feedback appropriate for control dynamics.

Introduction

When we perform visually guided actions in the world, there are often multiple sources of visual feedback information available. Thus, an important part of visual-motor learning is becoming able to attend and utilize the most relevant visual feedback for a control task (Gibson, 1966). Clearly we must have the capability to learn the mapping from the most relevant visual feedback to control actions, or we would not be able to gain control expertise in natural conditions. Indeed, some recent studies have demonstrated that, through experience with a task, we can learn to identify and use more effective control signals from the available visual feedback (e.g., Fajen, 2008; Smith, Flach, Dittman, & Stanard, 2001). 

The purpose of the present study was to investigate how our visual-motor system recruits an entirely novel visual feedback signal for a manual control task. In the previous studies by Smith et al. (2001) and Fajen (2008), learning involved attunement to new combinations of available visual motion cues. In this study, we tested whether an arbitrary visual cue (color) with no previously learned association with manual control can through experience become an effective visual feedback signal to improve continuous closed-loop control performance. Assuming that such cue recruitment is possible, one can ask what factors determine the effectiveness of the novel cue. Previous studies have reported that we rely on different visual feedback depending on the dynamics of the control task (Fajen, 2008; Li, Sweet, & Stone, 2005, 2006). In the present study, we thus examined whether the recruitment of a novel cue similarly depends on its relationship to the control dynamics. 

We used a manual control task in which participants used a joystick to keep a line centered on a CRT display as the line underwent quasi-random movement. This task has been studied previously (Li et al., 2005, 2006; Sweet, 1999) and is a simplified version of real vehicular control tasks such as stabilizing the horizon when controlling an aircraft. Two types of joystick control dynamics were tested in the current study: in the velocity control condition, the joystick displacement was proportional to the rate of change of line position; in the acceleration control condition, the joystick displacement was proportional to the rate of change of line velocity. Velocity control is commonly experienced in our daily life, e.g., when driving a car, the rate of change of the car's moving direction is proportional to the steering wheel angle displacement. Acceleration control, as experienced in flying an airplane, is less common and more challenging but can be mastered with practice. 

The visual feedback for this control task comes from the moving line, with the main control signals being the line's instantaneous position and velocity. The contributions of position and velocity signals to control performance are not constant, but rather depend on the dynamics of the controller (i.e., the joystick in the current study). Previous work on manual control has established that human operators rely primarily on position feedback for velocity control, and on velocity feedback for acceleration control (e.g., Li et al., 2005, 2006; McRuer & Krendel, 1974; McRuer, Krendel, & Reisener, 1965; Sweet, 1999). 

To introduce a new potential feedback signal for the manual control task, we changed the color of the moving line. Three color-feedback conditions were tested. In the no-color condition, the color of the line remained constant (gray) regardless of the line movement. In the position-color condition, the color of the line was correlated with the distance of the line from the center of the screen, i.e., the line was gray when at the center of the display and became increasingly red as it moved either above or below the display center ( Figure 1a). In the velocity-color condition, however, the color of the line was coupled to the velocity of the line rather than its position, i.e., the line was gray when its instantaneous velocity was zero and became increasingly red as its instantaneous velocity increased regardless of its moving direction ( Figure 1b). 

Figure 1

 

Color was introduced as a potential feedback signal for a manual control task. (a) In the position-color condition, the line was gray when it was positioned at the center of the screen but became increasingly red as its position deviated from the center of the screen. (b) In the velocity-color condition, the line was gray when it was stationary and became increasingly red as a function of its speed.

The two different types of color feedback (position and velocity) were used in order to vary the relationship between the introduced novel color cue and the visual feedback for velocity and acceleration control dynamics. When the added color cue is correlated with line position, it coincides with the primary visual feedback that operators use for velocity control but not acceleration control. Conversely, when the added color cue is correlated with line velocity, it coincides with the primary visual feedback for acceleration but not velocity control. Our experimental design therefore allowed us to discern whether the effectiveness of a novel cue in manual control depended on the sensory information itself (i.e., position color vs. velocity color) or its relationship to the visual feedback appropriate for the dynamics of the control task. 

Methods

Participants

Seven staff members and students (four naive to the specific goals of the present study) at the University of Hong Kong participated in the experiment. All had normal or corrected-to-normal visual acuity and normal color vision. 

Stimuli

The stimulus was a Gaussian-blurred horizontal line (67°H × 4.2°V, blur standard deviation = 1.6°) on a black background (0.27 cd/m ²), displayed on a CRT monitor (67°H × 52°V) with 60-Hz refresh rate. During a trial, the line's vertical position on the CRT monitor was perturbed by the sum of 10 harmonically independent sinusoids, and the participant's task was to use a joystick (B&G Systems, FlyBox) to keep the line centered on the monitor screen. The input line position perturbation u had the following form as a function of time t:  

$u\left(t\right)=\sum _{i=1}^{10}D\frac{a_{i}2\pi k_i}{240}\sin (\frac{2\pi k_i}{240}t+{\rho }_i).$

(1)

Table 1 lists the actual values of a, k, and resulting frequencies ( ω = 2 πk/240) used for the study. D was set to a value of 0.8° for the velocity control and 0.2° for the acceleration control (due to the greater difficulty of the latter). The phase offset ( ρ _i) was randomly varied from − π to π. The use of harmonically independent sum of sines not only made the line motion on the screen appear random but also allowed for a frequency-based analysis of the linear component of the control response while minimizing artifacts from non-linearities that might produce harmonic distortion (Elkind, 1956; McRuer & Krendel, 1959; Stark, Iida, & Willis, 1961). The average speed of the uncorrected input disturbance was 2.25°/s (peak: 8.51°/s). 

Table 1

 

Magnitudes and frequencies of the 10 harmonically independent sinusoids in the input position perturbation u.

i	a i	k i	ω i (Hz)
1	2	5	0.021
2	2	8	0.033
3	2	13	0.054
4	2	23	0.096
5	2	37	0.154
6	2	59	0.246
7	0.2	101	0.421
8	0.2	179	0.746
9	0.2	311	1.296
10	0.2	521	2.171

Two types of joystick control dynamics were tested; the displacement was proportional either to the rate of change of line position ( velocity control) or to the rate of change of line velocity ( acceleration control). The control dynamics of the joystick ( Yc) was implemented in software by the display computer with  

$Yc=\frac{1}{s}$

(2)

for velocity control and  

$Yc=\frac{1}{s(s+0.2)}$

(3)

for acceleration control, where s is the Laplace Transform variable. Note that our acceleration control is not a perfect acceleration control system of 1/ s ². We added a damping factor of 0.2 s to reduce task difficulty. The joystick position was sampled at 60 Hz, i.e., every frame of the display. The joystick displacement values ranged from −1 to 1, corresponding to the full backward and forward positions, respectively. 

There were three color-feedback conditions. In the no-color condition, the moving line was not colored (i.e., gray) and its luminance (peak at 7.3 cd/m ²) was kept constant. In the position-color condition, the line appeared redder as it deviated from the center of the display and became reddest at the top or the bottom edge of the screen ( Figure 1a). In the velocity-color condition, the line appeared redder the faster it moved and became reddest at the speed of 52°/s ( Figure 1b). The color change was produced by increasing the power output of the red gun and decreasing the power output of the blue gun of the CRT monitor to ensure that the overall luminance of the line did not change. The reddest color that the line could become corresponded to x = 0.374 and y = 0.251 in CIE color space. 

Procedure

On each trial, a stationary line appeared and began moving when the participant pulled the joystick trigger. The line initially moved according to the sum-of-sines perturbation input, but this line motion was reduced as the participant's task was to move the joystick forward and backward to keep the line centered on the monitor screen. The duration of each trial was 245 s. 

Each participant ran two experimental sessions on two different days. Each session consisted of six conditions (3 color-feedback types × 2 control dynamics) with three trials per condition and typically lasted about 1.5 h. Trials were blocked by control dynamics and color-feedback type (3 trials per block), and the testing order was counterbalanced across sessions as well as across participants. There was a 30-s break between trials and a 4-min break between blocks. To ensure participants understood the task and became familiar with the control dynamics, they performed practice trials in the no-color condition before the experiment commenced. Pre-experiment practice continued until performance appeared stable, which required 6–9 trials for each control dynamics. 

Data analysis

We used several metrics of active control performance. Total performance error was measured as the root mean square (RMS) of the time series of recorded line position error from the perceived center of the screen in degrees of visual angle. We analyzed the data beginning 5 s after the start of the trial to ensure that we skipped the initial transient response. To examine the participant's control response specific to the input perturbation frequencies, we performed frequency (Bode) analyses to describe the human operator transfer function ( Yp) from our closed-loop performance data ( Figure 2). That is, we performed the Fourier analyses on both the visual position error and the joystick displacement to obtain the relevant signal amplitudes and phases. We then took the ratios of the amplitudes and the difference between the phases to compute the gain and phase lag, respectively, at each perturbation frequency. The RMS error, gain, and phase lag measures were averaged across the six trials in each condition. 

Figure 2

 

Simplified block diagram of the closed-loop manual control task. The human operator transfer function captures the operator's control compensation, and the controller dynamics specifies the joystick control dynamics.

Results

Baseline performance

Before considering the effect of the novel color cues, we first present sample control response data from the baseline no-color conditions. Figure 3 shows representative trials for velocity and acceleration control dynamics. The joystick response, considered as a function of time, was typically a scaled and delayed version of the input visual line position error signal, with a falloff in the response at the highest frequencies. The overall accuracy of performance, as expressed by the RMS error in line position, differed for the two types of control dynamics. As expected, the RMS error for velocity control was smaller than that for acceleration control (mean: 2.98° vs. 5.40°, t(6) = −11.92, p < 0.0001), consistent with the latter's greater difficulty. 

Figure 3

 

Typical raw performance data of the input line position error (solid) and the output joystick displacement (dashed) for (a) velocity and (b) acceleration control. Note that the output is a low-pass filtered and delayed version of the input. The delay was more obvious for acceleration control due to its greater difficulty.

Baseline performance for velocity and acceleration control also differed with respect to the human operator transfer function. Figure 4 plots the overall gain and phase of frequency response, averaged across seven participants, as a function of the input line perturbation frequency for the two types of control dynamics. For velocity control, the flat pattern of gain and the roll-off of phase with increased frequency show that participants behaved like a simple gain controller with a time delay, i.e., control response was in direct proportion to the input line position error. For acceleration control, in comparison, gain was not constant but rather increased with frequency at 20 dB/decade, and phase was shifted in the positive direction at the middle frequencies (i.e., in the 0.1–0.4 Hz range). These properties are characteristic of a control response that is proportional to the rate of change of position (i.e., velocity) rather than position itself. Use of velocity allows the controller to anticipate the position error signal, resulting in higher gain at higher frequencies and positively shifted phases, as we observed. These human operator transfer functions for velocity and acceleration control are consistent with findings from previous studies on manual control (McRuer & Krendel, 1974; McRuer et al., 1965; see a review in Jagacinski & Flach, 2003; Wickens, 1986). 

Figure 4

 

Frequency response (Bode) plots for the human operator transfer function (Yp) for the baseline no-color condition. The top panels depict mean gain and the bottom panels depict mean phase averaged across seven participants as a function of perturbation frequency for (a) velocity control and (b) acceleration control. Error bars represent SEs across participants (some of them are smaller than the data symbols).

Thus, as expected, the two types of control dynamics in the current study elicited different dynamic responses from participants that reflected their differential reliance on line position and velocity feedback information. We now consider the effects of the added novel color cues on control performance and how their effectiveness is related to control dynamics. 

Effects of color feedback

RMS error

Figure 5 plots the mean RMS error, averaged across the six trials and seven participants, as a function of color-feedback type for velocity and acceleration control, respectively. Two separate repeated-measures ANOVAs revealed that the main effect of color-feedback type was significant for both velocity control ( F(2, 12) = 5.09, p = 0.025) and acceleration control ( F(2, 12) = 4.46, p = 0.036). Post hoc Duncun tests showed that for velocity control, the RMS error for the position-color condition (2.79°) was significantly smaller than that for the no-color (2.98°, p = 0.041) or the velocity-color condition (3.04°, p = 0.012), while for acceleration control, the RMS error for the velocity-color condition (4.77°) was significantly smaller than that for the no-color (5.40°, p = 0.015) but not the position-color condition (5.17°, p = 0.086), and no significant difference in the RMS error was found between the latter two conditions ( p = 0.301). 

Figure 5

 

Mean RMS error against color-feedback condition for (a) velocity control and (b) acceleration control. Error bars are SEs across seven participants.

The overall pattern of the results indicates that the benefit from an introduced color cue depends on both control dynamics and how the color cue is provided. The color cue provides the most benefit when it is coupled to the visual feedback most effective for the employed type of control dynamics, i.e., line position feedback in the case of velocity control, and line velocity feedback in the case of acceleration control. The only finding that does not support this characterization is the lack of difference between velocity-color and position-color conditions for acceleration control, and for these conditions that trend is in a consistent direction. 

Gain

The RMS error measures the total performance error both visually and non-visually driven. The errors specific to the perturbation frequencies, however, are a better measure of the visually driven response specific to the moving line, and Fourier analysis allows us to segregate response-gain and response-delay effects. To analyze how participants' control response varied across color-feedback types at each of the input line perturbation frequencies, we computed the human operator transfer function (i.e., the ratio of the Fourier transform of the joystick displacement to that of the visual position error, see Figure 2). 

Figure 6 plots the mean response gain, averaged across the six trials and seven participants, as a function of frequency for each of the color-feedback and control dynamics conditions. We excluded the gains from the lowest three frequencies (<0.1 Hz) because our performance measurements were noisy at these ultra-low frequencies, due to the limited number of cycles available for analysis. The mean response gain averaged across the seven frequencies is also plotted as the rightmost points in Figure 6. 

Figure 6

 

Mean gain of the human operator transfer function ( Yp) against perturbation frequency for the three color-feedback conditions for (a) velocity control and (b) acceleration control. The rightmost points in the graphs plot the mean gain averaged across the seven frequencies. Error bars represent SEs across seven participants.

Due to the different gain characteristics in velocity and acceleration control (see Figure 4), two separate 7 (frequency) × 3 (color-feedback type) repeated-measures ANOVAs on the response gain were conducted. For velocity control, both the main effects of frequency and color-feedback type were significant ( F(6, 36) = 13.01, p < 0.0001 and F(2, 12) = 5.33, p = 0.022, respectively). Post hoc Duncun tests revealed that the overall response gain was significantly larger for the position-color (7.18 dB) than for the no-color (6.41 dB, p = 0.037) or the velocity-color condition (6.14 dB, p = 0.011), and the overall response gain for the no-color condition was not significantly different from that for the velocity-color condition. For acceleration control, both the main effects of frequency and color-feedback type were also significant ( F(6, 36) = 101.32, p < 0.0001 and F(2, 12) = 4.01, p = 0.046, respectively). Post hoc Duncun tests revealed that the overall response gain was significantly larger for the velocity-color (14.57 dB) than for the no-color (13.82 dB, p = 0.029) or the position-color condition (13.90 dB, p = 0.041), and the overall response gain for the no-color condition was not significantly different from that for the position-color condition. Thus, while for velocity control the gains were highest for position-color feedback, for acceleration control the gains were highest for velocity-color feedback. This pattern of results for response gain indicates again that the effect of the added novel color cue on control performance depends on both its relationship to the visual feedback and control dynamics, providing the most benefit when the colored-feedback cue is appropriate for the control dynamics. 

Phase lag

To quantify the effect of the added color cues on response delay, expressed in terms of phase lag, we plotted the mean phase lag, averaged across the six trials and seven participants, as a function of frequency for each of the color-feedback and control dynamics conditions in Figure 7. The mean phase lag averaged across the seven frequencies is also shown in Figure 7 (rightmost points). For both control dynamics, a 7 (frequency) × 3 (color-feedback type) ANOVA on the phase lag showed that only the main effect of frequency was significant ( F(6, 36) = 519.40, p < 0.0001 and F(6, 36) = 338.11, p < 0.0001, for velocity and acceleration control, respectively). As expected, the higher the frequencies, the larger the phase lag. The added color cues did not appear to have any significant effect on response delay. 

Figure 7

 

Mean phase of the human operator transfer function ( Yp) against perturbation frequency for the three color-feedback conditions for (a) velocity control and (b) acceleration control. The rightmost points in the graphs plot the mean phase averaged across the seven frequencies. Error bars represent SEs across seven participants.

Learning effects

The introduced novel color cue presumably had no pre-existing association with position or velocity cues, so some learning would have been necessary to use the novel cues to improve performance. To evaluate the learning process, we conducted a trial-by-trial analysis. In each of the two experimental sessions, participants performed three sequential trials in each of the color-feedback and control dynamics conditions. Figure 8 shows the mean RMS errors (upper panels) and gains (lower panels) for each condition plotted as a function of trial number: Trials 1–3 were from the first session, and trials 4–6 were from the second session. For the no-color condition, participants were trained up to stable performance before the experiment commenced, so we did not expect any learning effect. For the position-color and velocity-color conditions, the improvement from learning the new color cues could have emerged gradually across trials. This was not what we observed. No systematic improvement in performance is apparent in the graphs, for either the RMS or gain measures. Furthermore, some of the apparent trends are in the opposite direction as expected from learning. 

Figure 8

 

Mean RMS error against trial number for the three color-feedback conditions for (a) velocity control and (b) acceleration control. Mean gain against trial number for the three color-feedback conditions for (c) velocity control and (d) acceleration control. Error bars represent SEs across seven participants.

To statistically test for learning effects, we performed linear regressions across trials for each individual participant and condition. We analyzed RMS and gain measures but not phase lag, as effects of the color feedback were not observed for phase lag. Because the testing order of both color-feedback type and control dynamics was counterbalanced across the two sessions, interference could have prevented learning from accumulating across sessions. We therefore conducted the regressions using only the three trials within each session. None of the slopes were significantly different from zero for either the RMS errors ( t(19) < 1.63 for all tests, p > 0.11) or the gains ( t(19) < 1.07, p > 0.29). We thus were not able to detect the process of learning to use the novel color cues. The learning may have occurred too quickly to be detected or may have been obscured by interference across different color-feedback and control dynamics conditions in our counterbalanced design (see the Discussion section). 

Discussion

Our results indicate that participants can learn to use a new visual feedback cue (such as color) that has no previously learned association with manual control, to improve their performance in a closed-loop manual control task. Furthermore, the introduced new cue is effective only when it is coupled with the pre-existing visual feedback that participants normally use for the control task dynamics. Position-color feedback produced a 6% reduction in total RMS error and a 0.77-dB increase in response gain compared with no-color feedback for velocity control but no improvement for acceleration control. Conversely, velocity-color feedback produced a 12% reduction in total RMS error and a 0.75-dB increase in response gain compared with no-color feedback for acceleration control but did not improve performance for velocity control. These findings suggest that the recruitment of a new cue for visual-motor control is selective and depends on control dynamics. 

Our study is the first to investigate how the visual-motor system recruits an arbitrary new visual feedback cue for active control. Some other recent studies have demonstrated learning of new visual feedback cues. For example, Smith et al. (2001) found that subjects could learn through practice to use a novel combination of optical variables to improve performance in a ball-hitting task. Fajen (2008) also demonstrated that subjects could be trained to use new combinations of optic-flow cues in a simulated braking task. Our results demonstrate, however, that even a sensory cue not typically relevant to the control task—changes in color—can be recruited as an effective feedback signal to improve closed-loop control performance. 

In addition, our results show that the benefit of the novel cue depends on its relationship to the pre-existing visual feedback appropriate for the control dynamics. This is consistent with the previous findings showing that the visual-motor system adapts the use of visual feedback to the changes in the dynamics of the control task. Specifically, Li et al. (2005, 2006) found that while for velocity control dynamics, varying luminance contrast affected use of position more than velocity cues, for acceleration control dynamics, it affected use of velocity more than position cues. Similarly, Fajen (2008) found that subjects relied on different optic-flow cues depending on the dynamics of the braking task (i.e., mapping between brake position and deceleration). In the present study, the velocity-color cue analogously has greater contribution to performance under acceleration than velocity control, and conversely the position-color cue has greater contribution under velocity than acceleration control. Thus, converging evidence indicates that consistency between visual feedback cues and control dynamics is important for successful visual-motor performance. 

For the cue-recruitment process observed in our study, there are multiple possible underlying mechanisms. One possibility is that the introduced novel color cues contribute as an independent visual feedback signal to the controller, weighted according to whether the signal is appropriate for the control dynamics. Because the change of color in our conditions indicated only the magnitude but not the sign of change in line position or line velocity, the color cues were not sufficient in themselves for closed-loop control. However, if some minimal interaction with line position or motion cues resolves the sign ambiguity, the color cues could have contributed as a largely separate and independent visual feedback signal. A second possibility is that the benefit from color feedback is mediated by its effect on perception of line position and motion. Several perception studies have reported that our visual system can recruit irrelevant visual cues to resolve perceptually ambiguous stimuli through a process of associative learning with pre-existing cues (Haijiang, Saunders, Stone, & Backus, 2006; Sinha & Poggio, 1996). If such perceptual cue recruitment occurred in our study, the color cues could have contributed to the control performance by enhancing the perceptual information provided by the pre-existing position and motion cues. A third possibility is that the color feedback contributed to performance indirectly by changing the way position and motion cues are used for the control task. If participants were not making optimal use of the pre-existing position and motion cues, then some improvement in performance could be achieved by better utilizing these pre-existing cues. Color feedback could have served to direct attention to either the line position or motion, which would provide benefit only if the attended cue was appropriate for the control dynamics. The results from our current study cannot distinguish between these possibilities. 

Although we observed significant effects of the introduced color cues on both the RMS error and the response gain measures of the closed-loop control performance, the effects are relatively small (<15% reduction for RMS error and <1-dB increase for response gain on average). One possible reason for such small effects is that in our design, participants performed blocks of trials with different control dynamics and relationships between the colored feedback and control dynamics. A disadvantage is that when participants switch either control dynamics or color-feedback conditions, there are likely some interference effects across blocks. Interference across changes in control dynamics has been shown previously (e.g., Chernikoff, Duey, & Taylor, 1960). Interference between position-color and velocity-color feedback conditions is also plausible, as participants might persist in using color for position in the velocity-color condition, and vice versa. To prevent general interference effects from being confounds in our design, we counterbalanced the testing order of both control dynamics and color-feedback conditions across the two experimental sessions and across participants. However, interference effects could have reduced the benefits of the introduced novel color cues, in which case our experiment would have underestimated the effects of the novel color cues on active control performance. 

Improving the quality of visual feedback might be expected to improve both response gain and delay measures of control performance. For example, Li et al. (2005) found that luminance contrast affects both response gain and response delay. However, we observed no benefit from the color cues in response delay measures. This null result could be a result of the modest influence of the novel cues in general, or could be specific to color as a feedback cue. For example, processing of color changes may be too slow relative to processing of position and motion cues to provide any reduction in delay. Further testing of cue recruitment for other types of novel feedback signals could resolve this question. 

The findings from our current study have practical implications for the user interface design of active control systems. If a design engineer seeks to improve operator performance on a closed-loop manual control task by providing novel cues to enhance the pre-existing sensory feedback, our results indicate that they would have to consider how the properties of the enhanced visual feedback is related to control dynamics. Different ways of enhancing sensory feedback are not necessarily equivalent; to successfully promote improved control performance, the novel cues would have to enhance the pre-existing sensory feedback appropriate for the control dynamics. 

Acknowledgments

This work was supported by a grant from the Research Grants Council of Hong Kong (HKU 7471//06H) to L. Li. We thank Joseph Cheng for his assistance in computer programming and data collection, and Ben Backus for his helpful comments on a previous draft of this article. 

Commercial relationships: none. 

Corresponding author: Li Li. 

Email: [email protected]. 

Address: Department of Psychology, The University of Hong Kong, Pokfulam, Hong Kong SAR.