Slant cues are processed with different latencies for the online control of movement

Abstract

For the online control of movement, it is important to respond fast. The extent to which cues are effective in guiding our actions might therefore depend on how quickly they provide new information. We compared the latency to alter a movement when monocular and binocular cues indicated that the surface slant had changed. We found that subjects adjusted their movement in response to three types of information: information about the new slant from the monocular image, information about the new slant from binocular disparity, and information about the change in slant from the change in the monocular image. Responses to changes in the monocular image were approximately 40 ms faster than responses to a new slant estimate from binocular disparity and about 90 ms faster than responses to a new slant estimate from the monocular image. Considering these delays, adjustments of ongoing movements to changes in slant will usually be initiated by changes in the monocular image. The response will later be refined on the basis of combined binocular and monocular estimates of slant.

Introduction

When we want to place an object on a surface, we need to estimate the surface's slant to make sure that the object has about the same orientation as the surface before making contact. Information about this orientation is available from binocular disparity and from the monocular images. The information in the monocular images includes cues such as the shape of the surface's projection on the retina, changes in texture density across the retina, and motion parallax. Different slant cues are likely to be processed at different rates and so may provide information about changes at different latencies. Previous research suggested that differences in latency are ignored, so that cues with shorter latencies influence the combined estimate earlier (van Mierlo, Brenner, & Smeets, 2007). 

One way to examine how people use visual information to guide their action is by examining how they respond to perturbations of such information during their movement (Brenner & Smeets, 1997; Goodale, Pelisson, & Prablanc, 1986; Saunders & Knill, 2003; Veerman, Brenner, & Smeets, 2008). Different studies have reported different latency differences between monocular and binocular cues. In a study in which subjects had to respond to perturbations in surface slant (Greenwald, Knill, & Saunders, 2005), slant estimates based on binocular disparity appeared to influence corrections earlier than slant estimates based on monocular cues, so the authors concluded that binocular disparity was processed more quickly. This finding is surprising because Allison and Howard (2000) found that perceived slant shifted from being dominated by perspective to being dominated by disparity as exposure time to a test stimulus increased. Moreover, Brenner and Smeets (2006) found that subjects corrected movements faster in response to a jump in target depth when the jump was visible as a change in the height in the visual field than when the jump was only visible as a change in binocular disparity. 

Whereas Allison and Howard's (2000) and Brenner and Smeets' (2006) findings suggest that monocular cues are processed more quickly for estimating slant and distance, Greenwald et al.'s (2005) findings suggest that binocular disparity is processed more quickly for estimating slant changes. The reason for this discrepancy is not clear because the three studies differed considerably in various aspects. For example, Greenwald et al. showed alternating white and black frames for 167 ms before presenting the changed slant in order to mask the slant change. Allison and Howard and Brenner and Smeets did not mask the perturbations. Furthermore, in Greenwald et al.'s study, subjects moved a real object so that the visual information matched the proprioceptive information. In Brenner and Smeets' study, the visual position did not match the position that was felt, since subjects moved a cursor with a mouse. Moreover, in Brenner and Smeets' study, the perturbation was a change in position, whereas in Greenwald et al.'s it was a change in slant. Such differences make it impossible to tell which aspect is responsible for the different conclusions as to whether binocular information is processed faster or more slowly than monocular information. 

In this study, we investigated whether latency differences between responses to changes in binocular disparity and changes in the monocular image are visible in the online control of movement. As in Greenwald et al. (2005), subjects placed a cylinder on a surface of which the slant could change right after movement onset. Either the binocular disparities or the monocular images or both could indicate the change in slant. We determined how subjects altered the orientation of their hand in response to such a slant change. We blanked the screen before the slant changed on half of the trials to determine whether seeing the change allows one to respond faster. On such trials subjects could respond to the new slant but not to the transient. 

Method

Subjects

Eight subjects (three male, five female) participated in the experiment. Five subjects were naïve with respect to the purpose of the experiment. All subjects had normal or corrected-to-normal vision, with a stereo acuity below 1 arcmin (tested with Randot™ plates). 

Apparatus

Subjects sat behind a surface (45 cm by 45 cm) that was centered 60 cm in front of the midpoint of their body and 40 cm below their eye-level. This surface could be rotated around a transversal axis with the help of a computer-controlled motor (see Figure 1). They held a cylinder with a height of 6 cm and a diameter of 9.5 cm in their right hand. At the subjects' right side, 26 cm from their midsagittal plane, 60 cm in front of them, and 18 cm below eye-level, there was a second surface with a 2-mm deep indentation in the shape of the base of the cylinder. Subjects had to place the cylinder within this indentation at the start of each trial. 

Figure 1

 

Schematic representation of the set-up (not to scale).

Subjects did not see the real surface, starting position or cylinder. They saw a virtual surface, starting position, and cylinder. The three-dimensional virtual environment was created by presenting different images to the left and right eyes using a combination of two CRTs and mirrors (see Figure 1). The mirrors were semi-silvered with occluders attached behind them. We matched the virtual and real environments by removing the occluders and monocularly aligning the corners of a rectangle on the screen (as reflected by the mirror) with the 3D positions of four markers on a calibration rectangle that was placed above the real surface (as seen through the mirror) for that purpose. Using a 3D virtual environment enabled us to dynamically and independently manipulate the virtual surface's slant as specified by binocular disparity and by the monocular images. 

Throughout the experiment we recorded the 3D positions of three Infra-red Emitting diodes (IREDs) that were attached to the cylinder using an Optotrak 3020 system (Northern Digital, Inc.), so that we could generate images of the cylinder while the subject was moving it. Since motion parallax as a result of small head movements has been found to contribute to slant perception (Louw, Smeets, & Brenner, 2007), we determined the positions of the eyes relative to a bite-board before the experiment and recorded the position of the bite-board using the Optotrak system during the experiment. The bite-board was not attached to anything, so subjects were free to move their head. We continually adjusted the images to the positions of the subject's eyes so that the slant indicated by motion parallax was consistent with the slant indicated by static information from the monocular images. Note that this refers to the 3D position of the subjects' eyes in space. The direction of gaze was neither monitored nor instructed. 

The positions of the IREDs were sampled with a frequency of 250 Hz. Based on the coordinates of the IREDs on the cylinder and bite-board, a PC calculated the current position of the cylinder and eyes and sent these coordinates to two Apple G5's that each rendered an image of the cylinder and virtual surface for one of the eyes on a CRT monitor (1096 × 686 pixels, 47.3 by 30.0 cm). The new images were created with the frequency of the refresh rates of the two CRT monitors (160 Hz). Thus, we generated images that were appropriate for the actual position of the eyes and hand at each moment in time. The delay between a cylinder movement and the visual feedback was about 20 ms. 

Stimuli

A small pink virtual sphere indicated the starting position. The virtual surface was a square with sides of 10 cm and was visible as a red and gray checkerboard of 4 by 4 squares. The virtual object was a cylinder with 14 white and black stripes and a green top and bottom. It had the same dimensions as the real cylinder. The shapes of the projections of the squares of the checkerboard on the screens provided monocular information about surface slant. The differences between the computer images for the two eyes provided binocular information about surface slant (binocular disparity). Both information sources were also available for the cylinder ( Figure 2). 

Figure 2

 

Impression of the subject's view during the experiment, with from left to right the virtual surface, cylinder and starting point.

Procedure

Each trial began by the computer positioning the real surface in the orientation that the virtual surface would have at the end of the trial. This happened in 3 to 4 movements to prevent subjects from deducing the final orientation on the basis of the sound from the motor that positioned the real surface. Subsequently, the virtual surface and the virtual cylinder were presented. The virtual surface was slanted by 5° from the horizontal plane (positive angles indicate that the side of the virtual surface furthest from the subject's body is higher than the side nearest to the subject's body). Subjects had to move the cylinder to the starting position and, after hearing a beep, to accurately place the cylinder on the virtual surface. The beep was presented 500–800 ms after subjects placed the cylinder at the starting position. We regarded the moment that the cylinder had traveled 20 mm from the start position (in any direction) the moment the subject reacted. If subjects reacted before the beep or within 100 ms after presentation of the beep, the movement was considered to have started too early. The trial was then stopped and presented again. 

When subjects reacted, the slant of the virtual surface could change to either −5° or +15°. The binocular disparities could change at the same time as the monocular images (as they normally would) or only one of the two cues could change at that moment. If only one cue changed, the other did so 150 ms later to ensure that both the binocular disparities and the monocular images indicated the same slant at the end of the trial. This final slant was always consistent with the slant of the real surface that subjects felt at contact. When the two cues were in conflict for 150 ms, one cue always still indicated +5° while the other indicated a changed slant (either −5° or +15°). 

On half of the trials, the surface disappeared at the onset of the beep and only reappeared again at the moment of the first slant change, which meant that no surface was visible for about 620 ms. As a result, subject could not see the change on these trials (i.e., they could not see the transient for the first slant change). Figure 3 summarizes the 14 conditions. Each condition was presented at least 25 times to each subject. 

Figure 3

 

The six pairs of conditions in which the slant changed (conditions 1–6) and the two conditions in which it did not (conditions 7 and 8). The virtual surface always had a slant of 5° at the beginning of the trial. Black lines represent the slant indicated by binocular disparity. Red lines represent the slant indicated by the monocular images. Just after movement onset, the slant of the virtual surface could change by −10° (a) or +10° (b). This change was either in the monocular information (pairs 1 and 4), the binocular information (pairs 2 and 5), or both (pairs 3 and 6). If only one changed, the second changed 150 ms later. On half of the trials, the image was blanked from the auditory ‘go’ signal to the first slant change (conditions 4–6 and 8; gap not drawn to scale).

Data analysis

We used the orientation of the base of the cylinder in relation to the horizontal plane as our measure of cylinder orientation. This orientation was determined from the position data provided by the three IREDs on the cylinder. When one or more of these IREDs were missing, we interpolated their positions from the positions on the frames before and after the frame with the missing markers. We rejected a trial if it had more than 10 frames with missing markers in the period after the first slant change and before making contact with the surface, or if the movement time was more than 1.5 s. 

We determined the cylinder's angular velocity by fitting a 2nd order polynomial to the orientations of the cylinder during a 5-frame period centered on each frame and determining the derivative of this polynomial at that frame (Biegstraaten, Smeets, & Brenner, 2003). We synchronized the trials at the moment of the (first) slant change. For each subject, condition, and frame after the slant change, we then averaged the angular velocities and determined the corresponding standard errors. 

Since we found no indication that the sign of the slant change influences the latency of responses to a cue (see Figure 4), we analyzed the difference in angular velocity between conditions that have the same timing of the slant changes but a different direction of slant change (e.g., conditions 1a and 1b in Figure 3) rather than comparing each with the unperturbed conditions (7 and 8). We determined the onset of the response to the first slant change in three steps. First we searched from the moment of the slant change to find the first frame in which the angular velocities for the two directions of slant change differed by more than 2 standard errors (in this difference). Then we determined the maximal difference in angular velocity during the subsequent 100 ms and searched back from the frame at which this maximum occurred to find the frames at which the difference was 25% and 75% of the maximum. We considered the intersection of the line through these points with a line representing zero velocity difference to be the onset of the response (Veerman et al., 2008). We compared the onsets in the six conditions using a repeated-measures ANOVA and Fisher's PLSD (α = 0.05). 

Figure 4

 

Mean velocity of the cylinder's rotation. Time zero represents the moment that the first cue changed (except in conditions 7 and 8 where it is the time that it would have changed). In condition pairs 1 and 4, the monocular images changed first. In condition pairs 2 and 5, binocular disparity changed first. In each pair of conditions, the amplitude of the change was either −10° (a) or +10° (b). The shading indicates the standard error between subjects.

Although subjects obviously also responded to the second slant change in conditions in which a second change occurred 150 ms later than the first, we did not analyze such responses. Such an analysis would be quite complicated because by then subjects are already responding to the first change. The purpose of the second change was only to ensure that the feedback at the end of the trial felt correct. The 150-ms interval was long enough to ensure that we did not inadvertently consider responses to the second change to be late responses to the first. 

Results

On average, 56 of the 545 trials per subject were rejected because they contained too many frames with missing markers or because the movement time exceeded 1.5 s. Individual subjects' mean reaction times varied between 490 and 900 ms. Mean movement times, the time from reaction to the moment the cylinder made contact with the surface, ranged from 500 to 900 ms. 

Figure 4 shows the mean angular velocity of the cylinder over time. Zero on the time (horizontal) axis represents the moment that the slant (first) changed. The data confirm that responses in the two directions are similar (see the symmetry with respect to the unperturbed conditions 7 and 8). There are clear differences between the onsets and shapes of the responses in the different conditions. To get a better view of these differences, we determined the difference between the mean responses in the pairs of conditions that only differ in the sign of the slant change ( Figure 5). When the transient is present, subjects respond faster to the changes in the monocular images than to the change in binocular disparity. When there is no transient, subjects respond faster to a change in binocular disparity. When the monocular images and the binocular disparity change together, subjects responded as fast as they did to the fastest cue. 

Figure 5

 

Mean difference in angular velocity between conditions that have the same timing but a different sign of the slant change. The shading indicates the standard error between subjects.

When we compare the responses in which the transient was present with responses in which it was not ( Figure 6), we see that responses to changes in disparity are not affected by removing the transient, but responses to changes in the monocular images are. The responses to changes in the monocular images also appear to initially be weaker when the transient was present, but we cannot be sure of this because more variability in latency could also account for the less abrupt change in the average angular velocity. 

Figure 6

 

Comparing conditions in which the (first) slant change was visible (transient) and conditions in which it was not (no transient). Other details as in Figure 5.

Even when responding to the monocular images without the transients (condition 4), subjects responded fast enough to be sure that they were not responding to the change in binocular disparity that occurred 150 ms later. It took subjects about 200 ms to respond to a change in binocular disparity. The responses to a change in the monocular images when the screen had been blanked occurred well before 200 ms after the second change (350 ms after the first change). Thus, subjects responded to the changes in the monocular images as well as to the changed monocular images. 

These findings show that subjects can use (at least) three types of information to adjust ongoing movements. In order of increasing latency: the changes in the monocular images, the new information about slant from binocular disparity and the new information about slant from the monocular images. 

To investigate whether the differences that we see in Figures 5 and 6 are consistent across subjects, we determined the onsets of the responses for each subject ( Table 1). The averages of these response latencies are shown in Figure 7. The data in Table 1 and Figure 7 suggest that subjects can respond to three sources of information: the changes in the monocular images (condition pairs 1 and 3), the new slant from binocular disparity (2, 5, and 6), and the new slant from the monocular images (4). The ANOVA on the onsets of the responses revealed a significant main effect of Condition ( p < 0.01). Fisher PLSD tests confirmed the division into the three groups indicated by the rectangles in Figure 7. The onsets never differed significantly between members of the same group and always differed significantly between members of different groups. 

Table 1

 

Estimated latency for each pair of conditions (see Figure 3) for each subject (in seconds).

Subject	Pair
	1	2	3	4	5	6
1	0.15	0.15	0.14	0.26	0.21	0.21
2	0.17	0.21	0.16	0.20	0.20	0.17
3	0.17	0.20	0.18	0.20	0.22	0.19
4	0.15	0.20	0.20	0.29	0.30	0.19
5	0.19	0.23	0.16	0.30	0.24	0.22
6	0.18	0.21	0.17	0.30	0.16	0.18
7	0.15	0.20	0.17	0.23	0.18	0.19
8	0.17	0.23	0.17	0.27	0.22	0.22

Figure 7

 

Average onset of the response for each pair of conditions (as listed in Figure 3). The blue dotted rectangle indicates responses based on changes in the monocular images. The blue solid rectangle indicates responses based on a new estimate of slant from the monocular images. The red rectangle indicates responses based on a new estimate of slant from binocular disparity.

Discussion

Subjects alter the orientation of their hand to match the changing surface slant during their movement. When an object changes orientation its projection on the retina changes. In our virtual environment, the retinal projections of the squares of the checkerboard surface become more trapezoidal when the far side of the checkerboard moves downwards, because the lateral images sizes and separations decrease for the parts of the surface that move further away and increase for those that move closer. Our data suggest that subjects respond to these changes in each eye's image as well as to a changed image in each of the eyes. 

Response latency depends on saliency (Veerman et al., 2008). However, blanking the screen did not simply increase the latency by reducing the salience of all changes in slant because responses to changes in binocular disparity were not affected by blanking the screen (see Figure 6). Thus, blanking the screen specifically influenced responses to changes in the monocular image. 

Subjects responded 40 ms faster to changes in the monocular images than to new information about slant from binocular disparity and 90 ms faster to changes in the monocular images than to a new slant from the monocular images. When more than one source of information changed at the same time, the latency of the response was that of the fastest source. 

For a change in binocular disparity, the onset and the slope of subjects' responses were almost identical in the conditions with and without a visible transient, indicating that in both conditions subjects responded to the same kind of information. This suggests that subjects do not respond directly to the changes in binocular disparity, but only to the new disparities. This not necessarily mean that such changes are not detected because they may simply be detected with a longer latency than the disparities themselves. 

The latency differences that we find can explain the differences between the studies mentioned in the Introduction. The jumps in distance in Brenner and Smeets' study (2006) and the temporal modulation and step changes in Allison and Howard's study (2000) were clearly visible, so subjects could readily respond to changes in the monocular images. The perturbations in Greenwald et al.'s study (2005) were masked by a set of frames alternating between black and white for 167 ms, so responses were probably based on a changed estimate of slant. We found that responses to a new slant from binocular disparity were faster than responses to a new slant from the monocular images when the transient was not visible, but that responses to changes in the monocular images were even faster (when they were visible). 

In our study we intentionally chose natural conditions, such as an approximately horizontal surface in front of and below the subjects' eyes. We compared changes of the same magnitude for all cues. Under such conditions we find that a change in the monocular images is processed faster than a new binocular slant and that a new binocular slant is processed faster than a new slant from the monocular images. Perhaps conditions can be found in which the order of the latencies is different, but we believe that our findings are representative of many natural circumstances. 

Responses to changes in the monocular images appeared not only to be quicker, but also stronger than responses to a new slant (the slope of the pink curve versus the slope of the blue curve in the left plot of Figure 6). A possible explanation for this is that people respond to the motion in the retinal image by initiating a strong response in the direction of the change (before knowing when and where the change will end), whereas a new slant only initiates a response that is proportional to the change in slant (and the weight given to the source of information involved). In other words, subjects may initially match the way they rotate their hand to the motion of the surface and then adjust the end orientation of the ongoing response to the new slant estimates. This combination of responding to a derivative and to a combined slant estimate would nicely integrate optimal accuracy with fast responses. However, this poses a challenge to optimal cue combination theory, because the changes in the monocular images are temporary signals, so they cannot simply be averaged with other information to get a better estimate of slant. Thus, whenever the observer is moving relative to a surface, the whole dynamics of the interaction will have to be considered to predict how slant information is used. 

Acknowledgments

This study was conducted at the Human Movement Sciences Department of the VU Amsterdam, The Netherlands. This research was supported by The Netherlands Organisation for Scientific Research (NWO; MaGW Grant 452-02-007). 

Commercial relationships: none. 

Corresponding author: Christa M. van Mierlo. 

Email: [e-mail] or [e-mail]. 

Address: Faculteit der Bewegingswetenschappen/Faculty of Human Movement Sciences, Vrije Universiteit, Van der Boechorststraat 9, 1081 BT Amsterdam, The Netherlands.