Eye-movements aid the control of locomotion

Abstract

Eye-movements have long been considered a problem when trying to understand the visual control of locomotion. They transform the retinal image from a simple expanding pattern of moving texture elements (pure optic flow), into a complex combination of translation and rotation components (retinal flow). In this article we investigate whether there are measurable advantages to having an active free gaze, over a static gaze or tracking gaze, when steering along a winding path. We also examine patterns of free gaze behavior to determine preferred gaze strategies during active locomotion.

Participants were asked to steer along a computer-simulated textured roadway with free gaze, fixed gaze, or gaze tracking the center of the roadway. Deviation of position from the center of the road was recorded along with their point of gaze. It was found that visually tracking the middle of the road produced smaller steering errors than for fixed gaze. Participants performed best at the steering task when allowed to sample naturally from the road ahead with free gaze. There was some variation in the gaze strategies used, but sampling was predominantly of areas proximal to the center of the road. These results diverge from traditional models of flow analysis.

Introduction

Eye-movements are essential for sampling useful information from the world around us, information that feeds directly into the control of actions (Land & Hayhoe, 2001). When driving a car we travel many times faster than the speeds that our visual system evolved to control. With active gaze we can look far ahead to perceive future obstacles and identify potential hazards, then, at a moments notice, switch to looking only a meter or two ahead to concentrate on the negotiation of a tight bend using a high precision maneuver. Nowhere are there greater risks attached to perceptual errors than in high-speed locomotion, appropriate gaze behavior, therefore, is of critical importance. 

Consider the design of a robot to locomote around the world. The simplest solution would be to provide it with a single fixed camera that serves as a cyclopean ‘eye’ positioned in line with the locomotor axis. Using a fixed camera any global transformations of the visual array from frame to frame would be due to motion of the robot. As a result determining the direction of motion relative to the environment would not be difficult. This pattern of moving texture elements from surfaces in the world, as recorded on a fixed image plane, has been called optic flow (Gibson, 1958). If we set our robot off on a straight-line trajectory, with the shutter of the camera left open, it would record a series of radial streaks expanding from a singularity (Figure 1A. Gibson called this point the focus of expansion (FoE) of the optic flow field. With our simple robot traveling in a straight line the current direction of motion is specified by the FoE, so in principle it can be made to steer by ensuring that the FoE always lies in the required direction. This heuristic is both simple and effective, but is insufficient to mimic human locomotion since the human visual system includes mobile eyes. Eye-movements introduce additional motion components into the optic flow field that projects onto the retina. The first level of information available to the actor is therefore retinal flow rather than optic flow. If we rotate our eye, to look at an object on the ground, the real FoE present in optic flow is masked and a new singularity located at the point of fixation. How then can we use retinal flow to steer a course through the world? 

The transformation of retinal flow by eye-movements has been treated as a problem for the control of locomotion, and ways of extracting the original optic flow pattern have been proposed (Longuet-Higgins & Prazdny, 1980; Perrone & Stone, 1994). The majority of studies have considered the case of simple straight-line trajectories, investigating how the retinal flow could be partialled into the component due to the eye rotation and the component due to the linear translation. It has been suggested that this separation could be achieved using either decomposition algorithms based on statistical differences between the two components (Warren & Hannon, 1990) or by subtracting an estimate of the gaze rotation derived from extra-retinal information (Royden, Crowell, & Banks, 1994; Banks, Ehrlich, Backus, & Crowell, 1996). In either case, it is in principle, possible to recover the radial flow pattern (optic flow) containing the FoE (Figure1A) and specifying direction of motion or heading. 

Figure 1

 

Radial dot flow generated from a straight-line path across a ground plane. The direction of motion can be determined by finding the FoE, the point in the flow field where there is no horizontal or vertical motion. This may not be explicitly present, but can be extrapolated from the motion of other points in the retinal image. B) Curvilinear dot flow generated from a curved path across a ground plane, also with a fixed gaze.

There is, however, a complication. In many tasks we are not locomoting in a straight line, but changing heading, steering around obstacles, and following curved paths. This has two implications for controlling steering from retinal flow. Firstly, in curvilinear self-motion, the optic flow pattern generated contains both a translation and rotation component (Figure 1B). Gaze motion adds a further rotation to the retinal flow pattern. It is therefore more difficult to isolate the transformations due to eye-movements, and those due to locomotion since both add global rotation across the retinal flow field. Secondly, even if the separation of locomotion and gaze components is achieved, this is for a particular instant, and the locomotion information may be changing for a complex trajectory. 

In this context, if we require our robot to traverse complex paths, then implementing a mobile gaze system would seem to be creating a number of problems. Processing the images from the camera would become more complex. There may be ambiguous situations where a specific flow pattern could arise from different combinations of eye-movements and locomotion. To solve this ambiguity it may be necessary to monitor movement of the eye, and feed these signals into the visual system, incurring an additional processing load. As the processing load increases this may create greater risk in high speed locomotion that requires a reliable steering reaction. 

If eye-movements do cause a problem for steering control, then one might have expected humans to have evolved with the ability to lock gaze during locomotion. Owls have large eyes, which are highly sensitive to environmental motion, and are fixed in their sockets. They are still able to direct their gaze to different areas of the visual scene thanks to extra vertebrae in their neck that provides them with a wide range of head motion (270°). Some creatures, such as jumping spiders, have a hybrid system that combines fixed eyes that can be used to detect external movements with active eyes for scanning features of interest (Land, 1971). Without eye-movements it is very difficult to direct the most immediately useful and interesting information onto the most sensitive part of the eye, the fovea. Nearly all animals with good vision can move their eyes, and in most cases a mobile gaze is an essential feature of visual data acquisition. It is impossible for humans to maintain gaze fixed in the direction of travel without using a stationary visual target, or paralyzing the extra-ocular muscles to the eyes. 

It has been proposed that the rotation introduced into retinal flow by eye-movements can actually simplify the steering task (Kim & Turvey, 1999; Wann & Swapp, 2000). In this analysis, if you fixate a feature on your intended path then the form and distribution of the retinal flow lines are sufficient to inform you whether your current (curved) trajectory is appropriate, or to indicate significant understeer or oversteer. Wann & Land (2000) proposed, therefore, that the recovery of heading is not necessary for accurate steering as the task can be achieved using active gaze, retinal flow and/or extra-retinal gaze angle information. Wilkie & Wann (2002; 2003) and Wann & Wilkie (In Press), have demonstrated that a robust steering system can be relatively simple: Once a point on the intended path is fixated, a feedback loop (point attractor) acts to reduce the rotation components in either retinal flow or gaze angle information and this will ensure accurate steering (Wilkie & Wann, 2003). In this respect leaving our robot with a fixed camera would mean that we might expect only modest performance in a high speed locomotion context, and might also explain why engineered visual sensors are limited when compared with their biological counterparts. 

In the context of steering a course through the world, it is therefore possible to hypothesize that either fixed gaze or active (free) gaze may be of benefit. A fixed gaze would provide knowledge about instantaneous heading, because retinal flow would be equivalent to optic flow. Active gaze, however, would provide the ability to sample areas of the scene in the most appropriate manner for the locomotor conditions (i.e. speed, visibility and the type of control system) and provide high-resolution information of these areas. Furthermore, it may be that by directing gaze towards points you wish to pass through, you could simplify the retinal and extra-retinal information available for steering (Wann & Wilkie, In Press). 

In this paper we present an experiment where we attempt to determine how gaze influences and informs steering behavior. We specified where participants were allowed to look when steering around a series of bends, and monitored gaze patterns during steering. We compared steering errors arising with fixed gaze or tracking gaze behavior with those from free gaze. Finally we analyzed their steering and gaze behavior in the condition where they were allowed free gaze to see which parts of the scene they found most useful for achieving the steering task. 

Methods

Participants

Six participants took part in the experiment and all had either normal or corrected-to-normal vision. One further participant was initially involved, but due to persistent slips in tracking performance they were excluded from further analysis. All participants held a full driving license, and had driven for at least 7 years. The participants had prior experience of viewing wide field of view optic flow displays, but they were given 16 practice trials with active steering and free gaze before testing began to familiarize themselves with the experimental procedure and the control mapping of the steering device. 

Apparatus

The platform was a PC with Dual Xeon Processors, running tailor-made software with Direct X libraries under Windows 2000. Images were generated at a frame rate of 50 Hz, and projected onto large screen (2 × 1.5 m), 1m from the observer, therefore subtending a total angle of 90° × 73.7°. The projection system was a Hitachi Liquid Crystal Projector. The observers used both eyes to view the non-stereo image (bi-ocular viewing). A ground plane was displayed, which was clipped in software at a distance of 80m from the participant, this meant that the horizon was 1° below the true horizon. 

The participant controlled their direction of motion using a force-feedback steering wheel (Logitech Wingman) mounted in line with their body-axis. Movement of the steering wheel through the full 180° range provided control over direction of motion in the range: −15.75°/s to 15.75°/s. Rotation of the wheel increased the rate of change of heading with a minimum step size of 0.18°/s. The steering wheel supplied data at the same rate as the 50Hz display, therefore the maximum delay between movement of the wheel and screen position update was 2ms. Changes in wheel position were immediately applied to the simulated direction of motion as if rotated on a point with no application of vehicle dynamics such as friction, momentum or wheelbase. 

Head and body position were not stabilized, but an eye tracking system (ASL series 5000) was used to record where the observer was looking on each frame. This system detects the position of the pupil and the corneal reflection of the eye (CR) and uses the relationship between line of gaze and pupil/CR separation (Merchant, Morrissette & Porterfield, 1974). The eye-tracker was calibrated to the projection screen through the use of a regularly spaced 9-point grid. This grid was placed within the central area of the screen that contained the future course of the roadway (58° × 21°). The vertical extent of the calibration area did not need to be large because all the areas of interest were in the bottom half of the screen, below the horizon and the sky. Eye-movements were monitored within this calibration zone without measurable distortion. The resolution of the eye-tracker limits the maximum possible accuracy to ±0.2°. We established that the effective accuracy, using a multiple fixation task, was within ±0.32° of the actual point of gaze. The vertical resolution means that as the observer looks further ahead the distance estimates become less reliable. The practical limit of distinguishing look-ahead distances was therefore 20m ahead of the observer. 

Procedure

Each trial presented the viewer with a visual environment that contained a ground plane textured with gravel and a superimposed curving roadway. The edges of the roadway were 2m apart and generated using a sum-of-sines formula to generate the x and z co-ordinates:  

 

(1)

 

Roadway direction (D_r = 1 or −1, to generate a roadway to the right or left of the current position respectively) was chosen randomly for each trial requiring fast reactions and concentration for accurate performance. 

Observers were translated at a constant speed of 8ms⁻¹(29kph) across the ground, and were asked to steer to ensure they stayed within the roadway, ideally at the center of the road (1m from each edge). This task could be completed using a near road edge-centering strategy, as described in Land & Horwood (1995). We were interested in prospective judgments, rather than an iterative centering strategy, and the latter also would provide an advantage for a free gaze scanning. To prevent this strategy the leading edges of the roadway were cropped 17.5° down from horizon, but the ground plane was left visible. This meant that the participants could see road edges projecting from 4m or 0.5 seconds ahead, toward the horizon, but not closer than 4m. 

There were three viewing conditions: free gaze, fixed gaze and tracking gaze. With free gaze, participants could look wherever they wished in the scene. In the fixed gaze condition participants were required to fixate a static cross, maintaining their gaze on the ground 16m ahead (4° down from the horizon), 2 seconds ahead of their current position. This distance was chosen because pilot data indicated that this was the average forward look-ahead used in a free gaze situation. In the tracking gaze condition, the observer again had to fixate a cross, but this time it was always positioned in the center of the roadway 16m ahead of the participant. Fixation of the cross therefore caused the participant to track the cross from side to side as the road curved in front of them. Unless there was extreme deviation from the roadway (which was not observed) the tracking-cross stayed 4° down from the horizon. 

The three gaze conditions were reordered for each participant and tested in blocks. Each condition began with 12 practice trials before commencing with 16 experimental trials; each trial lasted 10 seconds. The position of the participant in the road was recorded, and the mean deviation from the center of the road was calculated for each frame of each trial. This provided the measure of steering accuracy (in meters) for each condition. 

The gaze of each participant was monitored and to ensure that fixation and tracking restrictions were adhered to we calculated deviations from the fixation point, and gains of tracking performance (See Table 1). All six participants fixated accurately and consistently (one additional participant who did not was excluded). The free gaze condition provided more diverse properties, so these trials were analyzed in terms of distance of gaze from the road center and distance look-ahead. 

Table 1

 

Average deviation of gaze for the fixation condition and horizontal gain during the tracking condition.

Participant	Fixation Deviation (°)	Tracking Gain
CB	0.54	0.98
CH	0.51	1.01
CW	0.46	1.10
HH	0.59	1.08
JB	0.50	0.93
RP	0.36	1.03
MT	0.52	0.78*

 

*MT was excluded for falling outside of the boundary of 0.85.

Results

Steering Behavior

To examine how gaze constraints influenced steering we calculated the deviation of the position for each participant from the (invisible) centerline of the road for each frame of each trial. The root mean squared (RMS) deviation provides an overall measure of precision (Figure 2). Participants managed to keep closer to the centerline when steering with free gaze than when their gaze was fixed (t(5) = 3.62, p < .01), exhibiting deviations approximately 2/3 the size. Tracking the middle of the road did impair performance when compared with free gaze (t(5) = 2.10, p < .05), but improved it relative to the fixed gaze condition (t(5) = 2.66, p < .05). 

Overall bias towards the inside or outside of the bend is captured by the mean constant error (CE: Figure 2). There was no significant bias for the Free and Tracking gaze conditions, however the Fixed gaze condition exhibited some bias towards the outside edge of the road (t(5) = 3.11, p < .05) consistent with insufficient steering before entering each bend. The bias in the fixation condition may therefore reflect steering that lacks appropriate anticipatory judgments of road curvature. 

Figure 2

 

Mean deviation from the centerline of six subjects for each gaze condition: Free gaze, Tracking gaze and Fixation. RMS = Root Mean Squared Error, CE = Constant Error, Error bars = Standard Errors.

As well as examining steering precision and bias, we can also compare the amount of time spent in different zones of the roadway. An average car is about 1.5m wide, so when driving down the center of our roadway there was ±0.25m of leeway to either side before a car wheel would encroach upon the edge of the road. Using this as a guide we created 4 zones from the center of the road outwards, and determined the time spent in each (Figure 3). It is apparent that the majority of the time is spent within the ‘safe’ center zone in the free gaze condition. Significantly less time in the center zone is observed for the fixed gaze condition (t(5) = 4.44, p < .01), and a significantly greater period of time is spent in the Outer or Off zone (t(5) = 3.74, p < .01). The differences between the Free and Tracking conditions were less obvious, suggesting that fixating the center of the road plays an important role in maintaining a safe road position. 

Figure 3

 

The amount of time spent in four zones from the center out to both road edges. The distribution of the road zones is shown on the right. The center (green) and side (yellow) zones are both ±0.25m in size, the outer (orange) zone is ±0.5m, and the off (red) zone is anywhere beyond the road edge.

Gaze Behavior

The results above confirm an advantage for free gaze over gaze fixation in a steering task. We can further explore the use of free gaze in this task and break gaze down into two main components, gaze position relative to the road, and gaze distance in the world. We analyzed the point of gaze of the 6 participants when steering round 32 bends. We took gaze data for the 2 seconds before the apex of each bend and calculated the intersection of the point of gaze with the ground plane. Examples of a single bend for 2 participants are shown in Figure 4 along with animations of the complete trials. It can be seen in these examples that the participants fixate near to the center of the road when entering the bend and gradually their gaze probes ahead to the bend exit as they move towards the apex. 

 

 

Figure 4a

 

Figure 4b

Figure 4

 

Left Column: Example gaze behavior over the 2s before the apex of the first bend (all units on graphs are in meters). A single trial is shown for two participants (HH & CH). Green lines indicate the road edges and invisible midline, and the red circles show points of fixation upon the ground (1 per frame). The blue lines connect sample locations of the participant (black dots) when looking to the main areas of fixation. Right Column: Animations showing locomotion in the left-hand trials. Only the 2s before each bend are shown, since this is the gaze behavior which was analyzed. The animations show point of gaze (red circle) connected by the blue line to each position on the road (black dot). The actual course taken is drawn as a black line over the road.

In a manner comparable to the steering analyses, we categorized the gaze location of each participant in terms of the percentage of time spent looking in five zones, from the inside edge of the road to the outside (See Figure 5). This was calculated from 16m before the apex, up to the point where they passed the apex of each bend. The mean distribution of gaze locations showed that while all zones were sampled, 80% of the time was spent looking at the center, or a zone proximal to the center of the road. One participant used the inside edge of the road to a slightly higher degree (CH: 15%) while another spent a negligible time looking at this feature (CB: 1%). Averaging across the zones we calculated the mean lateral distance of gaze from the center of the road across all trials (Table 2, central column) and this reinforces the findings with regard to the center of the road. 

Figure 5

 

Percentage of time spent looking at different lateral areas of the road when steering round bends. Each point (A–D) represents a zone of ±0.25m centered around this feature and gaze fixations within each zone are binned and the total durations summed before being converted to percentage scores. Data is shown for the six participants, with the final bar being the mean across them.

Table 2

 

Average gaze distance from the center of the road (positive values towards the inside of the bend), and the look ahead time for each participant.

Participant	Gaze Distance from Center (m)	Look Ahead (s)
CB	−0.19	1.07
CH	0.05	2.39
CW	0.07	1.88
HH	0.28	1.31
JB	−0.20	2.44
RP	0.10	1.72
Mean	0.02	1.80
S.D	0.18	0.56

We also examined the distance look-ahead for each participant. Land & Horwood (1995) suggested that there are two types of information to be retrieved: ‘position-in-lane feedback’ and ‘curvature of road’. The former ideally requires a close look-ahead distance to pick up precision information about the change in position of the road edges. In this experiment we removed the edges of the road up to 0.5s (4m) ahead of the current position. This reduced any advantage for the free gaze condition at discerning the current position-in-lane, but information from the roadway edges beyond 4 meters could still be relevant. There should also be an ideal distant look-ahead for picking up the curvature of the road. Land & Lee (1994) suggested that this might be 1–2s ahead of the current position on approaching a bend. 

We categorized the distance look-ahead in terms of the percentage of time spent looking in four zones: Near (0.5–1s), Middle (1–1.5s), Far (1.5–2s), V.Far (2–2.5s). The mean tendency across participants shows that a number of different depths are sampled, with the middle and far distances predominating (Figure 6, right bars). Some individuals, however, did prefer to look closer (e.g. CB), and some further away (e.g. CH). Once again we can average across the zones and calculate the mean temporal look ahead across all trials (Table 2). This suggests that both HH and CB used a close viewing strategy, fixating between 1–1.3s ahead, whereas on average CH and JB looked in the distance over 2.4s ahead. 

Figure 6

 

Percentage of time spent looking at different distances steering round bends. Near (4–8m : 0.5–1s), Middle (8–12m : 1–1.5s), Far (12–16m : 1.5–2s), V.Far (16 – 20m : 2–2.5s). Data is shown for the six participants, and also the mean across participants.

Discussion

The advantage that could arise from fixed gaze is that it greatly simplifies the extraction of heading information. The flow field projected on the retina reflects the locomotor trajectory, uncontaminated by gaze motion. For a curved path this should enable the perception of locomotor flow lines (Lee & Lishman, 1977) and in principle, matching of these flow lines to the projected edges of the road would be sufficient for accurate steering. The fixed gaze condition, however, did not result in accurate steering, even though we ensured that central vision was maintained close to the same areas sampled in the other conditions (16m or 2s ahead). This questions the utility of the locomotor flow line hypothesis and it also sheds some doubt on the use of an estimate of instantaneous heading. Wann & Land (2002) proposed that fixation of a point on your future path was an important feature of accurate steering. This would predict a significant improvement in the tracking condition, which imposes the Wann & Land strategy, as compared to fixed gaze. This was confirmed across all 6 participants for precision of steering errors and the time-in-zone measures. Land & Lee (1994) presented an earlier alternative proposal, that the tangent point, or apex of the bend, is a particularly useful feature when steering. The tangent point can provide information on the curvature of the road ahead. If participants habitually used this feature then we might have expected a significant improvement in the free gaze condition, over the tracking condition, as participants sought out the tangent point. There was some partial support for this in the RMS scores (Figure 2), but average performance displayed similar positional distributions for both conditions (Figure 3). Examination of gaze behavior when free to sample naturally revealed that a great deal more time was spent looking towards the center of the road than at the inside edge (Figure 5). 

The future path (road center) was not rigidly fixated, but on average 30% of the time was spent looking at the center zone, and another 50% looking either side of the center. In general we did not observe close gaze tracking of the road edges and no individual participant adhered closely to the tangent point strategy. It seems that if there is an advantage of maintaining gaze at the tangent point then the participants in this study did not find it. One critical difference may be that the Land & Lee (1994) study was completed at higher speeds, on real roads with blind bends. Our task was less threatening and did allow unlimited look-ahead. A further important difference between the two hypotheses is that the tangent point strategy provides a solution for maintaining a trajectory a fixed distance from the inside edge (lane following), whereas the path-fixation strategy allows any curved path to be chosen, including a ‘racing line’ of cutting the corner. Land & Tatler (2001) observed that on a well-practiced course a racing car driver did not look exactly at the tangent point, so it may be necessary to examine gaze behavior on an unpredictable course in order to resolve which strategy is used, and when. 

Irrespective of the precise strategy used in free gaze, we have illustrated that active gaze is essential for optimal sampling of relevant visual information, since it is under these conditions that the most accurate control of steering is observed. There were some individual differences in the patterns of fixation, but the general sampling strategy was to look ahead by 1–2.5s and to fixate a point close to the desired future path (Table 2). These temporal estimates are commensurate with Land & Lee (1994) but we differ with respect to the road features which are fixated. Each of these fixations would induce gaze rotation and introduce an additional rotation component into the retinal flow field. In previous investigations of the perception of heading this gaze rotation component would be seen as a problem or confound that required compensation (Lappe, Bremmer, & van den Berg, 1999). In contrast to this Wann & Wilkie (In Press) propose that the cascading fixations are the primary planning mechanism that enables path selection and the completion of complex trajectories of varying curvature. Fixation of points on the path can also enable the use of retinal flow to judge steering, without recovering optic flow or heading (Kim & Turvey, 1999; Wann & Swapp, 2000). 

Conclusions

Fixating an object eccentric to the current trajectory has been seen as a problem for locomotor control and so it was considered necessary to compensate for the gaze rotation. Although fixation of irrelevant features to the side of the road may result in errors, we found that fixation of the future path results in more effective steering even though the fixation introduces a rotation into the retinal flow field. Steering was consistently poor with fixed gaze, despite this condition ensuring that retinal flow was equivalent to optic flow. When participants were free to direct gaze, there were individual differences, but all participants chose to use eccentric fixation of the road some 1–2.5s ahead. If this distinct gaze pattern was disrupted then the accuracy of steering was impaired. In the imposed tracking task, steering was less accurate than with a free gaze, however similar ‘safe’ road positions were maintained in both cases. We conclude that active gaze is essential for effective locomotion through the world. Fixating points on an upcoming bend provides a mechanism for locomotor control, rather than a problem that requires compensation. This forward fixation strategy also provides a mechanism for route selection and learning. 

Acknowledgments

This research was supported by a grant from the United Kingdom Engineering and Physical Sciences Research Council (EPSRC). Commercial relationships: none.