The influence of retinal and extra-retinal motion cues on perceived object motion during self-motion

Abstract

Eye, head, and body movement are intimately linked. During self-motion, the eyes track objects by a combination of vestibular reflexes and smooth pursuit eye movements but although the world appears stable during saccadic gaze changes, it does not appear stable during physical self-motion. We determined the amount by which a fixated object needed to be moved in space in order to appear earth stationary to a linearly moving observer. Observers were oscillated sinusoidally either passively or under their own control, under lit and fully darkened conditions. The visual targets always needed to move (in space) in the same direction as the observer to be judged as earth stationary. Targets needed to be moved more in order to be judged as earth stationary when movement was in the dark, rather than in the light, and also when movement was passive rather than when it was active. Efference copy motor signals, visual movement, and non-visual cues all contribute significantly and approximately additively to the estimate of self-motion. Errors in perceived self-motion can produce subsequent illusory visual motion.

Introduction

During linear self-motion, a tracked earth-stable object appears to move not only relative to the observer but also in space. Although there is relative motion between all objects in the visual field during linear motion, it might be expected that these retinal motions would be both expected and compatible with the perception (held cognitively) that the observer is moving through a world of stable objects. 

The illusion of object motion during self-motion is related to two complementary illusions found when tracking a genuinely moving object in front of a stationary background. In this case, the object is perceived as moving slower than it really is (the Aubert–Fleischl phenomenon) while the background appears to be displaced in the opposite direction to the object's motion (the Filehne illusion); two illusory movements which have been documented for around a hundred years (Aubert, 1886; Filehne, 1922; Fleischl, 1882). The Aubert–Fleischl illusion is usually attributed to a failure in the compensatory eye movement system such that the eye movements involved, while physically accurate, may be taken into account centrally with a gain of only 70% (Bridgeman, 1989, 1995; Bridgeman & Graziano, 1989). The fact that the Filehne motion is in the opposite direction to linear self-motion suggests that too little motion is “subtracted” from the relative shift of the background on the retina, resulting in the residual motion being interpreted as a background displacement. 

Aubert–Fleischl- and Filehne-like effects are found not just as a result of eye-in-head movements but associated with displacement of the entire body (Freides, 1974; Harris et al., 2002; Jaekl et al., 2002; Jaekl, Jenkin, & Harris, 2005). In this case, the illusory motion is likely to be compounded by errors in non-retinal information concerning the observer's physical movement. Non-retinal information about linear self-motion includes the signals from the vestibular utricles concerning linear accelerations (for a review, see Benson 1982) and somatosensory cues from kinesthetic receptors in the musculature (Gianna, Heimbrand, & Gresty, 1996) and viscera (Mittelstaedt, 1997). If self-motion is under active control, then an efferent copy of the muscle commands may also generate a rapidly accessible signal concerning the intended movement (see, e.g., Helmholtz, 1866) and this anticipatory information can be combined with the slower returning afferents to estimate the actual motions undertaken more accurately (Von Holst & Mittelstaedt, 1950). 

Failure to correctly synthesize retinal and extra-retinal cues to the eye's displacement over time will inevitably result in failure to correctly parse the resultant retinal motion into that proportion corresponding to self-motion and the proportion originating from object motion (Wertheim, 1994) with consequent detrimental effects on the accuracy of perceived object motion. This introduces the near paradoxical situation whereby the correct compensation for apparent environmental motion is dependent upon the correct calculation of self-motion, and yet self-motion estimation itself relies, in part, upon the shift in the visual scene consequent on it—a situation described as “complementarity” (Wertheim, 1994). 

Two of the potential contributors to accurately judging the motion of an object during self-motion are the motor commands associated with active motion and the optic flow patterns induced by the self-motion itself. Integrating these cues would allow for a more accurate estimate of ego-displacement over time, and thus assist in parsing the retinal slip resulting from the motion of a target object from general motion of the ambient scene. Mesland and Wertheim (1995) reported a consistent underestimation of passive self-motion inferred through determining the thresholds for motion perception of visual stimuli presented in the periphery. They reported an underestimation of perceived ego velocity in a fully lit environment; an underestimation which increased when ambient vision was removed by performing the experiment in the dark. Wexler (2003) reported a similar underestimation for motion in the depth-plane for active motion; with this underestimate increasing when the subject was moved passively. We chose to employ a similar method to those of Wexler, but using a target whose motion was co-planar to the observer's motion. We also sought to combine the light/dark condition manipulation of Mesland and Wertheim (1995) with the active/passive condition manipulation of Wexler (2003). 

Using this method and design, the close interdependence between object and self-motion perception can be used as a probe for determining the accuracy with which self-motion itself was being calculated. In this way, the observer's task was orthogonal to the probe's objective: a perceptual measure of visual movement that taps the accuracy of self-motion estimation. We wished to determine the relative contribution of retinal and non-retinal cues through testing percepts during active and passive motion, as well as the contribution of optic flow by performing the task in the light and dark. We also sought to minimize retinal motion of the object as far as possible, by tracking the object throughout the testing. 

Method

Subjects

Ten observers (7 male, 3 female) between the ages of 22 and 40 took part in the experiment. Each had normal or corrected-to-normal vision and normal stereo acuity and reported no history of vestibular deficits. All aspects of the experiment conformed to the Ethical Guidelines specified by York University which conform to the Treaty of Helsinki, 1964. 

Equipment

Movement platform

A hoverbed was constructed by attaching four downward-pointing air-bearings at the corners of a platform mounted on a guide rail ( Figure 1A). The platform was padded to allow observers to lie comfortably on their backs looking straight up. The observer's head was held by a fixed, recessed, cushioned support that acted as a semi-rigid support and head restraint. The observer and bed could thus be moved as a single structure. The hoverbed was fitted with a motion sensor which sampled the position of the bed at 50 Hz with a resolution of 0.02 cm. Observers used their left hand to operate the buttons of a computer mouse which acted as their response device. With their right hand they could pull themselves along a support rail in the “active movement” condition ( Figure 1A). 

Figure 1

 

The experimental set up. (A) The observers lay on a platform mounted on air-bearings and steered by a guide rail with their head firmly supported. The motion of the platform was tracked. A support rail allowed the observers to pull themselves backward and forward (see text). A sheet of Plexiglas was mounted above the observer on which a disc was reflected so as to appear 130 cm in front of the observer and 73 cm in front of the background (the ceiling of the room). (B) The platform was attached at each end by springs and could be shifted either by the observer (as shown in A) or, via a cable, by the experimenter (as shown in B). The disc was moved on the screen by a variable proportion of the distance needed for it to be truly earth stationary. The disc could either remain stationary (and be reflected so as to appear at point “a” with gain = 1) or move smoothly “with” the observer's motion (toward point b: gain <1) or against the observer's motion (toward point c: gain >1).

The hoverbed was attached to anchor points at the head and foot by heavy-duty springs ( Figure 1B). These springs held the hoverbed in a central position about which it could be oscillated. For the active motion condition, observers propelled themselves “up and down” (from their perspective) at approximately 0.5 Hz with an amplitude of approximately ±10 cm pulling themselves along with their right hand by means of a handle fixed to the floor parallel to the guide rail. Subjects synchronized their movement to a metronome. For the passive motion condition, they were moved sinusoidally at the same frequency and amplitude by the experimenter using a cable attached to the hoverbed ( Figure 1B). 

Movement kinematics across conditions

In an attempt to match the movement kinematics across active and passive conditions, we chose not to employ a motorized mechanism for the passive motion condition. Instead we employed an analogue of the hand-propelled motion of the active condition with a “hand-pulled” motion involving the experimenter applying a manual force to a cable. This was an attempt to mimic the small inconsistencies with which each observer was likely to perform each active oscillation. This “noise matching” method, although likely to introduce variable error across trials, seems unlikely to introduce constant errors, although such a possibility cannot be discounted. 

With the motion constrained along a rigid guide rail, the two most likely sources of motion inconstancy across trials are in the domain of frequency and amplitude. Frequency was controlled through the motion being synchronized with an auditory cue provided by a metronome. In the passive conditions, the accuracy of this synchronization was monitored by the experimenter. The amplitude was maintained by reference to conspicuous markers on the cable and on the floor underneath the hoverbed, which allowed the distance of travel to be constantly monitored. The amplitude of motion in the active/dark condition was not actively monitored and remains a potential source of both variable and constant error. 

Duration of stimuli

The observer was free to perform as many oscillations as necessary to make their decision concerning the target's motion. As such the duration of any given trial was at the observer's discretion. This procedure was applied in order to ensure that judgments close to the point of subjective equality in this novel and unusual task were as accurate as possible. 

Visual display

Suspended directly above and parallel to the hoverbed at a height of 48 cm was a thin sheet of Plexiglas. The sheet was tilted around its long axis to form a partially reflecting but largely transparent surface ( Figure 1A). Reflected in the sheet was a computer screen positioned beside the observer. In the lit condition, the textured ceiling tiles of the laboratory in which the observers were lying could be seen clearly through, and on each side of, the Plexiglas (width 42°) at a distance of 203 cm. The computer display showed a white ellipse which, when reflected, was foreshortened to appear to the observer as a 0.44° disc at a distance of 130 cm (i.e., 73 cm in front of the background). The remainder of the computer screen and the surrounding floor were dark-masked so that no other reflected images were visible. The output from the motion sensor attached to the hoverbed was taken as input by a computer program, which shifted the ellipse linearly by a known proportion of the hoverbed's motion with a negligible delay. The ratio of the linear motion of the stimulus to that of the hoverbed could be manipulated as an experimental variable. The disc appeared to move in a plane parallel to the observer's motion and along their body centre line as they rode “up and down” (from their perspective) on the hoverbed. Apart from the background ceiling, there were no other proximal objects with which the motion of the disc could be compared. 

Motion gain

The movement of the dot can be described relative to the observer's motion. The dot's motion is expressed as the ratio of its movement (relative to the observer) compared to the amount it would have to move (relative to the observer) if it actually remained earth stationary. This ratio was called the “motion gain”. As such a motion gain of one corresponds to the disc being earth stationary. A motion gain of zero corresponds to the disc being observer stationary, i.e., matching the observer's motion. A gain of greater than one corresponds to the target moving in the opposite direction to the observer's motion, with an increasing gain describing the disc moving against the observer's motion with increasing amplitude. A gain of less than one corresponds to the disc moving in the same direction as the observer, with a decreasing gain describing the disc moving increasingly closer to the observer's amplitude of motion. 

General procedure

Observers performed four separate experimental sessions: “light” and “dark” room conditions and “active” and “passive” motion conditions. The order of these sessions was counterbalanced across the group. 

Observers fixated the virtual disc and judged the motion of the tracked disc relative to their own motion. They were asked to decide “Is the stimulus moving in space in the same direction as your own motion or in the opposite direction?” while moving sinusoidally and fixating the disc. Observers could perform as many oscillations as necessary in order to reach their decision concerning the disc's relative motion. The point of subjective equality (PSE) between judging the disc's motion as “with” or “against” observer motion was taken as the gain at which the disc was perceived as being stationary in space. 

We used the method of constant stimuli. For trials conducted in the light, the stimulus moved with a gain in the range of between 0.7 and 1.12 at intervals of 0.03. In the dark condition, the range was of gains was between 0.22 and 0.85 at intervals of 0.045. In both conditions, there were 15 stimulus gains. Each stimulus gain was presented 10 times in a randomized sequence of 150 trials per condition. A different range and increment of gains was necessary in the light and dark conditions to ensure that the range of presented stimuli effectively bracketed the group's responses—pilot data having indicated a significant change in the required range and centroid between lit and darkened conditions. 

For each observer, the proportion of times they judged the disc as having “motion with” their own motion was plotted as a function of the gain and fitted with a cumulative Gaussian:  

$y=\frac{100}{1+e^{-\left(\frac{x-x_o}{b}\right)}}\%,$

(1)

where b is the standard deviation and x ₀ is the gain corresponding to the point of subjective equality (PSE). Examples of the resulting curves are shown in Figure 2. 

Figure 2

 

Psychometric functions for each of the 10 participants showing the proportion of times target motion was judged as being “with” (in the same direction as) their motion, as a function of the gain of the target's motion (actual target movement/geometrically correct target movement). The 0.5 point was taken as the point of perceived stationarity. Conditions were either in the light (green and red curves) or dark (blue and grey curves), and motion was either active (top panel) or passive (bottom panel) motion. The broad curves on each plot indicate the group “average” for each condition.

Results

The curves for all subjects and all conditions are presented in Figure 2. Under all four conditions (light and dark, active, and passive motion), the gain at which the target was judged as stationary (the PSE) was statistically less than the geometrically correct value of unity. A series of Bonferroni-corrected paired t-test (with alpha level of .0125, two-tailed) were conducted comparing the gain values in each condition against unity. The results are summarized in Table 1. Each comparison showed that the gain at which perceived stationarity occurred was consistently different from (and less than) unity. In other words, when truly earth stationary (with a gain equal to 1), the target was consistently judged as moving in the “opposite” direction to the observer's motion. This result is summarized in Figure 3. 

Table 1

 

Summary of group mean and standard deviations for the gain at which target stationarity was judged in the light and dark conditions, while moving actively or being moved passively. Student t values and the corresponding p values indicate the probability of these gains differing from unity by chance.

Lighting condition	Motion condition	Gains for stationarity	t values	p values
Light	Active	0.91 ± 0.06	3.96	.003
Light	Passive	0.87 ± 0.04	9.42	<.001
Dark	Active	0.54 ± 0.09	14.69	<.001
Dark	Passive	0.42 ± 0.13	16.57	<.001

Figure 3

 

The mean gains at which the target was regarded as earth stationary for active and passive motion in the light and dark obtained by the method of constant stimuli. The horizontal dashed lines represent the gain values which would be required for the target to be earth stationary (top line) at gain = 1, visually stationary (centre line, i.e., zero parallax between the target and background) at gain = .36, and observer stationary (lower line) at gain = 0. The vertical gray bars are the range of gain values tested using the method of constant stimuli in the light and dark conditions.

The gains at which the target was judged as being stationary were compared across lighting conditions (light versus dark) and motion conditions (active versus passive) in a 2 × 2 repeated measures ANOVA. This revealed a reliable main effect of lighting with the motion gain at which target stationarity was perceived being significantly higher in the lit condition when compared to the dark room condition: F(1,9) = 122.74; p < .001. There was also a significant effect of motion condition, with object stationarity being judged at a reliably higher gain in the active condition compared to the passive motion: F(1,9) = 76.85; p < .001. 

There was also a reliable interaction between lighting and motion conditions ( F(1,9) = 9.54; p < .05) with the difference between the active and passive conditions in the lit condition (mean difference = .037; SE = .012) being reliably smaller than the corresponding difference in the dark condition (mean difference = 0.119; SE = 0019). 

Discussion

This study has shown that, paradoxically, in order to appear stable a target needs to move consistently in the same direction as the moving observer. The amount of required target motion is contingent upon whether the observer is in the light or dark (with the lit condition requiring less target movement) and, to a lesser extent, whether the observer is moving passively or actively (passive observer motion requiring more target movement than active motion for the target to be judged as stationary). Access to the efferent and afferent signals associated with active motion appear to contribute to judging whether objects are earth-stationary or not. 

Comparison with other studies

Overall our results broadly match those of similar studies in this area. For example, Mesland and Wertheim (1995) measured perceived self-motion using judgments of object motion. They determined upper and lower motion detection thresholds for observers seated upright and moved passively laterally on a motorized sledge from which they inferred the observers' perceived ego velocity (PEV). Under fully lit conditions, the PEV was slightly less than the actual motion; on average .95 of the true value. In darkened conditions the PEV shifted significantly to a gain of .83. In an experimental paradigm very similar to ours (Wexler, 2003) had observers judge “motion with the observer vs. motion against the observer” in an upright posture. The task was performed through the depth plane with observers moving toward and away from the visual target in dark-room conditions. Wexler (2003) included three conditions that were relevant to ours: active motion through thrusting the head and neck forward and backward in the depth plane; a similar passive motion in the same plane (through the observer being propelled in a wheelchair); “mismatched” motion where, although under the observer's active control, motion was effected through the observer propelling the wheels of a wheelchair in which the observer was seated. Wexler (2003) reports subjective stationarity at an equivalent gain of .62 for active motion and .43 for passive motion, which compares well with our findings of perceived stationarity gains of .54 and .42 respectively. In the “mismatched” motion condition, however, stationarity is perceived with a motion gain of only .48, i.e., there was more than a 50% underestimation of self-motion despite access to the motor commands associated with this active motion. It appears that this form of active locomotion provided no additional correcting signal over that available in Wexler's passive condition. This contrasts with our novel form of “active” locomotion where there was a reliable “corrective” influence of active motion. This may be as a result of the degree to which the effector of motion (the hand and arm in both cases) was “mismatched” to the resulting full-body motion. In our case the propelling arm motion was in the same plane and directly linked to the body motion. Wexler's “mismatched” motion condition involved the hands and arms moving along a circular path defined by the wheelchair's pushing rim. This suggests that the ability to integrate the efferent and afferent signals may be dependent on the concordance between the precise kinematics of the propelling motor act and the resulting motion. 

Effect of perceived target distance

One contributing explanation for the movement required to obtain a perceptually stationary target is that the distance of the target might have been misjudged. If self-motion were perfectly perceived, then it could be that the expected target motion was still incorrect because of errors in distance perception (Gogel, 1977, 1981; Hay & Sawyer, 1969; Shebilske & Proffitt, 1981). Disparity (Westheimer & McKee, 1978), accommodation (Fisher & Sanchez, 1997), and vergence (Ritter, 1977) all contribute toward the estimation of the disc's distance. Parallax motion is also a strong contributor to depth judgment (e.g., Braunstein & Andersen, 1981; Helmholtz, 1909/1962); however, our experimental paradigm specifically excluded parallax as a reliable depth cue. Misjudging the target's distance as being further away than it was could lead observers to expect it to displace less for any given self-motion. 

However, for the perceptually earth stationary gains illustrated in Figure 4A to be solely the result of mis-perceived target distance would require that the object was perceived at a depth significantly further than its actual distance. For the lit/passive condition, the target would have to have been misperceived at 1.2 times further than its true distance (i.e., at 156 cm rather than 130 cm) and at 2.4 times further (at 312 cm) in the dark/passive condition. Although depth misperception may be a contributory factor, we suggest that it is not likely to be the major cause of the observed biases. Depth overestimation cannot be invoked to explain the differences in bias between active and passive motion since distance perception should be the same in these two conditions. Instead we propose that it is the availability and integration of retinal and non-retinal cues to self-motion which best explains our results. 

Figure 4

 

The potential effect of (A) misperceived distance and (B) misperceived self-motion on the gain (actual motion/earth-stationary motion relative to subject) of target motion perceived as earth stationary by a moving observer. (A) As the distance of a target is misjudged as further than it really is (distance gain: perceived/actual distance), the predicted stimulus motion gain required for the stimulus to appear stationary is in the same direction (“with”) as the observer's movement and of a progressively smaller amplitude. (B) As the perceptual gain of self-motion (perceived/actual self-motion) increases beyond 1, the target motion required for perceived stability requires “motion against”. The horizontal lines on the graphs are the gains at which passive motion in the light (top line) and dark (bottom line) are regarded as earth stationary.

Potential effects of supine posture

Our observers performed all judgments along the body's long axis while in a supine orientation. This motion would stimulate the sacculus (Fernández & Goldberg, 1976). We cannot discount the possibility that this novel posture and the unusual form of locomotion we used could have influenced our results. Supine and upright postures may be equivalent in terms of motion perception. Loose, Probst, and Wist (1996) reported that backward tilts (at 45 degrees and 90 degrees) did not alter the threshold for motion perception in comparison with thresholds in an upright posture. Also the close correspondence between our results and those of studies involving upright observers in similar paradigms (e.g., Mesland & Wertheim, 1995; Wexler, 2003) seems to suggest some commonality of results across differing postures. Caution is advised however before generalizing our results to the perceptual consequences of all forms of locomotion. 

Effect of perceived distance of travel

Underestimation of the eye's motion in space has been suggested as the explanation for both the Filehne and Aubert–Fleischl phenomena, the existing accounts of which include assuming accurate coding of retinal motion (Wertheim, 1981, 1987) as well as allowing for errors in sensing retinal slip (Freeman & Banks, 1998). In the present and more general case, the eyes move both as part of the whole body movement as well as in their orbit. The movement with which the eyes track the disc is a combination of smooth pursuit and translational vestibulo-ocular reflex and has a gain considerably less than one (Liao, Walker, Joshi, Reschke, & Leigh, 2008). It is possible that if the object's movement were estimated from knowledge about how far the eyes had moved, this could provide a possible mechanism for the observations and errors we report. Direct sensory knowledge of how much the eyes have moved is however not available during vestibular driven compensatory eye movements (Bedell, Klopfenstein, & Yuan, 1989). The brain largely monitors such eye movements via an efference copy of the vestibular driving signal rather than feed forward proprioceptive input from eye muscles (Bridgeman & Stark, 1991). The gain of the VOR for up/down translations is largely unaffected by whether the room lights are on or not (Liao et al., 2008), further separating our perceptual observations from the associated eye movements. 

If the magnitude of self-motion were misestimated, this would affect the gain of object motion at which it appears earth stationary. If observers were to completely fail to register any self-motion, they would required a visual target to exactly match their own actual (now undetected) motion, i.e., the target would need to be “observer stationary” (moving with a gain of zero) for it to be perceived as earth stationary. If observers were to underestimate their motion by 50%, then a perceptually earth-stationary target would have to move with a gain of .5. Our results suggest that there is a progressive underestimation of self-motion as retinal and non-retinal cues are excluded. Removing the afferent and efferent signals associated with active self-motion results in more underestimation of self-motion. However, the greatest effect results from the removal of retinally sourced information concerning self-motion: the absence of optic flow resulted in self-motion being registered at only 52% of it's true value even when active motion cues are present. 

There has been some controversy as to whether self-motion is over or underestimated when motion cues are restricted. Studies using a comparison with remembered target distances suggest an overestimation of perceived movement (Berthoz, Israël, Georges-François, Grasso, & Tsuzuku, 1995; Harris, Jenkin, & Zikovitz, 2000; Israël, Chapuis, Glasauer, Charade, & Berthoz, 1993; Medendorp, Tweed, & Crawford, 2003; Redlick, Jenkin, & Harris, 2001) whereas tasks involving accumulating a distance estimate from the start position suggest an underestimation (Frenz & Lappe, 2005). These two conflicting observations have recently been resolved in a single model (Lappe, Jenkin, & Harris, 2007): the “leaky spatial integrator” where self-motion is calculated through aggregated estimates of displacement alone (i.e., in space and not over time), with these summations being consistent underestimations. This model accounts for biases in self-motion estimation depending on whether the dependent measure is the distance traveled from a starting point (“counting up” resulting in a distance over-estimation) or the remaining distance from an end point (“counting down” resulting in underestimation). In studies where observers must continuously monitor their movement throughout their judgments of motion for objects constantly in view (which is the case in our paradigm, as well as those of Mesland and Wertheim (1995) and Wexler (2003)) there is consistent evidence for self-motion underestimation, with this error increasing in magnitude as self-motion cues are excluded. As such, if Lappe and colleagues' (2007) account is correct, it would appear that impoverishing visual, somatosensory, motor and vestibular cues to self-motion results in an increased “leakage” of the integrator processes involved in calculating ego-displacement—even when the distance of travel is revisited continuously through sustained sinusoidal motion. This suggests that the integration process requires continuous input from an array of multi-sensory sources for displacement estimation to “plug” the integrator's loss of displacement accumulation: no one modality is sufficient by itself to achieve accurate self-motion estimates. 

Conclusions

Our findings show that the perception of body motion is a multi-sensory process, involving information from many aspects of visual and non-visual processing. Because these processes seem to be essentially additive, inadequate information available to any one of the contributing senses can cause the illusion that an earth-stationary object seems to move when viewed by a moving observer. This illusory movement, although prevalent in everyday life, appears to go almost unnoticed. Under laboratory conditions it provides a readily accessible vehicle for assessing the contribution of each of the sensory cues in resolving the complex shifts of the visual world into a single and coherent visual percept. 

Acknowledgments

Supported by the NASA Cooperative Agreement NCC9-58 with the National Space Biomedical Research Institute; the Canadian Space Agency; and grants from the Natural Sciences and Engineering Research Council of Canada to L.R. Harris. 

Commercial relationships: none. 

Corresponding author: Richard T. Dyde. 

Email: [e-mail]. 

Address: Centre for Vision Research, University of York, Toronto, Ontario, Canada M3J 1P3.