No visual dominance for remembered turns — Psychophysical experiments on the integration of visual and vestibular cues in virtual reality

Abstract

In most virtual reality (VR) applications turns are misperceived, which leads to disorientation. Here we focus on two cues providing no absolute spatial reference: optic flow and vestibular cues. We asked whether: (a) both visual and vestibular information are stored and can be reproduced later; and (b) if those modalities are integrated into one coherent percept or if the memory is modality specific. We used a VR setup including a motion simulator (Stewart platform) and a head-mounted display for presenting vestibular and visual stimuli, respectively. Subjects followed an invisible randomly generated path including heading changes between 8.5 and 17 degrees. Heading deviations from this path were presented as vestibular roll rotation. Hence the path was solely defined by vestibular (and proprioceptive) information. The subjects' task was to continuously adjust the roll axis of the platform to level position. They controlled their heading with a joystick and thereby maintained an upright position. After successfully following a vestibularly defined path twice, subjects were asked to reproduce it from memory. During the reproduction phase, the gain between the joystick control and the resulting visual and vestibular turns were independently varied. Subjects learned and memorized curves of the vestibularly defined virtual path and were able to reproduce the amplitudes of the turns. This demonstrates that vestibular signals can be used for spatial orientation in virtual reality. Since the modality with the bigger gain factor had a dominant effect on the reproduced turns, the integration of visual and vestibular information seems to follow a “max rule”, in which the larger signal is responsible for the perceived and memorized heading change.