Impaired stationarity perception is associated with increased virtual reality sickness

Abstract

Stationarity perception refers to the ability to accurately perceive the surrounding visual environment as world-fixed during self-motion. Perception of stationarity depends on mechanisms that evaluate the congruence between retinal/oculomotor signals and head movement signals. In a series of psychophysical experiments, we systematically varied the congruence between retinal/oculomotor and head movement signals to find the range of visual gains that is compatible with perception of a stationary environment. On each trial, human subjects wearing a head-mounted display execute a yaw head movement and report whether the visual gain was perceived to be too slow or fast. A psychometric fit to the data across trials reveals the visual gain most compatible with stationarity (a measure of accuracy) and the sensitivity to visual gain manipulation (a measure of precision). Across experiments, we varied 1) the spatial frequency of the visual stimulus, 2) the retinal location of the visual stimulus (central vs. peripheral), and 3) fixation behavior (scene-fixed vs. head-fixed). Stationarity perception is most precise and accurate during scene-fixed fixation. Effects of spatial frequency and retinal stimulus location become evident during head-fixed fixation, when retinal image motion is increased. Virtual Reality sickness assessed using the Simulator Sickness Questionnaire covaries with perceptual performance. Decreased accuracy is associated with an increase in the nausea subscore, while decreased precision is associated with an increase in the oculomotor and disorientation subscores.

Introduction

Movement of the eye relative to the stationary environment gives rise to full-field visual motion at the retina. Under most circumstances, this full-field motion is not perceived as movement of the environment in space, but is instead accurately perceived as movement of the eye, head, and/or body relative to the stationary environment. This perception of the environment as stationary, referred to herein as stationarity perception (Wertheim, 1994), depends on neural mechanisms that evaluate the agreement between self-motion estimated from visual versus nonvisual sources. 

The current study investigates sensory and motor factors that influence the precision and accuracy of visual and nonvisual self-motion estimates and thus the precision and accuracy of stationarity perception. This topic is important to investigate because failure of stationarity perception leads to the debilitating perception that the environment is moving independently. Perhaps the most significant adverse consequence of the failure of stationarity perception is that retinal image motion becomes an unreliable cue to self-motion. In the absence of reliable visual self-motion cues, estimation of self-motion can be highly impaired. Thus, false perception of environmental motion is generally debilitating, and is associated with clinical impairments including vertigo, as well as inappropriate oculomotor and postural responses. 

Failure of stationarity perception can occur naturally as a consequence of vestibular, oculomotor, and/or neurological disorders (Haarmeier, Thier, Repnow, & Petersen, 1997; Brandt, 2003; Bronstein, 2004). It can also occur in more unnatural situations, such as during vehicular travel (Bertolini & Straumann, 2016) or when viewing artificial visual displays (for example, virtual reality) (LaViola, 2000; Wilmott, Erkelens, Murdison, & Rio, 2022). Regardless of the cause, it is known that these situations can give rise to sickness, presenting with symptoms including nausea, sweating, and pallor, known variously as motion sickness, virtual reality sickness, cybersickness, or simulator sickness (Kennedy, Lane, Berbaum, & Lilienthal, 1993; LaViola, 2000; Bertolini & Straumann, 2016; Chang, Kim, & Yoo, 2020; Saredakis et al., 2020). Persistent visual-vestibular conflict is thought to be the main cause of such sickness (Reason & Brand, 1975; Reason, 1978), and this sickness has been hypothesized to be functionally adaptive, leading the organism to expel any ingested poisons that may be contributing to the mismatch (Treisman, 1977; Oman, 2012), for example, opioids (Lehnen et al., 2015), and adopt slower and more careful movements, thereby minimizing the risk of injury due to postural instability and allowing time for recalibration. 

The focus of this study is to better understand how stimulus factors impact the precision and accuracy of stationarity perception, and further how this precision and accuracy is related to sickness. Despite the hypothesized relationships between visual-vestibular conflict, stationarity perception, and sickness, no studies to date have demonstrated an association between psychophysical measures of stationarity perception and incidence of sickness. In addition, more research is needed to better understand how various sensory and motor factors impact both stationarity perception and sickness. 

To address these questions, the current study uses a psychophysical task that was introduced over 50 years ago (Wallach & Kravitz, 1965; Wallach, 1987) in which observers experience a head turn while a gain is applied to full-field visual motion. Through repeated presentations it is possible to measure both the precision and accuracy of stationarity perception and how it varies across conditions. Although the original paradigm was implemented using analog methods, more recent studies (Jaekl, Jenkin, & Harris, 2005; MacNeilage, 2007; Correia Grácio, Bos, Van Paassen, & Mulder, 2013; Garzorz & MacNeilage, 2017; Moroz, Garzorz, Folmer, & MacNeilage, 2019), including this one, present visual stimuli in virtual reality. 

The conditions investigated in the current study include the spatial frequency and retinal stimulus location (central vs. peripheral) of the visual stimulus. We hypothesize that spatial frequency should impact stationarity perception because it is known to impact perception of the speed of two-dimensional pattern motion (Campbell & Maffei, 1981; Ferrera & Wilson, 1991), the speed of reflexive eye movements (Waddington & Harris, 2015), as well as estimation of self-motion (Diener, Wist, Dichgans, & Brandt, 1976; Schor & Narayan, 1981; Palmisano & Gillam, 1998; Guo, Nakamura, Fujii, Seno, & Palmisano, 2021), with higher spatial frequency patterns generally leading to faster speed estimates. Thus, in the current experiment, we expect higher spatial frequency should lead to reduced visual gains perceived as stable. We hypothesize that retinal eccentricity should impact stationarity perception because motion processing is known to vary with retinal eccentricity (McKee & Nakayama, 1984) and eccentricity of the motion stimulus has been shown to impact perception of self-motion (Brandt, Dichgans, & Koenig, 1973; Palmisano & Gillam, 1998). Some studies report that self-motion perception is reduced with only central visual stimulation (Brandt et al., 1973). Others report that eccentricity is not a factor (Post, 1988), or that the effect of eccentricity depends on spatial frequency (Palmisano & Gillam, 1998). On balance, we hypothesize that visual self-motion estimates will be slower for central stimulation, leading to an increase in gains perceived as stable relative to peripheral stimulation. We also investigate the role of eye movements because these mediate retinal image motion (for example, through nulling of retinal slip in the case of vestibulo-ocular reflex (VOR)) and modulate efference copy signals. In line with prior findings (Garzorz & MacNeilage, 2017; Halow, Liu, Folmer, & MacNeilage, 2023), we expect that gain perceived as stable will be increased during scene-fixed compared to head-fixed fixation. 

Assuming that stationarity perception relies on mechanisms that compare visual to nonvisual self-motion signals, the present psychophysical task constitutes a useful tool for assessing how visual stimulus factors impact self-motion estimates. In the past, the impact of visual stimulus factors on self-motion processing has been assessed psychophysically using measures of vection (for example, Brandt et al., 1973; Kennedy, Hettinger, Harm, Ordy, & Dunlap, 1996; Palmisano & Gillam, 1998; Webb & Griffin, 2003; Guo et al., 2021), but interpretation of vection ratings can be problematic because of the likelihood of intersubject differences in the mapping between the perception and the reported rating. Instead, the current task allows assessing the effect of visual stimulus factors such as spatial frequency and eccentricity using a common measuring stick, namely the concurrent nonvisual (for example, vestibular) signals. 

More concretely, stationarity perception depends on a comparison of the visual and nonvisual self-motion estimates, which can be modeled simply as cross-modal discrimination (Garzorz & MacNeilage, 2017). This model predicts that when visual and nonvisual estimates are roughly equal to one another, the environment will be perceived as stationary; inequality will lead to failure of stationarity and the perception that the environment is drifting with or against the direction of head movement. Variability of stationarity judgments is equal to the sum of the variabilities on visual and nonvisual estimates. Thus, changes in gain perceived as stationary (i.e., accuracy) across conditions can be interpreted as changes in magnitude of the visual relative to the nonvisual self-motion estimate. Changes in variability (i.e., precision) of stationarity judgments can be interpreted as changes in noise on the visual (or nonvisual) self-motion estimate. This model is likely to be overly simplistic, but it allows interpreting changes in accuracy and precision of stationarity perception in a straightforward manner. 

At the same time we measure sickness using standard methods including the Simulator Sickness Questionnaire (SSQ) (Kennedy et al., 1993) and Discomfort Scores (Rebenitsch & Owen, 2014; Fernandes & Feiner, 2016). With the measures collected here, we can assess if there is a systematic relationship between performance on our stationarity perception task and measures of sickness. As we address in the discussion, assessing this linkage is crucial, as it may be a gateway to better understanding why sickness appears more often in certain situations, why certain sickness mitigation strategies are effective, and even lead to additional mitigation strategies, such as psychophysical training. 

Methods

Subjects

Twenty subjects (9 female, 11 male) were recruited for the study. One female subject was unable to complete the study due to severe sickness. These data were excluded from the analysis. The analyzed data includes 19 subjects, (8 female, 11 male) with a median age of 26 years (range, 20-47 years). All subjects reported normal or corrected-to-normal vision and no vestibular deficits. All subjects participated in both experiments. Experimental protocols were approved by the Institutional Review Board of the University of Nevada, Reno, and written informed consent was obtained from each subject before participation. 

Experimental setup

Visual stimuli were displayed using an HTC Vive Pro Eye head-mounted display (HMD), with a diagonal field of view of 110 degrees, a refresh rate of 90 Hz and a combined resolution of 2,880 × 1,600 pixels. Before each session, subjects would perform the Vive Pro Eye eye tracking calibration routine. In addition to performing an eye tracking calibration, this also measures the subject’s interpupillary distance and instructs the user to set the lenses in the headset to a position which is appropriate for their interpupillary distance. We used two base stations for head tracking, set diagonally from each other and approximately 8 ft from the HMD. The HMD was run using a Dell XPS 8930 computer with Intel Core i7-8700 CPU, 16 GB Ram, GeForce GTX 1070. Responses were recorded by subjects pressing the left or right arrow key on a standard Logitech keyboard. The virtual environment was programmed in Unity v2019.4.17f1. The visual scene was an Optokinetic drum (10-m diameter) with vertical black and white stripes with stripe width varying depending on condition (Figure 1, right). The fixation point was positioned 0.5 m in front of the subject and was 0.25 × 0.25 m. This fixation point would either stay world-fixed (i.e., move with the moving environment) during the scene-fixed fixation condition, or head-fixed (i.e., move with the subject’s head motion) during the head-fixed fixation condition (see fixation behavior condition descriptions below). The central/peripheral field of view mask used in this experiment followed the same behavior. 

Figure 1.

 

(Left) Conflict detection task. Subjects align a fixation point with a point in the center of the scene. This triggers a white circle to appear to the left or right of center, indicating the direction of rotation. Subjects make a yaw head rotation while maintaining fixation on the point. The trial ends, and subjects are prompted to respond, “With or Against?” Subjects respond, and the process repeats for 300 trials. (Right) Virtual Environment for both experiments. Subjects were seated in the center of an optokinetic drum in VR.

Procedure

For each of the conditions subjects performed the same task. The procedure for a single trial was as follows (also depicted in Figure 1). 

• 1. Subjects aligned a point at the center of the display (head-fixed point) with a fixation point located in the center of the visual scene (scene-fixed point). This was to ensure that subjects began each trial in the same orientation.
• 2. A head rotation cue in the form of a white dot appeared 15° to the left or right of the fixation point
• 3. After 0.3 seconds, the cue disappeared and the optokinetic drum environment was displayed
• 4. While maintaining fixation on either the head-fixed or scene-fixed point (depending on condition), subjects rotated their head left or right approximately 15° over approximately 1 second to face the head rotation cue.
• 5. When the rotation was completed, the visual environment disappeared and a response prompt appeared asking “With or Against?”.
• 6. The subject then responded using the arrows on a standard computer keyboard, by pressing the left arrow key to respond “With,” or the right arrow key to respond, “Against.” The response indicated whether the subject perceived the visual environment to be drifting with (visual gain too low) or against (visual gain too high) the direction of their head movement in the world reference frame.
• 7. The trial was over and the head-fixed and scene-fixed points appeared cuing the beginning of the next trial.

For this experiment, we wanted to keep head movements fairly consistent across trials and subjects. To do so, we placed constraints on the head movements subjects were allowed to make. Subjects received feedback on each trial if their head movement was too fast, too slow, too short, or too long. Acceptable head movements must last between 0.5 and 1.5 seconds, have a distance between 15° and 50°, and peak speed between 10 and 40°/s. If a subject made an “incorrect” head movement, the trial was discarded, and subjects completed a new trial. 

Trials were run in blocks of 300 trials, each block constituting one condition of a given experiment. Each block was collected on an independent visit on independent days to avoid carry-over effects of sickness. At the beginning of each block, subjects completed up to 20 practice trials to familiarize themselves with the type of head movements they needed to make and the particular demands of each condition. Subjects had the option to take a 30-second break after 100 and 200 trials. 

Across the 300 trials, visual gain was modified according to a staircase procedure. There were 2 interleaved staircases in each block, with 150 trials per staircase. Gain of 1 indicates visual scene motion that matched the tracked physical head motion exactly. A gain greater than 1 indicates that the visual scene moves faster than the head motion. This causes the scene to move “against,” or in the opposite direction of, head movement in the world reference frame. A gain of less than 1 causes the visual scene to move slower than the head motion. The scene would thus appear to move “with,” or in the same direction as, head motion in the world reference frame. One staircase began at a gain value of 1.3, and the other began at a gain value of 0.7 (Figure 3). Gain of the first staircase was modified using a 3-down-1-up rule (Leek, 2001), meaning that gain was reduced after three consecutive “against” responses, and increased after one ’with’ response. This rule converges to a gain value that elicits approximately 80% “against” responses (Leek, 2001). Gain of the second staircase was modified using a 3-up-1-down rule, meaning that gain was increased after three consecutive “with” responses, and decreased after one “against” response. This rule converges to a gain value that elicits approximately 20% “against” responses (Leek, 2001). Gains were increased/decreased respectively by 0.04 units of Log_e(Gain), until the second reversal in the staircase, at which point the gains were increased/decreased by 0.02 units. A reversal refers to a change in direction of the staircase. This procedure allows the staircase to quickly converge and concentrates sampling in a way that efficiently constrains the fit of the psychometric function. (Leek, 2001). From trial to trial, the selection of staircase, as well as the direction of head movement, was randomized. 

Before and after each block, subjects were asked to respond to the SSQ (Kennedy et al., 1993). Each question of the SSQ was presented to subjects in the VR environment, and they would use the keyboard to respond. Additionally, subjects were asked to rate their discomfort every 3 minutes on a scale of 1 to 10 (Rebenitsch & Owen, 2014; Fernandes & Feiner, 2016). Because the time per block was not constrained, the number of discomfort scores collected per block can vary. At the end of the condition, subjects were asked to give one final rating for their discomfort, which we referred to as the “end discomfort score.” 

Conditions: Experiment 1

In the first experiment, subjects completed four conditions following a 2 × 2 design with fixation behavior and spatial frequency of the visual stimulus manipulated across conditions (Figure 2, left). Fixation behavior was manipulated by instructing subjects to fixate either a head-fixed fixation point that remained centered on the display of the HMD, thereby reducing eye-in-head movement (Figure 2, top), or a scene-fixed fixation point that remained fixed relative to the rendered virtual environment, which required subjects to rotate the eyes counter to head movement to maintain fixation (Figure 2, bottom). To reiterate, while the head-fixed point moves relative to the scene in order to maintain its central position relative to the subject’s head, the scene-fixed target is anchored to the visual scene. This means that it had the same gain applied to it as the rest of the visual scene. Spatial frequency of the environment was manipulated by changing the width of the stripes in the virtual optokinetic drum (Figure 1) to yield patterns that are repeated at either 2 (high frequency) or 0.2 (low frequency) cycles per degree of visual angle. The order of presentation of these conditions was randomized for each subject. 

Figure 2.

 

Experimental designs. Each icon depicts a top-down view of the subject performing a head rotation. The wire-frame rectangle and framed stimuli represent the starting and ending positions of the HMD viewport, respectively. Red-dashed lines indicate fixation direction. (Left) Experiment 1. 2 × 2 design with factor fixation (head-fixed, scene-fixed) and spatial frequency (low - 0.2 cpd, high - 2 cpd). (Right) Experiment 2. 2 × 2 design with factors fixation (head-fixed, scene-fixed) and retinal stimulus location (central, peripheral). The central region subtended 40° of visual angle.

Conditions: Experiment 2

In the second experiment, four additional conditions were collected to satisfy a 2 × 2 design with fixation behavior and retinal stimulus location of the visual stimulus manipulated across conditions (Figure 2, right). For these conditions, only the high spatial frequency environment was used. In the central conditions, a peripheral mask was applied around the fixation point with an aperture of 40° diameter such that only the central portion of the visual stimulus was visible. In the peripheral conditions, a central mask was applied around the fixation point, again with a diameter of 40°, such that only the peripheral portion of the visual stimulus was visible. The decision to use a 40° aperture was based on a prior study (Bower, Bian, & Andersen, 2012) which found differences in discrimination thresholds at 0° and 40° eccentricity. The central and peripheral conditions were collected in random order. Low frequency stimuli were dropped in the second experiment due to time constraints on the number of conditions we could run. 

Analysis

To assess the influence of fixation and stimulus factors on stationarity perception, data from each subject and condition were fit with a separate cumulative Gaussian psychometric function using the Palamedes Toolbox (Prins & Kingdom, 2009). Specifically, we used the PAL_PFML_Fit function with lapse and guess rate parameters set to zero, with only the mean and the standard deviation of the cumulative Gaussian left as free parameters. For this analysis, we modeled the probability of response (with vs. against) as a function of the natural logarithm of the gain factor (Log_e(gain)) because gain is a ratio. Post-hoc assessment of goodness of fit (using the PAL_PFML_GoodnessOfFit function) confirmed that deviance (a measure of residual error) was significantly less when log rather than linear gain values were used (two-tailed paired t-test; t(151) = 2.09, p = 0.04). 

The parameters of the cumulative Gaussian fit are our dependent measures of interest (Figure 3). The mean is the point of subjective equality (PSE), and it indicates the log gain value that is expected to yield 50% with versus against responses. This is the single gain value that is most likely to be perceived as stationary, and it provides a measure of accuracy. Log gain values close to zero indicate accurate perception of stationarity (Log_e(1) = 0). The standard deviation of the cumulative Gaussian fit is the just-noticeable difference (JND). This is an estimate of the change in log gain relative to the PSE that will be just noticeable, i.e., the increase/decrease in value that leads to 84%/16% against responses. The JND provides a measure of precision or noise on stationarity judgments. 

For completeness, mean head and eye movement trajectories for each subject and condition are available for review in the Supplementary Materials. Only head yaw and horizontal eye traces are shown. Head and eye traces were resampled to 90 frames per second. Because these movements were generated by the subjects, there is variability in the duration and length of the trajectories. Keep in mind when reviewing these average trajectories that data at the end of each trajectory is less representative of the average movement because there are fewer samples. These trajectories suggest that subjects were generally able to maintain proper fixation during the scene-fixed fixation condition. However, during the head-fixed fixation condition several subjects failed to maintain perfect fixation. Data from these subjects was nevertheless included in all analyses. 

VR sickness as measured by the SSQ (Kennedy et al., 1993) was calculated for each subject and condition by subtracting the initial baseline response from the final response. Finally, separate subscores for nausea, oculomotor, disorientation, and total score were calculated as prescribed (Kennedy et al., 1993). Our methodology differs slightly from that of the original SSQ paper in that we use the difference between postexposure and baseline scores as our SSQ score, rather than simply the postexposure scores. We chose to do so following recent findings in the literature which have suggested that the assumption of a “zero” baseline score for subjects is “erroneous” (Brown, Spronck, & Powell, 2022) and that difference scores help eliminate reports of symptoms which existed prior to exposure (Bimberg, Weissker, & Kulik, 2020). Bar graphs of our final scores can be found in the Supplementary Materials. 

To quantify the association between stationarity perception and VR sickness, we used a linear mixed-effects model (LMM) design. LMMs have become increasingly popular in recent years due to their ability to handle unbalanced data sets and nonindependent data, such as repeated measures (Pinheiro & Bates, 2000; Oberg & Mahoney, 2007). These models can be used to predict variables using treatment effects and their interactions. Additionally, they provide a way to account for variation among samples (such as the variation arising among subjects) via a random effects structure (Oberg & Mahoney, 2007). To understand the impact of our experimental manipulations as well as a subject’s PSE or JND values on VR sickness reports, we fit a model which utilizes PSE, JND, our experimental manipulations and the interactions between them as our fixed effects and subject as our random effect. We used the R packages lme4 and lmertest (Bates, Mächler, Bolker, & Walker, 2015; Kuznetsova, Brockhoff, & Christensen, 2017) to run the analysis for this model. 

Conditional effects were not included as random effects because this prevented models from converging or led to singular fits, which should be avoided when fitting these models (Barr, Levy, Scheepers, & Tily, 2013). Additionally, we have no reason to suspect that these conditional effects would vary by subject. Histograms of the residuals appeared to indicate skewed distributions. This was not unexpected, given that few people report sickness but those that do often have very high scores, LMMs have been shown to remain robust despite violations of distributional assumptions (Schielzeth et al., 2020). For this reason, SSQ response data was not transformed prior to modeling. 

The relationship between VR sickness and the fixed effects of PSEs, JNDs, fixation, frequency, and RSL was modeled using the combined data from both experiments. Data from both experiments were combined for the LMM because sickness can be a subtle effect (Kennedy et al., 1996), meaning that many subjects experience few or no symptoms. We therefore combined the data sets to increase statistical power. It is also worth noting that combining datasets simplifies the analysis by allowing us to use 6 models rather than 12. The p values for the fixed effects of each model were obtained using the Satterthwaite method, which was chosen due its tendency to be more conservative on smaller datasets (Luke, 2017). 

Results

Experiment 1

The first experiment investigated the influence of fixation behavior (head-fixed vs scene-fixed) and spatial frequency (low vs. high) on psychophysical judgments of stationarity (Figure 2, left). The accuracy of stationarity judgments is captured by the mean of the cumulative Gaussian psychometric fit, also known as the PSE. These data are shown in Figure 4, left. A PSE of Log_e(Gain) equal to zero indicates accurate stationarity perception. 

Figure 3.

 

Example psychometric function and staircase from one subject and condition. The graph on the left shows the psychometric fit generated from the subject’s with/against response data. The PSE value for the fit is shown as a black dot, and the JND is shown as the red line extending to the 84% correct interval. The figure on the right shows the staircase from the same recording. On the 3Down1Up staircase (shown in blue), if the subject responds “against” three times, the gain is reduced. If the subject responds, “with” while a trial from this staircase is being presented, the gain is increased. The inverse of these rules are followed for the 1Down3Up staircase (shown in orange).

Figure 4.

 

Accuracy of stationarity perception - points of subjective equality (PSEs). Experiment 1 (left), Experiment 2 (right). Note that high-frequency full-field data from Experiment 1 are replotted alongside results of experiment 2 (translucent points) to facilitate comparison. The PSE is the mean of the cumulative Gaussian psychometric fit to data from each subject and condition. Log_e(Gain) of zero indicates that speed of visual motion was equal to head motion. Negative and positive values indicate that slower and faster speeds were required, respectively. Means of each condition are shown as a red circle, and individual subject PSEs are shown as points in grey. The blue lines indicate the standard deviation of the PSE values, extending one standard deviation above and below the mean. The left hand scale of each graph represents the natural log of the gain. The right hand scale of each graph is the gain in linear units.

In the scene-fixed conditions, PSEs were near zero, consistent with accurate stationarity perception. However, in the head-fixed conditions, PSEs were significantly decreased, F(1,19) = 33.8, p < 0.001 (Table 1) relative to the scene-fixed condition, meaning that slower visual speeds (i.e., reduced visual gains) were needed in order to perceive the visual environment as stationary. 

Table 1.

 

Results from the repeated measures ANOVA, the effects of fixation, spatial frequency and retinal stimulus location on accuracy (PSEs, left) and precision (JNDs, right) of stationarity perception. Experiment 1 (top), Experiment 2 (bottom). ^*Indicates statistical significance at p < 0.05.

 

Overall, PSEs with low and high spatial frequency stimuli were not significantly different, F(1,19) = 3.70, p = 0.071 (Table 1). The significant interaction between fixation and frequency, F(1,19) = 8.44, p = 0.009 (Table 1) suggests that any effect of spatial frequency depended on fixation. Specifically, PSEs with high spatial frequency were reduced relative to low spatial frequency when fixation was head-fixed, such that the eyes remained roughly fixed in the head and retinal image motion was therefore maximized. 

The precision of stationarity judgments is captured by the JNDs, and these also varied across conditions (Figure 5). The greatest precision (lowest JND) was observed during scene-fixed fixation with the low spatial frequency stimulus. The average JND in this condition was 0.12, meaning that a roughly 13% change in visual gain relative to the PSE was needed in order for the mismatch to be consistently noticed. JNDs in the head-fixed conditions were significantly increased relative to the scene-fixed conditions, F(1,19) = 14.6, p = 0.001 (Table 1), meaning that judgments were generally more noisy or uncertain during head-fixed fixation. Stationarity judgments were also significantly less precise in the high-spatial frequency compared to the low-spatial frequency conditions, F(1,19) = 12.2, p = 0.003 (Table 1), suggesting that scene content with low spatial frequency facilitates judgments of stationarity. 

Figure 5.

 

Precision of stationarity perception - just-noticeable differences (JNDs). Experiment 1 (left), Experiment 2 (right). Note that high-frequency full-field data from experiment 1 are replotted alongside results of experiment 2 (translucent points) to facilitate comparison. The JND is the standard deviation of the cumulative Gaussian psychometric fit to the data from each subject and condition. Values indicate the proportional increase/decrease needed relative to the PSE in order to reliably elicit responses of “against”/“with.” The red circles represent the mean JND for that condition, and individual subject JND values are shown as gray points. The standard deviation for the JND values are shown via the blue lines which extend one standard deviation above and below the mean. The left hand scale of each graph represents the natural log of the gain. The right hand scale of each graph is the gain in linear units.

Experiment 2

The second experiment investigated the influence of fixation behavior (head-fixed vs scene-fixed) and retinal stimulus location (central, peripheral, or full) on psychophysical judgments of stationarity. Data from the high spatial frequency condition of experiment 1 was analyzed along with data from experiment 2 using a 2 × 3 repeated measures ANOVA. Accuracy (Figure 4) was once again better (i.e., closer to zero) in the scene-fixed relative to the head-fixed condition, with lower gains (i.e., slower visual speeds) perceived as stationary during head-fixed fixation, and statistically significant differences observed across these conditions, F(1,19) = 48.02, p < 0.001 (Table 1). Retinal stimulus location, on the other hand, did not have a significant influence on accuracy, F(2,19) = 0.077, p = 0.926 (Table 1); PSEs were comparable regardless of retinal stimulus location. 

Precision (Figure 5), on the other hand, varied significantly depending on retinal stimulus location, F(2,19) = 5.58, p = 0.008 (Table 1), and these effects were mediated by fixation behavior, as suggested by the significant interaction, F(2,19) = 3.47, p = 0.042 (Table 1). Specifically, during head-fixed fixation, the best and worst precision was observed during the full and peripheral conditions, respectively. During scene-fixed fixation, however, differences across these conditions were less marked. Finally, results of Experiment 2 were consistent with those of Experiment 1 in demonstrating that precision is better overall during scene-fixed compared to head-fixed fixation, F(1,19) = 37.4, p < 0.001 (Table 1). 

Association between sickness and stationarity perception

A linear mixed-effect model (LMM) was fit to evaluate whether sickness, as assessed by the SSQ and Discomfort Scores, was associated with psychophysical performance, as assessed by PSEs and JNDs. Coefficients of the LMM and significance of associations are displayed in Table 2. 

Table 2.

 

Relationship between psychophysical measures of stationarity perception (PSEs, left; JNDs, right) and measures of sickness (SSQ and discomfort scores). Columns display the coefficients of the LMM along with the p-values from the Satterthwaite analysis. ^*Indicates statistical significance at p < 0.05.

 

The SSQ Nausea subscore was significantly associated with the accuracy of stationarity judgments (PSE). The coefficient of the fixed effect revealed a negative relationship between PSEs and Nausea (Table 2). This suggests that when PSEs are lower, subjects report higher levels of nausea. 

The SSQ Oculomotor and Disorientation subscores as well as the SSQ Total Score were significantly associated with the precision of stationarity judgments (JNDs). The coefficients of the fixed effects reveal a positive relationship between JNDs and the other three scores (Table 2). When JNDs were higher and subjects were more uncertain in their judgments of stationarity, reports of oculomotor discomfort, disorientation, and total sickness were higher. 

Neither average discomfort score, nor ending discomfort score were significantly associated with PSE or JND values. 

Discussion

Directly estimating self-motion from visual motion is contingent on the determination that the visual environment is stationary. Despite this importance, the mechanisms that the nervous system relies on to evaluate environmental stationarity remain poorly understood. Here we demonstrate that the precision and accuracy of stationarity judgments are jointly mediated by visual stimulus parameters and fixation behavior. We also observe systematic relationships between stationarity perception and simulator sickness. These findings have implications for enhancing stationarity perception and mitigating sickness in virtual and augmented reality applications. 

Effect of spatial frequency on stationarity perception

Manipulating spatial frequency content of the visual scene allows evaluating the relative contribution of different spatial frequency channels to the visual self-motion estimate. The greatest accuracy of stationarity perception was observed in the low spatial frequency condition (Figure 4). In other words, the visual self-motion estimate was most consistent with the nonvisual self-motion estimate when the low spatial frequency pattern was presented. This suggests that low spatial frequency channels are weighted highly in generating the visual self-motion estimate, which makes sense given the presence of large-scale spatial structure in the visual environment. The decreased gain perceived as stationary in the high spatial frequency conditions suggests increased speed of the visual self-motion estimate; this is consistent with previously reported effects of spatial frequency on visual speed estimation (Campbell & Maffei, 1981; Ferrera & Wilson, 1991) and vection (Diener et al., 1976; Palmisano & Gillam, 1998; Guo et al., 2021), and could explain the reduction in the gain perceived as matching. 

Differences in accuracy between low and high spatial frequency conditions were most evident in the head-fixed condition, in which subjects suppressed VOR eye movements, giving rise to maximal retinal image motion. In the scene-fixed condition, accuracy was comparable across conditions. We interpret this as an indication that the oculomotor contribution to the visual self-motion estimate in the scene-fixed condition (and equivalently the gain of the visually-enhanced VOR) is relatively unaffected by spatial frequency. The effect of spatial frequency is only observed in the head-fixed condition due to the increasing velocity of retinal image motion. 

Precision of the visual self-motion estimate was also best in the low spatial frequency conditions (Figure 5). This suggests that the visual-self-motion estimate is more noisy when only high spatial frequency stimuli are presented. If the high spatial frequency pattern leads to faster speed estimates, this could be an indication of noise proportional to the estimated speed. 

Effect of retinal stimulus location on stationarity perception

It is well-known that motion processing changes as a function of retinal eccentricity (McKee & Nakayama, 1984; Bower et al., 2012). It has also been shown that visually-induced perception of self-motion is affected by the retinal location (central vs. peripheral) of the optic flow stimulus (Brandt et al., 1973), though other studies have suggested that these effects depend on the relative depth (foreground vs. background) (Howard & Heckmann, 1989) or other properties of the central and peripheral stimuli (Andersen & Braunstein, 1985) including spatial frequency (Palmisano & Gillam, 1998). By manipulating the retinal eccentricity of the stimulus in the stationarity perception task, the goal was to assess the differential contributions of central and peripheral retinal signals to the overall visual self-motion estimate. 

We did not observe any effect of retinal stimulus location on the accuracy of stationarity perception (Figure 4). Visual gain perceived as stationary was reduced in the head-fixed conditions in Experiment 2, similar to what was observed in Experiment 1, especially with the high spatial frequency stimulus. This effect is likely to reflect a visual self-motion estimate that is biased toward increased speed with increased retinal image motion present in the head-fixed conditions. However, there were no further effects of retinal stimulus location, suggesting that this effect generalizes across retinal stimulus locations. 

The precision of stationarity perception was also worse in the head-fixed condition, with a significant influence of retinal stimulus location that was mediated by fixation behavior (Figure 5). Specifically, while the full-field stimulation led to the worst precision (largest JNDs) among the head-fixed conditions, it led to the best precision among the scene-fixed conditions. This may reflect an advantage of integrating retinal motion information across the visual field to generate a visual self-motion estimate under the conditions of this experiment. Precision was the worst overall in the head-fixed peripheral condition, suggesting that it was very challenging to estimate visual self-motion reliably in this condition. This makes sense given that sensitivity to motion of high spatial frequency stimuli has been shown to decrease with eccentricity due to the decrease in spatial resolution with increasing retinal eccentricity (Schor & Narayan, 1981; Johnston & Wright, 1985). 

Stationarity perception and simulator sickness

Simulator sickness, VR sickness, or cybersickness represents a significant barrier to large-scale adoption of VR technology. It is therefore important to develop a better understanding of the etiology of this sickness, and thereby a better understanding of circumstances that are more or less likely to induce it. Perhaps the most widely accepted explanation for what causes simulator sickness is visual-vestibular conflict which elicits visual and nonvisual (for example, vestibular) self-motion estimates that do not agree (Reason & Brand, 1975; Reason, 1978). This will occur in VR whenever the visual stimulus conveys self-motion, via optic flow, for example, while continuously navigating a virtual environment, even though the user is physically stationary. 

While many studies have demonstrated that conflict in VR induces sickness (for example, Akiduki et al., 2003; Keshavarz, Hecht, & Zschutschke, 2011; Ng, Chan, & Lau, 2020; Weech, Wall, & Barnett-Cowan, 2020), we are not aware of any study that has shown an association between psychophysical measures of stationarity perception and sickness. Perhaps closest to the current results, Kim et al. (Kim, Luu, & Palmisano, 2020) report an association between what they call perceived scene instability and sickness, but their measure of scene instability is quite distinct from ours because it gauges the magnitude of perceived conflict between the angle of the visually rendered and actual ground plane rather than biases and noise on perception of scene stationarity. Here we report that impaired performance on the stationarity perception task, in the form or decreased accuracy or precision, is associated with increased sickness as assessed with the SSQ. Specifically, we observe a dissociation in which impaired accuracy (PSE ≠ 0) is associated with increased scores on the Nausea subscale of the SSQ, while decreased precision (higher JND) is associated with increased oculomotor and disorientation subscale scores, as well as total score. 

While the associations are statistically significant, we cannot make claims about causal links between stationarity perception and sickness. For example, because we use an adaptive staircase procedure, experimental blocks that lead to nonunity gains perceived as stable entail greater exposure to nonunity gains. This increased exposure to nonunity gains (i.e., conflict) could be causing greater symptoms of sickness. Alternatively, certain stimulus conditions may elicit increased symptoms of sickness and sickness may cause impaired performance on the psychophysical task. Another alternative is that both effects on stationarity perception and sickness are caused by a third latent variable that is not measured here. 

Despite this uncertainty, the associations are informative and may both explain the effectiveness of existing sickness mitigation methods, as well as point the way toward new methods. For example, tunneling is a method whereby the field of view of the VR display is restricted when optic flow is presented, thereby reducing peripheral stimulation (Fernandes & Feiner, 2016; Al Zayer, Adhanom, MacNeilage, & Folmer, 2019; Adhanom, Navarro Griffin, Macneilage, & Folmer, 2020; Adhanom, Al-Zayer, Macneilage, & Folmer, 2021). This method has proven effective in reducing symptoms of sickness. In line with the observed associations, we find the greatest precision in the central conditions which mimic tunneling. 

Generally, it may be that stimulus conditions that support accurate and precise stationarity perception also lead to low levels of sickness in users. For example, we observe better psychophysical performance during scene-fixed rather than head-fixed fixation. Thus, it may be wise to encourage scene-fixed fixation during VR use as a method to reduce incidence of sickness. For example, this would suggest that graphical user interfaces rendered in VR should be world-fixed. Similarly, eliminating high-spatial frequency elements or increasing low spatial frequency elements of the visual scene may enhance stationarity perception and reduce sickness. Finally, it may be possible to train users to improve the precision and accuracy of stationarity perception, for example by providing feedback during a psychophysical task, and this may also lead to decreased incidence of sickness in these users. 

Comparison with prior studies using the stationarity task

Some version of the stationarity perception task has been used previously and values reported for accuracy and precision during active yaw head rotation with scene-fixed fixation differ considerably across these studies. Wallach was the first to use this task (Wallach & Kravitz, 1965; Wallach, 1987), and he reported that unity gains were perceived to be stable (i.e., a high degree of accuracy), and that an increase/decrease in gain of only 2% to 3% was perceived as moving (i.e., good precision). All subsequent studies that we are aware of report similar accuracy but considerably worse precision, perhaps due to the use of head-mounted displays, which have the potential to introduce latency, as well as limitations on tracking accuracy/precision, field of view, resolution, luminance, contrast, and so on. For these reasons, comparisons of performance across studies is difficult. We nevertheless provide a brief overview of comparable results here. Jaekl et al. (2005) report that gains close to unity (gain=1.09) were perceived as stable with a standard deviation of approximately 20% for reliably detecting increased or decreased gain. Steinicke, Bruder, Jerald, Frenz, & Lappe (2010) report an average PSE of gain=0.96 (accuracy) with approximately 30% increase/decrease in gain needed to detect increased/decreased gain. Correia Grácio et al. (2013) also report gains close to unity, but they do not report precision in a manner that is comparable to the present study. In general, we attribute differences in values reported across studies to methodological differences including display device, content of the visual scene, and method used to track head movement and render scene motion contingent on that head movement. 

In work from our own group, we have previously reported PSEs close to unity and JNDs of approximately 30% (Halow et al., 2023). JNDs were reduced considerably in the present study, with the best precision of approximately 13% in the scene-fixed condition with low spatial frequency. Since methods were broadly similar, we speculate that this difference across studies is due to differences in the visual stimuli (starfield vs. optokinetic drum) and/or tracking systems (Optitrack vs. HTC Vive lighthouse). As we show here, features of the visual stimulus which differ between starfield and drum can impact performance. Tracking systems could have impacted performance as well because Optitrack tracking is not the native tracking system for the Vive, and our previous implementation (Halow et al., 2023), therefore, presumably removed features such as prediction and smoothing that are most likely implemented to improve the consumer experience when using the native lighthouse tracking. Details of the tracking pipeline are proprietary, so these observations are purely speculative. Moving forward, it may be desirable to use tracking methods for which the entire tracking pipeline is accessible. This would allow investigating the impact of such tracking parameters on stationarity perception. 

The studies described above investigated stationarity perception during active yaw head turns with scene-fixed fixation (or uncontrolled fixation, which was presumably scene-fixed). Several studies (for example, Jaekl et al., 2005; Steinicke et al., 2010; Correia Grácio et al., 2013; Teng, Allison, & Wilcox, 2023) have investigated other degrees of freedom, meaning roll and pitch as well as linear head movements. Reported values for PSEs and JNDs vary considerably depending on the degree of freedom that is investigated. We do not review these results in detail here because they are beyond the scope of the present study. 

Conclusions

Stationarity perception provides a useful method to investigate how visual stimulus features impact visual self-motion estimates. For example, we can use this paradigm to further investigate how motion information is integrated by the visual system to estimate self-motion taking into account the impact of stimulus features such as contrast and luminance. This method has advantages over other methods, such as vection ratings, which do not entail comparison between the visual self-motion estimate and a well-defined norm (i.e., the vestibular stimulus). Ultimately, it would be fascinating to probe neural correlates of these behaviors in visually responsive brain areas through training of nonhuman primates to perform this task. 

We interpret our results assuming that stationarity perception is governed by simple cross-modal discrimination. In reality, more complex models may be more appropriate. For example, prior research has suggested a visually-referenced detection model (Jürgens & Becker, 2011). Similarly, prior expectations, for example, that the visual scene is in fact stationary (Wertheim, 1994; Duncker, 1929), may play a role and may influence performance (for example, Garzorz & MacNeilage, 2017). This task is well-suited to be modeled in a causal inference framework (Acerbi, Dokka, Angelaki, & Ma, 2018; Dokka, Park, Jansen, DeAngelis, & Angelaki, 2019; Noel et al., 2023). 

We have demonstrated an association between perceptual performance and sickness, but this association may not generalize beyond the current study. Further research is needed to test this generalization and also identify other perceptual measures that exhibit associations with these and other measures of sickness. This endeavor can shed light on the etiology of sickness and also inform sickness mitigation strategies. 

Acknowledgments

Supported by NIH P20GM103650 and NSF CHS 1911041. 

Commercial relationships: none. 

Corresponding author: Savannah J. Halow. 

Email: [e-mail]. 

Address: 1664 N. Virginia Street, Reno, NV 89557, Mail Stop 0296, USA.