Applications and implications for extended reality to improve binocular vision and stereopsis

Abstract

Extended reality (XR) devices, including virtual reality (VR), augmented reality (AR), and mixed reality (MR) devices, are immersive technologies that can swap or merge the natural environment with virtual content (e.g., videogames, movies, or other content). Although these devices are widely used for playing videogames and other applications, they have one distinct feature that makes them potentially very useful for the measurement and treatment of binocular vision anomalies—they can deliver different content to the two eyes simultaneously. Indeed, horizontally shifting the images in the two eyes (thereby creating binocular disparity) can provide the user with a compelling percept of depth through stereopsis. Because these devices are stereoscopic, they can also be used as high-tech synoptophores, in which the images to the two eyes differ in contrast, luminance, size, position, and content for measuring and treating binocular anomalies. The inclusion of eye tracking in VR adds an additional dimension to its utility in measuring and treating binocular vision anomalies, as well as other conditions. This paper describes the essential requirements for testing and treating binocular anomalies and reviews current studies in which XR devices have been used to measure and treat binocular vision anomalies.

Introduction

The benefits and costs of binocular vision

Primates are “blessed” with two frontal, mobile eyes. The advantages afforded by having two eyes, rather than a single eye, include stereoscopic depth perception due to the different viewpoints of the two eyes, binocular summation of luminance and contrast, redundancy of information, an expanded field of view, masking of scotomas (including the physiological blindspot in each eye), and facial symmetry (thought to be an important determinant of beauty) (Rhodes, 2006). Indeed, the evolution of binocular vision in many primates (including humans) has resulted in a significant investment in the development of cortical areas involved in binocular interactions and stereopsis (Parker, 2007). However, binocular vision comes with a cost. The movement of each eye is brought about by six extraocular muscles. In addition, normal binocular vision requires exquisite coordination of the movements of the two eyes, so that the retinal images fall on corresponding points in the two eyes. This is true for both version movements (i.e., both eyes moving in the same direction) and vergence movements (i.e., the two eyes moving in opposite directions to gain or maintain single binocular vision). Additionally, because accommodation (the ability to change focus) is linked to convergence, binocular vision requires coordination of both the intra- and extraocular neuromuscular systems. These requirements call for near perfection from binocular vision in humans. Consequently, a substantial proportion of people have deficits leading to anomalous binocular vision, including amblyopia, strabismus, high heterophoria, and accommodative–convergence problems (discussed below). Estimates of the prevalence of these anomalies vary substantially; one recent systematic review suggests that the pooled prevalence of amblyopia worldwide is about 4.3% (Mostafaie et al., 2020) and that of strabismus almost 2% (Hashemi et al., 2019). The failure to achieve normal binocular vision can result in reduced visual acuity in one eye, binocular suppression, reduced or absent stereopsis, diplopia, difficulty reading, asthenopia, and problems with eye–hand coordination (e.g., reaching and grasping). 

This article is intended to provide an overview of the current and potential uses of extended reality (XR) devices used to measure and treat binocular vision anomalies. XR devices include virtual reality (VR), augmented reality (AR), and mixed reality (MR) devices. These are immersive technologies that can swap or merge the natural environment with virtual content (e.g., videogames, movies, or other content). Although these devices are commonly used for playing videogames, they have one distinct feature that makes them potentially very useful for the measurement and treatment of binocular vision anomalies—they can deliver different content to the two eyes simultaneously. This paper describes the essential requirements for testing and treating binocular anomalies and reviews current studies in which XR devices have been used to measure and treat binocular vision anomalies. 

Conventional treatment for anomalous binocular vision

Amblyopia

Amblyopia is a common neurodevelopmental abnormality that results in physiological alterations in the visual pathways and impaired vision in one eye or, less commonly in developed than in rural regions, in both eyes (Meng et al., 2021). Aside from refractive error, amblyopia is the most common cause of vision loss in children. It is thought to primarily reflect alterations in visual cortex rather than in the retina (Kiorpes, 2019; Kiorpes & Movshon, 1996; Levi, 2020), including strong inhibition of the amblyopic eye by the dominant non-amblyopic fellow eye. Conventional clinical treatment of amblyopia, aimed at improving the visual acuity of the amblyopic eye, involves correction of any refractive error and patching (occluding) or penalizing the non-amblyopic eye. However, conventional treatment is slow, requiring months or even years of patching. Amblyopia is typically only treated in young children (less than 7 years old), and, although older children and even adults can benefit from patching, they are less responsive and require more prolonged patching than younger patients. Moreover, as many as 15% to 50% of children with amblyopia do not recover normal visual acuity, even after extensive treatment, and among those who do fully recover about one quarter regress in the first year (Birch, 2013; Holmes & Levi, 2018; Levi, 2020). 

These limitations have led to the development of several new approaches over the last decade or so, including perceptual learning, videogame play, and dichoptic/binocular treatment aimed at reducing the inhibition of the amblyopic eye and improve binocular vision and stereopsis (for reviews, see Bennett, Bex, Bauer, & Merabet, 2019; Birch, Kelly, & Wang, 2021; Brin, Chow, Carter, Oremus, Bobier, & Thompson, 2021; Coco-Martin et al., 2020; Falcone, Hunter, & Gaier, 2021; Foss, 2017; Hess & Thompson, 2015; Levi, 2020; Levi, 2022; Levi & Li, 2009). These approaches either use or could be adapted to use XR devices, which include VR, AR, and MR devices, which are immersive technologies and can swap or merge the natural environment with virtual content (e.g. videogames, movies). This review is focused on studies that have implemented these strategies in XR devices, primarily in VR. 

Strabismus

Strabismus (also known as heterotropia or, in the UK, as squint) is a condition in which the two eyes are misaligned; that is, both eyes do not look at the same place at the same time (for the American Optometric Association definition, see https://www.aoa.org/healthy-eyes/eye-and-vision-conditions/strabismus?sso=y). The degree of misalignment comes in a broad range of sizes, ranging from just a few degrees (microstrabismus) to very large angles, and in various directions—with one eye deviating inward (esotropia), outward (exotopia), up (hypertropia), or down (hypotropia) or rotated inward or outward around the visual axis (cyclotropia). The misalignment can be constant or intermittent (for example, only when fatigued) and can present at all viewing distances or only under specific viewing conditions—for example, only when looking up close (Duke-Elder, 1973). If strabismus occurs late in life (e.g., due to trauma or stroke), it results in diplopia, because the object of interest will be imaged on the fovea of one eye and on a non-corresponding area in the other. However, if the strabismus has its onset early in life, this results in suppression (inhibition) of the deviated eye. The locus and nature of the suppression depend on the direction of the eye turn. Conventional treatments include realigning the eyes using glasses (lenses and/or prisms), exercises (orthoptics), or surgery to realign the eyes. Classically, orthoptic treatment (also known as vision therapy) was often done using a clinical Wheatstone mirror stereoscopic arrangement (known as a troposcope, synoptophore, or amblyoscope) (Figure 1, left). This allows separate images to be presented to the two eyes (an example is shown in Figure 1, top right) and alignment of the images on corresponding areas in the two eyes by adjusting the positions of the mirrors to enable fusion. In addition, the troposcope is used to try to reduce suppression, encourage fusion and stereopsis, and increase the range of fusional vergence. These functions can be (and have been) readily achieved in XR display devices; an example is the VIVE Pro Eye head-mounted display (HMD) shown in Figure 1, bottom right. Although there are many smartphone apps developed exclusively for ophthalmological purposes—as many as 131 as of March 2020 (Aruljyothi, Janakiraman, Malligarjun, & Babu, 2021), with several aimed at treating amblyopia—they are not well suited to treating binocular vision anomalies because of limitations due to the small size of the (single) screen. 

Figure 1.

 

(Left) Vintage synoptophore (circa 1940s). (Right, top) Fusion slides, one viewed by each eye in the synoptophore. (Right, bottom) VIVE Pro Eye VR HMD.

Heterophoria

Heterophoria (sometimes described as a latent strabismus) refers to a misalignment of the visual axes of the two eyes under conditions when fusion is eliminated (e.g., when one eye is occluded or fusion is disrupted optically) due to a neuromuscular imbalance (Maddox, 1921). Almost everyone has heterophoria under some conditions, but heterophoria can result in headaches, eyestrain, blurred vision, and intermittent diplopia if there is insufficient reserve fusional capacity to overcome the muscle imbalance. The range of treatment options, when the heterophoria results in manifest symptoms, is similar to those applicable for strabismus including lenses, prisms, orthoptic exercises, and in certain cases even surgery (Borish, 2006). As with strabismus treatment, orthoptic exercises for heterophoria can be (and have been) readily achieved in XR display devices. 

Accommodative–convergence problems

Humans spend a considerable amount of time on near vision tasks such as reading or using a computer or smartphone. However, some individuals have considerable difficulty working up close due to difficulty with either accommodation (focus) or convergence. Additionally, because accommodation and convergence are inextricably linked, the accommodative–convergence relationship may not be appropriate for sustained efficient near viewing. Accommodative–convergence problems (including poor accommodative and vergence facility) can lead to similar symptoms to those discussed above (Banks, Hoffman, Kim, & Wetzstein, 2016; Hoffman, Girshick, Akeley, & Banks, 2008; Shibata, Kim, Hoffman, & Banks, 2011a) and to similar treatment modalities. 

VR devices might be useful for measuring and treating binocular vision anomalies

The VR environment is a 360° world that enables the user to move about, turn, and interact with its content. Although the virtual environment differs in a number of important aspects from the natural environment (Aizenman, Koulieris, Gibaldi, Sehgal, Levi, & Banks, 2022; Harris, Buckingham, Wilson, & Vine, 2019), VR devices have one distinct feature that makes them potentially very useful for the measurement and treatment of binocular vision anomalies in that they can deliver different content to the two eyes simultaneously. Indeed, by horizontally shifting the images in the two eyes (thereby creating binocular disparity), videogame developers and movie makers can provide the user with a compelling percept of depth through stereopsis. In other words, VR headsets (HMDs) are stereoscopic and can be used as high-tech synoptophores, in which the images to the two eyes differ in contrast, luminance, size, position, and content. The inclusion of eye tracking in VR adds an additional dimension to its utility in measuring and treating binocular vision anomalies, as well as other conditions, such as cortical visual impairment (CVI) (Bennett et al., 2019). Below, we describe the essential requirements for testing and treating binocular anomalies and review the current studies and reports in which VR devices have been used to measure and treat binocular vision anomalies. 

Requirements for measuring and treating binocular anomalies using VR

The key requirement for any instrument aimed at measuring and treating binocular anomalies is the capability of presenting and controlling different images to the two eyes at the same time. This allows for determining the subjective angle of alignment (where separate images in the two eyes are seen in the same visual direction), the presence or absence of binocular fusion and suppression, the luminance or contrast balance point (i.e., the relative luminance and/or contrast in the two eyes that result in equal perceptual input to the cortex), and assessment of the presence and quality of stereopsis. Additionally, by controlling the screen (and therefore retinal) positions of the images of each eye separately, the stimulus for fusional vergence can be varied. Many of these features can be found in standard clinical instruments, such as the synoptophore or troposcope. However, emerging research suggests several other features that may be important for both testing and treating binocular anomalies. These include a large field of view,¹ large binocular disparities, sensitive and repeatable measures of stereoacuity, eye-movement tracking, and the possibility of incorporating visuomotor integration into the treatment (Levi, 2022). 

The variability of retinal disparities encountered in the natural environment increases in size in proportion to the stimulus eccentricity (Aizenman et al., 2022; Sprague, Cooper, Tosic, & Banks, 2015), as does Panum's fusion area, which is the range of fusible retinal disparities (Hampton & Kertesz, 1983). Thus, for testing and treating binocular anomalies, a large field of view is essential to provide the possibility of peripheral fusion, even if central vision is reduced or suppressed in one eye. Importantly, some amblyopic patients who lack central stereopsis retain peripheral binocularity and stereopsis (Sireteanu, Fronius, & Singer, 1981). A large field of view also enables the presentation of large binocular disparities, which may be important for improving stereovision in both neurotypical (Levi & Li, 2019) and amblyopic patients (Godinez, Martin-González, Ibarrondo, & Levi, 2021; Li, Ngo, Nguyen, & Levi, 2011). Therefore, especially for clinical applications, it is crucial to have sensitive and repeatable measurements. To date, only a few studies have addressed this issue (see below). An important component of evaluating binocular vision is the objective assessment of eye position to establish the presence and magnitude of strabismus or heterophoria. Clinically, this is accomplished by evaluating the relative positions of the corneal reflections of a penlight (the Hirshberg test) or, more accurately, by the cover test (Duke-Elder, 1973). However, modern commercial HMDs such as the VIVE Pro Eye (Figure 1, lower right) offer the potential for reasonably accurate, objective eye tracking, which would make it possible to quantitatively measure the angle of strabismus and to use continuous eye tracking to control the retinal positions of the stimuli to the two eyes. To date, objective eye tracking has not been generally included in studies using XR headsets; however, a recent randomized control trial in children with amblyopia that combined eye tracking with anaglyph glasses to induce dominant eye blur around the fovea in dichoptic video content showed improved visual acuity in the amblyopic eye (Wygnanski-Jaffe, Kushner, Moshkovitz, Belkin, Yehezkel, & CureSight Pivotal Trial Group, 2022). 

Additionally, for extended treatment, it is crucial to have content that is engaging and highly motivating, is individually customized for success, progresses from easy to difficult, and provides rich feedback (Vedamurthy et al., 2016). It is also worth noting that individuals with amblyopia and reduced stereopsis have difficulties with visuomotor integration—reaching and grasping (Grant, Melmoth, Morgan, & Finlay, 2007; Grant & Moseley, 2011; Grant, Suttle, Melmoth, Conway, & Sloper, 2014; Niechwiej-Szwedo, Colpa, & Wong, 2019; Niechwiej-Szwedo, Goltz, Colpa, Chandrakumar, & Wong, 2017; Suttle, Melmoth, Finlay, Sloper, & Grant, 2011; Webber, 2018; Webber, Wood, Gole, & Brown, 2008) and locomoting in uneven terrain (Bonnen, Matthis, Gibaldi, Banks, Levi, & Hayhoe, 2021). Visuomotor activities can be easily incorporated into HMD VR games. Indeed, Vedamurthy et al. (2016) developed a VR “bug squashing” game that required participants to squash a virtual bug on a slanted surface by hitting it with a cylinder that they held in their hand. The perceived surface slant was determined by monocular texture cues and stereoscopic (disparity) cues that could be consistent or inconsistent. The participants, who were stereoblind or stereodeficient, were given rich visual, auditory, and tactile feedback. After 40 hours of game play, most participants’ stereoacuity improved, and these improvements were accompanied by a learned upweighting of their reliance on three-dimensional (3D) disparity cues relative to the two-dimensional texture cues. Finally, it is worth stating that the use of HMDs for treating binocular anomalies is not intended to replace traditional methods (patching/penalization) but rather to serve as an adjunct (Levi, 2020). 

Recent studies

Measurement of binocular anomalies using XR

Although a number of important recent advances have been made in the screening, diagnosis, and measurement of binocular anomalies (for a recent review, see Birch et al., 2021), only a few have been implemented in VR HMDs for clinical use. One is the Vivid Vision system, which is designed to deliver dichoptic visual stimulation for treating patients with binocular anomalies in commercial HMDs (https://www.seevividly.com/downloads/2022_Vivid%20Vision%20White%20Paper%20-%20Clinical%20Use%20v1.2.pdf). The system has a number of built-in tests that enable clinicians and patients to monitor progress during treatment, including tests for eye dominance, suppression, subjective angle of alignment, acuity, and stereo acuity. A recent study (Denkinger et al., 2022) found that the stereoacuity test was not susceptible to perceptual learning between sessions; however, it was unable to capture low thresholds (less than 20 arcsec) and was only weakly correlated with a “gold standard” global random dot stereoacuity test, Asteroid (McCaslin, Vancleef, Hubert, Read, & Port, 2020; Read et al., 2020). This system has been used to test and treat patients with impaired stereopsis (Backus, Dornbos, Tran, Blaha, & Gupta, 2018) and with amblyopia (Ziak, Holm, Halicka, Mojzis, & Pinero, 2017), discussed further in the next section. 

Another is the Visionary Tool (https://www.visionarytool.com), which is a commercial VR tool for testing and treating binocular anomalies. It incorporates a variety of tests and training options, including a test for binocular balance (discussed below), measurement of the subjective and objective angle of strabismus (with eye tracking), and an updated version of the scaffolding stereopsis training method described by Godinez (Godinez et al., 2021) and discussed below. Martin, Portela, Ding, Ibarrondo, and Levi (2020) evaluated the testability and reliability of the Visionary Tool binocular balance test in a large population (N = 100) ages 6 to 70 years old. Their results suggest that the test is both reliable and feasible in a clinical population and shows promise for differentiating patients from neurotypical controls with high confidence. 

Treatment of binocular anomalies using XR devices

XR devices (both VR and AR) have been used to treat patients with amblyopia, strabismus, heterophoria, and accommodative–convergence problems, as well as patients with stroke and traumatic brain injuries (TBIs), which are often accompanied by binocular vision problems such as diplopia and strabismus (Schaadt et al., 2014) (discussed below). 

Amblyopia

Over the past two decades or so, XR devices have been used in the treatment of amblyopia and strabismus (Table 1). Some of these applications have been reviewed elsewhere (Bennett et al., 2019; Birch et al., 2021; Boniquet-Sanchez & Sabater-Cruz, 2021; Coco-Martin et al., 2020; Foss, 2017). The earliest implementation was the development of an interactive binocular VR system, I-BiT, to treat amblyopia through interactive videogames or viewing movies (Cleary, Moody, Buchanan, Stewart, & Dutton, 2009; Eastgate et al., 2006; Foss, 2017; Herbison et al., 2013; Herbison et al., 2016; Waddingham et al., 2006). Other studies aimed at treating amblyopia have used commercial VR headsets with special-purpose software. For example, Ziak et al. (2017) instructed participants to play the Vivid Vision Diplopia game using an Oculus Rift DK2 headset (Oculus VR, Irvine, CA). Elhusseiny, Bishop, Staffa, Zurakowski, Hunter, and Mantagos (2021) used a ZEISS VR One Plus headset (ZEISS, Oberkochen, Germany) and prototype binocular therapeutic software. As illustrated in Table 1, the type of treatment, participant ages, and characteristics were quite varied, as were the improvements in visual acuity in the treatment groups, varying from none to as much as 5 lines (0.5 logMAR). 

Table 1.

 

Studies to date. Notes: Tx = treatment; A = anisometropic; S = strabismic; M = mixed; IXT = intermittent exotropia; VGP = videogame play.

 

Recently, Bao, Dong, Liu, Engel, and Jiang (2018) developed a novel method for viewing with both eyes, using a head-mounted video camera that feeds into an image-processing computer that in turn drives a head-mounted display. Specifically, the images in each eye are divided into a grid of square patches that are presented to the two eyes in a complementary fashion. In order to see the entire image in its full, intact form, subjects have to process inputs from both eyes cooperatively. Importantly, this study showed that adaptation to this altered input could change ocular dominance in healthy subjects and could improve visual acuity and contrast sensitivity of amblyopic subjects (Table 1). A more recent study (Banko et al., 2022) used an immersive stereo 3D AR environment (Leonar3Do International, Inc., Budapest, Hungary) to present stereoscopic videogames (capturing 3D caterpillars) on a passive 3D display. The authors reported improvements in visual acuity, stereoacuity, and contrast sensitivity. Interestingly, they reported that, in their pediatric population, astigmatism limited improvements in visual acuity, and they proposed a treatment regimen that combines standard treatment modalities (refractive correction and occlusion) with biofeedback fixation training (in patients with significant fixation instability in the amblyopic eye) and dichoptic and stereoscopic AR therapy. Their results and modeling also suggest that the worse the stereoacuity is at baseline, the more likely the patient will improve. However, for patients with no stereopsis at baseline, stable fixation with the amblyopic eye is important for improvements in stereoacuity following their training. 

Strabismus

XR has also recently been applied to the treatment of strabismus. A feasibility study of 20 minutes of VR and AR training of binocular vision in postoperative strabismus (Yang et al., 2022) resulted in a small (3%–6%) improvement in stereoacuity, which is not clinically significant. A 6-month study of dichoptic VR training resulted in improved eye position (measured both subjectively and objectively), and in some patients acquisition of stereopsis (Li et al., 2019). It is worth noting that, in studies aimed at treating amblyopia or poor stereopsis rather than strabismus per se, some patients with strabismus also show improved stereopsis (e.g., Godinez et al., 2021) (discussed below). 

Stereopsis

A number of studies have attempted to improve stereopsis in patients with amblyopia and/or strabismus by training stereopsis directly through perceptual learning or playing 3D videogames (Astle, McGraw, & Webb, 2011; Li, Tran, Bui, Antonucci, Ngo, & Levi, 2018; Portela-Camino, Martin-González, Ruiz-Alcocer, Illarramendi-Mendicute, & Garrido-Mercado, 2018; for a review, see Levi, 2022; Levi, Knill, & Bavelier, 2015); however, only two studies have applied VR devices in an attempt to improve stereopsis in patients with amblyopia, strabismus, or poor stereopsis. As noted earlier, Vedamurthy et al. (2016) had participants play a bug squashing game with virtual bugs presented in a special-purpose lab VR system. More recently, Godinez et al. (2021) and Portela-Camino et al. (2018) had participants play stereoscopic videogames using the Oculus Rift DK2 headset. Each of these studies resulted in significantly improved stereoacuity (by as much as 65%) with no change in visual acuity. 

Stroke and TBIs

Stroke can have a substantial impact on visuospatial perception (VSP). Impairments resulting from stroke include spatial neglect, hemianopia, gaze palsy, strabismus, and stereoblindness (Ross, 1983). It has been estimated that a majority of those with TBI experience oculomotor problems (Ciuffreda, Tannen, & Suter, 2020; Ciuffreda & Thiagarajan, 2022). AR has the potential to be especially useful for improving visuospatial perception by enabling the user to interact with virtual images in the real world (Gorman & Gustafsson, 2022). A recent study showed that stereopsis significantly predicted AR distance judgments and the ability to perceive holograms as 3D in neurotypical subjects. However, in stroke patients, the 3D judgment task was not predicted by stereovision. Patients with impaired VSP are less accurate in their ability to perceive the distance of virtual 3D objects (Hohler et al., 2021). The authors suggest that these findings may be useful in the rehabilitation of stroke patients with VSP deficits using AR devices. To date, the author is not aware of any studies that have implemented this suggestion. 

VR HMDs have been widely used in patients with TBI for improving balance deficits (Tefertiller et al., 2019), cognitive rehabilitation, attention and memory, and gait deficits, for example, in adults with TBIs (for a review, see Aida, Chau, & Dunn, 2018) and children for balance and physical rehabilitation (for a review, see Shen, Johnson, Chen, & Xiang, 2020). A recent study of 2458 US veterans with TBI found that almost a quarter were referred for vision-related rehabilitation (Winkler et al., 2022). This speaks to the importance of using vision as a diagnostic and progression monitoring tool for TBIs. A recent review (Simpson-Jones & Hunt, 2019) concluded that, “Promising interventions for vision rehabilitation following mild traumatic brain injury include the use of optical devices (e.g., prism glasses), vision or oculomotor therapy (e.g., targeted exercises to train eye movements), and a combination of optical devices and vision therapy.” However, none of the studies they reviewed involved the use of XR HMDs, and the author is not aware of any studies regarding the use of XR specifically for binocular anomalies associated with TBI. 

Some limitations and caveats regarding the use of XR devices for treating binocular anomalies

Some of the early problems commonly associated with older VR headsets, such as motion sickness, have been largely ameliorated by improvements in VR HMDs, including higher frame rates and predictive tracking. However, as early as the last century, there were concerns about potential problems for stereoscopic depth perception in VR (Wann, Rushton, & Mon-Williams, 1995), and Rushton and Riddell (1999) warned about the potentially harmful effects of VR in children. They suggested several approaches to better understand the plasticity of accommodation and vergence in the developing visual system. However, more than 20 years later, our understanding of the potential harmful effects is still quite rudimentary, and designing the appropriate studies for using XR devices by young children may be ethically challenging. Here are some of the known issues. 

The relationship between accommodation and convergence in the virtual environment is different from the natural environment

In the natural environment, accommodation and convergence are tightly linked and depend on the distance of the object of fixation. However, in order to see clearly in VR, we must accommodate (focus) to the virtual screen distance (1–2 meters in most modern consumer VR headsets), while converging on the object of regard, which might be at a very different distance (Kooi & Toet, 2004), leading to the well-known vergence-accommodation conflict (Banks et al., 2016; Hoffman et al., 2008; Shibata et al., 2011a; Wann et al., 1995). This vergence–accommodation conflict is known to cause discomfort and reduced performance (Hoffman et al., 2008; Koulieris, Bui, Banks, & Drettakis, 2017; Shibata, Kim, Hoffman, & Banks, 2011b). Interestingly, exercises aimed at altering the link between accommodation and convergence are sometimes used in the management of myopia and binocular anomalies (Cooper, Schulman, & Jamal, 2012), so it could actually be helpful in some patients if using VR actually improved the linkage between accommodation and convergence.² 

The statistics of eye movements and binocular disparity in the virtual environment are substantially different from that of the natural environment

The statistics of eye movements (fixation distances and directions) and binocular disparities across the visual field in popular videogames are inconsistent with the statistics of the natural environment (Aizenman et al., 2022) to which the human visual system is adapted (Sprague et al., 2015). These inconsistencies result in a greater likelihood of seeing double images and experiencing discomfort. Importantly, visual performance and comfort levels are better when the scene statistics are consistent with those of the natural environment than when they are not (Aizenman et al., 2022). On the other hand, successful therapies often use stimuli at or near the threshold of the patient's ability, which are not likely to be natural stimuli. So, although it may be important to develop tests and treatments using natural disparity scenes and to include content that uses natural disparities, it is not clear whether and how the “unnatural” disparity statistics might impact the effectiveness of therapy using XR devices. In other words, XR devices themselves may be quite good for visual therapy; however, the types of visual environments that are used in these therapies may require careful design. 

The virtual environment is more limited in size than the natural environment

When looking straight ahead, the normal binocular field of view (FoV) extends about 200° horizontally and 130° to 140° vertically. However, the FoV is considerably more limited in VR HMDs and varies substantially among the various HMDs. For example, Sauer, Sipatchin, Wahl, and Garcia (2022) tested six commercial HMDs and reported a range of horizontal binocular FoVs from 86° to 96° in five of them. Only one (StarVR One) had a horizontal binocular FoV approaching that of natural vision (182°); however, at the time of its release in 2018, the headset was listed at a price of $3200, considerably higher than most other standard consumer HMDs. Aizenman et al. (2022) have recently shown that, with optimal screen placement, the binocular field of view can be expanded. Table 2 provides the FoV and spatial and temporal resolutions of several common commercial VR HMDs. 

Table 2.

 

Characteristics of several commercial VR HMDs. Notes: Measurements are reported as provided by the manufacturers.

 

VR HMDs have limited spatial and temporal resolution

Testing visual acuity and especially fine stereoacuity in an HMD requires high spatial resolution. Visual acuity in neurotypical adults may be better than 20/20, requiring the ability to provide image detail less than 1 arcmin (a 20/20 letter is designed to subtend 5 arcmin at the retina), and stereoacuity thresholds may be as small as 3 arcsec or better (Westheimer & McKee, 1977). As shown in Table 2, most commercial HMDs have relatively limited spatial resolution. For example, with a resolution of 1440 pixels across a 100° FoV, the highest spatial frequency that can be faithfully displayed in an HMD is 7.2 c/°, far below the limit of normal spatial vision (30–60 c/°). Low spatial resolution also constrains the smallest disparities that can be displayed in an HMD. For example, the smallest disparity available in the Vivid Vision system stereo acuity test is 15 arcsec. Although this may be a problem for testing or training individuals with normal binocular vision (for example, it is not clear that we can train sensitivity to spatial frequencies that cannot be displayed in the headset), it is not necessarily a problem for screening or testing patients with binocular anomalies, and its limits are comparable to those of common clinical stereo tests such as the Randot circles test (limit 20 arcsec) or the ASTEROID Stereotest (limit 12 arcsec). Furthermore, headset manufacturers typically report only the resolution obtained when looking through the centers of the lenses; however, spatial resolution is not constant across the visual field. VR headsets have reduced resolution in the periphery (but not in the center) when looking straight ahead, and if the user looks toward one side then the resolution will drop markedly. Indeed, natural 3D scenes contain abundant information—much more than can be achieved in the VR environment—and it is not clear how many bits would be required for a 3D VR scene to perceptually match a 3D natural scene. Moreover, the limited luminance range available in VR headsets may be problematic in cases where the two eyes of a patient may need to be tested or trained under very different luminance levels in order to achieve balanced perceptual input (Ding & Levi, 2014). 

Early HMDs were associated with simulator sickness, dizziness, and nausea because of their low temporal resolution. However, many of the current commercial HMDs have higher frame rates than the early models (screen refresh rates vary from 60 to 120 Hz) (Table 2), resulting in shorter latencies, improved tolerance and, reduced simulator sickness. It seems likely that both spatial and temporal resolution will continue to improve in future HMD models. 

The virtual world may not be as interesting as the real world

A particular challenge with the use of VR HMDs for binocular training in young children is the choice of content. For example, children may grow bored with videogames (Kadhum et al., 2021) and simply give up. On the other hand, as noted elsewhere (Levi, 2022), dichoptic or 3D contrast-balanced video content (e.g., movies, TV shows) result in good compliance and outcomes (Xiao et al., 2020; Xiao et al., 2022). Jost et al. (2022) also reported excellent results using a similar approach, but without a headset or special glasses. Indeed, not only the choice of content but also the task and level of interaction may be what is important for children (and adults!) to engage in prolonged treatment. 

Other potential issues

In attempting to improve stereoscopic vision, it is important to note that depth constancy may be compromised and depth may be also be underestimated in VR (Hartle & Wilcox, 2022). Another potential consequence of VR HMD wear is increased choroidal thickness in young adults (Turnbull & Phillips, 2017). Although Turnbull and Phillips found no changes in binocular status of their subjects after 40 minutes of VR HMD wear, they hypothesized that the choroidal thickening resulted from the fixed virtual viewing distance of the HMD lenses (typically 1–2 meters), combined with convergence-induced accommodation in the virtual environment, creating a myopically defocused retinal image. It is not clear whether prolonged VR HMD wear would result in similar effects in young children or whether the effects (if any) would be long lasting. Finally, it is worth noting that some HMDs may not comfortably accommodate children and females with small heads (Stanney, Fidopiastis, & Foster, 2020) and may not allow for wearing glasses or for the appropriate interpupillary distance. 

Where do we go from here?

To quote the great Yogi Berra, “It's tough to make predictions, especially about the future.” However, it is clear that over time XR devices will improve and become less expensive. Spatial and temporal resolutions have already improved substantially in consumer headsets, and motion sickness and discomfort have been substantially reduced. A recent article suggests that heuristic evaluation by experts and users may be a useful tool for evaluating the potential of the application of ER for testing and treating binocular anomalies (Khaleghi, Aghaei, & Hosseinnia, 2022). The introduction of accurate eye tracking adds an important new dimension for measuring and treating binocular anomalies. Moreover, although there has been very limited use of AR in testing and treating binocular anomalies, this could provide a means to use natural scenes in treatment, as done by Bao et al. (2018), or by moving a virtual object near and far within the room the patient is actually sitting in. However, it seems important to carefully consider some of the issues raised above and to address them via carefully controlled studies, especially in young children. 

Acknowledgments

Supported by a grant from the National Eye Institute (R21EY030609). I thank Avi Aizenman, Ben Backus, Emily Cooper, Jian Ding, Angie Godinez, George Koulieris, Sunwoo Kwon, and Santiago Martin-Gonzáles for their very constructive and insightful comments and suggestions on an earlier version of this manuscript. 

Commercial relationships: none. 

Corresponding author: Dennis M. Levi. 

Email: [e-mail]. 

Address: Herbert Wertheim School of Optometry and Vision Science, University of California, Berkeley, Berkeley, CA, USA. 

Footnotes

1  Conventional instruments such as the synoptophore have a limited field of view (35° or less) (Barry, 2009; Stanworth, 1958).

Footnotes

2  One potential solution to the vergence–accommodation conflict is the use of light-field technology (https://www.computationalimaging.org/publications/the-light-field-stereoscope/); however, this is not commercially available at present.