Updating of visual orientation in a gravity-based reference frame

Abstract

The brain can use multiple reference frames to code line orientation, including head-, object-, and gravity-centered references. If these frames change orientation, their representations must be updated to keep register with actual line orientation. We tested this internal updating during head rotation in roll, exploiting the rod-and-frame effect: The illusory tilt of a vertical line surrounded by a tilted visual frame. If line orientation is stored relative to gravity, these distortions should also affect the updating process. Alternatively, if coding is head- or frame-centered, updating errors should be related to the changes in their orientation. Ten subjects were instructed to memorize the orientation of a briefly flashed line, surrounded by a tilted visual frame, then rotate their head, and subsequently judge the orientation of a second line relative to the memorized first while the frame was upright. Results showed that updating errors were mostly related to the amount of subjective distortion of gravity at both the initial and final head orientation, rather than to the amount of intervening head rotation. In some subjects, a smaller part of the updating error was also related to the change of visual frame orientation. We conclude that the brain relies primarily on a gravity-based reference to remember line orientation during head roll.

Introduction

Maintaining a veridical percept of objects in the world around us requires knowledge about the spatial relationships between objects in the environment, and between objects and ourselves (Burgess, 2006; Doeller, King, & Burgess, 2008; Filimon, 2015; Lambrey, Doeller, Berthoz, & Burgess, 2012; Mou, McNamara, Rump, & Xiao, 2006; Wang & Spelke, 2000). Relative orientation is one such relationship. If one of the references changes, the spatial relationship needs to be recomputed to maintain a correct registry with its true spatial orientation in the world. This is referred to as spatial updating, and has been studied extensively in relation to single points (Baker, Harper, & Snyder, 2013; Bloomberg, Jones, Segal, McFarlane, & Soul, 1988; Clemens, Selen, Koppen, & Medendorp, 2012; Gutteling, Selen, & Medendorp, 2015; Israël, Ventre-Dominey, & Denise, 1999; Klier & Angelaki, 2008; Tramper & Medendorp, 2015; Van Pelt, Van Gisbergen, & Medendorp, 2005). 

How do we maintain object orientation information across head rotation? Consider, for example, a line seen on a computer screen. To store its orientation, one could use the boundaries of the screen—an object-centered, or allocentric reference, which requires updating when the screen rotates, but not when the head tilts. Alternatively, the orientation of the line can be stored relative to the head—an egocentric reference frame, which requires updating when the head rotates (Medendorp, Smith, Tweed, & Crawford, 2002). As a third option, the orientation of the line can be stored relative to the direction of gravity, which determines our sense of upright in the world (Van Pelt et al., 2005; Yakusheva et al., 2007). Although gravity is an allocentric variable, its perception is egocentric, and modulated by vestibular signals (from semicircular canals and otolith organs) and visual context (Aubert, 1861; Kaptein & Van Gisbergen, 2004, 2005; Mittelstaedt, 1983; Vingerhoets, Medendorp, & Van Gisbergen, 2008; Witkin & Asch, 1948). Finally, it could be considered that the updating is not restricted to using a single reference frame, but that multiple reference frames are used in the maintenance of line orientation (cf. Tramper & Medendorp, 2015). Here, we investigated the role of allocentric and egocentric reference frames in the updating of line orientation across head rotations. 

To test between these reference frames, we exploited the systematic error that is observed when aligning a visual line to the gravitational vertical in the presence of panoramic visual cues—for example, when the line is surrounded by a square frame (Witkin & Asch, 1948). When the head is upright, line settings indicate a bias in the perception of gravity direction, which cyclically modulates with the orientation of the frame (Beh, Wenderoth, & Purcell, 1971). When performing the same task when the head is tilted but stationary, the size of this effect is increased, suggesting that the computation of the percept of gravity direction involves the combination of visual contextual cues and vestibular head-in-space signals (Alberts, de Brouwer, Selen, & Medendorp, 2016; Vingerhoets, De Vrijer, Van Gisbergen, & Medendorp, 2009). 

In the present study, we exploited this effect to examine the neural computations for spatial updating when torsional head rotations intervene between viewing a line, surrounded by a visual frame, and probing its remembered orientation with the frame upright. If the orientation of the line is stored in a gravity-based frame of reference, the corresponding memory will be affected by the perceived distortion of this frame. When a readout of this memory representation is obtained after the head rotation, the response will also incorporate the distortion of the perceived gravity direction at the new head position. Thus, the gravity-based model predicts that the updating error of line orientation is related to the difference in subjective distortion of the gravity frame when retrieving the orientation from memory and when storing the orientation in memory, rather than to the intervening head rotation itself. However, if line orientation is stored relative to the visual frame, in object-centered coordinates, these coordinates require updating if the frame is at a new orientation to keep correct registry with its true world-centered orientation. In this case, we would expect the readout of this memory after the head rotation to be related to the amount of intervening frame rotation, under the assumption that the brain can calculate this change in angle perfectly (Van Pelt et al., 2005). We also considered whether updating responses reflect a combination of these reference frames. 

Methods

Participants

Ten healthy subjects (two male, eight female, aged 22–34 years) participated in the experiment after giving their written informed consent, in accordance with the guidelines of the ethics committee of the Social Sciences Faculty of Radboud University, and in accordance with the Declaration of Helsinki. All subjects were naive with respect to the purpose of the study. Subjects reported to be free of vestibular or other neurological disorders, and had normal or corrected-to-normal vision. 

Experimental setup

Subjects were seated in a chair with two foam-padded platforms as headrests on either side of their head. The headrest on the subjects' right side was tilted 30° clockwise away from vertical; the headrest on their left side was directed vertically (Figure 1). The height and distance of the headrests were adjusted for each subject, allowing a 30° rotation around the head's roll axis. Stimuli were presented using a 55-in OLED screen (LG 55EA8809-ZC) with a resolution of 1920 by 1080 pixels and a 60-Hz refresh rate. The screen was placed in front of the subject at a horizontal distance of ∼170 cm. The stimulus was a gray 1-mm–wide line with a length of 25 cm (8° visual angle), presented with a random noise overlay and a Gaussian blur to prevent aliased edges of the stimulus from giving additional cues about the orientation of the line on screen. A square frame with sides of 45 cm (15° visual angle, line width 0.3 cm) was presented around the stimulus line. The bottom side of the frame was made slightly thicker (line width 1.3 cm) such that the frame orientation was unambiguous when oriented 45° clockwise or counterclockwise relative to upright. Stimuli were presented with a luminance of 0.22 cd/m². Subjects' responses were recorded with a keyboard. The experiment was performed in complete darkness, except for the stimuli on the screen. 

Figure 1

 

Updating task. After subjects tilted their head CW, they viewed a world-centered vertical visual line, surrounded by a square frame. They then rotated the head to upright, after which they indicated whether a probe line, varying in orientation and surrounded by an upright frame, was oriented CW (right arrow) or CCW (left arrow) with respect to the remembered stimulus line. In the stationary task, subjects were tested without intervening head rotation.

Experimental paradigm

Updating task

We used a two-alternative forced choice task to determine how subjects remember visual orientation across intervening head roll. A trial started with a high pitch tone (600 Hz), presented for 200 ms, that cued the subject to tilt the head 30° clockwise, against the headrest (Figure 1). With the head in this tilted position, the frame was displayed for a total duration of 550 ms, in an orientation randomly chosen from a set of nine angles between −45° and +45° in intervals of 11.25°. The stimulus line was presented 250 ms after the onset of the frame, and shown for 50 ms. After the frame disappeared, a mask consisting of randomly flashed lines with features similar to the frame lines was presented for 250 ms to wipe out iconic memory (Enns & Di Lollo, 2000). Next, the screen remained blank for 1.5 s and the subject was cued to rotate the head to upright, in alignment with the vertical headrest. Then, the visual frame reappeared, always at 0°, and a probe line was presented for 50 ms. Using a button press, the subject had to indicate whether the probe line was oriented clockwise (CW) or counterclockwise (CCW) with respect to the remembered stimulus line. After the response was given, the screen turned dark gray for 1.5 s to avoid carry-over effects of the frame orientations to the next trial. 

The stimulus line was presented at 0° orientation (i.e., vertical) in 80% of the trials; the other 20% were catch trials in which the line orientation was randomly drawn from a normal distribution (M = 0°, SD = 5°). The orientation of the probe line varied from trial to trial, and was determined dynamically with the psi-marginal adaptive procedure (Kontsevich & Tyler, 1999; Prins, 2013). There were 48 trials and 12 catches for each frame orientation, randomly interleaved, yielding a total of 540 trials. Trials were divided into 10 blocks with self-paced breaks in between, amounting to about 1 hr in total. Before the experiment began, subjects completed a 5-min training block to get used to the task, and to practice the optimal timing of their head movements to the sound cues. 

Stationary task

In order to predict performance in the updating task, we determined the subjects' percept of gravity direction at stationary 30° CW head tilt. This was used to predict the perceived direction of gravity before the head movement. Using the same range of frame angles as in the updating task, subjects indicated whether they perceived the stimulus line as tilted CW or CCW with respect to gravity. The orientation of the line varied from trial to trial following a psi-marginal adaptive staircase procedure (Kontsevich & Tyler, 1999; Prins, 2013). There were 48 trials per frame angle, randomly interleaved, resulting in a total of 432 trials split over eight blocks. In the same manner, we also tested the subjects' percept of gravity direction with the head and frame upright, in a single block of 48 trials. This measurement was used to predict the perceived direction of gravity after the head movement, in the updating task. Since the frame after the head movement in the updating task was always upright, perceived direction of gravity was only measured in this stationary task with the frame upright. The stationary and updating tasks were tested on different days. Subject 10 performed the stationary session first, for practical scheduling reasons. In all other subjects, the updating task came first. 

Data analysis

For each visual frame angle in the updating task, we determined at which orientation the probe line was perceived as equal to the orientation of the stimulus line. This orientation in degrees, also called the point of subjective equality (PSE) between the two lines, was taken as a measure of how a subject perceived the original line, but updated for the intervening head rotation. The individual percept was determined by taking the ratio of CW and CCW responses to each probe angle, and then fitting a cumulative Gaussian psychometric function through these ratios. 

The mean of the Gaussian was the value of the probe orientation at which the probability of a CW response is 50%. Thus, the mean was equal to the PSE for a given visual frame angle. The PSE was estimated for each frame angle separately, and the PSEs as a function of frame angle represented a subject's error pattern. 

We used a linear regression model to reveal the relative contributions of two possible reference frames to how visual orientation is represented in the brain. Line orientation could be represented and updated with respect to perceived gravity G, or to the visual frame F. The individually measured perceived gravity with the head tilted (G₀), the head upright (G₁), and the frame angle at encoding (F₀) and retrieval (F₁) were used to predict the error ε in the updating task (Equation 1). The measurement of G₀ serves as an estimate of the perceived direction of gravity at frame angle F₀, before the head movement, thus when the head is roll-tilted. The measurement G₁ is used as an estimate of the perceived direction of gravity after the head movement, thus when the head is upright. The frame angle after the head movement, F₁, was always 0°, and therefore G₁ contributes only a single data point per subject.  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\bf{\alpha}}\)\(\def\bupbeta{\bf{\beta}}\)\(\def\bupgamma{\bf{\gamma}}\)\(\def\bupdelta{\bf{\delta}}\)\(\def\bupvarepsilon{\bf{\varepsilon}}\)\(\def\bupzeta{\bf{\zeta}}\)\(\def\bupeta{\bf{\eta}}\)\(\def\buptheta{\bf{\theta}}\)\(\def\bupiota{\bf{\iota}}\)\(\def\bupkappa{\bf{\kappa}}\)\(\def\buplambda{\bf{\lambda}}\)\(\def\bupmu{\bf{\mu}}\)\(\def\bupnu{\bf{\nu}}\)\(\def\bupxi{\bf{\xi}}\)\(\def\bupomicron{\bf{\micron}}\)\(\def\buppi{\bf{\pi}}\)\(\def\buprho{\bf{\rho}}\)\(\def\bupsigma{\bf{\sigma}}\)\(\def\buptau{\bf{\tau}}\)\(\def\bupupsilon{\bf{\upsilon}}\)\(\def\bupphi{\bf{\phi}}\)\(\def\bupchi{\bf{\chi}}\)\(\def\buppsy{\bf{\psy}}\)\(\def\bupomega{\bf{\omega}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\begin{equation}\tag{1}\hskip-8pt\varepsilon \left( {{F_0},{F_{\rm{1}}}} \right){\rm{\ }} = {\beta _G}\left( {{G_{\rm{1}}}\left[ {{F_{\rm{1}}}} \right]{\rm-}{G_0}\left[ {{F_0}} \right]} \right){\rm{\ }} + {\beta _F}\left( {{F_{\rm{1}}}-{F_0}} \right){\rm{\ }} + {\beta _0}\end{equation}

 

The regression weights β_G, β_F, and β₀ were estimated with a least-squared error criterion. If a regression weight differed significantly from 0, the corresponding predictor contributed in a relevant way to the errors in the updating task. A weight of 1 would mean that the line orientation is kept within the corresponding reference frame across the update. A weight of 0 means that the corresponding reference frame did not contribute to the measured error pattern. 

Results

We studied the reference frame computations that underlie the updating of visual orientation across intervening head roll. Figure 2A illustrates the probability of a CW response as a function of probe line orientation in a single subject (S6), and the fitted psychometric curves, separately for the nine initial frame orientations. Circle size represents the number of trials for a given probe line orientation. The psychometric fits provide the point of subjective equality (marked by a square), which is where the probe orientation causes an equal proportion of CW and CCW responses. If subjects were to update the initial line orientation perfectly for the intervening head and frame rotation, the psychometric curve should reflect a step function, centered at zero. This is clearly not the case, suggesting an updating error of which the size varies depending on the orientation of the frame at initial line presentation. Figure 2B shows the updating error (±SD) as a function of the initial frame orientation (negative values correspond to CCW rotations), showing a cyclic modulation, with an offset reflecting the mean updating bias. 

Figure 2

 

(A) Psychometric results (data + best-fit curve) of a typical subject (S6) in the updating task. Dots show the line orientations that were tested, with dot size depicting the number of trials with that line orientation. F₀ is the frame orientation at presentation of the stimulus line. Squares show the updating error (PSE of the curve). (B) Response error shows a cyclic modulation with the orientation of initial frame orientation F₀. Shaded area, ±1 SD.

A similar pattern of updating errors was found in all 10 subjects (Figure 3A, orange lines). The peak-to-peak amplitude of the modulations ranged from 1.9° in S3 to 5.9° in S2. The updating bias (mean of the errors) varied between −6.0° for S4 and −2.7° for S3. The average pattern across subjects is shown in Figure 3B, showing a peak-to-peak amplitude of 2.7°, and a mean bias of −4.1°. 

Figure 3

 

(A) Measured (orange squares and lines, shaded area; ±1 SD) and predicted (purple squares and lines) updating performance as a function of initial frame orientation. (B) Averaged group performance (orange) superimposed on model prediction (purple) in the updating task.

Which reference frame underlies this pattern of updating errors? As pointed out in the Introduction, if the orientation of the line is stored in a gravity-based frame of reference, the corresponding memory will be affected by the perceived direction of gravity when viewing both the stimulus and probe line. Thus the gravity-based model predicts that the updating error of line orientation is related to the difference in subjective distortion of the gravity frame when storing and retrieving the line orientation from memory. 

To set up the predictions of this model, we measured for each subject the subjective perception of gravity direction in both the stimulus (storing in memory) and probe (retrieving from memory) conditions. Figure 4A shows the results, separately for each subject. In the stationary condition, when the head is tilted, the percept of gravity varies as a function of the frame orientation (black squares). Each subject shows a sine-like modulation of the frame effect, although there are some differences regarding the size of the effect and the frame angle that shows the peak of the illusion (Goodenough, Oltman, Sigman, Rosso, & Mertz, 1979; Wenderoth, 1974). Note that both define the prediction of the updating model. When the frame was upright (but head tilted) the average response error was around zero (M = 0.4°, SD = 1.6°, see Figure 4B). When both the head and frame were upright, there were small, individual biases in the perceived direction of gravity, ranging from −1.5° for S2 to −0.3° for S4 (M = −1. 0°, SD = 0.4°). 

Figure 4

 

(A) Perceived direction of gravity in the stationary task with the head tilted (black) and with the head upright (gray square) for all subjects. Error areas / bars; ±1 SD. (B) Averaged group data.

We used the responses in the stationary task to predict the updating errors, if subjects were to rely on a gravity-based reference frame for updating line orientation. This model (see Methods) provided a very good fit (R² = 0.89, F = 53.98, p < 0.001). 

Alternatively, if line orientation is stored relative to the visual frame, in object-centered coordinates, we would expect the updating errors to be related to the amount of intervening frame rotation (see Methods). We tested this by fitting a linear model, which provided a nonsignificant fit (R² = 0.04, F = 0.3, p = 0.6). 

We also fitted a combined linear model, predicting the updating error pattern using a weighted contribution of both reference frames in combination with a general offset. This model (purple squares in Figure 3) was able to explain more than 50% of the variance in 8 out of 10 subjects (R² values in Figure 3). The power of this model is further emphasized at the group level, explaining 96% of the variance (Figure 3B). At the group level, the adjusted R² (adjusted for the extra free parameter in the two-factor model) was 0.96 (F = 70.82, p < 0.001), which was larger than for both one-factor models (R² = 0.04 for visual frame only, 0.89 for perceived gravity only), supporting the combined model. At the individual level we found similar results for for 5 out of 10 subjects. 

The major contribution of the gravity-based reference frame to the updating performance is clearly reflected in the magnitude of the regression weight β_G (Figure 5, top panel). A weight of 1 means that a subject did not compensate for the change in perceived gravity between final head/frame orientation in the probe phase and initial head/frame orientation in the stimulus phase. For nine subjects, β_G was significantly positive (group values across all subjects: β_G = 1.04, t = 11.66, p < 0.001). 

Figure 5

 

Regression weights for perceived gravity (β_G), visual frame (β_F), and the intercept (β₀). Parameter values were fitted simultaneously, for each subject separately (S1–S10) and for the pooled group data (Group). Error bars, ±1 SD. *p < 0.05 and **p < 0.01.

The regression weight β_F revealed that the frame had only a minor contribution to the updating errors (Figure 5, middle panel). We found a small but significant effect on the group updating error (β_F = −0.01, t = −3.31, p = 0.016), but not at the level of individual subjects. The β_F value was negative, in the same direction as the tilt of the visual frame, indicating that subjects did not fully compensate for the intervening frame rotation. 

In general, the subjects' responses were systematically biased in the direction of the intervening head rotation (CCW), as reflected by the negative value of β₀ at the group level (β₀ = −1.83°, t = −8.72, p < 0.001). Of note, this bias in the updating results is not seen in the results of the stationary task. As explained in more detail in the Discussion, this offset may reflect an unknown bias in the mapping of the line from retinal coordinates to gravity-based coordinates. 

Discussion

We used an orientation updating task with intervening head rotations to test whether subjects code line orientation in an object-based reference frame or an egocentric reference frame based on perceived gravity. In order to quantify these reference frames, we used a stationary task to measure subjects' individual perceived direction of gravity under the influence of visual context. The performance in the stationary task served to predict the errors if a gravity-based reference frame was involved in the updating task. 

In the updating task, subjects memorized the orientation of a line inside a tilted frame while their heads were roll-rotated 30° CW. Then, they turned their heads upright, and judged the orientation of a second line, presented inside an upright frame, relative to the orientation of the memorized line. Errors in the updating task could largely be explained by the distortion of the gravity frame at the time of memorizing the orientation, and the distortion at the time of viewing the probe line. This is evidence that line orientation is kept in coordinates relative to perceived gravity, and is updated in this reference frame during intervening head rotation. We further found that a small part of the updating errors could be explained with the visual frame as underlying reference frame, which had a small but statistically significant effect in the combined model, albeit only at the group level. 

Our paradigm is based on the rod-and-frame illusion, showing that a vertical line is perceived as tilted when surrounded by a tilted visual frame (Beh et al., 1971; Witkin & Asch, 1948). This bias, which modulates sinusoidally with the orientation of the square frame, is interpreted as a bias in the internal representation of gravity direction (Figure 4). Li and Matin (2005) have shown that a square frame is not essential: A single peripheral line also causes a rod-and-frame effect. Mittelstaedt (1983) showed that subjects tend to localize their subjective gravity vector towards the direction of their own longitudinal head axis. The brain possibly interprets a tilted frame as an ambiguous head tilt signal, and thereby a biased gravity direction signal (Matin & Li, 1994). Because the percept of gravity relates sinusoidally to the orientation of the frame, it can predict the sinusoidal pattern of errors when the orientation of a visual line is memorized and updated in this frame. Previous work by Van Pelt et al. (2005) supports this notion. Their subjects made saccades to memorized targets in the frontal plane after an intervening head and body roll rotation. They showed that the direction errors of the saccade were closely related to the amount of subjective distortion of gravity direction at both the initial and final tilt angle, rather than to the amount of intervening rotation. 

In a model proposed by Vingerhoets et al. (2008), the bias can be explained by smaller weighting for otolith information when the head is tilted, and more reliance on visual information, leading to larger errors. Although in this study, we cannot directly compare head-tilt versus head-upright responses, the strong and consistent periodic errors in the stationary task (peak-to-peak distance: 3.2°) and updating task (peak-to-peak distance: 3.5°) can be explained by the stronger reliance on visual information when the vestibular signal is noisier with the head tilted (Burns & Blohm, 2010; De Vrijer, Medendorp, & Van Gisbergen, 2009; Tarnutzer, Bockisch, Straumann, & Olasagasti, 2009). 

In the updating task, the perceived gravity-based modulation was accompanied by an overall offset (mean across subjects: −4°, see Figure 3), which is much larger than could be expected based on the results in the stationary task (see Figure 4). A small part of this offset can be explained by the individual biases of subjects, which were present in the orientation judgments with both the head and the visual frame upright. These individual biases were negative for all subjects, but less than 1°, perhaps due to an aftereffect of prolonged CW head tilt in the previous blocks, leading to a spatial bias (mean offset: −1°) opposite to the head tilt (Day & Wade, 1966). The remaining difference of the offset (about 3°) requires a different explanation, and follows from experimental considerations. Figure 4 shows the perceived (or internal) direction of gravity, which modulates with the orientation of the frame. This modulation of the internal direction of gravity is exploited in the updating task, when testing for updating based on a gravity-based reference frame. In the updating task, an earth-vertical line is flashed, and subjects have to remember and update its orientation across head rotation. What is important to realize here is that knowing the internal direction of gravity, say 5° CW relative to actual gravity, does not mean that the orientation of the earth-vertical line is conversely coded as 5° CCW relative to internal gravity. This relative orientation follows from the nonlinear mapping of the line from retinal coordinates (when viewing it) to coordinates based on internal gravity (when storing it). As a result, the relative orientation could involve an unknown bias (e.g., due to uncorrected ocular counterroll; see de Vrijer et al., 2009; Otero-Millan & Kheradmand, 2016) and further varies with the frame-induced modulation of internal gravity. Both the bias and modulation of stored line orientation are revealed when the gravity-based reference is used in the updating task, as seen in Figure 3. 

We found that 89% of the variance in orientation error in the updating task can be explained by a gravity-based reference frame. Although we did not vary the direction and magnitude of the head rotation, any contribution of a head-centered reference frame is likely to be a relatively minor contribution to the coding of line orientation in this task. In that case, when subjects retrieve the line orientation from memory after the CCW head rotation, this memory is biased into the same direction, resulting in a negative offset. This explanation is in agreement with previous studies, showing a tendency for subjects to underestimate the magnitude of active self-displacement (Medendorp, Van Asselt, & Gielen, 1999) and active self-rotation (Jürgens, Boss, & Becker, 1999). To test this further, one would need to systematically vary the direction and magnitude of head rotations in an orientation updating task, and determine which part of the offset can be explained by errors in the estimation of head movement. 

Model fits revealed that an object-based reference frame could also contribute to spatial updating. This speaks to the notion that the brain defines line information in multiple reference frames in parallel, perhaps depending on their reliability (McGuire & Sabes, 2009; Tramper & Medendorp, 2015). In this study, the visual frame had only a minor contribution to the updating performance. Since the frame is only briefly visible during the stimulus and the probe phase, subjects may perceive it as two distinct objects rather than a single frame that could be used as a reference. We expect that subjects would rely more on the frame if it remained visible during the rotation, resulting in a larger contribution of the frame to the updating errors. 

At the neurophysiological level, a recent study reported gravity orientation tuning in the thalamus (Laurens, Kim, Dickman, & Angelaki, 2016), which could provide an allocentric reference for coding line orientation. Other work has shown that parieto-insular cortex, which receives projections from the thalamus, is involved in vestibular-based perception of verticality (Brandt & Dieterich, 1999). Functional magnetic resonance imaging (Walter & Dassonville, 2008) and transcranial magnetic stimulation (Lester & Dassonville, 2014) studies have provided evidence for involvement of the right superior parietal lobule in the integration of visual contextual information in the perceived gravity reference frame. It is possible that the locus of our effect is based on these areas in the dorsal stream, in accordance with a previous study that found effects of visual illusions in this pathway (De Brouwer, Smeets, Gutteling, Toni, & Medendorp, 2015). 

Conclusion

We found consistent errors in the coding of line orientation with a head movement intervening between encoding and retrieval. These orientation errors were cyclically modulated by a tilted visual frame surrounding the line. These observations originate from errors in perceived gravity with respect to the physical direction of gravity, induced by the tilted frame. In addition, line orientation was anchored to the visual frame itself, suggesting the simultaneous coding of line orientation in perceived gravity- and object-based frames of reference. 

Acknowledgments

NN was supported by a TopTalent grant from the Donders Institute. CFD's research is funded by the Netherlands Organization for Scientific Research (NWO-Vidi-452-12-009; NWO-Gravitation 024-001-006; NWO-MaGW 406-14-114; NWO-MaGW 406-15-291), the Kavli Foundation, the Centre of Excellence scheme of the Research Council of Norway – Centre for Biology of Memory and Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, the National Infrastructure scheme of the Research Council of Norway – NORBRAIN, and the European Research Council (ERC-StG RECONTEXT 261177; ERC-CoG GEOCOG 724836). WPM was supported by grants from The Netherlands Organization for Scientific Research (NWO-VICI: 453-11-001) and the European Research Council (EU-ERC-283567). 

Commercial relationships: none. 

Corresponding author: Nynke Niehof. 

Email: [email protected]. 

Address: Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.