Reweighting visual cues by touch

Abstract

It is well established that if multiple cues provide information about the same quantity, the information from these cues is combined by weighting each cue by the inverse of its variance. This implies that cue weights are determined by the cue variances only. However, this view is challenged by studies that showed that feedback about the actual value can induce changes in the cue weights when the feedback is consistent with one cue but not the other. We developed a paradigm that allowed us to measure the time course of this reweighting. Subjects placed an object flush onto a slanted surface. Monocular and binocular cues provided information about the slant and could be inconsistent with one another. Subjects received haptic feedback about whether they had oriented the object correctly when the object contacted the surface. This feedback was consistent with either the monocular or the binocular information. We found that the weight given to the visual cue that was consistent with the feedback increased relatively fast, leading to a mean weight change of 0.18 after 52 conflict trials. Thus, unless the haptic feedback somehow influences the reliability of the individual visual cues, the cue weights are not fully determined by the cue variances but also depend on the accuracy of each cue.

Introduction

Our percept of the environment is based on information from multiple sensory sources. Often, different sensory modalities or different cues within the same modality provide information about the same environmental quantity, such as the orientation of a surface. Many studies (e.g., Ernst & Banks, 2002; Knill & Saunders, 2003; van Beers, Sittig, & Denier van der Gon, 1999; van Beers, Wolpert, & Haggard, 2002) have shown that the brain combines the information from different sources in a statistically optimal way in the sense that the variance of the combined estimate is minimized. The equations for optimal integration can be derived in various ways, such as maximum likelihood estimation (Ernst & Banks, 2002; Ghahramani, Wolpert, & Jordan, 1997), Bayesian inference (Knill & Pouget, 2004; Ma, 2010), using information theory (Ghahramani et al., 1997), or by calculating the weighted average that minimizes the variance (Ghahramani et al., 1997; Landy, Maloney, Johnston, & Young, 1995; Oruç, Maloney, & Landy, 2003). As long as the uncertainty of each source is Gaussian and there is no informative prior information, all approaches lead to the same equations for the combined estimate. This optimal estimate can be interpreted as the weighted average of the individual estimates, where each source is weighted by the inverse of its variance. 

Although the framework of statistically optimal integration makes intuitive sense and is supported by a large amount of data (for reviews, see Ernst & Bülthoff, 2004; Knill & Pouget, 2004; Ma & Pouget, 2008), it has a counterintuitive aspect: The weights are fully determined by the variances of the individual cues and are independent of their accuracies. The accuracy is, however, often relevant when interacting with the environment. Consider, for instance, the situation that one wishes to place an object flush onto a slanted surface. Suppose that, before the object is placed, two visual cues provide information about the surface slant and that cue 1 has a bias of +4 deg and a variance of 5 deg², whereas cue 2 has a bias of −2 deg and a variance of 10 deg². The weight given to cue 1 will be twice as large as the weight of cue 2, which results in a combined estimate with a bias of +2 deg. If the object is placed according to this estimate, a systematic error of 2 deg will be made. This error will be readily available through haptic feedback when one feels that one edge of the object touches the surface first. How does such feedback influence subsequent perception of the surface orientation? One possibility is that the feedback will be used to recalibrate the individual cues so as to remove their biases (Atkins, Jacobs, & Knill, 2003). Another possibility is that the weights of the individual cues are changed such that the bias of the combined estimate is reduced. This study focuses on this second possibility. 

We conducted an experiment in which we examined reweighting of visual cues based on feedback about their accuracies while recalibration of the cues was prevented. The task was similar to the example in the previous paragraph: place objects flush onto a slanted surface, while two visual cues provide information about its slant. We created an artificial conflict between the visual cues and made the haptic feedback consistent with the same cue in all trials. We examined whether this would lead to an increase of the weight of the cue that was consistent with the haptic feedback. To prevent recalibration of the individual cues, the sign of the conflict varied randomly between trials with an average conflict of zero. 

Note that the two visual cues play a different role in this task than the haptic information. Since only vision provides information about surface slant until the object contacts the surface, the task of orienting the object will be based on visual information alone. In contrast, when the object touches the surface, haptic information is also available. This haptic information can be used to obtain a better estimate of the surface orientation, but its primary role is that it provides feedback about how one has performed because the part of the object that first touches the surface directly signals in which direction, if any, the object slant differs from the actual surface slant. We assume that such feedback can be used to adjust the relative weighting of the two visual cues. To understand how this could work, one should realize that the object's slant will generally differ somewhat from the visually estimated slant of the target surface due to inevitable movement variability (van Beers, Haggard, & Wolpert, 2004). As a result, it is not possible to unambiguously assign the perceived error at contact to misestimating the visual slant, as the error could also reflect inaccuracies in bringing the object in the desired orientation. However, one could observe the pairwise correlations of the haptic feedback with each of the visual signals. If one of the visual signals regularly deviates substantially from the haptic and the other visual signal, it has a higher probability to contain an error. The weight of this cue can then be reduced. This is only possible if the system has some degree of access to information from the individual visual cues, which has been shown to be the case for the cues considered here (Muller, Brenner, & Smeets, 2007). 

Three studies (Atkins, Fiser, & Jacobs, 2001; Ernst, Banks, & Bülthoff, 2000; Jacobs & Fine, 1999) followed a similar approach as introduced above, albeit for other tasks than placing an object on a slanted surface. Ernst et al. (2000) estimated the weights of disparity and texture as slant cues before and after subjects were exposed to a training phase in which they moved their hand over a surface while receiving haptic feedback that was consistent with one of the cues. They found that the weight of the cue that was consistent with the haptic feedback was larger after the training phase than before training. Atkins et al. (2001) and Jacobs and Fine (1999) obtained similar results for the weighting of visual cues to depth. 

These studies demonstrate that the relative weighting of different cues may not always be fully determined by their reliabilities (variances), as optimal integration assumes, but may also be influenced by feedback about the actual value. This finding has received surprisingly little attention, and many questions on this phenomenon remain unanswered. Here, we focus on the time course of the weight changes. All the previous studies used a long training phase (240 trials, taking 30–45 min in Ernst et al., 2000; 750 trials, divided over 2 days in Jacobs & Fine, 1999; and 504 trials, spread over 3 days in Atkins et al., 2001), and the weights were not estimated during the training phase but only after training. It is therefore unclear how the weights changed over time. Is this reweighting a slow process that requires hundreds of trials or is it a fast process that had converged long before the end of the training phase in the studies mentioned above? Here, we developed a new paradigm that allowed us to estimate how the weights evolved during the training phase. The task was similar to the example given above and is a modification of the task first developed by Knill (2005) to estimate weights of visual cues: subjects placed cylinders flush onto a slanted surface. The slant of the surface was defined by two visual cues (monocular and binocular) and there could be a conflict between these cues. We estimated the weights of the monocular and binocular cues from the slant of the cylinder just before it contacted the surface. Since the speed of reweighting may depend on the size of the conflict between the cues, we estimated the reweighting curves for two magnitudes of the conflict (20 and 10 deg). 

Methods

Subjects

Eight subjects (seven females, one male, between 23 and 32 years old) participated in both experiments. None of them reported any sensory or motor deficits, and all had normal or corrected-to-normal vision, with a stereo acuity better than 100 arcsec (assessed by the Stereo Fly testing package), and were unaware of the purposes of the study. 

Apparatus

We used the same setup as van Mierlo, Louw, Smeets, and Brenner (2009; see Figure 1). Subjects sat or stood behind a 40 cm by 40 cm surface that we will call the table. This table could rotate around its central axis oriented in the left–right direction. The rotation was driven by a computer-controlled motor. Rotating the surface led to different values of slant, where the slant was defined as 0 deg when the table was horizontal, and it was positive when the edge furthest from the subject was higher than the nearest edge. The rotation axis was about 60 cm from the subject's chest and about 37 cm below eye level. Subjects held a cylinder with a height of 6.0 cm and a diameter of 9.2 cm in their right hand. There was a fixed horizontal surface to the right of the table. This surface was 6.5 cm above the rotation axis of the table, and it had a 2-mm-deep indentation in the shape of the cylinder's base. Subjects placed the cylinder within this indentation at the start of each trial. There, the center of the cylinder was 28 cm to the right of the subjects' midsagittal plane and about 60 cm in front of their chest. 

Figure 1

 

(Left) Top view and (right) side view of the experimental setup (not to scale).

Subjects could not see the real table and cylinder but only virtual renderings thereof. The three-dimensional virtual environment was created by presenting different images to the left and right eyes using a combination of two CRTs (1096 × 686 pixels, 47.3 × 30.0 cm, 160 Hz) and mirrors (see Figure 1). This allowed us to manipulate the slant of the virtual table surface independently for monocular and binocular cues, as explained below. 

We used an Optotrak 3020 system (Northern Digital) to record the location and orientation of the cylinder and of the table and the locations of the two eyes at 250 Hz. For this purpose, three infrared emitting diodes (IREDs) were attached to the cylinder and four to the table. Another three IREDS were attached to a bite board that subjects held in their mouth during the entire experiment. The bite board did not restrain the subject's head because it was not attached to any external apparatus. We calibrated the positions of the eyes relative to the IREDs on the bite board before the experiment. The Optotrak output was sent to two Apple G5s that calculated a new image for each eye for each refresh (6.25 ms) of the CRT monitors, based on the latest Optotrak output. The delay between an actual position change and a displayed position change was about 20 ms. 

Stimuli

The virtual table was a square with sides of 10 cm located at the center of the real table. It consisted of a red and green checkerboard pattern of 4 by 4 squares (see Figure 2). The virtual cylinder had the same dimensions as the real cylinder, and it had 14 white and black stripes along its side and a green top and bottom with 14 black dots (see Figure 2). The shapes of the projections of the checkerboard squares on the CRT screens provided monocular information about table slant. Motion parallax as a result of small head movements, which also provides information about the slant (Louw, Smeets, & Brenner, 2007), was always consistent with this monocular information. The differences between the computer images for the two eyes provided binocular information about table slant (binocular disparity). The same sources of information were also available for the cylinder. 

Figure 2

 

Impression of the subjects' view during the experiment, with (left) the table surface and (right) the cylinder.

The monocular and binocular information about the virtual cylinder were always consistent: All cues indicated the actual cylinder orientation. In contrast, the monocular and binocular cues indicated either the same (no-conflict trials) or different (conflict trials) slants for the table. In order to render different slants for the two cues, we determined how a square surface with a slant defined by the monocular cues would look for a single (cyclopean) eye, and then rendered images for the two eyes that (on average) provide this retinal image, while the disparity had the value corresponding to the desired binocular slant (Knill & Saunders, 2003). This is equivalent to simulating a non-conflict trapezoid with the slant specified by binocular disparity that deforms slightly when the head moves. Subjects were not informed that there could be conflicts between the cues. 

Procedure

Within an experiment, there were two possible slant values, denoted by s ₁ and s ₂ (see below for the values in each experiment). The slant of the monocular cue (s _m) and that of the binocular cue (s _b) were chosen from these values. This produced four possible stimuli: [s _m, s _b] = {[s ₁, s ₁], [s ₁, s ₂], [s ₂, s ₁], [s ₂, s ₂]}, where the second and the third are conflict stimuli, and the first and the last are no-conflict stimuli. The slant of the actual table was the same as either s _m or s _b (see below). At the beginning of a trial, the motor positioned the table at this slant. This positioning happened in three movements (at least one in each direction) to prevent subjects from deducing the table slant from the sound made by the motor. Due to limitations in the control of the motor, the obtained slant could differ slightly from the target value (it varied with a standard deviation of about 0.4 deg). The difference was determined online in every trial, and it was added to the target values of the monocular and binocular slant cues, so that the actual table slant agreed exactly with at least one of the visual cues and the conflict between the cues had the same magnitude in all conflict trials. Trials in which the obtained slant differed more than 1 deg from the target value were discarded from the weight estimation analysis (87 of 4992 trials, 1.74%). 

Once the table was positioned and the subject had placed the cylinder at the start location, the virtual table surface and the virtual cylinder were shown. After a random delay between 200 and 500 ms, a beep instructed the subject to start the response. The instruction was to place the cylinder in a single continuous movement flush onto the table surface, roughly at the center of the virtual checkerboard. It was more important that the orientation matched that of the table than that it was placed exactly at the checkerboard's center. If subjects responded before or within 100 ms after presentation of the beep, the movement was considered to have started too early. The trial was then stopped and the subject had to start again. The virtual table surface and cylinder remained visible until the end of each trial (see Figure 2 for an impression of the subjects' view). Subjects typically held the cylinder on the table for less than 500 ms before moving it back to the start location. Well after the cylinder left the table (1650 ms after it contacted the table), the display turned black and the motor started to bring the table to its orientation for the next trial. 

A session consisted of 156 trials and took about 20 min. In each block of six trials, each conflict stimulus was presented twice and each no-conflict stimulus once, in a random order. In the first half of the session, the actual table slant, and therefore the haptic feedback, was consistent with one of the cues (monocular or binocular). In the second half, it was consistent with the other cue. Each subject performed two sessions of each experiment, one for each feedback order. 

In Experiment C20, the conflict between the slants was 20 deg, with s ₁ = −5 deg and s ₂ = 15 deg. Since subjects looked down at an angle of about 30 deg (relative to horizontal), these values correspond to slants relative to the line of sight of about 25 deg and 45 deg, respectively. In Experiment C10, the conflict between the slants was 10 deg, with s ₁ = −5 deg and s ₂ = 5 deg. The order in which subjects performed the four sessions (Experiment C10 or C20, starting with feedback consistent with monocular or binocular cues) was approximately counterbalanced across subjects. Each session was conducted on a different day. Subjects were unaware that there could be a conflict between the visual cues, that there were only two values of the table slant within a session, and that there was a change in the feedback during the session. 

Analysis

The goal of the analysis was to estimate the weights given to the monocular and binocular cues from the way in which subjects placed the cylinder onto the table. As a measure for the perceived slant, we used the slant of the cylinder just before it made contact with the table. We will refer to this as the contact slant. 

To find the contact slant, we calculated the distance between the (nearest point of the) cylinder and the (actual) table surface from the Optotrak output for each cylinder trajectory. The contact slant was defined as the slant on the last frame before the distance fell below 1 mm. Trials in which not all IREDs were seen by the Optotrak around the time of contact were discarded from analysis (58 of 4992 trials, 1.16%). We adjusted each contact slant by subtracting the (small) difference between the actual table slant and its target value in each trial, so that all contact slants could be related to the two target slants (−5 deg and either 5 or 15 deg). 

We estimated the cue weights from a set of observed contact slants using maximum likelihood estimation. We assumed that the perceived slant for each stimulus was a weighted average of the slants perceived on the basis of the monocular and binocular cues: 

${\widehat{s}}_{ij}=w_{\mathrm{m}}{\widehat{m}}_i+(1-w_{\mathrm{m}}){\widehat{b}}_j,$

(1)

where

$\widehat{s}$

_ij is the perceived slant for stimulus [s _m, s _b] = [s _i , s _j ], i, j ∈ {1, 2}, w _m is the weight of the monocular cue, 1 − w _m is the weight of the binocular cue, and

$\widehat{m}$

_i and

$\widehat{b}$

_j denote the perceived slant on the basis of the monocular and binocular cues, respectively. If we make the (rather unrestrictive) additional assumption that

$\widehat{m}$

₁ −

$\widehat{b}$

₁ =

$\widehat{m}$

₂ −

$\widehat{b}$

₂ (i.e., that the difference between the perceived slant on the basis of the monocular and binocular information is the same for both slant values), the following relations hold for the slant perceived for the four stimuli (see 1): 

$\{\begin{array}{c}{\widehat{s}}_{12}=w_{\mathrm{m}}{\widehat{s}}_{11}+(1-w_{\mathrm{m}}){\widehat{s}}_{22}\\ {\widehat{s}}_{21}=(1-w_{\mathrm{m}}){\widehat{s}}_{11}+w_{\mathrm{m}}{\widehat{s}}_{22}.\end{array}$

(2)

With N _ij being the number of responses (contact slants) for stimulus [s _m, s _b] = [s _i , s _j ] and S _ij being the sum of these slants, maximum likelihood estimation leads to the monocular weight estimate (see 1 for the derivation): 

$w_{\mathrm{m}\mathrm{,}\mathrm{M}\mathrm{L}\mathrm{E}}=\frac{(N_{12}+N_{22})(N_{11}{S}_{21}-N_{21}{S}_{11})+(N_{11}+N_{21})(N_{12}{S}_{22}-N_{22}{S}_{12})}{2N_{12}{N}_{21}(S_{22}-S_{11})+(N_{12}+N_{21})(N_{11}{S}_{22}-N_{22}{S}_{11})+(N_{11}-N_{22})(N_{21}{S}_{12}+N_{12}{S}_{21})}.$

(3)

Note that this method of estimating the weights is insensitive to cue-specific biases, to a general response bias, and to a linear compression of the range of responses (as has been reported for similar tasks; Knill, 2005; Knill & Kersten, 2004). This is because the weight w _m in the model assumed in Equation 2 is insensitive to linear changes of the perceived slants

$\widehat{s}$

_ij . 

We used Equation 3 to estimate the weight for each block of six consecutive trials for each session. We entered all responses in conflict trials within this block (four, minus the number of rejected ones) into Equation 3, as well as all responses in no-conflict trials in the entire session (52, minus the rejected ones). Including all no-conflict trials was justified by the observation that the responses for these stimuli generally did not change during an experimental session (see Results section). Including them all gave the estimator more power than including only the two in the six-trial block for which the weight was estimated. 

Note that the design of the experiments made it impossible for subjects to reduce their errors by simply biasing their responses in the opposite direction to the previous error(s) (van Beers, 2009), because the two conflict stimuli led to errors in opposite directions. For the same reason, errors could not be reduced by recalibration of the individual cues either (Atkins et al., 2003). 

Results

Experiment C20

Figure 3A shows the mean contact slants as a function of the trial number in Experiment C20. The left panel shows the slants when the haptic feedback was consistent with the binocular cue in the first half of the session and with the monocular cue in the second half. The right panel shows the results for the other feedback order. On average, subjects were quite accurate in the no-conflict trials (continuous green and red curves), as their contact slants were close to the slant specified by the visual cues (−5 deg and +15 deg, dashed horizontal lines). Importantly, the responses to these stimuli did not vary over time during the experiment (except, perhaps, for the red curve in the right panel; see below for an explanation). In contrast, the responses to the conflict stimuli (dashed blue and magenta curves) did vary during the session. For instance, the contact slants for the magenta stimulus (s _m = −5 deg, s _b = 15 deg) in the left panel were approximately constant in the first half but decreased during the second half. Since the haptic feedback was consistent with the monocular cue in this second half, this change had the effect of reducing the difference between the final cylinder slant and the slant of the table. At the same time, the contact slants for the blue stimulus (s _m = 15 deg, s _b = −5 deg) increased, which also had the effect of reducing the difference between cylinder and table slant. Equivalent effects can be seen in the right panel, where the changes occurred in the opposite order because the feedback order was reversed. 

Figure 3

 

Mean contact slants in Experiment C20 as a function of trial number for the two feedback orders (indicated at the top). Shaded areas represent across-subject standard errors. Horizontal dashed lines (red and green) give the slants of the individual cues. The vertical dashed line indicates the transition between the two feedback phases.

Figure 4A shows the mean weight of the monocular cue as a function of the trial number as estimated from the contact slants. The blue curve corresponds to the left panel in Figure 3 (feedback consistent with the binocular cue in the first half and with the monocular cue in the second half) and the red curve to the right panel in Figure 3. This figure shows that the monocular weight increased when the haptic feedback was consistent with the monocular cue (red curve in first half and blue curve in second half), whereas it decreased or was constant when the feedback was consistent with the binocular cue (blue curve in first half and red curve in second half). This agrees with the findings of earlier studies (Atkins et al., 2001; Ernst et al., 2000; Jacobs & Fine, 1999) that haptic feedback about the actual value can change the relative weighting of visual cues such that the visually perceived value agrees better with the felt one. 

Figure 4

 

(A) Mean monocular weights as a function of trial number in Experiment C20. Shaded areas represent across-subject standard errors. The blue curve corresponds to the left panel of Figure 3 and the red curve to the right panel. (B) Mean slope of the monocular weight as a function of trial number in the second half of the session in Experiment C20. (C) Mean RMS of differences between contact slants in consecutive trials with the same stimulus in Experiment C20. (D) The difference in slopes of the monocular weight as a function of trial number in the second half of the session for the two feedback orders (monocular minus binocular), plotted as a function of the ratio of the RMS values for the conflict and no-conflict trials. Each dot represents a subject.

We examined whether significant reweighting occurred by analyzing how the weight estimates changed during the second half of the experiment. The reason we restricted this analysis to the second half is that the first half was designed to control the state (i.e., the visual cue weights) of the subjects at the start of the second half. The state at the beginning of a new session could not be controlled as this state depends on the subject's history. As this history differed between subjects, some subjects could be expected to display considerable reweighting in the first half of a session, whereas others were expected to display little or no reweighting. In contrast, all subjects were expected to display substantial reweighting in the second half, because each subject experienced the same change in the cue with which the feedback was consistent. The second half of the session thus had maximal statistical power to detect reweighting, whereas that of the first half was substantially reduced. 

We found support for this argument in the data, as some relevant aspects of the subjects' history are known. For instance, there is one report that weights tend to be close to those at the end of a previous session in the same setup, even when that was conducted 24 h earlier (Ernst et al., 2000). An analysis of our data confirmed that this was also the case in this experiment (see 2). In addition to being influenced by the history of previous sessions, the history outside the setup may also play a role. The weights in our setup may, for instance, differ from those during everyday life. An analysis of the data from the first half of the sessions confirmed that this was the case: The weight of the monocular cue increased somewhat at the beginning of the session, regardless of the cue with which the feedback was consistent (see 2). Both effects made it impossible to obtain unbiased estimates of the reweighting we were interested in from the first half of the session. 

We used linear regression to analyze the weight changes during the second half of each session. For each subject, we regressed the monocular weight estimated for each block of 6 trials against the trial number. The mean slopes and their standard errors for the two feedback orders are shown in Figure 4B. The slope was significantly larger (p = 0.007, paired, one-tailed t-test) when the feedback was consistent with the monocular cue (blue) than when it was consistent with the binocular cue (red). The slope was significantly larger than zero (p = 0.042, one-tailed t-test) when the feedback was consistent with the monocular cue, but it was not significantly smaller than zero (p = 0.082, one-tailed t-test) when the feedback was consistent with the binocular cue. To examine whether the speed of reweighting depends on the cue with which the haptic feedback was consistent, we also tested whether the slopes were different after multiplying the slopes of one feedback phase by −1. This was not the case (p = 0.873, paired, two-tailed t-test). 

Before we can conclude that the haptic feedback led to changes in the monocular and binocular weights in this experiment, we will consider an alternative explanation. The cue conflict in this experiment was so large (20 deg) that subjects may have become aware that in many trials there was a conflict. If they recognized the conflict stimuli and discovered that there were only two different ones, they could have used trial-and-error strategies to find the table slant and, therefore, the desired response, for each conflict stimulus. Such a strategy would lead to apparent weight changes as observed, but this cannot be considered genuine reweighting. To examine whether subjects used this strategy, we analyzed the response variability in the conflict and no-conflict trials. In particular, the trial-and-error strategy would lead to incidental large differences in successive responses to the same conflict stimulus. This would not occur for no-conflict stimuli, as these would be recognized as such. We calculated the difference in the contact slant of successive trials of the same stimulus and calculated root mean square (RMS) values of these (the RMS was calculated from the differenced rather than the actual slants because differencing removes trends from the data, and trends are clearly present in the responses to the conflict stimuli due to the reweighting). This was done for each subject separately and for the trials in the second half of the session only. We then averaged the RMS values of the two conflict stimuli obtained in both sessions (with the different feedback orders) and did the same for the no-conflict stimuli. Figure 4C shows that the mean RMS values were significantly larger (p = 0.014, paired, one-tailed t-test) for the conflict stimuli than for the no-conflict stimuli. We also calculated the ratio of the RMS values for conflict and no-conflict stimuli for individual subjects. Figure 4D shows that subjects who had a large RMS ratio tended to have a large weight change (quantified as the difference between the slopes of the monocular weights for the two feedback orders), as the correlation between these measures was positive (0.54). This suggests that some subjects may have recognized the conflict stimuli and used trial-and-error strategies to determine their responses. Thus, the apparent reweighting that we found in this experiment could be the result of using cognitive strategies based on recognition of the conflict stimuli. We therefore do not have conclusive evidence for genuine reweighting in this experiment. 

We finally note that the trial-and-error searching for the correct response for the conflict stimuli is visible in the individual data of some subjects (see Figure 5). The subject in Figure 5B even displayed similar behavior to one of the no-conflict stimuli (with a slant of 15 deg, shown in red). The behavior of this subject explains why the red curve in the right panel of Figure 3 was lower in the second half than in the first half. 

Figure 5

 

Raw data (contact slant) of two subjects in Experiment C20, where the trial-and-error searching for the correct response is clearly visible in the second half. The subject whose data are shown in (B) even displayed this behavior for one of the no-conflict stimuli (the one marked in red).

Experiment C10

We applied the same analysis as above to the data of Experiment C10, in which the conflict was only 10 deg. Figure 6 shows the mean contact slants. The slants may look more variable here than in Experiment C20 (Figure 3), but that is a result of the different scaling of the vertical axes. The responses for the no-conflict stimuli (green and red curves) were quite stable during an experimental session. The slants for the conflict stimuli vary roughly in the same way over time as for Experiment C20 (Figure 3). The estimated weights of the monocular cue are shown as a function of the trial number for both feedback orders in Figure 7A. Although the changes are minimal in the first half, we see the same pattern as in Figure 4A in the second half: The monocular weight increased when the feedback was consistent with the monocular cue (blue) and it decreased when it was consistent with the binocular cue (red). 

Figure 6

 

Mean contact slants in Experiment C10. Same format as Figure 3.

Figure 7

 

(A) Mean monocular weights as a function of trial number in Experiment C10. Shaded areas represent across-subject standard errors. The blue curve corresponds to the left panel of Figure 6 and the red curve to the right panel. (B) Mean slope of the monocular weight as a function of trial number in the second half of the session in Experiment C10. (C) Mean RMS of differences between contact slants in consecutive trials with the same stimulus in Experiment C10. (D) The difference in slopes of the monocular weight as a function of trial number in the second half of the session for the two feedback orders (monocular minus binocular), plotted as a function of the ratio of the RMS values for the conflict and no-conflict trials. Each dot represents a subject.

For the same reason as in Experiment C20, we only analyzed the slopes of the monocular weight estimates as a function of the trial number in the second half of the session. The results are shown in Figure 7B. The slopes were significantly larger (p = 0.038, paired, one-tailed t-test) when the feedback was consistent with the monocular cue (blue) than when it was consistent with the binocular cue (red). The slope was significantly smaller than zero (p = 0.033, one-tailed t-test) when the feedback was consistent with the binocular cue, but it was not significantly higher than zero (p = 0.086, one-tailed t-test) when the feedback was consistent with the monocular cue. The speed of reweighting did not depend on the cue with which the haptic feedback was consistent, because the slopes were not significantly different (p = 0.414, paired, two-tailed t-test) when the slopes of one feedback phase were multiplied by −1. 

All the results so far in this experiment are very similar to those of Experiment C20. Indeed, a 2 (Experiment C20 vs. Experiment C10) by 2 (Cue with which feedback was consistent: monocular vs. binocular) factor repeated measures ANOVA on the slopes showed that only the main factor Cue was significant (F _(1,7) = 10.73; p = 0.014). The effect of Experiment (F _(1,7) = 0.078; p = 0.788) and the interaction between Experiment and Cue (F _(1,7) = 0.230; p = 0.646) were not significant. This suggests that the weight changes were not only qualitatively but also quantitatively similar in both experiments. The critical question, however, is whether some subjects also recognized the conflict stimuli and used a trial-and-error strategy in Experiment C10. The smaller conflict of 10 deg is expected to make this harder than in Experiment C20. The RMS of the contact slant changes in successive trials of the same stimuli (Figure 7C) was not significantly different for conflict and no-conflict stimuli (p = 0.114, one-tailed t-test). One could argue, however, that it is not surprising that the difference between the RMS of conflict and no-conflict stimuli was smaller here than in Experiment C20, because the conflict was only half as large. If the significant reweighting found here arose because some subjects recognized the conflict stimuli and used trial-and-error strategies to find the correct response, subjects who showed a large RMS ratio should have had large weight changes. Figure 7D shows that this was not the case. The subjects with the largest RMS ratios (which were considerably smaller than the largest RMS ratios in Experiment C20) had small to mediocre weight changes, and the correlation between RMS ratio and weight change was even negative (−0.14). This demonstrates that subjects did not use trial-and-error strategies to find the correct response for the conflict stimuli in this experiment. In other words, the reweighting found in this experiment most likely reflects genuine reweighting of the visual cues. 

A final question is how the weights changed over time. Were they, for instance, a linear or an exponential function of time? The weight curves in the right half of Figure 7A show too much variability to address this issue. It is therefore not possible to draw conclusions about the exact shape of these curves. The slopes of the linear regressions of weight against trial number (Figure 7B) suggest that the weight changed by about 0.0035 per conflict trial. Over the 52 conflict trials in the second half of the experiment, this led to a weight change of about 0.18. We also examined after how many trials in the second half of the session the slopes were significantly different for the two feedback types. To this end, we repeated the analysis on the slopes for a variable number of blocks of six trials. This analysis showed that the difference became significant (p < 0.05) after 32 conflict trials. 

Discussion

We developed a new paradigm in which subjects were trained to place an object on a slanted surface, where the feedback about their performance was consistent with one visual slant cue but not the other. The novel element of this paradigm was that it allowed us to estimate how the weights that subjects assigned to each cue evolved over time during the training phase. Our results confirm earlier findings (Atkins et al., 2001; Ernst et al., 2000; Jacobs & Fine, 1999) that such feedback can change the weights of visual cues to slant. They also show for the first time that this reweighting is a relatively fast process in which weights change a substantial amount per trial. There is no indication that the speed of reweighting depends on the cue (monocular or binocular) with which the feedback is consistent. 

Validity of the experimental findings

Our weight estimates were derived from the contact slants in response to various visual stimuli. In doing so, we assumed that the perceived slant was a weighted average of the slant estimated from monocular and binocular information (Equation 1). In addition, we assumed that the difference between the perceived slant on the basis of the monocular and binocular information was the same for both slant values of the no-conflict stimuli. This is a rather unrestrictive assumption that, nevertheless, may not hold. However, even if it did not hold, it will only have led to biases in the weight estimates, not to misestimates of how the weights changed over time. 

The assumption that the perceived slant is a weighted average of the slant estimated from the individual cues is commonly made to estimate cue weights, but it neglects the potential role of prior knowledge or belief about the surface slant. Can the apparent reweighting that we found be an artifact of subjects using (Bayesian) priors? The simplest form of a prior could be that subjects assume that the slant is close to a certain value, for instance, zero (corresponding to a horizontal orientation) or the mean of the slants experienced in previous trials. If subjects adopted such a prior, it will have led to a misestimate of the visual weights because the prior acts as a third cue with a non-zero weight. However, as long as such a prior is constant during an experimental session, it will not affect the detection of weight changes and the sign thereof. Our conclusions will, therefore, not be affected by the possible presence of such a prior; only the numerical values of the weight estimates will be somewhat biased [the observation that the difference between the contact slants for the two no-conflict trials was only marginally smaller than the difference between their simulated slants (see Figures 3 and 6) implies that such a prior did not play a large role; as a result, the biases in the weight estimates cannot be large]. A more sophisticated type of prior will be discussed in the Nature of reweighting section. 

The design of our experiment and analysis made it impossible that the observed weight changes are due to recalibration of individual cues (Atkins et al., 2003), to cue-specific biases, to a general response bias, to a linear compression of the range of responses (Knill, 2005; Knill & Kersten, 2004), or to trial-by-trial error correction based on the error in the previous trial (van Beers, 2009). A remaining possibility is that subjects recognized the conflict stimuli and used trial-and-error strategies to search for the correct response for each conflict stimulus. To check whether this occurred, we analyzed the variability in the contact slants in response to conflict and no-conflict stimuli and found evidence for it in Experiment C20 but not in Experiment C10. The fact that this method detected an effect in Experiment C20 indicates that the method is effective. The fact that it detected no effect in Experiment C10, for which the results were otherwise very similar, suggests that the results of that experiment are not caused by subjects recognizing the conflict stimuli. 

Since there is no artifact that can explain the results of Experiment C10, we conclude that those results reflect genuine reweighting of visual cues to slant. 

Time course of reweighting

The aim of this study was to determine the time course of reweighting of visual slant cues by haptic information about the actual slant. Our new paradigm proved to be successful as it allowed us to estimate how the weights evolved during adaptation. Although this method allows one to estimate weights at a high temporal resolution, it does not produce smooth curves of cue weight as a function of time (see the right half of Figures 4A and 7A) due to the substantial response variability (mean standard deviation for the no-conflict stimuli: 2.2 deg). Our results therefore do not allow us to draw conclusions about the exact shape of the weight curves. 

We used linear regression to quantify the average rate at which the weights changed. We found an average weight change of 0.0035 per conflict trial. However, we do not claim that the weights changed over time in a linear way. It is in fact impossible that they keep on changing linearly in long experiments as weights are bound between 0 and 1. It is more likely that they will move toward an asymptote, for instance, in an exponential fashion. For this reason, it is hard to compare the speed of reweighting found here to that reported previously. For instance, Ernst et al. (2000) found a mean weight change of 0.09 (average for texture and disparity feedback) after a training phase of 240 trials, which amounts to a mean weight change of 0.0004 per training trial. This is about a factor of 10 slower than in our experiment. This difference may at least partly be due to the fact that the weights had stabilized before the end of Ernst et al.'s training phase. This is consistent with the finding that the total reweighting found by Ernst et al. was not larger than that found in the present study (it was in fact smaller). 

Our results thus demonstrate that reweighting of visual cues by touch is a relatively fast process. It is not necessary to train subjects for hundreds of trials as we found significant reweighting after only 32 conflict trials. Nevertheless, it would be interesting to perform longer training sessions to find out at which values the weights will asymptote and how long it takes before the asymptote is approached. The paradigm developed here is very suitable for such experiments. Future experiments could also focus on the question of how the relative reliability of the visual cues (Burge, Girshick, & Banks, 2010; van Beers et al., 2002) and of the feedback signal determine the initial speed of reweighting. 

Nature of reweighting

All issues raised at the end of the previous paragraph will depend on the, as yet unknown, nature of the reweighting. One possibility is that the cue that was not consistent with the feedback in the conflict trials was gradually disregarded because it agreed poorly with the feedback. This would resemble the finding that in multisensory cue integration attention or conscious effort (Block & Bastian, 2010; Canon, 1970; Warren & Schmitt, 1978) can cause an increase of the weight given to one modality relative to that of the other. Similar effects have not been reported for different cues within the same modality when the conflict between the cues was as small as considered here, but they have been found for larger conflicts that are so large that they cause bistability (van Ee, van Dam, & Brouwer, 2005). 

A second possibility is that the reweighting is related to the statistical co-occurrence of signals (Ernst, 2007). If the haptic feedback is observed to be consistent with, say, the monocular cue on many consecutive trials, the mere correlation between haptics and monocular information may lead to an increase of the monocular weight. Indeed, Ho, Serwe, Trommershäuser, Maloney, and Landy (2009) observed such a reweighting of a visual cue by touch for judging the depth of seen and felt bumps. This effect was, however, only observed for a minority of the subjects, whereas the effect reported here appears to be systematic across subjects. 

A third possibility is that the reweighting found here is related to changes in the prior beliefs that subjects had about the shape of the stimuli. If they assumed that a stimulus was a square, they may have interpreted its slant differently than if they assumed it was a trapezoid. Extensive exposure to the statistics of target object shapes (Knill, 2007a, 2007b; Seydell, Knill, & Trommershäuser, 2010), but not the statistics of currently viewed shapes (Muller, Brenner, Smeets, 2009a), can induce changes in a subject's prior belief about the relative proportions of object shapes. This can be elegantly modeled using a mixture of priors on object shapes, where a proportion of objects are assumed to be isotropic (square in our experiments) while another proportion are assumed to have a non-isotropic shape (non-square trapezoids in our experiments). Extensive exposure to the statistics of object shapes can change the internal estimate of the relative proportion of isotropic and non-isotropic objects in the environment. As a result, the estimated cue weights can change (Knill, 2007a, 2007b; Seydell et al., 2010). It is possible that the reweighting that we found reflects such a change of the prior of object shapes. Indeed, a zero haptic error corresponded to a square object in all trials when the feedback was consistent with the monocular cue, whereas it corresponded to a square in only one-third of the trials (the no-conflict stimuli) when the feedback was consistent with the binocular cue. Thus, feedback about the actual slant, rather than direct exposure to the statistics of object shapes (Knill, 2007a, 2007b; Seydell et al., 2010), could influence the weight given to the monocular slant cue by influencing a prior about object shape. 

A fourth possibility is that, after reweighting, the cue weights were still statistically optimal given their variances but that the variances of the visual cues had changed as a result of the haptic feedback in the conflict trials. This possibility is related to the issue of how the brain determines and represents the reliability of a cue. Several ideas about this have been put forward. These can be divided into two categories (Ernst & Bülthoff, 2004). The first category consists of ideas that assume that the reliabilities are learned from experience, for instance, on the basis of correlations among different cues (Jacobs, 2002) or by making use of cues to uncertainty (Barthelmé & Mamassian, 2010). Ideas in the second category assume that the brain estimates the reliability online during the perceptual judgment itself (Ernst & Bülthoff, 2004) and that it represents this reliability explicitly. The possibility that the haptic feedback in the conflict trials changed the variances probably falls in the first category. However, it has been demonstrated convincingly that the reliability of a cue is represented, and therefore estimated, in each individual trial (Ma, 2010), which strongly supports the second category. Moreover, Barthelmé and Mamassian (2010) performed an experiment to distinguish between the two categories and also found evidence in favor of the second category. These results suggest that it is unlikely that the reweighting we found is related to changes in the variances of the individual visual cues. 

A final possibility for the nature of reweighting is that the weights are not determined by the cue variances only but also by the accuracy of each cue. For instance, the brain could aim to use weights that minimize the mean squared error, a measure that is closely related to task performance. Since the mean squared error equals the sum of the variance and the squared bias, such weights would depend on both the precision and the accuracy of each cue. In the example in the Introduction section, this would lead to a weight of cue 1 of 0.43 rather than 0.67 that is optimal given the variances only. Our results are consistent with this idea because the cue that was not consistent with the feedback was strongly biased in each conflict trial and would, therefore, have been down-weighted. The perceptual studies that found evidence that cues are weighted by the inverse of their variance are also consistent with this hypothesis because subjects did not receive feedback about their performance in those studies so that no information about the bias was available. As a result, the mean squared error equaled the variance. More research is required to test this hypothesis. 

Implications of this study

The main conclusion from this and earlier (Atkins et al., 2001; Ernst et al., 2000; Jacobs & Fine, 1999) work is that the weights of individual cues are not fixed, but weight assignment is flexible, maybe indicating that they are not only determined by the relative precision of the cues. When available, feedback providing information about the actual value plays a role as well. This role has been neglected in most previous cue integration studies that were of a purely perceptual nature and did not provide feedback about the actual value of the estimated parameter. However, feedback is generally present, and important, when interacting with the environment. We speculate that statistically optimal integration where the weights are determined by the cue variances only is the method that the brain uses to combine information in the absence of feedback, but a more general theory may apply to the situation in which feedback is present. Unraveling this general theory is crucial for our understanding of cue integration at the computational and neural level (Gu, Angelaki, & DeAngelis, 2008; Ma, Beck, Latham, & Pouget, 2006; Ma & Pouget, 2008). 

Appendix A

Estimating the weights using maximum likelihood estimation

Applying maximum likelihood estimation to estimate the weights from the data on the basis of Equation 1 is problematic because different single-cue slant estimates,

$\widehat{m}$

₁,

$\widehat{m}$

₂,

$\widehat{b}$

₁,

$\widehat{b}$

₂, can lead to the same two-cue slant estimates,

$\widehat{s}$

₁₁,

$\widehat{s}$

₁₂,

$\widehat{s}$

₂₁,

$\widehat{s}$

₂₂. In addition, marginalizing over

$\widehat{m}$

₁,

$\widehat{m}$

₂,

$\widehat{b}$

₁, and

$\widehat{b}$

₂ is not possible in closed form, whereas marginalizing numerically is very time consuming or not accurate. We therefore opted to make one additional and fairly unrestrictive assumption: We assumed that the difference between the perceived slants based on only monocular and on only binocular information is the same for both slant values:

$\widehat{m}$

₁ −

$\widehat{b}$

₁ =

$\widehat{m}$

₂ −

$\widehat{b}$

₂. If we make this assumption, we have 

$\begin{array}{cc}& w_{\mathrm{m}}{\widehat{s}}_{11}+(1-w_{\mathrm{m}}){\widehat{s}}_{22}\hfill \\ \\ & =w_{\mathrm{m}}(w_{\mathrm{m}}{\widehat{m}}_1+(1-w_{\mathrm{m}}){\widehat{b}}_1)+(1-w_{\mathrm{m}})(w_{\mathrm{m}}{\widehat{m}}_2+(1-w_{\mathrm{m}}){\widehat{b}}_2)\hfill \\ \\ & =w_{\mathrm{m}}(w_{\mathrm{m}}{\widehat{m}}_1+(1-w_{\mathrm{m}}){\widehat{b}}_1-{\widehat{m}}_1)+w_{\mathrm{m}}{\widehat{m}}_1+(1-w_{\mathrm{m}})(w_{\mathrm{m}}{\widehat{m}}_2+(1-w_{\mathrm{m}}){\widehat{b}}_2-{\widehat{b}}_2)+(1-w_{\mathrm{m}}){\widehat{b}}_2\hfill \\ \\ & =w_{\mathrm{m}}{\widehat{m}}_1+(1-w_{\mathrm{m}}){\widehat{b}}_2+w_{\mathrm{m}}(1-w_{\mathrm{m}})({\widehat{b}}_1-{\widehat{m}}_1)+w_{\mathrm{m}}(1-w_{\mathrm{m}})({\widehat{m}}_2-{\widehat{b}}_2)\hfill \\ \\ & ={\widehat{s}}_{12}+w_{\mathrm{m}}(1-w_{\mathrm{m}})({\widehat{b}}_1-{\widehat{m}}_1+{\widehat{m}}_2-{\widehat{b}}_2)\hfill \\ \\ & ={\widehat{s}}_{12}.\hfill \end{array}$

(A1)

This is the first of the pair of Equation 2. The other equation follows similarly. Notice that our assumption

$\widehat{m}$

₁ −

$\widehat{b}$

₁ =

$\widehat{m}$

₂ −

$\widehat{b}$

₂ is much less restrictive than the assumptions that others (Greenwald & Knill, 2009; Greenwald, Knill, & Saunders, 2005; Knill, 2005) made in estimating weights for comparable experiments. Specifically, we do not assume that the perceived slant depends linearly on the slants specified by the cues or that the response bias is independent of the cue values. 

We proceed by applying maximum likelihood estimation assuming that Equation 2 holds for the relation between the perceived slants for the conflict and no-conflict stimuli at a given short time interval in which the weights can be considered constant. We further assume that the responses to a stimulus in that time interval are independent and identically distributed as a Gaussian with a fixed variance σ ², for both conflict and no-conflict stimuli (Muller, Brenner, & Smeets, 2009b). Let r _ij ^(t) be the response in trial number t ∈ 1, …, N _ij of stimulus [s _m, s _b] = [s _i , s _j ]. Then, the log likelihood of observing the responses is 

$\begin{array}{cc}& L=-\frac{1}{2}(N_{11}+N_{12}+N_{21}+N_{22})\left(\log (2\pi {\sigma }^2)\right)\hfill \\ \\ & -\frac{1}{2{\sigma }^2}[\sum _{t=1}^{N_{11}}{(r_{11}^{\left(t\right)}-{\widehat{s}}_{11})}^2+\sum _{t=1}^{N_{12}}{(r_{12}^{\left(t\right)}-w_{\mathrm{m}}{\widehat{s}}_{11}-(1-w_{\mathrm{m}}){\widehat{s}}_{22})}^2+\sum _{t=1}^{N_{21}}{(r_{21}^{\left(t\right)}-(1-w_{\mathrm{m}}){\widehat{s}}_{11}-w_{\mathrm{m}}{\widehat{s}}_{22})}^2+\sum _{t=1}^{N_{22}}{(r_{22}^{\left(t\right)}-{\widehat{s}}_{22})}^2].\end{array}$

(A2)

The maximum of the log likelihood can be found by setting the following derivatives to zero: 

$\frac{\partial L}{\partial w_{\mathrm{m}}}=0,\frac{\partial L}{\partial {\widehat{s}}_{11}}=0,\frac{\partial L}{\partial {\widehat{s}}_{22}}=0.$

(A3)

This leads to the following system of equations: 

$\{\begin{array}{c}{w}_{\mathrm{m}}(N_{12}+N_{21})({\widehat{s}}_{11}-{\widehat{s}}_{22})+N_{12}{\widehat{s}}_{22}-N_{21}{\widehat{s}}_{11}+S_{21}-S_{12}=0\\ \\ S_{11}+w_{\mathrm{m}}{S}_{12}+(1-w_{\mathrm{m}})S_{21}-{\widehat{s}}_{11}[N_{11}+N_{12}{w^2}_{\mathrm{m}}+N_{21}{(1-w_{\mathrm{m}})}^2]-{\widehat{s}}_{22}(N_{12}+N_{21})w_{\mathrm{m}}(1-w_{\mathrm{m}})=0\\ \\ (1-w_{\mathrm{m}})S_{12}+w_{\mathrm{m}}{S}_{21}+S_{22}-{\widehat{s}}_{22}[N_{22}+N_{12}{(1-w_{\mathrm{m}})}^2+N_{21}{w^2}_{\mathrm{m}}]-{\widehat{s}}_{11}(N_{12}+N_{21})w_{\mathrm{m}}(1-w_{\mathrm{m}})=0,\end{array}$

(A4)

where 

$S_{ij}=\sum _{t=1}^{N_{ij}}{r}_{ij}^{\left(t\right)}.$

(A5)

Solving this system of equations gives the maximum likelihood estimate: 

$\begin{array}{cc}& w_{\mathrm{m}\mathrm{,}\mathrm{M}\mathrm{L}\mathrm{E}}=\frac{(N_{12}+N_{22})(N_{11}{S}_{21}-N_{21}{S}_{11})+(N_{11}+N_{21})(N_{12}{S}_{22}-N_{22}{S}_{12})}{2N_{12}{N}_{21}(S_{22}-S_{11})+(N_{12}+N_{21})(N_{11}{S}_{22}-N_{22}{S}_{11})+(N_{11}-N_{22})(N_{21}{S}_{12}+N_{12}{S}_{21})}\hfill \\ \\ & {\widehat{s}}_{11,\mathrm{M}\mathrm{L}\mathrm{E}}=\frac{N_{12}{N}_{21}(S_{11}-S_{22})+N_{12}{N}_{22}(S_{11}+S_{21})+N_{21}{N}_{22}(S_{11}+S_{12})}{N_{11}{N}_{12}{N}_{21}+N_{11}{N}_{12}{N}_{22}+N_{11}{N}_{21}{N}_{22}+N_{12}{N}_{21}{N}_{22}}\hfill \\ \\ & {\widehat{s}}_{22,\mathrm{M}\mathrm{L}\mathrm{E}}=\frac{N_{12}{N}_{21}(S_{22}-S_{11})+N_{11}{N}_{12}(S_{22}+S_{21})+N_{11}{N}_{21}(S_{12}+S_{22})}{N_{11}{N}_{12}{N}_{21}+N_{11}{N}_{12}{N}_{22}+N_{11}{N}_{21}{N}_{22}+N_{12}{N}_{21}{N}_{22}}.\hfill \end{array}$

(A6)

 

Appendix B

Reweighting during the first half of the session

For each subject, we determined the slope of the regression of the monocular weight as a function of trial number for the data from the first half of each session. Figures B1A and B1C show the mean slopes for Experiments C20 and C10, respectively. We used the same color-coding (red and blue) for the two feedback orders as in Figures 4 and 7. Therefore, if reweighting in the first half of the session was equivalent (in size and direction) to that in the second half, one would expect similar results to those in Figures 4B and 7B but with the signs flipped because the feedback was now consistent with the other cue. As predicted, the slope was significantly larger (p = 0.006, paired, one-tailed t-test) when the feedback was consistent with the monocular cue (red) than when it was consistent with the binocular cue (blue) for Experiment C20. Note, however, that the slope in the latter case was not negative, as one could have expected, but slightly positive. We will come back to this point in the last paragraph of this appendix. For Experiment C10 (Figure B1C), both mean slopes were close to zero and not significantly different from one another (p = 0.237, paired, one-tailed t-test). 

Figure B1

 

Changes in weight during the first half of the session. (A) Mean slope of the monocular weight as a function of trial number in Experiment C20. Same format as Figure 4B. (B) The same slopes as in (A) but separated into the slopes for subjects for whom the feedback was consistent with the same cue as at the end of the previous session (Not switched) and those for subjects for whom the feedback was consistent with the other cue than at the end of the previous session (Switched). The numbers below the bars indicate the number of subjects contributing to the bar. (C) Same as (A) but for Experiment C10. (D) Same as (B) but for Experiment C10.

Ernst et al. (2000) reported that in their experiment, a subject's weights at the beginning of a new session were close to those at the end of that subject's previous session in the same setup. We examined whether a similar effect was present in our data. To this end, we assigned individual sessions to one of two groups. For the Not switched group, the cue with which the feedback was consistent in the first half of the new session was the same as the cue with which it was consistent in the last part of the previous session. For the Switched group, feedback was consistent with the other cue than at the end of the previous session. Sessions that were the subject's first session in this study could not be assigned to either of these groups; these sessions were not included in this analysis. If weights at the beginning of a new session are close to those at the end of the previous session, one would expect large reweighting during the first half of the new session for the Switched group and little or no reweighting for the Not switched group. 

Figure B1B shows the mean regression slopes of both groups in Experiment C20. As predicted, the slope was significantly larger (p = 0.007, one-tailed t-test) when the feedback was consistent with the monocular cue (red) than when it was consistent with the binocular cue (blue) for the Switched group, whereas the difference was not significant (p = 0.81, one-tailed t-test) for the Not switched group. Unexpectedly, both slopes tended to be positive rather than near zero for this group. We discuss this finding in the last paragraph of this appendix. Figure B1D shows the results for Experiment C10. All slopes were close to zero, and in neither case was the difference between the red and blue bars significant. The insignificant effect for the Switched group is even in the wrong direction. 

In summary, the results of this analysis for Experiment C20 support the idea that subjects started a new session with weights that were close to the values at the end of the previous session. The results of the analysis of Experiment C10 do not add further support to this idea, but they are not inconsistent with it either; they simply do not allow us to draw any firm conclusions. This could be a result of the fact that weight estimates in Experiment C10 are intrinsically less precise due to the smaller conflict size, which leads to a smaller statistical power. Another factor that may play a role is that the majority of subjects did not switch in Experiment C10 (see Figure B1D; the number of subjects in each group is indicated under each bar). 

In addition to the above-mentioned differences, this analysis revealed another effect. If we average the red and blue bars in each of the panels of Figure B1, the result is always positive. This means that there was an overall tendency to increase the weight of the monocular cue during the first half of the session. This unspecific increase in the weight of the monocular cue can also be seen in the left halves of Figures 4A and 7A, where the red curves rise, but the blue ones do not descend. There was no such unspecific increase in the weight of the monocular cue during the second half of the session (see Figures 4B and 7B). It is not obvious why the monocular weight increased in this way, but since it only occurred during the first half of the session, it might be caused by the transition from natural vision to viewing stimuli in a virtual-reality-like setup. With natural vision, subjects give a certain weight to binocular information. Once they start the experiment and notice that they make errors, they may ascribe these errors to the unnatural binocular information they receive and somewhat reduce the weight given to the binocular information. 

Acknowledgments

This research was supported by the Netherlands Organisation for Scientific Research (NWO Grants 452-02-007 and 453-08-004). 

Commercial relationships: none. 

Corresponding author: Robert J. van Beers. 

Email: [e-mail]. 

Address: Research Institute MOVE, Faculty of Human Movement Sciences, VU University Amsterdam, Van der Boechorststraat 9, 1081 BT Amsterdam, The Netherlands.