**Abstract**:

**Abstract**
This study examines how people deal with inherently stochastic cues when estimating a latent environmental property. Seven cues to a hidden location were presented one at a time in rapid succession. The seven cues were sampled from seven different Gaussian distributions that shared a common mean but differed in precision (the reciprocal of variance). The experimental task was to estimate the common mean of the Gaussians from which the cues were drawn. Observers ran in two conditions on separate days. In the “decreasing precision” condition the seven cues were ordered from most precise to least precise. In the “increasing precision” condition this ordering was reversed. For each condition, we estimated the weight that each cue in the sequence had on observers' estimates and compared human performance to that of an ideal observer who maximizes expected gain. We found that observers integrated information from more than one cue, and that they adaptively gave more weight to more precise cues and less weight to less precise cues. However, they did not assign weights that would maximize their expected gain, even over the course of several hundred trials with corrective feedback. The cost to observers of their suboptimal performance was on average 16% of their maximum possible winnings.

*cues*. The challenge for an organism is to integrate the information provided by different cues to estimate important environmental properties with high precision

^{1}. The fundamental nature of this problem is reflected in the large literature on cue integration in sensory psychology (see Jacobs, 2002; Landy, Banks, & Knill, 2011, for brief reviews) and in cognitive psychology (see Slovic & Lichtenstein, 1971, for extensive review). The most important findings have been that observers (correctly) give more weight to more precise cues and that the precision of their estimates can approach the maximum precision possible for any rule of integration.

*environmental noise*) or within the nervous system (due to

*sensory noise*). In judging the location of a faint star, for example, one source of uncertainty is atmospheric fluctuations that affect the signal before reaching the retina, while another source of uncertainty is variance in processing the signal within the nervous system after reaching the retina. Figure 1 schematizes the factors that could potentially affect the precisions of cues (precision = reciprocal of variance) to some fixed environmental property

*x*. The dashed line marks the division between the external world and the organism's internal sensory world. The measurement of each cue may be perturbed by additive Gaussian environmental noise $\epsilon ie$ with mean 0 and variance

^{2}$Vie$, additive Gaussian sensory noise $\epsilon is$ with mean 0 and variance $Vis$, or both noise processes. In the general case, each cue available to the organism

*X*,

_{i}*i*= 1, … ,

*n*is perturbed by the sum

^{3}of the two sources of noise $\epsilon ie$ + $\epsilon is$, and this sum is itself a Gaussian random variable with mean

*x*, variance

*V*= $Vie$ + $Vis$, and precision

_{i}*π*= 1/

_{i}*V*. Importantly, and as shown in the Theory section, cues with higher precision should be given more weight when making an optimal decision.

_{i}*i*= 1,…,

*n*(e.g., Busemeyer, Myung, & McDaniel, 1993; Gigerenzer & Goldstein, 1996), the sensory literature tends to focus on integration of deterministic cues where the source of uncertainty is primarily internal to the organism: $Vis$ ≫$Vie$,

*i*= 1,…,

*n*(see Trommershäuser, Körding, & Landy, 2011). Furthermore, while cognitive studies typically address how people integrate a variable numbers of cues that are available simultaneously or sequentially, sensory studies are typically limited to how people integrate just two or three co-occurring cues.

*x*, was the environmental property of interest. On each trial, observers were given seven different cues

*X*,

_{i}*i*= 1, …, 7 as to the best location for the well by seven distinct surveying companies. Each cue was presented as a tick mark that flashed briefly to indicate a spatial location along the horizontal line.

*x*but with different variances

*V*,

_{i}*i*= 1, …, 7. The underlying variance of each cue was determined by presentation order. The common mean of the Gaussians (the true best location for the well) was varied at random from trial to trial. At the end of each trial, observers were provided with visual feedback indicating the true best location for the well. Observers in our study could potentially use this feedback to track the precision of each surveying company and adjust their weighting of the cues accordingly.

*sequential cue integration task*has the same formal structure as typical sensory cue integration tasks, it differs from typical sensory tasks in important respects. Unlike typical sensory integration tasks where the source of cue uncertainty is sensory, the location cues in this task have negligible and uniform sensory uncertainty. Instead, the key imprecision of cues in this study is external to the nervous system and dictated by the experimenter. Furthermore, unlike many sensory integration tasks where there tends to be enough information in a single trial for the organism to elicit an independent estimate of the precision of each cue (Ernst & Bülthoff, 2004), in this study the precision of each cue must be learned through repeated exposures across trials. As such, we are also interested in the learning rate of cue precisions and consequent weights.

*one*of the cues and ignore all of the remaining cues (the “veto” rule in Bülthoff & Mallot, 1988, or the “take the best” strategy in Gigerenzer & Goldstein, 1996). Even worse, the choice of which cue to follow may vary from trial to trial. In order to evaluate these possibilities, we compared human performance to an observer who uses only the single most precise cue. If participants outperform this baseline strategy, we infer that they are integrating cues

*effectively*(Boyaci, Doerschner, & Maloney, 2006). “Effective cue integration” indicates that the observer's estimation precision exceeds that of the single most precise cue, implying that the observer is integrating information from at least two cues.

*n*independent cues to some unknown value of interest,

*x*. Each cue

*X*,

_{i}*i*= 1, …,

*n*is an independent Gaussian random variable with common mean

*x*and possibly distinct variances

*V*,

_{i}*i*= 1, …,

*n*. We define the precision of each cue to be the reciprocal of its variance

*π*= 1/

_{i}*V*. A more variable cue is less precise, while a less variable cue is more precise. For the purposes of our experiment, we consider a case where the cues are unbiased, independent random variables. Oruç, Maloney, and Landy (2003) consider a more general case where the cues may be correlated, while Scarfe and Hibbard (2011) consider a case where the cues may be biased.

_{i}*x̂*=

*f*(

*X*

_{1}, …,

*X*) and taking the result

_{n}*x̂*to be an estimate of

*x*(see Oruç et al., 2003). We impose one restriction, however, that the expected value of the estimate be

*unbiased*:

*E*[

*x̂*] =

*x*. Among the unbiased rules of integration, we seek the one that maximizes the precision of the resulting estimate or, equivalently, minimizes its variance.

*n*cues with weights

*w*

_{1}+ … +

*w*= 1, as specified below. The resulting estimate

_{n}*x̂*in Equation 1 is unbiased Moreover, since the cues are independent random variables, the precision of the resulting estimate is (Oruç et al., 2003) where the choice of weights

*w*that maximizes precision can be shown to be (Landy et al., 1995) By substituting Equation 4 into Equation 3, we find that the precision of the resulting estimate, when integrating cues optimally, is simply the sum of the precisions of the individual cues:

_{i}*π*(

*x̂*) =

*π*

_{1}+ … +

*π*. The resulting estimate must therefore have a higher precision than any of the individual cues (Oruç et al., 2003, appendix), and the ideal observer who integrates cues optimally will always achieve a greater precision than an observer who uses only one of the cues, even if it's the one with greatest precision. Any other unbiased rule of integration, linear or nonlinear, will result in estimates whose precisions are less than that of the weighted linear rule of integration in Equations 1 and 4 (Oruç et al., 2003, appendix).

_{n}*n*= 2 and the two cues

*X*

_{1}and

*X*

_{2}have variances 2 and 1, respectively. Then their precisions are π

_{1}= 0.5 and π

_{2}= 1, and the optimal weights assigned by Equation 4 are

*w*

_{1}= 1/3 and

*w*

_{2}= 2/3. The more precise cue gets twice the weight of the less precise one. The precision of the estimate resulting from optimal cue integration is the sum of the precisions: 1.5. The variance of the resulting estimate is simply the reciprocal of the sum of the precisions: 1/1.5 = .666, which is less than that of either of the individual cues and, in fact, the lowest variance possible for any unbiased estimator.

*w*

_{1}=

*w*

_{2}= 0.5. Then the precision of the resulting estimate based on Equation 3 is: Even though this choice of equal weights did not reach the maximum possible precision of 1.5, the observer has achieved a greater precision than could be achieved by relying on any one cue alone. Boyaci et al. (2006) refer to this sort of cue integration as

*effective cue integration*: the precision of the observer's estimate is greater than the precision of any single cue: We can be certain that an observer who satisfies this condition is in fact integrating cues (Oruç et al., 2003). While optimal cue integration entails effective cue integration, one can integrate cues effectively without integrating them optimally.

*optimal cue integration*), and also test whether they integrate cues to achieve a greater precision than that of the most precise cue (

*effective cue integration*).

^{2}= 19,200) is five times greater than the variance of the best cue (61.97

^{2}= 3840). This means that the worst cue has one-fifth the precision of the best cue, and, consequently, it should ideally receive only one-fifth of the weight that the best cue receives (to maximize precision and expected gain).

*w*assigned to each of the seven cues

_{i}*X*using the following equation where

_{i}*x̂*denotes the observer's estimate and

*ε*∼ Φ(0,

*V*) is Gaussian random error with mean 0 and variance

*V*. The error term

*ε*captures any sensory or judgment uncertainty while the

*bias*term captures any chronic tendency that the observer might have to aim either to the left or to the right away from the best location for the well.

*ŵ*,

_{i}*i*= 1, …, 7 should show no patterned deviation from the optimal weights computed in Equation 4 and plotted in Figure 3B.

*ŵ*,

_{i}*i*= 1, …, 7 and the corresponding optimal weights

*w*,

_{i}*i*= 1, …, 7 shown in Figure 3B:

*i*= 1,…, 7 presented on the

*k*trial and the true best location for the well

^{th}*x*that is given as feedback. This “perfect memory observer” could then estimate the observed precision of each cue based on the first

^{k}*N*trials by maximum likelihood using the following equation

^{4}and then update the weights of the cues using these estimated precisions in place of the true

*π*in Equation 4. We computed this ideal learner by Monte Carlo simulation using 1000 runs and compared the rate of learning in this model to the learning performance of human observers.

_{i}*efficiency*, which is the ratio between their expected reward and the maximum reward possible. We computed the maximum reward of the ideal observer by Monte Carlo simulation using 1 million runs.

*ŵ*for each cue changes over the course of the experiment. These weights were elicited from data in non-overlapping blocks of 50 trials. Figure 4A (blue axes) shows results for the decreasing precision condition, while Figure 4B (red axes) shows results for the increasing precision condition.

_{i}*ŵ,w*) (i.e., the summed squared error between the estimated cue-weights of the ideal learner and the optimal cue-weights shown in Figure 3B) as a function of number of trials observed with 95% confidence intervals. The ideal learner converges to almost optimal in fewer than 100 trials (1 block = 50 trials).

*SEM*)

*of observers'*Δ(

*ŵ*,

*w*) in non-overlapping blocks of 50 trials, separately for the decreasing precision condition (blue) and for the increasing precision condition (red). While the ideal learner converges to optimal very rapidly, observers' estimated weights did not converge to optimal even after 600 trials. We fit lines to observers' data in Figure 5 and found no significant downward trend in either condition (decreasing precision condition:

*t*(9) = −0.53,

*p*> 0.05; increasing precision condition:

*t*(9) = −1.12,

*p*> 0.05). This indicates that while observers learned very rapidly to bring Δ(

*ŵ*,

*w*) in the direction of optimal, they were not able to further reduce Δ(

*ŵ*,

*w*) as the experiment progressed, even by the end of 600 trials. (See Figure A2 for individual results.)

*ŵ*as a function of temporal cue order for the last 500 trials in each condition (decreasing precision in blue, increasing in red). The dashed lines in corresponding colors are the weights that maximize precision from Figure 3B. Optimally, the most precise cue should receive five times more weight than the least precise cue, with the intermittent cues decreasing or increasing linearly in weight depending on the condition (Figure 3B). We fit regression lines to each set of weights using the following equation and plot the estimated fit as solid lines in blue (for decreasing precision condition) and red (for increasing precision condition).

_{i}*m̂*of all observers' trend lines are significantly negative in the decreasing precision condition (Mean = −0.017,

*SD*= 0.013,

*t*(9) = −4.29,

*p*= 0.002) and significantly positive in the increasing precision condition (Mean = 0.022,

*SD*= 0.024,

*t*(9) = 2.97,

*p*< 0.05). As the slopes in both conditions were significantly different from 0, we can rule out the possibility that observers were using the unweighted average of the cues as their estimate.

*m*= ±0.0317, all of the observers' estimated slopes are significantly greater in the increasing precision condition than in the decreasing precision condition,

*t*(9) = 5.57,

*p*< 0.001. This indicates that, on average, observers correctly assigned more weight to the more precise cues irrespective of their presentation order, though the differential weighting of the cues was not as great as the 5:1 range that is needed to be optimal.

*t*(9) = 11.15,

*p*< 0.001) and in the increasing precision condition (

*t*(9) = 4.78,

*p*= 0.001). This indicates that observers in our task integrated cues effectively: they used more than just the single most precise cue. However their estimation precisions were significantly lower than the maximum precision of the ideal observer (who integrates all cues commensurate with their individual precisions) both in the decreasing precision condition (

*t*(9) = −24.95,

*p*< 0.001) and in the increasing precision condition (

*t*(9) = −16.11,

*p*< 0.001). Thus, they were far from integrating all cues optimally. The errors bars mark which observers in each condition were less than optimal (all observers in all conditions) and which were

*effectively*integrating cues (all observers in the decreasing condition; all but P3, P8, P10 in the increasing condition).

*three*cues.

*This is the first demonstration we know of that observers effectively integrate more than two cues at a time*.

*SEM*) of each group's estimation precision in non-overlapping blocks of 50 trials. Observers' estimation precision gradually increased within each session (particularly in session 1). However, while there is a general trend toward improvement, observers' estimation precision is significantly lower than the maximum ideal precision of 4.2.

*efficient*they were at earning bonus money. Remember that observers were instructed to make as much money as possible. Any deviation from optimal performance in learning or in integrating cues could only reduce one's expected reward. The ideal observer's chance of winning $1 on any given trial was 75.9% (

*SD*= 18.2) once they correctly assessed cue precisions. On the other hand, the mean of observers' chances across the last 500 trials of each condition was 64% (

*SD*= 26.89), which means that their efficiency was 0.84. Observers thus lost about 16% of all bonus money available to them. For comparison, the expected chances of an observer who made use only of the single most precise surveying company (single most precise cue) was 53.4% (

*SD*= 31.2) for an efficiency of only 0.7. If one were to integrate the estimates of the two most precise surveying companies in an optimal fashion, one would still only obtain an expected chance of 64.3% (

*SD*= 26.05) for an efficiency of 0.85, which is only slightly better than what observers achieved.

*effective*(i.e., better performance than the single most precise cue) while still falling short of

*optimal*performance. While all observers demonstrated effective cue integration, all observers fell short of optimal cue integration even after extensive training (with feedback). This result is notable since a large body of work has accumulated suggesting that human observers can integrate cues in a near-optimal fashion even without corrective feedback (Jacobs, 2002; Landy, Maloney, Johnston, & Young, 1995). Indeed, a continuing tension in the field is that studies of cognitive decision-making and learning routinely find sub-optimal performance compared to an ideal observer (Berretty, Todd, & Martignon, 1999; Gigerenzer & Goldstein, 1996; Kahneman, Slovic, & Tversky, 1982), while observers in naturalistic visual perceptual tasks show evidence of near-optimal performance (Ernst & Banks, 2002).

*effective cue integration*in that all observers integrated information from more than one cue in guiding their estimates. Moreover, our results are the first findings we know of to demonstrate that observers effectively integrate more than two cues at a time.

*Psychological Science**,*21(4), 560–567. [CrossRef] [PubMed]

*Perception**,*28(2), 217–242. [CrossRef] [PubMed]

*. (pp. 235–254). New York: Oxford University Press.*

*Simple heuristics that make us smart**, 6(2):2, 106–118, http://www.journalofvision.org/content/6/2/2, doi:10.1167/6.2.2. [PubMed] [Article] [CrossRef]*

*Journal of Vision*

*Journal of the Optical Society of America A**,*5(10), 1749–1758. [CrossRef]

*Psychological Science**,*4(3), 190–195. [CrossRef]

*Journal of Experimental Psychology**,*54(3), 180–187. [CrossRef] [PubMed]

*. (p. 79). New York: Oxford University Press.*

*The Oxford dictionary of statistical terms**. Boca Raton, FL: Chapman & Hall.*

*An introduction to the bootstrap*

*Nature**,*415, 429–433. [CrossRef] [PubMed]

*Trends in Cognitive Sciences**,*8(4), 162–169. [CrossRef] [PubMed]

*Psychological Review**,*103(4), 650–669. [CrossRef] [PubMed]

*Cognitive Science**,*34, 10–50. [CrossRef] [PubMed]

*Trends in Cognitive Sciences**,*6(8), 345–350. [CrossRef] [PubMed]

*, 10(7):1213. [Abstract] [CrossRef]*

*Journal of Vision**. New York: Cambridge University Press.*

*Judgment under uncertainty: Heuristics and biases**. (pp. 5–29). New York, NY: Oxford University Press.*

*Sensory cue integration*

*Vision Research**,*35(3), 389–412. [CrossRef] [PubMed]

*Journal of Experimental Psychology**,*64(5), 482–488. [CrossRef]

*Vision Research**,*43(23), 2451–2468. [CrossRef] [PubMed]

*Spatial Vision**,*10(4), 437–442. [CrossRef] [PubMed]

*. (pp. 64–99). New York: Appleton-Century-Crofts.*

*Classical conditioning II: Current research and theory**, 11(7):12, 1–17, http://www.journalofvision.org/content/11/7/12, doi:10.1167/11.7.12. [PubMed] [Article] [CrossRef] [PubMed]*

*Journal of Vision*

*Organizational Behavior and Human Performance**,*6, 649–744. [CrossRef]

*. New York, NY: Oxford University Press.*

*Sensory cue integration**, 7(7):8, 1–11, http://www.journalofvision.org/content/7/7/8, doi:10.1167/7.7.8. [PubMed] [Article] [CrossRef] [PubMed]*

*Journal of Vision*

*Journal of Experimental Psychology**,*82(1, Pt. 1), 52–63. [CrossRef]

^{1}We use the term “precision” to refer to the reciprocal of variance. More generally, the term “precision matrix” refers to the matrix inverse of a covariance matrix (Dodge, 2003). The term “reliability” is sometimes used as a synonym for precision in the sensory cue integration literature (e.g., Backus & Banks, 1999).

^{3}To simplify the presentation, we ignore the possibility of correlation between cues and between environmental and sensory noise. See Oruç et al. (2003).

^{4}Many different learning algorithms based on the observed errors could be used instead. However, the point we will make is that the performance of an ideal observer converges to close to ideal in as few as 100 trials. The lack of convergence we observe in human data renders moot any precise comparison between human learning and ideal.