Abstract
Human visual systems integrate information across space to improve estimates about the environment. But the optimal strategy for integrating spatially correlated signals in natural scenes is not fully understood. We derive the optimal strategy for integrating local estimates from neighboring locations in space to compute a “global” estimate of a local scene property. Our framework is inspired by ideas from the cue combination literature. In cue combination, single-cue estimates are combined in a reliability-based weighted average. It is often assumed i) that measurement noise is independent, ii) that single-cue signals come from the same source (i.e. they are perfectly correlated), and iii) that single-cue estimates are unbiased. Here, we generalize these ideas to the problem of optimally pooling multiple local estimates from different nearby spatial locations. We assume i) that local estimates provide unbiased information and ii) that signals are only partially correlated at neighboring locations (as they are in natural images). The model specifies that local estimates should be linearly combined with weights that depend both on the reliability of local estimate and on the pattern of signal correlation across space. We applied this model to the problem of estimating 3D tilt. First, from a database of co-registered photographs and groundtruth distance maps of natural scenes, we sampled a large set of stimuli and validated that the model assumptions are approximately correct in natural scenes. Then, we computed the Bayes-optimal local tilt estimates from local image cues in the natural images. Finally, we combined the local estimates into global estimates using the optimal pooling model described above. Optimal global tilt estimates are better than optimal local tilt estimates at recovering groundtruth tilt in natural scenes. We also examine whether global estimates better predict human performance. The model provides a principled way to integrate local signals in real-world settings.
Acknowledgement: NIH R01-EY011747