Abstract
For integration of information from vision and haptics to be effective, the brain should only combine information referring to the same object. This could be achieved by considering the similarity of signals in the two sensory channels. For example, if there is a large conflict between two size estimates it is unlikely that they originate from the same object. Humans are adept at using tools such as pliers, however, which systematically change the relationship between (seen) object size and the opening of the hand. Here we show that the brain takes this change into account when using a tool, and integrates “conflicting” signals near-optimally. Subjects judged the separation between two planes in a two-interval forced-choice task. We first measured discrimination thresholds in visual- and haptic-alone conditions to predict performance when both cues were available. We then measured visual-haptic thresholds when subjects grasped the objects using a virtual tool, which simulated pliers. By moving the pivot, the pliers either magnified or minified the haptic signal. Haptic size at the hand was constant. Large and small (visual) objects were used, resulting in “correct-gain” conditions, in which the visual size and the opening of the hand, although different, were appropriate given the “gain” of the tool, as well as “incorrect-gain” conditions. In the “correct-gain” conditions, discrimination performance was better than single-cue performance, and was close to the optimal predictions. In the “incorrect-gain” conditions performance was at the level of, or worse than single-cue performance, suggesting that visual and haptic signals were not integrated. We conclude that the brain can take into account, dynamically, changes in the scaling of haptic and visual signals introduced by tools, and appropriately combines signals that were caused by the same object, independent of conflicts between the visual size and the opening of the hand.
Supported by EPSRC and ORSAS.