This special issue of Journal of Vision presents a collection of papers on sensorimotor processing and goal-directed movements, with special emphasis on how vision is employed in guiding hand movements. Until recently, research concerning sensory processing and research concerning motor control have followed parallel but independent paths. The partitioning of the two lines of research in practice partly derived from and partly fostered a bipartite view of sensorimotor processing in the brain – that a sensory/perceptual system creates a general purpose representation of the world which serves as the input to the motor systems (and other cognitive systems) that generate action/behavior as an output. On the one hand, this view has led researchers in perception to study how observers use sensory information to explicitly estimate various properties of objects and scenes without reference to how an observer might use that information to guide behavior. On the other, it has led researchers in motor control to treat the sensory input as an exact representation of the world.
Recent results from research on vision in natural tasks have seriously challenged this view, suggesting that the visual system does not generate a general-purpose representation of the world, but rather extracts information relevant to the task at hand (Droll, Hayhoe, Triesch & Sullivan,
2005; Land & Hayhoe,
2001). At the same time, researchers in motor control have developed an increasing understanding of how sensory limitations and sensory uncertainty can shape the motor strategies that humans employ to perform tasks. Moreover, many aspects of the problem of sensorimotor control are specific to the mapping from sensory signals to motor outputs and do not exist in either domain in isolation. Sensory feedback control of hand movements, coordinate transformations of spatial representations and the influence of processing speed and attention on sensory contributions to motor control are just a few of these. In short, to understand how human (and animal) actors use sensory information to guide motor behavior, we must study sensory and motor systems as an integrated whole rather than as decomposable modules in a sequence of discrete processing steps.
In recent years researchers have begun to focus more and more on sensorimotor integration. Besides the theoretical considerations alluded to above, two advances in the tools available to researchers have made it this possible. The first is the ready availability of equipment that allows experimenters to present complex visual stimuli in realistic 3D environments and to monitor motor behavior in real-time. The second, more profound, advance is the growing application of mathematical tools from statistical (including Bayesian) decision theory (Knill & Richards,
1996) and optimal control theory (Stengel,
1994)
. Indeed, statistical decision theory, which is now being widely applied to studies of visual perception, is fundamentally a theory of goal-directed actions based on uncertain sensory information (Ernst & Bülthoff,
2004; Maloney,
2002). With problems in both domains framed in a common mathematical language, it is much easier to think about perception and action together. As a consequence, we are seeing more work in sensorimotor integration drawing on insights gained from this type of modeling (Harris & Wolpert,
1998; Saunders & Knill,
2004; Todorov & Jordan,
2002; Trommershäuser et. al.,
2003).
Milner and Goodale (
1995) conjectured that visuomotor computations and visual perception rely on distinct neural substrates. Part of the argument for their conjecture is that motor tasks place unique computational demands on the visual system that are not relevant for most perceptual judgments. If we want to understand the visual computations that underlay motor behavior, we need to study vision in the context of motor control.
There are evident advantages to this integrated approach. First, limits on the accuracy and speed of processing impose limits on motor performance. These limits not only constrain performance, but shape the optimal strategies for movement planning and execution (Harris & Wolpert,
1998; Todorov & Jordan,
2002). Second, sensory processing is shaped by the demands of the motor tasks it serves. Thus, for example, visual and proprioceptive cues to hand position appear to be integrated differently for different components of the sensory transformation – vision dominating for path planning and proprioception dominating when calculating the muscle signals needed to move the hand along that path (Sober & Sabes,
2005). How sensory cues are integrated for online control should also vary over the course of a movement and does (Saunders & Knill,
2004).
Finally, the two factors – sensory limitations and task demands – do not provide independent constraints on sensorimotor control, but rather interact to constrain movement strategies. Recent work applying statistical decision theory to sensorimotor control illustrates how this interaction can be captured in the framework of statistical decision theory (Trommershäuser et al.,
2003): manipulating the explicit rewards associated with movement outcomes in psychophysical experiments provides a tool for studying how the human sensorimotor system resolves trade-offs between the sensory and task constraints.
To summarize, traditional disciplinary boundaries have led to an artificial division between the sensory and motor components of everyday activity. Motor control researchers are trained largely in the structure and function of the motor system, while vision researchers are trained in the structure and function of the visual system. The two communities of researcher publish in different specialized journals and present their work at special purpose disciplinary conferences. The current special issue represents an attempt to bring together in one place work in the two fields and in the interface between them that we hope will ultimately lead increased communication between the two disciplines in support of a newer and deeper understanding of sensorimotor integration.
National Institutes of Health EY13319 (DCK); National Institutes of Health EY08266 (LTM); Deutsche Forschungsgemeinschaft (Emmy-Noether-Programm) Grant TR, 528/1-2, 1-3 (JT). We especially thank Alice O’Donnell for help in preparing this introduction.
Commercial relationships: none.
Corresponding author: Laurence T. Maloney.
Address: Department of Psychology and Center for Neural Science, New York University,
New York, NY, USA.