The role of vision in detecting and correcting fingertip force errors during object lifting

Abstract

Vision provides many reliable cues about the likely weight of an object, allowing individuals to predict how heavy it will be. The forces used to lift an object for the first time reflect these predictions. This, however, leads to inevitable errors during lifts of objects that weigh unexpected amounts. Fortunately, these errors are rarely made twice in a row—lifters have the impressive ability to detect and correct large or small misapplications of fingertip forces, even while experiencing weight illusions. Although it has been assumed that we detect and correct these errors exclusively with our sense of touch, recent evidence has demonstrated a role for vision in this fingertip force scaling. Here, we demonstrate that even when stimulus set size, delay, and modality are controlled for, individuals are unable to skillfully scale their grip and load force rates over repeated lifts without vision. However, eliminating only the task-relevant visual information, while maintaining the rest of the visual world, shifts participants back into the normal, skilled mode of control. These findings clarify the role of visual information in the ostensibly haptic task of lifting objects, suggesting individuals use priors under conditions where uncertainty is high.

Introduction

In order to lift in a controlled, secure, and efficient manner, individuals make predictions about how much force will be required to overcome the effects of gravity. With objects that have been lifted before, these predictions are very accurate, utilizing a long-term “sensorimotor memory” (Johansson & Westling, 1984). Individuals apply an appropriate amount of force to successfully lift an object even after a substantial delay from when it was lifted last (Gordon, Westling, Cole, & Johansson, 1993). Of course, this reliance on sensorimotor memories can go only so far when operating in the real world. During initial interactions with objects where no direct previous experience of weight is available, these predictions must come from vision-based statistics in the environment (i.e., consistently reinforced occurrences). Typically, items that appear to be made from metal are heavy, whereas things that look plastic are usually light. Large bottles of water will be difficult to lift with one hand, while small bottles of water will invariably be easy to wield. Such is the consistency of these visual associations to a particular weight that even watching others lift objects of various weights will prime our motor system to react with an appropriate level of force (Alaerts, Senot et al., 2010). In addition, it is by combining such rules that we can (usually) skillfully lift objects that are new to us. Of course, these predictions will not always be correct, and we will sometimes encounter items that do not conform to the environment's statistics (outliers, in this terminology), and lifting objects that weigh more or less than we expected will likely result in a misapplication of force. Subtle “mistakes” are barely noticeable and usually corrected online during the lift. Large errors feel surprisingly dramatic: consider the comical perceptual and motor implications of trying to lift a full suitcase that was supposed to be empty and vice versa. However, irrespective of how cognitively penetrable these errors are, they will still be detected and corrected by the motor system in time for the next lift with astonishing accuracy, meaning lifters will rarely make the same error twice in a row. This fingertip “force-scaling” behavior is even resilient enough to operate while individuals are experiencing illusions of heaviness, such as the size- and material-weight illusions. 

In the size–weight illusion (SWI), individuals report that smaller objects feel substantially heavier than similar looking large objects that have been adjusted to weigh the same amount (Charpentier, 1891). To demonstrate this illusion, participants lift these objects (usually cubes) one at a time and verbally report some analogue of its weight (for more details regarding possible causes of the SWI, the interested reader is directed to Flanagan, Bittner, & Johansson, 2008; and a commentary by Ernst, 2009). When individuals lift these objects repeatedly, they apply a very characteristic pattern of fingertip forces. Initially, participants use excessive grip and load force rates to lift the larger SWI cube and lower rates of force to lift the smaller SWI cube. These force rates on early trials reflect their initial expectations that large objects will weigh more than small objects. In spite of these initial errors, participants rapidly detect and correct these mistakes (Flanagan & Beltzner, 2000). The initial differences in force rates applied to each cube are eliminated, and despite consistently reporting that the small cube feels heavier than the large cube, participants now begin to lift with almost identical force rates for the large and small cubes—forces that reflect the actual mass of the SWI stimuli. Not only do the fingertip forces not influence the perception of weight, but the motor system shows a dramatic immunity to the perceptual effects of the illusion. Participants may make errors in the opposite direction to their perceptions of heaviness, but their sensorimotor memories implacably direct the “correct” application of fingertip forces with practice (Grandy & Westwood, 2006). 

In order to scale their fingertip forces from the expected to the actual weight of, for example, SWI inducing cubes, individuals must be able to detect the errors that they make. Intuitively, it seems likely that they use their sense of touch to monitor any errors that are made while gripping and lifting objects. In fact, the haptic component of detecting and correcting errors over repeated lifts is well understood from a physiological viewpoint. In their detailed review, Johansson and Flanagan (2009) highlight the crucial role played by fast adapting type II (FA-II) tactile afferents in matching the expected liftoff time to the actual liftoff time—the crucial “transient mechanical events” (see Box 3 of Johansson & Flanagan, 2009). Although haptics no doubt play a fundamental role in the scaling of fingertip force rates, we have recently presented evidence that visual feedback of the lift is also required for this skilled behavior (Buckingham & Goodale, 2010a). Our task required participants to lift a single unchanging cube multiple times following a brief “preview period,” during which participants were allowed to see the size of the cube they thought that they would go on to lift. Seeing the small cube before the lift primed participants to expect a relatively lighter cube, making the cube that they eventually lifted feel heavier than it felt on the trials in which they previewed the larger cube. This finding demonstrated that individuals can, in fact, perceive the SWI without continuous visual experience of the objects' size, refuting Masin and Crestoni's (1988) assertion that the illusion required continuous perception (either in terms of vision or haptics). More importantly, however, it highlighted that our perception of heaviness is so dependent upon expectations that a single, unchanging object could be made to feel as if its weight was changing from trial to trial. In addition, and much to our surprise, participants failed to demonstrate the characteristic fingertip force scaling when lifting this cube: when they saw the large cube they applied higher rates of force than they did when they saw the small cube—and they made this error over and over again. This unexpected finding points toward a role for vision in detecting and correcting fingertip force errors that works in parallel with haptic feedback. 

Removing visual information has been a standard manipulation in visuomotor research for decades (see Elliott, 1990 for review). In recent years, however, in tasks as varied as reaching to grasp, pointing with one or both hands, and navigating through the environment, the specific role of vision has been clarified by manipulating a variety of other features of the experiment that could modulate the effects of removing visual information. 

The purpose of the current study was to clarify and determine the full extent of vision's involvement in the detection and correction of expectation-based fingertip force errors by manipulating various aspects of the no-vision lifting task in separate experiments. Throughout all these tasks, fingertip forces were measured on a trial by trial basis to determine if normal levels of fingertip force scaling could occur under any conditions when vision of repeated lifts were not permitted. In the first experiment, participants lifted (without vision) all of the objects that they previewed (cf., Buckingham & Goodale, 2010a). However, even when the discrepancy between the number of objects seen and the number of objects lifted was controlled for, participants still had difficulty scaling their fingertip force rates. In a second experiment, where the delay between the visual preview and the lift itself was eliminated, vision's general importance for the detection and correction of grip and load force errors was confirmed. To determine whether vision's importance was merely a consequence of the visual preview itself, we then asked participants to perform no-vision lifts after “previewing” the cubes haptically rather than visually. However, even with a haptic preview of the cube's size, participants were still poorer at force scaling than when they were allowed constant vision. Although the first three experiments did strongly suggest that vision is crucial to force scaling, a final experiment demonstrated that it does not matter how relevant the visual information is to the lift itself. Merely allowing participants visual information of the room itself but not the immediate lifting workspace was sufficient to bring their force scaling back to normal levels. 

General methods

In all experiments reported in this manuscript, participants were students with normal, or corrected-to-normal vision, recruited from the undergraduate Psychology research participation pool at the University of Western Ontario. In all studies, participants were naive to the experimental hypotheses, unaware of the SWI, and had not taken part in any previous object lifting study within the department (i.e., no individual took part in more than a single experiment presented in the current manuscript). All testing procedures were approved by the ethics board at the University of Western Ontario, and participants gave written informed consent prior to testing. 

All the tasks required participants to lift objects by grasping a handle with the thumb and index finger of their dominant hand without vision and give a rating of how heavy the object felt on every trial. Participants were told to lift the SWI cubes “straight upwards with your thumb and index finger on the handle in a smooth and confident fashion about 5 cm off the table, and hold it steady without unnecessary hefting”—instructions that prompted participants to lift in a predictive, feed-forward way, rather than with a probing, feedback-based strategy. These lifts were demonstrated by the experimenter before giving participants the opportunity to practice the lifting and perceptual rating tasks with two blue cylinders of similar (but not identical) size and mass to the SWI cubes. The experimenter was careful not to handle the SWI cubes in front of the participants, so they had no indication of how heavy each cube would be prior to lifting. 

In all the tasks described in the current manuscript, the same three SWI-inducing stimuli were used to elicit the illusion as in our previous work (Figure 2A of Buckingham & Goodale, 2010a). These stimuli were hollow wooden cubes of different sizes that had been painted black and filled with varying amounts of lead, so that they all weighed 700 g. The small cube was 5 cm × 5 cm × 5 cm, the medium cube was 7.5 cm × 7.5 cm × 7.5 cm, and the large cube was 10 cm × 10 cm × 10 cm. All the cubes had identical small brass and plastic mounts attached to the center of their top surface to facilitate the attachment of a pair of six-axis force–torque sensors (Nano17 F/T; ATI Industrial Automation, Garner, NC). These transducers were attached to a custom-built aluminum and plastic handle with opposing grip pads covered in painted sandpaper to facilitate a stable precision grip with the thumb and forefinger. The handle allowed the force transducers to be mounted to each cube and removed easily by the experimenter between lifts, adding 50 g to the total weight of the cube. Custom-built light sensors, which projected a small beam of light 3 mm above and parallel to the surface of the table, were used to provide an objective time stamp for the cube's liftoff. In all experiments, the cubes were presented in one of three pseudo-random orders, with all three cubes being lifted once, in a random order, on every three trials. Participants lifted each cube 15 times, for a total of 45 lifts in a single session lasting approximately 20 min. 

Data from each force transducer (representing the contribution of each finger to the lift) were sampled at 1,000 Hz to yield grip and load force vectors. Grip force was defined as the vector orthogonal to the surface of the grasp handle (i.e., squeezing), whereas load force was defined as the vector sum of the forces normal to the surface of the grasp handle (i.e., vertical lifting). The grip and load force traces for each finger were averaged, passed through a dual pass 4th-order Butterworth filter with a low-pass cutoff of 14 Hz, and differentiated with a 5-point central difference equation to yield grip force rate and load force rate. The maximum values from all measures represented the kinetic dependant variables. These values were first identified automatically as the numerical local maximum, and then confirmed by examining each force plot. By default, the maxima of the grip and load force rates occur well before liftoff has taken place, whereas the maxima of the forces themselves are roughly coincident with liftoff. Additionally, the time that elapsed between the load force increasing above 1 N and the actual liftoff was defined as the load phase duration. This temporal measure describes the dynamics of the lift, rather than its kinetics. Participants gave absolute magnitude representations (unconstrained numerical ratings, with larger numbers representing heavier feeling objects; Zwislocki & Goodman, 1980) of how heavy the cube felt after each lift. These numerical values were normalized to a z-score distribution based on each participant's mean and standard deviation. 

In keeping with the analyses strategy of our previous work (Buckingham & Goodale, 2010a), the perceptual, temporal, and kinetic measures within each experiment were examined in separate 3 (cube size) × 15 (lift number) repeated measures ANOVAs, presented in Table 1. To evaluate the role of the various factors during fingertip force scaling, we utilized two separate post hoc analyses. In order to directly examine the speed of the fingertip force scaling in each experiment, we compared the forces that were used to lift the large and small cubes over the first 5 trials with one-tailed Bonferroni-corrected t-tests, requiring a p-value of 0.02 to achieve statistical significance. The number of significant differences over these early lifts provided a simple metric of the amount of experience required by the motor system to adapt under the various conditions. Additionally, to provide a more fine-grained metric of the quality of the force scaling over the entire experiment, we fit the data points for each of the 15 lifts of each cube with a 4th-order polynomial in Sigmaplot 10.0 (Systat Software) representing the mean trends. We then calculated the average of the difference scores between the large and small cubes from the 50 interpolated time points that made up the regression line. If these lines failed to fully converge, the difference scores remained large, whereas smaller scores would (in combination with the speed analysis detailed above) indicate more accurate fingertip force scaling. Similar polynomial-based analysis methods are commonplace in psychophysics in questions more focused around means, rather than variance (Gescheider, 1997). Conceptually, combining the quality analysis with the standard inferential statistical analyses is similar to the dual “area under curve” and waveform peak/latency strategies, utilized by electroencephalogram researchers (see Luck, 2005) to more comprehensively describe their data. We would like to emphasize that neither of these measures in isolation provides a full picture of the fingertip force scaling. Only by combining these analyses can we examine the rapidity and the quality of scaling, allowing for a more comprehensive overview of the motor system's performance, allowing us to compare the variability and mean trends across the visual conditions. For comparison purposes, these new analyses were first performed on the full-vision data reported in Buckingham and Goodale (2010a). The values that were derived on the quality metric were taken as our full-vision control and compared with the peak grip and load force rate data with all of the various no-vision tasks reported herein with pairwise t-tests. 

Table 1

 

Inferential statistical analyses of the four experiments described in this manuscript, in addition to the full-vision condition reported in Buckingham and Goodale (2010a). The results of post hoc analyses are described in the individual results sections. S refers to main effect of size, L refers to main effect of lift number, and S × L refers to a size by lift number interaction.

Experiment	Full vision	1	2	3	4
Subtitle	Reprinted from Buckingham and Goodale (2010a, 2010b)	Lifting multiple SWI cubes without vision	No-vision SWI with no delay between seeing and lifting the cubes	No-vision SWI following haptic preview of object size	No-vision SWI with task-irrelevant vision maintained
Heaviness rating	S: F (2,38) = 186.5, p < 0.001**, η 2 = 0.91	S: F (2,34) = 226.9, p < 0.001**, η 2 = 0.93	S: F (2,40) = 52.4, p < 0.001**, η 2 = 0.72	S: F (2,38) = 252.9, p < 0.001**, η 2 = 0.93	S: F (2,44) = 314.1, p < 0.001**, η 2 = 0.37
	L: F (14,266) = 11.4, p < 0.001**, η 2 = 0.37	L: F (14,238) = 18.2, p < 0.001**, η 2 = 0.52	L: F (14,280) = 10.2, p < 0.001**, η 2 = 0.55	L: F (14,308) = 8.1, p < 0.001**, η 2 = 0.30	L: F (14,308) = 5.8, p < 0.001**, η 2 = 0.21
	S × L: F (28,532) = 1.6,p < 0.05*, η 2 = 0.08	S × L: F (28,476) = 1.6, p < 0.05*, η 2 = 0.09	S × L: F (28,560) = 1.3, p = 0.14, η 2 = 0.06	S × L: F (28,616) = 1.4, p = 0.09, η 2 = 0.07	S × L: F (28,616) = 1.0, p = 0.45, η 2 = 0.04
Grip force rate	S: F (2,38) = 8.7, p < 0.005**, η 2 = 0.31	S: F (2,34) = 22.0, p < 0.001**, η 2 = 0.56	S: F (2,40) = 17.7, p < 0.001**, η 2 = 0.47	S: F (2,38) = 10.1, p < 0.001**, η 2 = 0.35	S: F (2,44) = 1.6, p = 0.21, η 2 = 0.07
	L: F (14,266) = 0.57, p = 0.90, η 2 = 0.03	L: F (14,238) = 0.5, p = 0.93, η 2 = 0.03	L: F (14,280) = 1.8, p < 0.05*, η 2 = 0.08	L: F (14,308) = 1.1, p = 0.40, η 2 = 0.05	L: F (14,308) = 1.9, p < 0.05*, η 2 = 0.08
	S × L: F (28,532) = 1.7, p < 0.05*, η 2 = 0.08	S × L: F (28,476) = 0.7, p = 0.86, η 2 = 0.04	S × L: F (28,560) = 2.1, p < 0.005**, η 2 = 0.10	S × L: F (28,616) = 1.1, p = 0.30, η 2 = 0.06	S × L: F (28,616) = 3.2, p < 0.001**, η 2 = 0.13
Load force rate	S: F (2,38) = 6.1, p < 0.01**, η 2 = 0.24	S: F (2,34) = 21.5, p < 0.001**, η 2 = 0.56	S: F (2,40) = 11.0, p < 0.001**, η 2 = 0.36	S: F (2,38) = 6.8, p < 0.005**, η 2 = 0.26	S: F (2,44) = 5.2, p < 0.01**, η 2 = 0.19
	L: F (14,266) = 5.46, p < 0.001**, η 2 = 0.22	L: F (14,238) = 2.3, p < 0.01*, η 2 = 0.12	L: F (14,280) = 1.4, p = 0.16, η 2 = 0.07	L: F (14,308) = 4.4, p < 0.001**, η 2 = 0.19	L: F (14,308) = 3.4, p < 0.001**, η 2 = 0.13
	S × L: F (28,532) = 1.7, p < 0.05*, η 2 = 0.08	S × L: F (28,476) = 1.0, p = 0.44, η 2 = 0.06	S × L: F (28,560) = 1.4, p = 0.10, η 2 = 0.06	S × L: F (28,616) = 0.9, p = 0.86, η 2 = 0.04	S × L: F (28,616) = 2.4, p < 0.01**, η 2 = 0.10
Load phase duration	S: F (2,38) = 17.9, p < 0.001**, η 2 = 0.50	S: F (2,24) = 11.6, p < 0.001**, η 2 = 0.47	S: F (2,40) = 25.7, p < 0.001**, η 2 = 0.56	S: F (2,38) = 11.1, p < 0.001**, η 2 = 0.37	S: F (2,44) = 7.4, p < 0.005**, η 2 = 0.25
	L: F (14,266) = 2.68, p < 0.005**, η 2 = 0.13	L: F (14,168) = 1.3, p = 0.18, η 2 = 0.10	L: F (14,280) = 0.6, p = 0.88, η 2 = 0.03	L: F (14,308) = 2.0, p < 0.05*, η 2 = 0.09	L: F (14,308) = 3.2, p < 0.001**, η 2 = 0.13
	S × L: F (28,532) = 2.8, p < 0.05*, η 2 = 0.13	S × L: F (28,336) = 1.1, p = 0.37, η 2 = 0.08	S × L: F (28,560) = 2.2, p < 0.001**, η 2 = 0.10	S × L: F (28,616) = 1.3, p = 0.12, η 2 = 0.07	S × L: F (28,616) = 2.5, p < 0.001**, η 2 = 0.10

 

Note: * indicates significance at p < 0.05, ** indicates significance at p < 0.01.

To provide some context for the current findings, we will briefly describe the pattern of forces that individuals use to lift SWI cubes with full, unoccluded vision. In brief, the 19 participants who participated in the full-vision portion of our earlier work (Buckingham & Goodale, 2010a) were able to quickly detect and accurately correct expectation-based errors in their fingertip forces. This rapid scaling was characterized by an interaction between size and lift number (Table 1). It is worth emphasizing the consistency of this finding within the weight illusion literature; in all the studies that have examined the fingertip forces used to experience weight illusions, very similar patterns of scaling behavior are observed (Buckingham, Cant, & Goodale, 2009; Buckingham & Goodale, 2010b; Chang, Flanagan, & Goodale, 2008; Chouinard, Large, Chang, & Goodale, 2009; Flanagan & Beltzner, 2000). This rapid force scaling is generally confined to the force rates; the maximum forces themselves, which may peak after liftoff occurs, take substantially longer to scale to the cubes' actual mass (see Buckingham & Goodale, 2010b, for detailed discussion of this tangential point). Accordingly, the majority of the following discussion will focus on comparisons of the measures that occur before liftoff: the maximum grip and load force rates (2nd and 3rd rows of Figure 2). In terms of speed of the force scaling, participants applied significantly different grip and load force rates to the large and small cubes on only two out of the first five lifts. In terms of the quality of the force scaling, the average distance between the lines representing the force rates of the large and small cubes was 4.37 for the grip force rate and 3.41 for the load force rate. It is these values that were directly compared to the various no-vision tasks reported below with pairwise t-tests (Figure 3), in order to determine which manipulations were particularly crucial for “normal” levels of fingertip force scaling over repeated lifts. 

Experiment 1—Lifting multiple SWI cubes without vision

In the “no-vision” task reported in Buckingham and Goodale (2010a), participants only ever lifted a single cube following a preview phase where they saw an object that varied in size from one trial to the next. This procedure differs from a standard SWI paradigm, where the participant lifts multiple objects of different sizes. Although using a single cube allowed us to tightly control the physical aspects of the stimuli in our perceptual examination of the SWI, it did make it more difficult to interpret the role of vision in detecting and correcting grip and load force errors. It was unclear, for example, whether the deficit in force scaling was due to the lack of vision per se or to a difference between the number of objects that were seen (three) and the number of objects that were touched (the single lifted cube). It could have been the case, for example, that the sensorimotor system was expecting subtle differences in torques between the cubes that were not there (because, of course, the same cube was being lifted on every trial). In essence, it was unclear whether the deficit in fingertip force scaling that was demonstrated in Buckingham and Goodale (2010a) would generalize to a “standard” SWI task, and it was this ambiguity from our previous work that we chose to address first. To this end, we increased the number of SWI cubes from one to three, meaning that participants lifted (without vision) all three of the objects that they previewed. This new task would be more readily comparable to both our original full-vision control and much of the earlier work on how individuals lift SWI-inducing objects (e.g., Flanagan & Beltzner, 2000). If vision is crucial to fingertip force scaling over repeated lifts, participants will still be unable to correct their grip and load force errors, just as described in Buckingham and Goodale (2010a). 

Methods

In this first experiment, eighteen right-handed students took part for course credit (9 males, 9 females, mean age = 27.9 ± (SD) 9.9 years). Participants sat in front of a table wearing closed PLATO shutter goggles, with their dominant hand resting on the table beside the liftoff pad. The experimenter attached the force transducer handle to one of the cubes and placed it on the liftoff pad. The goggles then opened for 1 s. At the same time as they reclosed, an auditory go cue signaled participants to reach out and lift as described in the General methods section. Participants then gave their heaviness rating and the goggles closed in preparation for the next trial. The procedure was very similar to the no-vision condition reported in Buckingham and Goodale (2010a), except that no switch was made between the “seen cube” and the “lifted cube.” Instead, participants lifted one of three differently sized cubes that had identical mass—the classic SWI task performed without vision of the lift (Figure 1). 

Figure 1

 

(A) Schematic representations of the time courses for the full-vision and no-vision experiments reported in this manuscript. The blue line represents the presence or absence of vision, the black dotted line represents the haptic preview phase, and the vertical red dashed line highlights where liftoff occurred. In the first 3 experiments, vision was removed by shutter goggles, whereas in the final experiment, vision was removed by a cardboard occluder (the horizontal black dashed line shows the time course of the availability of task-irrelevant visual information). (B) Closer detail of Experiment 4, where vision was removed with a cardboard occluder. The first 5 panels detail the time course of the task, and the final panel shows the participant's viewpoint of the lift, with the occluder covering their lifting arm up to the midpoint of the forearm.

Results

When participants repeatedly lifted the three SWI-inducing cubes without vision, following a brief visual preview, they judged them to have different weights (i.e., a main effect of size was noted for perceptual rating, indicating that they experienced the SWI, see Table 1). More important for the current investigation was the fact that they were unable to scale their fingertip forces in the skillful way that they could with full vision, just as described in Buckingham and Goodale (2010a). This disruption was apparent only in the grip and load force rate measures, where no interactions between cube size and lift number were observed (Table 1). The lack on interactions in these measures indicated that the rapid scaling that characterizes normal repeated lifting of SWI cubes was not present. It is clear from Figure 2 that participants scaled their force rates less rapidly than when allowed full vision. Further, the quality of the force rate scaling was also substantially affected: pairwise comparisons showed a significantly lower quality of scaling in terms of the grip force rate (no-vision quality of scaling: 11.78 vs. full-vision quality of scaling: 4.37; p < 0.001) and load force rate (no-vision quality of scaling: 7.84 vs. full-vision quality of scaling: 3.41; p < 0.001) in the no-vision condition (Figure 3). These findings indicate that it is the removal of vision, rather than any discrepancy between the number of objects that were seen and lifted, that affected people's ability to detect and correct fingertip force rate errors. 

Figure 2

 

The perceptual (row 1) and kinetic (rows 2–4) measures for the small, medium, and large SWI cubes on each lift. Error bars represent standard errors of the means. Asterisks (*) indicate a significant difference between the forces applied to the small and large cubes on a particular lift.

Figure 3

 

Comparisons of the quality of the grip and load force scaling (i.e., the mean distance between the 4th-order polynomial functions representing the small and large cubes) across the four experiments and the full-vision control task reported in Buckingham and Goodale (2010a). Error bars represent standard errors of the means.

Experiment 2—No-vision SWI with no delay between seeing and lifting

Another factor that has been shown to be particularly crucial in action experiments that manipulate vision is the amount of time that elapses between seeing and doing. The no-vision task described in Buckingham and Goodale (2010a) required a 5-s delay between visual perception and action onset, and it is unclear whether such a lengthy period without vision may have impacted upon the sensorimotor memory required to lift with the correct fingertip forces. Within the reaching and grasping literature in particular, it has long been known that delays far smaller than 5 s can affect kinematics in a wide variety of contexts (e.g., Striemer, Chapman, & Goodale, 2009; Westwood & Goodale, 2003). Thus, in order to eliminate the possibility that delay, rather than vision, underlies the problems with force scaling, we removed the delay between visual perception and action onset. Should participants still have problems detecting and correcting their fingertip force errors under these conditions, it will provide strong indications that vision of the lift directly contributes to fingertip force scaling over repeated lifts. 

Methods

In this experiment, 23 right-handed students took part for course credit. Two participants were removed prior to analyses due to atypical lifting behavior on at least 10 lifts throughout the course of the experiment (i.e., multiple grip and load forces that were greater than 2 standard deviations above the mean), leaving a sample of 21 (12 males, 9 females, mean age = 18.7 ± (SD) 1.2 years). The general procedure was identical to Experiment 1, except that we eliminated any delay between seeing the cube and lifting it. Thus, the shutter goggles opened, and 1 s later, an auditory cue signaled participants to reach out and lift the cube. The entirety of the reach toward the cube was undertaken with full vision, and contact with the force transducer handle (specifically, an index finger grip force in excess of 0.5 N) triggered the shutter goggles to close. This procedure meant that participants had visual feedback right up until the point that they started lifting (N.B. not the point at which the liftoff itself occurred). 

Results

In this second experiment, participants also experienced a robust SWI (Table 1, Figure 2). Further, even without a delay between seeing and lifting, participants continued to show some deficits in scaling their fingertip force rates compared to when they were permitted full vision. The standard size × lift interaction was, in fact, observed for grip force rate, indicating that some scaling did occur with that measure. However, this interaction was not significant for the load force rate. Further, it is clear from Figure 2 and the post hoc analyses that participants took longer to adjust their fingertip forces from the expected to the actual mass of the cubes. Once again, the quality of the force rate scaling (Figure 3) was also detrimentally affected by the lack of vision for both the grip force (no-vision quality of scaling: 8.63 vs. full-vision quality of scaling: 4.37; p < 0.001) and load force (no-vision quality of scaling: 5.29 vs. full-vision quality of scaling: 3.41; p < 0.001). In summary, although the rapidity of the scaling of the grip and load force rates was roughly equivalent to lifts with vision, the overall quality of the force rate scaling was substantially worse in the absence of vision. This suggests that it was the lack of visual information rather than the delay between seeing and lifting that was responsible for the failure to properly scale fingertip forces in Buckingham and Goodale (2010a). 

Experiment 3—No-vision SWI following haptic preview of object size

It has been known for some time now that, during manual localization tasks, the relative importance of a particular modality can be altered by changing how reliable the information is (e.g., Kording & Wolpert, 2004). Under normal circumstances when reaching to grasp an object, all of the redundant visual, haptic, and proprioceptive sources of information are combined to form a single percept (Ernst & Bulthoff, 2004). However, when placed in opposition to one another, the senses compete, usually with vision (which is subject to less sensory noise than haptics or proprioception) dominating others (e.g., Hartcher-O'Brien, Gallace, Krings, Koppen, & Spence, 2006). It is possible that similar factors may be influencing lifters' performances in our no-vision lifting task. In Buckingham and Goodale (2010a), the expectations of heaviness were based purely upon vision—subjects did not have the opportunity to determine the size of the SWI cubes with any of their other senses. The unimodal, visual nature of the preview period may therefore have preferentially weighted the importance of vision at the expense of the other modalities—in other words, the failure to scale fingertip forces without vision may have been specific to, or even caused by, our visual preview task. If this were the case, it would certainly limit the impact of vision's role in object lifting. This point can be easily addressed by providing subjects with a haptic rather than visual preview period prior to their no-vision lift. 

Methods

In this experiment, 24 right-handed students took part for $5 compensation. Two participants were excluded from the study for failing to lift within the required time frame (i.e., the 4-s window in which the force transducer recorded) on multiple lifts, meaning that the kinetic measures were not recorded. A further two participants were also excluded in the analysis phase for atypical lifting behavior (as above), leaving a sample of 20 (7 males, 13 females, mean age = 24.6 ± (SD) 5.8 years). Here, vision was occluded for the entire experiment. Instead, participants “previewed” the size of the cube prior to the lift by exploring it with their dominant hand. Participants were told “to take as long as they needed to gauge the size of the cube” by exploring the top surface of the cube with their hand. Participants were not specifically instructed to use the size information to infer the cube's mass although, of course, they knew that a lift was imminent. Other than making sure they did not lift or tilt the cubes during this haptic preview period, the exploration was unconstrained in terms of time or style. The majority of participants haptically previewed the cube for up to 5 s on early trials (often, in a rather tentative fashion) and tended to reduce the preview time to less than 2 s on later trials. After the haptic preview, the participants placed their hand back on the table surface. The experimenter then attached the force transducer handle to the cube and an auditory cue signaled participants to lift as described in the General methods section. It should be emphasized that, in this experiment, participants never saw any of the cubes before, during, or after any of their lifts. Although participants were allowed to see the cubes they would be lifting before the experiment started (after the practice trials), during the task itself, they never saw the cubes again and only felt them with their hands. 

Results

In this experiment, where vision was not permitted at all, a haptic preview of object size was sufficient to induce an SWI. Further, when lifting the SWI cubes without vision following a haptic preview of their size, participants once again failed to scale their fingertip force rates as skillfully as they did when vision of the lift was permitted. No size × lift interactions were seen in the grip or load force rates, indicating that the rapid fingertip force scaling did not occur during these lifts. Furthermore, the overall quality of the scaling (Figure 3) was substantially lower than in the full-vision conditions for both grip (no-vision quality of scaling: 7.79 vs. full-vision quality of scaling: 4.37; p < 0.001) and load force rates (no-vision quality of scaling: 4.74 vs. full-vision quality of scaling: 3.41; p < 0.001), meaning that the quality of the scaling suffered without vision. Just as was shown in the previous experiment, the overall quality of the force rate scaling was substantially worse than when vision is permitted during a lift. The availability of vision still appears to be crucial for this aspect of object lifting, even when the lifter's expectations come from a non-visual source. 

Experiment 4—No-vision SWI with task-irrelevant vision maintained

In spite of controlling for a variety of factors, the results of our experiments continued to point toward a crucial role for vision in fingertip force scaling. However, it remained possible that the method with which vision was removed in our earlier no-vision task may have disrupted the force rate scaling. The use of the PLATO shutter goggles in Buckingham and Goodale (2010a) to remove vision of the entire environment may have affected performance in several ways. First, the dramatic transient of the entire visual world being removed may have had a distracting effect on participant's ability to anticipate the weight of an upcoming object. Second, this all-encompassing removal of vision may have altered the control mode under which the lift was performed. Although there is little research speaking directly to this effect, at least one study has indicated that removing vision of the entire world (i.e., rather than just the target) alters the temporal dynamics of online reach error corrections in a manual localization task (Heath, 2005). To address this final point, we altered the way in which vision was removed in our no-vision lifting task, using a shield to block only the visual information relevant to the success of the lift. 

Methods

In this final experiment, 28 right-handed students took part for $5 compensation. Four participants were excluded from the study for failing to lift within the required time frame on multiple lifts, and a further participant was excluded in the analysis phase for atypical lifting behavior, leaving a sample of 23 (8 males, 15 females, mean age = 24.4 ± (SD) 8.0 years). Here, shutter goggles were not utilized at all. Instead, participants sat with their eyes closed until the cube was placed on the table. Participants were then verbally informed they could open their eyes, at which point the experimenter placed a large three-sided cardboard box over the workspace in front of the participant to eliminate the task-relevant vision (i.e., the hand lifting the cube). Participants were able to see the cube on the table for as long as it took the experimenter to place the cover (approximately 1.5 s). Once this cover was in place, an auditory cue signaled participants to reach into the open edge of the cover and lift the SWI cube (Figure 1). The procedure achieved total visual occlusion of the object, the lifting hand, and most of the forearm. The larger sample in Experiment 4 was recruited to counter the possibility of an increase in variability of hand posture as participants lifted the cube without hitting the shield. 

Results

When lifting three SWI cubes inside a three-sided cover that removed the task-relevant visual information, participants, as expected, experienced a robust SWI. In contrast to the previous experiments, participants were clearly able to scale their fingertip force rates as skillfully as when full vision was permitted, showing significant size by trial interactions for both grip and load force rates. Somewhat surprisingly, this was not accompanied by a main effect of size for the grip force rate measure, which was present in all of the other experiments reported in the manuscript and the full-vision control. It seems likely that this lack of a size effect in this measure is a consequence of the surprisingly similar forces applied to all three of the cubes after the first few trials (i.e., the high-quality scaling—see below). In terms of the rapidity of the scaling, participants took only a single lift to correct their initial, expectation-based errors. Further, the overall quality of their scaling (Figure 3) was not significantly different to the full-vision performance in terms of grip force rate (no-vision quality of scaling: 3.21 vs. full-vision quality of scaling: 4.37; p = 0.09) or load force rate (no-vision quality of scaling: 3.51 vs. full-vision quality of scaling: 3.41; p = 0.81). Thus, rather unexpectedly, and even with a larger sample than in the previous experiments, participants' force rate scaling in this task was statistically equivalent to their performance when full vision of the entire task was available. Participants were, in fact, marginally better at scaling their grip force rates than they were when lifting with full vision, mirroring the aforementioned lack of a main effect of object size for this measure. To verify that the apparent improvement in fingertip force scaling was not based on the absence of a statistical difference, we performed the opposite contrasts, comparing the quality of the grip and load force rate scaling in this final experiment to those seen in Experiments 1–3. The results were unambiguous: in this final experiment, participants scaled their grip and load force rates with significantly higher quality than they managed in Experiment 1, 2, or 3 (all p-values < 0.001 for GFR and < 0.05 for LFR). 

Discussion

In the current study, we investigated the role of vision in detecting and correcting fingertip force errors over repeated lifts of SWI-inducing stimuli. The rationale behind this investigation stemmed from our earlier work (Buckingham & Goodale, 2010a), in which we noted that individuals persisted in lifting an unseen cube that they believed to be larger than it actually was with a greater rate of force than they did when they believed it to be smaller. However, it was unclear whether it was the lack of vision or one of the several other differences from a “standard” SWI experiment that was responsible for this inability to scale the fingertip force rates. To clarify the role of vision in this ostensibly haptic aspect of skillful object lifting, we manipulated the number of illusion-inducing cubes that were lifted, the delay between vision and the lift, the modality of the preview period, and the way in which vision was eliminated. To get a more descriptive picture of the skill with which the cubes were lifted over repeated trials, two metrics of performance were examined. First, we determined the speed with which the, comparatively large, initial expectation-based errors were corrected over the first few lifts. Second, we examined the degree to which more subtle errors in force application could be detected and corrected over a relatively large number of interactions (i.e., the quality of the force rate scaling). We then compared these metrics of skillful object lifting to our earlier data set that had been collected using lifts of the same stimuli with no visual occlusion, in order to provide a clearer picture of vision's involvement in fingertip force scaling. 

Although the current work is focused toward understanding sensory contributions to motor learning, we will first sum up the finding related to the perceptual SWI. Lifters' perceptual judgments of weight in all the experiments support our earlier contention that continuous visual experience is not necessary to experience weight illusions (Buckingham & Goodale, 2010a; cf., Masin & Crestoni, 1988): in all of the no-vision tasks, participants experienced robust SWIs. Cognitively based expectations can alter our perception of object weight in much the same way as visual context can alter our perception of object size. Furthermore, the modality of the expectation-inducing preview appears to be unimportant: feeling a large cube will induce illusion-causing expectations to the same degree as seeing a large cube (Figure 2, 1st row). Of course, demonstrating the, apparently substantial, role played by cognitive expectations in experiencing this illusion does not contradict earlier work by Ellis and Lederman (1993) suggesting that haptics must play a large role in experiencing the SWI. Rather, it is likely that many factors contribute to experiencing the “full” SWI. In fact, many studies have elegantly demonstrated that a somewhat weaker SWI can be induced by experiencing one modality in isolation (Ellis & Lederman, 1993; Kawai, Henigman, MacKenzie, Kuang, & Faust, 2007), while the largest illusion is experienced when vision and haptics are available. It is clear, however, that a substantial SWI can be experienced through expectations alone, irrespective of whether vision or haptics are used to induce the expectation in the first place. 

To determine individuals' abilities to detect and correct errors that they made during lifts, we measured their grip and load forces while they repeatedly lifted the same three identically weighted cubes. In the first experiment, where participants lifted three SWI-inducing objects without vision following a brief visual preview, there were clear deficits in detecting and correcting the force errors as compared to when they performed equivalent lifts with full vision. In the second experiment, where the delay between the vision preview and the lift was eliminated, participants were similarly poor at scaling their fingertip force rates. It could be argued that it may have been the way in which we controlled vision (i.e., with shutter goggles) that was somehow responsible for the participants' difficulties in scaling their fingertip forces (cf., Flanagan & Beltzner, 2000, where no shutter goggles were used). However, given that the goggles were also used in our full-vision control task to control vision between trials (Buckingham & Goodale, 2010a), where skilled fingertip force scaling clearly did occur, we do not find this to be a particularly likely scenario. Therefore, the findings from the first two experiments clearly indicate that vision plays a fundamental role in skillfully lifting objects, rather than the size of the stimuli set or short-term visual memory (as may have been the case in Buckingham & Goodale, 2010a). In order to determine if the importance of visual feedback in this task was somehow artificially enhanced by the modality of the visual preview, we allowed participants to feel rather than see the SWI cubes before lifting without vision. The results of this third task were consistent with the data collected from the first two experiments; that is, participants were significantly poorer (albeit no slower) at correcting their initial errors in fingertip force rates, continually applying higher rates of grip and load force to the large cube as compared to the small cube. Clearly, visual feedback is important irrespective of how the initial error-causing expectations of heaviness are induced. In a final experiment designed to gauge the specificity of the visual information that is crucial during the task, we allowed participants to lift the SWI cubes with only the task-relevant vision removed, obscuring the lift itself behind an opaque cover. Much to our surprise, it was under this condition alone that people were able to scale their fingertip forces just as well as they could with full vision. In other words, when they are allowed to see the surrounding environment during the lift, even in the absence of seeing the hand lifting the object, people are almost as skilled as they are during normal full-vision lifting. 

Taken together, it is clear that vision is indeed a requirement for the detection and skillful correction of expectation-based fingertip force errors—a task that has long been considered to be primarily haptic in nature (Johansson & Flanagan, 2009). Vision has shown to be a surprisingly good cue to the perception of an object's weight—for example, it has long been known that visual dynamic information is sufficient to gauge the weight of an object that someone watches being lifted (Bingham, 1987). This observation even extends to the motor system, where it has recently been demonstrated that an individual's motor system will react appropriately to the sight of visual mass cues (e.g., an empty bottle of water vs. a full bottle of water—Alaerts, Senot et al., 2010). There have even been recent hints of a role for vision in controlling fingertip forces online, which are clearly of relevance to the current work. For example, providing individuals with a visual display of their forces in the form of a computer-generated plot over time will improve their ability to maintain an isometric contraction of a certain force, and even small alterations in the gain of this plot will have substantial impacts upon their accuracy (e.g., Vaillancourt, Haibach, & Newell, 2006). Furthermore, in a recent study, Sarlenga, Baud-Bovy, and Danion (2010) have demonstrated that introducing a delay between object displacement and its graphical display changes the relationship between the timing of grip and load force application. Although this latter study would appear to imply a very specific role of visual feedback in helping to track the dynamics of the lifted object, the final study in our series, where we showed that task-relevant vision was not necessary, suggests otherwise. It could be suggested that our occluder failed to remove all of the task-relevant visual information, and participants were able to tune into whatever was left. For example, it has recently been shown that there are multiple visual cues that elicit a response that is tied to lifting objects that vary in weight (Alaerts, Swinnen, & Wenderoth, 2010). However, it is worth emphasizing at this point that all of our SWI cubes had identical mass, and the only visual cues that subjects would have been able to utilize are dynamic error information. This dynamic information, contained in the moving cube and hand, was completely covered by our occluder. Rather than specifically using the visual motion of the SWI cube to supplement their haptic feedback of lift errors, it instead appears that the availability of vision triggers a shift into a more skillful, feedback-based mode of control. 

In order to understand why this skilled mode of control is not utilized all of the time, we must first determine what occurs in the “unskilled” control mode (i.e., when wearing the closed shutter goggles). We believe that the answer lies in the direction of the errors made without vision—the surprising finding that participants continue to misapply force rates in the direction of their expectations of heaviness (i.e., more force applied to the large cube), rather than their explicit, haptic, experience of heaviness (i.e., that the large cube is comparatively light). It has recently been proposed that the motor system functions in a Bayesian fashion in these object lifting tasks, optimally integrating long-term expectations of heaviness (“priors,” in Bayesian terminology) with short-term veridical feedback from the previous lift (Brayanov & Smith, 2010; Buckingham & Goodale, 2010b). Indeed, a recent study has demonstrated a complex interplay between recent and longer term vision-based sensorimotor memories, which evolves over time as individuals prepare to lift objects (Loh, Kirsch, Rothwell, Lemon, & Davare, 2010). We propose that when the sensory world is diminished by removing vision, irrespective of vision's utility in carrying out the lift, the motor system defaults to its long-term priors at the expense of the veridical (or illusory) feedback of how heavy each cube felt during the previous lift. This means that, rather than fine-tuning the internal model of how much force would be required for that particular object, the model is constantly biased by long-term expectations of heaviness. The reason for this may simply be that under conditions where performance may suffer the best solution in the vast majority of cases would be to resort to using what almost always works: namely, higher rates of force to lift larger objects and lower rates of force to lift smaller objects. It is only when one encounters stimuli that defy the statistics of the natural world, such as SWI cubes, that this solution becomes maladaptive. 

To sum up, the current series of experiments has demonstrated that participants have difficulty scaling their fingertip force rates to the actual (identical) mass of SWI objects when they are forced to lift those objects without vision. However, given that subjects were able to scale their fingertip force rates when only the task-relevant visual information was occluded, it is unlikely that this effect arises from seeing the lifting hand or object per se. Instead, when all vision is removed, the motor system appears to rely more on long-term expectation-based priors (i.e., that big objects typically weigh more than small objects) and less on the immediate feedback from the lift. 

Acknowledgments

The authors would like to thank J. Ladich for constructing the SWI cubes, in addition to H. Yang for technical support. The authors would also like to thank N.A. Anwar, J. Lai, and several anonymous reviewers for their constructive comments on earlier drafts of the manuscript. 

The research project was funded by a Canadian Institutes of Health Research (CIHR) operating grant awarded to M. A. Goodale and a Natural Sciences and Engineering Research Council of Canada (NSERC) summer studentship awarded to N. Ranger. 

Commercial relationships: none. 

Corresponding author: Gavin Buckingham. 

Email: [e-mail]. 

Address: Centre for Brain and Mind, Natural Sciences Centre, University of Western Ontario, London, Ontario N6A 5C2, Canada.