Tactile force perception depends on the visual speed of the collision object

Abstract

Previous research on the interaction between vision and touch has employed static visual and continuous tactile stimuli, and has shown that two kinds of multimodal interaction effect exist: the averaging effect and the contrast effect. The averaging effect has been used to explain several kinds of stimuli interaction while the contrast effect is associated only with the size-weight illusion (A. Charpentier, 1891). Here, we describe a novel visuotactile interaction using visual motion information that can be explained with the contrast effect. We show that the magnitude of tactile force perception (MTFP) from an impact on the palm is significantly modified by the visual motion information of a virtual collision event. Our collision simulator generates visual stimuli independently from the corresponding tactile stimuli. The results show that visual speed modified MTFP even though the actual contact force remained constant: higher visual pre- and post-collision speeds induced lower tactile force perception. Finally, we propose a quantitative model of MTFP in which MTFP is expressed as a function of the visual velocity difference, suggesting that the gain of the tactile perception in the human brain is altered via MTFP modulation.

Introduction

We sense pressure, texture or vibration through mechanoreceptors in our hands (Johnson, Yoshioka, & Vega-Bermudez, 2000; Vallbo & Johansson, 1984). However, tactile perception does not depend solely on the sensation from these mechanoreceptors; several studies have demonstrated that it can be altered by visual or auditory information. 

Two kinds of multimodal interaction effect have been found in humans: the averaging effect and the contrast effect. The averaging effect involves the perception of an intermediate value from multimodal information. For example, the perceived object size is determined by a weighted average of visual and tactile information (Ernst & Banks, 2002). This type of interaction was found not only in size perception (Gepshtein, Burge, Ernst, & Banks, 2005; Helbig & Ernst, 2007) but also in texture perception (Guest, Catmur, Lloyd, & Spence, 2002; Lederman & Abbott, 1981; Lederman, Thorne, & Jones, 1986; Weisenberger & Poling, 2004) and in the perception of number of stimuli (Bresciani, Dammeier, & Ernst, 2006; Wozny, Beierholm, & Shams, 2008). On the other hand, when subjects, using their hands, lift two objects of the same mass but which have visually different volumes, they perceive the smaller object to be heavier. This phenomenon is known as the size-weight illusion or the Charpentier effect (Charpentier, 1891; Murray, Ellis, Bandomir, & Ross, 1999). If we use weighted average to explain this effect, then the visually larger object should be perceived as heavier than the visually smaller object, which is not the case. Thus, the size-weight illusion is caused by another type of interaction with a contrast effect. Until now, this type of interaction has been exclusively related to the size-weight illusion. However, here, we describe a multimodal interaction with a novel contrast effect caused by dynamic visual information. When a moving object collides with our body, we perceive a collision impact. For example, in a ball game, a moving ball exerts brief pressure on the player's hand when caught (collision event) and then the impact with the colliding object is perceived. We reveal experimentally that the visual speed of the object modifies the perceived tactile impact. 

In the present study, we simulated a ball-collision event on the human palm as a phenomenon caused by a brief tactile stimulus and a dynamic visual stimulus. By using such a dynamic event, the effect of the visual stimulus can be investigated independently of the effect of the tactile stimulus. We conducted two experiments in which a contact force stimulus was provided synchronously with a visual motion stimulus. The visual motion stimulus was independently controlled in each collision event while the contact force remained constant. In Experiment 1, to investigate the effect of visual speed on tactile perception, we presented a visual stimulus (a ball that would collide with the palm) which moved at a constant speed (pre-collision speed), bounced on the palm, and left the palm at a constant speed (post-collision speed). The results showed that not only pre-collision speed but also post-collision speed can contribute to tactile perception. Higher visual pre- and post-collision speeds induced significantly lower tactile force perception. 

In Experiment 2, to reveal any effect of visual acceleration on tactile perception, a moving visual stimulus with acceleration was presented. The results indicated that pre- and post-collision speeds, not acceleration, are critical factors for modulation of the magnitude of tactile force perception (MTFP). 

Finally, we propose a quantitative model of the tactile perception when dynamic visual information is taken into account, and demonstrate that this model can also explain quantitatively the size-weight illusion. 

Experiment 1

The purpose of Experiment 1 is to investigate how the visual speed of a colliding object affects the MTFP. We developed a virtual reality system to simulate a collision phenomenon on the palm ( Figure 1), in which the speed of the visual stimulus and the contact force of the tactile stimulus were independently controlled. Using a constant tactile stimulus and various visual speeds, we studied how visual information modulates MTFP. 

Figure 1

 

Apparatus. (a) An overall image of the apparatus. Subjects viewed the reflection of the visual stimulus, presented on a CRT. They couldn't see anything except the visual stimulus. (b) An image of the visual stimulus. The virtual ball (left) touches and bounces on the palm. Subjects saw this information through the shutter glasses. (c) Cylinder actuator from the subject's side view. It was set to present a collision force stimulus. Subject's right hand was fixed to the right side of a board by rubber belts. The real subject's hand was hidden from sight with a wooden board.

Methods

Subjects

Fifteen subjects (13 males and 2 females) participated in Experiment 1. In addition, two more subjects participated in the study, but they could not complete the experiment because they failed to produce consistent results when presented with the same stimuli. 

All subjects were naïve to the purpose of this study. They were right handed and had normal or corrected-to-normal vision. No subject reported abnormal tactile perception in daily life. Prior to the study, we obtained informed consent from all subjects, complying with the regulation of life science experiments conducted at Yokohama National University. 

Apparatus and stimuli

Subjects stood in a booth covered with black-out curtains and observed a visual stimulus presented on a CRT monitor (SONY) through two mirrors ( Figure 1a). The optical distance between the subject's eyes and the monitor was approximately 85 cm. Their right hand was hidden under a wooden board, and the CRT monitor was used to present an image of a right hand in the same position as the subject's real hand (with the appropriate depth cues) such that the virtual hand was likely to be perceived as the subject's real hand. The distance between the eyes and the real hand was approximately 65 cm. A stereoscopic system (STEREOGRAPHICS, Crystal EYES3) was used to present the visual stimulus and to perceptually match the distances of the virtual hand with the real one. 

Each subject observed a virtual hand and a virtual ball generated by OpenGL. The diameter of the virtual ball was 1.5 degree of visual angle (2 cm in diameter on the monitor). The virtual ball kept the same appearance in all experiments and moved horizontally on the monitor, first from the left to the right until it collided with the virtual hand and then moved immediately back to the left ( Figure 1b). 

A contact force to the subject's palm was applied using a cylinder actuator (ORIENTAL MOTOR CO. EZHC4A-05M) with an attached hard rubber sphere. The cylinder moved horizontally, touched the subject's hand, and moved away from the hand in synchrony with the virtual ball motion on the monitor. A contact force was applied to the palm of the right hand. The duration of the tactile stimulus was approximately 70 ms. For the purpose of ensuring that the contact position on the subject's hand remained the same throughout the experiments, the subject's hand was fastened by rubber belts ( Figure 1c). A noise-canceling headphone (BOSE, QuietComfort2) was used to mute the motor sound during the experiments. 

There were two phases per trial in the experiment: the reference phase and the test phase. In the reference phase, the cylinder actuator moved 10 mm horizontally from the start position, which was about 5 mm from the first contact position on the subject's palm. The collision speed (pre-collision speed) of the virtual ball was 32 cm/s, and the speed after the collision (post-collision speed) was 16 cm/s, resulting in a restitution coefficient of 0.5. The MTFP in the reference phase was defined as 100 and was used as the base for comparison with the MTFP in the test phase. 

In the test phase, five combinations of visual stimulus speeds (pre-collision speed/post-collision speed), i.e., 8/8, 8/48, 32/16, 64/8, and 64/48 ([cm/s]/[cm/s]), were adopted. These pairs of test conditions were selected from pilot experiments. The cylinder moved in the same manner as in the reference phase in all test conditions except for dummy trials. Ten dummy trials having a different cylinder movement were presented randomly in the test phase to prevent the subjects from noticing that all tactile stimuli might be constant. In the dummy trials, the visual stimuli were the same as the ones in the normal test trials; however, the cylinder movement was 12, 11, 9, or 8 mm. All test conditions were presented one time each in a single session. Five sessions were performed on each subject. 

Procedure

We measured the magnitude of tactile force perception with a magnitude estimation method. In a trial, we presented alternately a reference (control) stimulus pair (visual and tactile stimuli) and a test stimulus pair. After each test phase (when the visual stimulus in the test condition disappeared to the left side of the display), subjects were instructed to estimate the MTFP in the test phase in comparison with the reference MTFP set as 100. It was not permitted to recheck the stimuli presented. Before starting each experiment, subjects practiced for several trials including dummy conditions in order to familiarize themselves with the stimuli. The order of the stimuli was randomized in each session, and a short rest period was given between trials. Also a break time was given for each session. About 2 hours were needed to complete 5 sessions per subject, it was done in one day or two days separately. 

Results

Results from two subjects were excluded from our statistical analysis because their magnitude responses for each session deviated from the results of the other subjects. ¹ Figure 2 shows the average results for Experiment 1. The horizontal axis represents collision speeds of the virtual ball and the vertical axis represents MTFP. Plotted points indicate averaged values of MTFPs from 13 subjects. Figure 2 shows that the MTFPs were influenced by the speed of the visual stimulus both before and after the collision. When the post-collision speed was kept constant, higher pre-collision speeds induced significantly lower MTFP values. Similarly, when the pre-collision speed was kept constant, higher post-collision speeds induced significantly lower MTFPs (two-way repeated measures ANOVA; pre-collision speed, 8 and 64 cm/s, × post-collision speed, 8 and 48 cm/s). For pre-collision speed, F [1, 12] = 19.5, p = 0.001, η _p ² = 0.62. For post-collision speed, F [1, 12] = 9.7, p = 0.009, η _p ² = 0.45. In addition, there was no interaction effect ( F [1, 12] = 0.26, p > 0.1, η _p ² = 0.02). 

Figure 2

 

Experiment 1: Magnitude of tactile force perception (MTFP) under a constant contact force and as a function of visual pre-collision and post-collision speed. Symbols indicate averaged data for 13 subjects. It indicates that higher pre-collision speed and higher post-collision speed induce lower MTFP. Error bars indicate standard errors between subjects.

Moreover, we analyzed the results from the condition where the test stimulus matched the reference stimulus, that is the 32/16 cm/s (pre/post collision speed) condition. The reason why we made a comparison with the 32/16 cm/s condition and not with the reference MTFP(100) was to see the effect of removing the stimulus order. (There was no significant effect between MTFP from 32/16 cm/s condition and MTFP reference: 100.) Thus, the comparison of MTFPs obtained from test conditions was reasonable in order to rule out any order effect. Significant differences were observed when compared with the 8/8 collision speed condition ( t-test; t[12] = 3.44, p = 0.005) and 64/48 collision speed condition ( t[12] = 2.39, p = 0.034). On the other hand, there were no significant differences with the 8/48 speed condition ( t[12] = 0.90, p = 0.39) and 64/8 speed condition ( t[12] = 1.87, p = 0.085). 

To confirm the effects of visual information on MTFP, we conducted a control experiment using the same method as in Experiment 1 but with the CRT monitor off. The purpose of this experiment was to investigate if the effect in Experiment 1 was caused by visual information influencing the tactile perception and not caused by the tactile stimuli timing between the tactile collision in the reference phase and in the test phase. The tactile stimulus timing between the two phases was different when pre-collision speed in the test phase was 8 or 64. Also, if post-collision speed was different, the time span between the haptic collision timing in the test phase and MTFP response timing differed. These effects were tested in this control experiment. 

In the test phase, the virtual speed conditions were 8/8 and 64/48 cm/s (but visual stimulus was not presented). The tactile stimulus presentation timing was the same as in Experiment 1. The starting cue to present the stimulus and the ending cue to respond MTFP were presented orally to keep the same timing condition as in Experiment 1 under no visual condition. Ten subjects participated in the experiment. The results showed no significant difference between the 8/8 cm/s and 64/48 cm/s speed conditions ( t-test; t[9] = 1.93, p = 0.09) ensuring that there was no artifact effect caused by the apparatus or the tactile stimulus timing and the subject's response timing. 

These results clearly show that the MTFPs in Experiment 1 were modified by the visual motion information in the collision events. Not only the visual motion speed before the collision, but also the visual motion information after the collision contributed to the altered estimation of MTFP in the brain. 

Experiment 2

In Experiment 2, we investigated whether tactile perception in a collision event is influenced by the visual acceleration information of the collision object. 

Methods

Subjects

Seven subjects (6 males and 1 female) from Experiment 1 participated in Experiment 2. 

Apparatus and stimuli

The apparatus was the same as in Experiment 1. In the reference phase, visual and tactile stimuli were presented as described in Experiment 1. There were 8 conditions in the test phases: 4 uniform motion conditions and 4 accelerated motion conditions. For the uniform motion condition, the visual collision speed was set to 8, 16, 48, or 64 cm/s without acceleration. The 4 accelerated motion conditions were with a constant acceleration (−20, −16, 26, or 64 cm/s ²) and with an initial speed of 32 cm/s; i.e., the visual collision speed was set to 8, 16, 48, or 64 cm/s. In all conditions, the restitution coefficient of the visual stimuli was 0.5 and no acceleration was added after collision. For example, if the collision speed was 48 cm/s, then the post-collision speed was 24 cm/s. 

The tactile stimulus was applied as described in Experiment 1. In addition, 8 dummy trials were included per session, and 5 sessions were repeated for each subject. 

Procedure

Five sessions were performed for each subject. Subjects provided a numeric value corresponding to MTFP for each test stimulus in comparison with the reference MTFP. The details of the procedure were the same as in Experiment 1. 

Results

Figure 3 shows the averaged values of MTFPs from 7 subjects as a function of collision speed of the visual stimulus. The open symbols represent the average results for the accelerated motion conditions and the filled symbols represent the average results for the uniform motion conditions. Error bars indicate standard errors. These results are in accordance with the results in Experiment 1 in that the visual information modified MTFP despite the fact that the contact force remained constant. 

Figure 3

 

Experiment 2: Relationship between pre-collision speed and magnitude of tactile force perception (MTFP). Uniform motion conditions and accelerated motion conditions are indicated by filled and open symbols, respectively. Error bars indicate standard errors between subjects.

A significant difference was detected among the visual collision speed conditions (two-way repeated measures ANOVA; F [3,18] = 23.24, p < 0.001, η _p ² = 0.80), but not between the uniform motion and accelerated motion conditions ( F [1,6] = 3.72, p = 0.1, η _p ² = 0.38). In addition, there was no significant acceleration × collision speed interaction ( F [3,18] = 0.12, p > 0.1, η _p ² = 0.02). We conclude that the collision speed was a critical parameter in the estimation of MTFP, but the visual acceleration and the initial speed of the visual stimulus were not. By combining the results of Experiments 1 and 2, it is suggested that MTFP for a constant contact force can be predicted using a formula as a function of visual pre-collision and post-collision speeds. 

Discussion

This study has revealed that information on the visual speed of a collision object serves to modify the perception of its tactile force when hitting a subject's palm. Under natural conditions, the higher the speed of an object when it hits the palm, the greater the perceived impact on the palm. However, the present results show that visual collisions at higher speeds induced lower MTFPs despite the fact that the actual contact force was kept constant. This interaction can be considered as a contrast effect similar to what it occurs in the size-weight illusion. 

The results of Experiment 1 ( Figure 2) show that the visual motion speed not only before but also after the collision modulated the MTFPs. The results of Experiment 2 indicated no effect of acceleration on the MTFPs. To explain our findings, we focused on the visual velocity difference Δ V (a vector quantity) between the visual pre-collision velocity V _pre and the visual post-collision velocity V _post. In our experiments, V _pre becomes a positive value and V _post is a negative value in vector quantity because the visual object (ball) reversed direction after collision. Thus, Δ V becomes the summation of pre-collision velocity and post-collision velocity absolute values (i.e., differential velocity will become 48 in the 32/16 cm/s condition). Theoretically, Δ V is related to the impulse I in collision phenomena. Therefore, their relationship can be formulated as I = m · Δ V ∝ Δ V, where m represents the mass of the ball. We assume that m can be represented as a constant value because the same virtual ball was used in all our experiments. No significant difference was detected between the MTFP from the 32/16 condition and the MTFP from the 8/48 or 64/8 speed condition in Experiment 1, supporting the differential velocity hypothesis because all the conditions have similar Δ V ( Figure 4). 

Figure 4

 

Magnitude of tactile force perception (MTFP) as a function of differential velocity. Symbols represent averaged data. Error bars indicate SE between subjects. The regression curve was plotted using formula (7). n value is 0.059. Determination coefficients ( R ²) were 0.92 for Experiment 1 results.

Figure 4 shows the re-plotted results from Experiments 1 and 2 (only MTFPs for the uniform motion condition) as a function of the velocity difference Δ V, demonstrating that larger velocity differences induced lower MTFPs. Therefore, we assume that MTFP can be described by the following formula:  

$MTFP=K(\Delta V)f(CF),$

(1)

where K represents the gain factor as a function of visual velocity difference, Δ V, and f represents a function of contact force, CF. The absolute magnitude of tactile force perception in the reference phase MTFP _ref can be derived as  

$MTF{P}_{ref}=K_{ref}\left(\Delta V_{ref}\right)f(CF),$

(2)

where K _ref and Δ V _ref represent the gain factor and differential velocity in the reference phase, respectively. Similarly, the absolute magnitude of tactile force perception in the test phase MTFP _test can be represented as  

$MTF{P}_{test}=K_{test}\left(\Delta V_{test}\right)f(CF),$

(3)

where K _test and Δ V _test represent the gain factor and differential velocity in the test phase, respectively. Since we used a magnitude estimation method, the subjects estimated the MTFP _test in comparison with the MTFP _ref. Therefore, the subject's response ( res) can be described as a ratio of the two MTFPs as follows:  

$res=100\frac{MTF{P}_{test}}{MTF{P}_{ref}}.$

(4)

Note that the MTFP value in the reference was defined as 100 in the present study. Thus, res can be described as  

$res=100\frac{K_{test}\left(\Delta V_{test}\right)}{K_{ref}\left(\Delta V_{ref}\right)}.$

(5)

Additionally, based on our experimental results, we derive K using the following formula  

$K\propto (\frac{1}{\Delta V}{)}^n=(\frac{1}{|V_{post}-V_{pre}|}{)}^n,$

(6)

where n represents a power constant that accounts for the nonlinearity between perception and physical dimensions. For describing the contrast effect, K was defined by the reciprocal of visual differential velocity. 

Finally, we obtain the following formula:  

$res=100(\frac{\Delta V_{ref}}{\Delta V_{test}}{)}^n.$

(7)

The curve in Figure 4 was adapted to fit this formula for the results of Experiment 1 by setting n as 0.059. The determination coefficient was 0.92. Although the results from Experiment 2 were not used to determine the regression curve, their magnitude coincided well with the curve (the determination coefficient for the data from Experiment 2 was 0.95 using this curve). 

The visual information gain factor K as a function of the reciprocal of visual differential velocity was able to explain our results for MTFP. This suggests that the gain factor K optimizes the resolution of the tactile perception, which is allocated depending on the dynamic visual stimuli information. In the real world, a larger Δ V yields a greater impact on the palm. Therefore, the gain factor K is lowered to compress the perceptual response and to expand the range of pressure force perception when a larger Δ V is presented visually. Conversely, a lower Δ V makes the gain factor K higher to take into account a low contact force in high resolution level. 

Moreover, we investigated whether our gain control model can explain the size-weight illusion which produces the same interaction effect. If our model can explain the size-weight illusion, this would indicate the generality of our suggested model of contrast interaction. To investigate the size-weight illusion results, we compared our model with the regression function from Ellis and Lederman (1993) Experiment 1 (haptic-plus-vision conditions). In their study, the standard stimulus was an object weighing 350 g with a volume of 1093 cc, and the test stimuli volume were 149, 493, 1093, 2097, 3652, 5832 and 8615 cc but weighting the same 350 g. (In their experiment, other conditions were also tested but are not described here.) The subjects were instructed to set the standard stimulus as 100 and estimate the weight using a numeric value relative to 100. From their results, they proposed a regression equation for the size-weight illusion as 

$Lo{g}_{10}W=-0.09+1.25Lo{g}_{10}{H}_{test}-0.35Lo{g}_{10}V{O}_{test},$

(8)

where W represents the estimated weight, H_test represents the physical weight of the test objects, and VO_test represents the physical volume of the test objects. In their experimental settings, H_test was 350. Thus, the formula can be calculated as 

$Lo{g}_{10}W=-0.35Lo{g}_{10}V{O}_{test}+3.09.$

(9)

On the other hand, if we apply our gain control formula (7) to the size-weight illusion, physical volume should be the gain control factor. If the reference magnitude is defined as 100, then the magnitude of weight perception can be described as 

$W=100(\frac{V{O}_{ref}}{V{O}_{test}}{)}^n.$

(10)

Substituting 1093 for VO_ref, formula (10) can be described as 

$Lo{g}_{10}W=-n·Lo{g}_{10}V{O}_{test}+3.04n+2.0.$

(11)

This indicates that our gain control formula can be transformed into the same type of formula as (9) to represent the size-weight illusion. Therefore, our suggested model can fit the size-weight illusion results, confirming the generality of this gain mechanism. Furthermore, if we use 0.35 for the value n in formula (11), we obtain 

$Lo{g}_{10}W=-0.35Lo{g}_{10}V{O}_{test}+3.06,$

(12)

which is the same as formula (9). 

In this study, we have shown that visual information of a post-collision event modulates the perception of contact force. Some previous studies showed that information after an event could modulate previous perception: the flash-lag effect is one example. When a flash and a moving object are presented in the same location, subjects perceive mismatched positions between the two stimuli. To produce this effect, the visual moving stimuli, presented within several tens of milliseconds after the flash, modulate the flash position (Eagleman & Sejnowski, 2000). In our case, the calculated visual differential velocity changed MTFP gain. Therefore, the MTFP was determined in the time gap between the presentation of the contact force and the post-collision visual information. Our experiment included information from two modalities, whereas the flash-lag experiment included only visual information. However, similar effects were shown in previous studies; for example, for tactile apparent motion (Geldard & Sherrick, 1972) and for audiovisual information (Fujisaki, Shimojo, Kashino, & Nishida, 2004; Spence & Squire, 2003). It is no wonder that post-event visual information was also perceived as a part of the single collision event in the brain. As a result, such MTFP modulations, like the ones presented in this study, may occur. 

These considerations indicate that the phenomenon we describe occurred at a perceptual level. Visual information presented after the tactile stimuli could not modify the sensation level. Flanagan and Beltzner (2000) insist that the size-weight illusion is caused at a cognitive or perceptual level, not at a sensorimotor level, although the illusion originates from the difference between the weight expected from visual information and the actual object weight as a sensory mismatch (Ross, 1969). The similarity of the results, models and expected processing level between our illusion and the size-weight illusion indicates that these phenomena may be processed at the same level in our brain. 

Previous studies (Ernst & Banks, 2002; Gepshtein & Banks, 2003) demonstrated that a multimodal integration mechanism optimally averages visual information and haptic information with weighting factors corresponding to the variance and likelihood of each piece of information. These studies explained their results using an averaging manner in multimodal interaction. On the other hand, the contrast effect in multimodal interaction remains controversial (Ernst, 2009). In the present study, we found a novel interaction between contact force perception and visual motion information. Similarly to the size-weight illusion, the present results cannot be explained by the averaging effect. If visual and tactile collision impacts were integrated using weighted averaging, MTFP would have increased as visual velocity difference increased, but this was not the case. Therefore, our results indicate that there is a novel type of integration mechanism in which visual information modulates the gain of the tactile perception to extend the force perception range. In addition, we have shown that such a contrast effect can be described by a function of visual differential velocity. 

Acknowledgments

We are grateful to John S. Werner, Carlos Arce Lopera, and two anonymous reviewers for many helpful comments to improve this manuscript. 

Commercial relationships: none. 

Corresponding author: Katsunori Okajima. 

Email: [e-mail]. 

Address: Research Institute of Environment and Information Sciences, Yokohama National University, 79-7 Tokiwadai, Hodogaya-ku, Yokohama, Kanagawa, Japan. 

Footnote

Footnotes

¹  The two excluded subjects had high variability from 5 repeated measures on each condition. Thus, we decided that those two subjects were unsuitable to use in our analysis. However, if the two subjects were included in the two-way repeated measures ANOVA (pre-collision × post-collision), significant effect of pre-collision speed & post-collision speed and no interaction effect are shown, the same results as presented in Experiment 1.