Motor action reduces temporal asynchrony between perceived visual changes

Abstract

Abstract

Abstract:

Abstract  Perceiving a visual object requires binding sensory estimates of its various physical attributes. This process can be facilitated if changes of different attributes are perceived with little asynchronies when they are physically aligned, which is not always the case as revealed by temporal order judgment or perceptual synchronization tasks of visual attributes changes. In this study, we analyzed the effect of performing a motor action on the perceived relative timing between changes of position and color of a visual target by using a temporal order judgment (TOJ) task. Results showed that in the perceptual condition, the change of color must precede (−37.9 ms) the change of position in order to perceive a synchronous change of both target's visual attributes. This physical asynchrony vanished when the same changes took place near the end of a manual reaching action executed towards the visual target (−3.3 ms). The reduction of asynchrony was, however, not observed when participants performed TOJ of visual attributes change in the presence of concomitant tactile information (−36 ms) but with no action. The perceptual relative timing between visual changes was also unaffected when the timing was obtained by comparing each visual change to tactile information resulting from motor action (−33.5 ms) or external stimulation (−27.8 ms). Altogether, these results suggest that signals associated with the organization of a motor action, but not sensory information itself, contribute to reduce the differential delays when processing visual attributes of a single object. Furthermore, the effect of action was not observed when judging relative timing of object-related (visual) versus object-unrelated (tactile) sensory information.

Introduction

Maintaining a unitary percept of a visual object, despite the constant variation of its characteristics (position, color, shape) over time, requires continuously binding the various sensory estimates of its physical attributes. We know that different parts of the brain process different aspects of sensory information, though we still have a unitary conscious experience of external stimuli. Concerning the visual system, the dominant view is that there is a sharp division of labor between a visuo-motor system processing visual information through the dorsal pathway from the primary visual cortex (V1) to the posterior parietal cortex and a visuo-semantic system processing visual information through the ventral pathway from the primary visual cortex (V1) to the inferotemporal cortex (Milner & Goodale, 1995, 2008). Accordingly, components of sensory processing are considered as being dependent on behavioral purpose. Within this theoretical framework, action-related parameters such as size, location, and orientation are thought to be processed through the dorsal stream of the visual system (e.g., Pisella et al., 2009), whereas perceptual aspects for object identification such as shape, color, texture, and weight are thought to be processed through the ventral stream of the visual system (e.g., Op de Beeck, Haushofer, & Kanwisher, 2008). However, latencies in processing sensory information within the visual system are lower within the dorsal stream (around 40–80 ms within parietal areas) than within the ventral stream (around 100–150 ms within temporal areas, Tanné, Boussaoud, Boyer-Zeller, & Rouiller, 1995) depending mainly on whether cortical areas are activated by heavily myelinated, fast conducting magnocellular fibers or parvocellular fibers characterized by slower transfer of information (Nowak & Bullier, 1997). In agreement with these differential neural delays, psychophysical experiments have shown that the perception of stimuli changes is dependent on the estimated attributes. For instance, reaction times in a location discrimination task were reported to be within the range of 350–450 ms (Rossetti, Koga, & Mano, 1993; Tanaka & Shimojo, 1996), whereas reaction times to discriminate stimuli on the basis of semantic information were found to be within the range of 400–650 ms (Bartolo, Weisbecker, & Coello, 2007; Sereno & Rayner, 2003). In the same vein, Pisella, Arzi, and Rossetti (1998) showed that color information is processed about 80 ms slower than position information in a task where participants were instructed to interrupt their motor action when the target changed location or color. 

The direction of the perceived asynchrony does not always conform to the neurophysiological data. When pairing direction of motion with color many studies have found a color-motion asynchrony (CMA) in which color leads motion (e.g., Linares & López-Moliner, 2006; Moutoussis & Zeki, 1997; Nishida & Johnston, 2002; Viviani & Aymoz, 2001). However, an explanation of the illusion invoking an extra delay for processing the direction of motion has been proposed by some authors (Arnold, Clifford, & Wenderoth, 2001; Bedell, Chung, Ogmen, & Patel, 2003; Moutoussis & Zeki, 1997). Furthermore, the CMA has been shown to be task-dependent (e.g., whether color change occurred simultaneous or after motion onset, Gauch & Kerzel, 2008; see also Linares & López-Moliner, 2006; Nishida & Johnston, 2002) and these temporal variations in processing visual attributes might be related to the paradigms used (e.g., reaction time versus temporal order judgment, see Cardoso-Leite, Gorea, & Mamassian, 2007 for a discussion) or stimuli characteristics (e.g., stimuli saliency, Adams & Mamassian, 2004; Miller & Schwarz, 2006). 

Beside the discussion about the causal role of differential delays in the explanation of the CMA or related asynchronies, the parallel processing of separate attributes naturally raises the issue of their integration, i.e., how the independent attributes are bound together through an active process providing a unitary percept of visual objects (von der Malsburg, 1995; Roskies, 1999; Schmidt, 2009; Treisman, 1999). Several mechanisms have been proposed to account for sensory binding, focusing either on neural synchronization or temporal constraint associated with cognitive functions. At the neuronal level, integration could emerge from the temporal synchronization of neural activities that, in different networks, signal the processing of different attributes of the same event (König & Engel, 1995; Singer & Gray, 1995; von der Malsburg, 1995) using neural oscillations as binding mechanism (Engel & Singer, 2001). Another possibility would be to use conjunctive neural representations involving persistent patterns of weight strengths between neural units, so that different neural units are activated for different combinations of input features (O'Reilly, Busby, & Soto, 2003). At the functional level, theories have mainly focused on memory (Fougnie & Marois, 2009; Keizer, Colzato, & Hommel, 2008), attention (Kahneman, Treisman, & Gibbs, 1992; Treisman & Gelade, 1980) mechanisms, or time perception (Wenke & Haggard, 2009) to account for sensory or features binding. 

Despite the existence of different theoretical frameworks, the common idea is that attributes of the relevant events in the visual scene depend on a specific, though context dependent, time register before reaching consciousness (Aymoz & Viviani, 2004). By contrasting passive and active conditions, other authors have suggested that endogenous activity may also contribute to sensory binding. Haggard, Clark, and Kalogeras (2002), for instance, showed that performing a voluntary action influences the temporal perception of sensory events associated with that action. In their study, the perception of a sound was shifted 46 ms earlier in time relative to a baseline condition when it was preceded by a voluntary motor action. Interpreted within the framework of intentional binding by the authors, this effect underlines a contribution of “predictive models of motor control in constructing conscious experience of action.” Indeed, voluntary actions imply predictive mechanisms which are used to anticipate expected sensory consequences of acting, evaluate action feasibility (Jeannerod, 2006), and also guide actual motor behavior (Wolpert, Ghahramani, & Jordan, 1995). These predictive mechanisms are thought to influence the temporal perception of sensory and motor events (Haggard et al., 2002; Stetson, Cui, Montague, & Eagleman, 2006; Wenke & Haggard, 2009). Consequently, the perception of different object attributes is also expected to be influenced by whether or not these attributes vary in relation to voluntary motor action, particularly when they are processed with different latencies (Nowak & Bullier, 1997). In agreement with this, previous studies have shown that the temporal constraints associated with object color and position processing are affected by the observational context. For instance, Aymoz and Viviani (2004) showed that when embedding visual changes into an observed biological movement, the perception of object displacement did not lag the perception of color change as it is usually found in perceptual tasks. Thus, in the action condition the perceived temporal synchronization of attribute changes may not exclusively result from relative time delays associated with the neural organization of the visual system (Nowak & Bullier, 1997) or perceptual anticipation due to prediction of the visual event since the reduction of temporal asynchrony was not observed when the object displacement resulted from the impact of a nonbiological stimulus (Aymoz & Viviani, 2004). The authors' interpretation of this effect was that the moving hand calls into play the interaction between the visual and the motor system. It is indeed acknowledged that observed human actions are usually interpreted within a neural network similar to the one used for planning and executing actual motor actions (Grezes, Costes, & Decety, 1998). Generating a reliable representation of a human action may then provide an access to predicted sensory consequences of acting, and thus may contribute to the binding mechanism whereby all attributes of the relevant events in the visual scene are set into strict time register before reaching consciousness (Moore & Haggard, 2008). As a consequence of predictive motor mechanisms, the neural timing mechanism could be modulated during the action execution period in order to facilitate the processing and integration of action-related sensory signals (Wenke & Haggard, 2009). In case of variable delays in sensory and motor pathways, the temporal relationship between action and sensations is thought to be recalibrated to overcome changing latencies and to restore both the perception of object unity (Yarrow, Haggard, Heal, Brown, & Rothwell, 2001) and perceived events causality (Stetson et al., 2006) or agency (Haggard et al., 2002). 

However, the effect of an actual motor action on the perceived differential delays between changes of different sensory attributes has never been tested and this represents the aim of the present study. We initially used a temporal order judgment task of changes of color and position attributes of a visual target in a manual reaching task. As mentioned above, there is some consensus that in the context of action, perception of visual events obeys some principle of temporal predictability. In real life, our actions and the following expected sensory consequences are already calibrated so that we perceive this sequence without any conspicuous delay despite the neural delays inherent in the nervous system. For example, the retinal flow while moving our head or the visual detection of the effects of our hand actions are perceived without any apparent delay. Then, we tested in Experiment 1 whether the physical asynchrony between color and position changes expected in a perceptual condition was reduced when articulating the same changes with a motor action due to prediction mechanisms. However, reaching tasks imply also tactile information around the movement end-point which is known to contribute to the spatio-temporal aspects of sensorimotor performances. Indeed, signals from tactile receptors participate in encoding the position sense of the fingers (Gentilucci, Toni, Daprati, & Gangitano, 1997; Gordon & Soechting, 1995; Rabin & Gordon, 2004) and contribute to the timing control in sensorimotor synchronization tasks (Aschersleben, 2002; Billon, Semjen, Cole, & Gauthier, 1996; Stenneken, 2006). Furthermore, tactile information was found to be as accurate as the other sensory information (visual or auditory) in temporal order judgment (TOJ) tasks (Gallace & Spence, 2008; Hirsh & Sherrick, 1961). Then, to show that action-related information (internally predicted) and not subsequent tactile information is crucial for reducing perceptual asynchrony of visual attribute changes, we also tested whether the mere presence of tactile information without performing any action (not internally predicted) influences the perceived differential delays between color and position changes. Finally, cross-modal information can be affected differently by voluntary action. Tactile information, for instance, does not always vary congruently with visual information depending on object (soft or rigid) or action (touching, grasping, pulling, etc.) properties and is subjected to sensory attenuation effect when contingent to intentional motor productions (Blakemore, Frith, & Wolpert, 1999). We then tested in Experiment 2 whether the effect of voluntary action on sensory processing is still preserved when comparing cross-modal information (i.e., visual and tactile information) in a TOJ task involving or not intentional action. 

Experiment 1: Effect of motor action on perceived relative timing between position and color information

This experiment was designed to test the effect of performing an intentional action on the temporal order judgments of changes of position and color attributes of a visual target (motor condition) compared to a passive condition where no action was performed and the participant received (tactile condition) or not (perceptual condition) a tactile stimulation on the forefinger around the time position and color attributes of the visual target changed. 

Method

Participants

Ten participants (six females) aged between 21 and 32 years (mean age: 25.8 years ± 3.4 years) participated in the two experiments. None of them reported any sensory or motor deficits and all had normal or corrected-to-normal vision. They were recruited as volunteers and all were naïve to the purpose of the experiment (except for one participant who was one of the authors). Participants gave their informed agreement and were volunteers to take part to the experiment, which was performed in agreement with the local ethical committee guidelines and in accordance with the declaration of Helsinki. 

Stimuli and apparatus

Participants sat in a dimly lighted room in front of a 22-in CRT computer screen (Mitsubishi Diamond Pro 2070 SB, spatial resolution: 1024 × 768 pixels, sampling rate: 100 Hz) at a viewing distance of 30 cm. A response box was positioned on the table close to the body so that the participant could easily provide a response by pressing one of the two buttons. Custom software using Delphi routines was used to control the sequence of displayed images and to record the position of an ultrasonic marker for which the (x, y, z) coordinates were provided by a ZEBRIS 3D recording system (Zebris Medical GmbH, Isny, temporal resolution: 100 Hz, spatial resolution: 0.5 mm). The ultrasonic marker was placed on the participant's right forefinger and was used to trigger the trial and to change the image on the computer screen depending on its location in space. Each image contained on the lower part of the screen a starting location for the right forefinger (white circle, diameter: 0.5 cm) and a target on the upper part of the screen (red circle, diameter: 1 cm) distant by 24 cm. During the trial, the color and position of the target changed according to a predefined temporal sequence (perceptual condition), according to the position of the right forefinger during action (motor condition), or, finally, according to the time a tactile stimulus was delivered to the participant's forefinger (tactile condition, see below). Position change corresponded to a sudden 1 cm displacement of the target upwards. Color change corresponded to a sudden change from red (International Commission on Illumination-CIE x: 0.413, y: 0.213, z: 0.019) to green (CIE x: 0.358, y: 0.715, z: 0.119). The stimuli were isoluminant (18.8 cd/m²). 

In the tactile condition, the experimental setup was similar to the one described above for the perceptual and motor conditions except that instead of being initially positioned at the bottom of the computer screen, the participant's forefinger was positioned at the center of a computer loudspeaker (diameter 3.5 cm, frequency: 100 Hz, period: 5 ms) placed just beside target location on the computer screen. Without applying any particular pressure on the loudspeaker's membrane, tactile stimulation was provided by the deformation of the speaker membrane in response to a single electrical signal sent by the computer. In order to remove any auditory signal due to the activation of the loudspeaker, participants wore hearing protectors. A pre-experiment test revealed that while wearing hearing protectors, the snapping sound coming from the loudspeaker was not detected by the participant. 

Procedure

The task for the participants was to perform temporal order judgments in three conditions: a perceptual, a motor, and a tactile condition. In the perceptual and motor conditions, the participant positioned their right forefinger at the starting location on the screen and an auditory tone was provided between 500 and 1000 ms after the right forefinger reached the starting location, indicating the beginning of the trial. In the perceptual condition, the reference target attribute (position or color, in block sessions) changed following a delay of 700 ms after the tone. In the motor condition, following the tone the participants had to manually reach the target in about 700 ms (participants were familiarized with this movement duration before the experiment) and the reference target attribute (position or color) change occurred 100 ms before or after finger-target contact. We did so in order to avoid any interference of a concomitant tactile event with the reference visual change. 

In the tactile condition, the task for the participants was to perform temporal order judgments between color and position attributes changes of a visual target while receiving a concurrent tactile stimulation (see Figure 1). The participant positioned their right forefinger at the center of the loudspeaker fastened on the computer screen. In each trial, the beginning of the trial corresponded to the appearance of the target on the screen and the tactile stimulation was provided following a delay of 700 ms. The reference target attribute change (position or color) occurred 100 ms before or after the tactile stimulation. 

Figure 1

 

Time sequence of reference attribute (color or position) and test attribute (color or position) change in the perceptual, motor, and tactile conditions. The change of test attribute could occur within the time window of ±200 ms according to the occurrence of the reference attribute change. In the motor condition (middle panel), reference attribute change occurred at ±100 ms according to the movement end-point and for the tactile condition (lower panel), reference attribute change occurred at ±100 ms according to tactile stimulation.

In all conditions, a test target attribute (color or position) change occurred from −200 ms up to 200 ms relative to the reference change, by steps of 50 ms (resulting in nine possible stimuli onset asynchony, SOA). The occurrence of the test target attribute change was selected according to a pseudorandom order, with each SOA value being presented 10 times. The task for the participants was to indicate which attribute (position or color) changed first using one of the two buttons of the response box (two-alternative forced choice [AFC] paradigm). In the perceptual condition, participants performed 180 trials, Reference Target Attribute (position or color) × SOA (−200; −150; −100; −50; 0; 50; 100; 150; 200 ms) × 10 Trials). In the motor and tactile conditions, participants performed 360 trials, Reference Target Attributes (position or color) × Time for Change of Reference Target Attribute (±100 ms according to the occurrence of tactile information) × SOA (−200; −150; −100; −50; 0; 50; 100; 150; 200 ms) × 10 Trials. The task for the participants was to indicate which attribute (position or color) changed first using one of the two buttons of the response box (two-AFC paradigm) while ignoring the tactile stimulation. 

In all conditions, the participant had to respond as quickly and precisely as possible. To estimate the negative SOA in the condition involving a motor action, we derived the temporal value from the spatial location of the ultrasound marker computed online. These spatial locations were estimated from a pilot study, which revealed that spatial coordinates on average corresponding to −300 ms, −250 ms, −200 ms, −150 ms, −100 ms, and −50 ms were, respectively, 80 mm, 70 mm, 60 mm, 45 mm, 30 mm, and 15 mm from the target location along the vertical z axis of the computer screen. Assuming that movements were executed in about 700 ms, expected average negative SOA between change of test target attribute and finger contact was −150 ms and −50 ms, respectively, when the change of referent target attribute was fixed at either −100 ms (SOA: −300, −250, −200, −150, −100, −50, 0 ms) or 100 ms (SOA: −100, −50, 0, ms). Postexperiment analysis revealed that averaged SOA before finger contact was −129.9 ms (SD: 36.6 ms) and −58.8.6 ms (SD: 16.7 ms), respectively, when the change of the referent target attribute was fixed at −100 ms and 100 ms, these two values being not statistically different from the expected values, t(9) = 1.66, p = 0.13 and t(9) = 1.76, p = 0.11. 

Data analysis

In each condition, the set of data for each participant was the distribution of the proportion of trials in which the test change was perceived later than the reference change. Depending on the session, the reference attribute could be either position or color. For the sake of simplifying the graphical presentation, we normalized the responses and used position change as a reference (position change = zero). We thus plotted the proportion of time in which position was perceived as changing before color irrespective of the time-locked attribute. We applied this procedure to the perceptual, motor, and tactile conditions. In addition to this, we analyzed separately the two distributions coming from the two different locked attributes in order to compare them. Data were pooled over participants and the parameters (mean and standard deviation) of a cumulative Gaussian were estimated using maximum likelihood procedure. The point of subjective simultaneity (PSS) corresponded to the mean of the distribution, which served as a measure of the perceptual asynchrony between color and position change. If the changes were perceived synchronously when presented at the same time, the distribution would be centered on the value zero. In the three conditions, a positive PSS value (curve shifted to the right) denoted that position change must occur before color change for perceiving the two changes in synchrony (i.e., processing of position lagging processing of color) and a negative PSS value would imply the opposite. The standard deviation of the data estimated from the cumulative Gaussian (SD) expressed the sensitivity of the perceptual system, with a higher standard deviation indicating more variable perceptual judgments. We used parametric bootstrap method (Efron & Tibshirani, 1993) to obtain the 95% confidence intervals of the two parameters of the cumulative Gaussian functions after running 2000 simulations. By running simulations, we take into account the nonuniform sampling of data points to estimate the variability of the parameters of the fitted curve. This is important because data points are not equally spaced before and after the finger has touched the screen. 

Results

Temporal order judgment of color and position change in the perceptual condition

When comparing color change and position change in a temporal order judgment task, we found a mean PSS of −37.9 ms, 95% CI (−46/−30.5) and SD = 63.5 ms, 95% CI (48.3/75.6) indicating that color had to change before position in order to be considered as synchronous (see Figure 2). This effect was not affected by the fact that color change was fixed and position change variable, PSS = −36.0 ms, 95% CI (−47.5/−24.9) or the opposite, PSS = −39.9 ms, 95% CI (−50.6/−28.6). The same observation was made relating to the SD of the function, which was not affected by the fact that the color change was fixed and position change occurred ±200 ms around position change SD = 66.4 ms, 95% CI (57.8/75.1) or the opposite SD = 60.6 ms, 95% CI (35.3/77.7). Consequently, these results indicate that time processing for detecting position change was shorter than time processing for detecting color change, leading to the perception of asynchrony when both changes occurred simultaneously. The direction of this effect was very consistent across participants (when considering individual PSS differences between-participants, standard deviation was 25.07 ms). 

Figure 2

 

Proportion of position change perceived as the first change in the temporal order judgment task as a function of SOA, when comparing position and color change in the motor (plain line), tactile (dashed grey line), and perceptual (dashed black line) conditions. For the sake of simplifying graphical presentation, we normalized the responses and used position change as a time-marker (zero value).

Temporal order judgment of color and position change in the motor condition

When testing the effect of performing a motor action on the temporal order judgment of color and position changes, the mean estimated PSS was −3.3 ms, 95% CI (−8.9/2.6), and SD = 69.6 ms, 95% CI (60.5/78.3). Therefore, position change had to occur about 3 ms after color change for perceiving synchrony. A 95% confidence interval indicated that this value was not different from zero, suggesting that the asynchrony observed in the perceptual condition virtually disappeared in the motor condition. As a consequence, the PSS in the motor condition was significantly different from the PSS in the perceptual condition (as indicated by the absence of overlap between the 95% confidence intervals in perceptual and motor conditions, see Figure 2). The direction of this effect was very consistent across participants (when considering individual PSS differences between-participants, standard deviation was 19.7 ms). 

Considering separately the color fixed block and position fixed block, we found that the PSS was, respectively, 3 ms, 95% CI (−5.4/10.8) with SD = 66.5 ms, 95% CI (58.2/75.5) and −9.6 ms, 95% CI (−18.3/−1.1) with SD = 71.8 ms, 95% CI (57.2/85.9). These PSSs were significantly different from the average PSS found in the perceptual condition (as indicated by the 95% confidence interval in the perceptual condition). Considering the −100 ms and +100 ms blocks separately, we found that the PSS was, respectively, −8.9 ms, 95% CI (−17.4/−0.6) with SD = 77.4 ms, 95% CI (57.9/94.7) and 2.3 ms, 95% CI (−6.1/10.8) with SD = 62.8 ms, 95% CI (55.4/69.8). These PSSs were also significantly different from the average PSS found in the perceptual condition (as indicated by the 95% CI [-46/−30.5] in the perceptual condition). Interestingly, the SD of the distributions of the data in the perceptual and motor conditions were not statistically different in the perceptual and motor tasks as shown by the lack of overlap between the two 95% confidence intervals. This indicates that the motor action did not make the task more difficult and is consistent with the fact that participants judged the same sensory information in the motor condition as in the perceptual one. 

Temporal order judgment of color and position change in the tactile condition

When performing the TOJ task between changes of color and position visual attributes in the presence of tactile stimulation, participants' performances revealed a mean PSS of −36 ms, 95% CI (−39.9/−31.9); with SD = 68.9 ms, 95% CI (63.4/73.9). Therefore, color change had to occur about 36 ms earlier than position change in order for the participant to perceive the two changes simultaneously. Confidence intervals indicated that this value was significantly different from 0 ms but not significantly different from the PSS found in the perceptual condition, 37.9 ms, 95% CI (−46/−30.5), see Figure 2. The direction of this effect was very consistent across participants (when considering individual PSS, between-participants standard deviation was 13 ms). 

When considering separately color fixed block and position fixed block, we found a PSS of, respectively, −36.1 ms, 95% CI (−41.6/−30.5); with SD = 63.4 ms, 95% CI (54.4/71.1) and −36 ms, 95% CI (−42.1/−30.3); with SD = 71.8 ms, 95% CI (66.1/81.6). These PSSs were not significantly different from the average PSS found in the perceptual condition, as indicated by the confidence intervals, 95% CI (−46/−30.5). Considering the −100 ms and +100 ms blocks separately, we found that the PSS was, respectively, −32.9 ms, 95% CI (−38.6/−27.3); with SD = 71 ms, 95% CI (62.5/78.7) and −39.1 ms, 95% CI (−44.5/−33.2); with SD = 66.5 ms, 95% CI (58.5/73.7). These PSSs were also not significantly different from the average PSS found in the perceptual condition as indicated by the confidence intervals, 95% CI (46/−30.5). Interestingly, the standard deviations of the distributions in the perceptual, motor, and tactile conditions were not statistically different, as revealed by the 95% confidence interval. This indicates that the tactile stimulation did not make the task less or more difficult and seems to confirm the fact that participants have judged the visual information similarly in the tactile condition as in the perceptual and motor conditions. The presence of the tactile stimulation did not reduce the asynchrony observed in the perceptual condition, suggesting that it weakly contributed to the reduction of asynchrony in the motor task (−3.3 ms). 

Discussion

The aim of the present study was to evaluate the effect of performing a manual reaching task on the temporal order judgments of position and color changes of a visual target. We tested in particular whether the perceived relative timing of sensory events was modified when timely locked to a self-generated action compared to a baseline perceptual condition or a condition where tactile stimulation was provided around the time visual attributes changed but in the absence of any intentional motor action. When perceptually judging the relative change between position and color attributes of a visual object, temporal order judgments showed that color change must precede position change by 37.9 ms in order for the participant to perceive a simultaneous change of the two attributes. This finding is in line with previous studies that have reported asynchrony in TOJ tasks when estimating variations of color and position attributes of a visual stimulus (Pisella, Arzi, & Rossetti, 1998; Tanaka & Shimojo, 1996), suggesting different integration times of these attributes to form a single percept. These observed asynchronies could be accounted for by the differential delays characterizing the visual neural pathways associated with color and position processing (Nowak & Bullier, 1997). Indeed, color is thought to be processed within the ventral stream of the visual system, which received the visual inputs with a delay of at least 40 ms compared to the dorsal stream of the visual system responsible for processing position and movement changes (Nowak & Bullier, 1997; Tanné et al., 1995). Contrary to our findings in which color lags change of position, color has been reported to lead motion in the context of the CMA (e.g., Nishida & Johnston, 2002). This clear discrepancy is not easy to reconcile by invoking ventral and dorsal differences as both position change and movement direction change are arguably dorsally processed. Neither can the time marker model proposed in Nishida and Johnston (2002) easily accommodate our results as both color and position changes would correspond to first-order temporal changes (i.e., the change is ascertained by comparing two samples). However the CMA arises when direction has to be paired with color. In this context, direction needs integration time and realizing the direction of a movement might involve different processing than ascertaining a change of position. For example, in hitting movements, direction has been found to be perceived separately than motion (Brouwer, Middelburg, Smeets, & Brenner 2003). Admittedly, we used a change of position instead of smooth motion. Nevertheless, our change of position could have activated motion areas just as if it was apparent motion (Maffei, Macaluso, Indovina, Orban, & Lacquaniti, 2010). Furthermore, it is worth noting that previous studies have reported longer asynchrony in the perception of color and position changes. Pisella et al. (1998), for instance, found that color information was processed about 80 ms slower than position information. However, this result was obtained in a visuo-manual adaptation task where participants were requested to prevent from responding when either the color or the position of the target changed at movement onset. The task thus evaluated visuo-motor delays as well as timing of inhibition processes which may explain the longer asynchrony that we found in our task, which focused essentially on perceptual processes. 

Importantly, the pattern of results in the TOJ task was different when the participants estimated the temporal order of color and position changes while performing a voluntary motor action. In the action condition, the asynchrony reported above nearly vanished and simultaneous changes were perceived when the two changes occurred while separated by only 3.3 ms, a value not statistically different from 0 ms. Then, self-generated signals, associated with the production of a voluntary motor action, seem to have contributed to the performance in the TOJ task with the effect of reducing the relative timing between the perception of position and color changes of visual objects when occurring near a movement end-point. This effect cannot be accounted for by tactile feedback experienced at the movement end-point, since asynchrony between perceived position and color change was 36 ms in the tactile condition, a value close to the one observed in the perceptual condition (39.7 ms). Consequently, tactile information did not seem to play a crucial role in the perception of relative change of position and color attributes of a visual target when executing an intentional motor action. The reduction of asynchrony between perceived color change and perceived position change in the motor condition was then probably more dependent on a motor-related signals and predictive mechanisms rather than sensory-related signals resulting from the production of intentional action. This finding corroborates the previous observations that performing a voluntary action influences the temporal perception of action consequences in the environment (Haggard et al., 2002; Stetson et al., 2006). As a whole, these experiments have suggested that the perception of sensory events (e.g., a sound) occurs earlier in time (about 50 ms) when its occurrence is preceded by a manual reaching action (Haggard et al., 2002). Furthermore, the perceptual temporal discrimination threshold when judging the occurrence of two stimuli increased when associated with an intentional action (Wenke & Haggard, 2009). This natural variation of the temporal relationship between action parameters and action-related sensory effects was interpreted within the framework of intentional binding, suggesting that delays in sensory and motor signals processing are reduced (Haggard et al., 2002) or recalibrated (Stetson et al., 2006). However the reduction of delays due to recalibration cannot explain the reported pattern. We could initially consider two possibilities: Either the time processing for color was shortened for the action or lengthened for position changes. The first one, although a possibility, is more difficult to interpret. However, position is much more relevant for goal-directed actions. This could imply that a change of position requires the visuo-motor system to further update or remap the new position, while this would not be mandatory for a color change. 

In the following experiment, we tested whether asynchrony while performing an action also reduces when performing a TOJ task involving cross-modal stimuli, i.e., when judging the relative timing between a visual attribute change and tactile stimulation. In the context of intentional action, the relation between visual and tactile information is thought to be less predictable than the relation between various visual attributes. Indeed, tactile information in intentional action does not always vary congruently with visual information depending on object (soft or rigid) or action (touching, grasping, pulling, etc.) properties. In agreement with this, previous studies have shown that the effect of action on estimation of time intervals (Morrone, Ross, & Burr, 2005) or sensory recalibration in temporal order judgment tasks (Fujisaki, Shimojo, Kashino, & Nishida, 2004; Stetson et al., 2006; Vroomen, Keetels, de Gelder, & Bertelson, 2004) is weaker when involving cross-modal instead of unimodal sensory information. Furthermore, voluntary action produces an attenuation of body-related sensory information (Blakemore et al., 1999) which renders the processing of tactile information less accurate (Shergill, Bays, Frith, & Wolpert, 2003). We then tested in the following study whether asynchrony in the temporal order judgment task also reduced while performing an action (motor condition) and comparing cross-modal, i.e., visual and tactile, sensory information. In the tactile condition, judgments of relative timing between a visual attribute change and tactile stimulation were tested also in a passive condition, i.e., with no motor action. In this condition, tactile stimulations were provided by a loudspeaker as in Experiment 1. 

Experiment 2: Effect of motor action on perceived relative timing between position or color and tactile information

In Experiment 2, we tested whether the reduction of asynchrony in the TOJ of perceived object-related attributes while performing an action is still observed when the TOJ task concerns object-related (color or position) versus object-unrelated (tactile) information. The task was to evaluate whether the color (or position) change or the finger-target contact occurred first and was performed under active (motor) or passive (tactile) conditions. In the passive tactile condition, the task was to evaluate which of the color (or position) change or the finger tactile concurrent stimulation occurred first, while performing no motor action. 

Procedure and data analysis

The task for the participants was to perform a temporal order judgment task comparing color or position attributes change of a visual target with finger-target contact at the movement end-point (motor condition) or with finger tactile concurrent stimulation (tactile condition) while remaining still. In the motor condition, as in Experiment 1, participants positioned their right index finger at the starting location on the screen and an auditory tone was provided between 500 and 1000 ms after the right forefinger reached the starting location indicating the beginning of the trial. Following the tone the participants had to manually reach the target in about 700 ms and one of the target attributes (color or position) changed within a temporal window of ±200 ms relative to when finger-target contact occurred (see Figure 3). In the tactile condition, the participant positioned their right forefinger at the center of the loudspeaker positioned on the computer screen close to the target location. In each trial, the beginning of the trial corresponded to the appearance of the target on the screen and the tactile stimulation was provided following a delay of 700 ms. One test target attribute (color or position) changed within a temporal window of ±200 ms relative to the tactile stimulation (referent attribute), which was delivered on the forefinger (see Figure 3). In the motor condition, the task for the participants was to indicate whether the visual change (position or color) occurred first, or whether this was the contact of the forefinger with the target that occurred first. Negative SOA were obtained as in Experiment 1 from the spatial location of the hand during movement execution. Spatial coordinates on average corresponding to −200 ms, −150 ms, −100 ms, and −50 ms were, respectively, 60 mm, 45 mm, 30 mm, and 15 mm from the target location along the vertical z axis of the computer screen. Assuming that movements were executed in about 700 ms, expected average negative SOA between change of test target attribute and finger contact was −100 ms (SOA: −200, −150, −100, −50, 0 ms). Postexperiment analysis revealed that averaged SOA before finger contact was −102.4 (SD: 20.7 ms), this value being not statistically different from the expected value, t(9) = 0.36, p = 0.73. 

Figure 3

 

Time sequence of test attribute (color or position) change according to finger-target contact with the screen or tactile stimulation (reference attribute). The change of test attribute could occur within the time window of ±200 ms according to the finger-target contact with the screen or tactile stimulation.

In the tactile condition, the task for the participants was to indicate whether the visual change (position or color) occurred first or whether this was the tactile stimulation that occurred first. Responses were provided by using one of the two buttons of the response box (two-AFC paradigm). In each of these conditions presented in block sessions, the participants performed 90 trials, SOAs (−200; −150; −100; −50; 0; 50; 100; 150; 200 ms) × 10 Trials. 

In the motor condition (color or position), the set of data for each participant was the distribution of the proportion of trials in which the finger-target contact was perceived first (used as reference for normalized responses). In the tactile (color or position) conditions, the set of data for each participant was the distribution of the proportion of trials in which the finger tactile stimulation was perceived first (used as reference for normalized responses). A positive PSS value corresponded to a processing of finger-target contact or tactile stimulation lagging the processing of the test attribute change (position or color) and a negative PSS value indicated that the processing of finger-target contact was faster than the processing of the test attribute change (position or color). Data analysis was performed as in Experiment 1. 

Result

Temporal order judgment of color or position change and finger-target contact in the motor condition

In the motor condition, when analyzing temporal order judgments of color change and finger-target contact, we found a PSS of 2.6 ms, 95% CI (−10.3/15.5); with SD = 129.1 ms, 95% CI (114.7/145.3). Therefore, color must change slightly after finger-target contact in order for the participants to consider the two events as occurring synchronously. Similarly, position must change 36.1 ms, 95% CI (22.9/49.2); SD = 136 ms, 95% CI (118.8/156.1) after finger-target contact in order for the participants to consider the two events as occurring synchronously (see Figure 4a). A 95% confidence interval showed that the PSS was shorter when estimating color change than when estimating position change relative to finger-target contact (the difference being 33.5 ms). By contrast, a 95% confidence interval suggested that the SD of the psychometric functions did not differ, though they were shallower than in the perceptual, tactile, and motor conditions of Experiment 1, respectively, 95% CI (48.3/75.6), 95% CI (63.4/73.9), and 95% CI (60.5/78.3). 

Figure 4

 

(A) Proportion of finger-target contact perceived as the first change in the temporal order judgment task as a function of SOA, when comparing finger-target contact to position (plain line) or color (dashed line) visual change. For the sake of graphical presentation, we normalized the responses and used finger-target contact as a reference (zero value). (B) Proportion of tactile stimulation perceived as the first change in the temporal order judgment task as a function of the SOA, when comparing tactile stimulation to position (plain line) or color (dashed line) visual change. For the sake of simplifying graphical presentation, we normalized the responses and used tactile stimulation as a time-marker (zero value).

Temporal order judgment of color or position change and finger-tactile stimulation in the tactile condition

In the tactile condition, when analyzing temporal order judgments of color change and finger-tactile stimulation, we found a PSS of −36.2 ms, 95% CI (−48.3/−23.8); with SD = 124.1 ms, 95% CI (109.8/139.8). Therefore, color must change significantly before finger-tactile stimulation in order for the participants to consider the two events as occurring synchronously. Position must change −8.4 ms, 95% CI (−21/3.6); with SD = 136 ms, 95% CI (110.5/138.8) before finger-tactile stimulation in order for the participants to consider the two events as occurring synchronously (see Figure 4b). A 95% confidence interval showed that the PSS was shorter when estimating position change than when estimating color change relative to finger-tactile stimulation (the difference being 27.8 ms). By contrast, a 95% confidence interval suggested that the SDs of the psychometric functions did not differ, though they were shallower than in the perceptual, tactile, and motor condition of Experiment 1, respectively, 95% CI (48.3/75.6), 95% CI (63.4/73.9), and 95% CI (60.5/78.3). 

The fact that in this experiment the SDs of the psychophysics functions were shallower in the motor and tactile conditions than in the perceptual, motor, and tactile conditions of Experiment 1 was indeed expected if participants were making intersensory judgements, as required by the task, when compared to unimodal TOJ judgements (see Miyazaki, Yamamoto, Uchida, & Kitazawa, 2006). Therefore, this is consistent with participants comparing cross-modal information. 

Discussion

In this study, we tested whether asynchrony in a temporal order judgment task, as observed in the perceptual condition of Experiment 1, also reduced while performing an action in the context of a cross-modal TOJ task, i.e., when judging temporal order of visual (color or position) and tactile stimuli. We found that the asynchrony between perceived color change and tactile information (2.6 ms) was different from the asynchrony between perceived position change and tactile information (36.1 ms). The difference between these two values (33.5 ms) was close to the one observed in the perceptual task of Experiment 1 (39.7 ms). Thus, despite the fact that visual and tactile stimuli were associated with the execution of an intentional action, no reduction in relative timing processing of sensory events was observed. The absence of any reduction in asynchrony due to action in cross-modal TOJ was confirmed by the results obtained in the tactile condition of Experiment 2, where TOJ of visual attribute change (color or position) and tactile stimuli was performed. In this condition, asynchrony between perceived color change and tactile information (−36.2 ms) was different from the asynchrony between perceived position change and tactile information (−8.4 ms). The difference between these two values (27.8 ms) was again close to the one observed in the perceptual task of Experiment 1 (39.7 ms) and also to the one observed in the motor condition of Experiment 2 (33.5 ms). Then, although the sensory changes due to intentional action can be anticipated in the action condition, asynchrony between visual and tactile information was not less than in the perceptual condition (Experiment 1), suggesting that executing an intentional action did not modify the perception of a temporal relationship of cross-modal sensory signals. However, the motor and tactile conditions revealed that tactile information was processed about 40 ms slower when associated with an intentional action, which made the time processing for tactile information closer to the time required for processing color information, though this did not modify the relative time of processing color and position attributes. The increase of time processing for tactile information might be the consequence of the sensory attenuation effects associated with intentional action (Blakemore et al., 1999), which would render tactile information processing more difficult resulting in an increase in time processing 

One explanation for the lack of effect of voluntary action on cross-modal sensory perception might be that tactile information plays little role as a time marker in processing visual consequences of acting. Another interpretation could be that the highly variable tactile information depending on the execution context makes the perception of the time correspondence between visual and tactile information difficult in natural conditions and thus the elaboration of predicting mechanisms. In agreement with this, the standard deviations of the data used to compute the cumulative Gaussian functions were larger in the cross-modal condition compared to the unimodal condition, making the slopes of the fitted function shallower and suggesting then that the former condition was perceptually more difficult. 

General discussion

To summarize, when considered together the present data showed that intentional action can modify how we perceive external events that occur temporally in the vicinity of action end-point. Indeed, when passively observing a visual stimulus, change of its position was perceived 37.9 ms earlier than change of its color. This finding is in line with previous studies that have reported asynchrony in temporal order judgment tasks of color and position attributes of a visual target (Pisella, Arzi, & Rossetti, 1998; Tanaka & Shimojo, 1996), suggesting different processing times of these different attributes to form a single percept. At the same time, it challenges recent models that attempt to account for the CMA (e.g., Nishida & Johnston, 2002) by invoking the impossibility of pairing first (color change) and second-order (e.g., direction) changes which would lead to perceive changes of direction as lagging the changes of color. Unlike what is usually the case in this illusion, we have found that color lags the change of position, with both changes arguably being first-order changes. When perceiving these very same visual changes while performing an intentional action, the latter had the effect of broadly reducing this asynchrony. Indeed, when changing the color and position of a visual target at the vicinity of a reaching movement end-point, the relative timing to obtain the perception of simultaneous change was close to 0 ms (Experiment 1, motor condition). This effect was not due to the presence of a concurrent tactile stimulation (Experiment 1, tactile condition), since the mere presence of a tactile stimulation at the time the temporal order judgment of color and position were performed, in the absence of intentional action, was not sufficient to reduce the asynchrony observed in the perceptual condition. The positive effect of performing a motor action on perceptual judgments confirms previous studies that have revealed an effect of intentional action on temporal perception of motor and sensory events, so that timing errors between sensory information are reduced (López-Moliner & Linares, 2006) or a perceptual event is shifted earlier in time towards the action that is supposed to have caused it (Haggard, 2005; Haggard et al., 2002). In the same vein, it has been demonstrated that temporal sensory discrimination threshold increases when associated with an intentional action (Wenke & Haggard, 2009). The lack of effect of tactile information on temporal order judgment of visual events is also compatible with the previous observation that imposing a movement kinematically identical to a self-initiated action did not induce intentional binding (Engbert, Wohlschläger, & Haggard, 2008; Haggard & Clark, 2003; Haggard, Clark, & Kalogeras, 2002; Wohlschläger, Engbert, & Haggard, 2003). Thus, the effect of action on temporal order judgment seems intrinsically dependent on voluntary action and not effector kinematics or somatosensory feedback (Cravo, Claessens, & Baldo, 2011). 

One possible interpretation for these effects of action on sensorimotor processing is that action modifies how attentional processes are allocated to the target (Roskies, 1999; Whitney, 2009) with a direct consequence on how sensory information is processed and integrated (Holcombe & Cavanagh, 2008; Posner & Dehaene, 1994; Treisman, 1999; Treisman & Gelade, 1980; Wolfe & Cave, 1999). This is indeed what suggests the premotor theory of visual perception, which considers that (eye or manual) action planning towards a visual target and orientation of attention towards that target represent a unique process improving visual perception on target's properties (Deubel & Schneider, 1996; Rizzolatti & Craighero, 1998). In line with this interpretation, Holcombe and Cavanagh (2008) found that perceived asynchrony between change in color and change in motion direction of a visual dot reduced when changes were synchronized with the presentation of an external cue. One may thus consider the possibility that that the finger approaching the target in the motor condition played as an exogenous cue affecting the perception of the relative timing between color and position changes. However, assuming that the position of the finger played as an exogenous cue, one would have also expected an effect of the finger position on the perception of the relative timing between visual attribute and tactile information, or even in the passive condition, which was not observed. Furthermore, the fact that the standard deviation of the data estimated from the cumulative Gaussian in the perceptual, tactile and motor conditions (Experiment 1) were equivalent argue against a pure attentional effect, since this indicates that performing an action toward a visual target did not modify the sensitivity of the visual system when processing the target's properties. Finally, studies on intentional binding have suggested that the change in time processing of a sensory event when associated with intentional action does not depend exclusively on attentional mechanisms, since when participants were prevented from focusing on a particular event, they show a very strong binding effect even for this event, which did not receive specific attention (Haggard & Cole, 2007). 

An alternative explanation can be proposed, which considers that the effect of voluntary action on temporal order judgment of sensory events emerged from a specific neural mechanism producing intentional binding of actions and their effects in conscious awareness (Aymoz & Viviani, 2004; Haggard et al., 2002). This mechanism can be related to the fact that “events surrounding voluntary action are bound by a specific cognitive function of the central nervous system” (Haggard et al., 2002) or that “the temporal interval between action and consequences is perceived as shorter because the consequence of the action is anticipated and therefore processed faster” (Baldo et al., 2007). Prediction mechanisms are thus thought to constrain visual perception in relation to action by taking into account the internal delays associated with the processing of different sensory information and timely synchronized according to the prediction made by the motor system (Stetson et al., 2006). In agreement with this, previous studies have shown that object-attributes binding is intrinsically related to voluntary action and occurs mainly when a temporal contiguity exists between an action and its effect (Cravo, Claessens, & Baldo, 2011; Engbert, Wohlschläger, & Haggard, 2008; Haggard et al., 2002; Wohlschläger, Engbert, & Haggard, 2003). In these studies, the authors revealed that voluntary action produces a contraction of time by which the perception of an action initiation and its consequences are temporally attracted towards each other (intentional binding). Buehner and Humphreys (2009) have further suggested that a causality relationship between action and sensory consequences is necessary for the intentional binding effect to occur. In line with this, Desantis, Roussel, and Waszak (2011) recently showed that having the belief of being responsible for sensory effect as a consequence of action increases the degree of intentional binding. In the same vein, Engbert, Wohlschläger, and Haggard (2008) showed that temporal binding not only relates movements with the effects they actually produce but also with effects they are expected to produce. 

Though the present study focused on perceived relative timing between sensory changes, the intentional binding framework can quite effectively account for our data since we found that asynchrony between color and position changes when presented simultaneously reduced substantially when the visual events occur near the end of a reaching action. Whether action modifies (or recalibrates) the processing delay associated with each sensory attribute or the time window within which attributes changes are considered as simultaneous, or even shifts the whole sensory processing in time, are alternative interpretations for the effect of action in perceived relative timing of visual attributes, which remain to be more carefully investigated in the future (see however Wenke & Haggard, 2009). Whatever the nature of the effect of interacting physically with the target, the present data revealed, for the first time, that endogenous signals associated with the planning and execution of an intentional motor action and expected sensory consequences improve the perception of visual objects attributes changes. 

The facilitating effect of producing an intentional action seems weakened, however, when considering cross-modal information. Indeed, asynchrony between color and finger-target contact and between position and finger-target contact showed a difference of 33.5 ms, a value close to the asynchrony observed in the perceptual condition (37.9 ms). The same asynchrony was observed when performing the TOJ task in the absence of intentional action (27.7 ms). Though this remains to be properly investigated, a possible interpretation for this lack of asynchrony reduction in the cross-modal condition could be the absence of consistent time correspondence between visual and tactile sensory variation during intentional action, which might prevent the elaboration of reliable predicting mechanisms. This would explain why the asynchrony was reduced by an intentional action when comparing two pieces of object-related information instead of one piece of object-related and one of object-unrelated information. An alternative interpretation could be related to the sensory attenuation effect that has been reported for tactile information processing in the context of intentional action. Blakemore et al. (1999) reported indeed that self-produced tactile stimulation is generally perceived with a weaker intensity than when produced by an external cause. This attenuation effect is thought to facilitate the distinction between self- and externally generated sensory signals leading to the sense of agency (Bays, Flanagan, & Wolpert, 2005; Bays, Wolpert, & Flanagan, 2006). In the present study, a sensory tactile attenuation phenomenon may thus explain the lack of action effect on perceptual asynchrony between tactile and visual information. 

In conclusion, the present study provides new insight into the effect of performing a voluntary action onto the perception of action consequences. In particular, we found that temporal constraints associated with the integration of sensory information of visual attributes are reduced when performing a reaching action compared to a simple perceptual task. Our results suggest then that the facilitation effect gained from performing an actual motor action on visual attributes processing seems to rely on the predictions carried by the internal signals associated with self-generated action and not cross-modal sensory calibration processes. It remains to test whether these effects can be generalized to other object-related attributes. Furthermore, testing temporal order judgments of visual attributes changes but with various time delays between action ending and sensory effects would represent a relevant paradigm for probing agency. 

Acknowledgments

The first (XC) and third (YC) authors were supported by the University Lille-Nord de France. The second author (JLM) was supported by Grant PSI2010-15867 from the Ministry of Science and Innovation of the Spanish government and an ICREA Academia Distinguished Professorship award. 

Corresponding author: Yann Coello. 

Email: [e-mail]. 

Address: Research Unit on Cognitive and Affective Sciences, Université Charles De Gaulle, Gaulle-Lille3, 59653 Villeneuve d'Ascq, France. 

Commercial relationships: none.