Open Access
Article  |   April 2023
Force illusion induced by visual illusion: Illusory curve in cursor path is interpreted as unintended force
Author Affiliations
Journal of Vision April 2023, Vol.23, 5. doi:https://doi.org/10.1167/jov.23.4.5
  • Views
  • PDF
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Takumi Yokosaka, Yusuke Ujitoko, Takahiro Kawabe; Force illusion induced by visual illusion: Illusory curve in cursor path is interpreted as unintended force. Journal of Vision 2023;23(4):5. https://doi.org/10.1167/jov.23.4.5.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Discrepancies between expected and actual visual outcomes of motor action can produce an illusory sensation of unintended force. In the present study, we addressed whether the force illusion could be induced even when the discrepancy was brought about by the illusory appearance of the actual outcome. Specifically, the apparent path of a cursor controlled by the participants was modulated by the direction of noise motion presented inside the cursor. We showed that a greater noise motion inside the cursor caused a greater apparent curve of the cursor path and, also, higher rating scores for an unintended force. We also found that the unintended force was influenced strongly by the visibility of the cursor, suggesting that the apparent curve of the cursor path was a more important factor in generating the unintended force than the noise motion itself inside the cursor. Our results suggest that the illusory force, which is mediated by cross-modal mechanisms susceptible to visual illusion, can be exploited in extended reality systems as a novel technique for giving users a sensation of force.

Introduction
When an agent executes a motor action, the brain of the agent tries to expect the outcome of that motor action, to compare the expected and actual outcomes, and to associate the motor action with its outcome when the actual outcome is judged to be consistent with the expected one (Blakemore, Frith, & Wolpert, 1999; Haggard, 2017). When the spatiotemporal discrepancy between the expected and actual outcomes is large, the brain perceptually infers the reason for the discrepancy. One representative interpretation for the discrepancy inferred by the brain in such a situation is that the outcome is not caused by the motor action of the agent. That is, the large discrepancy between the expected and actual outcomes causes a decrease in the sense of agency (Asai & Tanno, 2007; Farrer, Valentin, & Hupé, 2013; Kawabe, 2013; Kawabe, Roseboom, & Nishida, 2013; Sato & Yasuda, 2005; Shanks, Pearson, & Dickinson, 1989; Wen, Yamashita, & Asama, 2015b, 2015a). Another interpretation of this discrepancy, although related to that just described, is that the outcome is altered by external forces or by the physical properties of an external object. Taking advantage of this sort of perceptual interpretation by the brain, previous studies have proposed some techniques in extended reality systems for giving users an illusory sensation of force and weight on the basis of the discrepancy between the expected and actual outcomes of motor action. For example, regarding the sensation of force, it has been reported that participants felt illusory force when a delay was inserted while they moved a cursor (Smith, 1972; Takamuku & Gomi, 2015). A method has also been proposed to make participants feel an illusory force by magnifying the error between a target path and the user’s cursor path, which likely increased a spatiotemporal discrepancy between the expected and actual outcomes of the participant’s motor action (Nomoto, Ban, Narumi, Tanikawa, & Hirose, 2016). Many previous studies in the field of pseudo-haptics have also reported that a virtual weight can be produced by a slowing down (Ban & Ujitoko, 2018; Dominjon, Lecuyer, Burkhardt, Richard, & Richir, 2005; Gomez Jauregui et al., 2014; Rietzler, Geiselhart, Gugenheimer, & Rukzio, 2018; Samad, Gatti, Hermes, Benko, & Parise, 2019; Taima, Ban, Narumi, Tanikawa, & Hirose, 2014) or a delay insertion (Honda, Hagura, Yoshioka, & Imamizu, 2013; Osumi et al., 2018) during the user’s manipulation of an object. Lécuyer, Burkhardt, & Etienne (2004) assume that a horizontal reaction force can be visually induced by accelerating or decelerating the cursor and proposed a method for modulating bumpiness and holes based on the idea (Lécuyer et al. 2004). van Mensvoort et al. (2002) also assumed that modulating the mouse cursor position makes the mouse feel as if it is being pulled or pushed in that direction, and proposed an bumps/holes and weight modulation method based on the idea (Mensvoort, Vos, Hermes, & Liere, 2010). The illusory sensation of force and weight might occur because the brain directly interprets the discrepancy between the expected and actual outcomes as stemming from an external force or the weight of an external object (Honda et al., 2013), or because the brain chooses the appropriate internal physical model which is consistent with the discrepancy between the expected and actual outcomes of the agent’s motor action (Takamuku & Gomi, 2015). 
It is an interesting question whether the association of the expected and actual outcomes of motor action is altered when the appearance of the actual outcome is modulated by a visual illusion. Some previous studies have tried to answer this question and their answers were in the affirmative. If the actual outcomes, whose appearance is modulated by visual illusion, influence the association, the motor action will be adjusted in accordance with the illusory appearance of the actual outcome. Consistent with this prediction, it has been shown that a saccadic landing point was displaced in the direction of the illusory shift of a saccadic target as the actual outcome of saccadic eye movements (Carey, 2001; Kosovicheva, Wolfe, & Whitney, 2014; Lisi & Cavanagh, 2015). Moreover, it has also been shown that not only saccadic eye movements, but also pointing actions, can be influenced by the illusory appearance of a pointing target (which can also be considered as an actual outcome) (Ueda, Abekawa, Ito, & Gomi, 2019). The results of the previous studies indicate that the brain tries to associate motor action with its outcome even when the appearance of that outcome is modulated by visual illusions. 
Although the computation for the association of the expected and actual outcomes has been found to be influenced by the illusory appearance of the actual outcome of motor action, it has not been examined in detail whether the discrepancy between expected and actual outcomes could cause an illusory sensation of external force when that discrepancy is brought about by the illusory appearance of the actual outcome. To examine this, we conducted a series of psychophysical experiments to examine whether illusory force could be perceived when the apparent path of a mouse cursor was modulated by a visual illusion. We adopted stimuli inspired by motion-based position illusion, sometimes called the motion-induced position shift (MIPS) or the curveball illusion (De Valois & De Valois, 1991; Durant & Zanker, 2009; Lisi & Cavanagh, 2015; Ramachandran & Anstis, 1990; Shapiro, Lu, Huang, Knight, & Ennis, 2010; Tse & Hsieh, 2006; Whitney, 2002). In these types of visual illusions, the perceived position of a two-dimensional patch is shifted in the direction of motion that is presented within the patch. In the present study, we presented drifting random noise within a patch as a cursor that our experimental participants manipulated. In our preliminary observations, consistent with the curve ball illusion (Shapiro et al., 2010), we found that the cursor path was distorted perceptually in the direction of the noise motion presented inside the cursor while the participants moved the cursor along a straight path (Supplementary Movie 1). We hypothesized that the participants would feel an unintended force upon the cursor if the actual cursor path were perceived to differ from the expected cursor path due to a visual illusion. Note that “unintended force” here refers to a force that the participants experienced as if an external force was being applied to the cursor under their control; it does not imply that the participants themselves unintentionally produced a force. In Experiment 1, we examined whether the distortion of the cursor path was actually perceived to curve in the direction of motion presented inside the cursor. In Experiment 2, we examined whether and how strongly the force illusion occurred with the stimuli used in Experiment 1. In Experiment 3, we tried to disentangle the contribution to the illusory sensation of force of the illusory curve of the cursor path from the contribution of the motion inside the cursor itself. At the end of this article, we discuss the possibilities and limitations of the force illusion that affect its use as a technique for giving users force sensations in the extended reality systems. 
Experiment 1
The purpose of the first experiment was two-fold. First, we wanted to check whether drifting noise within a circular patch used as a cursor could create an illusion in which the cursor path would seem to be curved in the direction of the drifting noise. Second, we wanted to test how the amplitude of the noise drift could modulate the magnitude of the illusory curve of the cursor path. As shown in Figure 1, participants were asked to move the cursor from a starting point on the left (or right) side of the screen to a goal point on opposite side of it. During the participants’ manipulation of the cursor, the noise within the cursor drifted upward or downward at a certain amplitude. We expected that the noise drift would produce the illusory curve of the cursor path and its magnitude would increase with the drift amplitude of the noise within the patch. To measure the magnitude of the illusory curve of the cursor path, the actual path of the cursor was modulated on the basis of the staircase method so that it counteracted the illusory curve. The amplitude of the modulation of the actual cursor path when the participants’ responses converged was considered the magnitude of the illusory curve of the cursor path. 
Figure 1.
 
Experimental setup. The white arrows represent the motion directions of the cursor and the reference object, and were not actually displayed as experimental stimuli.
Figure 1.
 
Experimental setup. The white arrows represent the motion directions of the cursor and the reference object, and were not actually displayed as experimental stimuli.
Participants
We adopted a between-subjects design in which nine groups of participants were tested with different noise drift conditions each of which was related to the different levels of the amplitude of the noise drift as follows: −7.00, −5.25, −3.50, −1.75, 0.00, 1.75, 3.50, 5.25, or 7.00 mm. Using a statistical calculator, Morepower 6.0 (Campbell & Thompson, 2012), we calculated the minimum sample size for a between-subjects design under following three conditions: a medium-effect size (Cohen’s F = 0.25), a power of 80%, and an alpha of 5%. The minimum sample size that satisfies these conditions was 252 (i.e., 28 participants per condition). We recruited 270 participants in total for 30 participants per condition, which can be divided by 6 participant attributes ([female and male] × [20s, 30s, and 40s]). Each group consisted of 30 participants (15 females and 15 males). The mean ± standard deviations of their ages for the nine amplitude conditions were 36.33 ± 8.62, 36.10 ± 8.19, 35.23 ± 8.30, 35.93 ± 8.35, 35.27 ± 7.70, 34.50 ± 9.50, 35.47 ± 8.44, 35.40 ± 8.57, and 35.53 ± 8.48, respectively. A Japanese crowdsourcing research company recruited participants online and they were paid for their participation. The participants were unaware of the specific purpose of the experiment. Ethical approval for this study was obtained from the ethics committee at Nippon Telegraph and Telephone Corporation (Approval number: R02-009 by NTT Communication Science Laboratories Ethics Committee). The experiments were conducted according to principles that have their origin in the Helsinki Declaration. Written informed consent was obtained from all observers in this study. 
Apparatus
The experiments in this study were carried out online using the participants’ own personal computers (PC) and mice. Because our experimental script could only be run on a PC, smartphones or tablet PCs, which have neither a physical keyboard nor a mouse, could not be used in this experiment. 
Stimuli
As shown in Figure 1, the stimuli consisted of a background square filled with gray-scale random noise (as background noise), a circular patch (as a cursor) within which noise drifted in a certain direction, two squares to represent start and goal points, a fixation point, and a black square as a reference object. We generated and presented these stimuli using p5js (McCarthy, Reas, & Fry, 2015, a JavaScript library; https://p5js.org/). To control the spatial dimension of these stimuli in millimeters, we calculated the size of one pixel on each participant’s PC by asking them to adjust the size of a rectangle on the monitor so that the rectangle had the same size as their credit card (or a card with an equal size to a credit card) before the experiment began. Hereafter, we describe the spatial dimension of visual stimuli in millimeters because the pixel size of the stimuli varied slightly depending on the participant’s monitor. 
The background noise was centered within the monitor and its size was 35 × 35  mm. The background noise was randomly generated before each trial, and the intensity of each pixel was drawn from a uniform distribution having the range [255/4, 255 × 3/4]. In preliminary testing, we noticed that the noise was spatially blurred depending on the monitor and/or PC that the participants used. To counteract potential effects of undesired blurring on our experiments, an isotropic Gaussian blur filter with a semantic differential equal to 1 pixel was applied to the noise. We believe that the filtering eliminated the undesired blurring effect because the magnitude of the undesired blur was apparently smaller than the blur from the Gaussian blur filter we applied to the stimuli. 
The cursor, which had a circular shape with a diameter of 10 mm, was filled with the noise. The noise was randomly generated before each trial using the same procedure as the background noise. A gray ring frame 1.2 mm thick and with an intensity of 255/2 was added to the cursor. The drifting noise within the cursor was implemented in the following ways. First, a noise whose vertical size was larger than the cursor was generated, although the entire area of the noise was not presented as stimuli. Second, a part of this larger area of noise was cropped and pasted within the cursor, such that the vertical position of the cropped noise ycut was determined depending on x, the position of the cursor in the horizontal axis of the stimuli (Figure 2). Specifically, the cropped area was determined by the following formula:  
\begin{equation} y_{cut} = A_{noise}\,exp\left(-\frac{(c_{x}-x)^{2}}{2(W/8)^{2}}\right),\quad \end{equation}
(1)
where Anoise is the number of pixels equivalent to the amplitude of noise drift [mm], cx is the horizontal center of the screen (i.e., the horizontal center of the background noise), and W is a pixel size equivalent to 35 mm (i.e., the size of the background noise). Note that the position of the cropped noise was changed only vertically and in coordinates relative to the cursor position, not in world coordinates (i.e., when Anoise was zero, the noise was glued to the cursor). The maximum amplitude of noise drift Anoise for each participant was chosen from one of the following nine levels: −7.00, −5.25, −3.50, −1.75, 0.00, 1.75, 3.50, 5.25, and 7.00 mm. When the amplitude of noise drift Anoise was positive, the lower (i.e., positive side of the vertical axis) of the larger area of noise was cropped out; in this case the noise drift within the cursor was perceived to drift upward (i.e., toward the negative side of the vertical axis). To explain how the cropping changed the stimulus appearance, some snapshots of stimuli for the −7.0 mm and 7.00 mm drift conditions are shown in Figure 2
Figure 2.
 
Position ycut of cropped noise for −7.0-, 0.0-, and 7.0-mm drift conditions. White circles denote cropped areas. Solid black lines denote Gaussian functions that define the vertical positions of the cropped noise (Equation 1).
Figure 2.
 
Position ycut of cropped noise for −7.0-, 0.0-, and 7.0-mm drift conditions. White circles denote cropped areas. Solid black lines denote Gaussian functions that define the vertical positions of the cropped noise (Equation 1).
To measure the magnitude of the illusory curve of the cursor path, the actual path of the cursor was modulated on the basis of the staircase method so that it counteracted the illusory curve. The actual cursor position in the vertical axis y′ was defined by the following equation using the current horizontal (x) and vertical (y) positions of the cursor: 
\begin{equation} y^{\prime } = y - A_{path}\,exp\left(-\frac{(c_{x}-x)^{2}}{2(W/8)^{2}}\right),\quad \end{equation}
(2)
where Apath is an arbitrary value that can be experimentally manipulated in a staircase manner and adjusted in response to the participant’s judgments in previous trials. Note that the vertical direction on the display corresponds to the direction away from the participant. 
Two squares, the start and goal points for the participant’s cursor motion, were displayed 70 mm apart on left and right (or right and left) sides of the background noise. The spatial side of the two squares (i.e., of the start and goal points) was determined pseudo-randomly for each trial. 
An eye-shaped fixation point was placed 30 mm below the center of the background noise. An 8 × 8-mm black reference object moved 30 mm below the fixation point. The reference object, which served as a reference for the movement of the cursor, moved 70 mm laterally with a cosine modulation of speed. It took 2 seconds for the reference object to move by 70 mm. The reference object’s x-position Rx was defined by the following equation:  
\begin{equation} R_{x} = \left\lbrace \begin{array}{@{}l@{\quad }l@{}}c_{x} - W cos(2\pi t/T) & \text{(left $\rightarrow $ right)} \\[.5ex] c_{x} + W cos(2\pi t/T) & \text{(right $\rightarrow $ left)} \end{array}\right. \end{equation}
(3)
where t is the elapsed time (ms) since the reference object started moving. Because t ranges from 0 to 2,000 ms and T is 4,000 ms, this equation realizes a half cycle of the cosine. The reference object moved from the same spatial side as the start point. For each frame, the horizontal position of the cursor was compared with the horizontal position of the reference object. When the horizontal deviation between the cursor and the reference object exceeded ±15 mm, the participant was asked to restart the trial. 
A demo code for rendering our visual stimuli is available on github (https://github.com/TYokosaka/Force-Illusion-Induced-by-Visual-Illusion). 
Procedure
We wanted to equalize the stimulus size across the participants’ monitors, which were expected to have different spatial resolutions in pixels. Therefore, before the formal test, we asked the participants to adjust the size of a rectangle on the screen with the left and right keys so that the size of the rectangle was the same as the size of a credit card or a card of equal size, following the method used by Li, Joo, Yeatman, and Reinecke (2020). Because the method assumes that the participants’ monitors have square pixels, participants were asked to adjust only the length of one side. We controlled the stimulus size in millimeters based on the measured screen pixel size. After that, to control the distance between the screen and the participants’ eyes, we instructed the participants to observe the screen from a distance of 600 mm. 
At the beginning of the experiment, participants were provided with written instructions to explain the details of the experiment. After reading the instructions, they moved on to a practice session consisting of the multiple trials described below. 
  • In the first trial, the participants practiced moving the cursor (without noise drifting) from the starting point (on the left side) to the goal point (on the right side) while matching the horizontal position of the cursor with that of the reference object (although in the instruction text, we described the cursor as a “ring” for the sake of simplicity, we call it the cursor in this manuscript). Both Apath and Anoise were zero.
  • In the second trial of the session, the participants observed the stimuli in which the path of the cursor under their control was curved upward when the cursor passed over the background noise; here we set Apath to the number of pixels corresponding to 7.0 mm in the Equation 2. The direction of movement of the cursor was the same as in the first trial. Anoise was zero.
  • In the third trial of the session, the participants were shown a written instruction asking them to judge whether the path of the cursor seemed to curve upward or downward in a two-alternative forced-choice manner. After the instruction had been read, a fixation point was displayed at the center of the screen, and the participants moved the cursor while looking at the fixation point, not at the cursor or the reference object. No eye movements were measured during the task. The direction of movement, Apath, and Anoise were the same as in the second trial.
  • In the fourth trial of the session, the participants were shown a written instruction explaining that the spatial side of the starting and end points would be interchanged. After reading that, they moved the cursor from the starting point (right side) to the goal point (left side), judged the direction of the curve of the cursor path, and reported their judgment by clicking assigned buttons on the display. In this trial, participants observed stimuli in which the path of the cursor under the participant’s control was curved downward when it passed over the background noise; here we set Apath to the number of pixels corresponding to −7.0 mm and Anoise to zero.
  • In the fifth trial, the participants observed stimuli in which the path of the cursor was straight and the noise within the cursor drifted; here Apath was set to zero and Anoise was set to the number of pixels corresponding with 7.0 mm in the Equation 1. The participants were asked to judge the direction (upward or downward) of the curve for the cursor path. Note that participants received no explanation about either the existence of the drifting noise within the cursor or the potential illusory effects of the drifting noise on the appearance of the cursor path.
  • In the sixth and seventh trials, participants underwent the same procedure with Anoise set to the number of pixels corresponding with −7.0 mm and 0.0 mm, respectively (Apath set to zero).
After completing all seven trials of the practice session, the participants moved on to a session consisting of main trials. The parameters are also summarized in Table 1
Table 1.
 
Parameters in the practice session in Experiment 1.
Table 1.
 
Parameters in the practice session in Experiment 1.
In a similar fashion to the practice session, the participants’ task in the main trials was to move the cursor from the start point to the goal point while matching the horizontal position of the cursor with that of the reference object. The participants were again instructed to fixate on the central fixation point. After moving the cursor to the goal point, the participants judged whether the path of the cursor curved up or down as it moved across the background noise and reported their judgment by clicking assigned buttons. For each trial, a description “Please place the ring here and wait” was displayed above the starting point. Two seconds after the participant successfully placed the cursor on the starting point, the reference object appeared and started moving. Participants moved the cursor toward the goal point so that the horizontal position of the cursor matched that of the reference object. If the deviation of the horizontal position between the cursor and the reference object exceeded ±15 mm, the screen returned to the beginning of the trial and displayed the message, “Please try again since the ring movement was not in sync with the movement of the reference object.” When the cursor successfully reached the goal point, the screen for the participants to provide their judgment was displayed. We presented a button that read, “The path of the ring appeared to curve upwards,” on the upper side of the answer screen while we also presented another button that read, “The path of the ring appeared to curve downwards,” on the lower side. When a participant clicked on either of the buttons, they could proceed to the next trial. The participants were also allowed to go back to the beginning of the trial and observe the visual stimuli again without reporting their judgment. 
We used a randomly interleaved staircase method with two staircases. One of the Anoise values (−7.00, −5.25, −3.50, −1.75, 0.00, 1.75, 3.50, 5.25, and 7.00 mm) representing the amplitude of noise drift was assigned to the staircase as a between-subjects factor. Although participants might adapt to the fixed perturbations (i.e., Anoise), we chose this between-subjects design because it was difficult in the online experiment to have the participants perform a sufficiently large number of trials to obtain enough data to perform a proper analysis after randomizing Anoise. Two staircases were presented in a random order: for one staircase, the initial value of Apath in Equation 2 was the number of pixels corresponding with 8.0 mm, and for the other staircase, the initial value of Apath was the number of pixels corresponding with −8.0 mm. For each staircase, if the participant selected “The path of the ring appeared to curve upwards,” the value of Apath was reduced in the next step of the staircase. In contrast, if the participant chose “The path of the ring appeared to curve downwards,” the value of Apath was increased in the next step of the staircase. The degree of increment or decrement in Apath was the number of pixels corresponding with 1.0 mm until the second reversal of the participants’ judgment, 0.75 mm until the fourth reversal, and 0.5 mm until the sixth reversal. The experiment was terminated when the responses in both staircases were reversed six times or when the total number of trials for both staircases reached 50. 
Data analysis
To estimate the magnitude of the illusory curve of the cursor path, we fitted a cumulative logistic distribution function to the response data and calculated the point of subjective equality (the 50% point of the distribution) of Apath for each participant using the psignifit package for Python (Schütt, Harmeling, Macke, & Wichmann, 2016). 
To test whether the amplitude of the noise drift affected the magnitude of the illusory curve of the cursor path across participants, we conducted a nonparametric version of the analysis of variance (ANOVA). After carrying out an aligned rank transform (ART) (Wobbrock, Findlater, Gergle, & Higgins, 2011) for the magnitude of the illusory curve of the cursor path, we conducted a one-way repeated-measures ANOVA with the amplitude of the noise drift as a between-subjects factor. Multiple comparisons tests within the ART paradigm (ART-C) (Elkin, Kay, Higgins, & Wobbrock, 2021) with Bonferroni correction were then performed for the factor of the amplitude of the noise drift. 
Results and discussion
Figure 3A shows the distribution of the estimates of Apath canceling the illusory curve of the cursor path as a function of the drift amplitude of the noise. The data for noise drift conditions of −3.5, −1.75, and 1.75 [mm] violated the assumption of normality (Shapiro–Wilk test of normality, Bonferroni-corrected p < 0.05). As such, there was an issue that it was not appropriate for us to analyze the data by an ANOVA in a purely parametric manner. To solve this issue, we conducted an ART before the ANOVAs. An ANOVA using data from an ART is often called an “ART-ANOVA.” It has been suggested that ART-ANOVA could be applied to nonparametric as well as parametric data. Moreover, as with a parametric ANOVA, ART-ANOVA enables us to predesign the number of samples to be tested. For the reasons described elsewhere in this article, we believe that it is appropriate to use ART-ANOVA for our data. The results of ART-ANOVA showed the significant main effect of the amplitude of the noise drift, \(F(8, 261) = 25.79, p \lt 0.001, \eta _{p}^{2} = 0.44\). Results of multiple comparison tests (Figure 3B) showed that the magnitudes of the illusory curve of the cursor path induced by a larger random noise drift (−7.0 to −3.5 mm and 3.5 to 7.0 mm) tended to be larger than those induced by a smaller random noise drift (−1.75 to 1.75 mm). To estimate a gain of response for the amplitude of noise drift, we conducted a linear regression analysis and calculated the slope. The slope and intercept were −0.32, t(7) = −14.78, p < 0.001, and 0.20, t(7) = 2.11, p = 0.07, and R2 was 0.97. The slope value suggests that an actual cursor modulation (i.e., Apath), with an amplitude approximately one-third of the noise drift, and in the opposite direction to the noise drift, is required to cancel out the illusory cursor curve. 
Figure 3.
 
Results of Experiment 1. (A) Distribution of the estimate of Apath canceling the illusory curve of the cursor path as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values.
Figure 3.
 
Results of Experiment 1. (A) Distribution of the estimate of Apath canceling the illusory curve of the cursor path as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values.
We found that the illusory curve of the cursor path could be induced by the noise drift within the cursor. We believe this report is the first of the phenomenon in which the path of a mouse cursor that participants actively manipulated could be perceptually altered by motion information within the cursor, while many previous studies have reported that the position of a static patch (De Valois & De Valois, 1991; Durant & Zanker, 2009; Ramachandran & Anstis, 1990; Whitney, 2002) and the trajectory of a moving patch (Lisi & Cavanagh, 2015; Shapiro et al., 2010; Tse & Hsieh, 2006) could be perceptually modulated by motion information within the patch. 
We also found that the estimate of the actual cursor curve Apath canceling the illusory curve was approximately one-third of the amount of noise drift. One possible explanation for a gain lower than 1 might be that participants compensated for the illusory curve by actually moving their mouse in the opposite direction to the illusory curve. If this is the case, the estimate of the actual cursor curve Apath would be smaller, because the illusory curve is canceled out by both the experimentally manipulated cursor curve Apath and the compensatory movement of the participant’s mouse. This possibility is expected to be verified by recording and analyzing mouse trajectories in future studies. 
Using the stimuli of this experiment, in the next experiment we investigated whether the illusory curve of the cursor path could cause an unintended force (i.e., a force that the participants experienced as if external force was being applied to the cursor under their control). 
Experiment 2
The purpose of this experiment was to check whether the participants could feel an unintended force for the illusory curve in the path of the cursor under their control, and also whether the magnitude of the illusory force would increase with the magnitude of the illusory curve of the cursor path. We used stimuli in which the noise within the cursor drifted vertically while the actual path of the cursor was not modulated. Similar to the protocol for Experiment 1, the participants were asked to move the cursor by using the mouse and then rate how strongly they had felt an unintended vertical force being applied to the cursor. Because the cursor is, so to speak, an agent that externally moves according to the participants’ intention, we believe that cursor movements that do not conform to the participants’ intention can be interpreted as stemming from an external force applied to the agent (i.e., the cursor) as mentioned in the Introduction. To ascertain whether the strength of force which the participants felt was dependent on the magnitude of the illusory curve in the cursor path, we also asked the participants to rate the magnitude of the apparent curve in the cursor path separately. 
Participants
The protocols for consent, recruitment, and ethics were identical to those used in Experiment 1. By using a statistical calculator, Morepower 6.0 (Campbell & Thompson, 2012), we calculated the minimum sample size required for a within-subjects design with a medium-effect size, Cohen’s F = 0.25, a power of 80%, and an alpha of 5%. The minimum sample size that satisfies these conditions for nine noise drift levels was 32. Because the data available from the crowdsourcing company we used in this study are known to have many outliers (Kawabe, 2021), we decided to recruit a larger number. Forty-eight people (24 females and 24 males), who had not participated in Experiment 1, participated in this experiment. The mean ± standard deviation of their ages was 35.48 ± 7.88 years. 
Stimuli
The stimuli were identical to those used in Experiment 1 except for the following. In contrast with the stimuli in Experiment 1, the actual path of the cursor was not modulated (i.e., Apath was always zero). That is, the physical cursor path followed the manipulation by the participants. 
Procedure
At the beginning of the experiment, participants were provided with written instructions explaining the details of the experiment. After reading the instructions, they moved on to a practice session following the multiple trials described. 
  • The first trial was identical to that in Experiment 1.
  • In the second trial of the session, a fixation point was displayed at the center of the screen, and the participants moved the cursor while looking at the fixation point, not at the cursor or the reference object. No eye movements were measured during the task. The direction of movement of the cursor was from the left side to the right side and Anoise was zero.
  • In the third trial of the session, the participants moved the cursor and observed the cursor moving in a physically straight line while the noise within it drifted; here the direction of movement of the cursor was the same as in the first trial and Anoise was set to the number of pixels corresponding with 7.0 mm in the Equation 1. Here, participants were given the explanation that “the ring may appear to curve upward or downward” and that they “may feel as if an unintended force is applied to the ring in an upward or downward direction.”
  • In the fourth trial of the session, the participants were shown a written instruction asking them to evaluate the magnitude of the cursor’s path change and the magnitude of the unintended force, respectively. We adopted a three-point semantic differential scale for the magnitude ratings. For the magnitude rating of the cursor’s path change, “no path change at all” was placed at the left end of the scale bar, and “trajectory appeared to change greatly” was placed at the right end of the scale bar, with three anchors at equal intervals. For the magnitude rating of unintended force, “I felt no force at all” was placed at the left end of the scale bar, and “I felt a large force” was placed at the right end of the scale bar, with three anchors at equal intervals. The direction of movement of the cursor and Anoise were the same as in the third trial.
  • In the fifth trial, the participants were shown a written instruction explaining that the spatial side of the starting and ending points would be interchanged. After reading that, they moved the cursor from the starting point on the right side of the monitor to the goal point on the left side, rated the magnitude of the illusory curve of the cursor path and the magnitude of any unintended force by clicking anchors on the scale bars. In this trial, participants observed stimuli in which the path of the cursor under their control was curved downward when it passed over the background noise; here we set Anoise to the number of pixels corresponding with −7.0 mm.
  • In the sixth trial, participants then underwent a trial with Anoise set to the number of pixels corresponding with 0.0 mm.
After completing all six trials of the practice session, the participants moved onto a session consisting of main trials. The parameters are also summarized in Table 2
Table 2.
 
Parameters in the practice session in Experiment 2.
Table 2.
 
Parameters in the practice session in Experiment 2.
The procedure in the main trial was identical to that used in Experiment 1 except for the following. After the participants moved the cursor from the start to end points, they were asked to rate the magnitude of any unintended force they had felt being applied to the cursor as well as the magnitude of the illusory curve of the cursor path. In a similar fashion to Experiment 1, immediately after the cursor successfully reached the goal point, a screen for the rating was displayed. After the participant completed ratings for both magnitudes, they could proceed to the next trial. Participants were also allowed to go back to the beginning of the trial because the experimental task involved them both controlling the mouse and making judgments about the cursor’s appearance, and thus there was a concern that the participants may not always be equally attentive to both the mouse control and the cursor’s appearance. When participants went back to the beginning of the trial without rating the magnitudes, they moved the cursor and observed the visual stimuli again. On each trial, Anoise was chosen from the following nine conditions: −7.00, −5.25, −3.50, −1.75, 0.00, 1.75, 3.50, 5.25, and 7.00 mm. These nine conditions were presented in a random order and repeated in four sets, so that each participant performed a total of 36 trials. 
Data analysis
We averaged the rating scores for the magnitude ratings of the unintended force and the cursor’s path change across four repetitions, respectively, for each Anoise condition and for each participant. To test whether the amplitude of the noise drift affected the averaged rating scores of unintended force across participants, we conducted a nonparametric version of the ANOVA. Rating scores having upper and lower limits do not exhibit normality under the conditions in which mean scores are located near the upper or lower limit. Therefore, after we carried out an ART (Wobbrock et al., 2011) for the rating scores of unintended force, we conducted a one-way repeated-measures ANOVA with the amplitude of noise drift as a within-subject factor. Multiple comparison tests within the ART paradigm (ART-C) (Elkin et al., 2021) with Bonferroni correction were then performed for the factor of the amplitude of the noise drift. To examine the relationship between the rating score for the magnitude of the unintended force and the rating score for the magnitude of the apparent curve in the cursor path, we computed the Spearman’s rank correlation coefficient between them. 
Results and discussion
Figure 4A shows the distribution of the averaged rating scores for the unintended force as a function of amplitude of the noise drift. All data violated the assumption of normality (Shapiro-Wilk test of normality, Bonferroni-corrected p < 0.05). The results of ART-ANOVA showed the significant main effect of the amplitude of the noise drift, \(F(8, 376) = 21.72, p \lt 0.001, \eta _{p}^{2} = 0.32\). Results of multiple comparison tests (Figure 4B) showed that the rating scores for the unintended force induced by a larger random noise drift (−7.0 to −3.5 mm and 3.5 to 7.0 mm) tended to be larger than those induced by a smaller random noise drift (−1.75 to 1.75 mm). Figure 4C shows the relationship between the rating score for the unintended force and the rating score for the apparent curve in the cursor path. A Spearman’s correlation coefficient rs between the rating score for the magnitude of the unintended force and the rating score for the magnitude of the apparent curve in the cursor path was 0.86, p < 0.01. Figure 4D shows the relationship between the rating score for the unintended force in this experiment and the magnitude of the illusory curve in the cursor path in Experiment 1. In this graph, we averaged each rating score across participants. The Spearman’s correlation coefficient rs between them was 0.85, p < 0.01. These results suggest that a stronger sensation of unintended force co-occurs with a stronger illusion for the cursor path. 
Figure 4.
 
Results of Experiment 2. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values. (C) Bubble plot of the rating scores for the unintended force and those of the apparent curve of the cursor path. Marker size represents the number of participants. (D) Scatter plot of the averaged rating scores for the unintended force in Experiment 2 as a function of the absolute magnitude of the illusory curve in the cursor path in Experiment 1. Each marker represents each random noise drift condition.
Figure 4.
 
Results of Experiment 2. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values. (C) Bubble plot of the rating scores for the unintended force and those of the apparent curve of the cursor path. Marker size represents the number of participants. (D) Scatter plot of the averaged rating scores for the unintended force in Experiment 2 as a function of the absolute magnitude of the illusory curve in the cursor path in Experiment 1. Each marker represents each random noise drift condition.
In summary, the greater the amplitude of the noise drift within the cursor, the greater the sensation of an unintended force being applied to it. Because force applied to the mouse was constant among conditions with different amplitudes of the noise drift, the unintended force the participants reported was the product of an illusion. The magnitude of the illusory force was also highly correlated with the magnitude of the illusory curve of the cursor path. 
There was another issue that needed to be resolved to support the conclusions reached in the previous paragraph. The magnitude of the illusory curve in the cursor path increased with the amplitude of the noise drift within the cursor, which made it difficult to disentangle which factor the magnitude of illusion in the cursor path or the amplitude of the noise drift within the cursor was critical to the illusion of an unintended force being applied to the cursor. To address the issue, we performed the Experiment 3
Experiment 3
The purpose of this experiment was to check whether the magnitude of the illusory force was determined by the amplitude of the noise drift within the cursor, by manipulating the visibility of the ring frame around the cursor. In Experiments 1 and 2, the color of the cursor ring frame was set to gray because we wanted to induce a strong cursor path illusion by reducing the visibility of the cursor against the background noise. In this experiment, we tested an additional condition in which a black ring frame was applied to the cursor. It is known that the MIPS becomes weaker as the visibility of the edge of the patch containing the motion stimulus increases (Hisakata, Hayashi, & Murakami, 2016; Ramachandran & Anstis, 1990). The similar effect of edge visibility on the landing position of saccadic eye movements has been reported when the saccadic target position was modulated by MIPS (Kosovicheva et al., 2014). In our preliminary observations, we confirmed that the cursor with the black ring frame, which had greater visibility against the background noise than the cursor with the gray ring frame, strongly suppressed the illusory curve in the cursor path (Supplementary Movie 2). We expected that the black ring condition would decrease the magnitude of the unintended force because it decreased the magnitude of the apparent curve in the cursor path. In contrast, if the amplitude of the noise drift within the cursor was important, the visibility of the ring frame would not modulate the magnitude of the unintended force. 
Participants
The protocols for consent, recruitment, and ethics were identical to those used in Experiment 1. By using a statistical calculator, Morepower 6.0 (Campbell & Thompson, 2012), we calculated the minimum sample size for a 3 × 2 ANOVA as a within-subjects factor under the following conditions: a medium effect size, Cohen’s F = 0.25, a power of 80%, and an alpha of 5%. The minimum sample size that satisfies these conditions for all factors was 126. One hundred twenty-six people (63 females and 63 males), who had not participated in Experiments 1 and 2, participated in this experiment. The mean ± standard deviation of their ages was 35.24 ± 8.46 years. 
Stimuli
Stimuli were identical to those as used in Experiments 1 and 2 except for the followings. In addition to the condition with a gray ring applied to the cursor, we tested a condition in which a black ring was applied to the cursor. The grayscale intensity of the blackness of the ring was 0. 
Procedure
At the beginning of the experiment, participants were provided with written instructions explaining the details of the experiment. After reading the instructions, they moved on to a practice session following the multiple trials described. 
  • The first four trials were identical to those in Experiment 2.
  • In the fifth trial, the participants moved the cursor with the black ring. Anoise was set to the number of pixels corresponding with 7.0 mm. After they moved the cursor, they rated the magnitude of any unintended force they had felt being applied to the cursor as well as the magnitude of the apparent curve of the cursor path.
  • In the sixth and seventh trial, participants then underwent the same procedure with the gray and black ring conditions wherein Anoise was 0.
After completing all seven trials of the practice session, the participants moved onto a session consisting of main trials. The parameters are also summarized in Table 3
Table 3.
 
Parameters in the practice session in Experiment 3.
Table 3.
 
Parameters in the practice session in Experiment 3.
In the main trials, Anoise was chosen from the three conditions: 0.00, 3.50, and 7.00 mm. The color of the ring frame applied to the cursor was also chosen from gray (with low visibility) and black (with higher visibility). These six conditions (three Anoise conditions × two visibility conditions) were presented in random order and repeated in six sets, so that each participant performed a total of 36 trials. 
Data analysis
We averaged the rating score separately for the unintended force and the rating score for the apparent curves of the cursor path across six repetitions for each condition and for each participant. To test whether the amplitude of the noise drift and the visibility of the cursor affected the averaged rating scores for the unintended force across participants, we again conducted a nonparametric version of the ANOVA. In a similar fashion to Experiment 2, we carried out an ART (Wobbrock et al., 2011) for the rating scores for the unintended force and then conducted a two-way repeated-measures ANOVA with the amplitude of the noise drift and the visibility of the cursor as within-subject factors. Multiple comparison tests within the ART paradigm (ART-C) (Elkin et al., 2021) with Bonferroni correction were then performed for the factor of the amplitude of the noise drift. To examine the strength of the relationship between the rating score for the unintended force and the rating score for the apparent curve in the cursor path, we computed the Spearman’s rank correlation coefficient between them. 
Results and discussion
Figure 5A shows the distribution of the rating score for the unintended force as a function of the amplitude of the noise drift. All data violated the assumption of normality, Shapiro–Wilk test of normality, Bonferroni-corrected p < 0.05. The results of ART-ANOVA showed the significant main effects of the amplitude of the noise drift condition, \(F(2, 250) = 89.75, p \lt 0.001, \eta _{p}^{2} = 0.42\), and the visibility condition, \(F(1, 125) = 101.21, p \lt 0.001, \eta _{p}^{2} = 0.44\). The interaction between the visibility and the amplitude of the noise drift was also significant, \(F(2, 250) = 54.51, p \lt 0.001, \eta _{p}^{2} = 0.30\). The simple main effects of the visibility were significant for the 3.5-mm condition, \(F(1, 125) = 65.46, p \lt 0.01, \eta _{p}^{2} \lt 0.34\), and the 7.0-mm condition, \(F(1, 125) = 82.83, p \lt 0.01, \eta _{p}^{2} \lt 0.40\), but not for the 0-mm condition, \(F(1, 125) = 0.02, p = 0.88, \eta _{p}^{2} \lt 0.01\). The simple main effects of the amplitude of the noise drift were significant for the lower visibility (gray ring) condition, \(F(2, 250) = 90.87, p \lt 0.01, \eta _{p}^{2} \lt 0.42\), and the higher visibility (black ring) condition, \(F(2, 250) = 24.72, p \lt 0.01, \eta _{p}^{2} \lt 0.17\). Multiple comparison tests showed the significant differences for all pairs of the amplitude of the noise shift conditions for both the lower visibility condition (Table 4) and the higher visibility condition (Table 5). Figure 5B shows the relationship between the rating score for the unintended force and the rating score for the apparent curve in the cursor path. The Spearman’s correlation coefficient rs between the two types of the rating scores was 0.87 (p < 0.01). 
Table 4.
 
Results of multiple comparison tests for the low visibility conditions.
Table 4.
 
Results of multiple comparison tests for the low visibility conditions.
Table 5.
 
Results of multiple comparison tests for the high visibility conditions.
Table 5.
 
Results of multiple comparison tests for the high visibility conditions.
Figure 5.
 
Results of Experiment 3. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Gray and black boxes denote the rating scores in the lower visibility (gray ring) and higher visibility (black ring) conditions, respectively. Solid black lines on the gray boxes and solid white lines on the black boxes denote median values. Dashed gray lines denote mean values. (B) Bubble plot of the rating score for the unintended force and the rating score for the apparent curve of the cursor path. Marker size represents the number of participants.
Figure 5.
 
Results of Experiment 3. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Gray and black boxes denote the rating scores in the lower visibility (gray ring) and higher visibility (black ring) conditions, respectively. Solid black lines on the gray boxes and solid white lines on the black boxes denote median values. Dashed gray lines denote mean values. (B) Bubble plot of the rating score for the unintended force and the rating score for the apparent curve of the cursor path. Marker size represents the number of participants.
These results showed that the visibility of the ring frame applied to the cursor strongly altered the rating score for the unintended force. Specifically, the high visibility black ring frame significantly decreased the rating score, in comparison with the low visibility gray ring frame. The amplitude of the noise drift was matched between the two ring visibility conditions. Thus, the results indicate that the major factor in the modulation of the subjective magnitude of the unintended force was the illusory curve in the cursor path due to the noise drift within the cursor and that the contribution of the amplitude of the noise drift itself was relatively minor. 
General discussion
In the present study, we examined whether participants experience an illusory sensation of force being applied to a cursor that they manipulated when the path of the cursor was modulated by a visual illusion. In Experiment 1, we found that participants reported that the path of the cursor was curved in the direction of the noise drifts inside the cursor. In Experiment 2, we showed that participants experienced the illusion of an unintended force being applied to the cursor when the apparent cursor path was modulated by the illusion reported in Experiment 1. Experiment 3 showed that, rather than the amplitude of the noise drift within the cursor, the magnitude of the illusory curve of the cursor path played a major role in generating the illusion of an unintended force. 
Our results indicate that the mechanism responsible for the illusory force is affected by illusory distortion of the actual outcome of motor action. It has been reported that illusory changes in position and direction of motion affected motor control including saccadic eye movements and pointing actions (Carey, 2001; Lisi & Cavanagh, 2015; Kosovicheva et al., 2014; Ueda et al., 2019), and that the sense of agency was also influenced by visual feedback based on cross-modal illusions (Tanaka & Watanabe, 2021). Extending the findings in the previous studies, the present study has reported that the illusory force was caused when the actual outcome of motor action was modulated by a visual illusion. Because the illusory force was influenced by the magnitude of the illusory curve in the cursor path, the mechanism for generating the illusory force may not be related to an action pathway, as proposed by Goodale and Milner (1992) and Westwood and Goodale (2011), which is reportedly insusceptible to visual illusion. Rather, it is likely that the mechanism responsible for the illusory force is involved with the cross-modal processing that integrates neural signals of expected and actual outcomes of motor action. 
Other possible mechanisms than cross-modal integration may be assumed to be involved in the generation of the illusory force. For example, there is a possibility that the illusory force is computed in visual processing only on the basis of the apparent curve of the cursor path. In this scenario, when the participant who visually predicts a straight path for the cursor observes the cursor’s distorted path, he or she possibly makes a higher-level visual interpretation that an external force changed the path of the cursor. Asides from the higher-order visual interpretation, a cross-modal transfer (Biocca, Kim, & Choi, 2001) is another possible candidate to explain the illusory force. In line with the theory of cross-modal transfer, when the visual sensation of force is generated by the cursor’s illusory path changes, a haptic sensation of force can also co-occur based on the association between visual and haptic modalities. The possibility that these mechanisms contributed to the illusory force has not been distinguished in previous studies or in the present study, and the improvement of experimental designs to isolate the contributions of these mechanisms is warranted in the future. 
It may be necessary to discuss the further role of participants’ expectations for the cursor path in the generation of the illusory force. In our study, we asked the participants to move the cursor from the start to goal points along a straight path. Hence, the participants could expect that the cursor would move along a straight path under their control. We suggest that, in this situation, the brain could easily compare the discrepancy between the expected and actual paths of the cursor because they have only to detect a deviation in the apparent cursor path from the expected straight path. In this respect, it would be intriguing to check whether the illusory force can be weakened when the participant cannot easily anticipate how the cursor will move. For example, the illusory force may be weakened when the participant is asked to move the cursor along a sinusoidal or more complex path because moving the cursor along more complex paths tends to make the comparison between the expected and actual paths noisier, and thus, detection of the deviation between the expected and actual paths gets harder. 
A noise drift could affect participants’ mouse trajectories for two reasons, although we did not measure actual mouse trajectories in this experiment. First, the mouse trajectory could change as a result of the sensation of force induced by noise drift during reaching movements. In a previous study, it was reported that participants’ cursor trajectories were altered by modulating the mechanical force feedback (Shadmehr & Mussa-Ivaldi, 1994). If this motor compensation could be observed in our stimulus setup, it would provide evidence that participants actually felt a force owing to the noise drift. Second, however, there was a possibility that the unexpected change in visual cursor trajectories might have induced participants to move their mouse to counteract the apparent curve, without feeling any force. Indeed, earlier studies have reported that motor correction to achieve reaching movement occurs when representations of visual hand position are modulated (Sarlegna et al., 2003; Saunders & Knill, 2003; Smeets & Brenner, 2003). It will be important to investigate the effect of random drift on mouse cursor trajectory while disentangling these possibilities. 
In the present study, we adopted random noise rather than Gabor patches as stimuli. The reason for this was that it was difficult to conduct gamma correction of luminance emitted from the monitor in online experiments. Without gamma correction, there was no guarantee that the background would have a neutral mean luminance on each participant’s monitor, which raised the concern that the difference in luminance between the cursor and the background might serve as a position cue that could weaken the illusion for the cursor path. In the present study, by using random noise for the cursor and its background, we eliminated the involvement of the luminance cue in the task, and hence, possibly enhanced the illusory curve of the cursor path. Conducting laboratory experiments on the difference in the magnitude of the illusory force between random noise and the Gabor patches would be of interest for future research, even though it is beyond the scope of the present study. 
One might suspect possible artifacts owing to a response bias caused by directly asking about the unintended force. One way to alleviate bias in participants’ responses about the force experience might be to present other questions unrelated to the force experience, for example, in the case of our experiment, about the magnitude of the cursor’s roughness. Also, one might suspect the bias resulting from our instruction asking the participants to judge the magnitude of the force. Under this instruction, the participants might infer that they should respond to a greater force in specific conditions. A possible way to alleviate the bias is to ask participants to judge the similarity of the force between stimuli rather than the magnitude of the force itself. We can estimate the relative magnitude of the force for stimuli by using some scaling methods, such as a multidimensional scaling analysis and a maximum likelihood difference scaling analysis (Maloney & Yang, 2003), which can construct a magnitude scale based on the similarity ratings. 
Finally, we want to discuss the technical contributions and limitations of the present study. The phenomenon reported in the present study is possibly applicable to psychophysical studies on online sensorimotor control. It would be interesting to check how the illusory shift of the cursor path can change online motor control along a relatively long path of cursor movement. Moreover, based on the phenomenon reported in the present study, it is possible to propose a novel extended reality technique for changing the apparent path of a cursor without modifying its physical path and, also, for giving users the illusory sensation of force being applied to the cursor that they are controlling. There are several advantages to using the illusory path of the cursor rather than the physical modulation of the cursor path that has been proposed in traditional pseudo-haptics research. One advantage is that the software does not need to compensate for misalignment between the hand (mouse) position and the cursor position. In the classical pseudo-haptics method, if the cursor position is modulated for a long time, the gap between the actual hand position and the cursor position becomes larger. The illusory motion method, which does not require modulation of the actual cursor position, does not cause this misalignment to occur, and does not need to compensate for it. Another advantage of using illusions is that by overlaying illusory stimuli on real objects such as hands through augmented reality technologies, force sensation can be induced without modulating their positions. The strengths described allow us to develop several applications. For example, our method can contribute to virtual reality applications that convey feelings such as the forces derived from objects and the weight of objects. Our method can also enable navigation applications to guide the user's attention in the desired direction by presenting an illusory force. 
The limitation of the technique is that the cursor needs to be observed in the peripheral visual field. This issue of cursor eccentricity may be resolved by modifying the size and/or spatial frequency of the cursor. Future studies need to extend the present scientific findings into techniques for adding force sensation based on the visual illusion wherein users directly view the cursor as they usually do in real life. 
The visual design of the cursor and its surround can also be improved. In the experiments for the present study, we moved random noises within the cursor. In contrast, it would be an interesting direction for future studies to check whether the drift of other texture patterns than noise within the cursor can cause the illusory force. For example, a Gabor patch is one of the common visual patterns for inducing an illusory position shift or trajectory change. In our demo script (https://github.com/TYokosaka/Force-Illusion-Induced-by-Visual-Illusion), we can experience an illusion similar to that investigated in this study, by manipulating a drifting Gabor patch cursor. The necessity of using background noise can also be examined in future studies. 
Acknowledgments
The authors thank Seitaro Kaneko for his help in the implementation of our preliminary scripts for experimentation. 
T.Y., Y.U., and T.K. conceived the experiment. T.Y. and T.K. implemented the experimental system. T.Y. analyzed the results. All authors interpreted the data, wrote the manuscript, and reviewed it. 
Commercial relationships: none. 
Corresponding author: Takumi Yokosaka. 
Address: NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, 3-1, Morinosato Wakamiya, Atsugi, Kanagawa 243-0198, Japan. 
References
Asai T., Tanno Y. (2007). The relationship between the sense of self-agency and schizotypal personality traits. Journal of Motor Behavior, 39(3), 162–168, http://www.tandfonline.com/doi/abs/10.3200/JMBR.39.3.162-168.
Ban Y., Ujitoko Y. (2018). Enhancing the pseudo-haptic effect on the touch panel using the virtual string. 2018 IEEE International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS). March 25–28, 2018, doi:10.1109/haptics.2018.8357188.
Biocca F., Kim J., Choi Y. (2001). Visual touch in virtual environments: An exploratory study of presence, multimodal interfaces, and cross-modal sensory illusions. Presence, 10(3), 247–265, doi:10.1162/105474601300343595.
Blakemore S.-J., Frith C. D., Wolpert D. M. (1999). Spatio-temporal prediction modulates the perception of self-produced stimuli. Journal of Cognitive Neuroscience, 11(5), 551–559, doi:10.1162/089892999563607.
Campbell J., Thompson V. (2012). Morepower 6.0 for anova with relational confidence intervals and Bayesian analysis. Behavior Research Methods, 44, 1255–1265, doi:10.3758/s13428-012-0186-0.
Carey D. P. (2001). Do action systems resist visual illusions? Trend in Cognitive Science, 5(3), 109–113, doi:10.1016/s1364-6613(00)01592-8.
De Valois R. L., De Valois K. K. (1991). Vernier acuity with stationary moving gabors. Vision Research, 31(9), 1619–1626, doi:10.1016/0042-6989(91)90138-u.
Dominjon L., Lecuyer A., Burkhardt J., Richard P., Richir S. (2005). Influence of control/display ratio on the perception of mass of manipulated objects in virtual environments. In: IEEE Proceedings VR 2005. Virtual Reality, 2005. IEEE, doi:10.1109/vr.2005.1492749.
Durant S., Zanker J. M. (2009). The movement of motion-defined contours can bias perceived position. Biology Letters, doi:10.1098/rsbl.2008.0622.
Elkin L. A., Kay M., Higgins J. J., Wobbrock J. O. (2021). An aligned rank transform procedure for multifactor contrast tests. In: The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 754–768.
Farrer C., Valentin G., Hupé J. (2013). The time windows of the sense of agency. Consciousness and Cognition, 22(4), 1431–1441, doi:10.1016/j.concog.2013.09.010.
Gomez Jauregui D. A., Argelaguet F., Olivier A.-H., Marchal M., Multon F., Lecuyer A. (2014). Toward “pseudo-haptic avatars”: Modifying the visual animation of self-avatar can simulate the perception of weight lifting. IEEE Transactions on Visualization and Computer Graphics, 20(4), 654–661, doi:10.1109/tvcg.2014.45.
Goodale M. A., Milner A. (1992). Separate visual pathways for perception and action. Trend in Neuroscience, 15, 20–25, doi:10.1016/0166-2236(92)90344-8.
Haggard P. (2017). Sense of agency in the human brain. Nature Reviews Neuroscience, 18, 196–207, doi:10.1038/nrn.2017.14.
Hisakata R., Hayashi D., Murakami I. (2016). Motion-induced position shift in stereoscopic and dichoptic viewing. Journal of Vision, 16, 3, doi:10.1167/16.13.3.
Honda T., Hagura N., Yoshioka T., Imamizu H. (2013). Imposed visual feedback delay of an action changes mass perception based on the sensory prediction error. Frontiers in Psychology, 4, 760, doi:10.3389/fpsyg.2013.00760.
Kawabe T. (2013). Inferring sense of agency from the quantitative aspect of action outcome. Consciousness and Cognition, 22(2), 407–412, doi:10.1016/j.concog.2013.01.006.
Kawabe T. (2021). Perceptual properties of the poisson effect. Frontiers in Psychology, 11, 2020, doi:10.3389/fpsyg.2020.612368.
Kawabe T., Roseboom W., Nishida S. (2013). The sense of agency is action–Effect causality perception based on cross-modal grouping. Proceedings of the Royal Society B: Biological Sciences, 280(1763), 20130991, doi:10.1098/rspb.2013.0991.
Kosovicheva A. A., Wolfe B. A., Whitney D. (2014). Visual motion shifts saccade targets. Attention, Perception, & Psychophysics, 76, 1778–1788, doi:10.3758/s13414-014-0680-9.
Li Q., Joo S. J., Yeatman J. D., Reinecke K. (2020). Controlling for participants' viewing distance in large-scale, psychophysical online experiments using a virtual chinrest. Scientific Reports, 10(1), 1–11.
Lisi M., Cavanagh P. (2015). Dissociation between the perceptual and saccadic localization of moving objects. Current Biology, 25(19), 2535–2540, doi:10.1016/j.cub.2015.08.021.
Lécuyer A., Burkhardt J.-M., Etienne L. (2004). Feeling bumps and holes without a haptic interface, doi:10.1145/985692.985723.
Maloney L. T., Yang J. N. (2003). Maximum likelihood difference scaling. Journal of Vision, 3, 5, doi:10.1167/3.8.5.
McCarthy L., Reas C., Fry B. (2015). Getting started with P5.js: Making interactive graphics in JavaScript and processing. Make: Technology on Your Time, Maker Media, Incorporated, https://books.google.co.jp/books?id=027ooQEACAAJ.
van Mensvoort I. M. K. (2002). What you see is what you feel. DIS ‘02: Proceedings of the 4th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (pp. 345--348), doi:10.1145/778712.778761.
Mensvoort K. V., Vos P., Hermes D. J., Liere R. V. (2010). Perception of mechanically and optically simulated bumps and holes. ACM Transactions of Applied Perception, 7(2), 1–24, doi:10.1145/1670671.1670674.
Nomoto A., Ban Y., Narumi T., Tanikawa T., Hirose M. (2016). Supporting precise manual-handling task using visuo-haptic interaction. AH ‘16: Proceedings of the 7th Augmented Human International Conference 2016, 10, doi:10.1145/2875194.2875216.
Osumi M., Nobusako S., Zama T., Taniguchi M., Shimada S., Morioka S. (2018). Sensorimotor incongruence alters limb perception and movement. Human Movement Science, 57, 251–257, doi:10.1016/j.humov.2017.09.003.
Ramachandran V. S., Anstis S. M. (1990). Illusory displacement of equiluminous kinetic edges. Perception, 19(5), doi:10.1068/p190611.
Rietzler M., Geiselhart F., Gugenheimer J., Rukzio E. (2018). Breaking the tracking. CHI ‘18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–12), doi:10.1145/3173574.3173702.
Samad M., Gatti E., Hermes A., Benko H., Parise C. (2019). Pseudo-haptic weight. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, doi:10.1145/3290605.3300550.
Sarlegna F., Blouin J., Bresciani J.-P., Bourdin C., Vercher J.-L., Gauthier G. M. (2003). Target and hand position information in the online control of goal-directed arm movements. Experimental Brain Research, 151, 524–535, doi:10.1007/s00221-003-1504-7.
Sato A., Yasuda A. (2005). Illusion of sense of self-agency: Discrepancy between the predicted and actual sensory consequences of actions modulates the sense of self-agency, but not the sense of self-ownership. Cognition, 94(3), 241–255, doi:10.1016/j.cognition.2004.04.003.
Saunders J. A., Knill D. C. (2003). Humans use continuous visual feedback from the hand to control fast reaching movements. Experimental Brain Research, 152, 341–352, doi:10.1007/s00221-003-1525-2.
Schütt H. H., Harmeling S., Macke J. H., Wichmann F. A. (2016). Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data. Vision Research, 122, 105–123, https://www.sciencedirect.com/science/article/pii/S0042698916000390, https://doi.org/10.1016/j.visres.2016.02.002.
Shadmehr R., Mussa-Ivaldi F. (1994). Adaptive representation of dynamics during learning of a motor task. Journal of Neuroscience, 14, 3208–3224, doi:10.1523/jneurosci.14-05-03208.1994.
Shanks D. R., Pearson S. M., Dickinson A. (1989). Temporal contiguity and the judgement of causality by human subjects. Quarterly Journal of Experimental Psychology B: Comparative and Physiological Psychology, 41(2), 139–159, doi:10.1080/14640748908401189.
Shapiro A., Lu Z.-L., Huang C.-B., Knight E., Ennis R. (2010). Transitions between central and peripheral vision create spatial/temporal distortions: A hypothesis concerning the perceived break of the curveball. PLoS One, 5(10), e13296, doi:10.1371/journal.pone.0013296.
Smeets J., Brenner E. (2003). Fast corrections of movements with a computer mouse. Spatial Vision, 16, 365–376, doi:10.1163/156856803322467581.
Smith W. M. (1972). Feedback: Real-time delayed vision of one's own tracking behavior. Science, 176(4037), 939–940, doi:10.1126/science.176.4037.939.
Taima Y., Ban Y., Narumi T., Tanikawa T., Hirose M. (2014). Controlling fatigue while lifting objects using pseudo-haptics in a mixed reality space. In: 2014 IEEE Haptics Symposium (HAPTICS). IEEE, doi:10.1109/haptics.2014.6775451.
Takamuku S., Gomi H. (2015). What you feel is what you see: inverse dynamics estimation underlies the resistive sensation of a delayed cursor. Proceedings of the Royal Society B: Biological Sciences, 282(1811), 20150864, doi:10.1098/rspb.2015.0864.
Tanaka K., Watanabe K. (2021). Sense of agency with illusory visual events. Journal of Experimental Psychology, Human Perception and Performance, 47, 228–251, doi:10.1037/xhp0000886.
Tse P., Hsieh P.-J. (2006). The infinite regress illusion reveals faulty integration of local and global motion signals. Vision Research, 46, 3881–3885, doi:10.1016/j.visres.2006.06.010.
Ueda H., Abekawa N., Ito S., Gomi H. (2019). Distinct temporal developments of visual motion and position representations for multi-stream visuomotor coordination. Scientific Reports, 9, 12104, doi:10.1038/s41598-019-48535-0.
Wen W., Yamashita A., Asama H. (2015a). The influence of action-outcome delay and arousal on sense of agency and the intentional binding effect. Consciousness and Cognition, 36, 87–95, http://www.sciencedirect.com/science/article/pii/S1053810015001415, doi:10.1016/j.concog.2015.06.004.
Wen W., Yamashita A., Asama H., (2015b). The sense of agency during continuous action: Performance is more important than action-feedback association. PLoS One, 10(4), e0125226, doi:10.1371/journal.pone.0125226.
Westwood D. A., Goodale M. A. (2011). Converging evidence for diverging pathways: Neuropsychology and psychophysics tell the same story. Vision Research, 51, 804–811, doi:10.1016/j.visres.2010.10.014.
Whitney D. (2002). The influence of visual motion on perceived position. Trends in Cognitive Science, 6, 211–216, doi:10.1016/s1364-6613(02)01887-9.
Wobbrock J. O., Findlater L., Gergle D., Higgins J. J. (2011). The aligned rank transform for nonparametric factorial analyses using only anova procedures. Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems – CHI’11, 143–146, doi:10.1145/1978942.1978963.
Figure 1.
 
Experimental setup. The white arrows represent the motion directions of the cursor and the reference object, and were not actually displayed as experimental stimuli.
Figure 1.
 
Experimental setup. The white arrows represent the motion directions of the cursor and the reference object, and were not actually displayed as experimental stimuli.
Figure 2.
 
Position ycut of cropped noise for −7.0-, 0.0-, and 7.0-mm drift conditions. White circles denote cropped areas. Solid black lines denote Gaussian functions that define the vertical positions of the cropped noise (Equation 1).
Figure 2.
 
Position ycut of cropped noise for −7.0-, 0.0-, and 7.0-mm drift conditions. White circles denote cropped areas. Solid black lines denote Gaussian functions that define the vertical positions of the cropped noise (Equation 1).
Figure 3.
 
Results of Experiment 1. (A) Distribution of the estimate of Apath canceling the illusory curve of the cursor path as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values.
Figure 3.
 
Results of Experiment 1. (A) Distribution of the estimate of Apath canceling the illusory curve of the cursor path as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values.
Figure 4.
 
Results of Experiment 2. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values. (C) Bubble plot of the rating scores for the unintended force and those of the apparent curve of the cursor path. Marker size represents the number of participants. (D) Scatter plot of the averaged rating scores for the unintended force in Experiment 2 as a function of the absolute magnitude of the illusory curve in the cursor path in Experiment 1. Each marker represents each random noise drift condition.
Figure 4.
 
Results of Experiment 2. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Solid black and dashed gray lines denote median and mean values, respectively. (B) Results of multiple comparison tests. Color scale denotes Cohen’s d and asterisks denote Bonferroni-corrected p values. (C) Bubble plot of the rating scores for the unintended force and those of the apparent curve of the cursor path. Marker size represents the number of participants. (D) Scatter plot of the averaged rating scores for the unintended force in Experiment 2 as a function of the absolute magnitude of the illusory curve in the cursor path in Experiment 1. Each marker represents each random noise drift condition.
Figure 5.
 
Results of Experiment 3. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Gray and black boxes denote the rating scores in the lower visibility (gray ring) and higher visibility (black ring) conditions, respectively. Solid black lines on the gray boxes and solid white lines on the black boxes denote median values. Dashed gray lines denote mean values. (B) Bubble plot of the rating score for the unintended force and the rating score for the apparent curve of the cursor path. Marker size represents the number of participants.
Figure 5.
 
Results of Experiment 3. (A) Distribution of mean rating scores for the unintended force as a function of the amplitude of the noise drift. Gray and black boxes denote the rating scores in the lower visibility (gray ring) and higher visibility (black ring) conditions, respectively. Solid black lines on the gray boxes and solid white lines on the black boxes denote median values. Dashed gray lines denote mean values. (B) Bubble plot of the rating score for the unintended force and the rating score for the apparent curve of the cursor path. Marker size represents the number of participants.
Table 1.
 
Parameters in the practice session in Experiment 1.
Table 1.
 
Parameters in the practice session in Experiment 1.
Table 2.
 
Parameters in the practice session in Experiment 2.
Table 2.
 
Parameters in the practice session in Experiment 2.
Table 3.
 
Parameters in the practice session in Experiment 3.
Table 3.
 
Parameters in the practice session in Experiment 3.
Table 4.
 
Results of multiple comparison tests for the low visibility conditions.
Table 4.
 
Results of multiple comparison tests for the low visibility conditions.
Table 5.
 
Results of multiple comparison tests for the high visibility conditions.
Table 5.
 
Results of multiple comparison tests for the high visibility conditions.
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×