Cognition modulates action-to-perception transfer in ambiguous perception

Abstract

Can cognition penetrate action-to-perception transfer? Participants observed a structure-from-motion cylinder of ambiguous rotation direction. Beforehand, they experienced one of two mechanical models: An unambiguous cylinder was connected to a rod by either a belt (cylinder and rod rotating in the same direction) or by gears (both rotating in opposite directions). During ambiguous cylinder presentation, mechanics and rod were invisible, making both conditions visually identical. Observers inferred the rod's direction from their moment-by-moment subjective perceptual interpretation of the ambiguous cylinder. They reported the (hidden) rod's direction by rotating a manipulandum in either the same or the opposite direction. With respect to their effect on perceptual stability, the resulting match/nonmatch between perceived cylinder rotation and manipulandum rotation showed a significant interaction with the cognitive model they had previously been biased with. For the “belt” model, congruency between cylinder perception and manual action is induced by same-direction report. Here, we found that same-direction movement stabilized the perceived motion direction, replicating a known congruency effect. For the “gear” model, congruency between perception and action is—in contrast—induced by opposite-direction report. Here, no effect of perception-action congruency was found: Perceptual congruency and cognitive model nullified each other. Hence, an observer's internal model of a machine's operation guides action-to-perception transfer.

Introduction

Actions typically have perceptual consequences. Moving one's hand results in changes of the location of the retinal image of the hand, pushing a mouse forward makes a pointer go up, pulling a control stick backward makes a plane climb, pulling a cord down opens the blinds, turning a key releases a lock, turning a screw pushes it forward or backward, etc. These mappings from action to outcome are vastly different, sometimes even conflicting, yet they appear nearly self-evident to us. Apparently, we have internalized models of the complex mappings between actions and their effects and can recruit them in a context-specific manner. In the present study, we ask whether these cognitive models of action consequences penetrate into perception itself. 

Besides affecting perception through changing the external world, action may also directly impact internal perceptual representations. Practicing a movement leads to improved visual discrimination of the same movement (Casile & Giese, 2006), action and the perception of action may rely on the same primitives (Mataric, 2000), and as humans, we are all “experts” on biological motion perception (Troje, 2008, 2013). The notion of shared action-perception representations has been formalized as the theory of common coding (Prinz, 1997) and extended into the theory of event coding (Müsseler, 1999). 

Ambiguous stimuli present an excellent means of isolating direct effects of action on perception from effects that are mediated through changes in the outside world. For example, when two identical disks move across the screen on the same trajectory but in the opposite direction (Metzger, 1934), the direction of a concurrently performed hand action biased the percept to either the two disks moving across or bouncing off one another (Mitsumatsu, 2009). Wohlschläger (2000) demonstrated that planning or executing a hand movement biased a rotating ambiguous motion display in the direction of manual rotation. Similarly, the perceptual interpretation of an ambiguous (bistable) rotating cylinder was stabilized, when viewers reported their perceived direction with congruent manual rotation (Beets et al., 2010). Comparable results were found when instead of visually ambiguous displays, binocular rivalry (Maruya, Yang, & Blake, 2007) or unambiguous stimuli with high perceptual uncertainty (Keetels & Stekelenburg, 2014) were used. These studies describe a congruency effect, whereby a match between action and perception (e.g., rotation in the same direction) leads to increased perceptual stability, as compared to an incongruent relationship (rotation in the opposite direction). 

The results on ambiguous stimuli provide evidence that action control and action perception can be coupled bidirectionally, even if the action has no consequence in the outside world (the stimulus remains unchanged, only its perception reverses). This is closely linked to the core notion of the ideomotor theory (Greenwald, 1970), common coding (Prinz, 1997), and the theory of event coding (Müsseler, 1999; Hommel, Müsseler, Aschersleben, & Prinz, 2001) that posit joint representations of perception and action. These joint representations may be acquired through experience, as the execution of an action yields a specific perception. In the absence of unambiguous perceptual evidence, action control may therefore bias perception towards an interpretation that is most consistent with the concurrently performed action. More specifically, the action may generate a forward model (Wolpert & Miall, 1996) of the action's expected perceptual consequences, and ambiguity is resolved to the perceptual interpretation that most closely matches the model. 

All of the aforementioned theories, as well as their later modifications (see e.g., Schütz-Bosbach & Prinz, 2007; Zwickel & Prinz, 2012) leave the mechanisms of the action-to-perception transfer comparably unspecified. They do not make any theoretical assumptions or predictions as to where the shared representations should be located in the processing hierarchy. In the context of rivalry, some studies, like Wohlschläger (2000) or Beets et al. (2010), show results that suggest a cognitive involvement, where the perceptual task or action goal has a crucial role in how the transfer effect takes place. In contrast, studies employing rivalry paradigms that can distinguish between conscious and unconscious effects of the action-related visual stimulus (Maruya et al., 2007; Veto, Schütz, & Einhäuser, 2018) demonstrate that effects of action on perception can also take place outside of awareness. This suggests a low-level source of the effect and leaves open the possibility that the action-to-perception transfer is caused by direct information flow between the two domains. In sum, the available experimental evidence offers diverging evidence as for the level of processing hierarchy and where the shared representations are located. It seems likely that shared sensory-motor representations can occur at multiple levels, of which some are under cognitive control. Hence, we ask whether a shared sensory-motor representation can be modified by the cognitive model of the expected perception-action relation. 

There is, however, a major challenge when addressing the effect of cognitive models on perception-action coupling, as two effects are necessarily confounded in most paradigms: (a) the effect of coupling between visible motion and action, and (b) the effect of coupling between the internal model of motion and action. Here, we separate these two effects to overcome the implicit assumption that cognitive model and perception are closely matched, and to assess the impact of cognition on action-to-perception transfer. As in Beets et al. (2010), we used an ambiguous structure-from-motion cylinder. In our experiment, we biased observers with one of two possible models of the particular mechanics that link the action to the observed visual consequence. We then tested whether the induced internal model modulated the effect of action on perception. Unlike in previous studies, however, observers did not report the perceived direction of the ambiguous cylinder itself, but of a visual representation of the manipulandum lever used to report the percept. Observers were taught that the ambiguous cylinder and the lever were either coupled through a belt or through gears (Figure 1a). This results in four (2 × 2) conditions (Table 1): the internal model (levels: “gear”, “belt”) and the match between perceived cylinder rotation and manipulandum rotation (levels: “congruent”, “incongruent”). 

Figure 1

 

Stimuli, percept, and task. (a) First, in separate blocks (Blocks 1 and 6, see Table 2), participants were introduced to the mechanical model (“belt” or “gear” layout). For 30 s, they controlled the displayed motion with the manipulandum. Then, 20 s of unambiguous motion followed (the cylinder and mechanical model rotated with occasional switches in direction), where observers had to report the rotation of the red lever (the red rod attached to the wheel) in accordance with the subsequent experimental block (“same direction instruction” or “opposite direction instruction”). For the last 20 s of training, the red lever disappeared and the mechanics was covered by a virtual desk, while the task remained unchanged. (b) All test blocks showed the same, ambiguous, motion cylinder for 3 min each. (c) Two possible perceptual interpretations of the test stimulus (clockwise and counterclockwise). Participants had to respond to the imagined motion of the red lever, as it related to their current percept. (d) Instruction (manipulandum rotation in the same or opposite direction as that of the red lever in the mental model). Note that in the “belt” condition, the same/opposite direction instruction leads to congruency/incongruency between perceived and performed rotation, while this relationship is reversed in the “gear” condition.

Table 1

 

Conditions. Notes: There are four (2 × 2) experimental conditions, defined by the factors internal model (belt, gear) and match (congruent, incongruent). Note that match and instruction in the belt and gear conditions are inversely related. Depending on whether perceptual congruency or internal model dominates, different predictions on perceptual (de)stabilization result (right columns).

 

The two alternative hypotheses result in distinct predictions. If the congruency between perceived motion and executed movement determines the percept, perceptual stability should be independent of the internal model: Congruency between movement and motion would yield perceptual stabilization, incongruency destabilization (Figure 2, left). Such results would be in line with the finding that the coupling between domains can take effect also outside of awareness (Maruya et al., 2007; Veto et al., 2018), and thus would further suggest a low-level source of action-to-perception transfer. If, however, the transfer depends on the internal model, the congruency effect should reverse in the gear condition (Figure 2, right); that is, we predict an interaction between the internal model and match factors. This would suggest that the action-to-perception transfer is based on the internal representation of movement, with closer connections to the internally simulated consequences of action than to the visually observed motion. Even if the effect does not reverse completely, any interaction between internal model and perception-action match would point to cognitive penetration of action-perception coupling. 

Figure 2

 

Hypotheses. Expected effects of internal model and match (between percept and action) on perceptual stability, if action-perception coupling is not under cognitive influence (left) or dominated by the cognitive model (right).

Methods

Participants

Thirty-two naive participants (15 males, 17 females; 25 ± 4.6 years; four left- and 28 right-handed) were included in the analysis. In one additional observer, no reversal of their rotation occurred in several blocks; this was detected when visually inspecting data quality, and the data of this observer was excluded prior to any further analysis. One further observer was assigned to a wrong group by technical error, which was realized during the experiment and their data was not analyzed or inspected any further. Procedures conformed to the Declaration of Helsinki and were approved by the Ethikkommission FB04 of Philipps-University Marburg (#2011-04K). Participants gave written informed consent prior to their participation. All participants had normal or corrected-to-normal vision. 

Setup and stimuli

Stimuli were generated using Unity3D (Unity Technologies, San Francisco, CA) and MATLAB (Mathworks, Natick, MA) with the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997), and presented on an Asus GL502 laptop computer (screen resolution: 1,920 × 1,080 at 60 Hz; viewing distance: 73 cm). Manual responses were recorded by a custom-built manipulandum device (Figure 1d shows an image of the device), measuring the angular position of the rotating handle via a Kübler Sendix 5020 incremental rotary encoder (Fritz Kübler GmbH, Villingen-Schwenningen, Germany). For all participants, the manipulandum was placed on the right side of the chair. 

For training blocks, stimuli were rendered with a perspective camera and other depth-cues present. The three-dimensional model of the cylinder consisted of small spheres placed at equal distances from a vertical axis, with randomly defined vertical and angular positions relative to the axis. The total size of the display extended 14.4 × 11.7 degrees of visual angle, with the diameter of each dot being 0.08°. The mechanical model consisted of either two wheels connected by a belt, or two adjacent cogwheels. The wheels moved according to the type of connection, that is, same direction in the belt condition and opposite direction in the gear condition. One of the wheels was placed directly below the cylinder and always moved together with it, as if they were fastened together. This wheel, as well as the cylinder, was shown in the center of the screen. The second wheel was to the right, with a vertical red rod attached to the top side (resembling the handle of the manipulandum; see Figure 1a). 

For test blocks, the same cylinder object was depicted in the center (6.1° × 12.7°) as an orthographic projection, without depth-cues or the attached mechanical model (Figure 1b). This way, the direction of rotation was completely ambiguous and up to the perceptual interpretation of the viewer. The spheres of the cylinder were shown in a homogeneous color (appearing as two-dimensional dots) and their size did not change along their movement trajectory. Thus, the front and rear surfaces of the cylinder were identical and showed no cues of occlusion. However, due to the dynamics of the dot movements, this cylinder formation is consistently perceived as a three-dimensional rotating object (structure-from-motion), where the apparent direction of rotation is ambiguous and its perception alternates (see e.g., Beets et al., 2010). 

Procedure

For each participant, the experiment consisted of two halves, one with the belt and the other with the gear stimulus condition (order counterbalanced between participants). Each half of the whole experiment started with a training block that introduced the stimulus and mechanical model of the applicable condition, followed by four test blocks with the ambiguous stimulus (Table 2). 

Table 2

 

Design matrix. Notes: Within a sequence of test blocks, order of reporting conditions follows an ABBA pattern (either SOOS or OSSO; S = same direction instruction; O = opposite direction instruction). The starting of the sequence and all other variables of the design were counterbalanced between participants, leading to a total of eight possible block order combinations; that is, each block order (I–VIII) was assigned to four of the 32 observers.

 

The training blocks were designed to gradually introduce model and task to the participant. In the first 30 s, the movement of the unambiguous cylinder and the attached mechanical model were directly connected to the manipulandum lever. Participants were instructed to move the lever as they wished and to observe the mechanical workings of the model. Then, for a 20-s interval, the model rotated at a constant velocity, changing direction every 6 ± 2 s. Participants had to either mimic the movement of the red rod on the attached wheel, or rotate in the opposite direction (according to what the instruction would be in the subsequent test block). In the last 20 s of the training block, participants continued with their previous task but the red rod disappeared, and the mechanics were occluded by a virtual desk. This way, the movement of the cylinder was still unambiguous, but the task of the participant was already identical to what they would do in the subsequent test block. To make certain that the correct response was practiced, a salient red rod appeared directly to the left of the cylinder, when the response direction was incorrect. Furthermore, the experimenter was also present during the training block and verified that by the end of the instruction, all participants understood the current task. 

Test blocks always showed the ambiguous stimulus (Figure 1b), moving at a constant velocity (90°/s). Depending on the condition of the given block, participants had to move the manipulandum lever in the same or opposite direction as the red lever on the mechanics (as seen in the training block) would rotate (Figure 1c and d). Test blocks lasted 3 min each. Before each block, the starting position of the manipulandum lever was set to the 12 o'clock position. 

The order of training stimulus (belt or gear mechanics) between the two halves of the experiment, the order of test block instructions within one half of the experiment (same or opposite direction; always in an ABBA order), as well as the order of test block instructions between the two halves of the experiment were counterbalanced between participants (Table 2). 

Analysis

Perceptual stability is operationalized as the median duration for which a percept (of either rotation direction) was perceived. Specifically, manipulandum rotation velocity data were segmented into periods of rotation in one direction or the other, as well as periods with no movement (no change of position for at least two subsequent sample points). For each observer and condition, perceptual stability was defined as the median duration of all nonzero velocity segments. 

Comparisons between conditions were made using a within-subjects repeated measures analysis of variance, with internal model (belt or gear) and match (congruent and incongruent, as in the relation between stimulus and action) factors. Effects were considered significant at a 0.05 alpha level, while a Bonferroni-adjusted alpha-level of 0.05 / 4 = 0.0125 was used for post hoc t tests. 

All reported tests are two-sided; MATLAB (R2015a) was used for data processing, SPSS (version 24) for statistical analysis. 

Results

In all experimental conditions, observers viewed an ambiguous rotating cylinder (Figure 1b) and reported perceived spinning direction by rotating a manipulandum lever either in the same or in the opposite direction as the lever of the assumed mechanism would move (Figure 1c and d). Importantly, the test blocks were visually exactly identical for all experimental conditions. 

We measured perceptual stability and tested whether the internal model (gear vs. belt) and match (congruent vs. incongruent) factors influenced this measure. We found no significant main effect of either factor (internal model: F[1, 31] = 0.517, η_p² = 0.016, p = 0.478; match: F[1, 31] = 2.697, η_p² = 0.080, p = 0.111), while the two factors showed a significant interaction, F(1, 31) = 4.763, η_p² = 0.133, p = 0.037. Paired samples two-sided t tests revealed that the congruency effect was significant only in the belt condition (perceptual stability: mean_{Belt-Congruent} = 4.40 s; mean_{Belt-Incongruent} = 3.24 s; t[31] = 2.759, p = 0.010), but not in the gear condition (mean_{Gear-Congruent} = 3.39 s; mean_{Gear-Incongruent} = 3.67 s; t[31] = 0.661, p = 0.513; Figure 3). 

Figure 3

 

Results. Perceptual stability for each condition, averaged across participants (N = 32; mean perceptual stability across observers, where perceptual stability for each observer is the median duration for which a percept was perceived). Error bars show standard errors of the mean.

Discussion

The interaction between the internal model and match factors revealed that the cognitive model of the coupling between an action and its observable effect significantly influenced the action-to-perception transfer. Results from the belt condition replicated the known congruency effect (Beets et al., 2010), while the lack of effect in the gear condition (in the presence of an interaction) showed that the influence of the assumed mechanical model counteracted the natural congruency bias. Thus, cognition plays a significant role in action-to-perception transfer, while it is not the sole source of the effect. 

In the framework of the common-coding theory (Prinz, 1997), our results can be interpreted as evidence that the shared representations between perception and action occur on a cognitively accessible level of processing. This is in line with the observation that action-perception transfer can depend on the relevance of an action for the perceptual task (Beets et al., 2010). A direct influence of the cognitive model on action-to-perception transfer might also be of adaptive value in real-life situations, in particular when tools similar to the one used here are involved: Evoking a cognitive model allows better predictions of an action's consequences and may therefore result in better performance or quicker learning of a complex manual task (Lupyan, 2015). Nonetheless, our results do not exclude that, on some level, shared action-perception representations exist that are under less cognitive control and form independently of awareness (Maruya et al., 2007; Veto et al., 2018). Such effects might play a role in the present experiment, too, possibly explaining why the reversal of the congruency effect in the gear condition was incomplete. In addition, it is open, whether levels of perceptual processing exist that are entirely impenetrable to cognition (Pylyshyn, 1999), but still accessible by action. In a representation-based framework, the results on action-perception transfer taken together necessitate different representational levels, of which only some being modified by executive functions, awareness, or cognition. Results of the present study, as well as the divergent findings of earlier studies on the necessity of task relevance (e.g., Beets et al., 2010) and on the possibility of an action-to-perception transfer outside of awareness (e.g., Maruya et al., 2007), show that no single mechanism at any given stage of processing can account for all the observed phenomena. 

A complementary view posits that the quality of perception arises from expected relation between our actions and their sensory effects (O'Regan & Noë, 2001). Perception, cognition, and action then become intimately related through the model that is generated by observing the sensory consequences of an action. In this case, perceptual qualities and the cognitive model can be viewed as consequences of the embodied action-perception relation. This is consistent with a recently proposed action-oriented framework, in which perception and cognition are formed together, with action being the key organizing force behind both (Engel, Maye, Kurthen, & König, 2013). In a simple system like the gear/belt mechanics, it would appear conceivable that instruction led the observers to simply learn the coupling from action to perception without forming a cognitive model. However, we deliberately chose an experimental design that reversed the congruency instruction without re-exposing the observers to the action-perception coupling; instead, we exposed each observer to their second mechanical model only after he or she had completed all blocks with the first model (order of mechanics balanced across observers; Table 2). Observers therefore needed to apply their internal mechanical model to reverse the instruction without practicing the action-perception contingency. This makes it likely that observers indeed have formed a cognitive model during instruction, which they consistently applied until a different model became evident through a new, externally available, action-perception contingency. 

Acknowledgments

Our study was supported by the German Research Foundation through the projects “CRC/Transregio 135 – Cardinal mechanisms of perception: Prediction, valuation, categorization” and “International Research Training Group 1901 – The Brain in Action.” NFT's contribution was funded by a Natural Sciences and Engineering Research Council Discovery Grant. 

Commercial relationships: none. 

Corresponding author: Peter Veto. 

Email: . 

Address: Institute of Physics - Physics of Cognition Group, Chemnitz University of Technology, Chemnitz, Germany.