Action adaptation during natural unfolding social scenes influences action recognition and inferences made about actor beliefs

Abstract

When observing another individual's actions, we can both recognize their actions and infer their beliefs concerning the physical and social environment. The extent to which visual adaptation influences action recognition and conceptually later stages of processing involved in deriving the belief state of the actor remains unknown. To explore this we used virtual reality (life-size photorealistic actors presented in stereoscopic three dimensions) to see how visual adaptation influences the perception of individuals in naturally unfolding social scenes at increasingly higher levels of action understanding. We presented scenes in which one actor picked up boxes (of varying number and weight), after which a second actor picked up a single box. Adaptation to the first actor's behavior systematically changed perception of the second actor. Aftereffects increased with the duration of the first actor's behavior, declined exponentially over time, and were independent of view direction. Inferences about the second actor's expectation of box weight were also distorted by adaptation to the first actor. Distortions in action recognition and actor expectations did not, however, extend across different actions, indicating that adaptation is not acting at an action-independent abstract level but rather at an action-dependent level. We conclude that although adaptation influences more complex inferences about belief states of individuals, this is likely to be a result of adaptation at an earlier action recognition stage rather than adaptation operating at a higher, more abstract level in mentalizing or simulation systems.

Introduction

The actions of other individuals can be understood on multiple levels. We can detect and recognize another individual's actions, understand the goal of their actions, and infer their intentions, beliefs, and desires from just action information. Our ability to make sense of the actions of other individuals is critical for operating successfully in a complex social environment (Frith & Frith, 2012). It is important, therefore, that we have a clear understanding of how we process action information in natural social scenes. 

Analysis of actions at increasingly higher levels of cognitive understanding occurs in a hierarchy of processing stages across a network of brain areas (Decety & Grezes, 1999; Giese & Poggio, 2003; Hamilton & Grafton, 2006; Lestou, Pollick, & Kourtzi, 2008; Van Overwalle & Baetens, 2009; Wurm & Lingnau, 2015), and our evaluation of complex social scenes involves simultaneous processing at these multiple stages. Following category-nonspecific visual analysis in earlier regions of the visual system, the first stage in the bodily action processing network is thought to be within the superior temporal sulcus (STS). Here, neurons are found that respond selectively to the sight of specific actions (Chitty, Perrett, Mistlin, & Potter, 1985; Oram & Perrett, 1996; Perrett et al., 1985, 1989), and this region in humans is thought to underlie the recognition of actions (Allison, Puce, & McCarthy, 2000; Grossman et al., 2000; Puce, Allison, Bentin, Gore, & McCarthy, 1998; Puce & Perrett, 2003). In addition, the STS may represent elementary forms of action intentions (Jellema, Baker, Wicker, & Perrett, 2000): STS neurons have been shown to be sensitive to actions embedded in specific action sequences and to code for actions most likely to happen in the immediate future given the immediate perceptual history (Jellema & Perrett, 2003; Perrett, Xiao, Barraclough, Keysers, & Oram, 2009), which puts it in a position to represent the goal directedness of actions (Jellema & Perrett, 2005). However, a fuller understanding of actor intentions and beliefs is likely to be derived during subsequent stages (Hamilton & Grafton, 2006; Van Overwalle & Baetens, 2009), within either the inferior parietal cortex (e.g., Becchio et al., 2012; Fogassi et al., 2005; Van Overwalle, 2009) or the frontal motor regions (e.g., Becchio et al., 2012; Iacoboni et al., 2005) or through interaction between these regions. 

Adaptation is a ubiquitous sensory processing mechanism, but it is unclear how it influences the processing of action information at these different levels of understanding. In psychophysics experiments, adaptation consists of prolonged exposure to a stimulus with closely defined parameters, which is thought to result in a suppression of neural mechanisms that underlie the coding of the specific characteristics of the stimulus. Perceptual judgments of test stimuli presented following adaptation can be profoundly biased, and the character of these aftereffects can help illustrate the nature of the underlying neural mechanisms. Recently it has become clear that adaptation not only influences early visual processing but also has an effect on increasingly higher and more abstract representations, including face identity (Leopold, O'Toole, Vetter, & Blanz, 2001), face emotion (Benton et al., 2007), face trustworthiness (Keefe, Dzhelyova, Perrett, & Barraclough, 2013), action recognition (Barraclough & Jellema, 2011; Barraclough, Keith, Xiao, Oram, & Perrett, 2009; de la Rosa, Streuber, Giese, Bulthoff, & Curio, 2014), implied actions (Lorteije et al., 2007), action gender (Troje, Sadr, Geyer, & Nakayama, 2006), action emotion (Roether, Omlor, Christensen, & Giese, 2009; Wincenciak, Ingham, Jellema, & Barraclough, 2016), and even multimodal representations of actor identity and emotion (e.g., Hills, Elward, & Lewis, 2010; Konkle, Wang, Hayward, & Moore, 2009; Pye & Bestelmeyer, 2015; Skuk & Schweinberger, 2013; Zaske, Schweinberger, & Kawahara, 2010). In this study we set out to investigate the extent to which perceptual adaptation can influence increasingly higher levels of action understanding. 

Previous research has indicated that visual adaptation can influence action recognition, but it is unknown whether the more complex inferences we make about the internal mental state of the actors themselves can also be influenced. Visual adaptation to hand actions (Barraclough et al., 2009) and whole-body actions (Barraclough & Jellema, 2011) influences the subsequent recognition of hand and whole-body actions, respectively. These effects are likely to be due to the selective adaptation of single neurons within the STS that are responsive to specific actions (Kuravi, Caggiano, Giese, & Vogels, 2016). Although adaptation paradigms have been used to demonstrate that action aftereffects can be modulated by social context (de la Rosa et al., 2014), these effects are probably due to a top-down modulation of action recognition mechanisms. Adaptation can influence the perception of more abstract concepts conveyed by actors, such as gender (Roether et al., 2009) and emotion (Troje et al., 2006). However, both of these studies used point light “biological motion” stimuli, and it is not clear whether the emotional action adaptation observed by Roether et al. (2009) is due to adaptation to the perceived emotional state of the actors portrayed by the point light stimuli or whether these results can be explained by a more simple adaptation to the complex motion patterns of the stimuli or the kinematics of the biological motion figure. In this study, we therefore tested the effect of visual adaptation on action recognition mechanisms, but we also tested the effect of adaptation on a conceptually later stage of processing involved in deriving the belief state of the actor. To do this, we adapted a paradigm used previously to demonstrate action adaptation (Barraclough et al., 2009) while using stimuli similar to those of Grèzes et al. (2004), who examined the neural mechanisms underlying both action perception and actors' beliefs about their physical environment. In the Grèzes et al. (2004) study, actors picked boxes up off the floor and exhibited different variations in action kinematics depending on the weight of the box. During the observation of such stimuli, our action recognition mechanisms are able to distinguish such subtle differences in kinematics (Grèzes, Frith, & Passingham, 2004; Runeson & Frykholm, 1981, 1983). Furthermore, when actors had correct or false expectations (or beliefs) about the weights of the boxes they were about to lift, they executed their actions with different kinematics. This information is available to observers, who can then use it to extract the actors' beliefs, or expectations, about the box weights (see Grèzes et al., 2004; Runeson & Frykholm, 1983). In our study, to assess how adaptation influenced action recognition judgements, we examined the effect of adaptation to actors picking up boxes of different weights on the judgment of the weight of boxes picked up by subsequent actors (cf. Barraclough et al., 2009). To assess how adaptation influenced inferences about the belief state of the actor, we examined how adaptation to actors with correct—and incorrect—expectations of the weights of boxes influenced judgements made about subsequent actors' expectations of box weight. 

Previous adaptation studies have used either unnaturalistic, oversimplified stimuli presented on small screens or contrived experimental designs. This can lead to difficulty in determining whether these previous results can be generalized to the natural social environments we navigate. Experiments have been conducted this way for good reason. Stimuli are simplified (e.g., biological motion stimuli, cropped videos of body parts, isolated actors presented out of context) in order to isolate and study the processing of specific features of the actions represented. In addition, typical adaptation experiments with action stimuli (and face stimuli) have followed established methods for the assessment of mechanisms underlying the processing of more simple stimuli (e.g., oriented lines, moving dot fields). A typical trial might involve a presentation of an adapting stimulus on a blank screen for a period of time, its removal from the screen for a period of time, and then a brief or flashed presentation of a subsequent test stimulus on the screen again. In contrast to these previous studies, we sought to measure aftereffects under naturalistic viewing conditions. In the experiments we describe here, participants viewed two actors in the same scene; one actor was assigned as the “adapting” actor and the other as the “test” actor. Scenes with both actors in the room unfolded in a naturalistic fashion: Actions were performed in sequences without the scene containing the actors instantly appearing and disappearing from the screen, as for all previous experiments. Furthermore, given that some previous evidence (e.g., Snow et al., 2011) has indicated that adaptation might occur differently after observation of two-dimensional (2D) objects rendered on a small screen compared with the observation of real objects, we wanted to assess whether the mode of presentation influenced whole-body aftereffects. We therefore examined adaptation under highly naturalistic viewing conditions using virtual reality to present three-dimensional (3D) life-size orthostereoscopic actors as well as under more typical conditions experienced in the psychophysics lab, where the actors were presented in 2D on a small screen. 

We examined the effect of adaptation on action recognition and inferences of actor belief state in a series of six experiments. In Experiment 1 we assessed whether the way that action scenes were viewed influenced action recognition aftereffects by presenting action scenes in life-size orthostereoscopic 3D, life-size 2D, and small-scale 2D formats. Aftereffects resulting from adaptation of stimuli processed in the early visual cortex show sensitivity to changes in low-level characteristics such as stimulus position, size, and orientation. In contrast, higher level aftereffects, resulting from adaptation at later stages of processing, show relative insensitivity to changes in lower level stimulus characteristics. For example, whole-body aftereffects are observed irrespective of the viewpoint from which adapting and test actors are viewed (Barraclough & Jellema, 2011). In order to evaluate whether action recognition aftereffects were due to adaptation of lower or higher level processing mechanisms, in Experiment 2 we examined whether action aftereffects arose from adaptation to view-dependent or view-independent mechanisms by measuring aftereffects when the adapting and test actors were seen from the same or different perspectives. Visual aftereffects typically show characteristic build-up in strength with increasing exposure to the adapting stimulus and decay over time (e.g., Barraclough, Ingham, & Page, 2012; Barraclough et al., 2009; Hershenson, 1989; Leopold, Rhodes, Muller, & Jeffery, 2005; Rhodes, Jeffery, Clifford, & Leopold, 2007). To test whether action recognition aftereffects would show dynamics similar to those seen previously, we varied both the number of adapting actions executed (Experiment 3) and the duration of the interval between the actions executed by the adapting and test actors (Experiment 4). In Experiments 5 and 6 we evaluated whether visual adaptation influenced inferences about the belief state of individuals in the social scene. In Experiment 5 we investigated whether visual adaptation to the perceived belief state of one actor generated perceptual aftereffects that distorted the perceived belief state of another actor in the same scene. In addition, we tested whether the way that action scenes were viewed influenced aftereffects by presenting scenes in life-size orthostereoscopic 3D and small-scale 2D formats. Finally, in Experiment 6 we tested whether distortions in the perception of an individual's belief state following visual adaptation were due to adaptation at a high level of understanding, at which the belief state of individuals is derived. 

General method

Participants

Participants were either University of Hull or University of York students or staff and either were paid or volunteered to take part in experiments. All had normal or corrected-to-normal vision, and stereoacuity was better than 40 arcsec for 3D stimulus presentation. Experiments were approved by the ethics committee of the Department of Psychology at the University of Hull and the University of York and were performed in accordance with the ethical standards laid down in the 1990 Declaration of Helsinki. Written consent to take part in the studies was obtained from all participants. All participants were naive to the purposes of the study. 

Stimuli

Stimuli were filmed with a custom 3D camera built by combining two TM900 digital cameras (Panasonic, Kadoma, Japan; each filming at 1920 × 1080 pixels and 50 frames/s progressive scan) mounted on a solid aluminum plate and set to a 66-mm parallel interaxial distance. During filming the camera was placed at a height of 1.22 m and a distance of 4.1 m from the virtual screen plane in order to ensure the best possible correspondence between the camera viewpoint of the scene and final display to the participants. All objects were viewed at a depth behind the screen plane (i.e., positive parallax). 

Stimulus set 1: Box-lifting stimuli; actors with correct expectations of box weights

Two males and two females were filmed lifting boxes onto a table from the floor (see Figure 1a). Actors performed the lifts when facing either the left table or the right table, and all actions were filmed when the actor faced the respective table from both the left and right sides of the table. For all actions, actors were always informed about the weight of the boxes prior to lifting. We refer to actions performed at the left table as adapting actions and actions performed at the right table as test actions. Adapting actions were lifts of 2- and 18-kg boxes or no action (standing still). Boxes (one, two, four, or eight) were arranged in a line (along the depth dimension), and actions were always directed toward the farthest box first. Each lift took 4.5 s to execute while guided by a timing beep. Test actions were lifts of 6-, 8-, 10-, 12-, and 14-kg boxes. Boxes were always positioned equidistant from the front and back of the table (along the depth dimension) such that the mean position of the adapting and test actors in depth was the same across all actions. 

Figure 1

 

Schematic of the experimental designs. Each panel shows cropped individual stills at different time points in a continuous movie presented during an example trial. (a) Experimental design for Experiments 1 through 4. The adapting actor was on the left and test actors were on the right. Durations of the adapting phase and durations of the interstimulus interval varied across experiments. During Experiment 2 (not illustrated), adapting and test actors executed actions on either side of the static table in order to change the view of the action. (b) Experimental design for Experiment 6. The adapting actor was on the right and test actors were on the left. Experiment 5 (not illustrated) was similar; however, the adapting action was a box-lifting action.

A pilot experiment was conducted to examine whether the kinematics of the lifting actions allowed observers to discriminate the weight of the box being lifted. Observers' ratings of box weight were found to correlate highly with the physical weight of the box (slope = 0.33, R² = 0.98). For adapting actions during Experiments 1 through 4 we selected the actions executed by the actor (female) whose lifts of the 2- and 18-kg boxes were perceived as the lightest and heaviest, respectively (slope = 0.48). The actions executed by the remaining three actors (two males, one female) were used as test actions, such that the identity of the adapting and test actors was always different. 

Stimulus set 2: Box-lifting stimuli; actors with incorrect expectations of box weights

Two male and three female actors were filmed lifting boxes onto a table from the floor. The actors were positioned to the left of either the left or right table and were viewed from behind at 45° to the camera in order to occlude all facial information. During filming, actors were deceived such that they had incorrect expectations (beliefs) about the weight of the box they were to lift (cf. Grèzes et al., 2004). Adapting actions and test actions were filmed while they were performed at both the left and right tables. For adapting stimuli, actors executed 20 box lifts at the left table. On the majority of lifts (18/20) the actor had the correct expectation that the box weighed 10 kg; however, two boxes had unexpected weights (2 and 18 kg; i.e., the actor overestimated or underestimated the weight of the box, respectively). For test stimuli, actors completed 40 box lifts at the right table. On the majority of lifts (36/40) the actor had the correct expectation that the box weighed 10 kg; however, four boxes had different weights (6, 8, 12, and 14 kg; i.e., the actor had differing degrees of over- and underestimation of the box weight). 

Pilot testing indicated that observers could discriminate test actor expectation (from large overestimations through true estimations to large underestimations) from the movies of test actions (correlation between actor expectation and observers' rating, slope = 0.36, r² = 0.96). The actor who conveyed the greatest degree of under- or overestimation of the box weights was selected as the adapting actor, and the remaining four actors were used as test actors. 

Stimulus set 3: Box-lifting and lever-pulling stimuli; actors with incorrect expectations of force required

Two female actors were filmed lifting boxes onto a table from the floor and pulling a lever emerging from the surface of a cabinet. The actors were viewed from behind at 45° to the camera in order to occlude all facial information. To generate stimuli in which actors had a true or false expectation about the force required to execute the action (lift or pull), we organized separate blocks of filming to manipulate actor expectation. Adapting actions were lever pulls and were filmed with the actor on the right side of the room (see Figure 1b). Actors had a correct estimation of the force necessary to pull the lever (1 or 9 kg) during 90% of pulls; during 10% of pulls, the weight was unexpectedly changed to 5 kg to generate overestimates (during 9-kg pulls) and underestimates (during 1-kg pulls). Test actions were box lifts executed while facing the table on the left side of the room. Actors executed box lifts when they had the correct expectation of the box weight (2.5, 5, 7.5, 10, 12.5, 15, and 17.5 kg) during 90% of lifts. During 10% of lifts (once per weight) the weight was changed to 10 kg, resulting in differing degrees of over- and underestimation. Actors also completed lifts of 10-kg boxes in which the box weight was held constant. In total, actors completed 350 lifts, providing five instances for each level of under- or overestimation and 50 instances of the lift of the 10-kg box when 10 kg was expected. 

Although pull action weights (1–9 kg) were nominally smaller than lift action weights (2–18 kg), the force required to execute the actions was approximately similar. The lever in the cabinet was geared to increase the force required to execute the pull action. In addition, pulls were executed with one hand rather than two hands, making this action harder to execute than the two-handed box lift. 

Stimulus presentation

Stimuli were presented to participants in three ways: (a) life size with orthostereoscopic 3D, (b) life size in 2D, and (c) small scale in 2D. Life-size orthostereoscopic 3D and life-size 2D stimuli, subtending 4.57 m × 2.44 m (50° × 33°), were presented on a 5.33 m × 2.44 m (58° × 33°) rear-projection screen (two Mirage 2000 Christie Digital DLP projectors [Christie, Cypress, CA] were used, effective full-screen resolution = 2240 × 1024 pixels, 100-Hz refresh rate). Stimuli were viewed through active liquid-crystal display shutter glasses (nuVision 60 GX [McNaughton Inc., Beaverton, OR], triggered by StereoGraphics Infra-Red emitters [StereoGraphics, San Rafael, CA]). Participants sat such that their eyes were positioned at a height of 1.22 m and a distance of 4.1 m from the screen plane (to the cyclopean eye). Stimuli viewed from this position were orthostereoscopic such that the stereo images of the stimuli were viewed in real-world dimensions. Distortions in perceived depth induced in participants with interpupillary distances greater or less than the interaxial distance of the camera would be too small to be reliably detected based on normal stereoacuity (10 arcsec; Dodgson, 2004). For life-size 2D presentation, participants viewed only the left movie channel through the liquid-crystal display shutter glasses. Small-scale 2D stimuli for Experiment 1 were presented on a Philips 202P40 22-in. cathode ray tube monitor (1600 × 1200 pixels, 100-Hz refresh rate [Philips, Amsterdam, Netherlands]; stimuli were scaled such that they subtended 56° × 33° with a viewing distance of 43 cm). Small-scale 2D stimuli for other experiments were presented on a 24-in. thin-film transistor monitor (Acer GD245HQ, 1920 × 1080 pixels, 100-Hz refresh rate [Acer, New Taipei, Taiwan]; stimuli were scaled such that they subtended 49° × 29° at a viewing distance of 57 cm). 

Experimental procedures

A personal computer running MATLAB 2010a (MathWorks, Natick, MA) and Psychtoolbox (Brainard, 1997) controlled the experiments, displayed the stimuli, and recorded participant responses. A custom movie playback engine was used to display 3D movies. The details of the experimental procedures for each of the six experiments are described in the subsections below. 

Experiment 1: Adaptation to actions in naturalistic environments

Experiment 1 tested whether observers experienced visual aftereffects in recognizing an individual's actions when the actions were executed in a naturalistic social scene containing another person. In addition, we assessed whether the way that action scenes were viewed influenced action recognition aftereffects by presenting action scenes in life-size orthostereoscopic 3D, life-size 2D, and small-scale 2D formats. 

Method

Participants

Twenty four participants (13 females, 11 males; mean age = 22.5 years, SD = 3.9) took part in the experiment. 

Stimuli

Movies were made in which the actions of two individual (adapting and test) actors occurred in succession in a naturalistic fashion. In a typical adaptation experiment, the adapting stimulus is presented and then disappears from the screen, and then a test stimulus is presented (e.g., Barraclough & Jellema, 2011; de la Rosa et al., 2014; Troje et al., 2006). In contrast to these previous studies, both the adapting and test actors remained in the scene throughout the trial. Different stimuli from Set 1 were combined using video compositing (Vegas Pro 10, Sony, Tokyo, Japan) to construct new movies in which the adapting actor appeared on the left and a test actor appeared on the right in a natural-looking room scene. This process ensured that the whole sequence of actions appeared in a realistic fashion, with no intervals in the video stream or unnatural changes in either of the actors' behavior. However, the relationship between the adapting and test actions was tightly controlled, as in previous artificial presentation of stimulus sequences during adaptation experiments (e.g., Barraclough et al., 2012; Troje et al., 2006). 

Procedure

The experiment was a within-subject factorial design with three viewing conditions (life-size orthostereoscopic 3D, life-size 2D, and small-scale 2D) and three adaptation conditions (adapt heavy: lift 18 kg, Movie 1; adapt light: lift 2 kg, Movie 2; no adaptation, Movie 3). In order to measure shifts in action recognition with the presence of an adapting actor, it was necessary to ensure that participants believed they were viewing different test actors on different trials. With naturalistic actions executed by identifiable actors, each action execution can have idiosyncratic kinematics and thus be highly recognizable. We needed a large number of trials in our experimental design to get accurate measures of perception with and without adaptation, but we also wanted to ensure that unique action executions were not viewed by participants so often that they became familiar and participants realized that they were repeatedly viewing the same test action. Therefore, we used five test action conditions (lift 6, 8, 10, 12, and 14 kg) and three test actor identities (two males, one female). Test action condition and test actor identity were not conditions of interest and therefore were not analyzed. 

Each of the three viewing conditions was tested on separate days, and viewing condition was counterbalanced across participants. On each day, the no-adaptation condition was always tested first in a single block. Then, the adapt heavy and adapt light conditions were tested in separate subsequent blocks, where adaptation condition was counterbalanced across participants and viewing conditions. Within a block of testing, all other conditions (five test action conditions and three test actor identities) were randomized such that there were in total 15 trials/block. 

At the start of each adaptation block (adapt heavy, adapt light), an initial preadaptation video was displayed three times in which the adapting actor, alone in the room, lifted all eight boxes of the same adapting weight (24 lifts). Following a 1-s black screen, the first trial commenced. Each trial began with 2 s of adapting and test actor inactivity (to allow stereo fusion; Hoffman, Girshick, Akeley, & Banks, 2008). This was followed by the adapting actor lifting eight boxes onto the table, a 1-s inter-action interval (duration between the offset of the last adapting action and the onset of the test action) during which both actors stood still, the test actor lifting his or her box onto the table, and finally 0.5 s during which both actors stood still (see the lower sequence in Figure 1a). At the end of the trial the screen turned black, indicating that the participant had to respond during a 1.5-s interval. If no response was registered during the interval, an error beep was played and the trial was repeated randomly later in the block. 

The no-adaptation block began with both the adapting and test actors standing still in front of a single box for the same period of time it would have taken the adapting actors to execute the eight lift actions (38 s). The trial structure was identical to that in the adapting trials except that the adapting actor stood still throughout the entirety of each trial (see the upper sequence in Figure 1a). At the end of each trial, participants made their responses on a wireless numeric keyboard and indicated the weight of the box lifted by the test actor, choosing between 2, 4, 6, 8, 10, 12, 14, 16, and 18 kg. Breaks were given between each block of testing, and the experiment took approximately 45 min to complete. 

We calculated aftereffects for each of the three viewing conditions (life-size orthostereoscopic 3D, life-size 2D, and small-scale 2D) and each of the two adaptation conditions (adapt heavy, adapt light). For each participant and viewing condition, we subtracted the mean ratings of the weights of boxes lifted by the test actors for control trials from the mean ratings of the weights of the boxes lifted by the test actors following adapting actors lifting light or heavy boxes. This generated both “heavy” and “light” aftereffects for each viewing condition and participant. Positive values indicated that test actors appeared to lift heavier boxes, negative values indicated that test actors appeared to lift lighter boxes, and zero indicated no effect of the adapting actor. 

Results

Experiment 1 examined whether action recognition was distorted by the actions of another individual in a naturalistic social scene. This was examined for life-size orthostereoscopic 3D, life-size 2D, and small-scale 2D presentation formats. A 2 × 3 analysis of variance (ANOVA) with adaptation condition and viewing condition as within-subject factors showed that the actions executed by the adapting actor had a significant influence on the recognition of the actions executed by the test actors—magnitude of aftereffects, main effect of adaptation condition: F(1, 23) = 69.88, p < 0.001, η_p² = 0.75 (see Figure 2a). Test actors appeared to lift heavier boxes after the adapting actor lifting light boxes and appeared to lift lighter boxes after the adapting actor lifting heavy boxes. The presentation format did not significantly influence aftereffect magnitudes—main effect of viewing condition, F(2, 46) = 1.80, p = 0.177, η_p² = 0.07; interaction between adaptation condition and viewing condition, F(2, 46) = 0.43, p = 0.655, η_p² = 0.02—indicating that action recognition aftereffects measured in more traditional psychophysical laboratory environments are of equivalent magnitude to those measured during more naturalistic viewing conditions. The distortions in the perception of the actions of the test actors resulting from the actions executed by the adapting actors are large and consistent with repulsive action aftereffects observed following adaptation to hand actions (Barraclough et al., 2009; de la Rosa et al., 2014) and whole-body actions (Barraclough & Jellema, 2011). 

Figure 2

 

Action recognition aftereffects. Positive values indicate that test actors appeared to lift heavier boxes than they actually lifted, negative values indicated that test actors appeared to lift lighter boxes, and zero indicated no effect of the adapting actor. (a) Aftereffects observed with different viewing conditions (Experiment 1). (b) Effect of viewpoint of the adapting and test actors on aftereffect magnitudes (Experiment 2). (c) Increasing adaptation duration increases aftereffect magnitude (Experiment 3), which is best modeled by exponential functions (light aftereffect, R = 0.91; heavy aftereffect, R = 0.91), plotted. (d) Increasing the interval between adapting and test actions reduces aftereffect magnitude (Experiment 4), best modeled by an exponential decay function (light aftereffect, R = 0.91) and a logarithmic function (heavy aftereffect, R = 0.86), plotted. Error bars indicate standard error of the mean.

Experiment 2: View dependence of action aftereffects

Experiment 2 examined whether the action aftereffects observed in Experiment 1 resulted from adaptation to view-dependent or view-independent mechanisms. View-independent aftereffects would suggest adaptation of high-level representations. We measured action aftereffects when the adapting and test actions were seen from similar or different perspectives. 

Method

Participants

Seventeen participants (nine females, eight males; mean age = 22.0 years, SD = 2.4) took part in the experiment. (The number of participants was based on a power analysis of the aftereffects measured during Experiment 1.) 

Stimuli

Movies were made in which the actions of two individual (adapting and test) actors occurred in succession in a naturalistic fashion. Movies were similar to those used in Experiment 1. However, three viewpoint conditions were presented in which (a) adapting and test actors faced left (same view left), (b) adapting and test actors faced right (same view right), and (c) the adapting actor faced right and the test actor faced left (different view). 

Procedure

The experiment was a within-subject factorial design with three adaptation conditions (adapt heavy: lift 18 kg; adapt light: lift 2 kg; no adaptation) and three viewpoint conditions (same view left, same view right, different view). In addition, there were three test action conditions (lift 6, 10, and 14 kg) and two test actor identities (one male, one female). Test action condition and test actor identity were not conditions of interest and therefore were not analyzed. However, they were varied in order to ensure that participants viewed different actions during experimental testing. No-adaptation and adaptation conditions were presented in separate blocks of testing. 

No-adaptation control blocks were completed both before and after an adaptation block. Results from both blocks were averaged together to provide one measure of participants' ratings of each weight of box, each test actor identity, and each view of the test actors. Control blocks were conducted both before and after the adaptation block. There were 12 movies used during each control block: 2 test actors (male, female) × 3 test action conditions (6, 10, and 14 kg) × 2 test actor views (face left, face right). During each control block, each of the 12 movies was presented five times in a pseudorandom order. Test procedures were the same as for Experiment 1 no-adaptation blocks; however, the intertrial interval was 3 s, and trials began with the test actor standing still for 2 s followed by the action without the presence of any adapting actor in the room. 

Between the two control blocks was a single adaptation block. During the adaptation block 36 different movies were presented: 2 adaptation conditions (adapt heavy, adapt light) × 3 viewpoints (same view left, same view right, different view) × 2 test actors (male, female) × 3 test action conditions (6, 10, and 14 kg). Movies (trials) occurred once in a random order. All other test procedures were as for the adaptation blocks in Experiment 1 except no preadaptation was used, the intertrial interval was 3 s, heavy and light adapting conditions were interleaved in the same block rather than being separated into their own blocks, a break of 1 to 2 min was given to participants 18 trials into the block to help reduce participant fatigue, and all testing was performed with participants viewing movies presented on a 24-in. thin-film transistor screen (see Method). 

We calculated aftereffects for each of the three viewing conditions (same view left, same view right, different views) and each of the two adaptation conditions (adapt heavy, adapt light). We subtracted the mean ratings of the weights of boxes lifted by the test actors for control trials when they were facing left from the mean ratings of the weights of the boxes lifted by the test actors for adaptation trials when the test actors faced left (same view left, different views) in each adaptation condition to generate both heavy and light aftereffects. In addition, we subtracted the mean ratings of the weights of boxes lifted by the test actors for control trials when they were facing right from the mean ratings of the weights of the boxes lifted by the test actors for adaptation trials when the test actor faced right (same view right) in each adaptation condition to generate both a heavy and a light aftereffect. Positive values indicated that test actors appeared to lift heavier boxes, negative values indicated that test actors appeared to lift lighter boxes, and zero indicated no effect of the adapting actor. 

Results

A 2 × 3 within-subject ANOVA was used to test the differences between the magnitudes of aftereffects under the different adaptation conditions and viewpoint conditions (Figure 2b). Aftereffects generated when the adapting actor lifted heavy or light boxes were significantly different: main effect of adaptation condition, F(1, 16) = 53.61, p < 0.001, η_p² = 0.77. There was no main effect of viewpoint, F(2, 32) = 1.45, p = 0.25, η_p² = 0.08, nor was there a significant interaction between viewpoint and adaptation condition, F(2, 32) = 0.06, p = 0.94, η_p² = 0.00. Importantly, significant differences in light and heavy aftereffects were still observed irrespective of the direction the actors faced (ts > 4.77, ps < 0.001; see Figure 2b). In conclusion, these results show that the action recognition aftereffects are viewpoint independent (cf. Barraclough & Jellema, 2011) and thus result from changes to high-level action processing mechanisms. 

Experiment 3: Effect of action exposure on aftereffect magnitude

Increasing adaptation exposure typically increases the magnitude of aftereffects in a logarithmic fashion (e.g., Barraclough et al., 2009, 2012; Hershenson, 1989, 1993; Leopold et al., 2005). In Experiment 3, we therefore examined whether the action recognition aftereffects observed in Experiments 1 and 2 were dependent on the duration of exposure to the adapting actor as he or she lifted increasing numbers of boxes. 

Method

Participants

Seventeen participants (11 females, six males; mean age = 22.3 years, SD = 3.8) took part in the experiment. (The number of participants was based on a power analysis of the aftereffects measured during Experiment 1.) 

Stimuli

Movies were made in which the actions of two individual (adapting and test) actors occurred in succession in a naturalistic fashion. Movies were similar to those used in Experiment 1; however, test actors were combined with the adapting actor lifting different numbers of boxes (one, two, four, and eight) onto the table. 

Procedure

The experiment was a within-subject factorial design with three adaptation conditions (adapt heavy: lift 18 kg; adapt light: lift 2 kg; no adaptation) and four adapting action repetitions (one, two, four, and eight repeats). In addition, there were three test action conditions (lift 6, 10, and 14 kg) and three test actor identities (two males, one female). Test action condition and test actor identity were not conditions of interest and therefore were not analyzed; however, they were varied in order to ensure that participants viewed different actions during experimental testing. No-adaptation and adaptation conditions were presented in separate blocks of testing. 

As in Experiment 2, no-adaptation control blocks were completed both before and after an adaptation block. These control blocks were used to provide a measure of participants' ratings of the weight of boxes lifted by test actors in the absence of an adapting actor in the scene. There were nine movies used during each control block: 3 test actors (two males, one female) × 3 test action conditions (6, 10, and 14 kg). During each control block, each of the nine movies was presented five times in a pseudorandom order. All other procedures were the same as in Experiment 2. 

Between the two control blocks was a single adaptation block. During the adaptation block 72 different movies (2 adaptation conditions × 4 adaptation action repetitions × 3 test actors × 3 test action conditions) were presented. Movies (trials) occurred once in a random order. All other procedures were the same as for the adaptation blocks in Experiment 2, except two breaks of 1 to 2 min were given to participants after 24 trials to help reduce participant fatigue. 

We calculated aftereffects for each of the four adapting action repetition conditions and each of the two adaptation conditions. The mean ratings of the weights of boxes lifted by the test actors for control trials were subtracted from the mean ratings of the weights of the boxes lifted by the test actors in each adaptation condition to generate eight aftereffects (four heavy and four light). Positive values indicated that test actors appeared to lift heavier boxes, negative values indicated that test actors appeared to lift lighter boxes, and zero indicated no effect of the adapting actor. 

Results

An ANOVA with adaptation condition and adapting action repetition as within-subject factors was used to test the magnitudes of the different aftereffects. As in Experiments 1 and 2, the behavior of the adapting actor had a significant influence on the perception of the actions executed by the test actors: main effect of adaptation condition, F(1, 16) = 11.90, p = 0.003, η_p² = 0.43. There was no main effect of adaptation action repetition, F(3, 48) = 1.00, p = 0.399, η_p² = 0.06. However, importantly, there was an interaction between adaptation condition and adaptation action repetition, F(2.07, 33.08) = 16.36, p < 0.001, η_p² = 0.51 (Greenhouse-Geisser correction applied), indicating that aftereffects increased magnitude with adapting action repetition (Figure 2c). Previously examined aftereffects (Barraclough et al., 2009, 2012; Hershenson, 1989, 1993; Leopold et al., 2005) increase with adapting stimulus exposure and can be best modeled with a logarithmic function. To test this we fitted (using an unconstrained nonlinear minimization of the sum or squared residuals; MATLAB 2016; MathWorks) simple (linear, exponential, logarithmic) functions to the data to determine the best model of the action recognition aftereffect increase with adapting action exposure. Exponential functions provided the best models for both the data following adaptation to lifts of light boxes (R = 0.91) and the data following adaptation to lifts of heavy boxes (R = 0.91). (R values for the alternate functions are listed in Supplementary Table S1.) 

Experiment 4: Duration of action aftereffects

Adaptation aftereffects typically decay exponentially over time (e.g., Barraclough et al., 2009, 2012; Hershenson, 1989; Kloth & Schweinberger, 2008; Leopold et al., 2005; Magnussen & Johnsen, 1986). In Experiment 4, we therefore examined whether the aftereffects observed during previous experiments decayed in an exponential fashion by varying the interval between the last adapting action and the test action. 

Method

Participants

Seventeen participants (13 females, four males; mean age = 20.3 years, SD = 2.0) took part in the experiment. The number of participants was based on a power analysis of the aftereffects measured during Experiment 1.) 

Stimuli

Movies were made in which the actions of two individual (adapting and test) actors occurred in succession in a naturalistic fashion. Movies were similar to those used in Experiment 1; however, we varied the inter-action interval—that is, the duration between the offset of the last adapting action and the onset of the test action (1, 2, 4, and 8 s). 

Procedure

The experiment was a within-subject factorial design with three adaptation conditions (adapt heavy: lift 18 kg; adapt light: lift 2 kg; no adaptation) and four inter-action intervals (1, 2, 4, and 8 s). In addition, there were three test action conditions (lift 6, 10, and 14 kg) and three test actor identities (two males, one female). Test action condition and test actor identity were not conditions of interest and therefore were not analyzed. They were, however, varied in order to ensure that participants viewed different actions during experimental testing. No-adaptation and adaptation conditions were presented in separate blocks of testing. 

As in Experiment 2, no-adaptation control blocks were completed both before and after an adaptation block. These control blocks were used to provide a measure of participants' ratings of the weight of boxes lifted by test actors in the absence of an adapting actor in the scene. There were nine movies used during each control block: 3 test actors (two males, one female) × 3 test action conditions (6, 10, and 14 kg). During each control block, each of the nine movies was presented five times in a pseudorandom order. All other procedures were the same as for Experiment 2. 

Between the two control blocks was a single adaptation block. During the adaptation block 72 different movies were presented: 2 adaptation conditions × 4 inter-action intervals × 3 test actors × 3 test action conditions. Movies (trials) occurred once in a random order. All other procedures were as for the adaptation blocks in Experiment 2, except two breaks of 1 to 2 min were given to participants after 24 trials to help reduce participant fatigue. 

We calculated aftereffects for each of the four inter-action interval conditions and each of the two adaptation conditions. We subtracted the mean ratings of the weights of boxes lifted by the test actors for control trials from the mean ratings of the weights of the boxes lifted by the test actors in each adaptation condition to generate eight aftereffects (four heavy and four light). Positive values indicated that test actors appeared to lift heavier boxes, negative values indicated that test actors appeared to lift lighter boxes, and zero indicated no effect of the adapting actor. 

Results

An ANOVA with adaptation condition and interaction interval as within-subject factors was used to test the magnitudes the different aftereffects. As in previous experiments, the behavior of the adapting actor had a significant influence on the perception of the actions executed by the test actors: main effect of adaptation condition, F(1, 16) = 29.40, p < 0.001, η_p² = 0.65. There was no main effect of inter-action interval, F(3, 48) = 2.66, p = 0.850, η_p² = 0.02; however, there was an interaction between adaptation condition and inter-action interval, F(3, 48) = 3.1, p = 0.036, η_p² = 0.16. Previously examined aftereffects (Barraclough et al., 2009, 2012; Hershenson, 1989, 1993; Leopold et al., 2005) have shown a decrease in aftereffect magnitude over time best modeled by an exponential decay function. To test this we fitted (using an unconstrained nonlinear minimization of the sum or squared residuals; MATLAB 2016) simple (linear, exponential, logarithmic) functions to the data to determine the best model of the action recognition aftereffect decay. The decay over time (Figure 2d) could be best modeled by an exponential function fitted to the data following adaptation to lifts of light boxes (R = 0.91) and a logarithmic function fitted to the data following adaptation to lifts of heavy boxes (R = 0.86). (R values for the alternate functions tested are listed in Supplementary Table S2.) 

Experiment 5: Effect of adaptation on the actor's belief state

The previous experiments demonstrated that adaptation in naturalistic environments can generate aftereffects that influence action recognition. However, actions can also provide information about the internal mental state of the actor (e.g., Becchio, Sartori, & Castiello, 2010; Grèzes et al., 2004; Sartori, Becchio, Bara, & Castiello, 2009). Inference of mental states derived from action kinematics is thought to be a separate processing stage following action recognition (Frith & Frith, 1999; Hamilton & Grafton, 2006; Saxe & Kanwisher, 2003; Van Overwalle & Baetens, 2009). In Experiment 5, we tested whether visual adaptation could influence this later stage of processing and influence inferences about the expectations individuals have about box weight. Observers viewed in both life-size orthostereoscopic 3D and small-scale 2D presentation formats adapting actors with different expectations of the weight of the box they were to pick up. Observers then made judgments about the expectations of the test actors subsequently picking up boxes. 

Method

Participants

Twenty-one participants (13 females, eight males; mean age = 20.9 years, SD = 0.7) took part in the experiment in which stimuli were presented in 2D, whereas 23 different participants (six females, 17 males; mean age = 24.7 years, SD = 5.5) took part in the experiment in which stimuli were presented in 3D. 

Stimuli

Movies were made in which the actions of two individual actors (adapting and test) occurred in succession in a naturalistic fashion. In the movies both the adapting and test actors remained in the scene throughout the trial. Different stimuli from Set 2 were combined using video compositing to construct new movies in which the adapting actor appeared on the left and a test actor appeared on the right in a natural-looking room scene. In each movie, the adapting actor action was always repeated eight times. Although this resulted in a somewhat unnaturalistic jump in actor posture from the last frame of the action clip to the first frame of the next clip, it was not possible to generate a sequence of multiple similar actions because actors could not be deceived about the weight of the box eight times in a row. 

Procedure

Experiment 5 had a factorial design with viewing condition (life-size orthostereoscopic 3D, small-scale 2D) as a between-subjects factor and adaptation condition (adapt overestimation, Movie 4; adapt underestimation, Movie 5; no action [control], Movie 6) as the within-subject factor. In addition, there were four test actor identities (two males, two females) and five test action conditions (lift 6, 8, 10, 12, and 14 kg. (Because the expectation of the test actors was that the box weight would be 10 kg, each test weight corresponds to different over- and underestimations; see Method.) Test action condition and test actor identity were not conditions of interest; however, they were varied in order to ensure that participants viewed different actions during experimental testing. No-adaptation control blocks were completed both before and after two adaptation blocks. 

There were 20 movies used during each control block: 4 test actors (two males, two female) × 5 test action conditions (6, 8, 10, 12, and 14 kg). Each of the 20 videos was presented in a pseudorandom order and presented four times in total during each control block (total = 80 trials). In control blocks, trials began with the test actor standing still for 2 s followed by the action without the presence of any adapting actor in the room. The screen turned black, indicating that the participant was to respond during a 3-s interval. 

Participants were required to judge whether the test actor had a true or false expectation of the weight of the box and, if false, whether and to what extent the weight was over- or underestimated. Responses were made on a linear scale of symbols printed on keyboard keys: “- - - -,” “- - -,” “- -,” “-,” “0,” “+,” “+ +,” “+ + +,” and “+ + + +.” The symbols represented a continuum from an extreme underestimation (“- - - -”) through true estimation (“0”) to an extreme overestimation (“+ + + +”). For analysis, these symbols were converted to a range of −4 to +4. 

Adaptation conditions were tested in two counterbalanced blocks where in each block the adapting actor was either over- or underestimating the weight of the lifted box. Within a block, all other conditions (four test actor identities and five test weights) were randomized. At the start of each adaptation block, an initial preadaptation movie of the adapting actor lifting the box 24 times was displayed. The first trial began as soon as the preadaptation phase had finished. Each trial began with 2 s of adapting and test actor inactivity, followed by the adapting actor lifting his or her box onto the table eight times and a 1-s inter-action interval during which both actors stood still. After this, the test actor lifted his or her box. The screen turned black, indicating that the participant was to respond during a 3-s interval. During adaptation blocks, participants were required to judge whether the test actor had a true or false expectation of the weight of the box as for the control blocks. 

Linear functions were fitted to the mean estimates of the test stimuli across both control conditions to assess participants' ability to distinguish over- and underestimations. Data from one participant (female, age = 26 years) tested with the 3D presentation format were excluded from further analysis because the participant could not distinguish over- and underestimations correctly (slope of linear function >2 SD from the mean). To calculate aftereffects, for each participant and each presentation format we subtracted the mean estimate of the expectation of test actors during control trials from the mean estimate of the expectation of test actors during each adaptation condition. Positive values indicated that test actors appeared to overestimate the weight of the box they were to lift, negative values indicated that test actors appeared to underestimate the weight of the box they were to lift, and zero indicated no effect of adaptation. 

Results

An ANOVA with adaptation condition and presentation format as factors showed that estimates of test actor expectation were significantly changed by the behavior of the adapting actor—main effect of adaptation condition, F(1, 41) = 31.6, p < 0.0001, η_p² = 0.44 (Figure 3a)—where test actors appeared to underestimate box weight following adaption to actors overestimating box weight and test actors appeared to overestimate box weight following adaption to actors underestimating box weights. There was no main effect of presentation format, F(1, 41) = 3.13, p = 0.084, η_p² = 0.07. However, aftereffects were smaller during life-size 3D orthostereoscopic presentation—interaction between adaptation condition and presentation format, F(1, 41) = 7.32, p = 0.010, η_p² = 0.15—although for both 3D and 2D presentation formats, adaptation condition resulted in significantly different aftereffects—paired t tests: 3D t(21) = 2.17, p = 0.042; 2D t(20) = 5.60, p < 0.001. In conclusion, these results show that inferences about the belief states of individuals are influenced by adaptation to the belief states of other individuals in the social scene. 

Figure 3

 

Effect of adaptation on inferences about belief state. (a) Changes in the perception of the test actors' expectations of box weights following adaptation to another individual's box-lifting actions. (b) Changes in the perception of actors' expectations of box weight following adaptation to lever-pulling actions. Error bars indicate standard error of the mean.

Experiment 6: A test of cross-action adaptation

Experiment 5 demonstrated that action adaptation influences inferences about the belief state of individuals, but it was not clear to which of two possible effects the results should be attributed. First, adaptation could have occurred at an abstract level where the belief states of other individuals are derived. Alternatively, adaptation could have occurred at an earlier level where the action kinematics are processed, which then could modulate inferences about belief states. The stimuli from Set 2 where the adapting actors underestimate the weight of the box produce kinematics consistent with lifting a heavier box; the stimuli where the actors overestimate the weight of the box produce kinematics consistent with lifting a lighter box. Thus, adaptation to an underestimation (or heavy lift) results in a perception of an overestimation (or lighter lift), and thus the two mechanisms are indistinguishable. 

In Experiment 6 we tested the level of action understanding at which adaptation occurred by examining whether adaptation to actor expectations transferred across different actions. Representations of the belief state of others are rather abstract and do not rely on specific or easily predicted action kinematics, as mental state can be derived from many different stimuli (e.g., Heider & Simmel, 1944). The transference of adaptation from one action to another would indicate adaptation occurring at such an abstract representation of actor mental state. 

Method

Participants

Twenty-one participants (19 females, two males; mean age = 24.0 years, SD = 3.6) took part in the experiment. 

Stimuli

As in Experiment 5, movies were made in which the actions of two individual actors (adapting and test) occurred in succession in a naturalistic fashion. In the movies, both the adapting and test actors remained in the scene throughout the trial. Different stimuli from Set 3 were combined using video compositing to construct new movies in which the adapting actor appeared on the right and a test actor appeared on the left in a natural-looking room scene (Figure 1b). In each movie, as in Experiment 5, the adapting actor action was repeated eight times. 

Procedure

Experiment 6 was a within-subject test of the effect of adaptation to the belief state of an actor pulling a lever on the belief state of another actor lifting a box. The adapting actor pulled levers emerging from a box (see General method), where they underestimated, overestimated, or had a correct estimation (control) of the force required to execute the pull action. Test actors lifted boxes onto a table, where they had seven levels of overestimation and underestimation of the force required to execute the lift action (2.5, 5, 7.5, 10, 12.5, 15, and 17.5 kg). Given that test actor underestimations may be perceived as the test actor lifting heavy boxes and test actor overestimations may be perceived as the test actor lifting light boxes, any aftereffects that may be observed could be due to a hypothetical force adaptation rather than adaptation to the belief state. To control for this we included two additional adaptation conditions where the adapting actors pulled a heavy (8 kg) lever and a light (1 kg) lever, during which the adapting actor had a correct estimation of the force required to execute the action. 

Using pilot data we matched the adapting stimuli on their perceived force required while varying the perceived belief state. That is, in the underestimate and heavy adaptation conditions the perceived weight was roughly equivalent, but in the underestimation condition the perceived underestimation was greater. Similarly, in the overestimation and light conditions the perceived weight was roughly equivalent, but in the overestimation condition the perceived overestimation was greater (Table 1). 

Table 1

 

Perceived weight and perceived expectation in each of the adaptation conditions.

 

Pilot data showed that the pull action in which the adapting actor overestimated the force required (expected 9 kg, executed 5 kg) and the pull action in which the adapting actor pulled a light lever (expected and executed 1 kg) appeared to be executed with equivalent force (t = 0.27, p = 0.79). In addition, the pull action in which the adapting actor underestimated the force required (expected 1 kg, executed 5 kg) and the pull action in which the adapting actor pulled a heavy lever (expected and executed 9 kg) appeared to be executed with equivalent force (t = 0.97, p = 0.37). Thus, in total there were five adapting action conditions (adapt overestimation of pull force, Movie 7; adapt underestimation of pull force, Movie 8; adapt pull heavy lever, Movie 9; adapt pull light lever, Movie 10) and a control (no adaptation) condition (Movie 11). 

No-adaptation control blocks were completed both before and after an adaptation block. During control blocks, no adapting actor was present in the room (upper sequence in Figure 1b); however, the stimuli and procedure were as for the adapting block. During the adapting block both the adapting and test actors were present in the room (lower sequence in Figure 1b); all conditions (four adapting stimuli, seven test stimuli) were presented three times in a pseudorandom order. On any one adaptation trial, initially, the actors stood still for 2 s. Then the adapting action was repeated eight times, followed by a 1-s interstimulus interval during which both actors stood still. Finally, the test actor lifted the box. The screen turning black indicated to the participant to respond during a 4-s interval. Participant response range and data analysis were as for Experiment 5. 

Results

A one-way repeated measures ANOVA showed that aftereffects generated by the different adapting stimuli were not significantly different from each other, F(4, 80) = 0.40, p = 0.81, η_p² = 0.02 (see Figure 3b). All aftereffects were small and nonsignificant—all ts(20) < 0.81, ps > 0.32—showing that adaptation to action force, and actor belief state, does not transfer from one action to another. In conclusion, Experiment 6 shows that adaptation to belief state does not occur at a high abstract level where the precise kinematics of the actions are irrelevant. Rather, changes in belief state resulting from visual adaptation appear dependent on the specific kinematics of the actions being executed (cf. Wincenciak et al., 2016). 

General discussion

In a series of six experiments we examined the effect of adaptation on action recognition and inferences of actor belief state in naturalistic environments. Experiments 1 through 4 showed that action recognition aftereffects occurred when two individuals were present in a social scene. Here, the recognition of the actions executed by one individual is changed by adaptation to the actions previously executed by the other individual. These action recognition aftereffects are similarly sized when measured using small screens typically found in psychophysics laboratories, as when measured using life-size orthostereoscopic 3D presentation in a virtual reality environment (Experiment 1). Action recognition aftereffects were similarly sized irrespective of the view from which the two actors were seen (Experiment 2), indicating that the aftereffects rely on high-level view-independent mechanisms (Barraclough & Jellema, 2011; Jellema & Perrett, 2006). The action recognition aftereffects showed both a build-up with adaptation exposure (Experiment 3) and decay over time (Experiment 4). These dynamics are characteristic of other previously observed visual aftereffects and indicate that the effects we observe here are likely due to adaptation rather than other potential mechanisms. Experiment 5 showed that inferences about the belief state of individuals are influenced by visual adaptation to the belief state of another individual in the same social scene. Although Experiment 5 suggested a possible high-level abstract adaptation that affects action understanding, during Experiment 6 belief state aftereffects disappeared when the adapting and test actors were executing different actions. The results of Experiment 6 showed that aftereffects do not transfer from one action to another and therefore ruled out the possibility of high-level abstract adaptation to belief states. We believe that the most parsimonious explanation is that action recognition adaptation modulates subsequent inferences about the belief state of individuals. 

Applicability of results to natural viewing

During Experiments 1 through 4 the social scenes unfolded in a naturalistic fashion in which the adapting actor would execute a number of actions and then the test actor would execute his or her action. Unlike previous adaptation experiments with action stimuli (Barraclough & Jellema, 2011; Barraclough et al., 2009; de la Rosa et al., 2014; Roether et al., 2009; Troje et al., 2006)—and indeed with face stimuli (e.g., Chen, Yang, Wang, & Fang, 2010; Leopold et al., 2001; Rhodes, Jeffery, Watson, Clifford, & Nakayama, 2003)—where stimuli were disembodied, were limited to dots of light, or would appear and disappear from the screen in quick succession, the current scenes were more akin to the natural environments we typically experience. Our results indicate that during natural viewing of social scenes in which more than one individual is present, visual adaptation can change our ability to recognize the actions of individuals accurately. For Experiments 5 and 6, the social scenes were somewhat less realistic: Here the actions executed by the adapting actor in the scene had to be repeated eight times rather than having the adapting actor execute eight different actions. This was necessary due to the nature of the stimuli: It was not possible to recruit actors who could be fooled repeatedly about the weight of boxes they were to lift. This limitation does potentially limit the generalizability of the distortions in belief state observed with visual adaptation during Experiment 5. 

During Experiments 1 and 5 we compared aftereffect magnitudes when participants viewed the natural scenes via different presentation formats. Action recognition aftereffects were similarly sized when scenes were presented in life-size orthostereoscopic 3D, in small-scale 2D, and in life-size 2D. The full orthostereoscopic 3D presentation allows a very naturalistic experience of the social scene in which the participant experiences the actors appearing in the same room as them. Actors and physical environment were of the correct size and correct depth information, with realistic lighting and shadows cast by the actors in the scene. Together, these cues provide a compelling sense of the presence of the scene. Although the virtual reality experience is not identical to our natural experience, our results highlight the importance of action (and presumably other) aftereffects appearing during our daily experience. In addition, these results show that the action recognition aftereffects studied using typical psychophysics equipment in the laboratory are informative about our daily experiences. Belief state aftereffects were also observed during life-size orthostereoscopic 3D presentation; however, they were much smaller than those observed with a small-scale 2D presentation format. A somewhat parallel effect has been seen previously, where functional magnetic resonance adaptation to the real 3D presentation of objects is markedly reduced compared with the presentation of the same objects on a screen in 2D (Snow et al., 2011). Although our stimuli are more complex and social, and although functional magnetic resonance adaptation cannot be directly compared with the psychophysical adaptation we measured, these results do indicate differences in the way the brain responds to repeated presentation of realistic 3D stimuli and 2D images of stimuli. Furthermore, other research suggests that we interact in distinct ways with our 2D and 3D world (e.g., Holmes & Heath, 2013). Our results raise the concern that studying the more complex higher order and abstract forms of action understanding with unnaturalistic viewing formats in typical psychophysics and neuroimaging laboratories does not automatically inform us about the performance of action understanding in real-life conditions. 

Sensitivity to viewpoint

Action recognition aftereffects were similarly sized irrespective of the view from which the adapting and test actors were seen. When the test actor was seen from a perspective 180° rotated away from the view of the adapting actor, aftereffects were not significantly different from the aftereffects when the test and adapting actors were seen from the same perspective. A similar view independence has been seen previously for whole-body walking aftereffects (Barraclough & Jellema, 2011). In contrast, some other high-level aftereffects for hand actions (Barraclough et al., 2009) and face expressions and identities (Benton, Jennings, & Chatting, 2006) show a mixture of viewpoint independence and viewpoint dependence. The differences seen in the degree of viewpoint dependence observed in previous results and this study may be due to the nature of the stimuli presented. The greater viewpoint dependence is observed in the studies examining processing of body parts (hands, faces), whereas viewpoint independence is observed in those studies examining whole-body aftereffects. This increase in viewpoint independence may reflect the transition from processing body parts to more comprehensive representations of bodies in a hierarchical fashion (Fleischer, Caggiano, Thier, & Giese, 2013; Giese & Poggio, 2003; Jellema & Perrett, 2006). 

Adaptation dynamics

The action recognition aftereffects we observe here have much in common with other demonstrations of action aftereffects and other visual aftereffects. We see an increase in aftereffect magnitude with repetition of adapting actor action and an exponential decay (for the light aftereffect) over time, inconsistent with a simple priming effect but consistent with studies investigating the dynamics of tilt (Magnussen & Johnsen, 1986), motion (Hershenson, 1989), face identity (Leopold et al., 2005), face configuration (Rhodes et al., 2007), gender (Ghuman, McDaniel, & Martin, 2010), biological motion (Troje et al., 2006), hand action (Barraclough et al., 2009), and whole-body aftereffects (Barraclough et al., 2012). Because the time course of the action recognition aftereffects we observe follows this classic time course, it suggests that the adapted mechanism is perceptual in nature and is neither an artifact of participant behavior during the experimental task nor perhaps due to other postperceptual processes (see below). 

Adaptation to belief state

Adaptation was also shown to influence judgments about the belief state of individuals (Experiment 5). On the face of it, these effects might appear consistent with repulsive aftereffects similar to those observed following adaptation to hand actions (Barraclough et al., 2009; de la Rosa et al., 2014) or whole-body actions (Experiments 1 through 4; Barraclough & Jellema, 2011). Interpretation of the mechanisms underlying these changes in the inferences about the belief state of individuals, however, is difficult. We therefore conducted Experiment 6 to clarify the putative site of the adaptation effect. This experiment showed that adaptation to belief state did not transfer from one action to another. Such supracategorical adaptation would be expected if there was adaptation at an abstract level of the belief state. Indeed, areas of the cortex involved in making inferences about the mind of other individuals (theory of mind) respond to a range of very different stimuli conveying the same meaning (e.g., Castelli et al., 2010; Castelli, Frith, Happe, & Frith, 2002; Saxe & Kanwisher, 2003). That adaptation to one action does not transfer to a different action indicates that adaptation is not acting at an action-independent abstract level but rather at an action-dependent level. Rather, we believe that adaptation at a conceptually lower level where the actions themselves are distinguished (Barraclough & Jellema, 2011; Barraclough et al., 2009; Kuravi, Caggiano, Giese, & Vogels, 2016) is the most parsimonious explanation for the effects we observed in Experiment 5. Adaptation to the adapting actor underestimating the weight of the box will also result in adaptation to kinematics associated with lifting an unexpectedly heavier box. The perceived change in the test actors' judgments of the weight of the boxes such that they appear to overestimate the box weights may be due to a change in the representation of the kinematics of box lifting such that they appear more akin to the kinematics associated with lifting an unexpectedly lighter box. Similar but opposite effects are also observed following adaptation to adapting actors overestimating box weights. In short, changes in belief state attribution can be explained by changes in perceived kinematics—a less abstract task. 

Possible alternate mechanisms underlying action adaptation

Lower level retinotopic adaptation

There is a possibility that there might be some influence of retinotopic adaptation or adaptation at earlier stages in the visual system. Adaptation of retinotopic mechanisms might occur if the participants fixated the screens in such a way that the position of the adapting actor in visual space subtended one region of the retina and then, when observing the test actor, their fixation was such that the test actor subtended the same region of visual space. Although we do not rule out the possibility that adaptation at a low level in the visual system may occur in parallel, we do not believe they are the dominating effects we observe here. First, the stimuli were presented for a long period of time (>30 s), and although we did not measure eye movements, the participant freely viewed the stimuli; therefore, it is likely that they moved their eyes during testing and did not fixate in the rather idiosyncratic fashion described above. Second, the low-level characteristics (e.g., shape, color, form, motion) of the adapting and test actors were very different: They were different identities and sizes, wore different clothes, and so on. Finally, during Experiment 2 when the adapting and test stimuli were seen from very different views, similarly sized aftereffects were observed even though the actors could not overlap the same retinotopic regions. 

Postperceptual processes

One possibility is that the results we observe are due to postperceptual processes in which the cognitive decisions of the participants are influenced (Morgan, Melmoth, & Solomon, 2013). A defining characteristic of perceptual adaptation is that aftereffects build up logarithmically with adaptation stimulus exposure and decline exponentially with time (Gibson & Radner, 1937; Hershenson, 1989; Leopold et al., 2005). A postperceptual decision-making process is unlikely to show these specific characteristic dynamics. Consistent with an explanation based on perceptual adaptation, Experiments 3 and 4 showed that action recognition aftereffects increased with adapting action recognition and declined with time. Although the build-up with adaptation was best modeled by exponential functions, the aftereffects with one, two, and four repeats of the adaptation action were small and nonsignificant, making drawing conclusions about the best-fitting function difficult. Overall, however, the relationship between aftereffect magnitude and adaptation duration and duration between the adapting and test stimuli is strongly suggestive of genuine perceptual adaptation processes. 

Even stronger evidence against a postperceptual explanation for the results we observe comes from Experiment 6. In this experiment we found that adaptation to one action did not influence perception of another action. If adaptation effects were due to a shift in participants' decisions made about the belief state of individuals, then we would expect this to be independent of the actions that they observed; however, cross-action adaptation was not observed. Indeed, aftereffects were dependent on the kinematics of the actions executed by the adaptation and test actors. 

Conclusions

During observation of two individuals in naturalistic scenes, perceptual adaptation aftereffects influence action recognition and inferences about the belief state of individuals. These action recognition aftereffects are view independent and show the same characteristic dynamics as seen with more simple stimuli and with some less naturalistic experimental designs. These distortions in social perception that occur whilst viewing naturalistic scenes can be explained by visual adaptation of mechanisms coding specific action kinematics; they are neither attributable to low-level adaptation, nor response shifts due to top-down influences (Morgan et al., 2013), nor do they need to invoke simulation mechanisms (Gallese & Goldman, 1998) nor higher-order Theory-of-Mind processes (Gallagher & Frith, 2003). Our results suggest that the perception of an agent's action in a naturalistic social scene, and the interpretation of the agent's mental state, is dependent not only on what that agent is doing but also on the adaptive effect induced by other agents in the social environment. 

Acknowledgments

We thank Miriam Barber, Helen Raassum, Rachel Bush, and Erin McWilliams for help with testing participants and Katie Slocombe for reading earlier drafts of this article. This research was funded by a grant from the Economic and Social Research Council (RES-062-23-2797) awarded to Nick E. Barraclough. Joanna Wincenciak was supported by an 80th Anniversary Scholarship from the University of Hull. 

Commercial relationships: none. 

Corresponding author: Nick E. Barraclough. 

Email: [email protected]. 

Address: Department of Psychology, University of York, York, UK.