Expecting the unexpected: Temporal expectation increases the flash-grab effect

Abstract

In the flash-grab effect, when a disk is flashed on a moving background at the moment it reverses direction, the perceived location of the disk is strongly displaced in the direction of the motion that follows the reversal. Here, we ask whether increased expectation of the reversal reduces its effect on the motion-induced shift, as suggested by predictive coding models with first order predictions. Across four experiments we find that when the reversal is expected, the illusion gets stronger, not weaker. We rule out accumulating motion adaptation as a contributing factor. The pattern of results cannot be accounted for by first-order predictions of location. Instead, it appears that second-order predictions of event timing play a role. Specifically, we conclude that temporal expectation causes a transient increase in temporal attention, boosting the strength of the motion signal and thereby increasing the strength of the illusion.

Introduction

Neural transmission and processing delays pose a challenge for determining the instantaneous position of moving objects. Our ability to accurately perceive and interact with moving objects as though there were no delay (Brenner, Smeets, & de Lussanet, 1998) suggests that the brain is able to compensate for these delays. One explanation for how the brain might do this is by predicting the position of a moving object based on its previous position and velocity, extrapolating along the object's previous motion trajectory (Nijhawan, 1994). 

Such motion extrapolation mechanisms are thought to underlie a range of visual motion illusions, in which stationary objects are mislocalized due to motion signals. This includes the well-known flash-lag effect, in which a briefly flashed stationary stimulus appears to lag behind a moving stimulus, even though they are physically aligned (Nijhawan, 1994). Related illusions include the flash-drag (Whitney & Cavanagh, 2000), flash-jump (Cai & Schlag, 2001), and flash-grab illusions (Cavanagh & Anstis, 2013). Over the past 25 years, there has been considerable debate about the mechanisms underlying these phenomena (Eagleman & Sejnowski, 2000; Krekelberg & Lappe, 2001; Whitney & Cavanagh, 2000). Although the source of these motion-induced position shifts remains controversial, recent evidence has provided a missing link in the extrapolation account: a mechanism that corrects for failed extrapolation (Blom, Liang, & Hogendoorn, 2019). Blom et al. (2019) found that mislocalization in the flash-grab effect not only depends on motion after the reversal, but also crucially involves a component opposite to the motion vector preceding the reversal. This correction-for-extrapolation mechanism shifts the extrapolated position back along its trajectory when the extrapolated representation is not confirmed by sensory input. This new result provides additional support for predictive extrapolation mechanisms (Nijhawan, 2008). 

At the neural level, extrapolation mechanisms have been demonstrated at multiple stages in the visual hierarchy in both animals and in humans. For example, motion extrapolation has been reported in the retinae of rabbits, mice, and salamanders (Berry, Brivanlou, Jordan, & Meister, 1999; Schwartz, Taylor, Fisher, Harris, & Berry, 2007), in macaque LGN (Sillito, Jones, Gerstein, & West, 1994), in both cat and macaque V1 (Jancke, Chavane, Naaman, & Grinvald, 2004; Subramaniyan et al., 2018), and in macaque V4 (Sundberg, Fallah, & Reynolds, 2006). In humans, extrapolation mechanisms have been demonstrated extensively in the motor domain (Brenner et al., 1998), and motion prediction has also been shown in the early visual system using fMRI (Ekman, Kok, & de Lange, 2017), EEG (Hogendoorn & Burkitt, 2018; Hogendoorn, Verstraten, & Cavanagh, 2015), and psychophysics (Hogendoorn, Carlson, & Verstraten, 2008; van Heusden, Harris, Garrido, & Hogendoorn, 2019; van Heusden, Rolfs, Cavanagh, & Hogendoorn, 2018). 

Conceptually, these low-level predictive extrapolation mechanisms might be expected to interact with higher order predictions, such as those formed by statistical learning (Dale, Duran, & Morehead, 2012) or cognitive processes such as attention and expectation (Gordon, Tsuchiya, Koenig-Robert, & Hohwy, 2019). Here, we examine the interaction between predictive motion extrapolation and temporal expectation. To do this, we use the flash-grab effect, a visual illusion in which a target is briefly flashed on top of a moving texture as it reverses direction (Cavanagh & Anstis, 2013). This results in the flashed target being perceived as displaced in the direction of motion following the reversal. We have previously argued that one way to understand the flash-grab effect is as a consequence of violated extrapolation (van Heusden et al., 2018). In this interpretation, the moving background activates motion extrapolation mechanisms, which (due to neural delays) briefly continue extrapolating the position of the background even after it reverses. As a result, by the time the reversal is detected, there will be a mismatch between the predicted and actual positions of the background. Together with the motion sequence that immediately follows the reversal, this causes a strong transient velocity signal in the direction opposite to the initially extrapolated direction, as the system corrects for the failed extrapolation and “catches up” (for graphical representations, see Figure 1). Crucially, such velocity signals have been argued to bias position representations of concurrently presented static objects (Eagleman & Sejnowski, 2007)—in this case causing the perceived position of the target to shift in the direction of the second motion sequence. Here, we investigate how the magnitude of this illusion depends on the temporal precision with which the observer is expecting the reversal. 

Figure 1

 

Schematic illustration of extrapolation in the flash-grab effect (reproduced from Van Heusden et al., 2019). Each panel shows position as a function of time, with the upper plot showing velocity as a function of time. Solid gray traces indicate the position and velocity of the physical stimulus as presented on the screen, and dotted black traces indicate (predictive) neural representations of the same stimulus. This mirrors representations in the retina as demonstrated by Schwartz et al. (2007), but could equally apply to any position representation in the visual hierarchy. (A) In order to accurately localize a moving object despite neural transmission delays, the visual system uses concurrent velocity signals to extrapolate the real-time position of the object (blue lines). (B) When an object unexpectedly reverses direction, at any given level of representation, some time elapses before the reversal is detected. During that time, the object will continue to be (erroneously) extrapolated into positions where it is never presented, creating a prediction error. (C) As the represented position shifts from the predicted trajectory to a new trajectory, a brief spike in the corrective velocity (dotted red trace) will generate an exaggerated position shift. If a (stationary) flash is presented at the same time as the reversal, then the position of the flash will interact with the (large) transient velocity signal and be mislocaliaed, resulting in the flash-grab effect (red lines).

Previous research on temporal expectation in the related flash-lag illusion suggests that the perceived position of the flash might be modulated by its temporal predictability. For example, Vreven and Verghese (2005) used temporal cues to influence the predictability of the flash, and showed that the flash-lag effect is reduced when the timing of the flash is predictable. Spatial predictability has similarly been found to reduce the strength of the flash-lag effect (Baldo, Kihara, Namba, & Klein, 2002; Baldo & Namba, 2002; Namba & Baldo, 2004). In the flash-grab illusion, it is known that spatial predictability also decreases the flash-grab effect (Adamian & Cavanagh, 2016), but nothing is known about temporal certainty. 

More broadly, it is well known in many domains of behavioral research that when target stimuli occur at constant and/or predictable times, observers can anticipate the stimuli and show enhanced motor processing (in the form of reaction time) compared to unpredictable conditions (Niemi & Näätänen, 1981; Nobre, Correa, & Coull, 2007). Notably, both temporal uncertainty and reaction time decrease with longer presentation times, reflecting the increasing hazard function (Nobre et al., 2007): The conditional probability of an event occurring increases, given that it has not yet occurred (Luce, 1986). Evidence from perceptual discrimination tasks (Correa, Lupiáñez, & Tudela, 2005; Rohenkohl, Cravo, Wyart, & Nobre, 2012; Rolke & Hofmann, 2007) and event-related potentials reveals that temporal expectation may also enhance visual perception processing (Correa, Lupiáñez, Madrid, & Tudela, 2006; Doherty, Rao, Mesulam, & Nobre, 2005; Rohenkohl & Nobre, 2011). Doherty et al. (2005) specifically investigated spatial and temporal expectations on attentional orienting to moving stimuli, using an occluder and varying the temporal interval after which a moving object reappeared from behind the occluder. They showed that spatial and temporal expectation interact synergistically to improve position-specific processing at the predicted time. 

In the context of the flash-grab effect, if temporal expectation transiently boosts attention to early visual processing (Doherty et al., 2005), then it might be expected to similarly boost the strength of the motion signal and the magnitude of the extrapolation errors that arise when the reversal is detected. Eagleman and Sejnowski (2007) argued that this motion signal directly biases localization judgments, causing the related flash-lag, flash-drag, flash-jump, and Frohlich illusions. The transient allocation of attention would amplify these signals and the resultant mislocalization. Indeed, Shioiri, Yamamoto, Oshida, Matsubara, and Yaguchi (2010) showed that the magnitude of the flash-lag effect can be used to index the amount of available attentional resources, and Cavanagh and Anstis (2013) showed that attention is necessary for the flash-grab effect to be observed. In the light of these results, the motion-induced mislocalization of the target in our flash-grab paradigm should be larger if the reversal is expected than it would have been if the reversal had been unexpected. 

Conversely, under a hierarchical predictive coding framework (Rao & Ballard, 1999), the possible effect of expectation on the flash-grab effect speaks to the distinction between first-order predictions about the content of a sensory signal and second-order predictions about the precision (or “predictability”) of a signal as articulated by Feldman and Friston (2010). If only first-order predictions were taking place, we might hypothesize a decrease in the flash-grab effect with increasing expectation, as sudden changes in sensory content can be rapidly explained away by predicting these changes. Second-order predictions, in contrast, do not represent sensory content, but alter the integration weights of first-order predictions relative to new sensory information. In this way, second-order prediction can be considered the transient increase in the relative weight for sensory input when the event is expected. This is functionally equivalent to the effect of transient increases in temporal attention. 

Here, we present a set of four experiments studying the effect of temporal expectation on the flash-grab effect to discriminate between these two alternative possibilities. In Experiment 1, we reanalyzed three flash-grab datasets, and show that in each case, flash-grab magnitude increased with increasing hazard function: The more a reversal was expected, the larger the illusion. We subsequently carried out three new experiments to test the hypothesis that this increase was due to increased temporal expectation. In Experiment 2, we show that it is not the absolute time before a target is presented, but the relative timing of a given trial within a block that determines flash-grab magnitude. We define the interval between when the motion starts and time at which the target is presented as “time-to-target.” In Experiment 3, we show that when expectation is held constant by drawing trial target times from an exponential, rather than uniform, distribution (thereby flattening the hazard function), the influence of time-to-target on flash-grab magnitude is reduced. Finally, in Experiment 4 we show that when time-to-target is held constant over a series of multiple identical trials, allowing more precise temporal expectation to develop, the flash-grab magnitude gradually increases. Conversely, oddball trials with unexpectedly early reversals generate a substantially reduced illusion. 

Altogether, the results show that temporal expectation affects the magnitude of the flash-grab effect, such that the more predictable the timing of the reversal, the stronger the illusion. Within the predictive coding framework, this is inconsistent with the view that the brain only makes first-order predictions, instead requiring second-order predictions about the expected timing of changes to the environment. In other words, instead of explaining away sudden changes in sensory input, the effect of expectation on flash-grab magnitude is better understood as temporal attention transiently boosting motion signals at the time of target presentation. 

Experiment 1: Analysis of effect of time-to-target on flash-grab magnitude

Here, we analyze the effects of time-to-target from three experiments. Two of these are experiments from published articles from our lab that varied time-to-target but did not analyze this factor: van Heusden et al. (2018) in Experiment 1A and van Heusden et al. (2019) in Experiment 1B. To these we add a new dataset (Experiment 1C). In each case, observers viewed a flash-grab stimulus and reported the perceived position of a red disc flashed on a rotating annulus as the annulus reversed direction. Here, we analyze the magnitude of the illusion as a function of the time at which the motion reversal and flash occurred (time-to-target). 

Methods

For full details of experimental methods in Experiments 1A and 1B, including stimulus geometry and procedure in Experiments 1A and 1B, please see van Heusden et al. (2018) and van Heusden et al. (2019), respectively. General details are summarized below. 

Observers

Experiments 1A, 1B, and 1C included data from eight, 17, and 12 observers, respectively. All observers in all experiments gave informed consent prior to participation and were compensated for their participation. All data was collected in accordance with the Declaration of Helsinki. 

Apparatus

Each experiment was presented using MATLAB (MathWorks, Natick, MA) and the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997). In Experiments 1A and 1C, eye position was recorded using an EyeLink 1000 remote eye-tracker (SR Research, Mississauga, Canada) but that data is not analyzed here. In Experiment 1B, the stimulus was viewed through a mirror stereoscope to be able to manipulate the information presented to each eye. 

Stimulus and procedure

In each experiment, observers viewed a flash-grab stimulus made up of an annulus with alternating black and white segments. The annulus rotated either clockwise or counterclockwise at an angular velocity of 200° per second, and reversed motion direction after a variable amount of time (1,000, 1,100, 1,200, 1,300, 1,400, or 1,500 ms in Experiments 1A and 1B; 1,000, 1,100, 1,200, 1,300, or 1,400 in experiment 1C). In each experiment, an equal number of trials were presented with each value of time-to-target—that is, the distribution of time-to-target was uniform. The target stimulus, a small red disc, was presented for 10 ms on the annulus at the time of the reversal, in one of three possible positions (160°, 180°, or 200° polar offset from the top of the annulus: Figure 2). After the presentation of the target, the annulus continued to rotate in the opposite direction for 400 ms, after which the annulus turned gray. Participants were instructed to report the perceived position of the target after each trial using the computer mouse. If observers did not perceive the target, they were instructed to click at the location of the fixation; these trials were discarded. 

Figure 2

 

Schematic trial sequence of Experiments 1A, 1B, and 1C. Observers view a rotating annulus for a variable amount of time (first panel). At a given moment, a target red disc is briefly flashed in one of three possible target locations (second panel; the solid disc indicates the target on this trial, and the translucent discs indicate the two possible alternative positions). After the target is presented, the annulus continues to rotate in the direction opposite to the first motion sequence for 400 ms (Panel 3). At the end of each trial, the observer uses the mouse to move a red disc into the position where he/she perceived the target. This is typically shifted in the direction of the second motion sequence (the flash-grab effect; Panel 4).

Experiments 1A and 1C included additional trials in a different condition in which observers responded by making saccades instead of mouse clicks. These were not included in the current analysis. Experiment 1B was presented in a range of dichoptic viewing conditions, in which the first motion sequence, the second motion sequence, and the target disc could be independently presented to either or both eyes in a number of different combinations. All dichoptic conditions are collapsed in the current analysis. 

Additional details of Experiment 1C

In Experiment 1C, the annulus consisted of 18 alternating dark and light segments with a white fixation dot (diameter 0.5 degrees of visual angle [dva]), presented in the center of the screen. The annulus was presented on a gray background at 20% contrast, with an inner radius of 9.3 dva and an outer radius of 13.7 dva. The annulus rotated at an angular velocity of 200° of polar angle per second. The target stimulus was a red disc with diameter 3.12 dva, presented at the time of reversal for 10 ms (two frames) superimposed on the annulus at a radius of 11.5 dva from the fixation point. The target was always presented at the edge between a light-dark or dark-light segment at one of three locations: 160°, 180°, or 200° of polar angle offset from the top of the annulus. Overall target location, timing of reversal, and direction of motion (clockwise or counterclockwise) were randomized across trials. 

On alternating blocks, observers either made saccades to the target or reported the perceived position of the target using a mouse click after the end of the trial. Only mouse reports are analyzed here. Observers completed a total of 720 mouse-report trials each. 

There were three experimental conditions, designed to elicit different proportions of express saccades in the saccade-report blocks (the so-called “gap effect”; Saslow, 1967), but also included in the mouse-report blocks for completeness. In Condition 1, on gap trials, the fixation dot disappeared 200 ms before target presentation and annulus reversal. In Condition 2, on simultaneous trials, the fixation point disappeared at the same time as target presentation and annulus reversal. In Condition 3, on overlap trials, the fixation point remained present for the entire trial. The three trial types were randomly mixed throughout blocks, with equal frequency of each. As no effect of the manipulation was observed, the three conditions are collapsed in the current analysis. 

Results

In each experiment, we calculated flash-grab magnitude on individual trials as the difference in polar angle between the reported position of the target and the physical position at which it had been presented. Error in the direction of the second motion sequence (i.e., after reversal) was taken as positive. Mean flash-grab magnitude was calculated for individual observers as a function of time-to-target, averaging across all other experimental manipulations in Experiments 1B and 1C. Within each experiment, mean flash-grab magnitudes as a function of time-to-target within each observer were baseline-corrected by subtracting the mean flash-grab magnitude across all time-to-target conditions for that observer. Results are shown in Figure 3, plotted around the aggregate mean of all observers and all conditions. 

Figure 3

 

(A–C) Mean flash-grab magnitude as a function of time-to-target in three different datasets: Experiment 1A (A), Experiment 1B (B), and Experiment 1C (C). Open circles indicate averages across all observers within each condition. In each case, error bars indicate standard errors of the mean across all trials within that condition, after baseline correction to remove interobserver differences in mean flash-grab magnitude. The dotted line indicates a linear best-fit to the data. (D) The best-fit slopes observed in Experiments 1A through 1C, showing 95% CIs for each. The open marker (overall) indicates the best estimate of the slope across all three experiments based on a random-effects model, with its 95% CI.

To test for a systematic effect of time-to-target on baseline-corrected flash-grab magnitude, in each experiment, we entered mean flash-grab magnitudes across observers into a linear regression. This revealed a significant linear effect of time-to-target on flash grab magnitude in each experiment: 

•  
  Experiment 1A: Best-fit slope of 1.1° of polar angle per second (95% CI [0.2, 2.2]); significant with t = 3.2, p = 0.03.
•  
  Experiment 1B: Best-fit slope of 3.0° of polar angle per second (95% CI [1.9, 4.2]); significant with t = 7.33, p = 0.002.
•  
  Experiment 1C: Best-fit slope of 3.5° of polar angle per second (95% CI [1.5, 5.5]); significant with t = 5.52, p = 0.012.

Across all three experiments, the mean best-fit slope is 2.4° of polar angle per second (95% CI [0.9, 3.9]; Figure 3D). Note that the three experiments differed on relevant experimental details, including the geometry of the display, which will likely affect the precise numerical value of this relationship. Most importantly, all three studies revealed a significant positive slope relating time-to-target to flash-grab magnitude. 

Each of these experiments used the flash-grab paradigm with a uniform distribution of time-to-target durations. This means that the hazard function (the chance of a reversal occurring at a given point in the trial, given that it has not yet occurred in that trial) increases as the trial progresses. Because of this, an observer might increasingly expect the reversal as the trial progresses. This suggests that, contrary to an explanation that only deals with first order predictions, expecting the reversal increases, rather than decreases, the magnitude of the flash-grab effect. In the following three experiments, we further test this hypothesis and rule out alternative explanations. 

Experiment 2: Absolute or relative time-to-target

In Experiment 2, we further test the hypothesis that flash-grab magnitude increases with increasing time-to-target due to increasing expectation. An alternative explanation for the positive relationship observed in Experiments 1A through 1C could be adaptation: Viewing an adapting stimulus for longer typically leads to a stronger or longer aftereffect. This is true for the motion aftereffect, for example (Mather, Verstraten, & Anstis, 1998). In the motion aftereffect, adaptation to motion of a pattern in a particular direction causes a subsequently presented static pattern to appear to move in the opposite direction. Although the motion aftereffect is not thought to be the main reason for the flash-grab effect (Cavanagh & Anstis, 2013), motion adaptation has been shown to induce shifts in the perceived position of static objects (Nishida & Johnston, 1999), and therefore the motion aftereffect could conceivably be playing a role in the mislocalization observed in the flash-grab effect. To distinguish between explanations in terms of adaptation versus expectation, here we investigate whether flash-grab magnitude is determined by absolute or relative time-to-target. If flash-grab magnitude depends on expectation, it should depend on the range of time-to-target durations in a given block. For example, in a block with reversals occurring at 1,000–1,500 ms, a reversal at 1,100 ms would be relatively unexpected, whereas that same time-to-target in a block of with reversals occurring from 600–1,200 ms would be more expected. Conversely, if the relationship observed in Experiments 1A through 1C is due to increasing adaptation to the background motion, then absolute duration should determine flash-grab magnitude, and the duration relative to other trials in the experimental block (e.g., the trial context) should be irrelevant. Experiment 2 aims to dissociate these two explanations. 

Methods

Fifteen observers (age: 19–40, eight male) took part in the experiment. Stimuli were presented using identical apparatus to Experiment 1C, with the exception that eye position was not recorded. 

The experiment was designed to manipulate time-to-target in three separate conditions, to test whether relative or absolute time-to-target influenced the magnitude of the illusion. Stimuli were identical to Experiment 1C, with the exception that the time at which the target was presented varied from 600–1,900 ms after motion onset. This was a larger time range than Experiment 1, and the time-range was split into three conditions. The target was always presented at the same time as the annulus reversing direction. In Condition 1, the reversal (and target presentation) occurred at 600, 700, 800, 900, 1,000, or 1,100 ms; in Condition 2 the reversal (and target) occurred after 1,000, 1,100, 1,200, 1,300, 1,400, or 1,500 ms; and in Condition 3 the reversal (and target) happened after 1,400, 1,500, 1,600, 1,700, 1,800, or 1,900 ms. Within each block, each value of time-to-target was presented equally often. Each observer completed 664 trials in each condition. Conditions were presented in separate sessions and the order of conditions was counterbalanced across observers. 

Results

As before, the difference in polar angle between the reported position and the physical position of the disc was taken as a measure of the flash-grab effect (positive numbers indicating mislocalization in the direction of motion after the reversal). The mean magnitude of the flash-grab effect for each observer was calculated as a function of time-to-target separately in each of the three conditions. Mean flash-grab magnitudes within each observer were baseline-corrected by subtracting the mean flash-grab magnitude across all conditions for that observer. Mean flash-grab magnitudes as a function of time-to-target in each of the three conditions are shown in Figure 4. 

Figure 4

 

Mean flash-grab magnitude as a function of time-to-target in Experiment 2. Open circles represent averages across 15 observers. Error bars indicated standard errrors of the mean across observers, after baseline correction to remove interobserver difference in mean overall flash-grab magnitude. Colors differentiate trials by condition, which differed in the range of time-to-target values tested. In Condition 1 (blue), trials were presented with 600–1,100 ms time-to-target. In Condition 2 (red), trials were presented with 1,000–1,500 ms time-to-target. In Condition 3 (orange), trials were presented with 1,400–1,900 ms time-to-target. Dashed lines indicate best-fit lines for each individual condition.

First, to evaluate whether absolute time-to-target had an effect on flash-grab magnitude, we entered all datapoints across all three conditions into a linear regression. This revealed no significant relationship between the two variables (best-fit slope 0.4°/s, t = 1.44, p = 0.17; 95% CI [−0.2, 1.0]). 

Subsequently, to test whether relative time-to-target had an effect on flash-grab magnitude, we entered the datapoints from each condition in separate linear regressions. This revealed a significant increase of the illusion with time for the shortest time-to-target range (600–1100 ms; slope coefficient 4.2°/s, t = 5.62, p = 0.005; 95% CI [2.1, 6.3]). The slope coefficients for the two longer ranges, although both positive, did not reach significance (both p > 0.2). We believe this is due to the greater power in Experiments 1A through 1C, where individual data points incorporated 3,200, 2,805, and 1,728 trials respectively, as compared with 1,660 in Experiment 2. Furthermore, in Experiments 1A through 1C, observers were only presented with a single range of time-to-target values throughout the entire experiment, which would become highly learned over the course of the experiment. In Experiment 2, observers relearned a new range of time-to-target values at the start of each session. 

Finally, to test whether flash-grab magnitude could be influenced by relative time-to-target even when absolute time-to-target was held constant, we analyzed the time-to-target values where the conditions overlapped. To do so, we carried out two separate 2 × 2 repeated measures analyses of variance (ANOVA) with factors condition (Condition 1 vs. Condition 2; Condition 2 vs. Condition 3) and time-to-target (1,000 vs. 1,100; 1,400 vs. 1,500, respectively). The first ANOVA (Condition 1 vs. Condition 2) revealed a significant main effect of condition (F = 7.83, df = 1, p = 0.008) without a significant effect of time-to-target (F = 0.04, df = 1, p = 0.85). The second ANOVA (Condition 2 vs. Condition 3) did not find any significant main effects (condition: F = 0.90, df = 1, p = 0.68; time-to-target: F = 0.67, df = 1, p = 0.42). 

Altogether, the results reveal insufficient evidence to reject the (null) hypothesis that absolute time-to-target does not have an effect on flash-grab magnitude. Rather, it is relative time-to-target within blocks, particularly at shorter durations, that influences flash-grab magnitude. For the shorter durations, this is therefore consistent with the interpretation that expectation, rather than adaptation, increases the strength of the flash-grab effect. 

Experiment 3: Keeping expectations constant

Experiment 2 demonstrated that the increase of flash-grab magnitude with increasing time-to-target was not attributable to the absolute amount of time before presentation of the target (and the concurrent reversal). To directly test the hypothesis that the increasing hazard function influences flash-grab magnitude, in Experiment 3 we manipulate the shape of the distribution of time-to-target values from which trials are drawn. In the uniform condition, trials were drawn from a uniform distribution of time-to-target, in which (as in previous experiments) the probability of a target increases as one progresses further into a trial. Conversely, in the exponential condition, time-to-target values were drawn from an exponential distribution, such that the probability of a target at any given time-point, conditional upon reaching that point in the trial with no prior target, remains constant. If our hypothesis were true, the flat hazard function of the exponential condition would cause the relationship between time-to-target and flash-grab magnitude to weaken. 

Methods

Sixteen observers (age: 19–31, five male) participated in the experiment. All observers had normal or corrected-to-normal vision and gave informed consent prior to participation in the experiment. Apparatus and stimuli were identical to Experiment 2, with just one exception: The shape of the distribution from which the time-to-target was randomly drawn on a given trial. In the uniform condition, this could be one of six discrete values: 600, 700, 800, 900, 1,000, or 1,100 ms, with equal probability. In the exponential condition, time-to-target was instead drawn from a continuous exponential distribution. The mean (850 ms) and standard deviation (171 ms) was held constant across the two conditions. As before, observers reported the perceived position of a red disc flashed in one of three physical locations on the reversing annulus. All observers completed 578 trials in each of the two conditions within a single session. The order of conditions was counterbalanced across observers, and upon debriefing none of the participants reported noticing a difference between conditions. 

Results

On each trial, the difference in polar angle between the reported position and the physical position of the disc was taken as a measure of the flash-grab effect. As before, positive numbers indicated perceived displacement in the direction of motion after the reversal. Flash-grab magnitudes within each observer were baseline-corrected by subtracting the mean flash-grab magnitude across all conditions for that observer. Trials from the exponential condition were binned into six equal quantiles to match the six time-to-target values in the uniform condition. Results are shown in Figure 5, plotted around the aggregate mean of all observers and all conditions. 

Figure 5

 

Mean flash-grab magnitude as a function of time-to-target in Experiment 3, drawn from either uniform (blue) or exponential (red) distributions. Open circles indicate averages across 16 observers and error bars indicate standard errors of the mean across observers, after baseline correction to remove interobserver differences in mean flash-grab magnitude. Dashed lines indicate best-fit lines for each condition.

Mean baseline-corrected flash-grab magnitudes as a function of time-to-target across observers in each of the two conditions were entered into separate linear regressions. This showed a significant linear dependence on time-to-target in both the uniform condition (slope = 2.6°/s; 95% CI [1.0, 4.1]; t = 4.6, p = 0.01) and the exponential condition (slope = 1.7°/s; 95% CI [1.1, 2.3]; t = 7.87, p = 0.001). Importantly, when data from both conditions were entered stepwise into a single multiple linear regression, the interaction between the two conditions was significant (t = 7.4, p < 0.001), showing that the slope in the uniform condition was significantly greater than the slope in the exponential condition. 

Flattening the hazard function by drawing trials from an exponential distribution therefore reduced, but did not eliminate, the influence of time-to-target on flash-grab magnitude. This is only partially consistent with our hypothesis that temporal expectation increases the flash-grab effect, which would predict that the effect of time-to-target on flash-grab magnitude should be eliminated in the exponential condition. Upon inspection of Figure 5, it appears that the linear fit to the exponential condition is strongly driven by flash-grab magnitude in the last bin, with flash-grab magnitude relatively constant over the remaining bins. One possible (but post hoc) explanation for the residual relationship could therefore be the presence of rare trials in the exponential condition with unusually long time-to-target, which draw the observer's attention due to their long delay, thereby increasing the strength of the illusion in those trials. Altogether though, we interpret the reduced influence of time-to-target on flash-grab magnitude in the exponential condition as generally consistent with the effect of expectation. 

Experiment 4: Violation of expectations

In this final experiment, we test a final prediction of the hypothesis that temporal expectation increases the strength of the flash-grab effect, namely: If expectation increases the flash-grab effect, then an unexpected reversal should produce a small flash-grab effect. To test this prediction, in Experiment 4 we presented observers with sequences of trials with constant time-to-target, allowing them to gradually build up a precise temporal expectation of the reversal, thereby manipulating second order predictions. Then, we presented single trials with unexpectedly shorter time-to-target. The rationale behind this manipulation was that as observers view repeated trials with identical time-to-target, the temporal precision of their expectation would increase. When a trial features an unexpected, short time-to-target, temporal expectation at that time in the trial is low. In this way, the experiment would allow us to study the effects of both the gradual increase of temporal expectation with repetition, as well as the effect of an unexpected reversal. 

Methods

Ten observers (age: 18–22, three male) naive to the purposes of the experiment took part after providing informed consent. All had normal or corrected-to-normal vision. Apparatus and stimuli were identical to Experiments 2 and 3, with one exception. Trials were presented in repeated sequences of 11 trials, with the first 10 trials presented with 1,100 ms time-to-target and every 11th trial presented with 600 ms time-to-target. Sequences were presented back-to-back without interruption. Upon debriefing, all observers reported being unaware of any recurring differences between trials. At least at the level of conscious report therefore, observers did not come to expect the short time-to-target that was presented every 11th trial. Observers completed a total of seven blocks (924 trials in total) in a single session. 

Results

On each trial, the difference between the physical and reported position of the target was taken as flash-grab magnitude, with differences in the direction of the second motion sequence taken as positive. As before, flash-grab magnitudes within each observer were baseline-corrected by subtracting the mean flash-grab magnitude (across all trials) for that observer. We then calculated average flash-grab magnitude for each sequence position for each observer. Results are shown in Figure 6, plotted around the aggregate mean of all observers and all conditions. 

Figure 6

 

Mean flash-grab magnitude as a function of sequence position in Experiment 4. Observers viewed repeating sequences of 11 trials, with the first 10 trials having a constant time-to-target (plotted in black) and each 11th trial presented at an unexpectedly shorter time-to-target (plotted in red). Open circles indicate average flash-grab magnitude for each sequence position across 10 observers. Error bars indicate standard errors of the mean across observers, after baseline correction to remove interobserver difference in mean flash-grab magnitude across all conditions. The dashed line indicates a best-fit line for the first 10 trials.

To test whether flash-grab magnitude depended on sequence position, mean baseline-corrected flash-grab magnitude for each observer and each sequence position was entered into a repeated-measures ANOVA. This revealed a significant effect of sequence position (F = 3.9, df = 10, p < 0.001). To understand the nature of the effect, we subsequently entered mean baseline-corrected flash-grab magnitudes across observers for each sequence position into a multiple linear regression, with sequence position and time-to-target as regressors. This revealed that flash-grab magnitude for sequence position 11 was significantly reduced compared to sequence positions 1–10 (t = −4.68, p = 0.002). The coefficient for flash-grab magnitude across the first 10 positions was positive (0.08, 95% CI [− 0.08, 0.17]) but did not quite reach significance (t = 2.08, p = 0.070). To further investigate the robustness of the sequence dependence, we fitted linear functions to each of the individual observers, yielding 10 slope coefficients (one for each observer). A t test revealed that across all 10 observers, the mean slope coefficient across the first 10 trials was significantly greater than 0 (t = 2.98, df = 9, p = 0.016). 

Altogether, the pattern of results is consistent with our interpretation in terms of expectation: As observers view sequential trials with identical time-to-target, their temporal expectation becomes more precise, and the flash-grab magnitude gradually increases. Conversely, when the final trial in the sequence is presented with shorter time-to-target, it is not expected and the flash-grab magnitude is correspondingly reduced. 

Discussion

This flash-grab illusion is triggered when a continuously moving background reverses direction. Here we manipulated the temporal expectation of the reversal in four different experiments to see if increasing the expectation of the reversal would reduce its effect. Across the three initial experiments (two reanalyses of previously published data [van Heusden et al., 2018, van Heusden et al., 2019] and a new dataset) we found the opposite result: an increase in the magnitude of the flash-grab illusion with increasing time to the motion reversal. We then conducted three additional experiments to directly test whether the increase in the magnitude of the flash-grab effect was due to temporal expectation. In Experiment 2, we showed that flash-grab magnitude was independent of the absolute duration that elapsed before the target was presented, effectively ruling out an explanation based on a build-up of adaptation to the background motion. In Experiment 3 we showed that reducing the predictability of the reversal attenuated the illusion. Finally, in Experiment 4 we showed that illusion-strength gradually increases as temporal expectation becomes more precise and is then substantially reduced when this temporal expectation is unexpectedly violated. 

Altogether, we provide convergent evidence that temporal expectation increases the magnitude of the flash-grab effect. This indicates that the expectation of the motion reversal does not attenuate its effect, it increases it. A predictive coding account that fails to include second order predictions would fall short in trying to explain these results. Broadly speaking, the purpose of a predictive network is to minimize surprise (as operationalized by prediction error; Friston, 2018). In our data, expecting the reversal increases rather than reduces the effect of its “surprise.” To explain this pattern of results, a predictive coding framework must implement second-order predictions, which predict the reliability of the first-order predictions relative to new sensory input (Feldman & Friston, 2010). When that reliability is low, sensory input is given a higher weight. This description is functionally equivalent to temporal attention transiently boosting incoming motion signals. 

We have previously shown that the flash-grab effect can be separated into one component going backward along the initial trajectory (i.e., correction-for-extrapolation), as well as a second corrective velocity component along the new trajectory (Blom et al., 2019). Such forward shifts would be predictions of where real-world targets would be if they were actually moving. This would be functional for targeting actions at moving objects, as we have previously shown for saccades (van Heusden et al., 2018). Fully characterizing the interplay between violations of old predictions and the formation of new predictions will require further investigation. 

It is informative to compare our results to previous studies that have investigated the effect of temporal expectation on a related illusion, the flash-lag effect (Nijhawan, 1994). For example, Linares, Lopez-Moliner, and Johnston (2007) used various values of time-to-target and found that mislocalization in the flash-lag effect (FLE) was larger when the flash was presented after longer time intervals (800 ms) compared with shorter time intervals (200 ms), as we observed in Experiments 1 through 3. They suggest that the increase in the illusion might be due to the duration of the preflash trajectory, concluding that the effect evolves over time until saturating. However, that interpretation cannot explain the flash-grab results from our Experiment 2, in which the relative, rather than absolute, time-to-target influences the magnitude of the illusion. In any case, the effect of time-to-target is not consistent for the FLE. Using a slightly different paradigm, Vreven and Verghese (2005) showed the opposite pattern, with longer time-to-target resulting in a weaker FLE (see also Eagleman & Sejnowski, 2000). 

Interestingly, the results of temporal expectation in the flash-grab effect are opposite to what has previously been reported for spatial predictability (Adamian & Cavanagh, 2016). Evidently, spatial and temporal predictability have different effects on the illusion. Evidence from behavioral and event-related potential studies investigating the relative contributions of spatial and temporal expectation suggest that temporal expectation functions synergistically with spatial expectation to enhance perceptual processing (Doherty et al., 2005; Rohenkohl, Gould, Pessoa, & Nobre, 2014). Although the present study did not investigate spatial expectation, in our tasks the spatial predictability of the target could be considered high, as the target always occurred towards the bottom of the annulus. 

The most likely mechanism by which temporal expectation influences the flash-grab effect is temporal attention. Visual expectation and attention are considered closely related (Summerfield & Egner, 2009). Previous studies involving perceptual discrimination tasks have shown that expectation can guide attention to selectively attend to specific time intervals (Correa et al., 2005; Doherty et al., 2005; Summerfield & Egner, 2009). It has been shown that attention is required for the flash-grab effect (Cavanagh & Anstis, 2013). It may be that orienting temporal attention to the moment of the reversal (and the flash) increases the gain on motion processing at that instant, increasing its impact on the perceptual representation. When the reversal occurs at an unexpected time, less temporal attention would have been allocated to the background motion, and a reduction in the illusion would be expected. This explanation is consistent with the pattern of results observed in Experiment 4, in which an unexpected reversal results in the flash being perceived closer to its physical location. Under predictive coding accounts, temporally specific expectations about precision and attentional gain control are considered to be synonymous (for a detailed account, see Feldman & Friston, 2010). 

Based on human and animal recordings, one proposed mechanism by which temporal expectation might modulate attention to influence perception could include low-frequency oscillatory mechanisms, in which neuronal excitability aligns with the time at which the stimulus is expected to appear (Lakatos, Karmos, Mehta, Ulbert, & Schroeder, 2008; Rohenkohl & Nobre, 2011). The encoding of precision in predictive coding has also been attributed to oscillatory mechanisms (for a detailed account see Kanai, Komura, Shipp, & Friston, 2015). Understanding the neural mechanisms involved in temporal expectation will require more research and will provide further insight into how temporal expectation and attention modulate perception. 

To summarize, this study provides evidence that varying the temporal predictability of the reversal and flash in the flash-grab illusion modulates the magnitude of the illusion: The illusion becomes stronger as temporal expectation increases and vice versa. This shows that (at least in the case of visual motion) higher order predictions amplify, rather than attenuate, the sensory effects of an expected motion reversal. To be consistent with hierarchical predictive coding accounts, those accounts must incorporate second-order predictions about the reliability of sensory input: predictions that modulate the gain on sensory input. An alternative way of phrasing this is that temporal attention transiently boosts motion signals at the time of target presentation, in turn increasing the magnitude of the flash-grab effect. 

Acknowledgments

EvH, TB, KC, and HH were supported by the Australian Government through the Australian Research Council's Discovery Projects funding scheme (project DP180102268). PC was supported by grants from Dartmouth College and from Natural Sciences and Engineering Research Council Canada. 

Commercial relationships: none. 

Corresponding author: Kate M. Coffey. 

Email: . 

Address: Melbourne School of Psychological Sciences, The University of Melbourne, Melbourne, Australia.