Stretching time: Relativistic lag-induced shifts in perceived audiovisual synchrony using cluttered displays

Abstract

When presented with temporally displaced audiovisual events, observers shift their point of subjective simultaneity (PSS) in the direction of this prior lag. This effect, known as temporal recalibration (TR), has been inferred previously using single audiovisual events. Here we investigate TR using an audiovisual synchrony search paradigm employing spatiotemporally cluttered visual displays. By manipulating the relative modulation frequency of the adaptor (0.72 Hz) and test (0.36, 0.72 Hz) we find that following lag-adaptation, PSS shifts preserve the relative phase—not the latency—of the adapted lag. Applying this cross-frequency design to a classic simultaneity discrimination task, we find TR is unaffected by the relative frequency of adaptor and test in terms of latency rather than phase. This dissociation implies that under conditions of low spatial certainty TR obeys a relativistic (phase-conserving) temporal scaling law, whereas high spatial certainty affords PSS shifts, which operate in absolute (latency-conserving) temporal coordinates.

Introduction

In natural scenes, although multisensory information may originate from a single source, it is likely to be signaled asynchronously in primary auditory and visual cortices due to differences in the propagation speeds of sound and light, and to differences in neural processing latencies for audition and vision (Corey & Hudspeth, 1979; Lennie, 1981). It has been suggested that the human brain may compensate for such variability by shifting its point of subjective simultaneity (PSS). Indeed, mere exposure to asynchronous audiovisual events shifts observers' PSS in the direction of the prior audiovisual asynchrony. This phenomenon, known as audiovisual temporal recalibration (TR), has received burgeoning interest in recent years (Fujisaki, Shimojo, Kashino, & Nishida, 2004; Roseboom & Arnold, 2011; Van der Burg, Alais, & Cass, 2013; Vroomen, Keetels, de Gelder, & Bertelson, 2004). In these studies participants passively observe two events—usually a flash and a beep—displaced in time by a particular magnitude and sign (adapted lag stimulus onset asynchrony [SOA]). After this period of lag exposure, participants are then shown another pair of auditory and visual events whose onsets are displaced by a particular SOA (test SOA), and they must decide whether these are synchronous or not (simultaneity judgment [SJ]). By plotting the proportion of SJs as a function of test SOA, an approximately normally shaped distribution of responses emerges, the mean of which corresponds to the PSS. The signature of TR is a shift in the PSS in the direction of the prior adapted lag. 

This study aims to investigate the coding principles underlying TR by applying our recently published search method designed to measure audiovisual simultaneity performance in spatiotemporally cluttered visual environments (Van der Burg, Cass, & Alais, 2014). According to the latency hypothesis, TR is a consequence of the brain modifying the relative latencies of auditorily and visually related neural signals, thereby reducing temporal displacement evident in prior audiovisual events (Roach, Heron, Whitaker, & McGraw, 2011). An alternate possibility is that TR is a consequence of variation in the relative temporal phase of auditorily and visually evoked neural oscillations. A recent neuroimaging study found evidence that may support such a phase hypothesis, showing shifts in sound evoked steady-state oscillations as a consequence of adaptation to audiovisual asynchrony (Kösem, Gramfort, & van Wassenhove, 2014). To date, however, no evidence exists to differentiate between latency and phase-based coding schemes for TR. 

To address this question, we use periodically modulating audiovisual stimuli in both the adaptor and test stages of our experiments (Experiments 2 and 3). For periodically modulating stimuli, temporal offset (lag) can be defined in either absolute latency-based or relative phase-based coordinates (degrees). If the latency hypothesis is correct, then PSS shifts due to lag adaptation should be unrelated to the relative temporal frequency of adaptor and test. Conversely, if the phase hypothesis is correct, lag adaptation should produce shifts in PSS, which preserve phase relationships (i.e., vary as a proportion of modulation wavelength) regardless of the relative frequency of adaptor and test stages. 

Materials and methods

Participants

Eleven participants took part in the study (Experiment 1: three women, three men; M = 32 years; Experiment 2: two women, four men; M = 37 years; Experiment 3: three women, three men; M = 36 years). These sample sizes are typical in TR research. All participants reported normal or corrected-to-normal vision and normal hearing. Eight participants were naive to the purposes of the experiment and were paid for their participation, and three were the authors (JC participated in all experiments; DO in Experiment 1; EVdB in Experiments 2 and 3). The experimental procedure was explained and written consent was obtained, with participants advised they were free to terminate the experiment at any time. Participants were treated in accordance with the University of Western Sydney's Human Research Ethics Committee. 

Stimuli and apparatus

Stimuli were created using e-Prime 2.0 software (Psychology Software Tools, Inc., Sharpsburg, PA). The experiment was run on a PC and LaCie Electron 22 blue III monitor (75 Hz refresh rate). Participants were seated in a dimly lit room approximately 57cm from the monitor and wore headphones (Sennheisser HD600, Sennheisser, Wedemark, Germany). 

Adaptation period

The adaptation stimulus consisted of a circular disk (diameter = 1° visual angle), centered on fixation against a black background (<0.5 cd/m²). The disk modulated in luminance between from 8.9 to 68.8 cd/m². The auditory stimulus consisted of a 500-Hz tone (44.1-kHz sample rate) presented binaurally via headphones modulating in intensity between silence and 75 dB. Both the tone intensity and disk luminance modulated with a square-wave profile with a fundamental frequency of 0.72 Hz. 

In Experiment 1 we manipulated the relative phase of the adapting tone and disk, corresponding to SOAs of −106, 0, and +106 ms, where negative values indicate that increments in auditory modulation precede increments in luminance by 106 ms, and positive values indicate luminance increments precede tone intensity by 106 ms (Figure 1b). In Experiments 2 and 3, a single adaptation SOA was employed (+106 ms), and the location of the adapting disk was presented at fixation. For a given adaptation condition, participants were initially exposed to 120 s of continuous exposure to a particular SOA, followed by a test stimulus (see below), then a repeated sequence of re-exposure to the initial adaptation stimulus for 7 s, followed by a test stimulus. This trial cycle of adaptation followed by test procedure was repeated for a total of 100 cycles per condition per participant. Each adaptation condition was blocked within a given testing session, the order of which was randomized across testing sessions. 

Figure 1

 

Spatial and temporal properties of adaptor and test stimulus displays used in Experiment 1. (a) A centrally positioned luminance modulating disk and binaural intensity modulating tone (0.72-Hz fundamental frequency) were presented for 2 min prior to the onset of the first test stimulus at one of three different audiovisual lags: (b) tone leading by 106 ms (blue traces); synchronously (red traces); and disk leading by 106 ms (green trace). Subsequent top-up adaptor stimuli were presented prior to each test stimulus for five modulation cycles. (c) Example visual test stimulus. (d) Time course of audiovisual test stimuli. The circles plotted on the modulation traces represent the phases allocated to the disks in the display. Note that the spatial allocation of phases to disks was random, not sequential. In any given display, the luminance of each disk was modulated at a different time (i.e., with a unique temporal phase), and a single disk was physically synchronized with the tone (0° temporal phase).

Movies 1–3.

 

In all movies the adaptation stimulus (a repeatedly presented single disk and tone pairing) is followed by an example of our synchrony search task in which one must judge which of the 20 disks' luminance onset is uniquely synchronized with the tone onset. The only difference between each movie is the asynchrony presented during the adaptation period. Movie 1: 0-ms lag adaptation. Movie 2: approximately −100 ms (tone first) adaptation. Movie 3: approximately +100 ms (disk first) adaptation. The search stimulus in each case is identical. Note that these animations were recorded at 28 frames/s. Note that the audiovisual latencies presented will depend upon the specific timing properties of the audiovisual system on which it is viewed.

 

 

 

Test procedure

In Experiments 1 and 2 the test stimulus consisted of 20 luminance-defined disks (diameter = 1° visual angle) arranged equidistantly on an imaginary circle (radius = 6.8°) centered on fixation against a black background (<0.5 cd/m⁻²; see Figure 1c). A small fixation point (8-mm diameter; 68.8 cd/m²) appeared at the center of the screen and was present throughout the experiment. In Experiment 1 each disk modulated with a square-wave profile at 0.72 Hz, the same frequency as that of the adapting stimulus. In Experiment 2, audiovisual test stimuli modulated at 0.72 Hz for half of the trials (total of 100 trials), and 0.36 Hz for the remaining half (100 trials). In all experiments the test stimulus was presented for three modulation cycles per trial. The luminance modulation of each disk in the test display was assigned a unique temporal phase with respect to the tone, such that only one disk was physically synchronized with the tone. As shown in Figure 1d, the luminance increments associated with nine of the disks preceded the tone onset, and 10 occurred afterwards. This disk modulation sequence was composed of temporally equal-spaced steps (range = 360 ms prior to 400 ms following zero in 40-ms steps). On each test trial, the spatial location of each unique temporal phase was randomly assigned to the 20 disk locations. In the test phase of each trial cycle in Experiment 3 the 500-Hz tone appeared in conjunction with a single disk (6.8° to the right of fixation), both modulating abruptly at either 0.36 or 0.72 Hz with SOAs ranging from −239 to +280 ms in 40-ms steps. On each trial the observer had to indicate via a binary key press whether auditory and visual events presented in the test phase were synchronous or not. A total of 10 test trials per SOA and test frequency condition were presented to each participant in each experiment for a total of 280 trials per participant. The various test SOAs and modulation frequencies were presented in random sequence and with equal probability. No performance feedback was provided to subjects. 

Procedure

Participants were instructed to maintain fixation on the fixation dot throughout the adaptation and test phases of the experiment. In Experiments 1 and 2, on each experimental trial the test array of 20 disks was immediately replaced by the numbers 1–20, with each number centered corresponding to the location of a particular disk. The participants' task was to identify the location of the disk that was synchronized with the tone by entering a number on a numeric keypad. In Experiment 3, the single test disk was immediately replaced by the text: “1 = synchronous; 0 = asynchronous.” In all experiments the presentation of subsequent trials was contingent upon a valid key press. The order of conditions in Experiment 1 was randomized across subjects (3 adapting SOAs × 2 adapting locations) and blocked within experimental sessions. Experiment 1 was completed first, followed by Experiment 2, then Experiment 3. 

Results

Figure 2a, b plots the proportion of synchrony responses for each tone-relative disk onset time (SOA) in each of the three adaptation lag conditions (−106, 0, and 106 ms). An SOA of 0 ms refers to the single disk within a given trial's display whose luminance modulation is physically synchronized with the tone. Negative test SOA values represent disks whose luminance increments are lagged relative to the onset of the tone; positive SOAs indicate disks with luminance increments that precede increments in tone intensity. Synchrony responses in each of the six adaptation conditions were fitted with a Gaussian:  

 

 

Figure 2

 

Results of Experiment 1. (a) Proportion of synchrony responses for each of the 20 tone-relative disk onset times presented during each test display (see Figure 1c) following exposure to each of the three audiovisual lag conditions: SOA −106 (blue diamonds), 0 (red circles) and 106 ms (green squares). The abscissa represents the temporal displacement between a given disk onset and the onset of the tone: negative values, target disks lag the tone onset; positive values, target disks lead the tone onset. The onset of the auditory signal is represented by a test SOA of 0 ms. The six topmost graphs show the results for individual participants, beneath which are the same results averaged across participants. Continuous shaded curves show the best-fitting Gaussians (see Equation 1) applied to individual and participant-averaged data. (b) Mean PSS, (c) mean SD, and (d) mean amplitude derived from Equation 1 fitted to each participant's data in each of the three adaptation lag conditions. Colors in (b–c) correspond to different audiovisual adaptor lags. Blue: tone leads by 106 ms; Red: synchronous disk and tone (0 ms); Green: vision leads by 106 ms. Error bars are standard errors of means measured between subjects. Dashed lines represent unadapted Gaussian fits (a) and associated parameter estimates (b–d). Gray shaded regions are unadapted standard errors.

Figure 3

 

Temporal properties of adaptor and test stimulus displays used in Experiment 2. (a) A 0.72-Hz square-wave audiovisual adapting stimulus was presented initially for 2 min, followed by 7-s top-up prior to each subsequent test stimulus. Luminance increments preceded auditory intensity increments by +106 ms. (b) Time course of audiovisual test stimuli. Test stimuli were identical in all respects to Experiment 1 except that two test frequencies were employed (0.36 and 0.72 Hz). The 0.36-Hz stimulus presented visual stimuli with half the temporal density of the adapting stimulus.

Here, A represents the amplitude, and SD is the standard deviation corresponding to the precision of audiovisual simultaneity estimates. PSS, the key parameter of interest here, is the defined as the lag (test SOA) associated with the fitted Gaussian's peak. 

Mean estimates of PSS for each adaptation condition are plotted in Figure 2b. A highly significant main effect of adaptation lag was observed on PSS, F(2, 5) = 81.889, p < 0.001. To examine which audiovisual lags exerted shifts in PSS relative to the synchronous audiovisual adaptation condition, we conducted two separate pairwise repeated-measures t tests: All audiovisual asynchronies generated significant shifts in PSS relative to synchronous adaptation conditions: −106 ms lag: t(5) = −4.414, p = 0.007; +106 ms lag: t(5) = −9.036, p < 0.001. Separate within-subjects ANOVAs failed to find significant differences in fitted SDs or amplitudes across the different adaptation conditions, SD: F(2, 5) = 1.79, p = 0.278; A: F(2, 5) = 0.548, p = 0.616. Visual inspection of individual data in Figure 2a reveals that the comparatively large mean SD value obtained in the +106 lag adaptation was largely a consequence of poor performance from a single observer in this condition (see Subject 2 in Figure 2a). 

To examine whether the adaptation period itself had any effect on PSS (accuracy), SD (precision), or amplitude, we fitted synchrony performance measured without periods of lag adaptation (Van der Burg et al., 2014). Mean estimates of each fitted parameter are plotted in Figure 2b through d (see horizontal dashed lines). A Mann–Whitney U test failed to find a significant difference between estimates of PSS derived under unadapted and physically synchronous (0-ms lag) adaptation conditions (p = 0.792). Two separate independent samples Kruskal-Wallis tests comparing fitted estimates of Gaussian SD and A indicated that neither parameter varied across any condition tested (unadapted, −106, 0, and +106 ms lag adaptation), SD: p = 0.916; A: p = 0.840. 

This is the first demonstration of TR using an audiovisual synchrony search task within a highly cluttered visual environment (Van der Burg, et al., 2014). Expressed as a proportion of the adapted lag, PSS shifts rarely exceed 50% (Fujisaki et al., 2004; Heron, Roach, Hanson, McGraw, & Whitaker, 2012; Vroomen et al., 2004). Our experiment yields larger shifts in PSS (∼75 ms, tone leading; 120 ms, flash leading) corresponding to shifts of 70% and 115% of the adapted lag respectively. One possible reason we observe such large shifts may be that the search task itself is rather difficult. Although on any given trial one of the visual search objects was always synchronized with the tone, the temporal proximity of other objects (those with temporally adjacent modulation phase) may spuriously bind with the auditory signal. Indeed, spurious audiovisual bindings are frequently observed in spatiotemporally cluttered visual contexts when competing visual events occur within ∼80 ms of the auditory event; the so-called window of simultaneity (Van der Burg, et al., 2014; Van der Burg, Cass, Olivers, Theeuwes, & Alais, 2010; Van der Burg, Olivers, Bronkhurst, & Theeuwes, 2008). 

Experiment 2

In Experiment 2 we attempted to make the SJ task less difficult by halving the frequency of the period defining each audiovisual event in the test display, thereby increasing the temporal distance between temporally adjacent events from 40 to 80 ms (i.e., decreasing the temporal density of visual events). If the large recalibration effects we observed in Experiment 1 resulted from high task difficulty, then increasing the temporal distance between adjacent elements might be expected to reduce the magnitude of lag induced PSS shift. 

This manipulation also has implications for both the latency and phase compensation accounts of TR. According to the latency hypothesis, shifts in PSS are due to variations in the relative latency of auditory and visual sensory representation, and therefore, should be unaffected by the relative temporal frequency of adaptor and test. Conversely, if TR is due to shifts in the relative phase of auditorily and/or visually evoked oscillations, rather than their latencies, then we expect adaptation-induced shifts in PSS to increase by a factor of two when the target stimulus is half that of the adapting stimulus relative to when adaptor and test have identical frequencies. 

Results

Figure 4a plots the mean proportion of synchrony responses following 106-ms lag adaptation for each tone-relative disk onset time (SOA) for audiovisual test stimuli modulating at 0.36 and 0.72 Hz. Mean estimates of PSS at each test frequency for each individual are shown in Table 1. Figure 4b illustrates the group mean PSS at each test frequency. A two-tailed t test with test frequency as the within-subjects variable shows that PSS shifts are significantly greater at the lower temporal frequency tested: t(5) = 19.641, p < 0.001. More specifically, the test stimulus modulating at half the frequency of the adapting audiovisual stimulus generates an approximately 2 fold increase in PSS relative to adaptors and tests with the same frequency. Interestingly, when expressed in terms of relative phase rather than latency (Figure 4c, d), no significant differences in PSS are observed as a function of test frequency, t(5) = 2.273, p = 0.072. In light of the small sample size used in this study, it is possible that a significant difference in PSS may emerge as a function of test frequency in phase-centered coordinates with increased statistical power. Given that we are interested in discerning the effect of latency versus phase-based coordinate systems, we ran an additional between-subjects ANOVA to compare the magnitude of PSS shifts across test frequencies in latency and phase-based reference frames. This analysis revealed a significant interaction between these factors: F(1, 5) = 52.71, p < 0.001, confirming that PSS shifts are significantly larger when expressed in latency-centered, relative to phase-centered, coordinates. 

Figure 4

 

Results of Experiment 2. (a) Plots the group mean proportion of synchrony responses for each of the 20 tone-relative disk onsets following adaptation to a +106-ms (vision leading) audiovisual lag whose audiovisual components modulated with fundamental frequency of 0.72 Hz. The abscissa represents differences in latency between the onset of the various disks in the test display and the tone. Negative values represent disk modulations lagged relative to the onset of the tone, and positive values represent disk modulations advanced with respect to the tone. Continuous shaded curves show the best fitting Gaussians (see Equation 1) applied to participant-averaged data. Red and blue circles and curves show the proportion of disks chosen as synchronous for a given latency when the disks and tone in the test display modulated at 0.72 and 0.36 Hz, respectively. (b) Lag-adapted points of subjective simultaneity derived using 0.72- and 0.36-Hz tests (red and blue, respectively). (c) Same as (a) except that the abscissa now represents tone-disk onset variation in terms of modulation phase (relative timing coordinates), not latency (absolute timing coordinates). (d) Same as (b) except that PSS is expressed in terms of disk-tone modulation phase, not latency. Error bars are standard errors of means measured between subjects.

Table 1

 

Individual and averaged parameter estimates for Experiment 2. Note: Individual point of subjective simultaneity (PSS) and standard deviations (SD) parameter estimates for latency- and phase-based trials in 0.36- and 0.72-Hz test conditions, respectively. Values in bold text are means (M) and mean standard errors (SE) observed across subjects in each condition.

 

This positive shift in PSS observed with the lower frequency test stimulus contradicts what we would expect if the magnitude of recalibration scales with task difficulty, as reducing the temporal frequency of the test stimuli ought to make the task less difficult as fewer competing visual events would fall within the temporal window of audiovisual simultaneity (Van der Burg et al., 2014). On the contrary, in terms of relative latency, we find that SJ performance actually becomes less accurate (i.e., PSS shifts deviate further from 0-ms SOA) in the lower temporal density condition (0.36 Hz). Unlike the observed shifts in PSS, no significant differences in audiovisual search precision (Gaussian SD) were observed as a function of test frequency when expressed in either latency- or phase-based coordinates, latency: t(5) = 1.489, p = 0.197; phase: t(5) = −1.640, p = 0.162. 

Despite being less accurate in their estimates of audiovisual synchrony, participants reported finding synchrony search in the slowly modulating trial conditions (i.e., the 0.36 Hz) easier than in the fast modulating (0.72 Hz) condition. Combined with the absence of any difference in precision across these conditions, this may seem surprising. Given the absence of any performance feedback during the task, however, it is likely that subjects were unaware of their inaccuracy when making assessments of their performance. As for their precision, if we assume that the window of simultaneity is unaffected by the temporal density of the visual stimulus (Van der Burg et al., 2014), the fewer visual events that are likely to have occurred within this ∼100-ms window in the 0.36-Hz condition implies reduced competition between visual events, and might therefore explain participants' belief that the task was less difficult. 

The dependency we observe between PSS and test temporal frequency is incompatible with what we would expect if TR were due to latency compensation, which predicts equivalent adaptation lag-induced PSS shifts across different test frequencies. That adaptation-induced PSS shifts (a) increase in magnitude (in latency coordinates) when test stimuli modulate at half the rate of the adapting lag stimulus, and (b) are independent of test frequency when expressed in terms of relative phase, provides the first empirical support for the idea that TR is constrained by relative (phase-based) rather than absolute (latency-based) temporal coordinates. 

These psychophysical results are consistent with a recent steady-state magnetoencephalographic (MEG) study showing that neural oscillations evoked by phase-lagged periodic auditory and that visual stimuli become temporally aligned over time (Kösem et al., 2014). Interestingly, these shifts in phase of the evoked cortical responses were accomplished by auditorily evoked, not visually evoked, oscillations. The current study is unable to establish whether the PSS shifts we observed are due to shifts in auditory and/or visual sensory representations. 

Our methodology differs in several respects to previous studies on audiovisual temporal recalibration. Most critically, perhaps, is that our search task differs qualitatively from classical methods. Whereas the classical approach requires subjects to establish whether or not the single auditory and visual events presented within a given trial are synchronous or not (synchrony discrimination), our task required subjects to search for the disk location that is synchronous with the tone. Arguably, unlike the classic SJ task, our search task required that the observers establish a percept of (at least approximate) synchrony between the tone and at least one of the 20 simultaneously presented visual events. One strategy to perform this task might therefore be to encode audiovisual events by attending to the relative phase of each disk (or a subset). The apparent phase-based behavior of TR we observed in Experiment 2 may require such a phase-based process. 

Experiment 3

In Experiment 3 we used a classic SJ task to establish whether the phase-scaled shifts in PSS observed in Experiment 2 generalize to a classic synchrony discrimination task of the kind favored by previous TR studies. Experiment 3 was identical in all respects to Experiment 2, except that a single tone and disk were used in the test stage and participants had to decide whether these audiovisual modulation profiles were temporally aligned or not. 

Results

Figure 5a shows the proportion of SJ responses for audiovisual stimuli modulating at 0.36 and 0.72 Hz as a function of test SOA following adaptation to +106-ms audiovisual lag. Table 2 lists individual participants' fitted estimates of PSS and SD for each test frequency in both latency-based (absolute) and phase-based (relativistic) reference frames. Mean estimates of latency-based PSS are 99 and 97 ms for 0.36 and 0.72 Hz, respectively (see Table 2; Figure 5b). A repeated-measures t test comparing PSS estimates across the two test frequencies revealed no significant difference: t(5) = −0.297; p = 0.778. 

Figure 5

 

Results of Experiment 3. (a) Plots the group mean proportion of synchrony responses for each of the single disk's tone-relative SOAs following adaptation to a +106-ms (vision leading) audiovisual lag 0.72 Hz whose audiovisual components modulated with fundamental frequency of 0.72 Hz. Negative values indicate the auditory modulation was advanced relative to the visual modulation, with positive values representing visual modulations advanced relative to the tone. Continuous shaded curves show the best-fitting Gaussians (see Equation 1) applied to participant-averaged data. Red and blue circles and curves show proportion disks chosen as synchronous for a given latency when the test disk and tone modulated at 0.72 and 0.36 Hz, respectively. (b) Lag-adapted points of subjective simultaneity derived using 0.72- and 0.36-Hz tests (red and blue, respectively). (c–d) Same as (a–b) except that the abscissa now represents differences in disk-tone modulation phase, not latency. Error bars are standard errors of means measured between subjects.

Table 2

 

Individual and averaged parameter estimates for Experiment 3. Note: Individual point of subjective simultaneity (PSS) and standard deviations (SD) parameter estimates for latency- and phase-based trials in 0.36- and 0.72-Hz test conditions, respectively. Values in bold text are means (M) and mean standard errors (SE) observed across subjects in each condition.

 

Plotting these data instead as a function of relative phase of audiovisual events yields mean PSS estimates of 12.8° and 25.2° for 0.36- and 0.72-Hz test stimuli, respectively. A repeated-measures t test on these PSS estimates in phase-based coordinates yielded a highly significant effect of test frequency when expressed as a function of modulation phase: t(5) = 9.052; p < 0.001. This invariance in adapted PSS in latency coordinates (Figure 5a, b) combined with a strong dependency with respect to relative phase (Figure 5c, d) is what we would expect if recalibration were due to latency and not phase compensation. Unlike Experiment 2, here the fitted estimates of task precision (SD) were significantly narrower in 0.36-Hz relative to the 0.72-Hz condition in both latency-based, t(5) = 7.133, p = 0.001, and phase-based, t(5) = 11.301, p < 0.001, temporal coordinates. 

General discussion

This study shows that adapting to a particular audiovisual lag distorts perception of event timing such that observers' PSS shifts in the direction of the adapted lag. This lag-induced TR is evident in both our synchrony search task (Experiments 1 and 2) as well as a binary simultaneity judgment task (Experiment 3) similar to that used previously (Fujisaki et al., 2004; Vroomen et al., 2004). Interestingly, our cross-frequency design reveals a previously unobserved dissociation between the magnitude of recalibration and the task used to measure PSS. When performing synchrony search, participants systematically selected audiovisual events as being synchronous when they shared a similar phase relationship to the adapting stimulus, regardless of the relative modulation frequency of adaptor and test (Experiment 2). That PSS shifts preserve the audiovisual phase relationship of the adapting stimulus rather than its latency relationship implies that this form of TR operates within a relativistic rather than an absolute temporal reference frame. 

By contrast, when making binary discrimination judgments regarding the synchrony (or asynchrony) of a single flash and tone (Experiment 3), PSS shifts showed a strong dependency with test frequency when expressed in terms of audiovisual modulation phase, not onset latency. Unlike the recalibration effects observed in Experiment 2 using synchrony search, the PSS shifts inferred using the more classic synchrony discrimination task (Experiment 3) are consistent with an absolute (latency-preserving) temporal reference frame. 

In addition to dissociations between the effect of different reference frames on our measure of accuracy (PSS) in Experiments 2 and 3, our precision measure (SD) behaves differently again. Specifically, whereas no frequency-dependent SD effects were observed in our search task, simultaneity discrimination judgments (Experiment 3) were more precise at the lower frequency tested. These differences in performance precision were evident regardless of the reference frame used. 

These results imply new constraints for models of TR. To date the most sophisticated model encodes latency via a population of neurons each sensitive to a band-limited subset of audiovisual delays (Roach et al., 2011). Our results indicate that such a model is valid only in contexts of low spatial uncertainty (Experiment 3) and cannot account for the phase-invariant behavior observed in Experiment 2. Whether their delay-line model requires modification to incorporate the attentional demands of the task or whether an altogether independent system is required to encode phase is not yet clear. 

Both the phase and latency scaling of the recalibration effects we observe psychophysically are in keeping with the results of Kösem et al. (2014), who found shifts in alignment of audiovisually evoked steady-state MEG oscillations consistent with the direction and magnitude of recalibration observed here. Indeed such shifts, whether psychophysical or cortically evoked, can be expressed in terms of both latency and phase. Future physiological studies might benefit from the cross-frequency design employed here to determine whether their observed shifts in auditory cortical oscillations vary as a function of adaptor-test modulation wavelength (phase) or latency. 

An alternative, though not necessarily mutually exclusive possibility is that the manifestations of TR observed in this study, whether it be phase- or latency-invariant, are the result of adaptation-induced shifts in decisional criteria regarding which particular set audiovisual lags are to be deemed “simultaneous” (Yarrow, Jahn, Durant, & Arnold, 2011). Why such criterion shifts should scale with phase in certain cases, and latency in others is unknown, although it is conceivable that this may reflect different strategies demanded of synchrony search, on the one hand (Experiments 1 and 2), and simultaneity discrimination on the other (Experiment 3). 

A possible explanation for the phase-invariant results of Experiment 2 is that subjects might adopt a search strategy, which preserves the rhythmic structure of the adapting and test SOAs rather than latency differences per se. This idea is directly analogous to musical performance, which requires that performers preserve rhythmic structure (i.e., temporal phase) despite variations in tempo. In the case of musical rhythm, slowing the tempo by a factor of two requires the performer to increase the latency of consecutive notes (or beats) by a factor of two. Equivalently, in Experiment 2 we found that halving the modulation rate of the test search arrays (equivalent to decreasing the tempo by a factor of two) produced a 2-fold increase in the PSS latency, while approximating the phase relationship of the adapting SOAs regardless of modulation rate. Whether such conservation of rhythmic structure—audiovisual synchrony search or musical performance—relies on strategic bias at a decisional or an earlier sensory level of analysis, or whether it involves neural mechanisms, which operate in phase-based rather than latency-based coordinates, remain open questions. The regular periodic modulation of our stimuli may indeed facilitate the preservation of rhythmic (phase) structure of the adapting asynchrony. 

It should also be noted that the square-wave modulation profiles used in this study are unusual in the study of temporal recalibration, which typically employs (periodic or aperiodic) transient impulse trains. The extent to which our results are contingent upon this difference is unknown. One reason to believe that the modulation profile might be an important factor is the large magnitude of (latency contingent) PSS shifts we observed in Experiment 3, which are at least 25% larger than those reported previously (Fujisaki et al., 2004; Heron et al., 2012; Vroomen et al., 2004) using an otherwise equivalent simultaneity discrimination paradigm. To investigate these possibilities, future studies might consider manipulating the temporal regularity and/or duty cycle of adapting and test modulation. 

The extent to which these various strategies and stimulus factors may relate to the degree of spatial and/or temporal uncertainty implicit in our tasks is also unknown. It is conceivable, for example, that the diffuse attention required during synchrony search encourages recalibration in phase-based coordinates, while focused attention (classic simultaneity judgment) favors a latency-based reference frame. It will be for future studies to examine whether this behavior is specific to audiovisual TR or whether it generalizes to other sensory and/or motor domains. 

Acknowledgments

This work was funded by Discovery Projects awarded to JC (DP120101474) and EVdB (DE130101663) by the Australian Research Council. 

Commercial relationships: none. 

Corresponding author: John Cass. 

Email: [email protected]. 

Address: School of Social Sciences and Psychology, University of Western Sydney, Milperra, Australia.