Remapping time across space

Abstract

Abstract

Abstract:

Abstract  Multiple lines of evidence indicate that visual attention's temporal properties differ between the left and right visual fields (LVF and RVF). Notably, recent electroencephalograph recordings indicate that event-related potentials peak earlier for LVF than for RVF targets on bilateral-stream rapid serial visual presentation (RSVP) identification tasks. Might this hastened neural response render LVF targets perceptually available sooner than RVF targets? If so, how might the visual system reconcile these timing differences to estimate simultaneity across the LVF and RVF? We approached these questions by presenting bilateral-stream RSVP displays that contained opposite-hemifield targets and requiring participants to judge both the targets' temporal order and simultaneity. The temporal order judgments (TOJs) revealed that participants perceived LVF targets ∼134 ms sooner than RVF targets. This LVF hastening approximates a full cycle of visual attention's canonical ∼10 Hz (∼100 ms) temporal resolution. In contrast, performance on the simultaneity task did not exhibit the LVF hastening observed on the TOJ task, despite identical retinal stimulation across the two tasks. This finding rules out a stimulus-driven “bottom-up” explanation for the task-specific behavior. Moreover, error patterns across the two tasks revealed that, within the decision stage of simultaneity judgments, participants remapped LVF targets, but not RVF targets, to a later time in the RSVP sequence. Such hemifield-specific remapping would effectively compensate for the cross-hemifield asymmetries in neural response latencies that could otherwise impair simultaneity estimates.

Introduction

Neural computational work by John Tsotsos (1990) prompted Jeremy Wolfe to conclude that “processing everything, everywhere, all at once requires a brain that will not fit in the human head” (Wolfe et al., 2006, p. 177). Evolution's response to this neural computational limit is attention—the selection of sensory information. Attention selects—or samples—sensory information more slowly than our visual system samples the environment. For example, humans can discriminate a continuous light from one flickering at 30–50 Hz (Kelly, 1961; Rovamo & Raninen, 1984; Andrews, White, Binder, & Purves, 1996), yet our ability to select a flickering source's individual phase components for further analysis is limited to just ∼10 Hz (Rogers-Ramachandran & Ramachandran, 1998; Verstraten, Cavanagh, & Labianca, 2000; Battelli, Cavanagh, Martini, & Barton, 2003; Aghdaee & Cavanagh, 2007). This difference highlights the distinction between the temporal resolution of stimulus-driven vision versus that of visual attention. Visual attention's comparatively low temporal resolution (∼10 Hz) has important consequences in everyday life. For instance, while driving at 70 miles per hour (112.65 km/h) visual attention's ∼10 Hz temporal resolution permits only one sample every 10.27 feet (3.13 m), potentially a life and death difference in a crash. Here, we investigated visual attention's temporal resolution by requiring participants to judge the relative timing of two targets embedded amid distractors in rapid serial visual presentation (RSVP) displays. 

Prior RSVP experiments have demonstrated that participants often incorrectly report the temporal order of two targets separated by ∼100 ms, i.e., approximately one cycle of visual attention's 10 Hz temporal resolution (Reeves & Sperling, 1986; Chun & Potter, 1995; Chun, 1997; Isaak, Shapiro, & Martin, 1999; Spalek, Falcon, & Di Lollo, 2006; Akyürek, Riddell, Toffanin, & Hommel, 2007; Olivers, Hilkenmeier, & Scharlau, 2011; Spalek, Lagroix, Yanko, & Di Lollo, 2012). Accuracy on these temporal order judgment (TOJ) tasks improves at larger target asynchronies. The relatively low TOJ accuracy at the ∼100 ms asynchrony could be explained by either of two hypotheses that Olivers et al. (2011) recently suggested. One hypothesis maintains that at brief (∼100 ms) asynchronies the two targets merge into a single attentional episode, rendering the physically sequential targets perceptually simultaneous. An alternative hypothesis allows for the possibility that participants perceive the targets sequentially but that the perceived temporal order can be reversed if the second target receives more attention than the first. This alternative hypothesis is physiologically plausible as recent single-cell and local field potentials recorded from the primate visual cortex demonstrate that attention reduces neural response latencies and increases neural contrast gain and response gain (Hudson, Schiff, Victor, & Purpura, 2009; Sundberg, Mitchell, Gawne, & Reynolds, 2012). Sundberg et al. (2012) “speculate that latencies may be reduced at higher contrast because stronger stimulus inputs drive neurons more rapidly to spiking threshold, while attention may reduce latencies by placing neurons in a more depolarized state closer to threshold before stimulus onset” (p. 16040). Although further experiments are needed to determine which of Olivers et al.'s hypotheses better explains TOJ errors at brief target asynchronies, both hypotheses posit that this phenomenon reflects the influence of visual attention. 

Visual attention's influence depends strongly on lateral hemifield. Behavioral data demonstrate this in two ways. First, several studies have revealed a bilateral attentional advantage, i.e., better performance for targets distributed across the left and right visual fields than for targets restricted to just one lateral hemifield (Awh & Pashler, 2000; Alvarez & Cavanagh, 2005; Delvenne, 2005; Chakravarthi & Cavanagh, 2009; Reardon, Kelly, & Matthews, 2009; Delvenne, Castronovo, Demeyere, & Humphreys, 2011).¹ This bilateral attentional advantage suggests independence between the neural resources that govern attention to the left visual field (LVF) and those governing attention to the right visual field (RVF). Second, tasks that push the limits of attention's temporal resolution often demonstrate significantly better performance for LVF- than for RVF-attended targets. Specifically, LVF attentional advantages have been documented on dual-stream RSVP letter identification tasks (Holländer, Corballis, & Hamm, 2005; Scalf, Banich, Kramer, Kunjan, & Simon, 2007; Verleger et al., 2009; Smigasiewicz et al., 2010; Verleger et al., 2010; Verleger, Smigasiewicz, & Möller, 2011; Verleger, Dittmer, & Smigasiewicz, 2013) and random-dot motion discrimination tasks (Bosworth, Petrich, & Dobkins, 2012) and when judging the simultaneity of dynamically varying Gabor patches (Kelly & Matthews, 2011; Matthews, Vawter, & Kelly, 2012). Critically, the same dynamically varying Gabor patches generated no hemifield effects when the same participants judged spatial frequency differences rather than simultaneity (Kelly & Matthews, 2011; Matthews et al., 2012). Together, these behavioral studies support the possibility that the neural resources governing attention to the LVF may be particularly specialized for fine temporal tasks. 

Consistent with this behavioral evidence, physiological manipulations involving transcranial magnetic stimulation (TMS) (Müri et al., 2002; Woo, Kim, & Lee, 2009) and clinical reports of split brain (Forster, Corballis, & Corballis, 2000) and right parietal lobe patients (Battelli et al., 2001; Battelli et al., 2003) suggest that the right parietal lobe may be specialized for temporal judgments. Indeed, Rorden, Mattingley, Karnath, and Driver (1997) found impaired TOJs in patients with left-sided visual extinction caused by right parietal lobe damage. Such findings have led to speculation about a “when” pathway (Battelli, Pascual-Leone, & Cavanagh, 2007; Battelli, Walsh, Pascual-Leone, & Cavanagh, 2008; Davis, Christie, & Rorden, 2009) that is distinct from the “what” (ventral) and “where” (dorsal) pathways (Mishkin & Ungerleider, 1982). Further support for a right parietal lobe “when” pathway comes from event-related potential (ERP) data very relevant to the present psychophysical study. Specifically, Verleger et al. (2011) and Verleger et al. (2013) found that N2pc (parietal contralateral) ERPs—a marker of selective attention (Hopf et al., 2000)—peaked ∼50 ms earlier for LVF than for RVF targets on a dual-stream RSVP identification task. 

Verleger et al.'s (2011) and Verleger et al.'s (2013) hastened ERPs to LVF targets raise important perceptual questions. Are LVF targets perceived sooner than RVF targets? If so, how might the visual system reconcile these timing differences to estimate simultaneity across the LVF and RVF? We approached these questions by presenting dual-stream RSVP displays similar to Verleger et al.'s (2011) and Verleger et al.'s (2013) and requiring each participant to judge both the targets' temporal order and simultaneity. Two surprising findings emerged. First, the TOJ task revealed that participants perceived the LVF targets ∼134 ms sooner than the RVF targets. This LVF hastening approximates a full cycle of visual attention's canonical ∼10 Hz (∼100 ms) temporal resolution (Verstraten et al., 2000; VanRullen & Dubois, 2011). Second, the simultaneity judgments exhibited error patterns that indicated participants remapped LVF targets, but not RVF targets, to a later time in the RSVP sequence. Such hemifield-specific remapping would effectively compensate for the cross-hemifield asymmetries in neural response latencies that could otherwise impair simultaneity estimates. 

Method

Participants

Denison University's Human Subject Committee approved the experiments in this study, which were conducted with the understanding and written consent of each participant. Twenty-three Denison University undergraduates with normal or corrected-to-normal vision completed the study. All were naïve regarding the purpose of the experiment, and several expressed surprise when lateral hemifield effects were mentioned during the debriefing. We did not ask our participants to indicate handedness. Although Efron (1963) reported that handedness was correlated with hemifield differences in simultaneity and TOJs, subsequent and more complete psychophysical (Newman & Albino, 1977) and fMRI (Badzakova-Trajkov, Häberling, Roberts, & Corballis, 2010) findings have argued against this handedness effect. 

Apparatus

Experiments were conducted on Dell OptiPlex780 desktop computers, each with a Microsoft Windows 7 Enterprise operating system. SuperLab 4.5 presentation software (Cedrus) controlled 17-in (43.18-cm) flat screen Dell 2009W displays, each with a 60-Hz vertical refresh rate and 1680 × 1050 spatial resolution. Although head position was not stabilized, viewing distance was typically 57 cm from the monitor. 

Stimuli

The stimulus on each trial was a dual-stream RSVP sequence (Figure 1 and Movie 1). Each sequence comprised 40 15-Hz frames (67 ms/frame; 2.667 s total), containing a black fixation cross (0.5° × 0.5°) centered in a white surround. Across each 40-frame sequence, 20 bilateral stimulus pairs were presented, one on each odd-numbered frame. Even-numbered frames contained only the fixation cross. Consequently, visual transients occurred at 15 Hz (every 67 ms), but new stimulus information occurred at 7.5 Hz (every 133 ms). Within each new stimulus pair, 3.5° separated (center-to-center, horizontally) the fixation cross from each laterally flanking stimulus. The flanking stimuli were either Arabic numbers or capitalized English letters (Calibri font, stroke-width 0.02°) extending 2.0° vertically and a maximal 2.0° horizontally. 

Figure 1

 

Each RSVP sequence contained 20 bilaterally presented stimulus pairs that included black letter distractors and two targets—an exogenous (salient) red letter and an endogenous (less salient) black digit. The two targets were presented in opposite lateral hemifields, either synchronously or at various asynchronies. The sequence above schematizes an exogenous RVF red letter (“D,” shown here in white) preceding an endogenous LVF black digit (“3”) by 268 ms. Participants judged the targets' temporal order (“letter first” vs. “number first”) in one session and the targets' simultaneity (“same” time vs. “different” times) in a separate session. For the schematic above, the correct TOJ and simultaneity responses are, respectively, “letter first,” and “different.” Movie 1 shows a sample RSVP trial.

 

Movie 1.

 

A sample RSVP trial.

Each RSVP sequence contained two targets (one in each lateral hemifield) and 38 distractors (19 in each lateral hemifield). One of the targets was an endogenous² (less salient) black digit, randomly selected in each trial from the set 1, 2, 3, 4, 5, and 6. The other target was an exogenous red letter, randomly selected in each trial from the set D, F, G, J, K, and L. The distractors comprised all other letters, randomly sequenced, and always presented in black. 

Tasks

Each participant completed a simultaneity task and a TOJ task. The simultaneity task required participants to report whether the exogenous (red letter) target and the endogenous (black number) target flashed at the “same” time or at “different” times. The TOJ task required participants to report which of the two targets flashed first, “letter” versus “number.” Note that a motor response bias favoring either response option would be unrelated to the targets' lateral hemifield configuration—the variable of interest. This precaution eliminated motor response biases as an explanation for any subsequently observed lateral hemifield effects. Participants responded on a standard computer keyboard with no restrictions on which finger or hand to use. 

Procedure

TOJs occurred on one day, and simultaneity judgments occurred on a different day, counter-balanced across participants. Retinal stimulation remained identical across the two sessions. Within each daily session, each participant completed 22 practice trials, followed by five 120-trial blocks (600 trials for analysis). Each 120-trial block comprised 40 synchronized trials and 80 asynchronized trials. The 80 asynchronized trials comprised four instances each of 20 experimental conditions. These comprised five target asynchronies (133, 267, 400, 533, and 667 ms) crossed with two exogenous-target-hemifield configurations (hereafter, LVF-exo vs. RVF-exo) crossed with two target orders (hereafter, Exo-first vs. Endo-first). Among the 40 synchronized trials, half contained LVF-exo (RVF-endo) targets, and half contained RVF-exo (LVF-endo) targets. All trials contained at least one target in the 11th stimulus pair, which occurred 1.333 s into the RSVP sequence. All conditions were randomized anew within each 120-trial block. 

On the simultaneity task, participants were informed initially—and subsequently reminded between trials blocks—that asynchronized and synchronized trials would occur in a 2:1 ratio. On the TOJ task, participants were informed that “letter-first” and “number-first” trials would occur equally often. To maintain motivation on each task, immediate visual feedback identified each response as correct or incorrect. “Letter-first” and “number-first” responses on TOJ trials with synchronized targets were not objectively classifiable as correct or incorrect. Consequently, half of the synchronized TOJ trials were predesignated arbitrarily as “letter first” and half as “number first” to avoid biasing participants' responses with feedback. 

Data analysis

On the TOJ task, we assessed how hemifield affected the perception of relative target timing by constructing psychometric functions separately for the LVF- and RVF-exo target conditions. Within each of those conditions, the psychometric function's ordinate reflected the proportion of “letter-first” responses, i.e., Exo-target-first responses. The abscissa comprised target asynchronies ranging between −667 (Endo-target first) and +667 (Exo-target first) ms in 133-ms steps. A least-squares procedure was then used to fit the data with a sigmoid of the form  

 

where K and Xo determine the slope and midpoint of the sigmoid, respectively. The correlation between the best fitting sigmoid and the data, as indexed by the Pearson correlation coefficient (r), was statistically significant (p < 0.05) within each target-hemifield condition for each participant. Because each fit was significant, it was possible to use Xo from the sigmoidal fits to determine each participant's point of subjective equality (PSE). The PSE corresponds to the temporal asynchrony at which the participant made “letter-first” and “number-first” responses equally often, i.e., 0.5 on the ordinate.³ The temporal resolution of our PSE algorithm was 0.53 ms (1886 Hz). We derived this value in a simulation that assumed 51 rather than 50 “letter-first” responses in the 100 synchronized trials and nonbiased responding on the 200 asynchronized trials. In addition to the PSEs, which indexed time perception, we also computed the 75% just noticeable differences, which indexed time discrimination. These discrimination thresholds were defined as half the stimulus (target-asynchrony) change required to alter the response rate from 0.25 to 0.75. 

On the simultaneity task, we assessed how hemifield affected the perception of relative target timing by computing, separately for the LVF- and RVF-exo target conditions, the criterion value according to signal detection procedures (Green & Swets, 1966). Conceptually, the criterion value reflects the decision boundary between the two response options (the black vertical line in Figure 2). “Same” versus “different” responses on our simultaneity task occurred, respectively, when the level of sensory evidence for asynchronized targets fell below versus exceeded the criterion value. Computationally, the criterion value was derived from the proportion of hits and false alarms, respectively defined here as correct and incorrect “different” responses. Specifically, we z-transformed the proportions of hits and false alarms, then computed the following likelihood ratio:  

 

 

Figure 2

 

The criterion from signal detection theory. On our simultaneity task, asynchronized targets occurred twice as often as synchronized targets, reflected here by the relative heights of the corresponding probability distributions (Wickens, 2002). The neutral criterion (vertical black line) occurs at the sensory evidence level at which the likelihood ratio of the asynchronized-to-synchronized distributions (the ratio of y-axis values) matched our 2:1 asynchronized-to-synchronized target ratio. Criterion values shifted left and right of this neutral point respectively generate lower and higher likelihood ratios, which correspond to biases favoring the “different” and “same” responses, respectively.

Likelihood ratios determined in this manner indexed the participant's PSE in each condition. PSE values equaling two reflect “different”-to-“same” response ratios in proportion to the 2:1 ratio of asynchronized-to-synchronized trials, i.e., nonbiased responding. Higher and lower PSEs reflect overusage of the “same” and “different” response options, respectively. In addition to the PSEs, which indexed time perception, we also computed d', which indexed time discrimination. Conceptually, d' reflects the horizontal separation (peak-to-peak) between the synchronized- and asynchronized-target probability distributions in Figure 2. Computationally, d' corresponds to (zHits) – (zFalse Alarms) with d' = 0.6745 reflecting the nonbiased 75% discrimination threshold. 

Results

Figure 3 shows the primary result from our TOJ experiment—a hastened perception of LVF targets. Specifically, the hatched gray ovals reveal that—at brief target asynchronies—“letter-first” response rates in the LVF (blue circles) matched those occurring ∼133 ms later in the RVF (red squares). Likewise, the dotted arrows drawn downward from the equal-probability line (0.50) reveal that LVF and RVF targets generated PSEs, respectively, corresponding to 83 ms before and 51 ms after physical equality. These PSEs are plotted in Figure 4. The hemifield difference indicates that, on average, participants perceived LVF targets ∼134 ms earlier than RVF targets, t(22) = 5.689, p = 0.00001, η_p = 0.595; data from individual participants are available in the supplementary information. Note that this significant LVF perceptual hastening occurred despite error feedback that equally reinforced LVF-first and RVF-first responses. Indeed, our nonbiased feedback offered cues to an optimal distribution of left-first and right-first responses, yet participants used the feedback suboptimally by favoring the left-target-first response. 

Figure 3

 

Temporal order judgment psychometric functions. Participants reported which target appeared first, the exogenous letter target versus the endogenous number target. Positive x-axis values reflect trials in which the exogenous target flashed first. Negative x-axis values reflect trials in which the exogenous target flashed second. Blue circles and the blue curve (model) represent LVF-exo target trials. Red squares and the red curve (model) represent RVF-exo target trials. Error bars reflect +1 SEM (N = 23).

Figure 4

 

Points of subjective equality on the TOJ task. Error bars reflect +1 SEM (N = 23).

The 134 ms LVF hastening is a large effect, approximating a full cycle of visual attention's canonical ∼10 Hz (∼100 ms) temporal resolution (Verstraten et al., 2000; VanRullen & Dubois, 2011). A temporal discrepancy of this magnitude raises an important question. How might the 134-ms LVF hastening influence simultaneity judgments across the lateral hemifields? The most parsimonious possibility would be to assume that simultaneity judgments and TOJs are governed by the same neural events. To the extent this were true, one would expect simultaneity judgments to exhibit the same LVF hastening observed on the TOJ task. This predicts that the proportion of “different” responses on the simultaneity task would be lowest, i.e., perceived simultaneity would be greatest, whenever LVF targets physically lag 133 ms behind RVF targets. The corresponding target asynchronies would be +133 ms on RVF-exo trials (when LVF-endo targets come second) and −133 ms on LVF-exo trials (when LVF-exo targets come second). Pink and light blue arrows, respectively, for RVF- and LVF-exo trials, show these predicted target asynchronies in Figure 5, to which we now turn. 

Figure 5

 

Simultaneity task results. Participants reported whether the exogenous letter target and the endogenous number target appeared at the same time or at different times. The symbols and x-axis values are the same as in Figure 3. Arrows in pink (RVF-exo trials) and light blue (LVF-exo trials) indicate the predicted target asynchronies at which “different” responses would be lowest if the LVF hastening that affected temporal-order judgments also affected simultaneity judgments. The black arrow indicates the target asynchrony at which “different” responses were actually lowest. Error bars reflect +1 SEM (N = 23).

Data from the simultaneity task disconfirm the LVF hastening predicted from the TOJ task. As Figure 5 reveals, the proportion of “different” responses was lowest when the two targets were physically synchronized (black arrow, 0 ms asynchrony), not at the predicted asynchronies (pink and light blue arrows, ±133 ms asynchronies). Critically, because retinal stimulation remained identical across the TOJ and simultaneity tasks, factors beyond the stimulus-driven “bottom-up” neural response must have caused the difference between the observed and predicted outcomes. This implies that the TOJ and simultaneity tasks are not governed by the same decision rule. 

Information about the decision rule on the simultaneity task is revealed by comparing performance across the two smallest target asynchronies. Pairwise comparison between the LVF-exo trials (blue circles) at plus versus minus 133 ms reveals significantly more “different” responses, t(22) = 4.376, p = 0.0002, η_p = 0.465, when LVF-exo targets came second (−133 ms; M = 68.478; SE = 2.837) than when LVF-exo targets came first (+133 ms; M = 51.087; SE = 3.913). This pattern is exactly opposite to what would be predicted from the LVF hastening observed on the TOJ task. That is, LVF hastening should have reduced “different” responses when LVF-exo targets came second (−133 ms) and should have increased “different” responses when LVF-exo targets came first (+133 ms). The observed pattern not only disconfirms LVF-exo target hastening, it also reveals that participants performed the equivalent of remapping LVF-exo targets to a later point in the RSVP sequence. Critically, this remapping was specific to LVF-exo targets and did not occur for RVF-exo targets. Pairwise comparison between the RVF-exo trials (red squares) at plus versus minus 133 ms reveals significantly more “different” responses, t(22) = 6.359, p < 0.0001, η_p = 0.648, when RVF-exo targets came second (−133 ms; M = 64.565; SE = 2.433) than when RVF-exo targets came first (+133 ms; M = 39.130; SE = 3.439). That difference is consistent with the LVF hastening observed on the TOJ task. Overall, the data shown in Figure 5 reveal that, within the decision stage of simultaneity judgments, participants effectively remapped LVF-exo targets, but not RVF-exo targets, to a later time in the RSVP sequence. In the General discussion, we model this remapping by adding a delay to the time at which LVF-exo targets are estimated to arrive at the decision stage of the simultaneity task. 

Further information about the decision stage of the simultaneity task arises from another aspect seen in Figure 5. Specifically, LVF and RVF response rates (blue circles and red squares) generally overlapped except when the targets were synchronized (0 ms), t(22) = 3.152, p = 0.005, η_p = 0.311 and when the exogenous-letter target led by the briefest asynchrony (+133 ms), t(22) = 2.295, p = 0.032, η_p = 0.193. At each of those temporal intervals, participants made “different” responses significantly less often in RVF-exo trials (red squares) than in LVF-exo trials (blue circles). This suggests longer PSEs, i.e., requiring larger temporal asynchronies to make a “different” response in RVF-exo trials than in LVF-exo trials. 

To more formally quantify such hemifield-specific criterion shifts, we computed PSEs separately for RVF- and LVF-exo trials. These appear in Figure 6, separately for endo-target–first trials (top panel) and exo-target–first trials (bottom panel). Here, the ordinates represent likelihood ratios based on signal detection theory (Green & Swets, 1966). On our simultaneity task, lower likelihood ratios reflect more frequent use of the “different” response; higher likelihood ratios reflect more frequent use of the “same” response (see Method). Figure 6 reveals that participants perceived the targets' relative timing as “same” more frequently (higher likelihood ratios) on RVF-exo trials than on LVF-exo trials with nonoverlapping likelihood ratios occurring at the briefest asynchrony (133 ms). The PSEs—shown by the arrows drawn downward from the point at which each Weibull function intersects the neutral likelihood ratio of two—were also longer for RVF- than for LVF-exo trials. These PSEs are plotted in Figure 7. Relative to PSEs in the LVF-exo condition (blue bars), RVF-exo PSEs (red bars) increased by ∼55 ms when the endo target physically led and by ∼75 ms when the exo target physically led. Therefore, regardless of whether endo or exo targets physically led, participants demanded greater sensory evidence (longer target asynchronies) on RVF- than LVF-exo trials to report that the targets differed in time. In short, Figures 6 and 7 demonstrate compelling evidence for hemifield-dependent PSE shifts in our simultaneity task. In the General discussion, we model this hemifield-dependent PSE shift on RVF-exo trials by increasing the criterion (i.e., the temporal asynchrony required to make a “different” response) within the simultaneity task's decision rule. 

Figure 6

 

Simultaneity task likelihood ratios. The symbols are the same as in Figure 5. The point of subjective equality (PSE) on the simultaneity task occurred when the likelihood ratio equaled two because asynchronized targets were presented twice as often as simultaneous targets. Likelihood ratios greater and less than two respectively reflect biases favoring “same” and “different” responses. Error bars reflect ±1 SEM (N = 23).

Figure 7

 

Simultaneity task PSEs, expressed in milliseconds. PSEs were greater for RVF- than LVF-exo trials, regardless of whether endo or exo targets were shown first.

Movie 2 shows the percentage of errors that occurred in all experimental conditions tested here. Visual inspection reveals that error rates on both the TOJ and simultaneity tasks tended to decrease when target asynchronies exceeded 133 ms as one would expect. By contrast, two counterintuitive phenomena can be observed in the data near the abscissa's center. First, on the simultaneity task, there was a significant crossover interaction between exo-hemifield and target asynchrony, F(1, 22) = 7.715, p = 0.011, η_p = 0.260 (see diagonal green ovals within Movie 2). Specifically, LVF-exo errors exceeded RVF-exo errors when the targets were physically synchronized, but the reverse error pattern occurred when the exogenous target led by 133 ms. This interaction is what one would expect if, to make a “different” response on the simultaneity task, participants demanded significantly larger target asynchronies (i.e., longer PSEs) on RVF-exo trials than on LVF-exo trials. Second, when the exogenous target led by 133 ms (see vertical gray oval within Movie 2), simultaneity-task errors significantly exceeded TOJ-task errors within each hemifield condition, t(22) > 6.240, p < 0.001, η_p > 0.639. This is counterintuitive because discerning the targets' temporal order would seem to require greater time-specificity than discerning whether the targets were asynchronized at all. Additionally, this error pattern did not occur when the endogenous target led by 133 ms (data points at −133 ms). 

 

Movie 2.

 

Blue and red symbols respectively indicate LVF- and RVF-exo trials. Closed and open circles respectively indicate simultaneity-task trials with asynchronized versus synchronized targets. Triangles indicate TOJ task trials. At the 0-ms asynchrony, TOJ trials are omitted because “letter-first” and “target-first” responses cannot be classified as correct or incorrect. Diagonal green ovals mark the crossover interaction on the simultaneity task. The vertical gray oval indicates the target asynchrony at which participants made fewer TOJ errors than simultaneity-judgment errors.

The different pattern of errors associated with the TOJ and simultaneity tasks is made visually apparent by the solid and dotted lines connecting the plus and minus 133 ms target-asynchrony data points in Figure 8. A repeated-measures ANOVA on task (TOJ, simultaneity), Exo-hemifield (LVF, RVF), and target order (Endo-first, Exo-first) revealed a significant three-way interaction, F(1, 22) = 11.991, p = 0.002, η_p = 0.353). Separate ANOVAs within each task revealed an Exo-hemifield by target-order interaction on the TOJ task, F(1, 22) = 34.601, p < 0.001, η_p = 0.611, but no interaction on the simultaneity task, F(1, 22) = 1.564, p = 0.224, η_p = 0.066.These distinct task-specific error patterns occurred despite identical retinal stimulation across the two tasks, thereby ruling out stimulus-driven “bottom-up” neural responses as the cause. In the General discussion, we account for this three-way interaction by modeling task-specific decision rules. 

Figure 8

 

Spatiotemporal distribution of errors on the TOJ and simultaneity tasks. Symbols are the same as in Movie 2. Dotted lines (simultaneity task) and solid lines (TOJ task) highlight error patterns that distinguish the two tasks from each other. Error bars reflect ±1 SEM (N = 23).

The three-way interaction readily seen in Figure 8 also has important implications for our finding that participants exhibited a left-target-first bias on the TOJ task (Figure 3) but not on the simultaneity task (Figure 5). One possible explanation for that difference pertains to the feedback provided on each task. Specifically, when LVF and RVF targets were synchronized, we provided valid feedback on the simultaneity task (because objectively correct responses were possible) but random feedback on the TOJ task (because objectively correct responses were not possible). In principle then, participants could have adjusted their responses to match objective reality on the simultaneity task (given valid feedback) but not the TOJ task (given random feedback). However, at the briefest asynchronies (plus and minus 133 ms, Figure 8) identically valid feedback occurred on the simultaneity (dotted lines) and TOJ (solid lines) tasks, yet the significant three-way interaction reveals task-specific error patterns nevertheless. Likewise, within the plus 133 ms asynchrony alone, simultaneity errors significantly exceeded those on the TOJ task, despite identically valid feedback across the two tasks, LVF-exo: F(1, 22) = 38.932, p < 0.001, η_p = 0.639; RVF-exo: F(1, 22) = 48.259, p < 0.001, η_p = 0.687 (see vertical gray oval within Movie 2). These findings demonstrate that task differences in the validity of feedback were not necessary to generate task-specific error patterns. Most strikingly, within the simultaneity task alone, errors at the plus 133 ms asynchrony significantly exceeded those when the stimuli were synchronized (0 ms asynchrony) despite identically valid feedback across those conditions, LVF-exo: F(1, 22) = 11.674, p = 0.002, η_p = 0.347; RVF-exo: F(1, 22) = 67.990, p < 0.001, η_p = 0.756 (see diagonal green ovals within Movie 2). This directly demonstrates the participants' failure to optimally use valid feedback on the simultaneity task to realign their responses with objective reality. 

Lastly, although our research question addressed time perception rather than time discrimination, Table 1 contains 75% discrimination thresholds for our TOJ and simultaneity tasks to inform future experiments or replication. 

Table 1

 

Seventy-five percent discrimination thresholds, in milliseconds. Simultaneity-task thresholds correspond to d' = 0.6745.

	LVF exo target	RVF exo target
Temporal order judgment task	197	157
Simultaneity task: Endo target first	157	134
Simultaneity task: Exo target first	218	213

General discussion

The present experiments assessed lateral hemifield differences in attention's temporal properties. Specifically, participants judged the temporal order and simultaneity of opposite-hemifield targets embedded in dual-stream RSVP displays. The experiments generated the following main findings: 

1.  
   A ∼134 ms hastened perception of LVF targets on the TOJ task (Figures 3 and 4);
2.  
   Patterns of simultaneity-task responses that disconfirm those predicted from the hastened perception of LVF targets on the TOJ task (Figure 4 vs. Figure 5);
3.  
   Greater PSEs when judging simultaneity on RVF-exo trials relative to LVF-exo trials (Figures 6 and 7);
4.  
   Significantly more errors on the simultaneity task than on the TOJ task when Exo-targets preceded Endo-targets by 133 ms (Movie 2);
5.  
   A significant three-way (task by Exo-hemifield by target order) interaction in error rates at the briefest (133 ms) target asynchrony (dotted vs. solid lines in Figure 8).

These outcomes occurred while retinal stimulation remained identical across the TOJ and simultaneity tasks. This finding rules out a stimulus-driven “bottom-up” explanation for the task-specific behavior. We next consider models that account for these findings via task-specific decision-stage algorithms that receive identical stimulus-driven “bottom-up” inputs. Subsequently, we discuss how the present findings relate to prior studies. 

Modeling the TOJ and simultaneity task results

One might model the present TOJ task results with an algorithm or decision rule, its inputs, and its output. The inputs to the decision rule include an estimated time of arrival (ETA) for our dual-stream RSVP displays' left and right visual field targets, LVF_ETA and RVF_ETA. For these ETAs, no unusual neural activity would be needed. The early visual pathway's retinotopic architecture would readily register each target's lateral hemifield, and illusions like the classic Pulfrich effect⁴ demonstrate that the early visual pathway labels stimuli with the functional equivalent of ETAs. The ETAs could be combined with a bias to generate an output—a TOJ response (“LVF first” vs. “RVF first”)—according to the following decision rule: 

•  
  If ([LVF_ETA – RVF_ETA] + bias) < 0, “LVF first,” else “RVF first.”

The bias would be expressed in milliseconds and would correspond to a participant's PSE. When the bias is zero, the output would be determined entirely by the ETAs assigned in the early visual pathway. Negatively and positively signed bias values would generate, respectively, inclinations to favor “LVF first” and “RVF first” responses. In principle then, the hastened perception of LVF targets on the present TOJ task (Figures 3 and 4) could reflect hastened early visual system responses to LVF targets, a decision-stage bias, or both. A decision-stage bias favoring the LVF-first response seems unlikely, however, because our TOJ task required participants to report “letter first” versus “number first,” not “LVF first” versus “RVF first.” By contrast, a hastened early visual pathway response to LVF targets seems plausible given prior physiological data showing ERPs peaking earlier for LVF than for RVF targets on a dual-stream RSVP identification task (Verleger et al., 2011; Verleger et al., 2013). 

While a hastened early visual pathway response to LVF targets would be sufficient to explain our TOJ results, additional factors are needed to model our simultaneity results. This is because our simultaneity results disconfirmed two predictions that would be made if the LVF hastening on the TOJ task also occurred on the simultaneity task. First, on the simultaneity task, “different” responses were lowest (perceived simultaneity was greatest) at the 0 ms target asynchrony, not at the predicted ±133 ms asynchronies. (See Figure 5, black arrow vs. pink and light blue arrows.) Second, LVF-exo trials on our simultaneity task generated a performance asymmetry opposite to that predicted from the hastened perception of LVF targets on the TOJ task. Specifically, participants perceived brief asynchronies as “different” significantly more often when LVF-exo targets followed (−133 ms), rather than preceded (+133 ms), RVF-endo targets (Figure 5). Moreover, hastened early visual pathway responses to LVF targets would not explain why our participants exhibited greater PSEs when judging simultaneity on RVF-exo trials relative to LVF-exo trials (Figures 6 and 7). Taken together, these results require a model of the present simultaneity task that incorporates the exo-target's hemifield. 

One such model of the present simultaneity task contains the following two-step decision rule: 

1.  
   If the exogenous target is in the LVF, add a delay to LVF_ETA, else increase the bias;
2.  
   If | LVF_ETA – RVF_ETA | > bias, “different,” else “same.”

The delay added to LVF_ETAs in step one represents the remapping of time that is reflected in the title of this article. We previously noted this LVF-specific remapping in the results for the simultaneity task (see Figure 5, LVF-exo simultaneity trials). Similarly, the bias increase in step one represents the observed RVF-exo–specific PSE shift also previously noted in the results for the simultaneity task (see Figures 5 through 7, RVF-exo simultaneity trials). In step one, determining the exogenous target's hemifield would not require any unusual neural activity. Indeed, Buschman and Miller's (2007) electrophysiological recordings suggest that exogenous and endogenous targets are registered by distinct neural synchrony rates and distinct cortical sequences of synchronized neural firing. Further support for distinct neural responses to endogenous versus exogenous targets comes from hemispatial-neglect patients' dissociations between these target types (Esterman, McGlinchey-Berroth, & Milberg, 2000), and from TMS experiments linking the temporoparietal junction with exogenous but not endogenous attention (Chica, Bartolomeo, & Valero-Cabré, 2011). In step two, the bias would be expressed in milliseconds and would correspond to a participant's PSE as on the TOJ task. Unlike the bias on the TOJ task, however, the bias in step two of the simultaneity task would be nonnegative. This is because a sufficiently large target asynchrony in either direction (the absolute value of LVF_ETA – RVF_ETA) would constitute evidence for a “different” response on the simultaneity task. Larger biases would indicate that participants demand greater sensory evidence (longer PSEs) for a target asynchrony before making a “different” response. 

This two-step model explains the above-noted significant three-way (task by Exo-hemifield by target-order) interaction in error rates at the briefest (133 ms) target asynchrony (solid vs. dotted lines in Figure 8). To appreciate this, consider how the error rates change across the two tasks within a given exo-hemifield condition. When the exogenous target is in the LVF on simultaneity trials, the model adds a delay to the LVF_ETA to counter the early visual pathway's hastened response to LVF targets. The consequence of this LVF delay, though, depends on target order. Specifically, the delay increases the signal that participants must detect (the target asynchrony) when LVF-exo targets come second but decreases the signal to be detected when the LVF-exo targets come first. This explains our empirical observation that in LVF-exo trials when the task changed from TOJ (blue triangles) to simultaneity (blue circles) error rates declined when exo-targets came second (−133 ms) but increased when exo-targets came first (+133 ms). On RVF-exo trials, our model increases the value of the bias within the simultaneity decision. This drives simultaneity error rates above TOJ error rates because the LVF_ETA – RVF_ETA difference remains identical in the two tasks but must overcome an elevated bias within the simultaneity decision. The elevated bias explains our empirical observation that in RVF-exo trials when the task changed from TOJ (red triangles) to simultaneity (red circles) RVF-exo errors increased regardless of target order and despite identical retinal stimulation across the two tasks.⁵ 

We emphasize that the present simultaneity data could be modeled in other ways as well. The model proposed above, however, offers three desirable features. First, it parsimoniously explains the diverse range of empirical findings from our simultaneity experiment. Second, it relies only on physiological properties empirically documented by single unit recordings in monkeys (Buschman & Miller, 2007) or in humans via EEG recordings (Verleger et al., 2011; Verleger et al., 2013), TMS (Chica et al., 2011), and clinical observations of hemispatial-neglect patients (Esterman et al., 2000). Perhaps most importantly, the proposed model demonstrates a simple computational principle by which the visual system could successfully recover simultaneity from an asymmetry it imposes on itself—a hastened response to LVF targets. 

Relation to earlier studies

The present findings confirm and extend those of prior dual-stream RSVP studies that demonstrated LVF advantages. As noted in the Introduction, prior dual-stream RSVP studies showed that participants more accurately identify LVF targets than RVF targets (Holländer et al., 2005; Scalf et al., 2007; Verleger et al., 2009; Smigasiewicz et al., 2010; Verleger et al., 2010; Verleger et al., 2011; Verleger et al., 2013). The present study, which addressed relative timing judgments rather than target identification, also demonstrated a LVF advantage. Specifically, participants in our TOJ task perceived LVF targets approximately 134 ms earlier than RVF targets. An important distinction between our TOJ task and the previous target-identification tasks pertains to memory. The previous target-identification tasks required maintaining a memory of specific target letters or target numbers amid the RSVP stream's subsequently interfering distractors. Our TOJ task did not require participants to maintain a memory of specific target letters or target numbers, yet a significant LVF advantage emerged nevertheless. This implies that the LVF advantage on RSVP tasks does not depend on the neural events that maintain target identity in memory. 

Our findings also demonstrate that the LVF advantage on RSVP tasks need not be related to the so-called attentional blink (AB). The AB is the phenomenon that occurs in RSVP experiments when the identification of one target impairs the identification of a second target (Raymond, Shapiro, & Arnell, 1992; Chun & Potter, 1995). The impairment in the second target's identification, however, typically occurs only when it follows the first target by 200 to 500 ms. If the second target follows the first by less than 200 ms, its identification often remains unimpaired—a phenomenon known as “lag 1 sparing” (Visser, Zuvic, Bischof, & Di Lollo, 1999). By contrast, the present TOJ task exhibited a salient LVF advantage (hastening) at 0 ms and 133 ms target asynchronies—durations well within the typical lag 1 sparing interval. Moreover, the present LVF advantage diminished at longer asynchronies (i.e., “lags”) when the AB is typically maximal. Therefore, the LVF advantage on the present RSVP task is likely a distinct phenomenon from the AB given the distinct time courses. 

Another interesting time-course difference arises when contrasting the present perceptual data and ERP data previously obtained with similar stimuli. Using dual-stream RSVP displays similar to those tested here, Verleger et al. (2011) and Verleger et al. (2013) found that N2pc (parietal contralateral) ERPs peaked ∼50 ms earlier for LVF than for RVF targets on an identification task. It may seem surprising that the magnitude of LVF perceptual hastening on the present TOJ task (∼134 ms) exceeded that of the LVF physiological hastening (∼50 ms) on an identification task. This is surprising because the neural events that register time of arrival on TOJ tasks would plausibly occur earlier in the visual pathway than those that register a target's letter or number identity. One possible explanation for the larger perceptual (∼134 ms) than ERP (∼50 ms) magnitude would arise if a physiological LVF hastening—or equivalently, a RVF delay—pushed RVF targets into a subsequent attentional cycle. This possibility assumes that attention operates in discrete time samples. Indeed, theories of discrete attention have been developed elsewhere (Niebur, Koch, & Rosin, 1993; VanRullen & Koch, 2003; Womelsdorf & Fries, 2007; Wyble, Bowman, & Nieuwenstein, 2009; VanRullen, Busch, Drewes, & Dubois, 2011; VanRullen & Dubois, 2011; Wyble, Potter, Bowman, & Nieuwenstein, 2011). Evidence supporting these theories comes from psychophysical experiments on intrusion errors (Vul, Hanus, & Kanwisher, 2009), the so-called “blinking spot light of attention” (VanRullen, Carlson, & Cavanagh, 2007), the flash lag effect (Chakravarthi & VanRullen, 2012), and the wagon-wheel illusion (Purves, Paydarfar, & Andrews, 1996; Simpson, Shahani, & Manahilov, 2005; VanRullen, Reddy, & Koch, 2005; VanRullen, Pascual-Leone, & Battelli, 2008).⁶ Regarding a potential neural locus, it is notable that the wagon-wheel illusion can be disrupted by TMS to the parietal lobe (VanRullen et al., 2008). The parietal lobe's role in time sensitivity also has been implicated by Verleger et al.'s (2011) and Verleger et al.'s (2013) ERPs on RSVP tasks, fMRI studies exploring TOJs (Davis et al., 2009), and by the impaired TOJs in patients exhibiting visual extinction (Rorden et al., 1997; Baylis, Simon, Baylis, & Rorden, 2002; Rorden, Jelsone, Simon-Dack, Baylis, & Baylis, 2009). 

Broadly, the present findings confirm recent reports of a dissociation between the neural events that mediate judgments about simultaneity and temporal order (van Eijk, Kohlrausch, Juola, & van de Par, 2008; Weiss & Scharlau, 2011; Love, Petrini, Cheng, & Pollick, 2013). These task-specific neural events can be modeled in various ways. The task-specific decision rules modeled above parsimoniously explain the main features of the behavioral data in a physiologically plausible manner. 

Conclusion

The present experiments assessed lateral hemifield differences in attention's temporal properties. Our TOJ experiment revealed that participants perceived LVF targets ∼134 ms sooner than RVF targets. This LVF hemifield hastening, which approximates a full cycle of visual attention's canonical ∼10 Hz (∼100 ms) temporal resolution, was not evident when the same participants viewed the same RSVP displays but judged simultaneity. Together, these experiments demonstrate that the visual system remaps a hastened neural response to LVF targets by using hemifield-specific rules within the decision stage of simultaneity judgments. Such hemifield-specific remapping would effectively compensate for the cross-hemifield asymmetries in neural response latencies that could otherwise impair simultaneity estimates. 

Supplementary Materials

xlsx S1 

Acknowledgments

The authors thank Dr. Joo-Hyun Song, Dr. Karen Gunther, and two anonymous reviewers for helpful suggestions. 

Commercial relationships: none. 

Corresponding author: Nestor Matthews. 

Email: [e-mail]. 

Address: Department of Psychology, Denison University, Granville, OH, USA. 

Footnotes

¹  Pillow and Rubin (2002) reported a perceptual completion task that generated superior unilateral performance, which depended on perceptual grouping but not visual attention.

Footnotes

²  “Endogenous” and “exogenous” here, respectively, correspond to the biology-related designations of inside versus outside the body. The black digit targets form a group distinct from the black letter distractors only within the neural networks of those who have learned the English letter and Arabic number categories—an endogenous distinction. The red letter targets differ spectrally from the black letter distractors—an exogenous distinction.

Footnotes

³  Although we plotted the proportion of “letter-first” responses, the difference between the LVF and RVF PSEs would be the same if one instead plotted the proportion of “number-first” responses.

Footnotes

⁴  In the classic Pulfrich effect, a pendulum swinging from left to right is perceived as having an illusory three-dimensional rotation when a light-attenuating filter is placed before one of the two eyes. The neural response to the stimulus in the filtered eye's view is slowed relative to that of the unfiltered eye's view. This time delay results in artificial interocular positional differences that generate responses in cortical neurons tuned to various binocular disparities, thereby giving rise to the illusory three-dimensional rotation.

Footnotes

⁵  The significantly elevated RVF-exo error rate on the simultaneity task initially suggests a suboptimal decision rule across the 40 RVF-exo trials that each participant completed at the briefest asynchronies (±133 ms). Yet the same decision rule significantly reduced the error rate (incorrect “different” responses) on the 100 RVF-exo trials that each participant completed when the two targets were physically synchronous (see Figure 8). Overall then, the decision rule was advantageous at the smallest temporal intervals tested here.

Footnotes

⁶  A different account of the wagon-wheel illusion involves perceptual rivalry between spuriously activated motion detectors (Kline, Holcombe, & Eagleman, 2004; Kline & Eagleman, 2008; Piantoni, Kline, & Eagleman, 2010).