Continuous psychophysics shows millisecond-scale visual processing delays are faithfully preserved in movement dynamics

Abstract

Image differences between the eyes can cause interocular discrepancies in the speed of visual processing. Millisecond-scale differences in visual processing speed can cause dramatic misperceptions of the depth and three-dimensional direction of moving objects. Here, we develop a monocular and binocular continuous target-tracking psychophysics paradigm that can quantify such tiny differences in visual processing speed. Human observers continuously tracked a target undergoing Brownian motion with a range of luminance levels in each eye. Suitable analyses recover the time course of the visuomotor response in each condition, the dependence of visual processing speed on luminance level, and the temporal evolution of processing differences between the eyes. Importantly, using a direct within-observer comparison, we show that continuous target-tracking and traditional forced-choice psychophysical methods provide estimates of interocular delays that agree on average to within a fraction of a millisecond. Thus, visual processing delays are preserved in the movement dynamics of the hand. Finally, we show analytically, and partially confirm experimentally, that differences between the temporal impulse response functions in the two eyes predict how lateral target motion causes misperceptions of motion in depth and associated tracking responses. Because continuous target tracking can accurately recover millisecond-scale differences in visual processing speed and has multiple advantages over traditional psychophysics, it should facilitate the study of temporal processing in the future.

Introduction

The binocular visual system combines spatial and temporal information from the eyes to estimate the structure of the three-dimensional (3D) environment, the 3D motion of objects in the environment, and self-motion through the environment. A large body of research has focused on how spatial differences in the left- and right-eye images (i.e., binocular disparities) drive the estimation of 3D structure and motion (Banks, Gepshtein, & Landy, 2004; Burge & Geisler, 2014; Cormack, Czuba, Knöll, & Huk, 2017; Cormack, Stevenson, & Schor, 1991; Cumming & DeAngelis, 2001; DeAngelis, Ohzawa, & Freeman, 1991; Iyer & Burge, 2018; Julesz, 1964; Ogle, 1952; Ohzawa, DeAngelis, & Freeman, 1990; Tyler & Julesz, 1978; Wheatstone, 1838). A smaller but still substantial body of research has investigated how temporal differences in the processing of left- and right-eye images impact the estimation of motion-in-depth (Burge, Rodriguez-Lopez, & Dorronsoro, 2019; Carney, Paradiso, & Freeman, 1989; Lages, Mamassian, & Graf, 2003; Lit, 1949; Morgan & Thompson, 1975; Pulfrich, 1922; Reynaud & Hess, 2017; Rodriguez-Lopez, Chin, & Burge, 2023; Rogers & Anstis, 1972; Wilson & Anstis, 1969). Despite this long-standing interest (Watson, 1986), there have been few psychophysical attempts to measure the time course of visual processing and how it impacts the perception of motion in depth. 

Traditional psychophysical data collection techniques, which require an observer to view a series of individual trials and to respond to each trial with a binary choice, usually do not reveal continuous temporal information about visual processing. Although recovering such information is possible in principle, traditional techniques are slow. Collecting sufficient data to recover time-course information is therefore often impractical. More generally, the slow pace of data collection places strong constraints on the number of conditions that can be practically run on each subject and hence on the scope of the experimental questions that can be practically addressed using within-subjects designs. 

Continuous target-tracking psychophysics, a new stimulus–response data collection technique, simultaneously provides information about the time course of visual processing and improves the rate of data collection (Bonnen, Burge, Yates, Pillow, & Cormack, 2015). With this technique, a property of a stimulus (e.g., position) is tracked via an input device (e.g., a computer mouse).¹ Changes in the stimulus property of interest (e.g., contrast) are then associated with commensurate changes in the tracking response. The manner in which performance changes with the stimulus allows one to make inferences about various aspects of spatial and—as we show here—temporal processing. 

There are many advantages of target-tracking psychophysics. First, as we will show, target tracking and traditional forced-choice psychophysics make near-identical estimates of temporal processing delays. Second, target tracking provides time-course information that traditional forced-choice psychophysics typically does not; this additional information provides a richer picture of the temporal characteristics of the system and can lead to enhanced predictive power (see Discussion). Third, target tracking provides a direct continuous measure of behavioral performance; other non-invasive continuous measures such as electroencephalographty and electroretinography (e.g., Kremers, Aher, Parry, Patel, & Frishman, 2022) provide more direct measures of neural activity but must be related to performance via other means. Fourth, target tracking is low tech and does not require specialized equipment other than a display screen and a mouse. (If target tracking in depth is desired, then a stereo-capable display is required.) Fifth, target tracking requires little instruction or practice and is therefore suitable for collecting large amounts of high-quality data from virtually any observer; this last advantage may make target tracking particularly advantageous for work with developmental, clinical, and other non-traditional populations. 

The main goal of this paper is to demonstrate the utility of continuous psychophysics for measuring the time course of visual processing. To do so, we estimate interocular differences in visual processing using three methods—monocular target-tracking psychophysics (Figure 1A), traditional forced-choice psychophysics (Figure 1B), and binocular target tracking in depth (Figure 1C)—and then compare results across methods. 

Figure 1.

 

The classic Pulfrich effect and three approaches to measuring interocular delays. (A) Monocular target tracking. A target undergoing a random walk in X on the screen is tracked while viewed with the left eye alone and also while viewed with the right eye alone. Although the target is always perceived in the plane of the screen (i.e., no illusory depth is perceived), comparing monocular tracking performance between the eyes can yield estimates of interocular delay that match those obtained with traditional psychophysics. (B) Traditional forced-choice psychophysics. When a moving target is viewed with an unequal amount of light in the two eyes, the distance and 3D direction of horizontal motion are perceived incorrectly. With a neutral-density filter in front of the left eye, sinusoidal target motion (i.e., motion similar to that of a pendulum) in the screen plane is misperceived as motion along a near-elliptical trajectory in depth. The image in the darker eye is processed with a delay relative to the brighter eye. For rightward motion, this interocular delay causes the effective target image position in the darker eye (gray dot) to lag behind (i.e., to be shifted leftward) relative to the target image position in the brighter eye (white dot). For leftward motion, the target image position in the darker eye is shifted rightward (not shown). The binocular visual system computes the disparity from these effective left- and right-eye images, and the target is perceived behind the screen for rightward motion (and in front of the screen for leftward motion). In a traditional forced-choice psychophysical experiment, observers report their percept (e.g., “front left” or “front right”) with a button press. This method recovers estimates of interocular delay. (C) Binocular target tracking. A target undergoing a random walk in X and Z is tracked while being viewed with both eyes. When the left eye is dark, the interocular delay causes a target moving rightward to be perceived farther away than it is and vice versa. Analyzing how target motion in X affects response motion is Z can also reveal interocular differences in processing.

To develop a stringent test for monocular target tracking, we make use of the Pulfrich effect, a well-known stereo-motion phenomenon (Lit, 1949; Pulfrich, 1922). When the image in one eye is darker than the image in the other, motion in the frontal plane, like that of a clock pendulum, is misperceived as near-elliptical motion in depth (Figure 1B). The effect occurs because the image with less light is processed more slowly. The interocular mismatch (i.e., delay) in processing speed causes, for moving objects, an effective neural disparity that leads to illusory percepts of depth. Interocular delays as small as a few milliseconds can cause large perceptual effects. Traditional forced-choice experiments which would, in this case, require observers to report whether a target stimulus was moving leftward or rightward when it appeared to be in front of the screen (see Figure 1B), can accurately estimate these millisecond differences in processing speed. The Pulfrich effect therefore provides a demanding test for continuous psychophysics. 

We show that continuous target-tracking psychophysics can be used to quantify millisecond-scale differences in visual processing between the eyes. We also prove a novel relationship between the temporal differences between left- and right-eye monocular processing and key properties of binocular target tracking in depth. These results indicate that sensory delays in visual processing are faithfully preserved in the movement dynamics of the hand. Target tracking has exquisite temporal sensitivity, provides estimates that are matched to those provided by traditional psychophysics with millisecond accuracy, and holds promise for a range of scientific and clinical applications. 

Results

Monocular target-tracking psychophysics

First, we estimated the visual processing delays caused by luminance reductions with target tracking under monocular viewing conditions. Human observers tracked a white vertical target bar undergoing a random walk in X (i.e., horizontally) with a small mouse cursor dot (Figure 2A). The task was to follow the target as accurately as possible. Tracking was performed monocularly under six viewing conditions: left eye alone or right eye alone at each of three different luminance levels (see Methods). The task was performed without difficulty; the human tracking response was a smoothed and delayed approximation of the target motion (Figure 2B). 

Figure 2.

 

On-screen display and monocular tracking performance. (A) Target tracking stimulus. The target bar underwent a horizontal random walk in X or a random walk in both X and Z, depending on the experiment. The task was to track the target with a small dark mouse cursor. Motion direction and speed are indicated by arrows and dashed shapes; note that they are for illustrative purposes and were not present in the actual stimuli. (B) Target trajectory in X (solid curve) and tracked trajectory (dashed curve) across time. The response trajectory is a smoothed, delayed version of the target trajectory. Note that the delay in the human tracking response is approximately constant throughout the trial. (C) Target and response trajectories in Z. Note that the tracking delays are more pronounced in Z than in X.

Monocular cross-correlograms—the cross-correlation of the target and response velocities—are shown for all observers in the highest and lowest luminance conditions (Figure 3A). The cross-correlogram approximates the shape of the temporal impulse response function of the visuomotor system, assuming the system is linear. The latency of the initial response, as quantified by the first point of the cross-correlogram that rises out of the noise ranges between 150 and 200 ms. The rise is steep such that the peak correlation occurs 50 to 75 ms after the cross-correlation exceeds the correlation noise (see Methods). The temporal integration period ranges between 100 and 300 ms across observers, as quantified by bandwidth (i.e., full-width at half-height). Note that, because the motion statistics were matched across visual conditions, motor noise should be constant across conditions (Harris & Wolpert, 1998). (Jerk, the time derivative of response acceleration, does not vary systematically across conditions.).² More systematic aspects of motor response should also remain constant across visual conditions. So, although the visuomotor response in any given condition will be affected by the characteristics of both the visual system and the motor system, comparisons across visual conditions should reveal changes in visual processing only. 

Figure 3.

 

Monocular target tracking results. (A) Monocular target tracking in X for all observers. The cross-correlograms in the left column reflect left-eye-alone target tracking performance when the image was bright (i.e., maximum luminance: OD = 0.0; light blue curve) and when the image was dark (i.e., one-quarter maximum luminance: OD = 0.6; dark blue curve). The cross-correlograms in the right column similarly reflect right-eye-alone tracking performance when the image was bright and when the image was dark (light and dark red curves, respectively). The cross-correlograms are nearly overlapping. However, the dark curves are shifted rightward by a small but consistent amount. This shift indicates increased processing delays. Insets show the systematic delay between the rising edges of the cross-correlograms. The shaded bands in the main panel shows ±1 SD of the correlation noise, computed from lags less than 0 ms. (B) Interocular tracking delays as a function of optical density difference. Negative optical density differences (∆O = −0.6 and −0.3) show delays that were computed between conditions with dark left-eye images and bright right-eye images. Positive differences (∆O = +0.3 and +0.6) show delays that were computed between conditions with bright left-eye images and dark right-eye images. In general, delay is systematically related to the interocular difference in optical density. (Please see Figure 6A for error bars on these data).

On quick examination, the monocular cross-correlograms appear to be the same for the high- and low-luminance conditions. Closer examination, however, reveals a small but systematic shift between the cross-correlograms in the two luminance conditions. This shift is clearest and most consistent in the rising edges of the cross-correlograms (Figure 3A, insets). The visuomotor responses in the high-luminance conditions are faster by several milliseconds than the responses in the low-luminance conditions. Thus, assuming that the dynamics of the motor system itself are unchanged by the visual stimulus, the results imply that the speed of visual processing is faster in high-luminance than in low-luminance conditions. 

Figure 3B shows these differences in processing speed (i.e., delays) as a function of the luminance difference between the left and right eyes (i.e., simulated optical density difference; see Methods). We estimated the delays by computing the cross-correlation of the cross-correlograms in two luminance conditions (high luminance vs. low luminance); the delay yielding the maximum correlation was taken as the estimate of relative delay. This method makes use of all of the raw data, is robust, and is relatively unladen by assumptions. Across all observers, lower luminance caused slower visuomotor processing. In the conditions with the largest luminance differences, the delays ranged between 4 and 10 ms. Interestingly, the estimated delays reported in Figure 3B are in general agreement with previous psychophysical and neurophysiological investigations of the Pulfrich effect, which have shown that every log-unit reduction of luminance reduces the speed of processing by approximately 10 ms (Carney et al., 1989; Rogers & Anstis, 1972). Moreover, the fact that these results are systematic and regular suggests that continuous target tracking is sensitive enough to estimate relative delays on the order of milliseconds. 

The question, however, is whether these measured differences in visuomotor processing speed reflect differences in visual processing speed that are associated with stimuli of different luminance. To answer this question, we examined how estimates of temporal processing delays based on target tracking are related to estimates of temporal processing delays obtained with traditional forced-choice techniques. With forced-choice techniques, observer responses do not meaningfully depend on the dynamics of the motor system, so a quantitative comparison of delays in the two paradigms is a stringent test of whether target tracking can be used to measure millisecond-scale differences in visual processing. 

Traditional forced-choice psychophysics

To directly compare estimates of processing speed from tracking and traditional psychophysics, we used a standard paradigm to measure interocular delays associated with the Pulfrich effect (Burge et al., 2019; Rodriguez-Lopez, Dorronsoro, & Burge, 2020). The luminance levels in each eye were matched to those used in the monocular tracking experiment. The primary differences between the experiments were the motion trajectories followed by the stimulus and the method of response. Rather than a random walk, the target stimulus followed a sinusoidal trajectory on each eye's monitor (Figures 4A to 4C). Rather than continuously tracking the target stimulus with a cursor, observers viewed the target and made a binary response reporting an aspect of the 3D motion percept. When luminance differences between the eyes were non-zero, the target appeared to follow a near-elliptical 3D motion trajectory in depth. 

Figure 4.

 

Binocular target stimulus, on-screen positions, and disparity-specified 3D target trajectories for forced-choice psychophysics experiment. (A) On-screen target image positions over time for the left and right eyes. When the left-eye image is delayed on-screen relative to the right-eye image (top), disparity specifies “front left” 3D motion (bottom). (B) When no on-screen delays are present, disparity specifies motion in the plane of the screen. (C) When the left-eye image is advanced on-screen relative to the right-eye image, disparity specifies “front right” 3D motion (bottom). (D) Binocular target stimulus. Arrows and dashed bars show motion direction and speed; they are for illustrative purposes and were not present in the actual stimuli. Free-fuse to see stimulus in 3D: Cross-fusers will see a depiction of “front right” (i.e., counterclockwise) 3D motion; divergent fusers will see a depiction of “front left” (i.e., clockwise) 3D motion. (E) Example psychometric functions from the first human observer for interocular differences in optical density: ∆O = −0.6 OD, −0.3 OD, 0.0 OD, +0.3 OD, +0.6 OD. These differences in optical density correspond to luminance differences ranging from the left eye having 75% less light than the right eye to the right eye having 75% less light than the left eye. To cancel the induced neural delays, the required on-screen interocular delays (arrows) change systematically with the luminance differences.

To change the stereoscopically defined 3D motion trajectory, on-screen disparity was manipulated to simulate on-screen interocular delay (see Methods). To simulate the left-eye image being delayed on-screen relative to the right-eye image, on-screen disparity specified that the target was undergoing “front left” motion (i.e., clockwise motion when viewed from above) (Figure 4A). To simulate no delay, disparity specified that the target was moving in the plane of the screen (Figure 4B). And, to simulate a left-eye image that is advanced on-screen relative to the right-eye image, on-screen disparity specified that the target was undergoing “front right” motion (i.e., counterclockwise motion when viewed from above) (Figure 4C). The size and shape of the target bar, the size and shape of the picket-fence reference bars, the luminance levels, and the peripheral 1/f noise were identical to those presented in the tracking experiment (Figure 4D). The only differences were the shape of the mean luminance gray region (circular vs. rectangular) and the number of picket-fence bars (five vs. thirteen). 

The task was to report whether the target appeared to be moving leftward or rightward when it appeared to be in front of the screen. For a given interocular luminance difference, we measured a psychometric function: the proportion of trials that observers reported “front right” as a function of the on-screen interocular delay (Figure 4E). See Methods for details. 

The goal of the experiment was to find, for each luminance difference, the critical on-screen delay that makes the target appear to move in the plane of the screen. This critical on-screen delay—the point of subjective equality (PSE) on the psychometric function—should be equal in magnitude and opposite in sign to the neural delay caused by the luminance difference between the eyes in each visual condition. We found that the critical on-screen delays (i.e., the PSEs) required to null the neural delays changed systematically with the associated luminance differences in all five observers (Figure 5). 

Figure 5.

 

Traditional forced-choice psychophysics results. Critical on-screen interocular delays that resulted in percepts of zero motion in depth as a function of the interocular differences in optical density: ∆O = −0.6 OD, −0.3 OD, 0.0 OD, +0.3 OD, +0.6 OD. These optical density differences correspond to the left-eye stimulus being 75% and 50% darker than the right-eye stimulus, the stimuli in both eyes having the same luminance, and the right-eye stimulus being 50% and 75% darker than the left-eye stimulus, respectively. Negative and positive on-screen delays correspond, respectively, to the left-eye stimulus being delayed and advanced relative to the right-eye on-screen stimulus. (Please see Figure 6A for error bars on these data).

Target-tracking versus forced-choice psychophysics

Figure 6A shows the estimates from the traditional forced-choice experiment plotted directly against the estimates from the continuous target-tracking experiment. Error bars in both directions are bootstrapped standard errors. The data are tightly clustered about the unity line, indicating millisecond-scale agreement between estimates of interocular delay provided by the two experiments (Figure 6A). Across conditions and observers, the differences between the estimated delays with traditional and tracking psychophysics were very small. The mean difference in delay was −0.16 ms (95% confidence interval, –1.04 to 0.71) with a standard deviation of 2.06 ms (Figure 6A, upper-right inset). The mean difference in delay was not statistically different from zero. For reference, Figure 6B overlays the estimates of interocular delay from each of the two experiments (Note that Figure 6B replots data from Figures 3B and 5.) 

Figure 6.

 

Interocular delays from target tracking versus traditional forced-choice psychophysics. (A) Interocular delays measured with target tracking plotted against interocular delays measured with forced-choice psychophysics in equivalent conditions. The correlation is 0.89 (p = 4.4 × 10⁻⁹). The red tint indicates conditions in which the right eye was darker, and the blue tint indicates conditions in which the left eye was darker. Error bars indicate 68% bootstrapped confidence intervals from 1000 datasets resampled with replacement; if error bars are not visible, they are smaller than the size of the data symbol. The histogram (upper-right corner) shows the distribution of differences between the forced-choice delays and the tracking delays; the mean difference was only −0.16 ms (SD = 2.06 ms). (B) Interocular delay as a function of interocular difference in optical density as measured with target tracking (white symbols) and forced-choice psychophysics (gray symbols) for individual observers in each of five different conditions: ∆O = −0.6 OD, −0.3 OD, 0.0 OD, +0.3 OD, +0.6 OD. Positive on-screen delays indicate that the left eye was processed more slowly. (Data replotted and overlayed from Figures 3B and 5).

Given the enormous differences between the two psychophysical methods, the agreement between the respective estimates of interocular delay is striking. One method—traditional forced-choice psychophysics—presented a target stimulus following a stereotyped motion trajectory in depth (i.e., a near-elliptical path through a 3D volume of space) and obtained a binary response. This binary response reflected an aspect of the observer's percept, and the response is completely independent of the temporal properties of the motor system. The other method—target-tracking psychophysics—presented a target stimulus following an unpredictable motion trajectory (i.e., a random walk in the two-dimensional plane of the display monitor) and obtained the continuous motor response of the observer. This continuous response necessarily reflects the temporal properties of both the visual and motor systems, and yet the estimates of interocular delay from the two methods agree to within a fraction of a millisecond on average and to within a few milliseconds in each individual condition. The fact that substantially different stimuli and substantially different experimental paradigms yield near-identical estimates strongly suggests that both paradigms are measuring the same underlying quantity. 

Binocular target-tracking psychophysics

The preceding experiments show that monocular target tracking and traditional forced-choice psychophysics yield similar estimates of millisecond-scale processing differences between the eyes. Does binocular target tracking afford similarly precise measurements of interocular differences in visual processing? In this experiment, human observers binocularly viewed and tracked a target bar with a small cursor undergoing a random walk in the XZ plane (see Figure 1C). Either the left- and right-eye on-screen images were equally bright or the image for one eye was substantially darker than the other: ∆O = −0.6 OD, 0.0 OD, and 0.6 OD (see Methods). These luminance conditions were matched to those used in the monocular tracking experiment. 

Binocular target-tracking: Empirical results

Binocular tracking of horizontal target motion is similar to monocular tracking of horizontal motion for all luminance conditions (X vs. X) (Figure 7). The latency of the initial response ranged between 150 and 200 ms across observers, and the temporal integration period ranged between 100 and 300 ms. Binocular target tracking in depth is uniformly more sluggish (Z vs. Z) (Figure 7). In each observer, the latency of the initial response in Z occurred approximately 50 ms later than the initial response in X, and the period of temporal integration in Z was nearly double the temporal integration period in X. These results held when both eyes had the same luminance (Figure 7A), when the left eye was darker than the right (Figure 7B), and when the right eye was darker than the left (Figure 7C). Hence, neither the X versus X nor the Z versus Z cross-correlograms provide information about differences in temporal processing between the eyes, as these cross-correlograms are nearly identical down the columns of Figure 7. The sluggishness of the response in Z to target motion in Z also replicates the primary finding from Bonnen, Huk, and Cormack (2017) and is generally consistent with other results that have shown that changes in depth are processed more slowly than changes in horizontal position. The current results, however, do not shed light on the underlying reasons for the sluggishness of the Z motion processing. 

Figure 7.

 

Binocular tracking. Temporal cross-correlograms between X target motion and X response motion (X vs. X), Z target motion and Z response motion (Z vs. Z), and X target motion and Z response motion (X vs. Z) for all five human observers for three visual conditions: (A) Both on-screen images are equally bright (∆O = 0.0 OD). (B) The left-eye image is dark (i.e., 75% of maximum luminance) and the right-eye image is bright (∆O = −0.6 OD). (C) The left-eye image is bright and the right-eye image is dark (i.e., 75% of maximum luminance; ∆O = +0.6 OD). Tracking in X is comparatively swift, tracking in Z is more sluggish, and the impact of horizontal target motion on the depth response (X vs. Z) depends systematically on the luminance differences between the eyes.

Recall that, in the context of the Pulfrich effect (see Figure 4), interocular processing delays cause horizontally moving targets to be misperceived as moving in depth. We examined whether this signature of the Pulfrich effect is present in binocular tracking in depth; that is, we examined whether horizontal target motion is associated with response movements in depth. The cross-correlations between X target motion and Z response motion clearly show that such associations depend on the luminance condition (X vs. Z) (Figures 7A to 7C). When the left eye is dark (Figure 7B), there tends to be an initial positive lobe, followed by a second negative lobe. When the right eye is dark (Figure 7C), the shapes of the X versus Z cross-correlograms are approximately mirror reversed. The dependence of the X versus Z cross-correlograms on luminance condition suggests that they may be useful for recovering differences in the time course of visual processing between the eyes. Note that, when both eyes are bright (Figure 7A), there are small but systematic deviations from zero, suggesting a baseline (∆O = 0.0 OD) asymmetry in left- and right-eye processing (see below). 

Binocular target tracking: Model and simulation

We developed a model to determine whether the X versus Z cross-correlograms can be used to recover the time course of differences in visual processing between the eyes. The model incorporates projective geometry, the left- and right-eye impulse response functions, and the statistics of target motion. We show analytically that the cross-correlation of X target motion and Z response motion is proportional to the difference between the temporal impulse response functions associated with the left and right eyes. That is,  

\begin{eqnarray} {\dot x_T} \otimes { {{\dot{\tilde z}}_T}} \propto {h_R} - {h_L} \qquad \end{eqnarray}

(1)

where \({\dot x_T}\) is the X velocity of the target, \({{\dot {\tilde z}}_T}\) is the Z velocity of the response, and h_L and h_R are the left- and right-eye impulse response functions, respectively, under the assumption of a linear system (see Supplementary Material for derivation). Thus, it may be possible to use binocular 3D target tracking to estimate the temporal evolution of interocular differences in visual processing. 

Simulations confirm the analytic results. The simulations were performed as follows. First, for a given virtual random walk in X and Z, we calculated the corresponding left- and right-eye on-screen image positions of the target on each time step using projective geometry. Next, we convolved the on-screen image positions with idealized left- and right-eye impulse response functions to obtain the left- and right-eye effective image positions. Lower luminance was assumed to shift the corresponding impulse response function in time. Finally, we back-projected the left- and right-eye effective image positions into 3D space (see Methods). The 3D tracking response was assumed to equal the position of the back-projected location in space. 

Simulated tracking performance for an observer with different impulse response functions in the two eyes—one shifted to simulate the effect of lower luminance—is shown in Figure 8. The impulse response functions are shown in Figure 8A. The simulated observer smoothly tracks the horizontal target motion component with a consistent interocular delay, as in the monocular tracking experiment (Figure 8B, top). Tracking performance in depth, however, is distinguished by obvious inaccuracies (Figure 8B, bottom). The over- and undershooting of the response in depth relative to the target depth depends systematically on horizontal target motion (Figure 8B, rectangular boxes). This dependence is the hallmark of the Pulfrich effect and can be quantified by the cross-correlogram of target X motion with response Z motion. The thick curves in Figure 8C show this cross-correlogram. Importantly, the difference between the left- and right-eye temporal impulse response functions, shown by the thin curves in Figure 8C, beautifully predicts the X versus Z cross-correlogram, as indicated by Equation 1. 

Figure 8.

 

Simulated binocular tracking through depth with interocular luminance differences. (A) Impulse response functions for the model observer's left and right eyes when the left eye is dark and the right eye is bright. (B) Resulting tracking performance in X and Z when the left eye is dark and processed with a delay relative to the right eye. Target motion in X is tracked smoothly and accurately, but large tracking inaccuracies are apparent in Z. Large over- and underestimates of depth (i.e., Z position) are caused by rightward and leftward target movements, respectively, the hallmark of the classic Pulfrich effect. The vertical rectangles highlight regions of the time series where these effects are most noticeable. (C) The cross-correlation of X target motion and Z response motion (thick curves) from the simulated data and the difference between the left- and right-eye impulse response functions (thin curves) are plotted for two different conditions with one eye dark and the other eye bright (colors). The simulated data agree with the mathematical prediction in Equation 1. (D) Empirical (thin curves) versus predicted (thick curves) data for the first human observer. The predictions are obtained by first fitting log-Gaussian functions to the cross-correlograms for relevant conditions in the monocular tracking experiment (see Figure 3A) and then taking the difference of the fits (see Equation 1).

Binocular target tracking: Empirical results and model fits

To examine whether the empirical binocular target tracking data are predicted by discrepancies between the left- and right-eye impulse response functions, we fit the cross-correlograms in the bright and dark conditions of the monocular tracking experiment (see Figure 3A) with log-Gaussian functions, took the difference of the fits (see Equation 1), and compared the result to the empirically determined X versus Z cross-correlograms. Data from the first human observer have clear similarities to the predictions (noisy vs. smooth curves) (Figure 8D). 

Recall that the monocular left- and right-eye cross-correlograms reflect the processing dynamics of both the visual and the motor systems. However, if the dynamics of the motor system are stable across two monocular conditions (e.g., left eye dark, right eye bright), then the difference between the corresponding cross-correlograms should reflect only differences in visual processing (i.e., the differences between the left- and right-eye impulse response functions; see Equation 1). Thus, assuming that nonlinearities associated with binocular combination can be safely ignored in the context of this analysis (and no other complicating factors are present), the difference between the left- and right-eye cross-correlograms should predict the cross-correlogram between target X motion and response Z motion during binocular tracking, as it does in the simulated dataset. 

The differences between fits to the monocular cross-correlograms (see Methods) nicely predict the binocular X versus Z cross-correlograms in the first human observer (Figure 8D, smooth curves). (The baseline X vs. Z asymmetry in the left-eye bright/right-eye bright condition was subtracted off before making the comparison; see Figure 7A.) In this observer, binocular tracking performance is predicted by discrepancies between the left- and right-eye impulse response functions. Thus, when the left- and right-eye on-screen images have different luminance levels, the impact of horizontal target motion on the human response in depth is well accounted for by differences in the monocular cross-correlograms. 

Across observers, however, the prediction quality is mixed, and in some cases it is poor (Figure 9). The first lobes of the cross-correlograms are reasonably predicted in some observers but not at all in others. The fact that the initial lobes are well captured in the first two observers is, however, intriguing. Additional lobes of the cross-correlograms are poorly accounted for in all observers, except perhaps for observer S1. The prediction quality and the raw data are particularly bad for observer S3, who reported struggling to track the target in depth, although one would be hard-pressed to detect these struggles from the Z versus Z cross-correlograms. We speculate that these additional lobes, which tend to occur half a second or more after the onset stimulus motion, are driven by fundamentally different processes (e.g., feedback delays due to binocular combination) (Goldreich, Krauzlis, & Lisberger, 1992) than those that are measured in the monocular tracking and traditional forced-choice psychophysics experiments. Hence, although XZ correlation data from the binocular conditions are well suited, in theory, to estimate the time course of temporal processing differences between the eyes (see Figures 8A to 8C), the discrepancies between model and human cross-correlograms (especially after ∼500 ms) make such analysis impractical with the present framework. 

Figure 9.

 

Binocular tracking in depth for all human observers (Note that the data and predictions in the leftmost panel are identical to the data and predictions in Figure 8D). Horizontal target motion systematically affects human tracking responses in depth curves (thin, noisy curves). The blue and red curves depict how the X versus Z cross-correlograms in the unequal luminance conditions ∆O = −0.6 OD and ∆O = +0.6 OD changed from baseline (i.e., both eyes bright; see Methods). Blue curves represent the X versus Z cross-correlograms when the left eye is dark and the right eye is bright. Red curves represent the X versus Z cross-correlograms when the left eye is bright and the right eye is dark. Predictions from the difference between the monocular cross-correlograms (thick, smooth curves) in the corresponding visual conditions (dark eye: OD = 0.6; bright eye: OD = 0.0) account for the first lobe of the binocular X versus Z cross-correlograms in roughly half of the conditions. The second lobes and third lobes of the cross-correlograms are poorly accounted for. The uneven ability of the model (see Equation 1) to predict the data suggests that other unmodeled factors contribute to binocular tracking performance.

The analytical results, simulations, and partial experimental validation regarding tracking in depth are intriguing, but the discrepancies between the predictions and results suggest that more work must be done. The fact that the additional lobes are unaccounted for by the predictions suggests the involvement of other processes in binocular combination or in binocular tracking in depth. These phenomena are potentially quite interesting, but a full analysis of them must await targeted experiments and analyses in the future. 

Methods

Participants

Five human observers participated in the experiment: four males and one female. Two were authors and the rest were naïve to the purposes of the experiment. The observers were 26, 31, 36, 41, and 55 years old at the time of the measurements. All had normal or corrected-to-normal visual acuity (20/20), and normal stereoacuity (20 arcsec) as determined by the titmus stereo test. All participants provided informed consent in accordance with the tenets of the Declaration of Helsinki, and the protocol was approved by the Institutional Review Board at the University of Pennsylvania. 

Apparatus

Stimuli were displayed on a custom-built four-mirror stereoscope. Left- and right-eye on-screen images were presented on two identical VIEWPixx monitors (VPixx Technologies, Saint-Bruno, QC, Canada). The gamma function of each monitor was linearized using custom software routines. The monitors were 52.2 × 29.1 cm, with a spatial resolution of 1920 × 1080 pixels, a refresh rate of 120 Hz, and a maximum luminance of 105.9 cd/m². After light loss due to mirror reflections, the maximum luminance was 93.9 cd/m². A single AMD FirePro D500 graphics card (Advanced Micro Devices, Santa Clara, CA) with 3-GB GDDR5 VRAM controlled both monitors to ensure that the left- and right-eye images were presented simultaneously. To overcome bandwidth limitations of the monitor cables, custom firmware was written so that a single color channel drove each monitor; the red channel drove the left monitor and the green channel drove the right monitor. The single-channel drive to each monitor was then split to all three channels for grayscale presentation. 

Observers viewed the monitors through a pair of mirror cubes positioned one interocular distance apart (Supplementary Figure S1). Heads were stabilized with a chin and forehead rest. The mirrors were adjusted such that the vergence distance matched the distance of the monitors. The monitors were positioned at a distance of 100 cm. This distance was confirmed both by a laser ruler measurement and by a visual comparison with a real target at 100 cm. The mirror cubes had 2.5-cm openings which yielded a field of view of 15° × 15°. At this distance, each pixel subtended 1.09 arcmin. Stimulus presentation was controlled via the Psychophysics Toolbox-3 (Brainard, 1997). Anti-aliasing enabled accurate presentations of disparities as small as 15 to 20 arcsec. 

Neutral-density filters

To control the stimulus luminance for each eye and to induce luminance differences between the eyes we placed “virtual” neutral-density filters in front of the eyes. First, we converted optical density to transmittance, the proportion of incident light that is passed through the filter, using the standard expression T = 10^–OD, where T is transmittance and OD is optical density. Then, we reduced the monitor luminance by a scale factor equal to the transmittance. We have previously verified that real and virtual neutral-density filters with equivalent strengths yield identical performance (Burge et al., 2019). The optical density of the neutral-density filters took on values of 0.0, 0.3, and 0.6. For the experiments with binocular viewing conditions (binocular tracking and forced-choice psychophysics), the interocular difference in optical density  

\begin{eqnarray}\Delta O = O{D_R} - O{D_L} \qquad \end{eqnarray}

(2)

quantifies the luminance difference between the eyes. In all binocular conditions, at least one eye had an optical density of 0.0. Interocular differences in optical density thus ranged from −0.6 to 0.6: ∆O = −0.6 OD, −0.3 OD, 0.0 OD, +0.3 OD, +0.6 OD. At the extremes of this range, the stimulus to one eye had only 25% the luminance of the stimulus to the other eye. 

Procedure: Tracking experiments

Tracking data and forced-choice data were collected from each human observer in a within-subjects design. In the tracking experiments, the observers controlled a mouse with their preferred hand in a comfortable position; all were right handed. Observers initiated each run by clicking the mouse, which caused the target bar to appear in the center of the screen. After a stationary period of 500 ms, the target followed a random-walk trajectory for the next 11 seconds. The task was to track the target as accurately as possible with a small, dark mouse cursor. 

In the monocular tracking experiment, data were collected in left-eye-alone and right-eye-alone conditions. For each eye, tracking data were collected at one-quarter maximum luminance, at half maximum luminance, and at maximum luminance. These values are equivalent to placing neutral-density filters with optical densities of 0.6, 0.3, and 0.0, respectively, in front of the viewing eye. The non-viewing eye was occluded with an eye patch. Data were collected in 20 intermixed blocks of 12 runs each for a total 40 runs per condition. 

In the binocular target tracking experiment, data were collected in each of three luminance conditions that were the analogs of the conditions in the monocular tracking experiment: left eye at one-quarter maximum luminance with the right eye at maximum luminance (i.e., left eye dark and right eye bright; ∆O = −0.6 OD); both left and right eye at maximum luminance (∆O = 0.0 OD); and left eye at maximum luminance and right eye at one-quarter maximum luminance (i.e., left eye bright and right eye dark; ∆O = +0.6 OD). Data were collected in 10 intermixed blocks of 12 runs each, for a total of 40 runs per condition. Left-eye-alone monocular tracking blocks, right-eye-alone monocular tracking blocks, and binocular tracking blocks were interleaved and counterbalanced. 

Procedure: Forced-choice experiments

In the forced-choice experiment, the observer's task was to report, with a button press, whether the target bar was moving leftward or rightward when it appeared to be nearer than the screen on its virtual trajectory in depth. Using a one-interval two-alternative forced choice procedure, nine-level psychometric functions were measured in each condition using the method of constant stimuli. The levels of on-screen interocular delay ranged between ±10.0 ms at maximum. The range and particular levels were set according to the sensitivity with which each human observer discriminated interocular delay. This was done to ensure good sampling of the psychometric functions. The psychometric functions were fit with a cumulative Gaussian via maximum-likelihood methods. The 50% point on the psychometric function—the point of subjective equality (PSE)—indicates the on-screen interocular delay required to null the interocular difference in processing speed. Observers ran 180 trials per condition in counterbalanced blocks of 90 trials each. 

Stimuli: Tracking experiments

For the tracking experiments, the target bar was a moving white vertical bar on a gray background. The target bar, which subtended 0.25° × 1.00° of visual angle, was vertically flanked by two stationary sets of 13 picket fence bars (Figure 2A). The gray background subtended 10.0° × 7.5° of visual angle. 

The target bar performed a random walk in X only or in X and Z. The X and Z positions of the target on each time step t + 1 were generated as follows  

\begin{eqnarray}x\left( {t + 1} \right) = x\left( t \right) + {\varepsilon _x};{\rm{ }}{\varepsilon _x}\sim N\left( {0,Q} \right) \qquad \end{eqnarray}

(3a)

 

\begin{eqnarray}z\left( {t + 1} \right) = z\left( t \right) + {\rm{ }}{\varepsilon _z};{\rm{ }}{\varepsilon _z}\sim N\left( {0,Q} \right) \qquad \end{eqnarray}

(3b)

where ε_x and ε_z are samples of Gaussian noise that determine the change in target position from the current to the next time step, and Q is the so-called drift variance. The drift variance controls the mean magnitude of the position change on each time step and hence the overall variability of the random walk. The variance of the walk positions across multiple walk instantiations, σ²(t) = Qt, is equal to the product of the drift variance and the number of elapsed time steps, t. The value of the drift variance in our task (0.8 mm per time step = 2.75 arcmin per time step) was chosen to be as large as possible such that each walk would traverse as much ground as possible while maintaining the expectation that less than one walk out of 500 (i.e., less than one per human observer throughout the experiment) would escape the horizontal extent of the gray background area (176 × 131 mm; 10.0° × 7.5°) before the 11-second trial completed. Ninety-five percent of the visited horizontal positions were within 50 mm of the starting point in X and in Z. Ninety-five percent of the visited positions in depth corresponded to disparities smaller than 11 arcmin, and less than one walk out of 500 generated a disparity larger than 20 arcmin. Even these largest disparities are much smaller than the upper disparity processing limit for stereopsis. 

In many experiments involving stereopsis, the distance of the virtual stimulus is specified by retinal disparity, which is directly determined by the stimulus position on each of two eye-specific monitors. The on-screen stimulus positions corresponding to a particular target position in stereoscopic space can be obtained using similar triangles (Supplementary Figure S2). The left- and right-eye on-screen image positions corresponding to a virtual target position are given by  

\begin{eqnarray} && {x_L} = {x_T} - \left( {\frac{{{z_T} - {z_S}}}{{{z_T}}}} \right)\left( {{x_T} + I/2} \right) \nonumber \\[.5em] && = {x_T}\left( {\frac{{{z_S}}}{{{z_T}}}} \right) + \frac{I}{2}\left( {\frac{{{z_S}}}{{{z_T}}}} \right) - \frac{I}{2} \qquad \end{eqnarray}

(4a)

 

\begin{eqnarray} && {x_R} = {x_T} - \left( {\frac{{{z_T} - {z_S}}}{{{z_T}}}} \right)\left( {{x_T} - I/2} \right) \nonumber \\[.5em] && = {x_T}\left( {\frac{{{z_S}}}{{{z_T}}}} \right) - \frac{I}{2}\left( {\frac{{{z_S}}}{{{z_T}}}} \right) + \frac{I}{2} \qquad \end{eqnarray}

(4b)

where z_S is the screen distance, and I is the interocular distance. 

Model: Tracking response in 3D

The effective left- and right-eye images are obtained by convolving the on-screen left- and right-eye images with eye-specific temporal impulse response functions  

\begin{eqnarray}{{\tilde x}_L}\left( t \right) = {x_L}\left( t \right) * {h_L}\left( t \right) \qquad \end{eqnarray}

(5a)

 

\begin{eqnarray}{{\tilde x}_R}\left( t \right) = {x_R}\left( t \right) * {h_R}\left( t \right) \qquad \end{eqnarray}

(5b)

where h_L(t) and h_R(t) are the left- and right-eye temporal impulse response functions, respectively. Convolving the left- and right-eye target velocities with the impulse response function of each respective eye gives the velocities of the effective left- and right-eye images. 

The predicted 3D response position is obtained by back-projecting the effective left- and right-eye images into 3D space. The X and Z positions of the 3D response are given by  

\begin{eqnarray}{{\tilde x}_T} = \frac{I}{2}\left( {\frac{{{{{\tilde x}}_L} + {{{\tilde x}}_R}}}{{{{{\tilde x}}_L} - {{{\tilde x}}_R} + I}}} \right) \qquad \end{eqnarray}

(6a)

 

\begin{eqnarray}{\tilde z_T} = I\left( {\frac{{{z_S}}}{{{{{\tilde x}}_L} - {{{\tilde x}}_R} + I}}} \right) \qquad \end{eqnarray}

(6b)

For the monocular tracking conditions, the stimuli only changed position in X and the observers wore an eye patch over one eye (see Figure 1B). 

The response velocities in X and Z are obtained by differentiating the X and Z response positions with respect to time  

\begin{eqnarray}{{\dot {\tilde x}}_T} = \frac{I}{2}\left[ {\frac{{2\left( {{{{\tilde x}}_L}{{{\dot {\tilde x}}}_R} - {{{\dot {\tilde x}}}_L}{{{\tilde x}}_R}} \right) + I\left( {{{{\dot {\tilde x}}}_L} + {{{\dot {\tilde x}}}_R}} \right)}}{{{{\left( {{{{\tilde x}}_L} - {{{\tilde x}}_R} + I} \right)}^2}}}} \right] \qquad \end{eqnarray}

(7a)

 

\begin{eqnarray}{{\dot {\tilde z}}_T} = I{z_S}\left( {\frac{{{{{\dot {\tilde x}}}_R} - {{{\dot {\tilde x}}}_L}}}{{{{\left( {{{{\tilde x}}_L} - {{{\tilde x}}_R} + I} \right)}^2}}}} \right) \qquad \end{eqnarray}

(7b)

where \({{\dot {\tilde x}}_L}\) and \({{\dot {\tilde x}}_R}\) are the velocities of the effective target images in the left and right eyes, respectively. The target velocities in X and Z are given by functions having an identical form, provided that the velocities of the left- and right-eye on-screen images are substituted for the velocities of the effective target images in the two eyes. 

To determine the impulse response function relating the target and response, we computed the zero-mean normalized cross-correlations between the target and response velocities  

\begin{eqnarray}\begin{array}{@{}l@{}} \rho \left( {\tau ;{\dot x},{\dot {\tilde x}}} \right) = \frac{1}{{\left\| {{\dot x}\left( t \right)} \right\|\left\| {{\dot {\tilde x}}\left( t \right)} \right\|}}\! \left[ {\sum\limits_{t = 1}^N {\left( {{\dot x}\left( t \right) - \bar{\dot x}} \right)\left( {{\dot {\tilde x}}\left( {t + \tau } \right) - \bar {\dot {\tilde x}}} \right)} } \right]\qquad \end{array}\!\!\!\!\!\!\!\!\nonumber\\ \end{eqnarray}

(8a)

 

\begin{eqnarray}\begin{array}{@{}l@{}} \rho \left( {\tau ;{\dot z},{\dot {\tilde z}}} \right) = \frac{1}{{\left\| {\dot z\left( t \right)} \right\|\left\| {{\dot {\tilde z}}\left( t \right)} \right\|}} \!\left[ {\sum\limits_{t = 1}^N {\left( {\dot z\left( t \right) - {\bar {\dot z}}} \right)\left( {{\dot {\tilde z}}\left( {t + \tau } \right) - \bar {\dot {\tilde z}}} \right)} } \right]\end{array}\!\!\!\!\!\!\!\!\nonumber\\ \qquad \end{eqnarray}

(8b)

where τ is the lag, \(\dot x\) and \({\dot {\tilde x}}\) are the target and response velocities in X, and \({\dot z}\) and \({\dot {\tilde z}}\) are the target and response velocities in Z. We also computed the zero-mean normalized cross-correlation between the X target velocity and the Z response velocity:  

\begin{eqnarray}\begin{array}{@{}l@{}} \rho \left( {\tau ;{\dot x},{\dot {\tilde z}}} \right) = \frac{1}{{\left\| {{\dot x}\left( t \right)} \right\|\left\| {{\dot {\tilde z}}\left( t \right)} \right\|}} \left[ {\sum\limits_{t = 1}^N {\left( {{\dot x}\left( t \right) - {\bar {\dot x}}} \right)\left( {{\dot {\tilde z}}\left( {t + \tau } \right) - {\bar {\dot {\tilde z}}}} \right)} } \right]\end{array}\!\!\!\!\!\!\!\!\nonumber\\ \qquad \end{eqnarray}

(9)

to determine how X target motion impacts Z response motion. The influence of X target motion on Z response motion is the hallmark of the Pulfrich effect. 

The normalized cross-correlation is obtained by performing a series of normalized dot products between two time series at each of many different lags. It quantifies the similarity of two time series as a function of time lag. Assuming a linear system, when the input time series (i.e., the target velocities) is white noise, as it is here, the cross-correlation with the response gives the impulse response function of the system. When computing the normalized cross-correlations, we excluded the first second of each 11-second tracking run to ensure that only steady-state performance was analyzed. (The time-series data indicate that most subjects had acquired the target within the first 500 ms of each trial.) First, we computed the normalized cross-correlation in each run (Equations 8 and 9). Then, we averaged these cross-correlograms across runs in each condition. It is these mean cross-correlation functions (i.e., the cross-correlograms) that are presented in the figures. 

Predicting binocular from monocular tracking performance

Predicting binocular tracking performance from monocular tracking performance was a three-step process. To determine how lateral target motion influences the depth response we first fit the monocular cross-correlograms with log-Gaussian shaped functions using least-squares regression. In nearly all cases, the log-Gaussian functions provided excellent fits to the monocular cross-correlograms when the curves exceeded 2 SD of the correlation noise, which was computed from lags less than 0 ms. Second, we computed the difference between the fits in the bright condition (OD = 0.0) in one eye and the dark condition (OD = 0.6) in the other eye (e.g., left eye dark, right eye bright). Third, we subtracted the X versus Z cross-correlogram in the condition where both eyes were bright from the cross-correlograms in the conditions of interest (i.e., one eye dark, one eye bright). This subtraction removed each observer's baseline asymmetry and isolated the changes caused by the interocular luminance differences. 

Stimuli: Forced-choice experiments

For the forced-choice psychophysics, we simulated the classic pendulum Pulfrich stimulus on the display. For each trial, the left- and right-eye on-screen target positions in degrees of visual angle were given by  

\begin{eqnarray}{x_L}\left( t \right) = E\cos \left( {2\pi \omega \cdot \left( {t + \Delta t} \right) + {\phi _0}} \right) \qquad \end{eqnarray}

(10a)

 

\begin{eqnarray}{x_R}\left( t \right) = E\cos \left( {2\pi \omega \cdot \left( t \right) + {\phi _0}} \right) \qquad \end{eqnarray}

(10b)

where E is the target movement amplitude in degrees of visual angle, ω is the temporal frequency of the target movement, ϕ₀ is the starting phase, t is time in seconds, and ∆t is the on-screen delay between the left- and right-eye target images. When the interocular on-screen delay is non-zero, a spatial binocular disparity results, and the target follows a near-elliptical trajectory of motion in depth. Negative values indicate that the left-eye on-screen image is delayed relative to the right-eye image. Positive values indicate that the left-eye on-screen image is advanced. 

The images of both eyes were presented coincidently on each monitor refresh with an on-screen binocular disparity that was equivalent to the desired on-screen delay. We calculated the equivalent on-screen disparity \(\Delta x = \dot x\Delta t\) from the target velocity and the desired on-screen delay and then shifted the on-screen spatial positions of the left- and right-eye images. The equivalent on-screen disparity as a function of time is given by 

\begin{eqnarray} && \Delta x\left( t \right) = {x_R}\left( t \right) - {x_L}\left( t \right) \nonumber \\ && = 2E\sin \left( {\pi \omega \cdot (\Delta t)} \right)\sin \left( {\pi \omega \cdot (2t + \Delta t) + {\phi _0}} \right) \qquad \end{eqnarray}

(11)

where negative disparities are crossed (i.e., nearer than the screen) and positive disparities are uncrossed (i.e., farther than the screen). The maximum disparity magnitude occurs when the perceived stimulus is directly in front of the observer and the lateral movement is at its maximum speed. When the stimulus is moving to the right, the maximum disparity in visual angle is given by ∆x_max = 2Esin(πω). 

The movement amplitude was 2.5° of visual angle (i.e., 5.0° total change in visual angle; see Figure 4D); the temporal frequency was 1 cycle per second; and the starting phase, ϕ₀, was randomly chosen to be either 0 or π. Restricting the starting phase to these two values forced the stimuli to start either 2.5° to the right or left of center, respectively, on each trial. 

Two sets of five vertical 0.25° × 1.00° bars in a picket-fence arrangement flanked the region of the screen traversed by the target bar (Figure 4D). The picket fences were defined by disparity to be in the plane of the screen and served as a stereoscopic reference of the screen distance. A 1/f noise texture, also defined by disparity to be in the plane of the screen, covered the periphery of the display to help anchor vergence and serve as a further stereoscopic reference to the screen distance. A small fixation dot was located at the center of the screen, at which observers were instructed to fixate. 

Discussion

We have shown that continuous target tracking in two and three dimensions can measure the time course of visual processing. We have also shown that millisecond-scale differences in visual processing between the eyes can be estimated with precision that is comparable to traditional psychophysics. Here, we discuss some of the advantages of the target-tracking paradigm over traditional psychophysics, consider future methodological challenges, speculate about the potential for clinical application, and reflect on how the current results fit in the historical development of the science. 

Time-course information provides potential for enhanced explanatory power

The ability to recover the time course of information processing provides the potential for new explanatory power. For example, in the forced-choice psychophysics task, when sinusoidal target motion was presented in the plane of the screen and one eye was darker than the other, rather than the expected trajectory (Figure 10A, black), one observer spontaneously reported perceiving elliptical trajectories in depth that were not aligned with the screen (Figure 10A, colors). Upon follow-up debriefing, two additional observers reported perceiving misaligned elliptical trajectories. 

Figure 10.

 

Anomalous Pulfrich percepts. (A) Some observers spontaneously reported perceiving anomalous near-elliptical motion trajectories that were not aligned with the screen (blue or red). Other observers perceived trajectories that were aligned with the screen (black). (B) Effective neural image positions as a function of time can elicit the anomalous perceived motion trajectories in (A). Left-eye (blue) and right-eye (red) neural image positions when the processing was delayed and damped in the left eye (left subplot) or right eye (right subplot). (Note that the color schemes are consistent across all subpanels.) (C) Temporal impulse response functions that have different temporal integration periods in the two eyes can account for the neural image positions in (B). Impulse response functions with longer temporal integration periods tend to dampen the amplitude of the effective image position in that eye. (Left subplot) Right-eye processing was fast and left-eye processing was delayed (i.e., had a longer time to peak) with a longer temporal integration period (i.e., had a larger full width at half-height). (Middle subplot) Slow-eye processing was delayed but not damped relative to the fast-eye processing, and so did not cause anomalous Pulfrich percepts. (Right subplot) Left-eye processing was fast and right-eye processing was delayed with a longer temporal integration period.

The most likely proximate cause of these anomalous Pulfrich percepts is that the amplitude of the effective image motion in the more slowly processed eye is not merely delayed but also damped relative to the effective image motion in the other eye (Figure 10B). Shape differences between the impulse response functions in the two eyes can lead to such interocular differences in the amplitude of the effective neural image positions across time (Figure 10C). Consider the case where the left eye is darker than the right eye, and the left-eye impulse response function peaks at a longer latency and has a longer temporal integration period than the right-eye impulse response function (Figure 10C, left). The effective position of the left-eye image will be delayed and damped relative to the right-eye image (Figure 10B, left). Stereogeometry dictates that, as a consequence, observers will perceive the target stimulus undergoing “front left” motion along a trajectory that is rotated such that its primary axis is askew to the screen (Figure 10A, blue). 

These ideas have recently been tested empirically; however, rather than using luminance differences between the eyes to induce temporal differences in visual processing, spatial frequency differences were used (Chin & Burge, 2022). High spatial frequencies reliably cause i) more delayed neural and psychophysical responses than low spatial frequencies (Bair & Movshon, 2004; Harwerth & Levi, 1978), and ii) temporal integration periods with longer durations than low frequencies (Bair & Movshon, 2004; Vassilev, Mihaylova, & Bonnet, 2002). Traditional forced-choice and continuous target-tracking psychophysical experiments were conducted. 

In the forced-choice experiment, observers viewed dichoptically presented oscillating Gabors. The left- and right-eye Gabors were identical in all respects except that the carrier spatial frequency was higher in one eye than in the other (e.g., 3 cycles per degree in one eye and 1 cycle per degree in the other). The task was to report whether the apparent trajectory in depth was oriented left-side or right-side back from the plane of the screen (Figure 10A, red and blue). The prediction was that the eye with the higher spatial frequency would have a longer temporal integration period (Figure 10C). This longer temporal integration period would cause increased neural damping (Figure 10B). And the increased neural damping would, in turn, cause the apparent motion trajectory to be misaligned with the plane of the screen (Figure 10A). This prediction was borne out in the data. However, the evidence that the perceptual effects were due to temporal integration periods was quite indirect. 

In the continuous psychophysics experiment, participants tracked Gabor targets that were identical to those used in the forced-choice experiment. The prediction was that the higher frequency Gabors would be associated with cross-correlograms having longer temporal integration periods (Figure 10C). This prediction was also borne out in the data. Also, it was shown that the durations of the temporal integration periods estimated directly from the tracking experiment correlated well with the durations that were inferred from the forced-choice experiment. Continuous psychophysics thereby provided more direct evidence than traditional psychophysics for the proposed explanatory account of the anomalous Pulfrich effect (Figure 10). 

Hence, the results from the current article and from Chin and Burge (2022) together provide evidence that traditional and continuous psychophysics paradigms yield similar estimates of i) processing delay (or latency), and ii) the duration of temporal integration. The rich temporal information provided by continuous psychophysics can, in some cases, provide more direct evidence of the underlying causes of perceptual phenomena than can traditional psychophysics. Traditional psychophysics typically provides snapshots of visual processing and perception, whereas continuous psychophysics provides something more akin to video clips. 

Preservation of visual processing differences in the visuomotor response

The millisecond-scale differences in processing that underlie the Pulfrich effect have almost certainly arisen by early visual cortex (Carney et al., 1989; Vassilev et al., 2002; Wolpert, Miall, Cumming, & Boniface, 1993). Indeed, there is a widespread view that these processing differences underlying the Pulfrich effect have their physiological origin in the retina itself (Mansfield & Daugman, 1978; Prestrude, 1971; Wolpert et al., 1993; also see Bernhard, 1940). We have shown that temporal delays in visual processing are faithfully preserved in the movement dynamics of the hand. To cause a motor response, electrical impulses must travel down multiple myelinated axon sheaths, and chemical communication must occur at multiple synaptic junctions as signals move from the cortex to the brain stem and down the arm to the hand; the motion of the hand represents the culmination of all of these processes. Despite the myriad signal transformations and delays that occur after early visual cortex, the tiny interocular sensory delays that cause the Pulfrich effect are preserved in the motor response. 

The fact that these small differences are preserved in the visuomotor response of the hand is striking. There is a clear evolutionary advantage for motor responses to occur as rapidly and consistently as visual processing allows. But there is no guarantee that this will happen. For example, the delay associated with a signal at one stage of processing may be magnified as the signal proceeds through subsequent stages of processing (Sternberg & Knoll, 1973). The fact that measurements from continuous target-tracking psychophysics are sensitive to small changes in temporal processing bodes well for its applicability to a wide array of research questions that depend on precise temporal information. 

Implications and applications

Traditional forced-choice psychophysical techniques are the gold standard in sensory–perceptual experiments. Forced-choice paradigms are our current best tools for minimizing measurement error. and they also sit atop a strong theoretical foundation for data analysis that is grounded in signal detection theory (Green & Swets, 1966). The price one pays for using traditional psychophysics, however, is that experimental data take a long time to collect, and the experiments themselves can be tedious for observers to participate in. Although merely inconvenient in many laboratory settings, time and tedium become very real problems if one wants to make measurements in children or in clinical populations that might be unwilling or unable to produce a satisfactory number of trials (Candy & Cormack, 2022). Moreover, instructions that seem simple for experienced psychophysical observers, such as “press the left arrow key when the stimulus appears to be moving left when it is in front of the screen, and the right arrow if it appears to be going the other way” might be confusing to young children and unintelligible to preverbal infants. Indeed, because infants naturally follow moving visual targets, continuous psychophysics should enable the behavioral assessment of visual function early in the developmental arc (Mestre, Neupane, Manh, Tarczy-Hornoch, & Candy, 2023). Finally, if one wishes to study effects that have large individual differences, then a technique is needed in which it is possible to obtain a satisfactory amount of data from a single observer in a short amount of time so that it is, in turn, possible to collect good data from a very large number of observers. 

In clinical settings, vision is often assessed imprecisely, in large part because of severe time restrictions. Multiple visual functions (e.g., acuity, binocular function, color vision) are assessed with different tests requiring different instructions. There may be advantages to having patients perform only a single task–target tracking–and to have the target defined by different levels of spatial detail, binocular disparity, or color on different tracking runs (e.g., Chin & Burge, 2022). It has been recently shown, for example, that the contrast sensitivity function can be rapidly measured using continuous psychophysics (Mooney, Alam, Hill, & Prusky, 2020). Certain pathologies (e.g., psychophysical, physical, developmental) are likely to have sensory–motor processing delays as an indicating factor. For example, visual processing delays associated with amblyopia have recently been measured using continuous psychophysics (Gurman & Reynaud, 2024). Given its accuracy, efficiency, and relative ease of use, continuous psychophysics may have substantial utility as a tool for clinical assessment. 

The arms and hands are large, heavy, and sluggish. The eyes are smaller, lighter, and more responsive. The eyes thus have the potential to increase the fidelity of tracking performance. When the eyes are tracking random walk motion, for example, the onset latency of smooth pursuit eye movements tends to be in the range of 80 to 120 ms (Mulligan, Stevenson, & Cormack, 2013; Tavassoli & Ringach, 2009). Although smooth pursuit latencies depend on the properties of the target being tracked, these latencies are consistently less than the 150- to 200-ms onset latencies of the motor response of the hand in the current tracking experiments (see Figure 3). Delays in neural processing of sensory information have been shown to determine the delays of smooth pursuit eye movements (Lee, Joshua, Medina, & Lisberger, 2016). Finally, there tends to be considerably less intersubject variability with smooth pursuit than with hand movements (Tavassoli & Ringach, 2009). By using eye movements, it should be possible to adapt continuous target tracking methods for use in young children and animal models. Young children and many animals reflexively follow moving targets with their eyes, potentially obviating the need for verbal instruction (in the case of children) or extensive training (in the case of animals). Indeed, continuous target tracking with eye movement monitoring has been used to measure how macaques and marmosets process optic flow (Knöll, Pillow, & Huk, 2018) and to measure both vergence and accommodative responses in infants as young as 5 weeks old (Downey, Pace, Seemiller, Candy, & Cormack, 2017). Using eye movements instead of, or in addition to, hand movements as the response effector in continuous target-tracking experiments may therefore have a number of advantages, and these probable advantages will only increase as eye-tracking technology continues to improve. 

Binocular eye tracking, in particular, might have interesting applications for assessing temporal processing differences between the eyes. First, binocular target tracking is potentially more efficient than monocular tracking; a single binocular run can yield information about interocular processing differences, whereas at least two distinct runs (one for each eye) must be performed with monocular tracking. Second, with binocular tracking, the interocular comparison is performed automatically by the binocular visual system before the motor system is engaged. With monocular target tracking, the comparison is performed only post hoc by the experimenter after the motor response has been collected in two different conditions. Third, binocular tracking eliminates the possibility that state changes (e.g., alertness, motivation) between runs could corrupt estimates of interocular delay. Finally, as with the Pulfrich effect, small changes in the processing by one eye (e.g., due to a pathology) might produce detectable interocular differences even though the change in the monocular response per se would be impossible to detect without some previously recorded baseline. Of course, for these benefits to be fully realized, the reasons underlying the mismatches between the binocular tracking predictions and data (see Figure 9) must first be understood. Nevertheless, assessing the efficiency of monocular versus binocular target tracking for such purposes could be a useful direction for future work. 

The continuously changing stimulus and response also afford new possibilities for the application of analytical methods from systems neuroscience—or control theory (Bonnen et al., 2015; Burge, Ernst, & Banks, 2008; Straub & Rothkopf, 2022)—to psychophysical data. In research on human perception and behavior, it is common to compare human performance to that of a normative model of the task (Burge, 2020; Burge & Jaini, 2017; Ernst & Banks, 2002; Geisler, 2011; Kording & Wolpert, 2004). But, normative models of psychophysical tasks are rarely constructed to predict continuous responses over time. In systems neuroscience research, however, popular models for neural system identification are designed to recover computational-level descriptions of how stimuli drive neural response over time. Such models (e.g., the generalized linear model) are commonly applied to continuous-time stimuli and responses; however, they have rarely been applied to human psychophysical data, perhaps because human datasets collected with traditional forced-choice methods have insufficient time-course information for the full statistical power of the models to be realized (Knoblauch & Maloney, 2008; Macke & Wichmann, 2010; Murray, 2012). The continuous target-tracking paradigm, paired with recent developments linking normative models to human performance (Burge & Geisler, 2015; Chin & Burge, 2020; Kim & Burge, 2018; Kim & Burge, 2020) and to methods for neural systems identification (Burg et al., 2021; Burge & Jaini, 2017; Iyer & Burge, 2019; Jaini & Burge, 2017; Park, Archer, Priebe, & Pillow, 2013), represents an exciting direction for future research. 

The reverse Pulfrich effect

It has long been known that interocular differences in luminance cause interocular differences in processing speed; the darker image is processed more slowly (Lit, 1949; Pulfrich, 1922). More recently, it was discovered that interocular differences in blur also cause interocular differences in processing speed: The blurrier image is processed more quickly (Burge et al., 2019; Rodriguez-Lopez et al., 2020). Given that blur decreases contrast and given that decreases in contrast tend to decrease processing speed (Albrecht, 1995; Bair & Movshon, 2004; Levi, Harwerth, & Manny, 1979; Nachmias, 1967; Reynaud & Hess, 2017; Shapley & Victor, 1978; Vassilev et al., 2002), it may at first seem surprising that blur increases rather than decreases processing speed. However, it is also known that, compared with lower spatial frequencies, higher spatial frequencies are processed with longer delays (Albrecht, 1995; Bair & Movshon, 2004; Levi et al., 1979; Min, Reynaud, & Hess, 2020; Nachmias, 1967; Reynaud & Hess, 2017; Shapley & Victor, 1978; Vassilev et al., 2002), and temporal integration periods (Bair & Movshon, 2004; Chin & Burge, 2022). Hence, because blur reduces the contrast of high spatial frequencies (fine detail) more than low spatial frequencies (coarse detail) (Burge & Geisler, 2011; Burge & Geisler, 2012; Campbell & Green, 1965; Navarro, Artal, & Williams, 1993), a blurrier image, having fewer fine details, is processed more quickly. Experiments have confirmed that this explanation accounts for the reverse Pulfrich effect (Burge et al., 2019; Chin & Burge, 2022). 

Many stimulus properties impact the temporal aspects of visual processing: luminance, contrast, spatial frequency, blur, color. The visual field location at which a given stimulus property is processed also impacts processing delay and the duration of temporal integration (Burge & Dyer, 2023; Kelly, 1984). The computational rules relating image properties to processing speed remain to be discovered. The target-tracking paradigm, because of the rich temporal information it provides, may be well suited for investigating this problem, as well. 

Conclusions

In this paper, we used continuous target-tracking psychophysics to demonstrate that small (i.e., less than 10 ms) delays in visual information processing, likely arising in the retina, are preserved throughout the entire sensory–motor loop. We showed that differences in visual processing can be recovered from the motor responses of the hand. Continuous psychophysics is well positioned to become an indispensable tool in the experimental toolkit for measuring the temporal dynamics of visual processing, an important but understudied topic in vision research. 

Acknowledgments

The authors thank Stephanie Shields for extensive comments on a draft version of the manuscript. The authors also thank Kathryn Bonnen for useful discussion. This work was supported by a grant from the National Eye Institute and Office of Behavioral and Social Science Research, National Institutes of Health (R01-EY028571 to JB), and discretionary funds from the University of Pennsylvania (to JB). 

Commercial relationships: none. 

Corresponding author: Johannes Burge. 

Email: [e-mail]. 

Address: Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA. 

Footnotes

1  Visuomotor tracking itself is not new. What is new is that visuomotor continuous target-tracking psychophysics–based visuomotor measures of performance can be directly connected to measures of performance obtained from traditional forced-choice psychophysical techniques.

Footnotes

2  Visual and motor noise can reduce peak and average correlation values but should not impact the cross-correlogram shape (see Methods). Thus, noise should not bias estimates of delay or of the temporal integration period.