Spatial integration of optic flow information in direction of heading judgments

Abstract

While we know that humans are extremely sensitive to optic flow information about direction of heading, we do not know how they integrate information across the visual field. We adapted the standard cue perturbation paradigm to investigate how young adult observers integrate optic flow information from different regions of the visual field to judge direction of heading. First, subjects judged direction of heading when viewing a three-dimensional field of random dots simulating linear translation through the world. We independently perturbed the flow in one visual field quadrant to indicate a different direction of heading relative to the other three quadrants. We then used subjects' judgments of direction of heading to estimate the relative influence of flow information in each quadrant on perception. Human subjects behaved similarly to the ideal observer in terms of integrating motion information across the visual field with one exception: Subjects overweighted information in the upper half of the visual field. The upper-field bias was robust under several different stimulus conditions, suggesting that it may represent a physiological adaptation to the uneven distribution of task-relevant motion information in our visual world.

Introduction

The act of walking or steering a vehicle through a stationary environment produces a complex flow pattern on the retina (optic flow) that provides information about one's direction of heading. In the simplest scenario, when the direction of gaze is fixed, the optic flow radiates outward from a central focus of expansion. The position of the focus of expansion on the retina specifies projection of the direction of heading (hereafter referred to as DOH) vector in the image (Gibson, 1950). For constant flow patterns (simulating gaze-fixed movement along a straight line), humans are able to judge direction of heading with an accuracy of less than 0.2° when visual features near the DOH are on the retina (Warren & Kurtz, 1992; Crowell & Banks, 1996). This suggests a remarkable sensitivity to the information provided by optic flow about direction of heading. In this article, we investigate how young adult humans integrate information from different parts of the visual field to make direction of heading judgments. While human observers can effectively compute direction of heading with sparse optic flow input (Warren & Hannon, 1990), optimal performance requires integrating noisy sensory motion signals across the visual field. Here, we test the hypothesis that humans optimally integrate optic flow information from different regions of the visual field, much like they have been shown to integrate qualitatively different sensory cues (Ernst & Banks, 2002; Knill et al., 2003; Alais & Burr, 2004; Gu, Watkins, Angelaki, & DeAngelis, 2006; Brouwer & Knill, 2009; Issen & Knill, 2012). 

It is well known that the information content of any part of a flow field depends on its position relative to the DOH and that variability in human subjects' ability to judge direction of heading from isolated patches of a flow field varies with the information content within the patch (Warren & Kurtz, 1992; Crowell & Banks, 1996). However, little is known about the precise strategy undertaken by the visual system to integrate local motion information and form a global percept over large portions of the visual field. This may involve utilization of all information available across the visual field or, instead, selective and preferential processing of information from smaller portions of the visual field to guide the global percept. Because the information provided by different parts of the flow field depends critically on the spatial relationship between the location of flow and the true direction of heading (Crowell & Banks, 1996), the hypothesis predicts that the influence of subregions of optic flow information on subjects' heading judgments will depend in predictable ways on the spatial relationships between the subregions and the DOH. To test this optimal global integration hypothesis, we used a two-phase approach. First, we measured the performance of human observers in determining forward heading direction (up to 9° eccentricity) using visual information in the near periphery (between 10° and 20° eccentricity). By presenting optic flow suggesting slightly different directions of heading in different quadrants of the visual field (on some trials), we were able to measure the relative influence of the information in each quadrant on subjects' direction of heading judgments. Second, we simulated the performance on the experimental task of an ideal observer whose local estimates of motion were corrupted by sensory noise. The hypothesis predicts that the pattern of quadrant influences on subjects' judgments (e.g., as a function of quadrant position relative to the simulated direction of heading) will mimic that of the ideal observer. Systematic deviations between subjects' behavior and that of the ideal observer would shed light on any suboptimal integration strategies that the visual system might use. 

Experiment 1: Estimating direction of heading from flow fields—are humans ideal integrators?

Method

Participants

Nine naive adult subjects (five males, four females; aged 18–26 years) were recruited at the University of Rochester. The study was approved by the University's Institutional Review Board. All subjects had self-reported normal or corrected-to-normal vision and gave written informed consent to participate. Three of the participants (two males, one female) did not complete the task due to difficulty maintaining fixation or focus, and their data were excluded from further analysis. 

Overview of experimental logic

Subjects were seated in front of a 180°, floor-to-ceiling cylindrical screen, on which were projected dot flow patterns on a dark background, simulating straight-line trajectories through a three-dimensional cloud of points (Figure 1A). Near-forward self-motion was simulated in all cases, with directions of heading sampled from an 18° circle centered at the straight-ahead vector (relative to subjects' head positions). Since the region of the stimulus in the immediate vicinity of the focus of expansion provides the most reliable information about direction of heading, a large gray circle covered the central 20° of the field of view (i.e., with a radius of 10°). In addition, a physical aperture occluded parts of the stimulus that extended beyond 20° eccentricity. Therefore, the focus of expansion on any trial was occluded by the gray circle, and all visible moving dots appeared within an annulus spanning 10° and 20° eccentricity. A square reference grid was projected on the screen at all times. The lines bisecting the field of view horizontally and vertically were red, while other gridlines (positioned every 5° of horizontal and retinal eccentricity) were blue. This allowed subjects to align their perception of the direction of heading with a visual landmark that remained visible throughout the response period. 

Figure 1

 

(A) Photograph of a young adult observer participating in Experiment 1. The pointer used to indicate perceived direction of heading (DOH) is arrowed in back. (B) Schematic representation of stimulus display and possible anchor points. (C) Schematic representation of the optic flow in a perturbation trial (Panel 1) and the two component flow patterns from which it was generated (Panel 2: DOH located 6° to the LL; Panel 3: DOH located 3° to the LL).

The display was centered at eye level for each subject (see Figure 1B for schematic representation of the stimulus). At the start of each trial, a fixation cross was displayed at the intersection of the two red reference lines. Subjects maintained fixation while this cross was displayed for 1 s. An expanding dot pattern was then presented for 1 s, after which subjects indicated the perceived direction of heading using a custom-built device that functioned as a pointer (Figure 1A). The pointer was tracked by an Optotrak system (NDI, Waterloo, Ontario, Canada), and a red cursor was shown at the point on the screen toward which it was pointing. Subjects indicated their selected direction of heading by pressing a button on a gamepad held in the other hand. 

A subset of trials was designated as test trials, in which the simulated direction of heading was chosen from one of 12 positions along the diagonals of the display (at 3°, 6°, and 9° of eccentricity; Figure 1B). Test trials included 

•  
  anchor trials, in which the optic flow in all four quadrants of the display depicted a common direction of heading, and
•  
  perturbation trials, in which the optic flow in three of the quadrants depicted one of the four 6° eccentric directions of heading, while the other quadrant (randomly chosen) depicted a direction of heading that was 3° more or less eccentric than that depicted in the other three quadrants (see example in Figure 1C).

The remaining trials were designated as wild-card trials, in which the simulated direction of heading was chosen randomly and uniformly from within a 9°-radius circle centered on the fixation cross. 

Feedback was given at the end of each wild-card trial by displaying a small circled cross at the true direction of heading. No feedback was given on the test trials, both to avoid overlearning of the 12 oversampled test positions and because perturbation trials contained no single correct direction of heading. Because the simulated DOHs for the two anchor trials were separated by only 3° of visual angle and because the central occluder obscured the focus of expansion, the patterns did not look unnatural and subjects did not notice any conflict (or shear) in the perturbation trials. 

There were 32 perturbation conditions, which were generated as follows (see Figure 1C for a schematic diagram of a perturbation condition): 

1.  
   One of the four quadrants contained the direction of heading location. This is referred to as the DOH quadrant. The base direction of heading was 6° off fixation along an imaginary 45° line bisecting the quadrant.
2.  
   Independently, one of the four quadrants was selected to contain the perturbation. This is referred to as the perturbation quadrant. The optic flow in the other three (nonperturbation) quadrants depicted the base direction of heading.
3.  
   The flow pattern in the perturbation quadrant was manipulated to indicate a direction of heading 3° more or less eccentric than the base direction of heading. With four options for the DOH quadrant, four options for the perturbation quadrant, and two options for the perturbation direction, the experiment consisted of 32 configurations for perturbation trials.

By performing multiple linear regressions on the two-dimensional positions of user estimates for perturbation and anchor trials, it was possible to independently determine the influence of each visual field quadrant on the x and y components of user estimates of heading, as described under Data analysis (below). 

Equipment and setup

Participants were seated in an adjustable office chair in front of the cylindrical projection screen. A chin rest and forehead rest were adjusted so that the participant could lean forward into the head rest and view the screen comfortably. Stimuli were projected from a Christie Matrix 2500 projector (Christie Digital Systems USA, Cypress, CA) that was mounted on the ceiling 2.5 m away from the screen. Participants were seated 2.2 m away from the horizontal center of the screen and wore an eye patch on the right eye so that stimuli appeared monocularly to the left eye only. Monocular viewing eliminated the contribution of disparity information for interpretation of the stimulus, resulting in a more compelling perception of the stimulus as representing movement through a three-dimensional cloud of dots. Participants interacted with the display using a handheld pointer fitted with six Optotrak markers (Figure 1A). The marker positions were tracked at a frame rate of 60 Hz by an Optotrak Certus camera system. The system was calibrated by having the subject align the pointer tip with a series of 16 projected targets. After a successful calibration, the Optotrak system tracked the markers on the pointer and rendered an onscreen cursor aligned with the pointer tip. The subject either verified that the target tracked closely with the pointer tip or repeated the calibration procedure. 

To ensure that the display was centered at eye level for each subject, two thin rings 20° and 21° of visual angle in radius were projected on the cylindrical projection screen. An Edmund Optics (Barrington, NJ) mounted adjustable diaphragm (165-mm outer diameter) was positioned in front of the subjects, who adjusted its aperture so that they could see the entire inner circle but none of the outer circle. In this way, we ensured that each subject perceived a field of view 40° in diameter centered around a natural fixation point at eye level. Eye gaze position was tracked at 250 Hz by an Eyelink infrared camera (SR Research, Ottawa, Ontario, Canada) using pupil location and corneal reflection. A five-point calibration preceded each block of 125 trials, and a single-point drift correction at the point of central fixation was enacted whenever the experimenter noticed that the starting gaze position was off center for a few consecutive trials. All subjects were right handed, and during the experiment they used the Optotrak-marked pointer in the right hand as if it were a laser pointer. Simultaneously, they held an Xbox (Microsoft, Redmond, WA) gamepad in the left hand to register their responses and pace the trials (Figure 1A). 

Stimuli

Stimuli consisted of green dots (0.4° diameter) on a black background. Because of the physical aperture placed in front of the subjects and the projected occluder covering the central 20° eccentricity, all signal information visible to the subject was between 10° and 20° eccentricity. Approximately 100 green dots were visible in this range at any given time. Dot spacing was chosen to be uniform in polar coordinates in the image plane. 

At the start of every trial, each dot was given a random radius from the center of the screen, drawn from a uniform distribution between 2.5° and 24° of visual angle, and a random theta from a uniform distribution between 0° and 360°. The dot was also given a random simulated depth within the cloud, chosen from a uniform distribution between 2 and 2.5 m from the viewer. The motion vector of each dot was determined by simulating viewer motion at a speed of 2 m/s toward these dots, resulting in dot speeds ranging between 8.2° and 22.4°/s and a mean dot velocity in the image plane between 14° and 16°/s, varying slightly from trial to trial. If a simulated dot's motion reached a boundary point (closer than 2 m to the viewer or outside 24° of visual angle) or surpassed a lifetime of 500 ms, the dot would be extinguished and a new dot would be drawn at a location randomly selected by the process described above. This caused the dots to look like they were twinkling and helped obscure shearing effects along the boundaries of the perturbation region. Each trial presented a flow field stimulus that was 1 s in duration. 

Testing protocol

Following setup and calibration procedures, the participants conducted a practice session of 61 trials of the task, which consisted of fixating centrally while watching the optic flow stimulus and then using the pointer and Xbox gamepad to select the perceived direction of heading. At the start of the practice session, the subject was able to see a projected image of blue gridlines spaced by 5° of visual angle, with the vertical and horizontal meridians marked in red and intersecting at the observer's calculated eye level. Each trial began with the appearance of a fixation cross at this intersection; if subjects fixated this cross precisely for 1 s, a dot flow pattern appeared, lasting for 1 s. At the conclusion of each trial, the fixation cross disappeared, leaving only the gridlines and central occluder. This signaled to the subject that he or she was free to move his or her eyes and register a response. To do this, the subject would hold the pointer with the point facing the screen, and a yellow cursor would be projected to the screen location to which he or she was pointing. When the cursor reached a location that indicated the subject's perceived direction of heading, he or she would press the gamepad trigger to record the response. Following this, a mask of randomly moving dots with the same average dot speed and spacing as the stimulus dots was displayed for 500 ms. Feedback was provided by displaying the subject's chosen location in yellow and the veridical DOH location as a green cursor, both for 500 ms. In the practice session, all trials were of the wild-card type, so the DOHs were randomly chosen from within the central 9° of the display and feedback was given on every trial. 

For the initiation and duration of the flow display, a fixation threshold of 3° radius was enforced, and if the calibrated eye position fell outside this range, a “fixation loss” message was displayed, instructing the subject to press the gamepad trigger button to continue. In the event of a fixation loss, the aborted trial type was reshuffled into the trial order to be repeated later. Aside from halting after fixation losses, trials were self-paced, such that subjects initiated the next trial by pressing the gamepad trigger. 

Following the training block and a short rest period, the experimenter answered any questions the participant might have had and explained that the test blocks would follow. The test blocks were longer in duration and randomly gave intermittent feedback for the preassigned wild-card trials, which were presented in randomized order (described below). For trials that did not give feedback, the default screen with occluder and gridlines was displayed until the subject initiated the next trial. Each experiment block consisted of 125 trials, randomly ordered as follows: 

•  
  37 wild-card trials, generated by randomly placing DOHs in each of 37 evenly divided areas within the central occluded 10° of the display;
•  
  24 anchor trials (two each of the 12 possible anchor locations; Figure 1B); and
•  
  64 perturbation trials (two each of the 32 possible perturbation trial types).

Each experiment block took about 12 min to complete. Six experiment blocks were completed in a 90-min session, with short breaks between each block. 

Terminology: Retinotopic and functional locations

For each trial, the four quadrants of visual space were assigned two labels: a retinotopic label, specifying the location of the quadrant on the retina, and a functional label (Figure 2), specifying the position of the quadrant relative to the quadrant containing the DOH. 

Figure 2

 

Illustration of functional location terminology. In Panels A and B, the quadrant containing the direction of heading (DOH) is shaded in pink, although in Panel A this occurs in the LL and in Panel B it occurs in the LR. The other quadrants are labeled according to their position relative to the DOH quadrant: cross-hemi (quadrant in the hemifield across that containing the DOH), same hemi (quadrant in the same hemifield as the DOH quadrant), and diagonal (quadrant located diagonally across from the DOH quadrant).

We labeled the retinotopic location of a quadrant as upper right (UR), lower right (LR), upper left (UL), or lower left (LL). We labeled the functional location as being the quadrant containing the direction of heading (DOH), the adjacent quadrant in the same hemifield (same-hemifield), the adjacent quadrant in the opposite hemifield (cross-hemifield), or the quadrant located diagonally across from the quadrant containing the DOH (Diagonal, Figure 2). 

Data analysis

For each trial, the subject's response coordinates were recorded in (x,y) space of an imaginary frontoparallel image plane along with the (x,y) coordinates specifying the direction of heading indicated by the optic flow in each of the four quadrants (UR, LR, UL, and LL). In order to estimate the relative influence of each spatial quadrant on the estimation of DOH, we modeled the x and y components of subjects' DOH estimates as linear functions of the x and y components of the DOH depicted by the optic flow in each of the four quadrants:  

 

 

 

where w_UR and λ_UR represent the strength of influence of the optic flow in the UR quadrant on subjects' estimates of direction of heading in the x and y directions, respectively, and similarly for the other three quadrants. As noted previously, the relative influence of optic flow should depend on the location of the flow relative to the DOH (functional quadrant); thus, we fit four separate models of the forms given by Equations 1 and 2 to subjects' direction of heading estimates for conditions in which the true direction of heading was in each of the four retinotopic quadrants (e.g., UR, UL). To directly conceptualize the relative influence of each quadrant, we normalized the weights to each quadrant so they would sum to 1. Therefore, the weights reported below represent the relative influence of the flow in each quadrant, independent of multiplicative biases in subjects' responses (e.g., compressing or expanding their estimates radially away from fixation).  

Construction of the ideal observer

To visualize the expected information-weighting behavior that subjects would exhibit for optimal integration of information across the visual field in this task, we simulated the performance of an ideal observer limited only by sensory noise in local motion estimates. The ideal observer computes the most likely direction of heading given a set of two-dimensional velocity measurements at discrete locations in the visual field. As we describe below, the ideal observer effectively computes a weighted template match between the noisy optic flow pattern observed and the expected noiseless optic flow for each possible direction of heading. It then picks the direction of heading with the best resulting match. In other words, the ideal observer takes a simplified input that contains noisy measurements of the velocity of each dot in the display and chooses a direction of heading that maximizes the likelihood function p(v⃗_m|H; x⃗), where v⃗_m is a composite vector containing the measured velocity vectors for each dot in a display, x⃗ is a vector containing the positions of each dot, and H is a two-dimensional vector representing the direction of heading. We further simplified the model to take as input only the directions of the local velocity vectors (v⃗_m contains unit-length vectors in the direction of measured flow for each dot) because local speed in the flow pattern contributes little information about direction of heading (relative to direction), both in theory (Crowell & Banks, 1996) and in practice (W. H. Warren, Blackwell, Kurtz, Hatsopoulos, & Kalish, 1991). Furthermore, direction discrimination thresholds are independent of dot speed over a broad range of speeds, including the dot speeds used in our stimuli; thus, the variance of sensory noise on direction can be assumed to be independent of speed (Crowell & Banks, 1996). 

Since the direction of heading uniquely specifies the direction of feature motion at each point containing a feature in the visual field, we can rewrite the likelihood function as  

 

where v⃗_H is a vector containing unit normal vectors representing the direction of motion of each dot predicted by the heading H. We assume that sensory measurements of dot direction are independent and follow a von Mises distribution centered on the true direction in the image, giving for the likelihood function  

 

where θ_i is the angle between the measured velocity vector for dot i and the velocity vector predicted by a given direction of heading and k_i is the precision of the sensory measurement (to a good approximation, the inverse variance of the sensory noise; Equation 7).  

The cosine term can be rewritten as a dot product between two unit normal vectors so that the likelihood function becomes  

 

 

The log likelihood is given by a simple sum of dot products:  

 

 

Thus, selecting a maximum likelihood estimate of the direction of heading given noisy local velocity measurements reduces to selecting the heading whose template optic flow best matches the measured flow pattern (using the weightings given by k_i). 

Direction discrimination thresholds vary both with eccentricity (Crowell & Banks, 1996; Gros, Blake, & Hiris, 1998) and with direction (the oblique effect; Ball & Sekuler, 1980; Heeley & Buchanan-Smith, 1992; Gros et al., 1998; Churchland, Gardner, Chou, Priebe, & Lisberger, 2003). They increase slightly with increasing eccentricity and can be twice as high for motions 45° away from either cardinal direction as they are for the cardinal directions. For the speeds contained in our stimuli, however, direction discrimination thresholds are largely invariant to speed (Crowell & Banks, 1996). We modeled these effects by assuming that the standard deviation of directional noise (a) increased linearly with eccentricity, with a maximum at 20° eccentricity that was 50% greater than the minimum at 10°, and (b) varied sinusoidally with direction, with minima at the cardinal directions and peaks 45° away from the cardinal directions:  

 

 

 

where r_i and θ_i are the retinal eccentricity and direction of dot i, respectively, and N is the number of dots in the display. The Display Formula factor accounts for the fact that motion discrimination thresholds do not improve as one increases the number of dots in a display past a minimally small number (Warren & Hannon, 1990; Eagle & Blake, 1995; one has to increase the noise per dot in the model by the square root of N to give equal performance with increasing number of dots).  

Equation 8 gives rise to motion discrimination thresholds of 5° (in a two-alternative forced-choice task) in cardinal directions and 10° in oblique directions for motion stimuli presented at 10° eccentricity (Gros et al., 1998). Because the ideal observer simply maximizes a weighted average of the dot product between measured and predicted velocity vectors for each dot in a display, its estimate for a given set of sensory measurements is independent of the absolute magnitude of the directional noise. 

To compute the weights given by the ideal observer to flow information in the different quadrants of a stimulus, we simulated the ideal observer for 10,000 trials using stimuli generated from the same stochastic process used to create experimental stimuli. We applied the same regression analysis used for subjects' data to the ideal observer's estimates of heading direction to compute the relative influence of the information in each quadrant on the ideal observer's direction of heading estimates. Because the stimulus information provided by a single dot, or collection of dots, depends only on its position relative to the true direction of heading, we simulated the ideal observer only for the condition in which the direction of heading was in the UR quadrant. 

Results

Ideal observer analysis

The information provided by a small patch of optic flow about direction of heading is a function of the direction of the flow (equivalently positioned on the retina relative to the direction of heading; Crowell & Banks, 1996). In general, the flow in a local patch of the retina provides good information about the position of the focus of expansion in a direction perpendicular to the direction of flow but poor information about the position of the focus of expansion along the direction of flow (Crowell & Banks, 1996). For example, a small patch of flow moving horizontally must be located at the same vertical location on the retina as the DOH; thus, it constrains the vertical position of the DOH to be close to that of the patch (depending on sensory noise) but only weakly constrains the horizontal position of the DOH (which must be to the side of the image away from the direction of flow). 

Here, we localized flow to four quadrants for purposes of measuring spatial variations in the relative influence of flow information on subjects' heading judgments. Figure 3 illustrates the information provided by the flow in each quadrant of the visual field as contour plots of the likelihood functions computed from the flow information in each quadrant of an experimental stimulus that simulated a DOH in the UR quadrant at (6°, 6°). The information provided by the entire flow field is characterized by the product of the four likelihood functions (Figure 3E). Assuming a flat prior on DOH, this equates to the posterior probability of the direction of heading, given noisy measurements of each dot's motion direction. The likelihood functions are approximately elliptical, and the spread of the likelihood function is inversely related to the reliability of the information. Thus, for example, the information in the diagonal quadrant (Figure 3C) is least reliable in the oblique direction between the center of the quadrant and the focus of expansion, but more accurately specifies the DOH along a perpendicular direction to this. Information in the cross-hemifield quadrant (Figure 3B) is most reliable in the y dimension and least reliable in the x dimension. Because the focus of expansion is located along the 45° axis, the uncertainty ellipses are oriented obliquely. The uncertainty ellipses are opposite in the same-hemifield quadrant (Figure 3D), where information about the focus of expansion is more reliable in the x dimension than in the y dimension. As a result, the uncertainty ellipses are narrower in the x dimension. The information in the same quadrant as the focus of expansion (Figure 3A) is relatively reliable in both the x and y dimensions, but when combining information from all four quadrants (Figure 3E), this reliability is increased and less distorted along the 45° axis. 

Figure 3

 

(A–D) Sample contour maps of the likelihood functions associated with the flow information in each of the four quadrants of an experimental stimulus (for a direction of heading 6° up and to the right). The shaded quadrant in each figure indicates the quadrant used to compute the likelihood functions. (E) The likelihood function derived from the entire flow field (equal to the product of the likelihood functions in Panels A through D).

In general, we can visualize the impact of perturbing the information in one or another quadrant by visualizing the effect the perturbation will have on that quadrant's likelihood function. The important thing to note is that the influence of a perturbation will depend not only on the quadrant containing the perturbed flow but also on the direction of the perturbation. Using the conditions in which the DOH was in the UR quadrant (illustrated in Figure 3) as an example, in Experiment 1 three quadrants contained flow specifying a DOH at (6°, 6°) while the fourth quadrant contained flow specifying direction of heading perturbed by ±3° in toward the fovea or out away from the fovea. The primary effect of these small perturbations was to shift the likelihood function associated with the information in the perturbed quadrant, with a greater shift occurring along the dimensions of greater reliability. For example, Figure 3B shows that information in the cross-hemifield quadrant is most reliable along the y dimension. Thus, a perturbation in this quadrant will shift the ideal observer's estimate more strongly in the y dimension than in the x dimension. Figure 3C indicates that if the diagonal quadrant contained the perturbation, its likelihood function would be shifted along its long axis and the ideal observer would show a minimal change in its estimate of the DOH, equal in the x and y directions. This change would be minimal because of the high uncertainty (represented by wide likelihood curves) of information in the diagonal quadrant. 

To help visualize how the optic flow in each retinal quadrant should influence subjects' DOH judgments in Experiment 1, we simulated the ideal observer on the stimuli used in that experiment for conditions in which the DOH was in the UR retinal quadrant. Applying the regression analysis to the model's estimates of DOH gives four weight vectors—one for each retinal quadrant. Figure 4 shows the weight vectors for each retinal quadrant, classified by the quadrant's functional location (DOH, cross-hemifield, same hemifield, or diagonal). The weight vectors represent the proportional shift in the ideal observers' estimates caused by the experimental perturbations in each of the four retinal quadrants (proportional to the magnitude of the perturbation). When the optic flow in the three quadrants not containing the DOH specify a direction of heading equal to (6°, 6°) but the DOH quadrant contains flow indicating a direction of heading equal to (9°, 9°), the ideal observer's DOH estimates are biased directly toward the (9°, 9°) direction of heading specified by the flow in the DOH quadrant (Figure 4A). By contrast, the ideal observer's estimates are minimally affected by conflicting flow information when it is contained in the quadrant diagonally opposite from the DOH quadrant. When the conflicting information is contained in the quadrant above or below the DOH quadrant, the ideal observer's estimates are pulled more in the direction of the conflicting quadrant than in the orthogonal direction. 

Figure 4

 

Weights given by human observers to flow information during direction of heading (DOH) judgments in Experiment 1. (A) Relative influence of different functional locations on DOH judgments. Solid lines indicate functional location weights measured in human subjects. Dashed lines indicate functional location weights obtained for the ideal observer. Ellipses represent standard errors of the mean. (B) Retinotopic location weights averaged over the four different locations for the DOH.

As shown in Figure 4A, the optimal integration hypothesis predicts that a quadrant's functional designation will strongly affect how the perturbed optic flow within the quadrant influences subjects' DOH judgments. By contrast, a quadrant's retinotopic location should not by itself affect how the optic flow information within the quadrant influences DOH judgments. That is, the weight vectors computed for each retinotopic quadrant should be equal when averaged over the four possible locations of the DOH. These predictions form the point of comparison between the optimal integration model and human subjects' behavior. 

Human subjects' performance

The influence of optic flow perturbations in any given quadrant on DOH judgments can depend on the quadrant's functional and retinotopic location. Furthermore, perturbations can influence judgments more in one direction than another. To test for effects of these three factors, we sorted trials into four subsets corresponding to conditions in which the simulated direction of heading was in the UR, LR, UL, and LL quadrants. For each of these four conditions, we applied the regression analysis described by Equations 1 and 2 to estimate the relative influence of the flow in each of the four retinal quadrants on the x and y components of subjects' direction of heading judgments. This gave 32 weights for each subject indexed by three factors: a quadrant's functional location (DOH, same hemifield, cross-hemifield, or diagonal), its retinotopic location (UR, LR, UL, LL), and the component of the DOH judgment influenced by the perturbation (horizontal, x; vertical, y). A three-way repeated measures analysis of variance on subjects' measured weights revealed three significant effects: a main effect of functional location, F(3, 15) = 10.89, p < 0.0005; a main effect of retinotopic location, F(3, 15) = 8.34, p < 0.0017; and an interaction between functional location and the direction of the measured effect of a perturbation (x or y), F(3, 15) = 6.43, p < 0.005. No other effects approached significance. 

Figure 4A illustrates the two effects related to functional position. The solid lines represent subjects' average weight vectors for each functional quadrant location averaged over the four possible retinotopic locations these could appear in (depending on the location of the DOH). The main effect of functional location appears here as differences in the magnitude of the weight vectors, dominated by large decreases in weights for the quadrant diagonally across from the DOH quadrant. The interaction between functional position and the direction of a perturbation's influence on direction of heading judgments appears as the difference in orientation of the weight vectors. 

The effect of functional location on each quadrant's influence on DOH judgments qualitatively matches the predictions of the ideal integrator model. The main effect of retinotopic location, however, is not predicted by the ideal integration hypothesis. To further explore this effect, we averaged the weight vectors for each retinotopic location across the four DOH conditions and ran a post hoc three-way repeated measures analysis of variance on the averaged weights with the hemifield (left, right) of the quadrant as one factor, its vertical position in the visual field (top, bottom) as a second factor, and the spatial dimension (x, y) in which the weight was computed as a third factor. The only significant effect was a main effect of vertical position within the visual field, F(1, 5) = 13.58, p < 0.014. As shown in Figure 4B, subjects weighted the flow information in quadrants above the midline more than flow information in quadrants below the visual field midline. 

The predicted influence of a quadrant's functional location on horizontal and vertical weights is clearly present in the results from human subjects (solid lines in Figure 4A), with the same-hemifield quadrant having a stronger influence on the horizontal component of subjects' heading direction estimates; the cross-hemifield quadrant having a stronger influence on the vertical component; the DOH quadrant having a strong, balanced influence on both; and the diagonal quadrant having a weak influence and contributing little to either component of subjects' estimates. The qualitative pattern of subjects' weight vectors provides a good match to that of the ideal observer. Most notably, as predicted by the ideal observer, perturbations in the optic flow in the quadrants adjacent to the one containing the DOH had asymmetric effects on subjects' judgments of the horizontal and vertical components of the direction of heading. These results are consistent with efficient integration of flow information across the visual field. 

Subjects differed from the ideal observer in two ways. First, the magnitude of weights (represented by vector length in Figure 4A) was less differentiated in human observers than in the ideal observer. This could be accounted for by an underrepresentation in the model of low-level sensory noise. The more striking difference, however, lay in the effect of retinotopic location on the measured weights. On average, subjects gave significantly more weight to information in the upper half of the visual field than to information in the lower half. This unexpected result prompted us to run Experiment 2 to test whether the upper-field bias was sufficiently robust to be evidenced under different task and stimulus conditions. Because most human movements are performed in the horizontal plane, this second experiment required subjects to judge headings restricted to the horizontal midline. We also added a second stimulus condition in which the three-dimensional dots making up the stimulus were rendered on ground and ceiling planes, simulating motion through a room. 

Experiment 2: Characterizing the upper-field bias in direction of heading estimation

Method

The method for Experiment 2 was identical to that for Experiment 1 except for the following. 

Participants

Fourteen young adults (two males, 12 females; aged 18–23 years), none of which had been tested as part of Experiment 1, participated in Experiment 2. All new subjects completed all components of Experiment 2, and none were excluded from analysis. 

Task and stimuli

In order to test the robustness of the upper-field bias observed in Experiment 1, we modified our stimuli so that the simulated directions of heading were constrained to lie along the horizontal midline of the display. Subjects were asked to judge the horizontal direction of heading along this midline. As in Experiment 1, stimuli comprised dots randomly distributed in three dimensions. The display retained the 10°-radius circular occluder used in Experiment 1 and added 40° × 5° horizontal and vertical bar occluders, centered on each red gridline. These eliminated shearing effects created at the borders between the hemifield-sized perturbations, when these occurred. The participants' task was to select the direction of heading for the entire stimulus. With perturbations affecting entire hemifields, if participants were to notice shearing effects, those instructions would become ambiguous: Participants would realize that sometimes there were two foci of expansion. This was not a problem in Experiment 1, where perturbations affected a single quadrant. In Experiment 2, shearing effects were also more prominent because the perturbation border was along the same meridian as the focus of expansion (in Experiment 1 it was offset by 45°). 

Two stimulus conditions were used: (a) a random cloud of dots and (b) random dots positioned on ground and ceiling planes (Figure 5). The random cloud stimulus was equivalent to the stimuli used in Experiment 1, with the exception that the dot density was constrained to be uniform in the image plane rather than uniform in polar coordinates. The ceiling/ground stimulus simulated motion down a long and wide hallway in which the back wall was occluded by the horizontal gray occluder bar. This second stimulus condition was used here to provide a more naturalistic scenario in which flow information is available both above and below fixation. The ceiling and ground planes were simulated to be 1 m above and below the central midline, respectively, with the nearest dots 200 m away and the farthest dots 2000 m away from the viewer. For this experiment, the randomized dot placement determines a unique depth, either on the ceiling plane (above the midline) or on the ground plane (below the midline). Dot motion simulated observer motion at 4 m/s to equate average dot velocity in the image plane with those in the cloud experiments. 

Figure 5

 

Schematic representation of the display and possible anchor locations for the horizontal direction of heading task used in Experiment 2

Procedure

Because of the simpler response options in Experiment 2, we replaced the Optotrak pointer with a computer mouse for subjects to record their responses. Movement of the cursor was locked to the horizontal midline, and subjects were told that all heading directions would be on this horizontal line. Subjects were also told that if it was difficult to achieve a sense of self-motion, the heading task was equivalent to guessing the center of expansion for the dots, which would always be occluded by either the central circle or the horizontal gray line. Subjects clicked the left mouse button to record their responses. Stimulus timing, fixation control, and feedback were the same as in Experiment 1. As in Experiment 1, there were three categories of trials in each experimental block; however, the parameters were defined differently. The following describes the number of trials of each type in an experimental block. 

•  
  37 wild-card trials: Directions of heading were chosen for wild-card trials by random selection (without replacement) of one of 37 equally spaced intervals within the 40° horizontal range. For each trial, the direction of heading was randomly chosen from a uniform distribution within the interval chosen for that trial. This enforced uniform sampling of the horizontal range on wild-card trials.
•  
  60 anchor trials: Directions of heading were randomly chosen from the set [−9, −6, −3, 3, 6, 9] degrees eccentricity along the horizontal midline (Figure 5), subject to the constraint that each anchor direction of heading was tested 10 times.
•  
  80 perturbation trials: On perturbation trials, the flow in either the upper or lower hemifield simulated a direction of heading of 6° right or left of fixation, while the flow in the other hemifield simulated a direction of heading of 3° or 9° on the same side of fixation. This created eight different perturbation conditions (upper/lower hemifield, left/right of fixation, ±3° perturbation), each of which was repeated at random 10 times.

As in Experiment 1, feedback about the correct direction of heading was given only on wild-card trials. Subjects ran in two sessions of one training block and five experimental blocks each. Half of the subjects were randomly selected to run a full session of the ceiling/ground stimulus on the first day followed by the cloud stimulus session on the second day; the remaining subjects conducted the sessions in the opposite order. 

Data analysis

To compute weights for the upper and lower visual fields, we fit subjects' heading judgments on anchor and perturbation trials using the regression equation  

 

where x_u is the horizontal direction of heading specified by the flow in the upper hemifield and x_L is the horizontal direction of heading specified by the flow in the lower hemifield. We calculated weights for the leftward and rightward headings and for the two stimulus conditions separately. The weights were normalized to sum to 1, so an equal distribution of influence to the upper and lower hemispheres of visual space would result in an upper-field weight of 0.5.  

Results

As shown in Figure 6, subjects gave a higher weight (greater than 0.5) on average to the upper visual field relative to the lower visual field. To determine the possible effects of heading orientation (left, right), task (cloud, ceiling/ground), and interaction on influencing weight to the upper visual field and to simultaneously determine whether the upper-field weighting was significantly different from 0.5, we fit a sum-coded linear regression with the task and side variables dummy coded as {–1,1}:  

 

where λ_U represents the magnitude of subjects' average upper-field bias (a negative value represents an underweighting of the upper field), λ_task represents the effect of task on the upper-field bias, λ_side represents the effect of the lateral position of the DOH on the upper-field bias, and λ_interaction represents the interaction of the two.  

Figure 6

 

Upper-field weights given by subjects in Experiment 2. Individual subject values are labeled S1 through S14; the overall mean and standard deviation across subjects are provided on the far right. Data are shown separately for leftward versus rightward headings and the cloud versus ceiling/ground tasks. Note the lack of significant effects. Hemi = hemifield.

The analysis yielded no significant main effects of task or DOH side with no significant interaction but a significant value for λ_U of 0.08 ± 0.02 (Table 1). This represents an average upper/lower field weighting of 0.58 to 0.42 over all stimulus conditions. Weights to the upper visual field were significantly correlated within subjects between the two tasks (r = 0.796, p = 0.0007 two tailed; averaging the weights for each subject between left and right presentations for each task). It should be noted that in this case, since subjects' weights are close to 50%, the assumption of normally distributed data is not unreasonable. Common statistical strategies for dealing with proportions, such as arcsine transformations or using the Wilcoxon test, do not change the significance of any of the terms. 

Table 1

 

Linear regression results of terms in Equation 10. There were no significant effects of task (cloud vs. ceiling/ground), direction of heading (DOH) side (left or right hemifields), or interaction. The intercept term λ_U (Equation 10) was significantly positive, meaning that weight to the upper visual field was significantly greater than 50%.

 

Discussion

The aim of this study was to measure how normally sighted young adult observers integrate information across the visual field in determining direction of heading from optic flow. In particular, we asked whether this information is integrated efficiently across wide areas of the visual field in the near periphery. An ideal observer model was constructed to gauge the efficiency of information integration in young adults. We found that human observers seem to integrate information across large areas of the visual field and that this integration is efficient and is weighted according to the relative fidelity of available information. In doing so, we eliminated other plausible but less efficient possible strategies, such as limiting information use to stimulus regions closest to the direction of heading. The ideal observer model also successfully predicted complex patterns of spatial information used in determining horizontal and vertical components of our global motion task. However, in a departure from the model, human observers exhibited an unexpected overreliance on the upper half of the visual field. 

Some deviation from the ideal observer model is to be expected due to limitations of the model, which contained several approximations. For instance, we input only dot direction information into the model, without speed information, based on evidence that speed information contributes relatively little to the direction of heading task (Warren et al., 1991; Crowell & Banks, 1996). We also used approximations for modeling the independent sensory noise for each dot based on the results of psychophysical experiments with stimuli that were similar, but not identical, to the stimuli in the present experiment (Ball & Sekuler, 1980; Heeley & Buchanan-Smith, 1992; Gros et al., 1998; Churchland et al., 2003). Finally, we used a similar approach to approximate the increase in motion discriminability in a display of 100 dots compared with a single dot (Warren & Hannon, 1990; Eagle & Blake, 1995). Additionally, our model assumed a flat prior—that is, that any visible direction of heading was equally likely to be presented as a stimulus. There is also emerging evidence that human observers may exhibit systematic biases in direction of heading perception, but these biases are not yet fully modeled in two-dimensional space (Cuturi & MacNeilage, 2013). Finally, deviations from the model could occur because of higher level attentional, cognitive, and motor response influences. 

Despite these limitations, the ideal observer model was able to predict several patterns in the heading judgments of human observers. For the horizontal component of heading discrimination, areas on the same horizontal (left, right) side of fixation as the direction of heading were predicted by the ideal observer to be the most utilized. Similarly, for the vertical component of heading discrimination, areas with the same vertical (upper, lower) side of fixation were predicted to be the most utilized. Both these patterns were exhibited by human observers in Experiment 1. Information-theoretic models of optic flow processing have previously been successful in predicting the role of central and peripheral vision in deviation-from-target (Warren & Kurtz, 1992) and two-alternative forced-choice horizontal heading discrimination tasks (Crowell & Banks, 1996). These models predicted human observer behavior better than spatially dependent hypotheses such as the peripheral-dominance hypothesis (Brandt, Dichgans, & Koenig, 1973) and the local cue/focus of expansion hypothesis (Gibson, 1950). The functionally dependent change in information weighting found in the present experiments further supports the hypothesis that global motion processing, like other domains of sensory integration, is driven primarily by the distribution of information in the stimulus. 

It is important to note that previous studies have shown human observers to be far better at estimating direction of heading when the focus of expansion was visible (Crowell & Banks, 1996); in the present experiments, the focus of expansion was occluded. If the focus of expansion had been visible, we predict that the quadrant containing this information would dominate influence in observer weighting. However, while there are certainly instances of navigating through natural scenes where the focus of expansion is visible, there are also many instances where it might be less prominent. This may arise when visual features are not present at and near the focus of expansion. Examples include driving toward a featureless building, walking through a snow-covered landscape, averting gaze away from the direction of heading, or navigating with a visual field deficit, such as in macular degeneration or homonymous hemianopia. A more common situation in which the focus of expansion might be occluded is when an observer is judging their direction of heading in a scene containing independently moving objects, which may cross the path of observer motion. Previous experiments have shown that while human observers are not as accurate in making heading judgments in these scenarios, they are still accurate within 3° of visual angle (Warren & Saunders, 1995; Royden & Hildreth, 1996). 

All subjects responded to the three-dimensional direction of heading task in Experiment 1 by incorporating information from large areas of the visual field spanning the horizontal and vertical midlines—a strategy that was automatic and unprompted. If instead subjects had been randomly attending to one quadrant at a time, the weight to each quadrant over several trials would be equal in the horizontal and vertical components. This is because in the trials contributing to the analysis of weights, each individual quadrant was consistent with a heading direction along the y = x or y = −x line. In our experiment, we had to occlude the focus of expansion in order to measure motion integration across wide areas of visual space. However, we would predict that even in situations where the focus of expansion is visible, human observers would still utilize information according to its relative reliability. This is supported by the fact that this strategy has been observed with many different models of multicue integration, including those using other sensory domains (Ernst & Banks, 2002; Knill, Friedman, & Geisler, 2003; Alais & Burr, 2004; Gu et al., 2006; Brouwer & Knill, 2009; Issen & Knill, 2012). 

The concern remains about how well our task predicts observer behavior in natural settings. First, there has been some debate regarding the extent to which optic flow contributes to human locomotion (Rushton, Harris, Lloyd, & Wann, 1998; Warren, Kay, Zosh, Duchon, & Sahuc, 2001; Harris & Bonas, 2002), and other functions have been proposed as purposes for processing optic flow, such as directing vergence eye movements and segmenting scene-relative object motion during self-movement (Busettini, Masson, & Miles, 1997; Warren & Rushton, 2009). However, recent evidence suggests that the use of optic flow in guiding locomotion depends on the fidelity of optic flow information available (Li & Niehorster, 2014; Li et al., 2014), an interpretation that is consistent with the cue combination literature discussed previously. 

In terms of the experimental task setup, the 1-s stimulus duration used in the present experiments may seem brief. However, this is an ecologically valid time frame for human drivers, for example, because quick corrections are often needed to counteract a skid or to adjust course after avoiding an obstacle. Experimentally, drivers are found to have average fixation durations of around 500 ms (Crundall, Underwood, & Chapman, 1999). Furthermore, studies of temporal motion integration showed that direction discrimination of global coherent motion asymptotes between 450 and 600 ms (Watamaniuk, Sekuler, & Williams, 1989; Watamaniuk & Sekuler, 1992), which indicates that our 1-s stimulus paradigm was sufficient for motion integration. 

Our experimental design using perturbation conditions revealed a spatial bias that has not been previously reported, namely that human observers appear to rely more heavily on the upper than the lower half of the visual field in determining direction of heading. This pattern of behavior was persistent across three different tasks, including a ceiling/ground stimulus condition, providing a counterpoint to previous theories that the primate visual system is specifically adapted to process ground plane information (Gibson, 1950; Previc, 1990). The cause of the upper-hemifield bias is unclear. There is no clear evidence suggesting that it could be due to an overrepresentation of the upper visual field, in terms of either cortical magnification in early visual areas or uneven connections with higher visual areas. In fact, the lower visual field is commonly reported as being slightly overrepresented in the primate dorsal lateral geniculate nucleus (Connolly & Van Essen, 1984), in the primary visual cortex (Van Essen, Newsome, & Maunsell, 1984; Tootell, Switkes, Silverman, & Hamilton, 1988), and in the middle temporal visual area (Maunsell & Van Essen, 1987). To rule out purely visual, nonattentional mechanisms, one would need to measure performance thresholds for determining direction of heading from the upper or lower visual field in isolation in order to identify whether performance fields in this task are largely constant between subjects. A study by Afraz, Pashkam, and Cavanagh (2010) found asymmetries in isoeccentric performance, which they called performance fields, for several low-level and high-level tasks. These performance fields were stable within each subject and task but more irregular between subjects. In our experiments, upper-field weights for the cloud and ceiling/ground tasks were highly correlated within subjects. This would suggest that if performance fields exist for determining motion direction, they would remain stable within subjects. If so, an individual subject's higher reliance on information in the upper visual field could in fact be an ideal strategy, and the pattern across subjects may be due to a large proportion of subjects having high-performance fields for motion direction in the upper visual field. 

The upper-field bias elicited here may also be due to a bias in covert attention. A study of speed–accuracy tradeoffs by Carrasco et al. (2001) in a feature (orientation) search task revealed temporal performance fields, or areas of slower information processing, when the target appeared along the vertical midline rather than other isoeccentric locations, with the slowest processing for target locations at the north position. Manipulations of covert attention revealed that attention had the largest effect for boosting the processing time for the region directly above fixation. It is less clear what would happen in a heading task, where subjects are distributing their attention over a wide area of the visual field in order to extract information from a brief presentation. However, the knowledge that covert attention can have asymmetric effects for isoeccentric locations in the visual field makes this a plausible explanation that could be tested in future studies. 

Conclusions

Young adult human observers appear to be very sensitive to the specific patterns of information content distributed across visual space when making direction of heading judgments. They differentially weight relevant regions of the visual field in determining the horizontal and vertical components of direction of heading. Overall, in most (but not all) respects, humans behave qualitatively like ideal observers when it comes to global processing of flow information, integrating information across the vertical and horizontal hemifields. Different regions of the visual field influence the horizontal and vertical components of human estimates differentially according to the fidelity of visual motion information they contain. However, subjects overweighted information in the upper visual field. Both the mechanisms and functional implications of this surprising result should be elucidated in future experiments. 

Acknowledgments

We thank Keith Parkins for excellent programming support and for assisting with the data collection. Our deep thanks also to Robbie Jacobs for his constructive suggestions during the revision of this article. This research was supported by an NEI Center Core grant to the Center for Visual Science (P30 EY001319), an unrestricted grant from the Research to Prevent Blindness Foundation to the Flaum Eye Institute, R01 grants from the NEI to D. K. (EY017939) and K. R. H. (EY021209), and a Collaborative Grant from the Schmitt Program on Integrative Brain Research (to K. R. H.). L. I. was partially supported by NEI training grant T32 EY007125 to the Center for Visual Science. K. R. H. is a Research to Prevent Blindness Foundation Lew R. Wasserman Merit Award recipient. 

Commercial relationships: none. 

Corresponding author: Krystel R. Huxlin. 

Email: [e-mail]. 

Address: Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, USA.