A relative frame of reference underlies reversed depth perception in anticorrelated random-dot stereograms

Abstract

Binocular disparity is represented by interocular cross-correlation of visual images in the striate and some extrastriate cortices. This correlation-based representation produces reversed depth perception in a binocularly anticorrelated random-dot stereogram (aRDS) when it is accompanied by an adjacent correlated RDS (cRDS). Removal of the cRDS or spatial separation between the aRDS and cRDS abolishes reversed depth perception. However, how an immediate plane supports reversed depth perception is unclear. One possible explanation is that the correlation-based representation generates reversed depth based on the relative disparity between the aRDS and cRDS rather than the absolute disparity of the aRDS. Here, we psychophysically tested this hypothesis. We found that participants perceived reversed depth in an aRDS with zero absolute disparity when it was surrounded by a cRDS with nonzero absolute disparity (i.e., nonzero relative disparity), suggesting a role of relative disparity on the depth reversal. In addition, manipulation of the absolute disparities of the central aRDS and surrounding cRDS caused depth perception to reverse with respect to the depth of the surround. Further, depth reversal persisted after swapping the locations of the two RDSs. A model of relative-disparity encoding explains all these results. We conclude that reversed depth perception in aRDSs occurs in a relative frame of reference and suggest that the visual system contains correlation-based representation that encodes relative disparity.

Introduction

Objects at depths in front of and beyond the fixation distance project their images onto relatively different locations of the left and right retinae, producing binocular disparity and allowing binocular depth perception. To achieve correct depth perception, the visual system needs to match the images of features from one eye to the corresponding images from the other eye (the stereo correspondence problem; Julesz, 1960; Marr & Poggio, 1979). Neuronal representation of binocular disparity based on a solution of the stereo correspondence problem (match-based representation) has been probed using binocularly anticorrelated random-dot stereograms (aRDSs), in which the corresponding dots in left-eye and right-eye images have opposite luminance contrasts (Figure 1A; Cumming & Parker, 1997; Janssen, Vogels, Liu, & Orban, 2003; Krug, Cumming, & Parker, 2004; Kumano, Tanabe, & Fujita, 2008; Takemura, Inoue, Kawano, Quaia, & Miles, 2001; Tanabe, Umeda, & Fujita, 2004; Theys, Srivastava, van Loon, Goffin, & Janssen, 2012). Because aRDSs lack a globally consistent binocular match, the correspondence problem cannot be resolved for aRDS (Julesz, 1960). Therefore, neurons that normally represent the solution should be insensitive to binocular disparity in aRDSs. Neurons in the primary visual cortex and mid-level stages of the dorsal visual pathway (middle temporal area [MT] and medial superior temporal area) of the monkey are sensitive to disparity in aRDSs and have tuning curves that are inversions of those for binocularly correlated RDSs (cRDSs; Cumming & Parker, 1997; Krug et al., 2004; Takemura et al., 2001). This inverted profile of disparity tuning suggests that neuronal responses in these areas reflect the cross-correlation between left-eye and right-eye images (Fleet, Wagner, & Heeger, 1996; Ohzawa, DeAngelis, & Freeman, 1990; Qian & Zhu, 1997). However, neurons in these areas do not rely solely on cross-correlation; the tuning amplitude is smaller for aRDSs than for cRDSs, which belies the equal amplitude predicted by pure cross-correlation computation (Haefner & Cumming, 2008; Read, Parker, & Cumming, 2002). The disparity selectivity for aRDSs is attenuated or abolished in mid-level and higher cortical areas within the dorsal and ventral visual pathways (area V4, inferior temporal area [IT], and anterior intraparietal area [AIP]), suggesting that the correspondence problem is progressively solved in these areas (Abdolrahmani, Doi, Shiozaki, & Fujita, 2016; Janssen et al., 2003; Kumano et al., 2008; Tanabe et al., 2004; Theys et al., 2012). The results obtained from functional magnetic resonance imaging studies in humans are consistent with those from the single-neuron studies in monkeys in that disparities in aRDSs modulate responses in V1 but not those in higher cortical areas within the ventral or dorsal pathways (Bridge & Parker, 2007; Preston, Li, Kourtzi, & Welchman, 2008; see Fujita & Doi, 2016, for detailed discussion). Thus, neuronal representations of disparity transition from correlation-based to match-based along the cortical hierarchy. 

Figure 1

 

(A) Correlated and anticorrelated RDSs. The central regions (inner white circles) of the two random-dot patterns in the top row constitute a correlated RDS (cRDS), whereas those in the bottom row constitute an anticorrelated RDS (aRDS). The central regions have crossed disparity when an observer fixates the white cross. Left and right columns contain stimuli for left and right eyes, respectively. In both RDSs, the surrounding annuli (outside of white circles) are cRDSs with zero disparity. (B) Schematic illustration of depth reversal based on absolute and relative disparities. Here, both the center and the surround have crossed disparities (i.e., closer than the fixation plane), but the center is located farther than the surround (i.e., the center has a smaller crossed disparity than that of the surround). If depth reversal occurs based on absolute disparity, the center will be perceived as farther away than the surround and the fixation plane (black arrow). If depth reversal occurs based on relative disparity, the center will be perceived as closer than the surround (white arrow).

Recent psychophysical studies provide evidence for a direct contribution of correlation-based representation to depth perception without a full transformation to match-based representation; under certain conditions, a patch of aRDS produces depth perception opposite to the direction of the disparity-defined depth (i.e., crossed disparity evokes “far” perception and uncrossed disparity evokes “near” perception; Doi, Takano, & Fujita, 2013; Doi, Tanabe, & Fujita, 2011; Tanabe, Yasuoka, & Fujita, 2008). Reversed depth perception requires a reference cRDS placed immediately adjacent to a patch of aRDS. The reversed depth perception is abolished when the adjacent cRDS is replaced with an RDS that does not evoke the perception of a surface in depth (an aRDS or a binocularly uncorrelated RDS, uRDS; Doi et al., 2011; Tanabe et al., 2008; see also Cumming, Shapiro, & Parker, 1998). A small gap (0.35°) between the aRDS and the cRDS also eliminates reversed depth perception (Kamihirata, Oga, Aoki, & Fujita, 2015). Correlation-based representation thus mediates depth perception only when an immediate reference plane is available. However, how reference planes contribute to reversed depth perception is unclear. 

A previous study proposed a gating mechanism by which the correlated reference plane boosts the disparity signal for aRDSs so that it can contribute to depth perception (Doi et al., 2011). In this account, depth perception is determined by neurons that encode the local absolute disparity (binocular disparity relative to the fixation point) of aRDSs, and reversed depth perception reflects the depth reversal based on absolute disparity (an absolute frame of reference). This absolute disparity model has successfully explained several characteristics of reversed depth perception such as effects of interocular delay, disparity magnitude, and temporal frequency (Doi & Fujita, 2014; Doi et al., 2011, 2013; Tanabe et al., 2008). This model builds on the neurophysiological findings in monkeys; neurons carrying binocular correlation-based signals such as those in V1 and MT encode local absolute disparity for RDSs that have a spatial configuration similar to those used in the psychophysical experiments (Cumming & Parker, 1997, 1999; Krug et al., 2004; Uka & DeAngelis, 2006). 

Another possible mechanism is that the reference plane provides a relative (object-centered) reference frame in which correlation-based representation generates depth perception. Consistent with this explanation, reversed depth perception in aRDSs shares characteristics with stereoacuity in cRDSs, which relies on the relative disparity between two objects. First, a binocularly uncorrelated reference stimulus increases discrimination thresholds for depth by more than an order of magnitude (Cottereau, McKee, Ales, & Norcia, 2012; Prince, Pointon, Cumming, & Parker, 2000), just as it abolishes reversed depth perception (Tanabe et al., 2008). Second, increasing the gap between a discrimination target and a reference stimulus degrades stereoacuity (Cottereau, McKee, & Norcia, 2012; Read, Phillipson, Serrano-Pedraza, Milner, & Parker, 2010), just as it gradually deteriorates and finally abolishes reversed depth perception (Kamihirata et al., 2015). Therefore, correlation-based representation of disparity may generate a reversed-depth percept in a spatial frame relative to a reference stimulus (relative frame of reference) but not relative to the fixation plane (absolute frame of reference). 

Here, we distinguished reversal of depth based on relative disparity from that based on absolute disparity. Previous studies on reversed depth perception exclusively used the reference cRDS of zero absolute disparity (Doi et al., 2011, 2013; Tanabe et al., 2008). As a result, the relative disparity between the target aRDSs and the reference cRDSs was always identical to the absolute disparity of the aRDSs. To resolve this issue, we independently manipulated the absolute disparities of the target aRDS and the reference cRDS. Consider an RDS comprising a central aRDS disk with crossed disparity and a surrounding ring of cRDS with an even larger crossed disparity (Figure 1B). When an individual judges whether the center is closer or farther than the surround, depth reversal in an absolute frame of reference leads to a depth judgment of “farther” (Figure 1B, black arrow). In contrast, depth reversal in a relative frame of reference results in a depth judgment of “closer” (Figure 1B, white arrow). We demonstrate that depth perception for aRDSs follows the latter pattern, reversing with respect to the reference stimulus, indicating that correlation-based representation generates depth perception in a relative frame of reference. 

Methods

Participants

Fourteen people participated in the study. Participants completed a screening test for stereo vision before beginning the experiments. In this test, they reported the relative depth between the central cRDS disk and surrounding cRDS ring (disk diameter, 4.8°; ring width, 1.6°). The disparity of the center (center disparity) was either −0.16° or 0.16° (negative and positive values indicate crossed and uncrossed disparities, respectively). The disparity of the surround (surround disparity) was fixed at zero. Eleven participants discriminated the depth correctly in more than 90% of the trials and proceeded to the experiments. The remaining three participants performed poorly on the screening test (<90% accuracy) and were therefore excluded from the experiments. 

The same six individuals, including an author (S. C. A.), participated in Experiments 1, 2, 3, and 4. Two additional participants were included in Experiment 1. Three other participants and an author (S. C. A.) were included in Experiment 5 (Experiment 1, n = 8; Experiments 2–4, n = 6; Experiment 5, n = 4). All but one (S. C. A.) participant were naïve to the purpose of the experiments. All participants had normal or corrected-to-normal vision. We obtained written informed consent from all participants and performed all experiments in accordance with the Declaration of Helsinki. 

Apparatus

Participants viewed visual stimuli on a full-flat cathode-ray tube monitor (Multiscan G520, Sony, Tokyo) placed 57 cm away from the eyes. The head was stabilized on a chin rest during experiments. The monitor had a spatial resolution of 1,152 × 864 pixels and subtended 38.8° × 29.5° of the visual field. The monitor refresh rate was 85 Hz. Visual stimuli were generated using OpenGL and the OpenGL Utility Toolkit (GLUT) and were presented with a graphics board (NVIDIA Quadro FX3700, Elsa Japan, Tokyo). We applied antialiasing to present visual stimuli at subpixel resolution and presented stimuli dichoptically using liquid-crystal active shutter glasses (RE7-CANE, Elsa, Aachen). We minimized interocular cross-talk by using only the red phosphors, which had the shortest decay time among the three types of phosphors used in the monitor. The amount of cross-talk, measured as the luminance of the ghost image (Tanabe et al., 2004), was less than 2%. 

Visual stimuli

Visual stimuli were dynamic RDSs composed of a central disk (diameter: 4.8°) and a surrounding ring (width: 1.6°; Figure 1A). The RDSs comprised an equal number of bright and dark square dots (bright: 2.5 cd/m²; dark: 0.0 cd/m²; measured through a shutter glass). The dots (size: 0.07° × 0.07°) were presented on a mid-luminance background (1.3 cd/m²; measured through a shutter glass). In each frame, the dots occupied 25% of the RDS area when the dots did not overlap each other. Dot position was randomized every two frames, resulting in a dot-pattern refresh rate of 42.5 Hz. The center of the RDS was 4.8° below the fixation target (nonius lines; see below) that was presented at the center of the screen. There was a gap of 0.8° between the fixation target and the edge of the RDS patch. 

Tasks

Participants were asked to determine whether the central disk was in front of or behind the surrounding ring (termed “near” and “far”, respectively). Each experiment consisted of three or four blocks of trials. To keep participant vergence angles constant, we used nonius lines as the fixation target. Dichoptically separate vertical lines (length: 0.4°) were presented above (right eye) and below (left eye) the center of a horizontal line that was binocularly presented (length: 0.8°). Thus, the nonius lines formed a cross when correctly fused. The lines were presented throughout the block, and participants were instructed to continuously fuse the left and right lines and maintain their fixation so that the nonius lines appeared to form a cross. 

After the RDS was presented for a fixed duration (varied by experiment, see below), it disappeared and participants reported their choice (“near” vs. “far”) by pressing designated keys within a 1-s (Experiments 1 and 2) or 500-ms (Experiments 3, 4, and 5) choice period. Upon pressing a key, the word near or far was displayed on the screen until the end of the choice period so that participants could verify their responses. During the choice period, they were allowed to change their choice by pressing the other key. We did not provide any feedback regarding the correctness of the choice. When a participant did not press a key during the choice period, a trial with the same stimulus condition was randomly inserted into the sequence of remaining trials. The next trial started 1 s after the end of the choice period of the preceding trial. 

In Experiment 1, we examined the depth perception for aRDSs that would evoke reversed depth perception only if the reversal occurs in a relative but not in an absolute frame of reference. The center was either a cRDS or an aRDS on a given trial, and the surround was always a cRDS. The center disparity was fixed at zero, and the surround disparity was varied across trials (−0.32°, −0.16°, −0.08°, −0.04°, 0.04°, 0.08°, 0.16°, or 0.32°). The stimulus was presented for 1 s in each trial. In each block, each of the eight surround disparities was combined with the cRDS center in five trials and with the aRDS center in 10 trials (eight surrounds × 15 = 120 trials). In addition to these trials, we included control trials to verify that the participants were performing the task using visual information from both the center and the surround. In these trials, the center was a cRDS of nonzero disparity (−0.32°, −0.08°, 0.08°, or 0.32°) and the surround was a cRDS of zero disparity. The participants would not be able to make a correct judgment in the control trials if they ignored the center disparity and used only the surround disparity to perform the task. Each of the four control conditions was repeated twice in each block (eight trials). Thus, a single block consisted of 128 randomly interleaved trials. Each participant performed three blocks. 

The control trials in Experiment 1 verified that participants made choices using both the center and surround disparities when the center was a cRDS (see the Results section). However, because aRDSs lack binocular correspondence, they might have relied solely on the surround disparity when the center was an aRDS. In Experiment 2, we replaced the aRDS with a uRDS, which has no disparity information, and examined whether participants exclusively used the surround disparity when the center was not binocularly correlated. The visual stimuli were the same as in Experiment 1, except that the center was either a cRDS or a uRDS in which dot patterns were independently generated for left-eye and right-eye images. The surround was always a cRDS. The surround of the cRDS-center stimuli had one of eight disparities (−0.32°, −0.16°, −0.08°, −0.04°, 0.04°, 0.08°, 0.16°, or 0.32°). The surround of the uRDS-center stimuli was either one of the same eight disparities or zero. Stimuli were presented for 1 s. Each surround disparity was tested five times for the cRDS center (eight surrounds × five = 40 trials) and 10 times for the uRDS center (nine surrounds × 10 = 90 trials). Control trials were the same as in Experiment 1 (eight trials). Thus, each block consisted of 138 randomly interleaved trials. Each participant performed three blocks. 

In Experiment 3, we addressed the possibility that the results in Experiment 1 were compromised by vergence eye movements during stimulus presentation. The center disparity deviates from its intended value if the vergence angle moves away from the fixation plane in response to surround disparity. If this was happening, it would complicate the interpretation of the results for Experiment 1 because it relies on the assumption that the center disparity was fixed at zero. In Experiment 3, we repeated the same experiment as Experiment 1 but with a stimulus duration of 94 ms. This duration is shorter than the latency of human vergence eye movements, thus eliminating possible changes in vergence angle during stimulus presentation (Masson, Busettini, & Miles, 1997; Yang, Bucci, & Kapoula, 2002). Each participant performed four blocks. 

In Experiment 4, we tested whether the depth reversal in aRDSs occurs relative to the surround by manipulating both center and surround disparities. The center was either a cRDS or an aRDS, and the surround was always a cRDS with one of three absolute disparities (−0.16°, 0.00°, or 0.16°). We used four center disparities to create relative disparities of −0.16°, −0.04°, 0.04°, or 0.16° between the center and the surround. We presented the stimulus for 94 ms in each trial. Each combination of center and surround disparities was repeated five times for the cRDS center (five × three centers × four surrounds = 60 trials) and 10 times for the aRDS center (120 trials). Thus, each block consisted of 180 trials arranged in a random order. Each participant performed three blocks. 

In Experiment 5, we examined whether the specific stimulus configuration used in Experiments 1, 3, and 4 (i.e., an aRDS surrounded by a cRDS) is necessary for reversed depth perception by repeating Experiment 4 with swapped centers and surrounds; the center was a cRDS and the surround was either a cRDS or an aRDS. Each participant performed three blocks. 

Data analysis

To quantify how much depth perception depended on a relative frame of reference in Experiment 4 and 5, we calculated the proportion of far choices for each stimulus condition in each subject and averaged these proportions across subjects. For each combination of surround disparity and the correlation of the center (Experiment 4) or surround (Experiment 5), we plotted the proportion of far choices as a function of center disparity and fitted a cumulative Gaussian function. The fitted functions in each correlation condition shared the standard deviation (i.e., the slope of the function) across surround disparities. Therefore, there were four free parameters (three means for three surround disparities and one standard deviation) in each correlation condition. Model parameters were determined by maximum likelihood estimation. All analyses were done with MATLAB (MathWorks, Natick, MA). 

Results

Experiment 1: Depth perception in aRDSs with zero absolute disparity and nonzero relative disparity

As a first step toward examining whether absolute or relative disparity underlies reversed depth, we presented an aRDS disk of zero absolute disparity surrounded by a cRDS ring of nonzero absolute disparity. We asked participants to report whether the stimulus center (an aRDS or a cRDS) was closer to (“near”) or farther from (“far”) the surrounding cRDS (Figure 2A). If reversed depth perception for aRDSs occurs based on relative disparity, participants would report a depth opposite the relative disparity between the center and surround (“near” and “far” choices for crossed and uncrossed surround disparities, respectively; Figure 1B). This reasoning predicts an inversion of psychometric functions (plots of the proportion of “far” choices against the disparity of the center relative to the surround) between cRDS and aRDS centers (Figure 2B, left). In contrast, if depth reversal occurs based on absolute disparity, the depth of the center would not be reversed because the center disparity is zero. Therefore, relative depth judgments would be insensitive to whether the center is a cRDS or an aRDS, leading to consistent psychometric functions for the two cases (Figure 2B, right). 

Figure 2

 

(A) Schematic of visual stimuli in Experiment 1. Participants discriminated depth in a central cRDS or aRDS, both of which were surrounded by a cRDS. The absolute disparity of both types of center was fixed at zero, whereas that of the surround varied across trials. (B) Predicted psychometric functions for depth reversal based on relative (left panel) and absolute (right panel) disparities. Psychometric functions are represented as the proportions of “far” choices plotted as a function of relative disparity between the center and surround. The color of the psychometric functions represents the binocular correlation of the center (blue: cRDS, red: aRDS). (C) Psychometric functions for each participant in Experiment 1. Line and marker color represents the binocular correlation of the center (blue: cRDS, red: aRDS).

When the center was a cRDS, the proportion of “far” choices increased from 0 to 1 at the transition from negative to positive relative disparity (i.e., from uncrossed to crossed surround disparity; Figure 2C, blue lines). Thus, for central cRDSs, participants perceived depth consistent with geometrically defined depth. In stark contrast, the proportion of “far” choices for aRDS centers decreased at the transition in most participants (Figure 2C, red lines). In all participants, the proportion of “far” choices decreased as relative disparity shifted from negative to positive over a fine range of binocular disparity (from −0.08° to 0.08°). However, details of the psychometric functions for aRDSs varied across participants. Psychometric functions of two participants (S. C. A., M. M.) agreed well with the prediction derived from depth reversal based on relative disparity (compare Figure 2B with Figure 2C). Choices of three participants were generally biased toward “near” or “far” when judging aRDS centers (near: F. T. and S. I. D.; far: S. A. H.). For some participants (Y. K., S. A. H., M. A. M., and Y. S. T.), the proportion of “far” choices deviated from the reversal of relative depth prediction when the amplitude of the relative disparity (i.e., surround disparity) was large (±0.32°). In control trials in which center cRDSs had either crossed or uncrossed disparity and surround disparity was fixed at zero, all participants correctly reported geometrically defined depth (mean proportion of correct choices or accuracy, 0.99 ± 0.01 SEM), confirming that participants performed the task by integrating both center and surround disparities. 

To quantify the magnitude of depth reversal induced by anticorrelation, we combined the data for all surround disparities and calculated accuracy separately for cRDSs and aRDSs (Figure 3). Here, correct choices were defined according to the sign of the relative disparity between the center and the surround (i.e., choices of “far” and “near” were correct for crossed and uncrossed surround disparities, respectively). For cRDSs, the proportion of correct choices was 1 or close to 1 in all participants (Figure 3, blue bars). For aRDSs, the proportion of correct choices was lower than chance in all but one participant (Y. S. T.; Figure 3, red bars; binominal test, p < 0.05). One participant (Y. S. T.) performed at chance level. Thus, as hypothesized, most participants perceived reversed depth for aRDSs with zero absolute disparity, suggesting that depth reversal takes place in a relative frame of reference. 

Figure 3

 

Each participant's accuracy in Experiment 1 for cRDSs (blue bars) and aRDSs (red bars). The gray area represents the range for chance-level performance calculated from 240 aRDS trials (binominal test, p < 0.05).

Experiment 2: Depth perception in RDSs without binocular correspondence

One may argue that depth judgments relying solely on surround disparity could lead to results similar to those obtained in Experiment 1. Suppose that participants ignored center disparity and simply reported the depth indicated by surround disparity (i.e., “near” for crossed surround disparity and “far” for uncrossed surround disparity). In such cases, the resulting proportion of correct choices would be lower than chance (i.e., below 0.5). Based on the results of Experiment 1, this explanation is unlikely because accuracy in the control trials, which required comparison between the cRDS centers and the cRDS surrounds, was nearly perfect. However, participants might have ignored center disparity when the center was an aRDS because aRDSs do not evoke perception of crisp surfaces (Tanabe et al., 2008). To test this possibility, we used uRDSs, which also do not produce crisp surface perception, and examined whether a uRDS surrounded by a cRDS of varying absolute disparity generates reversed depth perception. If participants simply report the depth of the surround and ignore the center disk for stimuli without crisp surfaces, accuracy would be similar between Experiments 1 and 2. Alternatively, if depth discrimination depends on relative disparity between the center and the surround, participants should not report reversed depth judgment for uRDSs because they lack disparity information and thus the relative disparity cannot be calculated. 

When the center was a cRDS, participants almost perfectly reported geometrically correct depth (Figure 4, blue lines; Figure 5, blue bars). They performed perfectly in the control trials, in which the cRDS center had either crossed or uncrossed disparity and the cRDS surround had zero disparity (mean accuracy, 1 ± 0.00 SEM), confirming that they used both center and surround disparities for cRDSs. When the center was a uRDS, the proportion of “far” choices made by four of the six participants (S. C. A., M. M., Y. K., and S. I. D.) did not appear to depend on the sign of the surround disparity; they tended to choose “near” regardless of the sign of the relative disparity (uRDSs were regarded as zero disparity in the calculation of relative disparity; Figure 4, black lines). This disparity-independent choice bias is reminiscent of “rivaldepth,” depth perception for stimuli without binocular correlation (O'Shea & Blake, 1987). The proportion of “far” choices in the remaining participants (M. A. M. and Y. S. T.) increased as the relative disparity transitioned from negative to positive (i.e., the surround disparity transitioned from uncrossed to crossed; Figure 4, black lines). We calculated accuracy for uRDSs by regarding their disparity as zero and excluding trials in which surround disparity was zero. We found that accuracy was either at the chance level (S. C. A., M. M., Y. K., and S. I. D.) or higher than chance (M. A. M. and Y. S. T.; binominal test, p < 0.001; Figure 5, black bars). No participant showed lower-than-chance accuracy, indicating that uRDSs surrounded by cRDSs with nonzero disparity did not produce reversed depth judgement. The results thus support the conclusion of Experiment 1 that reversed depth perception was generated based on the relative disparity between the aRDS center and the cRDS surround. 

Figure 4

 

Psychometric functions for each participant in Experiment 2. Participants discriminated the depth of a cRDS with zero disparity or a uRDS, both of which were surrounded by a cRDS with nonzero disparity. Psychometric functions are represented as the proportions of “far” choices plotted as a function of relative disparity (the absolute disparity of uRDSs was regarded as zero). Line color represents the binocular correlation of the center (blue: cRDS, black: uRDS).

Figure 5

 

Each participant's accuracy for cRDSs (blue bars) and uRDSs (black bars) in Experiment 2. The gray area represents the range for chance-level performance calculated from 240 uRDS trials (binominal test, p < 0.05).

Experiment 3: Depth perception in aRDSs with a short presentation period

The absolute disparity of visual stimuli changes when participants make vergence eye movements, which could potentially complicate the interpretation of the results in Experiment 1. For example, the aRDS center will have an uncrossed disparity rather than the desired zero disparity if participants converge their eyes in response to the surround with crossed disparity. In such cases, depth reversal based on both absolute and relative disparities leads to a choice of “near,” and we would not be able to determine which type of disparity underlies the reversed depth perception. Although we required participants to maintain fixation at the nonius lines to minimize vergence eye movements (see the Methods section), we did not record their eye movements. Therefore, uncontrolled vergence movements might have affected the results. In Experiment 3, we addressed this issue by using very short stimulus duration (94 ms), which is shorter than the latency of reflexive vergence eye movements (Masson et al., 1997; Yang et al., 2002). With this manipulation, we examined our interpretation of the results for Experiment 1 while avoiding any confounding effects of eye movement. 

Participants reported geometrically correct depth for cRDSs (Figure 6, blue lines; Figure 7, blue bars). In the control trials, all participants correctly reported depth of the cRDS centers with crossed or uncrossed disparity relative to the cRDS surround of zero disparity (mean accuracy, 0.98 ± 0.03 SEM). For the aRDSs, the proportion of “far” choices decreased as the relative disparity shifted from negative to positive (i.e., the surround disparity shifted from uncrossed to crossed) as in Experiment 1, and accuracy was less than chance in all participants (Figure 7, red bars; binominal test, p < 0.05). Thus, they perceived reversed depth in aRDSs with zero absolute disparity and nonzero relative disparity even when the stimulus duration was too short to elicit vergence eye movements. This indicates that reversed depth judgment for aRDSs of zero absolute disparity was unlikely to result from changes in absolute disparity induced by vergence eye movements. 

Figure 6

 

Psychometric functions for each participant in Experiment 3. Participants discriminated depth of a cRDS or an aRDS, both of which had zero disparity and were surrounded by a cRDS with nonzero disparity. The stimulus duration was shorter than the latency of vergence eye movements. Conventions are the same as in Figure 2.

Figure 7

 

Each participant's accuracy in Experiment 3. Conventions are the same as in Figure 3. The number of trials used to calculate the chance-level range was 320.

Experiment 4: Depth perception in aRDSs with varying combinations of center and surround disparities

If reversed depth perception is based on relative disparity, it should occur for varying pedestal disparities (absolute disparities of the entire stimulus). To test this prediction, we varied pedestal disparities of the stimuli comprising either a cRDS disk or an aRDS disk surrounded by a cRDS ring (Figure 8A). We predicted that changes in the surround disparity would horizontally shift psychometric functions plotted against the center disparity without changing their shape; when the surround has crossed or uncrossed disparity, the psychometric function would shift leftward or rightward, respectively (left column in Figure 8B). In contrast, if reversed depth perception is based on absolute disparity, the psychometric function would reverse at the center disparity of zero (right column in Figure 8B). 

Figure 8

 

(A) Schematic of visual stimuli in Experiment 4. Surround disparity was either uncrossed (top), zero (middle), or crossed (bottom). In each condition, the relative disparity between the center and the surround was −0.16°, −0.04°, 0.04°, or 0.16°. (B) Predicted psychometric functions based on depth reversal in a relative frame of reference (left column) and based on depth reversal in an absolute frame of reference (right column). The psychometric functions are plotted separately for each surround disparity (top: uncrossed surround disparity, middle: zero surround disparity, bottom: crossed surround disparity). Line color represents the binocular correlation of the center (blue: cRDS, red: aRDS).

When the center was a cRDS, participants correctly reported depth according to relative disparity; changes in surround disparity resulted in horizontal shifts of the psychometric functions plotted as a function of center disparity (Figure 9A and C, blue lines; 9A for data of individuals, 9C for the average across the six participants). As a result, the psychometric functions obtained with different surround disparities closely overlapped when plotted as a function of relative disparity (Figure 9B, blue lines), indicating that the amount of shift matched changes in surround disparity and that the sensitivity to relative disparity was invariant in this range of surround disparities (Figure 9D, blue line). When the center was an aRDS, changes in surround disparity also horizontally shifted the psychometric functions plotted against the center disparity (Figure 9A and C, red lines). As in cRDSs, the psychometric functions for aRDSs substantially overlapped with each other when plotted as a function of relative disparity (Figure 9B, red lines), indicating that the amount of shifts was consistent with the changes in surround disparity (Figure 9D, red line). However, in contrast to the psychometric functions for cRDSs, the proportion of far choices decreased as relative disparity moved from negative to positive. We quantified reversal of perceived depth for each surround disparity by calculating accuracy. For cRDSs, accuracy was close to 1 for all surround disparities (Figure 10, blue lines), whereas for aRDSs, all participants showed lower-than-chance accuracy for at least one surround disparity (Figure 10, red lines; binominal test, p < 0.05, Bonferroni corrected). Accuracy was lower than chance for all surround disparities in three participants (S. C. A., Y. K., Y. S. T.), for two surround disparities in two participants (M. A. M., M. M.), and for one surround disparity in the remaining participant (S. I. D.; Figure 10; binominal test, p < 0.05, Bonferroni corrected). Mean accuracy for aRDSs did not depend on surround disparity (Friedman test, p = 0.607). Thus, anticorrelation reverses depth perception according to relative disparity irrespective of pedestal disparity. 

Figure 9

 

Psychometric functions for each participant in Experiment 4. (A) Proportion of far choices plotted against center disparity. A psychometric function was generated separately for each surround disparity. Line color represents the binocular correlation of the center (blue: cRDS, red: aRDS). (B) The same data as in (A) are plotted against the relative disparity between the center and the surround. Conventions are the same as in (A). (C) Proportion of far choices averaged across subjects (M ± SEM) plotted against center disparity. (D) The mean (μ) of fitted cumulative Gaussian functions plotted against surround disparity.

Figure 10

 

Accuracy for each surround disparity in Experiment 4. Line color represents the binocular correlation of the center (blue: cRDS, red: aRDS). The gray area represents the range for chance-level performance calculated from 120 aRDS trials (binominal test, p < 0.05, Bonferroni corrected).

Experiment 5: Depth perception in cRDSs surrounded by aRDSs

Experiments 1, 3, and 4 tested participants with aRDSs surrounded by cRDSs. Whether this spatial configuration is essential for producing reversed depth perception is unclear. Therefore, we next examined whether anticorrelation of the surround, instead of the center, produces similar reversals of perceived depth. Specifically, we repeated Experiment 4 but swapped the binocular correlation of the center and the surround; the surround was either a cRDS or an aRDS, and the center was always a cRDS (Figure 11A). 

Figure 11

 

(A) Schematic of visual stimuli in Experiment 5. Surround disparity was either uncrossed (left), zero (middle), or crossed (right). (B) Psychometric functions for each participant in Experiment 5. Participants discriminated the depth of cRDSs that were surrounded by either cRDSs or aRDSs. The same disparities were assigned at the center and the surround as in Experiment 4. Conventions are the same as in Figure 9A except that the color represents binocular correlation of the surround (blue: cRDS, red: aRDS). (C) Proportion of far choices plotted against relative disparity. Conventions are the same as in (B). (D) Proportion of far choices averaged across subjects (M ± SEM) plotted against center disparity. The curves represent fitted cumulative Gaussian functions. (E) The mean (μ) of fitted cumulative Gaussian functions against surround disparity.

As in Experiment 4, when the surrounding ring was a cRDS, judgments of depth direction agreed with the sign of the relative disparity in all four participants (Figure 11B and D, blue lines; 11B for individuals, 11D for the average across the four subjects). The psychometric functions for the aRDS surround, plotted as a function of the center disparity, also shifted horizontally for different surround disparities but had slopes opposite to those for cRDS surround (Figure 11B and D, red lines). The amount of shifts for aRDSs was largely consistent with the changes in surround disparity (Figure 11B and E). When plotted as a function of relative disparity (Figure 11C, red lines) the psychometric functions for aRDSs either overlapped (S. C. A.) or horizontally shifted to a smaller extent (M. A. M., M. F., H. O. I.). We quantified reversal of perceived depth for the three surround disparities by calculating accuracy. For cRDSs, accuracy was well above chance for all three surround disparities (Figure 12, blue lines). For aRDSs, it was lower than chance for all surround disparities in three participants (S. C. A., M. F., M. A. M.; Figure 12, red lines; binominal test, p < 0.05, Bonferroni corrected) and was lower than chance for one surround disparity (0.16°) and close to the limit of chance level for the other surround disparities (−0.16° and 0.00°) in the remaining participant (H. O. I.; Figure 12, red line with open square data points; binominal test, p < 0.05, Bonferroni corrected). Thus, anticorrelation of the surrounding ring caused reversed depth perception according to relative disparity, just as it did for anticorrelation of the center examined in Experiment 4. Reversed depth perception is therefore unlikely to depend on the spatial arrangement of correlated and anticorrelated RDSs. 

Figure 12

 

Accuracy for each surround disparity in Experiment 5. Line color represents the binocular correlation of the surround RDSs (blue: cRDS, red: aRDS). The gray area represents the range for chance-level performance calculated from 120 aRDS trials (binominal test, p < 0.05, Bonferroni corrected).

Discussion

We examined the frame of reference underlying reversed depth perception that occurs when judging the depth of an aRDS in the presence of an adjacent cRDS. Participants perceived reversed depth in an aRDS with zero absolute disparity but with nonzero relative disparity to a surrounding cRDS (Experiments 1 and 3). Replacement of aRDSs with uRDSs abolished the reversal of depth judgments, confirming that reversed depth perception observed in Experiments 1 and 3 reflects comparison of disparities between the center and the surround (Experiment 2). Further, reversal of relative depth occurred across different combinations of absolute and relative disparities (Experiment 4). Finally, reversed depth perception occurred for both aRDSs surrounded by cRDSs and cRDSs surrounded by aRDSs (Experiments 4 and 5). Thus, when two RDSs are shown nearby, anticorrelation to one of the two RDSs reverses the perception of the relative depth between them. We suggest that binocular cross-correlation computation, which underlies reversed depth, produces depth perception in a relative frame of reference. 

Role of depth reference in reversed depth perception

Previous studies have shown that reversed depth perception in aRDSs requires an adjacent, binocularly correlated reference plane (Doi et al., 2011, 2013; Tanabe et al., 2008), but how the reference supports reversed depth perception had been unclear. Here, we demonstrated that what is reversed in reversed depth is the relative depth between two adjacent planes, not the absolute depth of the stimulus. This finding suggests that the critical role of the reference plane in reversed depth is to enable the calculation of relative disparity, rather than just boosting the responses to aRDSs (Doi et al., 2011). A correlated reference plane is important for fine stereoscopic depth perception (McKee & Levi, 1987; Westheimer, 1979), indicating that a relative frame of reference is constructed in both fine stereopsis and reversed depth perception. However, they are likely to have different neural substrates because fine stereopsis is largely dominated by the match-based disparity representations, whereas the correlation-based disparity representation contributes to coarse stereopsis (Doi et al., 2011). 

Our results provide an explanation for why a recent study failed to reproduce reversed depth perception in aRDSs surrounded by cRDSs (Hibbard, Scott-Brown, Haigh, & Adrain, 2014). In that experiment, the center and the surround were separated by a spatial gap of 0.35°. Spatially separating two stimuli drastically deteriorates stereoacuity (Cottereau, et al., 2012; Read et al., 2010), suggesting that the gap would prevent the visual system from forming a relative frame of reference. Thus, the absence of reversed depth in the previous study (Hibbard et al., 2014) might have resulted from a failure to construct a relative frame of reference. Indeed, recent experiments from our laboratory confirmed that reversed depth perception gradually deteriorates and is finally abolished as the width of the gap between the central and surrounding RDSs increases (Kamihirata et al., 2015). Other studies (Hayashi, Miyawaki, Maeda, & Tachi, 2003; Read & Eagle, 2000) have reported reversed depth perception in aRDSs that do not accompany any adjacent stimuli. In these experiments, however, aRDSs were superimposed on a binocularly correlated fixation point, which could then act as an adjacent depth reference. This contrasts with our experiments in which the fixation point was 0.8° away from the edge of the RDSs. 

A model of neural representation underlying reversed depth perception in aRDSs

The finding that anticorrelation reverses depth perception according to relative disparity suggests that some neurons in the visual cortex that make up the correlation-based representation encode relative disparity. The lower discrimination performance for aRDSs than for cRDSs further suggests that this representation would be less reliable for aRDSs (Figures 9–12). An extension of a previously proposed model of relative disparity encoding (Thomas, Cumming, & Parker, 2002) can generate such response properties (Figure 13). The model proposed by Thomas et al. (2002) produces a detector of relative disparity between an RDS's center and surround by combining absolute disparity detectors that operate on either the center or the surround (Figure 13A). For our extension, we assumed that the absolute disparity detectors are selective to disparity in aRDSs in a similar way as in V1 neurons; disparity tuning to aRDSs exhibits an inverted tuning shape with a lower amplitude of modulation compared with that to cRDSs (Figure 13B; Cumming & Parker, 1997; Haefner & Cumming, 2008). Figure 13C shows the responses of an example relative-disparity detector. When both the center and the surround are cRDSs (Figure 13C, left), the detector prefers a negative (near) relative disparity; the center-disparity tuning curve peaks at a negative value for a zero surround disparity, and changes in surround disparity produce corresponding peak shifts. When the center is replaced with aRDSs, this detector prefers a positive (far) relative disparity with a smaller response modulation (Figure 13C, right). As in cRDS centers, changes in the surround disparity shift the tuning curves accordingly. Thus, a simple extension of the existing model can create correlation-based encoding of relative disparity with weaker selectivity for aRDSs. 

Figure 13

 

A model for correlation-based representation of relative disparity. (A) The structure of the model. Each pair of absolute-disparity detectors consists of a unit that is selective for center disparity and a unit selective for surround disparity. The Gabor curves in the diagram represent disparity tuning curves, not receptive fields. These four units have the same Gabor disparity-tuning curves except for phases: Curves of two units in each pair have a different phase, which gives rise to a difference in preferred disparity (Δd), and curves of two units processing the same region of stimuli have orthogonal phases. A stimulus elicits responses of units, which are summed and squared separately for each pair. The outputs of pairs are again summed to define the response of the model neuron (R). This model neuron R shows a strong response when the difference in disparities between the center and the surround is Δd, for a range of different absolute disparities for the center and the surround. (B) Disparity tunings for absolute disparity detectors. Blue and red curves represent disparity tunings for cRDSs and aRDSs, respectively. (C) Responses of the model to center-cRDS (left) and -aRDS (right) stimuli. The surround was a cRDS. The top panels show the response of the model plotted against center and surround disparities. The bottom panels show the model's disparity tuning curves plotted against center disparity for different surround disparities. The left and right panels show responses of the model for cRDS- and aRDS-center, respectively. (D) Depth-discrimination performances based on the model. The difference of responses between “near” and “far” relative-disparity detectors (R_far – R_near) is plotted against center disparity. The “near” detector has been described in (C). The “far” detector is the same as the “near” detector, except that its center-disparity tuning was an inversion of that for the “near” detector and preferred “far” relative disparities. We regarded these differential responses as a proxy for the proportion of far choices. The four panels show response differences under four conditions of the correlation of the center and surround RDSs (top-left: center cRDS and surround cRDS; top-right: center aRDS and surround cRDS; bottom-left: center cRDS and surround aRDS; bottom-right: center aRDS and surround aRDS). The three curves represent response differences in different surround disparity conditions. The vertical dashed lines indicate the surround disparities.

A simple depth-discrimination scheme based on the model reproduces several aspects of psychophysical performance obtained in our experiments. We assumed that a decision-making process judges the relative depth of the center from the surround (“near” or “far”) by comparing the outputs of “near” and “far” relative-disparity detectors (Prince & Eagle, 2000; Shiozaki, Tanabe, Doi, & Fujita, 2012; see Shadlen et al., 1996, for the original formulation for motion-detection discrimination). Thus, the differential responses between “near” and “far” relative-disparity detectors were used as the decision variable that determines the proportion of “far” choices (Figure 13D). Both cRDS- and aRDS-center stimuli caused the same horizontal shift of psychometric functions along with surround disparities, but the curves for the two conditions have opposite slopes (Figure 13D, upper). In addition, as we found in Experiment 4, the slope of the predicted psychometric functions was shallower for aRDSs than for cRDSs because the response amplitudes of absolute disparity detectors were reduced for aRDSs. Essentially identical results were obtained for the central cRDS and surrounding aRDS configuration (Figure 13D, lower left), consistent with the results in Experiment 5. 

This simple model, however, has limitations. When both the center and surround are anticorrelated, the model predicts the psychometric functions identical to those for the center cRDS and surround cRDS stimuli except that they have shallower slopes (Figure 13D, lower right). In contrast, stimuli consisting only of aRDS do not evoke any depth perception, either normal or reversed (Cumming et al., 1998; Tanabe et al., 2008). Therefore, to explain the psychophysical performance under all possible combinations of the center and surround, we may still need to call for a gating mechanism that passes signals only when one of the two regions are binocularly correlated (Doi et al., 2011). Such a mechanism might be implemented by incorporating a thresholding operation in the transformation from the outputs of relative-disparity detectors into choices. In addition, the model cannot explain why adding a spatial gap between the center and surround eliminates reversed depth perception (Kamihirata et al., 2015) because the absolute disparity detectors in the model is agnostic to precise stimulus locations: They have no parameter defining exact spatial locations of their receptive field relative to the border between the center and surround. Further extension of the model is required to fully explain reversed depth perception. 

Cortical visual areas that support reversed depth perception in aRDSs

Our results suggest that neurons directly supporting reversed depth perception in aRDSs should have three properties: inverted disparity tuning to aRDSs, selectivity for relative disparity between adjacent cRDSs, and selectivity for relative disparity between aRDSs and adjacent cRDSs. The first two properties have been characterized in several cortical areas, providing clues for the neural substrates of reversed depth perception. Neurons in V1 show an inverted disparity-tuning curve for aRDSs but lack neuronal selectivity for relative disparity (Cumming & Parker, 1997, 1999). Activity in V1 is therefore unlikely to be a neural substrate that sufficiently supports depth judgments in random-dot patterns. Rather, V1 may provide signals from which downstream areas establish the representations directly relevant for depth perception (Fujita & Doi, 2016; Parker, 2004). 

MT neurons have been a strong candidate for the neural substrate for reversed depth perception, because nearly half of them have inverted disparity tuning curves in response to aRDSs (Krug et al., 2004), their responses are causally linked to coarse stereopsis (Uka & DeAngelis, 2006), and reversed depth perception is observed during coarse stereopsis (Doi et al., 2011). MT neurons are, however, selective to absolute, but not relative, disparity between two adjacent cRDS surfaces (Uka & DeAngelis, 2006), making them unlikely for mediating reversed depth. Similar arguments apply to areas V2, V3, and V3A where a subpopulation of neurons have inverted tunings for aRDSs (Allouni, Thomas, Solomon, Krug, & Parker, 2005; Okazaki & Fujita, 2010), but most neurons, if not all, encode absolute disparities (Anzai, Chowdhury, & DeAngelis, 2011; Thomas et al., 2002). 

A possible candidate is area V4. Many neurons in V4 are selective to relative disparity (Umeda, Tanabe, & Fujita, 2007) and are causally related to fine discrimination of relative disparity (Shiozaki et al., 2012). In addition, a subpopulation of V4 neurons has inverted disparity tuning to aRDSs, although the response modulation is weaker in V4 than in V1 or MT (Abdolrahmani et al., 2016; Kumano et al., 2008; Tanabe et al., 2004). A recent study has shown that the selectivity to disparity in aRDSs is almost lost when responses of V4 neurons are pooled for discriminating near/far disparities in cRDSs (Abdolrahmani et al., 2016). In the study, neurons were pooled based on the selectivity to disparity of center cRDSs. Because the disparity on the surround was fixed at zero in their experiments, both neurons selective to absolute and neurons selective to relative disparity are mixed in the pooled population. Selective pooling of V4 neurons based on the selectivity to relative disparity between the center and surround as well as the disparity modulation by aRDSs might retain encoding of the disparity in aRDSs and can support reversed depth perception. 

Neuronal sensitivity to disparity in aRDSs completely disappears in higher visual areas such as AIP in the parietal cortex (Theys et al., 2012) and IT in the temporal cortex (Janssen et al., 2003). Thus, it is unlikely that reversed depth perception in aRDSs is generated from neural activity in these higher visual areas. A previous study suggested that during fine, relative disparity discrimination, disparity signals in V4 are able to reach the decision mechanism without transiting IT (Shiozaki et al., 2012). Therefore, V4 might directly support reversed depth perception and fine discrimination of relative depth without further visual processing in IT. Taken together, current evidence points to V4 as a likely candidate of the neural substrate for reversed depth perception. 

How, then, does V4 support two aspects of stereopsis generated from different types of disparity representations—reversed depth perception generated by correlation-based representation and fine stereopsis generated by match-based representation (Doi et al., 2011)? One possibility is that the two aspects of depth perception are derived from different populations of V4 neurons. Indiscriminate pooling of the V4 neuronal activity for near/far discrimination of cRDSs gives rise to the match-based representation that can support fine stereopsis (Abdolrahmani et al., 2016). On the other hand, selective readout that preserves correlation-based representation in V4 is required to explain reversed depth perception as discussed above. Thus, V4 might be crucial for relative depth discrimination irrespective of the types of disparity representation, either correlation based or match based. 

Conclusions

We demonstrated that anticorrelation reverses depth based on relative disparity. This finding suggests that the critical role of binocularly correlated reference in reversed depth perception is to provide a relative frame of reference. Our observations can be explained by a model of relative disparity encoding, although a few aspects of reversed depth perception reported in previous studies require additional mechanisms to be fully explained. In summary, we showed that the visual system exploits correlation-based representation in a relative frame of reference for depth perception. Our finding provides a clue to examine the neural substrate of depth perception derived from correlation-based computation. 

Acknowledgments

This work was supported by grants to I. F. from the Ministry of Education, Culture, Sports, Science and Technology in Japan (KAKENHI JP15H01437, JP16H01673, JP16H03384, JP17H01381) and Ministry of Internal Affairs and Communications. We thank Takahiro Doi, Tomofumi Oga, Toshihide Yoshioka, and Bayu Gautama Wundari for helpful comments on the manuscript. 

Commercial relationships: none. 

Corresponding author: Ichiro Fujita. 

Email: . 

Address: Graduate School of Frontier Biosciences, Osaka University, Suita, Osaka, Japan.