**Abstract**:

**Abstract**
**Orientation signals, which are crucial to many aspects of visual function, are more complex and varied in the natural world than in the stimuli typically used for laboratory investigation. Gratings and lines have a single orientation, but in natural stimuli, local features have multiple orientations, and multiple orientations can occur even at the same location. Moreover, orientation cues can arise not only from pairwise spatial correlations, but from higher-order ones as well. To investigate these orientation cues and how they interact, we examined segmentation performance for visual textures in which the strengths of different kinds of orientation cues were varied independently, while controlling potential confounds such as differences in luminance statistics. Second-order cues (the kind present in gratings) at different orientations are largely processed independently: There is no cancellation of positive and negative signals at orientations that differ by 45°. Third-order orientation cues are readily detected and interact only minimally with second-order cues. However, they combine across orientations in a different way: Positive and negative signals largely cancel if the orientations differ by 90°. Two additional elements are superimposed on this picture. First, corners play a special role. When second-order orientation cues combine to produce corners, they provide a stronger signal for texture segregation than can be accounted for by their individual effects. Second, while the object versus background distinction does not influence processing of second-order orientation cues, this distinction influences the processing of third-order orientation cues.**

*fourth*-order cues in the sense used here, since a pair of points need to be inspected to extract the relevant local feature (such as contrast), and two features—and thus four points—need to be analyzed at the later filter stage. The shift in terminology is required because orientation can also be carried by correlations among

*triplets*of points—third-order in the sense used here—and the traditional terminology has no obvious parallel.

*β*'s, below), and four are third-order (the

*θ*'s, below). Each of these cues can be introduced independently or in combination, and without changing the distribution of intensities—thus eliminating a key confound.

*ensemble*, and the statistical properties described above are rigorously applicable to the

*ensemble*, not to individual images (Victor, 1994). We take the statistical view for several reasons. First, it allows for completely independent manipulation of the coordinates of interest. Rigorous control of correlations is possible at the ensemble level, but not for individual images: Since individual images are finite, the correlations estimated from the individual images will differ from that of the ensemble. Fortunately, for images of the size used, this difference is expected to be minor (Maddess, Nagai, Victor, & Taylor, 2007), so that individual images serve as good surrogates for the ensemble. Second, we are asking the subject to perform a statistical task: Even in principle, the “correct” choice on any given trial is simply the most likely choice—though the statistical evidence available to an ideal observer is quite strong. But most importantly, ensembles defined by characteristic statistics play a critical role in the normal function of the visual system. Intuitively, one recognizes a texture not by identifying a particular exemplar, but by recognizing the class to which it belongs. Experimentally, categorization of visual images into statistical classes is rapid, robust, and highly conserved across subjects (Julesz, Gilbert, & Victor, 1978; Victor & Conte, 1991), and the image statistics that support these classifications are demonstrably among the “tokens” of visual working memory (Victor & Conte, 2004).

^{2×2}kinds of blocks, there are only 10 free parameters. The reduction in the number of degrees of freedom occurs because the blocks can be placed in overlapping fashion, and they must match where they overlap. Linear combinations of these 16 probabilities provide 10 independent coordinates, which, together, fully specify the stimulus space. The coordinates fall into categories based on their order, i.e., the number of checks that must be simultaneously inspected to determine their values. As detailed below, there is one first-order coordinate (denoted

*γ*), four second-order coordinates (denoted

*β*

_{_},

*β*

_{|},

*β*

_{\}, and

*β*

_{/}), four third-order coordinates (denoted

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈}, and

*θ*

_{⌉}), and one fourth-order coordinate (denoted

*α*). All coordinates range from −1 to 1, and a completely random binary image corresponds to all 10 coordinates having the value 0. Our focus is on the four

*β*'s and the four

*θ*'s, since they are the potential carriers of orientation information. To describe these and the two nonoriented coordinates

*γ*and

*α*in detail, we use the convention that white checks are denoted by 1, and black checks are denoted by 0.

*γ*: It is the difference between the probability of a white check and the probability of a black check. If

*γ =*1, all checks are white; if

*γ*= −1, all checks are black; and if

*γ*= 0, both colors are equally likely.

*β*'s capture the pairwise (second-order) statistics: They are the difference between the probability that two neighboring checks match (i.e., both are white or both are black), and the probability that they do not match (i.e., one is white and one is black). If

*β*= 1, all checks match their nearest neighbor (in the direction indicated by the subscript), and if

*β*= −1, they all mismatch. The four subscripts (

*β*

_{_},

*β*

_{|},

*β*

_{\}, and

*β*

_{/}) correspond to the direction that is relevant to the match. For example,

*β*

_{_}= 1 means that all 1 × 2 blocks are either (0 0) or (1 1) and none are (0 1) or (1 0); such images will be dominated by horizontal stripes. If

*β*

_{_}= −1, all 1 × 2 blocks are either (0 1) or (1 0) and none are (0 0) or (1 1); in such images, horizontal rows will have alternating black and white checks. Values of

*β*

_{_}between 0 and 1 indicate a partial bias towards matching neighbors, while values between −1 and 0 indicate a partial bias towards mismatching neighbors, and

*β*

_{_}= 0 means that matching and mismatching neighbors are equally likely. Similarly,

*β*

_{|},

*β*

_{\}, and

*β*

_{/}capture the pairwise correlations in the vertical direction and the two oblique directions. Image patches with

*β*= ±0.4 are shown in Figure 2, upper panels.

*θ*'s capture the statistics of triplets of checks arranged in an ⌊-shaped configuration. Since there are four possible orientations for an ⌊-shaped configuration within a 2 × 2 windowpane, there are four

*θ*-statistics,

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈}, and

*θ*

_{⌉}. Each of them measures the third-order correlation within the corresponding ⌊-shaped region by comparing the probability that the region contains an even number of white checks versus an odd number of white checks. If

*θ*= 1, only an odd number of white checks (one or three) are present;

*θ*= −1 means the opposite. For example,

*θ*

_{⌋}means that only the configurations $ ( 1 1 1 ) $, $ ( 1 0 0 ) $, $ ( 0 1 0 ) $, or $ ( 0 0 1 ) $ are present (the fourth element of the 2 × 2 region is unconstrained); such images will have prominent white triangular-shaped regions pointing downward and to the right. If

*θ*

_{⌋}= −1, only the configurations $ ( 0 0 0 ) $, $ ( 0 1 1 ) $, $ ( 1 0 1 ) $, or $ ( 1 1 0 ) $ are present; such images will have prominent black triangular-shaped regions. Image patches with

*θ*= ±0.72 (for

*θ*

_{⌋},

*θ*

_{⌊}, and

*θ*

_{⌈}) are shown in Figure 2, lower panels.

*α*, captures the statistics of quadruplets of checks in a 2 × 2 block:

*α*= 1 means that an even number of them are white, and

*α*= −1 means that an odd number are white. This gamut has been studied extensively (Julesz et al., 1978; Victor, Chubb, & Conte, 2005; Victor & Conte, 1989, 1991, 1996, 2004), and is not our focus here.

*γ*,

*β*

_{_},

*β*

_{|},

*β*

_{\},

*β*

_{/},

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈},

*θ*

_{⌉},

*α*}, eight of them carry orientation information. They consist of the four second-order statistics {

*β*

_{_},

*β*

_{|},

*β*

_{\},

*β*

_{/}} and the four third-order statistics {

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈},

*θ*

_{⌉}}.

*β*'s or one of the

*θ*'s) was set at a nonzero value, either positive or negative. In each direction, four or five equally-spaced values were chosen to span the range from below threshold to well above threshold based on pilot experiments. For

*β*

_{_}and

*β*

_{|}, the maximum (absolute value) was 0.45, for

*β*

_{\}and

*β*

_{/}, the maximum was 0.75, and for the

*θ*s, the maximum was 1.0. (b) In a coordinate plane: A pair of coordinates was set at a nonzero value. This was done in all quadrants (i.e., in all sign combinations: both coordinates positive, both negative, and coordinates that were opposite in sign). The ratio of the coordinate values was fixed and chosen in approximate proportion to the above maximum values. Two values along each direction were studied. (c) Combinations of four coordinates of the same order (all four

*β*'s or all four

*θ*'s). All four coordinates had the same absolute value, and their signs were chosen either to match or to alternate as a function of orientation (see Figure 7). Four equally-spaced values were chosen, 0.075, 0.125, 0.175, and 0.225. (0.225 is 90% of the maximum possible value.) As described in detail in Victor and Conte (2012; see its table 2), the unspecified coordinates were assigned by first setting the values of all lower-order coordinates to zero, and then setting the remaining coordinates to values that maximized the entropy of the resulting images. In most cases, these other coordinate values were zero; in the cases in which the value was nonzero, it was below the perceptual threshold. For example, for a (

*β*

_{_},

*β*

_{|}) combination, the maximum-entropy value of

*α*is approximately $ \beta _ 2 $ + $ \beta | 2 $. The thresholds are <0.2 for this combination of

*β*'s, and the corresponding

*α*= 0.08 is far below its threshold, which is >∼0.5. Full details for the construction of the on-axis and coordinate-plane stimulus are provided in (Victor & Conte, 2012).

*β*'s, we mixed a stimulus specified by nonzero values of

*β*

_{_}and

*β*

_{\}with a stimulus specified by nonzero values of

*β*

_{|}and

*β*

_{/}. For the

*θ*'s, we mixed a stimulus specified by nonzero values of

*θ*

_{⌋}and

*θ*

_{⌊}with one specified by

*θ*

_{⌈}and

*θ*

_{⌉}(when

*θ*'s had the same signs), and we mixed a stimulus specified by

*θ*

_{⌋}and

*θ*

_{⌈}with one specified by

*θ*

_{⌊}and

*θ*

_{⌉}(when

*θ*'s had the alternating signs). These strategies ensured that the values of the unspecified coordinates were exactly zero, with the exception that for stimuli constructed as a mixture of four

*β*'s, the value of

*α*could be as high as 0.36 (for all

*β*'s set at 0.225, the largest value used). Since this value was not negligible, we assessed its effect on threshold in two subjects (MC and DT), by determining sensitivities to

*α*and its pairwise interactions with the

*β*'s.

^{2}, a refresh rate of 100 Hz, and a presentation duration of 120 ms, driven by a Cambridge Research ViSaGe system (Cambridge Research Systems, Ltd., Rochester, Kent, UK).

*β*and one

*θ*. Three of the seven subjects (MC, DT, and DC) participated in experiments to assess four-component combinations. On-axis sensitivities from DC were also measured, but to a much more limited extent. Of the seven subjects, MC is an experienced psychophysical observer, DC had no observing experience prior to the current study, and the other subjects had modest viewing experience (10 to 100 hours). All subjects other than MC and DT were naive to the purposes of the experiment. All subjects had visual acuities (corrected if necessary) of 20/20 or better.

*β*

_{_},

*β*

_{|},

*β*

_{\}) or two (

*θ*

_{⌋},

*θ*

_{⌊}) axes (four positive and four negative strengths), and in each of three combination directions (four strengths each). This resulted in 192 (

*β*'s) or 128 (

*θ*'s) on-axis stimuli and 96 combination stimuli, resulting in 288 (

*β*'s) or 224 (

*θ*'s) trials per block. We collected 15 such blocks per subject (4,320

*β*'s or 3,360

*θ*'s trials), grouped into five experimental sessions, yielding 120 responses for each set of coordinate values.

*r*. For rays along the coordinate axes,

*x*is the coordinate value; for the rays in the oblique directions,

*x*is the distance from the origin. In most cases, the Weibull shape parameter (the exponent

*b*) was in the range 2.2 to 2.7 for each ray, or had confidence limits that included this range. Therefore, we fit the entire dataset in each coordinate plane by a set of Weibull functions constrained to share a common exponent

_{r}*b*, but allowing the position parameter

*a*to vary across rays. (Note, however, that the exponent

_{r}*b*was allowed to vary

*between*planes; see Table 1, below). Next

*a*was taken as a measure of threshold, as

_{r}*x*=

*a*yields performance halfway between floor and ceiling (here, FC = 0.625). The 95%-confidence limits for

_{r}*a*were determined via 1,000-sample bootstraps. When performance was sufficiently close to chance for the entire ray, the upper confidence limit of these bootstraps was large (e.g., >10

_{r}^{5}); in these cases, the threshold was taken to be infinity. Unless otherwise noted, averages across subjects were calculated as harmonic means. (We used harmonic means to avoid divergences that would have resulted from averaging immeasurably large thresholds. The harmonic mean of thresholds is equivalent to the arithmetic mean of the sensitivities.)

Subject | Second-order statistics | Third-order statistics | |||||

(β_{_}, β_{|}) | (β_{_}, β_{\}) | (β_{\}, β_{/}) | Geometric mean | (θ_{⌋}, θ_{⌊}) | (θ_{⌋}, θ_{⌈}) | Geometric mean | |

MC | 2.35 | 2.82 | 2.54 | 2.56 | 2.09 | 2.54 | 2.31 |

DT | 2.78 | 2.96 | 3.14 | 2.95 | 3.28 | 3.49 | 3.38 |

JD | 3.00 | 3.65 | 3.50 | 3.37 | 4.55 | 4.21 | 4.37 |

DF | 2.83 | 2.81 | 2.71 | 2.78 | 3.33 | 3.13 | 3.23 |

KP | 2.83 | 3.42 | 3.23 | 3.15 | 3.06 | 3.27 | 3.16 |

TT | 2.92 | 2.89 | 2.72 | 2.84 | 3.42 | 3.41 | 3.41 |

Geometric mean | 2.78 | 3.08 | 2.95 | 2.93 | 3.21 | 3.30 | 3.25 |

*b*, but allowing for different values of

*a*for each combination. Here, for the combination directions, we used the convention that

_{r}*x*in Equation 1 is the common coordinate value of each component, since this facilitates the key comparisons below. (This convention is not the same as expressing thresholds in terms of distance from the origin. Since all four coordinates had the same absolute value, the Euclidean distance of a typical point (±

*x*, ±

*x*, ±

*x*, ±

*x*) from the origin is 2

*x*. Thus, to convert thresholds expressed as single coordinate values into thresholds expressed as distance from the origin, the numerical value should be doubled.)

_{i}_{,j}

*Q*

_{i}_{,j}

*c*, where

_{i}c_{j}*c*represents an individual image statistic (for mixtures of second-order statistics,

_{i}*c*

_{1}=

*β*

_{_},

*c*

_{2}=

*β*

_{|},

*c*

_{3}=

*β*

_{\}, and

*c*

_{4}=

*β*

_{/}; for mixtures of third-order statistics,

*c*

_{1}=

*θ*

_{⌋},

*c*

_{2}=

*θ*

_{⌊},

*c*

_{3}=

*θ*

_{⌈}, and

*c*

_{4}=

*θ*

_{⌉}) and the quantities

*Q*

_{i}_{,j}to describe how the image statistics

*c*and

_{i}*c*combine and interact. Without loss of generality, the threshold is set to 1 (since alternative values could be absorbed into the

_{j}*Q*

_{i}_{,j}). Thus, the model states that threshold is reached when

*Q*

_{i}_{,j}.

*Q*

_{i}_{,j}from the thresholds

*T*measured along all rays

_{r}*r*(including the on-axis and in-plane rays). That is, we adjusted the

*Q*

_{i}_{,j}so that along each ray

*r*, ∑

_{i}_{,j}

*Q*

_{i}_{,j}

*c*(

_{i}*T*)

_{r}*c*(

_{j}*T*) was as close as possible to 1, where

_{r}*c*(

_{i}*T*) is the value of the texture coordinate

_{r}*i*when threshold is reached in direction

*r*. The adjustment of the

*Q*

_{i}_{,j}was accomplished by minimizing which is a linear least-squares fitting procedure for the

*Q*

_{i}_{,j}. Note that

*F*= 0 only if the threshold

*T*in each direction

_{r}*r*is exactly predicted by Equation 2, namely, ∑

_{i}_{,j}

*Q*

_{i}_{,j}

*c*(

_{i}*T*)

_{r}*c*(

_{j}*T*) = 1. Once the parameters

_{r}*Q*

_{i}_{,j}are determined by minimizing Equation 3, Equation 2 provides a prediction of thresholds along any ray, including rays in directions that correspond to the four-component mixtures: it is the value

*T*for which

_{pred}*α*, which indicates the fraction of 2 × 2 blocks that contain an even number of white checks. Nonzero values of

*α*arise because the correlations among each pair of checks in a 2 × 2 block induces correlations among the quadruple of checks. For example, if all nearest-neighbor pairs tend to match, then the number of white checks within a 2 × 2 block is more likely to be even. For further details, see Victor and Conte [2012]). To take the fourth-order correlations into account, we include a fifth coordinate

*c*

_{5}=

*α*in Equation 2 and Equation 3, along with the four coordinates corresponding to the second-order statistics (

*c*

_{1}=

*β*

_{_},

*c*

_{2}=

*β*

_{|},

*c*

_{3}=

*β*

_{\}, and

*c*

_{4}=

*β*

_{/}). In two of the subjects (MC and DT) in which we measured responses to the four-component mixtures, we also determined the isodiscrimination contours in the planes spanned by

*α*and the

*β*s (Victor & Conte, 2012), and we used those results here to fit the five-coordinate version of Equation 3, and to predict thresholds via Equation 4. The remaining subject (DC) only participated in experiments involving the individual image statistics and the four-component mixtures. For this subject, we determined the model parameters by using his measured thresholds along the coordinate axes and the orientations of the ellipses obtained from MC or DT. This amounts to stretching the best-fitting ellipses from subject MC or DT along each coordinate axis to match DC's single-statistic thresholds (i.e., rescaling each

*Q*

_{i}_{,j}by a factor

*g*), and rescaling the interaction terms

_{i}*Q*

_{i}_{,j}by the geometric mean of these scaling factors $ g i g j $.

*(where*

_{k}q_{k}c_{k}*q*is the sensitivity of the channel to the statistic

_{k}*c*). The above model reduces to this case when the off-diagonal parameters

_{k}*Q*

_{i}_{,j}are chosen according to

*Q*

_{i}_{,j}= $ Q i , i Q j , j $. In this case, ∑

_{i}_{,j}

*Q*

_{i}_{,j}

*c*= (∑

_{i}c_{j}*)*

_{k}q_{k}c_{k}^{2}and Equation 2 is equivalent to

*Q*

_{i}_{,j}= 0 for

*i*≠

*j*in the above, so Equation 2 becomes

*q*and

_{k}*Q*

_{k}_{,k}for the above models were determined from the on-axis thresholds.

*β*

_{_},

*β*

_{|},

*β*

_{\},

*β*

_{/}}, which captured pairwise correlations in each of four orientations); four were third-order {

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈},

*θ*

_{□}}, which captured correlations among three checks, in each of four orientations. We first describe sensitivity to individual statistics, then examine their combinations.

*β*

_{_},

*β*

_{|}) than for the two oblique directions (

*β*

_{\},

*β*

_{/}), but there was little difference between the two directions within each category or between positive and negative excursions. These findings held across the

*N*= 6 subjects. The (geometric) mean threshold for cardinal directions was 0.286 (0.258 to 0.316, 95% confidence limits via

*t*test on log thresholds); for oblique directions it was 0.402 (0.362 to 0.446). This difference was highly significant (

*p*< 0.001, two-tailed paired

*t*test on log thresholds). Note that the cardinal and oblique statistics do not differ merely in orientation:

*β*

_{_}and

*β*

_{|}describe the correlation between checks that share an edge, while

*β*

_{\}and

*β*

_{/}describe the correlation between checks that share a corner. For either kind of image statistic, thresholds across subjects varied by only 10% (standard deviation from the mean on a log scale).

*p*> 0.05). There were no systematic differences in performance in the conditions in which the target was random and the background was structured versus conditions in which the target was structured and the background was random. There also was no consistent difference in performance for any of the four target positions.

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈}) were not significantly different (

*p*> 0.05 for pairwise comparisons of any two of the

*θ*'s, in either positive or negative excursions);

*θ*

_{⌉}was only tested in pilot fashion. However, there was a consistent difference between positive excursions (images with white triangular regions) and negative excursions (images with black triangular regions): Positive excursions had a slightly lower threshold (

*p*≈ 0.05 for

*θ*

_{⌋},

*p*< 0.005 for

*θ*

_{⌊},

*p*< 0.1 for

*θ*

_{⌈}, and

*p*< 0.01 when pooled). Across subjects, the threshold for positive excursions was 0.767 (0.663 to 0.887) and for negative excursions, 0.847 (0.721 to 0.995). This corresponds to a variation between subjects of 15% (standard deviation on a log scale).

*β*

_{_},

*β*

_{|}}, {

*β*

_{\},

*β*

_{/}}, and {

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈}}. For second-order statistics, thresholds for positive excursions (an increase in the number of neighbor-pairs that match) were indistinguishable from thresholds for negative excursions (an increase in the number of neighbor pairs that do not match). For third-order statistics, thresholds for positive excursions (bright triangles) were about 10% lower than thresholds for negative excursions (dark triangles).

*θ*-statistic, by definition, indicates the extent to which the texture is enriched with uniform three-check triangular regions (containing two checks on a side). The next-smallest triangular region, a six-check region (three checks on a side), contains within it three overlapping examples of the three-check triangle. Because the six-check triangle contains three instances of the three-check triangle (and therefore, three independent instances in which the

*θ*-bias is applied), the extent to which there is an enrichment of six-check triangles is given by

*θ*

^{3}. Similarly, a 10-check triangle (four checks on a side) contains six instances of the three-check trial, and therefore six independent applications of the

*θ*-bias, and is enriched by

*θ*

^{6}. These accelerating functions quantify the presence of triangles: For values of

*θ*near zero, they are no more frequent than chance; as

*θ*approaches −1 or +1, their frequency increases rapidly. Thus, if detection is based on these features, the transition from subthreshold to suprathreshold is expected to be more rapid than if detection is based on a feature whose presence is merely proportional to

*θ*. The rapidity of this transition is quantified by the Weibull exponent in Equation 1: If detection is based on a feature whose presence grows like

*θ*rather than

^{p}*θ*, then the Weibull exponent (for fraction correct as a function of

*x*= |

*θ*| in Equation 1) is expected to be

*p*times higher:

*p*≈ 0.05, one-tailed paired

*t*test,

*N*= 6). This is in contrast with the several-fold change in the exponent that would be expected if detection of structure was based on cooperative interactions of local statistics.

*pairs*of statistics. This motivates the analysis below, where we examine how threshold depends on the relative orientation and sign of two image statistics.

*θ*

_{⌋}= 0.4 and

*θ*

_{⌊}= −0.4. If these statistics are processed in a completely pooled fashion (as would be the case if the underlying mechanisms had no orientation tuning), then the image would be difficult to distinguish from a random one. This is because the two image statistics have magnitudes that are equal but are opposite in sign. (For further discussion specific to the Motoyoshi et al., [2007] model, see Discussion). On the other hand, if the two image statistics are processed independently (as would be the case if there are multiple mechanisms, each with its own orientation tuning), then the image would be readily discriminable from a random image, since cancellation would not occur. Thus, testing thresholds for a combination of image statistics that differ in orientation and sign will determine if the image statistics are processed in an orientation-specific way: If processing is pooled across orientations, there will be cancellation; if processing is orientation-specific, cancellation need not occur. A useful control for this analysis is the companion image in which the same two statistics have the same sign: This will not result in cancellation even if the statistics are processed in a pooled fashion.

*I*for any two image statistics, which compares the same-sign thresholds (

_{pool}*h*

_{+,+}and

*h*

_{−,−}) to the opposite-sign thresholds (

*h*

_{+,−}and

*h*

_{−,+}):

*I*= 1 means that the combination thresholds are the same, whether the statistics are present with the same sign or opposite sign, indicating independent processing (a lack of cancellation).

_{pool}*I*> 1 means that some cancellation has occurred.

_{pool}*I*= ∞ means that cancellation is complete (i.e., opposite-sign thresholds are infinite, and any potential orientation information is lost).

_{pool}*β*

_{_}and

*β*

_{|}), the results are particularly simple (Figure 3, first row): Thresholds are nearly identical whether they are combined with opposite sign or with the same sign.

*I*is very nearly 1 (

_{pool}*I*ranges from 0.97 to 1.21 for individual subjects,

_{pool}*I*= 1.09 from harmonic means,

_{pool}*N*= 6, Figure 6, below).

*β*

_{_}and

*β*

_{\}, second row of Figure 3), the individual thresholds are different (as expected from Figure 2A), but again thresholds for the combination are independent of whether the signs are opposite or the same (

*I*ranges 1.03 to 1.11, mean = 1.06, Figure 6, below).

_{pool}*β*

_{\}and

*β*

_{/}, elicits a different behavior. In contrast to the above two cases, the isodiscrimination contours are not aligned with the coordinate axes—instead, they are tilted, with the long axes extending into the quadrants in which

*β*

_{\}and

*β*

_{/}have opposite signs (Figure 3, third row). The tilt means that thresholds for the combinations with opposite sign are higher than for combinations with the same sign (

*I*ranges from 1.34 to 2.10, mean = 1.62,

_{pool}*N*= 6, Figure 6, below).

*β*

_{\}and

*β*

_{/}show a small but consistent deviation from the elliptical shape: Thresholds when both statistics are negative are lower than when both statistics are positive (blue arrow in Figure 3, bottom row). This deviation was seen, with varying degrees of prominence, in all six subjects. Interestingly, the images defined by

*β*

_{\}< 0 and

*β*

_{/}< 0 have the appearance of a maze, with many corners oriented along the cardinal axes (see region inside the blue arc in Figure 3, bottom row). But even though this quadrant of the space (

*β*

_{\}< 0 and

*β*

_{/}< 0) has “corners,” its degree of statistical structure is identical to what is present in the other quadrants of the space (i.e.,

*β*

_{\}> 0 or

*β*

_{/}> 0 or both (Victor & Conte, 2012). Thus, the lower thresholds indicate that pairings of second-order statistics that produce corners are processed more efficiently than other second-order pairings.

*θ*

_{⌋}and

*θ*

_{⌊}), and statistics that differ by 180° from each other (bottom row:

*θ*

_{⌋}and

*θ*

_{⌈}). In contrast to the findings for pairwise interactions of second-order statistics, thresholds are markedly elevated when the two statistics have opposite signs versus when they are the same. Correspondingly,

*I*ranged from 1.60 to 2.55 for individual subjects (mean = 2.05,

_{pool}*N*= 6) for statistics that differ by 90°, and from 1.15 to 1.74 (mean = 1.45,

*N*= 6) for statistics that differ by 180° (Figure 6, below).

*θ*

_{⌋},

*θ*

_{⌈}):

*I*ranged from 1.54 to ∞ for individual subjects (mean = 2.60) when the target was structured, but was 0.90 to 1.27 (mean = 1.12) when the background was structured. Note that for two subjects,

_{pool}*I*= ∞ in the structured-target condition. That is, for these individuals, thresholds were too high to measure reliably when image statistics had equal and opposite signs, implying nearly complete cancellation. No such cancellation occurred when the statistical structure was in the background, or when image statistics had the same sign. Across subjects,

_{pool}*I*for the two kinds of conditions (structured target vs. structured background) differed by approximately a factor of two for third-order statistics; for the second-order statistics, the difference was less than 30% (Figure 6).

_{pool}*β*(

*β*

_{_}or

*β*

_{|}) with one of the

*θ*'s are identical, other than a rotation or a reflection. Since we found no significant differences between the individual statistics that differ solely by a rotation, we focused on one example, (

*β*

_{_},

*θ*

_{⌋}) (first row of Figure 5).

*θ*'s (say

*θ*

_{⌋}). The two cases differ according to whether the second-order statistic involves the vertex of the ⌋ (i.e.,

*β*

_{\},

*θ*

_{⌋}), or alternatively, it spans across the vertex (i.e.,

*β*

_{/},

*θ*

_{⌋}). These cases are shown in the last two rows of Figure 5.

*I*is slightly less than 1: 0.90 for (

_{pool}*β*

_{_},

*θ*

_{⌋}), 0.81 for (

*β*

_{\},

*θ*

_{⌋}), and 0.77 for (

*β*

_{/},

*θ*

_{⌋}) (means across

*N*= 4 subjects, range 0.73 to 0.96). In qualitative terms, the tilt of the isodiscrimination contour translates into the statement that detection of pairwise correlation (

*β*> 0) is slightly enhanced in the context of large dark regions (

*θ*< 0), and detection of pairwise

*anti*correlation (

*β*< 0) is slightly enhanced in the context of large bright regions (

*θ*> 0), independent of their relative orientations.

*β*'s, or all four values of the

*θ*'s, had the same magnitude but might differ in sign. For independent processing, thresholds would be independent of sign. For pooled processing, thresholds would be markedly increased when statistics had opposite signs, because their influences would cancel. We first consider the experimentally-determined thresholds, and then the predictions of models based on pooled processing, independent processing, and an intermediate scenario.

*β*'s (filled circles) differed from the thresholds with same-sign

*β*'s, but the difference was modest. When the cardinal

*β*'s (

*β*

_{_},

*β*

_{|}) were positive and the oblique

*β*'s (

*β*

_{\},

*β*

_{/}) were negative, the thresholds ranged from being the same as that of the same-sign

*β*'s, to somewhat lower (range of ratios, 0.76 to 0.99). When the cardinal

*β*'s (

*β*

_{_},

*β*

_{|}) were negative and the oblique

*β*'s (

*β*

_{\},

*β*

_{/}) were positive, the thresholds ranged from being similar to that of the same-sign

*β*'s, to moderately higher (range, 1.00 to 1.74). In contrast, for the third-order statistics, thresholds with opposite-sign

*θ*'s were almost threefold higher than for same-sign thresholds (range, 2.62 to 2.95). There was virtually no difference between thresholds with all-positive

*θ*'s and all-negative

*θ*'s (ratio range, 0.94 to 1.03).

*c*to represent individual image statistics (for mixtures of second-order statistics,

_{k}*c*

_{1}=

*β*

_{_},

*c*

_{2}=

*β*

_{|},

*c*

_{3}=

*β*

_{\}, and

*c*

_{4}=

*β*

_{/}; for mixtures of third-order statistics,

*c*

_{1}=

*θ*

_{⌋},

*c*

_{2}=

*θ*

_{⌊},

*c*

_{3}=

*θ*

_{⌈}, and

*c*

_{4}=

*θ*

_{⌉}) and

*q*to represent the sensitivity to the statistic

_{k}*c*, this model states that threshold is reached when

_{k}*β*

_{_}and

*β*

_{|}were statistically indistinguishable, so they can be set to the same common value,

*q*. Similarly, the sensitivities for

_{card}*β*

_{\}and

*β*

_{/}were indistinguishable, and we set them to the common value,

*q*. With these substitutions, Equation 9 simplifies to

_{obl}*θ*'s (

*θ*

_{⌋},

*θ*

_{⌊},

*θ*

_{⌈}) are indistinguishable, and we assume that the threshold for

*θ*

_{⌉}shares the same value, which we designate by

*q*. So for third-order statistics, Equation 9 simplifies to

_{θ}*β*'s or the

*θ*'s to common multiples of ±1. For example, to determine the model's predicted threshold

*T*

_{++++}for a mixture of all positive

*β*'s, we set

*β*

_{_}=

*β*

_{|}=

*β*

_{\}=

*β*

_{/}=

*T*

_{++++}in Equation 10, to find that

*T*

_{+−+−}for a mixture of positive cardinal

*β*s and negative oblique

*β*'s, we set

*β*

_{_}=

*β*

_{|}=

*T*

_{+−+−}and

*β*

_{\}=

*β*

_{/}= −

*T*

_{+−+−}in Equation 10, to find that which is much larger than

*T*

_{++++}. These values are plotted in Figure 7A (downward triangles). As is shown, they are at odds with the measured thresholds: The measured same-sign thresholds are larger than predicted by the pooled model, and the opposite-sign thresholds are much smaller than predicted. For the third-order statistics, on the other hand, the model is at least qualitatively consistent with experimental findings (Figure 7B). For same-sign

*θ*'s, the predicted threshold (from Equation 11, by taking

*θ*

_{⌋}=

*θ*

_{⌊}=

*θ*

_{⌈}=

*θ*

_{⌉}=

*T*

_{++++}) is (1/4

*q*). This modestly underpredicts the experimental threshold. For opposite-sign

_{θ}*θ*'s, the predicted threshold is infinite, since when the

*θ*'s have opposite signs in pairs,

*θ*

_{⌋}+

*θ*

_{⌊}+

*θ*

_{⌈}+

*θ*

_{⌉}= 0. Measured thresholds are in fact infinite in some subjects, and in others, it is close to the maximal value that could be measured (Figure 4). In sum, the complete pooling model fails dramatically to predict the measured thresholds for combinations of second-order statistics (it predicts a large dependence on relative sign, when the data show only modest sign-dependence). For third-order predictions, the failure is only a quantitative one, as the predicted large sign-dependence is in fact observed.

*c*to represent individual image statistics and

_{k}*q*to represent the sensitivity to the statistic

_{k}*c*, this model states that threshold is reached when (see also Methods, Equation 6). Because of the similarity of the thresholds for the two cardinal

_{k}*β*'s, the two oblique

*β*'s, and the

*θ*'s, this equation reduces to for mixtures of second-order statistics and for mixtures of third-order statistics. As expected, the form of Equations 15 and 16 shows that the independent-processing model predicts thresholds for mixtures that are independent of whether the statistics have the same sign, or opposite signs.

*β*-condition is intermediate between the predictions of the pooled model and the independent model. For the condition in which the cardinal

*β*'s are positive and the oblique

*β*'s are negative, it correctly predicts thresholds that are lower than both the pooled and independent model. For the condition in which the cardinal

*β*'s are negative and the oblique

*β*'s are positive, it correctly predicts thresholds that are close to, but somewhat higher than, the thresholds predicted by the independent model. With the exception of the all-positive

*β*condition for subject DT, where there is a 50% deviation, all thresholds are correctly predicted within 20%, and many within 10%. A similar level of agreement is seen for the mixtures of third-order statistics (panel B): For same-sign

*θ*'s, the threshold values are correctly predicted at values intermediate between the predictions of the pooled and independent models. For opposite-sign

*θ*'s, the ellipsoid model predicts that thresholds are too high to measure; the experimental data show that they are much higher than for the same-sign

*θ*'s, but close to the limits that can be measured.

*anti*correlations in a particular direction. At third-order, there were subtle differences in sensitivity for positive and negative correlations, with sensitivity to positive correlations (corresponding to white oriented regions) about 10% greater than sensitivity to negative correlations (corresponding to black oriented regions). We emphasize that the positive and negative quantities considered here are the

*correlations*of image pixels, not the contrast polarity of the image tokens themselves (Motoyoshi & Kingdom, 2007).

*θ*-statistics, because—as we show below—such stimuli must generate an equal and symmetric distribution of signals in the ON and OFF pathways.

*T*in which

*θ*

_{⌋}and

*θ*

_{⌊}(a pair that differ by 90°) are present with equal magnitudes but opposite signs. Since the ON filters of Motoyoshi et al. (2007) are circularly symmetric, they must yield the same distribution of responses

*x*from this texture, and from the left-right mirror reflection of it, i.e., $ p T O N $(

*x*) = $ p m i r r o r ( T ) O N $(

*x*). Since

*θ*

_{⌋}and

*θ*

_{⌊}carry opposite signs, this left-right reflection—which interchanges one statistic with the other—inverts each of their values. Sign-inversion of the

*θ*s is equivalent to exchanging black for white (contrast inversion). Since the filters are assumed linear, contrast inversion of the image (consequent to exchanging

*T*for its mirror) results in an inversion of the distributions of signals that emerge from the filters, so $ p m i r r o r ( T ) O N $(

*x*) = $ p T O N $(−

*x*). Combining $ p T O N $(

*x*) = $ p m i r r o r ( T ) O N ( x ) $ with $ p m i r r o r ( T ) O N $ = $ p T O N $(−

*x*) yields $ p T O N $(

*x*) = $ p T O N $(−

*x*). Similarly, for the OFF pathway, $ p T O F F $(

*x*) = $ p T O F F $(−

*x*). Thus, prior to the nonlinearity, the signals on both ON and OFF pathways are even-symmetric (and consequently, have no skewness). Moreover, the distributions of signals on the ON and OFF pathways are identical, since linearity requires that $ p T O N $(−

*x*) = $ p T O F F $(

*x*). This means that opponent processing must lead to a null signal, regardless of an intervening nonlinearity prior to pooling.

*θ*-statistics that differ by 180° (such as

*θ*

_{⌋}and

*θ*

_{⌈}), a similar argument holds, based on a 180° rotation of the texture, rather than a mirror flip.

*fourth-order*stimuli in the current terminology, since in general, four points are needed to extract the orientation. As such, while those studies do not make use of the same kinds of stimuli used here, they demonstrated that high-order orientation cues are salient and identified the crucial stimulus characteristics (Landy & Henry, 2007; Landy & Oruc, 2002). However, these studies stopped short of determining selectivity compared to that of the ideal observer, or how cues combine—since it is not obvious how to use those stimuli to build a “calibrated” space of the sort used here. It is also worth noting that these higher-order cues are extracted by V1 neurons (Baker & Mareschal, 2001), and that simple models (consisting of a nonlinear subunit followed by an oriented filter) suffice to account for this. Conversely, the third-order stimuli used here are well-known (Julesz et al., 1978), but their capacity to carry orientation information, and their interaction with simple (i.e., second-order) orientation cues has not previously been investigated.

*Progress in Brain Research*, 134, 171–191. [PubMed]

*Vision Research*, 44, 257–277. [CrossRef] [PubMed]

*Biological Cybernetics*, 28 (3), 167–175. [CrossRef] [PubMed]

*Biological Cybernetics*, 29 (4), 201–214. [CrossRef] [PubMed]

*Vision Research*, 44 (27), 3223–3232. [CrossRef] [PubMed]

*Nature*, 399 (6737), 655–661. [PubMed]

*Nature Neuroscience*, 2 (8), 740–745. [PubMed]

*Visual pattern analyzers*. Oxford: Clarendon Press.

*Vision Research*

*,*32 (4), 719–743. [CrossRef] [PubMed]

*Journal of Vision*, 10 (3): 9, 1–16, http://www.journalofvision.org/content/10/3/9, doi:10.1167/10.3.9. [PubMed] [Article] [CrossRef] [PubMed]

*European Journal of Neuroscience*, 33, 130–142. [CrossRef]

*Journal of Neuroscience*, 24, 3313–3324. [CrossRef] [PubMed]

*Nature*, 290 (5802), 91–97. [CrossRef] [PubMed]

*Biological Cybernetics*, 31 (3), 137–140. [CrossRef] [PubMed]

*Perception, 36 ECVP Abstract Supplement*.

*Vision Research*, 42, 2311–2329. [CrossRef] [PubMed]

*Journal of Neurophysiology*, 95 (2), 862–881. [PubMed]

*Vision Research*, 35 (19), 2697–2722. [CrossRef] [PubMed]

*Journal of the Optical Society of America*, 32, 247–273. [CrossRef]

*Journal of the Optical Society of America A: Optics, Image Science, & Vision*, 24 (2), 278–293. [CrossRef]

*Vision Research*, 43, 2197–2205. [CrossRef] [PubMed]

*Vision Research*, 2007, 2047–2054. [CrossRef]

*Journal of Vision*, 10 (1): 3, 1–8, http://www.journalofvision.org/content/10/1/3, doi:10.1167/10.1.3. [PubMed] [Article] [CrossRef] [PubMed]

*Nature*, 447, 206–209. [CrossRef] [PubMed]

*Pattern Recognition Mechanisms, Experimental Brain Research*(Suppl. 11; pp. 117–151). Berlin: Springer-Verlag.

*Journal of the Optical Society of America*, A7, 783–789. [CrossRef]

*Kybernetik*, 16, 65–67. [CrossRef] [PubMed]

*Nature Neuroscience*, 8 (12), 1647–1650. [CrossRef] [PubMed]

*Vision Research*, 58, 59–67. [CrossRef] [PubMed]

*Perception & Psychophysics*, 43, 601–603. [CrossRef] [PubMed]

*Journal of Vision*, 10 (9): 2, 1–18, http://www.journalofvision.org/content/10/9/2, doi:10.1167/10.9.2. [PubMed] [Article] [CrossRef] [PubMed]

*Nature Neuroscience*, 7, 1129–1135. [CrossRef] [PubMed]

*Journal of Mathematical Psychology*, 1, 54–87. [CrossRef]

*Proceedings of the Royal Society of London B*, 278, 1365–1372. [CrossRef]

*Journal of the Optical Society of America A*, 11 (5), 1680–1684. [CrossRef]

*Vision Research*, 45 (3), 311–328. [CrossRef] [PubMed]

*Vision Neuroscience*, 2 (3), 297–313. [CrossRef]

*Vision Research*, 31 (9), 1457–1488. [CrossRef] [PubMed]

*Vision Research*, 36 (11), 1615–1631. [CrossRef] [PubMed]

*Vision Research*, 44 (6), 541–556. [CrossRef] [PubMed]

*Journal of the Optical Society of America A*, 29, 1313–1345. [CrossRef]

*Society for Neuroscience*( Washington, DC), Program No. 799.07. 2011. Online.

*Vision Research*, 38, 439–446. [CrossRef] [PubMed]