A two-stage model of orientation integration for Battenberg-modulated micropatterns

Abstract

Abstract

Abstract:

Abstract  The visual system pools information from local samples to calculate textural properties. We used a novel stimulus to investigate how signals are combined to improve estimates of global orientation. Stimuli were 29 × 29 element arrays of 4 c/deg log Gabors, spaced 1° apart. A proportion of these elements had a coherent orientation (horizontal/vertical) with the remainder assigned random orientations. The observer's task was to identify the global orientation. The spatial configuration of the signal was modulated by a checkerboard pattern of square checks containing potential signal elements. The other locations contained either randomly oriented elements (“noise check”) or were blank (“blank check”). The distribution of signal elements was manipulated by varying the size and location of the checks within a fixed-diameter stimulus. An ideal detector would only pool responses from potential signal elements. Humans did this for medium check sizes and for large check sizes when a signal was presented in the fovea. For small check sizes, however, the pooling occurred indiscriminately over relevant and irrelevant locations. For these check sizes, thresholds for the noise check and blank check conditions were similar, suggesting that the limiting noise is not induced by the response to the noise elements. The results are described by a model that filters the stimulus at the potential target orientations and then combines the signals over space in two stages. The first is a mandatory integration of local signals over a fixed area, limited by internal noise at each location. The second is a task-dependent combination of the outputs from the first stage.

Introduction

Combining orientation signals over space

The perception of coherent textures requires the integration of orientation signals over space. The definition of regions in an image that “belong” to the same texture is a necessary intermediate step to higher-level processes, such as finding boundaries between different textures (Marr, 1982; Landy & Graham, 2004). This study focuses on examining the strategies used for choosing which local samples to combine over space to calculate a global orientation estimate from a stimulus (which we shall call the “pooling strategy”) rather than focusing on addressing the process by which the individual local signals are combined (which we shall call the “combination process” as investigated previously by Dakin & Watt, 1997; Jones, Anderson, & Murphy, 2003; Webb, Ledgeway, & McGraw, 2010; Husk, Huang, & Hess, 2012). Our observers are tasked with making global orientation judgments for displays containing orientation micropatterns with various orientations. By manipulating the spatial layout of these micropatterns, it is possible to distinguish between some of the different spatial-pooling strategies that have been proposed previously. 

Signal-combination processes

Effects of spatial configuration aside, the combination process by which the visual system calculates a global orientation from an array of local orientations has been found to depend on the task set to the observer. Similar dependencies have also been reported in studies that investigated the integration of local motion signals. When the observer is required to distinguish between stimuli with weak horizontal or vertical orientation coherence (i.e., with a large difference between the two target orientations), observers filter the image at the two potential target orientations and then choose the orientation of the more strongly activated filter (Husk et al., 2012). A winner-takes-all process similar to this has been found in analogous motion studies performed in monkeys (Salzman & Newsome, 1994). 

Under conditions in which finer judgments of the global orientation of a texture need to be made, the observer calculates the vector-average of the local orientations (Dakin & Watt, 1997; Webb et al., 2010). Similar changes in the combination process used by observers based on the difference between the discriminated orientations have been demonstrated in the motion domain (Nichols & Newsome, 2002; Webb, Ledgeway, & McGraw, 2007). The large orientation differences used in the experiments reported here would be expected to cause the observer to max over filter outputs (the design of this study is similar to that of Husk et al., 2012). For our purposes however, it is not necessary to assume that the observer makes use of a particular combination process. Models that use a vector-averaging combination process produce very similar predictions to those made by the filter-maxing model presented in the body of this paper (see Appendix C). 

Pooling strategies and summation effects

Most signal-combination processes would predict an improvement in performance for detecting weak signals as the number of samples increases. Provided that the noise affecting each sample is at least partially independent, the limiting effect of the noise on performance can be reduced by exploiting the information from multiple samples. Pooling over additional samples in this manner will improve performance regardless of whether the observer is filter-maxing or vector-averaging. There are various possible strategies for pooling signals over space, which make different predictions for how performance should improve with the availability of additional signal samples. Previous work in which the number of samples available for combination is varied have reported conflicting results. Dakin (2001) found a completely flexible combination with respect to signal location over a proportion of the samples in the display. This was presented as an “information limit” for orientation integration. Other studies have shown either improvements reflecting ideal summation under a flexible pooling strategy up to some maximum integration area (Jones et al., 2003) or no benefit from increasing the number of samples whatsoever (Husk et al., 2012). 

This study

The summation effects resulting from increasing the number of samples available for integration are investigated here using psychophysics and computer modeling. The novel “orientation Battenberg” stimuli used allow for manipulation of the spatial arrangement of signal within a stimulus of fixed extent and eccentricity. This reduces the confounding effects of any inhomogeneities in sensitivity for performing the global orientation task (similar to the contrast Battenberg stimuli used by Meese, 2010). Jones et al. (2003) suggested that such an effect might have reduced the level of summation measured in their study. Our results show approximately linear summation over short distances (reflecting the summation of signal against a constant noise floor with an increasing number of samples) followed by a performance improvement consistent with ideal summation over longer distances (reflecting summation of both signal and the variances of per-location noise as the number of sampled locations increases). Previous investigations of the mechanisms underlying the perception of coherent texture have described models featuring an initial local integration stage in which the orientation statistics at each location are estimated, followed by further operations performed over those local estimates (Vorhees & Poggio, 1988; Sagi, 1990; Dakin & Watt, 1997). A two-stage model of this kind is supported by this study, the results of which suggest that observers perform mandatory local integration (affected by internal noise at each location) followed by flexible long-range pooling. 

Methods

Equipment

Stimuli were presented on a gamma-corrected CRT monitor using Psychtoolbox (Brainard, 1997; Kleiner, Brainard, & Pelli, 2007) running under MATLAB. The data collection for these experiments was split between two different equipment setups. The first was an Apple Macbook Pro with an NVIDIA GeForce 9600M graphics card presenting stimuli on a Philips MGD403 monitor; the second was an Apple Macbook Pro with an NVIDIA GeForce 8600M graphics card presenting stimuli on a Compaq monitor. The monitors had refresh rates of 75 Hz and 90 Hz and mean luminances of 77.2 and 26.9 cd/m², respectively. Observers viewed the monitors from a distance of 0.51 m, at this viewing distance giving 6 pixels per cycle for the 4 c/deg stimuli used here. 

Stimuli

Stimuli were 29 × 29 element arrays of 4 c/deg cosine-phase log-Gabor patches (spatial frequency and orientation bandwidths of 1.6 octaves and ±25°, respectively; see Meese, 2010), spaced 1 degree apart in a square grid. Stimuli were displayed at 80% delta-contrast  

 

where L is the stimulus image and L_mean its mean luminance. Each log-Gabor was either a potential signal element or a noise element. Potential signal elements had probability P(coherence) of assuming the target orientation (which was either horizontal or vertical); otherwise, they assumed an orientation drawn at random from a uniform distribution. Therefore, on a trial-by-trial basis, a particular coherence level did not guarantee that a certain number of signal elements would appear in the stimulus, but over many trials, the average (or “expected”) proportion of signal elements in the stimulus would be equal to the coherence level. All noise elements assumed random orientations. The range of potential element orientations was 0° to 179° (orientations were rounded to the nearest degree before stimulus generation). 

Two stimulus types were tested: “full” and “checked.” In the full stimuli, all elements were potential signal elements. For the checked stimuli, the potential signal elements were assigned to locations in the stimulus defined by a checkerboard (a square-wave plaid). This gave a stimulus tiled with square signal and nonsignal regions. Two types of checked stimuli were tested. For the “noise check” condition, the nonsignal regions contained randomly oriented elements. Therefore, each of the checked conditions contained the same total number of elements but approximately half as many signal elements as the full condition (see Table 1). For the “blank check” condition, the nonsignal regions were blank, and so the checked stimuli contained approximately half as many elements as the full stimuli. 

Table 1

 

Numbers and proportions of potential signal elements in the various checked “Battenberg” stimuli used in this study. Notes: The total number of elements in the full Battenberg stimulus was 841. Noise check stimuli always contained 841 elements with the nonsignal elements set to random orientations. Blank check stimuli did not contain any elements other than those that were potential signal elements.

Check size	Modulator phase: ϕ = 90°		Modulator phase: ϕ = 270°
	# signal elements	Proportion	# signal elements	Proportion
1 × 1	421	50.1%	420	49.9%
3 × 3	420	49.9%	421	50.1%
5 × 5	420	49.9%	421	50.1%
9 × 9	445	52.9%	396	47.1%
15 × 15	421	50.1%	420	49.9%

The spatial arrangement of the signal regions in the stimulus was manipulated by adjusting the frequency and the phase of the square-wave plaid modulator that defined the checkerboard. Decreasing or increasing the frequency made the signal regions larger or smaller, respectively, and this was used to create the different check sizes. These gave stimuli tiled with 1-, 3-, 5-, 9-, and 15-element square signal regions (i.e., the largest had 15 × 15 element “checks”). The phase of the modulation was also manipulated to test stimuli in both the cosine (ϕ = 90°) and anticosine (ϕ = 270°) phases. Thus, in the ϕ = 90° condition, the dispersion of signal was such that it was included in the central part of the display whereas in the ϕ = 270° condition there was no signal in the central region. Miniature example stimuli are shown in Figure 1. Full-size examples of each of the stimuli used in the experiments are available as Supplementary material (Figures S1–S21). 

Figure 1

 

Example of the stimulus design used in these experiments. The stimuli shown are 7 × 7 element arrays (smaller than the 29 × 29 arrays used in our experiments) with a 3 × 3 check size. The figure includes ϕ = 90° and 270° check versions of the noise check and blank check stimuli, each shown at 100% coherence.

Procedures

A blocked single-interval identification task was performed to find the orientation identification threshold for each check size, phase (ϕ = 90° vs. 270°), and checked Battenberg type (noise check vs. blank check). Thresholds were tracked using a pair of three-down one-up staircases (maximum 120 trials or 12 reversals), one for horizontal and the other for vertical signal trials. The staircases for the two signal orientations were interleaved randomly. Once the staircase for one orientation had terminated, dummy trials (in which no data were recorded) were still presented with that orientation until the staircase for the other orientation terminated. Staircases started at a high signal level to inform the observers of what stimulus to expect in each block. Stimuli were presented for 250 ms. Stimulus onset was accompanied by a beep. Observers maintained continuous central fixation with the aid of a black fixation dot that was shown between trials. The observers pressed a key on a keyboard to indicate whether the stimulus contained either “horizontal” or “vertical” coherence. The response was followed by a feedback beep that indicated whether it was correct or incorrect and then a 300-ms pause before the presentation of the next stimulus. Each observer performed four repetitions for each combination of check size (full, 1, 3, 5, 9, and 15), Battenberg modulator phase (ϕ = 90° or 270°), and Battenberg type (noise check or blank check). As the full stimulus was identical regardless of Battenberg type or modulator phase, each observer collected four times as much data for this condition (16 repetitions). These were averaged to give a single threshold per observer. 

Observers

Seven observers were used. Five were experienced psychophysical observers (ASB, DHB, JSH, RJS, and SAW), including two of the authors (ASB and JSH). Two were naïve undergraduates (LFE and VRP). All had either normal or corrected-to-normal vision. Observers DHB, RJS, and SAW were tested on the first equipment setup described above. Observers JSH, LFE, and VRP were tested on the second equipment setup. Observer ASB was tested on the first setup for the noise check and on the second setup for the blank check conditions. The study was conducted in accordance with the Declaration of Helsinki. 

Analysis

Data from the horizontal and vertical staircases were combined into a single psychometric function for each repetition, condition, and observer. This was then fitted by a cumulative normal function using Palamedes (Prins & Kingdom, 2009). The fitted function gave the probability of responding “horizontal” to either a vertical stimulus (plotted as negative coherence) or a horizontal stimulus (plotted as positive coherence). The coherence level at which the function reached P(“Horizontal”) = 0.5 gave the bias for the observer categorizing a stimulus as horizontal rather than vertical (the distributions of the biases found for each observer are presented in Appendix A), and the orientation identification threshold could be calculated as the difference between the coherence level at this point and that at which P(“Horizontal”) = 0.75. 

Results

Noise check

The orientation identification thresholds averaged across the seven observers are shown in Figure 2. The three rows present the data plotted as the expected number of signal elements in the stimulus at threshold (Figure 2a and b); the threshold probability of the elements in the signal region assuming the target orientation (Figure 2c and d), which is equivalent to the expected proportion of signal elements within the signal regions at threshold (hereafter termed the “coherence threshold”); and the expected proportion of signal elements across the entire stimulus at threshold (Figure 2e and f). In subsequent figures, the metric shown in Figure 2c and d is used with this coherence threshold for each checked stimulus plotted as a multiple relative to that for the full condition (which had a coherence threshold of approximately 10%). 

Figure 2

 

Thresholds from the identification task (averaged over seven observers) shown in three different ways. The results are plotted here as the expected number of signal elements in the stimulus at threshold (a–b), the expected percentage of signal elements in the potential signal regions at threshold (c–d), and the expected percentage of signal elements in the entire stimulus at threshold (e–f). The metric used in other results figures in this paper is that in c and d with these “coherence thresholds” expressed as multiples relative to that of the full (F) stimulus. Error bars show ±1 standard error here and in all other graphs.

Figure 2c presents the coherence thresholds for the noise check condition. For smaller check sizes (1 to 3), the thresholds for the checked stimuli were approximately double that for the full stimulus. This means that around the same total number of signal elements across the entire stimulus were required to reach the threshold performance level in each case (see Figure 2a). For medium sizes (5 to 9), the threshold elevation decreased to a factor of Display Formula . This is consistent with a strategy that uses information from potential signal regions but ignores irrelevant (noise-only) regions as the noise resulting from the combination of multiple equally noisy samples is proportional to the square root of the number of samples combined (a simple “sum of variances” rule). As the stimuli were blocked by check size and modulator phase, the observer could predict, in each trial, which areas of the display were potential signal regions and which would contain only irrelevant noise. 

One of the purposes of the Battenberg stimulus design is to reduce the effect of visual field inhomogeneities in sensitivity on the measurement of area summation (Meese & Summers, 2007; Meese, 2010). For stimuli with large check sizes relative to their extent, however, these effects will return. For the largest (15) check size in Figure 2c, performance diverged dependent on whether the stimulus was in the ϕ = 90° (foveal signal, peripheral noise) or ϕ = 270° (foveal noise, peripheral signal) phase. Coherence thresholds were almost as low for the ϕ = 90° stimulus as they were for the full stimulus. This means that the observers required around half as many signal elements across the entire stimulus when most of those elements were presented in the center of the display (Figure 2a). For the ϕ = 270° stimulus, coherence thresholds were approximately double that of the full stimulus. Therefore observers required the same number of signal elements in the largest ϕ = 270° stimulus as they did in the full stimulus (because the full stimulus has approximately twice the checked stimulus's signal area). This behavior would be consistent either with a relative insensitivity for this task in the periphery or a failure in segregating the noise present in the center of the display (this is discussed further below). 

Blank check

Figure 2d shows the coherence thresholds for the blank check condition. For the smaller check sizes in the blank check condition, coherence thresholds increase to approximately double that for the full stimulus in a similar manner to that seen in the noise check condition. This is unexpected as the predicted coherence threshold for this condition based on the stimulus properties alone would be a factor of Display Formula above that for the full condition (there are no noise-only elements, so the threshold would be proportional to the square root of the number of elements in the display: the sum of variances rule again). These results suggest that the performance-limiting noise for the noise check task does not arise from the responses to the interstitial noise elements; otherwise, the removal of these elements in the blank check condition would result in a decrease in threshold. On the other hand, however, the noise cannot be “late” and constant across conditions as this would predict the same performance level for all of the checked stimuli. Instead, these results suggest that observers are mandatorily integrating internal noise from blank display regions (or are limited by a noise source that is proportional to the monitored area) for the smaller check sizes but are able to exclude this noise for the larger check sizes. 

Figure 3

 

Coherence thresholds for the checked stimuli expressed as multiples of the full stimulus threshold, plotted with predictions from the SA (a–b), SI (c–d), and TS (e–f) models. Each row shows the same averaged data replotted from Figure 2c and d. The only fitted parameter was k, the size of the pooling region in the TS model (3 × 3). This fitting was performed by hand (Appendix B). RMS errors between the model predictions and the data are shown in dB.

Figure 4

 

Coherence thresholds plotted with predictions from the TN model fitted to the blank check data (a–b) and from the TA model with the best-fitting parameters for the noise check data (TA₁, c–d) and the best-fitting parameters for the blank check data (TA₂, e–f). Each row shows the same averaged data replotted from Figure 2c and d. The fitted parameters were the size of the pooling region (k), the standard deviation of the internal noise (σ_int), and the size of the integration aperture (a).

The similarities in the noise check and blank check data shown in Figure 2c and d can be confirmed by looking ahead to Figure 5, which compares the thresholds from these conditions directly by showing the amount of threshold elevation caused by the presence of the noise checks. Up to the largest (15) check size, thresholds for the ϕ = 90° stimuli are similar (threshold elevation factor of approximately one), suggesting that the pooling of samples (and segregation of noise) in these two conditions is similar. For the ϕ = 270° stimuli, however, threshold elevation increases with check size. Performance for the largest (15) ϕ = 270° stimulus in the blank check condition is a factor of Display Formula better than that in the noise check condition, meaning that at least part of the deficit for detecting that stimulus was due to inefficient pooling of noise in the noise check condition rather than a relative insensitivity for performing the task in the periphery (e.g., it may be more difficult for the observer to keep track of the signal and nonsignal regions in the ϕ = 270° noise check condition). 

Figure 5

 

Threshold elevation introduced by the noise check elements, calculated from the data in Figure 2c and d as the thresholds from the noise check condition divided by the thresholds from the blank check condition. The predicted threshold elevations from each of the models shown in Figures 3 and 4 are also plotted here. Only the SA and TS models predict any systematic elevation effect, and neither of those match the pattern seen in the data.

Modeling

Monte Carlo simulations

A set of models was developed to investigate the pooling strategy used by the observers. The initial aim of the modeling was to explain the surprising equivalence between the smallest check sizes in the noise check and blank check conditions (see Figure 2c and d), which would not be expected if the source of the limiting noise was in the responses to the individual micropattern elements (as the additional randomly oriented elements in the noise check condition would cause thresholds for those stimuli to be higher). Model predictions were obtained using stochastic Monte Carlo methods. A set of model observers was developed with different pooling strategies in MATLAB and run through 2,000 simulated trials per stimulus level of a “method of constant stimuli” version of the experiment. The coherence thresholds for each model observer were calculated using the simulated data and expressed as multiples of the full stimulus threshold, allowing them to be compared to the human results without the need for fitting. Three fitted parameters (the size of the local pooling regions k, the standard deviation of the internal noise σ_int, and the size of the global pooling region a) were then added in order to develop a model that provided a close account of the data. These parameters were each fitted by hand (see Appendix B). 

The filter-maxing combination processes

Several combination processes for the calculation of global orientation from individual local samples have been suggested previously. Here we implement the strongest candidate for this task, in which the observer selects the orientation of the most strongly activated oriented filter (after Jones et al., 2003; Husk et al., 2012). This choice is not crucial to our findings, however, and a set of models developed using a vector-averaging combination process made similar predictions (Appendix C). In our filter-maxing model, the stimulus is first filtered at the two potential target orientations. The filter elements are a pair of log-Gabor elements with the same tuning properties as those used in the generation of the stimuli (spatial frequency bandwidth of 1.6 octaves and orientation bandwidth of ±25°). These bandwidths are typical of those used previously in the literature to model simple cell responses and compatible with those found in neurophysiological investigations (De Valois & De Valois, 1990; Meese, 2010). As our filter elements were identical to the target elements in our stimuli, they also behaved as the ideal detector for those elements. The filter outputs are rectified and passed to the pooling stage. In the pooling stage, the outputs can be weighted according to their expected signal-to-noise ratio (dependent on the pooling strategy in operation, see below). The weighted filter responses are then summed over the image for each orientation, and these values are compared to each other. The model observer picks the orientation with the greater filter response. 

Pooling strategies

The simplest pooling strategy considered is the “sum all” (SA) model, in which the observer combines information from every element in the stimulus with an equal weighting regardless of whether it is a noise or potential signal element. The model with this strategy predicts that within each condition (noise check and blank check), there should be no effect of check size or modulator phase. The predicted coherence threshold for the noise check stimuli is approximately twice that for the full stimulus (Figure 3a), which is equivalent to requiring the same threshold proportion of coherent signal elements across the whole stimulus. For the blank check stimuli (Figure 3b), the predicted threshold elevation is a factor of Display Formula as there are no interstitial noise elements to limit performance in that condition. These predictions capture the performance for the small check sizes (1 to 3) in the noise check condition, the medium check sizes (5 to 9) in the blank check condition, and the largest ϕ = 270° size (15) in both the noise check and blank check conditions (see Figure 3a and b). The predictions fail, however, to describe the general form of either set of results, which both feature a transition from greater to lesser summation as the check size increases. 

The “sum ideally” (SI) strategy involves the combination of orientation information only from potential signal elements. This is equivalent to weighting the elements according to their expected signal-to-noise ratio as the responses to the elements in the noise regions provide zero signal. Models with this strategy predict threshold elevation of a factor of approximately Display Formula for both of the checked conditions (Figure 3c and d), which is identical to the prediction for the blank check stimuli with the SA model. For this reason, the results in the blank check condition, which were well described by SA models, are fit just as well by the SI models. In addition, the SI models predict the Display Formula summation for the medium check sizes in the noise check condition. However, this model also fails to capture the general form of the human data for either stimulus type. 

Two-stage hybrid models

The fact that the simpler candidate models featuring the two different pooling strategies (SA and SI) each predicted performance for different subsets of the results suggested that a more complete account could be provided by a model that combined their two behaviors. In the “two-stage (TS) hybrid models,” mandatory local combination over a k × k region (behaving like the SA model) is followed by flexible pooling of the outputs from those regions weighted by their expected signal-to-noise ratios, which were applied as a template (as in the SI model). The size of the local pooling region (k) was the only fitted parameter in this model. The best fit to the noise check data was provided by a k of 3 although the fit provided by a k of 2 was only marginally worse (see Appendix B, Figure A2). In the noise check condition, this model predicts an initial doubling of the coherence threshold for the small check sizes, followed by an improvement in performance to approach a factor of Display Formula for the medium and large check sizes (see Figure 3e). This captures the performance for all but the largest (15) check sizes. In the blank check condition (Figure 3f), the predictions are once again the same as for models with the SA and SI strategies. 

Internal noise

The SA, SI, and TS models all make predictions that are identical to each other for the blank check condition. This is because there are no interstitial noise elements in that condition that can be inappropriately pooled to elevate the coherence threshold. The results, however, show a doubling of threshold for the small check sizes that is the same in the noise check and blank check conditions. This is a larger performance deficit than can be accounted for by any of the three models (which all predict a Display Formula threshold increase for the blank check condition). A two-stage hybrid model, featuring additive Gaussian internal noise at each location (TN), does, however, predict similar performance for the blank check conditions as for the noise check conditions (see Figure 4a and b). The internal noise in the model is added after the rectification stage (meaning that the noisy local outputs can be negative). Because of the mandatory combination rule at the first pooling stage, this early noise model is equivalent to a model in which performance is limited by noise affecting the output of the first pooling stage. 

The standard deviation of the internal noise (σ_int) is now an additional parameter in the model. It is expressed as a percentage of the summed element-wise filter response to a matched log-Gabor element (i.e., relative to the maximum local filter output). As σ_int is increased from zero, it initially has the effect of increasing thresholds for the 1 × 1 blank check condition to bring them in line with those from the noise check condition. Once these have become equivalent, higher values of σ_int serve mainly to increase or decrease the sensitivity for all conditions. Calculating the relative coherence thresholds from the output of the model therefore largely factors out the effects of this model parameter once σ_int is sufficiently high (see Appendix B). Figure 4a and b shows the TN model prediction with the best-fitting k and σ_int values for the blank check data (note that the best-fitting k for the blank check data is 2, whereas k = 3 provided a marginally better fit to the noise check data as shown in Figure A2). For both the noise check and the blank check conditions, the TN model predicts an initial twofold summation for the 1 × 1 check size, which then decreases to approximately Display Formula as the check size increases. The predictions from this model account for the average human thresholds in the noise check condition for all but the largest check size (Figure 4a). In the blank check condition (Figure 4b) thresholds are lower than those predicted by the model for the medium (9) and largest (15) ϕ = 90° condition, but for all other conditions, the human behavior is captured by the model. 

Maximum integration aperture

In the TN model, there are several reasons why coherence thresholds could be elevated when signal is presented only in the periphery (compare ϕ = 90° and 270° thresholds for the largest check size stimulus in Figure 4). The simplest would be if the observer were only able to pool information from elements in the center of the display. This was tested using a “two-stage hybrid model with internal noise and maximum integration aperture” (TA), in which the model observer only had access to information from elements that were within a central a × a element square aperture (equivalent to a degrees or 4a carrier cycles). This would also be equivalent to a model that featured a step-edge decline in sensitivity at this eccentricity. More complex accounts of the effect of visual field eccentricity are not explored here as our stimuli were designed to factor out these effects when possible, meaning that any explanation we might provide would not be well constrained by our data. For example, further summation may be achieved by probability summation between multiple integration apertures at different locations across the visual field (resulting in a model similar to that proposed for the area summation of contrast by Baker & Meese, 2011). 

In order to show which features of our results the TA model could and could not account for, two separate fits were performed to determine the optimal set of parameters for the noise check (TA₁) and the blank check (TA₂) data. The TA₁ prediction is shown in Figure 4c and d. The best-fitting integration aperture size was 18 × 18 degrees (compared to other square apertures with integer dimensions, see Appendix B). The TA₁ model prediction is similar to that made by the TN model (for the same kernel size, see Appendix B) for all conditions except for the largest check size (15). For the 15 × 15 check stimuli, the predicted thresholds for the ϕ = 90° stimulus are reduced (when expressed relative to the full condition threshold), and the thresholds for the ϕ = 270° stimulus are elevated. This model prediction provides a good fit to the noise check data (RMSe = 0.54 dB) and for the smaller check sizes (1–3) in the blank check condition. For the larger check sizes in the blank check condition, however, the TA₁ model prediction systematically underestimates the sensitivity of the human observers (resulting in a relatively large RMSe of 1.78 dB). 

Fitting the aperture model to the blank check condition produces the TA₂ prediction shown in Figure 4e and f. As in the TN model, the best-fitting k for the blank check condition was 2 (as opposed to the value found from the TA₁ fit to the noise check data, which favored a k of 3). The best-fitting aperture size (a) was 19 × 19 degrees (1° wider than the size found by fitting to the noise check data). As would be expected, this version of the model provides an inferior fit to the noise check data compared to the TA₁ prediction (1.16 dB vs. 0.54 dB), underestimating both the threshold for the 3 × 3 check condition and the amount of separation between the two modulator phases at the largest check size. In the blank check condition, although the quality of the fit is improved from the TA₁ prediction (1.12 vs. 1.78 dB), sensitivity is still underestimated for the larger (9–15) check sizes. This indicates that there is no combination of parameters for this model architecture that can capture the performance for those conditions. 

Residual effects of check condition and modulator phase

Figure 5 replots the data from Figure 2c and d, showing directly the threshold elevation effect that the interstitial noise elements in the noise check condition have. Also shown in Figure 5 are the threshold elevation predictions from each of the six model variants presented earlier. All models except SA and TS predict no threshold elevation from the noise checks in either modulator phase. For the ϕ = 90° modulator phase the data agree with this prediction, with the ratio between the noise check and blank check thresholds remaining close to one for all check sizes. For the ϕ = 270° modulator phase, there is no threshold elevation introduced by the noise checks at the smallest (1) check size, but as the check size increases, the noise checks have the effect of raising thresholds relative to the blank check condition up to a maximum factor of Display Formula for the largest (15) check size. Only the SA and TS models have their performance limited by the interstitial randomly oriented elements in the noise check stimuli and therefore predict threshold elevation in the noise check condition. The pattern of elevation predicted, however, does not match that seen in the ϕ = 270° data. The key difference is that the data show a threshold elevation effect that both increases with check size and is dependent on the modulator phase whereas none of the models investigated here make this prediction. One possible explanation for this effect would be an increase in the size of the mandatory local integration region with eccentricity (linking this to the explanation of crowding provided by Parkes, Lund, Angelucci, Solomon, & Morgan, 2001). 

Discussion

Orientation integration is a noisy two-stage process

The results of this study suggest that the combination of orientation information over space is a noisy two-stage process (Figure 6). The key results reported here can be accounted for by a model that performs mandatory local integration affected by internal noise at each location followed by flexible pooling over the outputs from those regions (for the vector-averaging version of the model, these would seem equivalent to the involuntary and voluntary averaging discussed by Dakin, Bex, Cass, & Watt, 2009). This account is in agreement with previous studies that have found lower thresholds for stimuli with a greater signal area (Dakin, 2001; Jones et al., 2003). The effect found for the arrangement of the elements in the display contradicts the flexibility attributed to the pooling of local samples by Dakin (2001) although it is possible that the differences in terms of the task set to the observer (signal in noise here vs. fine discrimination in Dakin) would mean that they did not investigate the same signal-combination process (as discussed in the Introduction to this paper). The close spacing of the elements in our stimuli and their extension into the periphery would lead us to expect the individual elements to be crowded by each other. Under the account provided by Dakin et al. (2009), it is suggested that crowding only limits performance in tasks in which there is little orientation variability (e.g., a fine discrimination task) whereas for a signal-in-noise task, such as ours, performance should be limited by the number of pooled samples. If crowding were a limit on attentional resolution (Strasburger, 2005), however, then we would expect it to affect the ability of our observers to segregate potential signal and noise elements, possibly explaining the elevated thresholds in the ϕ = 270° noise check condition. In addition, it is not possible from our study to determine whether the observers were pooling all of the available samples in the stimulus or if they were making their decisions based on a subset of the local samples in the display (as was found by Dakin, 2001). 

Figure 6

 

Diagram of the TN model. This diagram shows how the “vertical” response to an example stimulus (which includes blank spaces in stimulus region n) is determined by filtering with a vertical filter element, mandatory local summation, and then global summation of the local outputs (from stimulus regions 1 to n) weighted by the expected signal-to-noise ratio at each location (w). For ease of presentation, the local summation region shown is 1 × 4 elements (our results suggest 3 × 3), and only the responses to the first and last stimulus regions are presented. The “horizontal” response would be calculated in an identical manner except with a horizontal filter element at the convolution stage. N_early, N_mid, and N_late show three possible locations for the limiting internal noise.

It is not entirely clear how the results presented here can be reconciled with those of Husk et al. (2012), who found no summation with increasing signal area for similar stimuli. The main difference between the stimuli used in the two studies is that this study used the Battenberg summation paradigm whereas Husk et al. increased the signal area of their stimuli by increasing diameter. It is possible that a combination of decreasing sensitivity for the local orientation-discrimination task and increasing the mandatory summation region size with eccentricity might flatten the threshold versus area functions. This question shall be addressed in future work, which will combine the Battenberg stimulus paradigm used here with a conventional area-summation design, comparing stimuli of different diameters. Performing this experiment at a variety of spatial frequencies will also allow us to determine whether the size of the mandatory integration region is linked to the scale of the elements that are being pooled. Previous studies that have investigated the processing of visual texture would lead us to predict this to be the case (Kingdom, Keeble, & Moulden, 1995; Kingdom & Keeble, 1999); however, if the mandatory pooling were related to crowding, we would expect its extent to be independent of the scale of the stimulus (Levi, Hariharan, & Klein, 2002). 

It is noteworthy that the results presented in Figure 2c and d bear a resemblance to those found in the contrast Battenberg study by Meese (2010) with short-range linear summation followed by long-range square-law summation. The explanation for the square-law summation differs, however, between the two studies. For the orientation result presented here, this nonlinear summation is explained by a flexible pooling strategy that segregates out the input from the nonsignal regions of the display whereas, in the contrast study by Meese, the output from those regions was included with that from the signal regions in the global combination process, and the nonlinear summation was accounted for by a square-law transducer. 

The nature of the limiting internal noise

Figure 6 shows three possible locations for the limiting noise in the modeling for this study (N_early, N_mid, or N_late). The similarity of the results from the noise check and blank check conditions indicate that the limiting noise is not driven by the response to individual elements. Instead, they suggest that a level of internal noise is pooled that is proportional to the number of locations being monitored, which includes blank locations that are being integrated at the first stage of the model. Jones et al. (2003) included “late noise” when modeling data from an orientation coherence experiment that used filtered noise as stimuli. In the modeling for that study, the noise was constant for different signal areas (this would be N_late in Figure 6). A model based on dominant noise at N_late would make the same prediction as the SA model shown in Figure 3a for both the noise check and blank check conditions. Such a model would not explain the results presented here as the improvement in performance seen for the medium check size stimuli requires that the observer is able to segregate out the limiting noise in irrelevant regions from the second combination stage. If dominant noise is contributed from each monitored location, then this could be performed by weighting the local outputs according to a template w as shown in Figure 6. 

In the internal noise model developed in this study (see Figure 4), the noise was implemented at each pooled location after the initial filtering stage (N_early); however, due to the mandatory local combination at the first combination stage, this is equivalent to adding noise to the combined local outputs (N_mid). From the results of this study, it is not possible to determine whether the limiting noise should be N_early or N_mid in Figure 6; however, N_mid seems more plausible as the level of early noise needed to exceed the external noise introduced by the randomly oriented elements in the noise check stimuli would be very high. In the best-fitting prediction to the noise check data (TA₁), the standard deviation of the early noise would be 85% of the mean response of the local detector to its ideal stimulus; in the best-fitting prediction to the blank check data (TA₂) prediction, it would be even higher (147%). There is also the possibility that the limiting internal noise could be multiplicative rather than the additive noise implemented in the modeling here. We shall address this question through the use of an equivalent noise paradigm in future work. 

Supplementary Materials

pdf S1 

pdf S2 

Acknowledgments

This work was supported by a grant from the Natural Sciences and Engineering Research Council (Canada) awarded to Robert Hess (#46528-11) and a grant from the Engineering and Physical Sciences Research Council (UK) awarded to Tim Meese and Mark Georgeson (#EP/H000038/1). The authors would like to thank three anonymous reviewers for their helpful criticisms and suggestions. 

Commercial relationships: none. 

Corresponding author: Alex S. Baldwin. 

Email: [e-mail]. 

Address: School of Life and Health Sciences, Aston University, Birmingham, UK. 

Appendix A: Identification task bias distributions

Figure A1 shows the distribution of the response biases found in the analysis of the data from the identification task. Across all observers and conditions, the mean bias was −4.66% with a standard deviation of 7.49%. This means that, on average, the observers were as likely to respond “horizontal” as they were to respond “vertical” to a stimulus with 4.66% vertical coherence. Looking at the data from the two main conditions (noise check and blank check) for each observer individually, most observers showed a significant response bias as decided by a one-sample t test (Table A1). For all observers except one, the direction of any significant bias was consistent across the two conditions. For ASB, the bias changed direction between the noise check and blank check condition, possibly due to this observer being tested on these conditions in two different labs several weeks apart. Aside from ASB, the observers tested in each lab showed the same mixture of biases: from horizontal (LFE and RJS) through mostly unbiased (SAW and VRP) to vertical (DHB and JSH). 

Figure A1

 

Distributions of response biases from each observer calculated as the level at which the fitted psychometric functions were at 50%. Bias values were pooled across all repetitions in all of the subconditions tested in each of the two main conditions: noise check versus blank check (shown in red and blue, respectively). Statistical properties of the distributions are reported in Table A1.

Table A1

 

Statistical properties of the bias distributions for each observer and major condition (noise check vs. blank check). Notes: A one-sample t test was performed for each in order to determine whether any bias present was significant.

Observer	Condition	Mean	SD	df	t	p	Sig.
ASB	Noise check	5.52%	3.63%	47	10.54	<0.001	**
	Blank check	−12.77%	7.60%	47	−11.64	<0.001	**
DHB	Noise check	1.54%	4.84%	47	2.20	0.033	*
	Blank check	3.29%	4.59%	47	4.96	<0.001	**
JSH	Noise check	6.32%	4.15%	47	10.54	<0.001	**
	Blank check	5.89%	6.55%	47	6.23	<0.001	**
LFE	Noise check	−8.71%	4.72%	47	−12.77	<0.001	**
	Blank check	−12.77%	7.60%	47	−2.53	0.015	*
RJS	Noise check	−3.90%	8.56%	47	−3.15	0.003	*
	Blank check	−5.42%	6.24%	47	−6.01	<0.001	**
SAW	Noise check	−1.96%	6.08%	47	−2.24	0.030	*
	Blank check	0.02%	5.49%	47	0.03	0.976
VRP	Noise check	0.97%	7.88%	47	0.85	0.398
	Blank check	−0.16%	6.19%	47	−0.18	0.859

Appendix B: Determination of parameter values

Predictions were generated from the TS model with a range of different kernel sizes (k) for the local mandatory integration stage (1 × 1 to 7 × 7 in integer steps). The model curves produced for the noise check condition and the RMS errors between those curves and the data are shown in Figure A2a and b. For the blank check condition, the model curves were identical for any kernel size, producing the same predictions as the SA and SI models. The best-fitting value for the k parameter is 3. Smaller k values predict too little summation for the small check sizes whereas larger k values predict too much summation for the larger check sizes. 

Figure A2

 

Panel (a) shows model predictions for the two-stage hybrid model with a range of different kernel sizes (k) against the data from the noise check condition (replotted from Figure 2c). Panel (b) shows the RMS error between the model predictions and the data for each value of k.

The predicted blank check condition thresholds generated by the TN model for a kernel size of 2 and a range of different internal noise levels (σ_int) are shown in Figure A3. Predictions with a kernel size of 3 are shown in Figure A4. As the internal noise level increases, the prediction for the blank check condition becomes similar to that for the noise check condition. In conditions of high internal noise, the blank check data are best fit by a kernel size (k) of 2. 

Figure A3

 

Panel (a) shows model predictions for the noisy two-stage hybrid model with a kernel size (k) of 2 and a range of different internal noise levels (σ_int) against the data from the blank check condition (replotted from Figure 2d). Panel (b) shows the RMS error between the model predictions and the data for each value of σ_int.

Figure A4

 

Panel (a) shows model predictions for the noisy two-stage hybrid model with a kernel size (k) of 3 and a range of different internal noise levels (σ_int) against the data from the blank check condition (replotted from Figure 2d). Panel (b) shows the RMS error between the model predictions and the data for each value of σ_int.

Contour plots of the RMS error between the noise check data and predictions from the TA model are shown in Figure A5. The parameters that produce the global minimum across the two surfaces are used to generate the TA₁ prediction shown in Figure 4c and d. The RMS errors between the blank check data and predictions from the TA model are shown in Figure A6. The parameters that produce the global minimum from these fits are used to generate the TA₂ prediction shown in Figure 4e and f. 

Figure A5

 

RMS error between the predictions from the TA model and the data for different combinations of the kernel size (k), internal noise (σ_int), and aperture size (a) parameters. The minimum on each plot is indicated by a “+.” The minimum across the two graphs (RMSe = 0.54 dB when k = 3, σ_int = 352, and a = 18) is the TA₁ model prediction shown in Figure 4c.

Figure A6

 

RMS error between the predictions from the TA model and the data for different combinations of the kernel size (k), internal noise (σ_int), and aperture size (a) parameters. The minimum on each plot is indicated by a “+.” The minimum across the two graphs (RMSe = 1.12 dB when k = 2, σ_int = 608, and a = 19) is the TA₂ model prediction shown in Figure 4f.

Appendix C: Vector-averaging model

In the vector-averaging model (Dakin & Watt, 1997), each pooled element is represented as a vector with magnitude m_i and orientation θ_i. It is assumed that the observer is able to extract the orientation of each element (in the model, this is implemented by taking the orientations directly from the stimulus-generation procedure). These are then combined using vector-averaging to get the average orientation  

 

 

Note that the local orientations are doubled before averaging and that the output of the vector-averaging operation is halved. This wraps the orientations at 180° (rather than at 360°, which is the usual limit) because each element in the display is symmetrical across its major and minor axes (a 90° element is identical to a 270° element). 

When every element is weighted equally, all elements are represented by unit vectors (m_x,y = 1). In cases in which the elements have different expected signal-to-noise ratios (e.g., the second stage of the two-stage hybrid model below), the magnitudes of the local vectors are each weighted by a template to control the contribution each local vector makes to the calculated average (see the section on pooling strategies). The model then picks the potential target orientation closest to the calculated average orientation. The predictions from vector-averaging models using the different pooling strategies given are shown in Figure A7a through d. These predictions have the same form as those made by the filter-maxing combination process in Figure 3. 

Figure A7

 

Coherence thresholds plotted with predictions from the vector-averaging versions of the SA (a–b), SI (c–d), and HM (e–f) models. Each row shows the same averaged data replotted from Figure 2c and d with the threshold for each check condition expressed as a multiple of that for the full stimulus.

For the hybrid model, the convolution kernel to simulate mandatory local combination is applied to the vector components  

 

 

 

 

 

 

 

 

 

 

These “blurred” components are then combined through vector-averaging.  

 

 

Predictions from this vector-averaging hybrid model are shown in Figure A7e and f. The kernel size (k) was chosen to be the same as that used in the filter-maxing model. A comparison between the predictions made by the vector-averaging two-stage hybrid model and the equivalent filter-maxing combination model (Figure 3e and f) shows them to be very similar. 

The internal noise model adds a sample of zero-mean (μ = 0) Gaussian noise to each element, drawn from a distribution with the requested standard deviation (σ_int)  

  ,

 

  .

 

Predictions from this model and the version of the model with a maximum integration aperture are shown in Figure A8. In each case, the parameters were set to be the same as those used in the filter-maxing models (Figure 4) except for the internal noise parameter (σ_int), which was set to a value that produced a prediction that was close to the one from the equivalent filter-maxing model (determined by fitting the vector-averaging model to the filter-maxing model and choosing the value of σ_int that gave the lowest RMS error). In all cases, the vector-averaging model was able to produce similar behavior to that seen in the filter-maxing model. 

Figure A8

 

Coherence thresholds plotted with predictions from the vector-averaging version of the TN model and from the vector-averaging version of the TA model with the best-fitting aperture and kernel size parameters from the filter-maxing model fits to the noise check (c–d) and blank check (e–f) data.