Human perception of subresolution fineness of dense textures based on image intensity statistics

Abstract

We are surrounded by many textures with fine dense structures, such as human hair and fabrics, whose individual elements are often finer than the spatial resolution limit of the visual system or that of a digitized image. Here we show that human observers have an ability to visually estimate subresolution fineness of those textures. We carried out a psychophysical experiment to show that observers could correctly discriminate differences in the fineness of hair-like dense line textures even when the thinnest line element was much finer than the resolution limit of the eye or that of the display. The physical image analysis of the textures, along with a theoretical analysis based on the central limit theorem, indicates that as the fineness of texture increases and the number of texture elements per resolvable unit increases, the intensity contrast of the texture decreases and the intensity histogram approaches a Gaussian shape. Subsequent psychophysical experiments showed that these image features indeed play critical roles in fineness perception; i.e., lowering the contrast made artificial and natural textures look finer, and this effect was most evident for textures with unimodal Gaussian-like intensity distributions. These findings indicate that the human visual system is able to estimate subresolution texture fineness on the basis of diagnostic image features correlated with subresolution fineness, such as the intensity contrast and the shape of the intensity histogram.

Introduction

Every day, we see objects and photographic images with very fine structures, such as human hair, fluffy fabric, and piles of dust. For a variety of reasons, e.g., the purchase of a comfortable carpet or a lovely kitten, we make judgments about the fineness of materials from their visual appearance. In some cases, the spatial scales of these materials are much finer than the spatial resolution limit of the human visual system (∼1 min arc) or the physical resolution of a digitized image. For instance, when a human hair (typically 0.08 mm in diameter) is viewed at a distance of 1 m, the visual angle (0.24 min) is a quarter of the minimum resolution of observers with 20/20 vision. Nevertheless, one may feel it possible to tell something about the thinness of human hair even from a few meters away. 

It is widely known in the computer graphics community that modeling subpixel fine structures is crucial for properly rendering fine structures. For instance, Figure 1 shows computer-generated hair images, in which light interactions among the strands were simulated with a volume rendering technique (Shinya, Shiraishi, Dobashi, Iwasaki, & Nishita, 2010). In the left image, the strands are drawn as line segments with a width of one pixel. In the right image, on the other hand, they are modeled as cylinders in 3D space with a radius of ∼0.3 pixels. Although the strands were drawn in the rendered image as a subpixel fine structure, they look finer and more realistic than those in the left image. Likewise, modeling subpixel fine structures is crucial for proper rendering of fabric, granular, and other fine materials (Zhao, Jakob, Marschner, & Bala, 2011, 2012; Zhao, Hašan, Ramamoorthi, & Bala, 2013; Yan et al., 2014; Khungurn, Schroeder, Zhao, Bala, & Marschner, 2015; Meng et al., 2015; Müller, Papas, Gross, Jarosz, & Novák, 2016; Zhao, Luan, & Bala, 2016). Such rendering techniques attempt to reproduce characteristic image patterns related to mesoscale/microscale structures that graphics researchers empirically know to be useful for enhancing the visual reality of rendered images. 

Figure 1

 

Computer-generated human hair images. Subpixel image rendering (Shinya, Shiraishi, Dobashi, Iwasaki, & Nishita, 2010) makes the hair texture in the right image (hair width: 0.3 pixels) look more natural and finer than that in the left image (hair width: 1 pixel).

Whereas there are good reasons to believe that the human visual system is sensitive to fine texture structures in the subresolution range, it remains scientifically unexplored whether our visual system veridically perceives subresolution fineness, and if so, how this can be possible. This study addressed these questions by means of psychophysical experiments and image analyses. 

Naturalistic images like those in Figure 1 contain many potential cues to hair fineness, such as loosely scattered hair strands on the side and irregular hair ends. To exclude these cues and focus on image cues included in dense textures, we first investigated human fineness perception with artificial texture images. These images consisted of a multiresolution sequence of one-dimensional (1D) hair-like random textures (Figure 2), made by successively applying low-pass filtering and down-sampling. We then compared our findings on the artificial textures with those on natural textures. 

Figure 2

 

One-dimensional random line textures with a variety of fineness levels used in the experiments at NTT. The value of X is the reciprocal of the line width in terms of the pixel width. In the experiments, each texture was presented in a square (256 × 256 pixels) on a calibrated monitor and viewed at a distance of 43 cm (2 min/pixel) or 344 cm (0.25 min/pixel). The table shows the width of one line per pixel, and those per minute at the two viewing distances, with numbers in red being the widths unresolvable due to the pixel resolution (first line) or due to visual acuity (second and third lines).

One possible strategy for the human vision system to judge subresolution fineness is to utilize image statistics correlated with the optical properties of physical fine structures. Past studies have suggested several statistical image cues that human vision might use for perception of glossiness (Nishida & Shinya, 1998; Adelson, 2001; Fleming, Dror, & Adelson, 2003; Motoyoshi, Nishida, Sharan, & Adelson, 2007), perception of transparency (Fleming & Bülthoff, 2005; Motoyoshi, 2010; Xiao et al., 2014; Kawabe, Maruya, & Nishida, 2015), perception of liquid viscosity (Kawabe, Maruya, Fleming, & Nishida, 2015; van Assen & Fleming, 2016), the gist of a natural scene (Oliva & Torralba, 2007), and so on. We suspect that fineness perception of dense textures with subresolution fine elements might also rely on diagnostic image statistics. Moreover, as discussed below, we found that certain properties of the luminance histogram are promising candidates. 

It should be noted that our study concerns fineness at the scale of the image. That is, we studied human judgment of the element size in terms of the visual angle, not in terms of the actual physical size. In addition, our study concerns the ability of humans to judge the subresolution fineness of elements that make up a dense texture. Moreover, because we fixed the texture parameters other than the spatial scale in the experiments, we can rephrase this as the ability to judge the density of texture elements at subresolution scale. Just like spatial frequency and wavelength are inseparable for sinusoidal gratings, element fineness and density are not separately considered in our case. In the experiments described below, we changed the texture fineness in this sense either by changing the image magnification (while keeping the image size fixed) or by changing the actual size of the material (while keeping the camera and display conditions fixed) and investigated human perception of the texture fineness in subresolution scale. 

Subsequently, in psychophysical experiments, we tested the ability of humans to discriminate subresolution fineness, first using artificial textures and then natural textures. After that, we analyzed the image statistics that were correlated with subresolution fineness and found certain promising candidates. Finally, we examined how manipulation of these image statistics affected subresolution fineness perception. Our findings suggest that the human visual system is able to estimate subresolution fineness by using correlated changes in visible image statistics, such as the intensity contrast and the shape of intensity histograms. 

Human ability to discriminate subresolution fineness

Experiment 1: Pairwise comparison of random textures

We explored to what extent human observers can evaluate the fineness of superfine structures. Experiments 1 and 2 used artificial textures, while Experiments 3 and 4 confirmed the basic findings of those experiments with natural textures. In particular, we explored the ability of humans to discriminate subresolution fineness by using a task consisting of pairwise comparisons of random textures. 

Methods

Similar experiments were conducted at two laboratories (NTT and Toho University) under slightly different conditions. 

Observers:

The participants were 12 paid volunteers at NTT, and one of the authors and nine student volunteers at Toho University. All had normal or corrected-to-normal vision. Except for the author, the participants were naïve to the purpose and methods of the experiment. The experiments were approved by the Ethical Committees at NTT Communication Science Laboratories and Toho University and were conducted in accordance with the Declaration of Helsinki. At NTT, we measured the visual acuity of all volunteers in Experiments 1, 2, 3, 4, and 6 (1). 

Stimuli:

To generate the series of random hair-like textures shown in Figure 2, we first created a large image I₀(x) by quasirandomly placing numerous thick white lines (intensity = 1) of width w₀ on a black field (intensity = 0). At NTT, w₀ = 4 and the lines did not overlap. At Toho University, w₀ = 40 and the white lines were allowed to overlap. The intensity was clipped to the range (0–1). 

In both cases, a rescaled digital image I(I;D) was then created by applying a rectangular low-pass filter L and down-sampling as follows:  

 

 

 

 

 

where i indicates the pixel position of the output image. D denotes the scale factor (0 < D < 1), and X represents the reciprocal of the line width measured in pixel widths, indicating the physical fineness level of the image.  

Apparatus:

At NTT, the texture stimuli were displayed using Matlab R2013b in conjunction with the Psychophysics Toolbox 3 (Brainard, 1997; Pelli, 1997). They were displayed on a calibrated 30-in EIZO color monitor (ColorEdge CG303W) controlled with an NVIDIA video card (Quadro 600) with a pixel resolution of 2560 × 1600 and a frame rate of 30 Hz. The intensity of each phosphor could be varied with 10-bit resolution. The maximum luminance of the monitor was 150.6 cd/m². In Experiments 1 and 2, a participant viewed the stimuli in a dark room at a viewing distance of 43 cm, where a single pixel subtended 2 min, or at 344 cm, where a single pixel subtended 0.25 min. 

At Toho University, the texture stimuli were displayed on a gamma-corrected 19-in CRT monitor (Mitsubishi RDF193H) controlled with a PC through a BITS++ video processor (Cambridge Research Systems). The maximum luminance of the monitor was 157 cd/m². Horizontal textures excluded interpixel interactions on the CRT monitor. An observer viewed the stimuli in a dark room at a viewing distance of 60 cm, where a single pixel subtended 2.1 min. 

Procedure:

In each trial, two texture images, quasirandomly chosen from X = 0.25, 0.5, 1, 2, 4, 8, and 16, were presented on the monitor. They were aligned above and below each other in random order. Each texture had 256 × 256 pixels. The participants were asked to select which of the texture images looked finest by pushing one of the keys. The stimuli were presented until the participant made a response. The task instructions were simply to compare fineness, and no response feedback was given. Each participant made 10 judgments for each of 21 texture pairs in a single block. At NTT, the judgments at the two viewing distances (43 and 344 cm) were blocked and the order of the blocks was counterbalanced across participants. 

Results and discussion

Figure 3 shows the results of Experiment 1 conducted at NTT. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column), based on 10 judgments by 12 observers. The left panel indicates the results at the viewing distance of 43 cm (0.5 pixel/min), and the right one indicates the results at the viewing distance of 344 cm (4 pixel/min). The image resolution was primarily limited by the pixel resolution at 43 cm, and by the visual acuity of the observers at 344 cm, since a visual acuity test showed that all the observers could perfectly resolve a square-wave grating of 15 cycles/° (0.5 line/min), but none could resolve a grating of 60 cycles/° (2 line/min) (1). The results of the pairwise comparison indicated that, at both viewing distances, the observers could judge the order of subresolution texture fineness with very small errors (Figure 3). The results obtained at Toho University were quite similar (2). The results suggest that the visual system can veridically judge the order of magnitude of subresolution texture fineness. 

Figure 3

 

Results of pairwise comparison of random-line textures. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column) based on 10 judgments by 12 observers. The white numbers in each cell indicate 95% confidence intervals computed using bootstrap estimates. The fineness judgment was nearly perfect. The results show that human observers accurately judged the order of magnitude of subresolution fineness.

Experiment 2: MLDS of random textures

Experiment 1 showed that the observers could correctly evaluate the fineness of the textures, even when the thinnest element was much thinner than the resolution limit of the visual system and that of the digitized image. However, because Experiment 1 was a pair-wise comparison, it remains unclear as to the quantitative differences in perceived fineness of the subresolution textures. To assess the magnitude of perceived fineness for textures more quantitatively, Experiment 2 estimated how perceptual fineness changes with physical subresolution fineness by using maximum likelihood difference scaling, MLDS (Maloney & Yang, 2003; Knoblauch & Maloney, 2008). 

Methods

Stimuli and procedure:

The same artificial textures as used in Experiment 1 were used in Experiment 2. At a viewing distance of either 43 or 344 cm, the participants viewed two pairs of textures and judged which one had a larger in magnitude within-pair fineness difference. The texture pairs were selected from X = 1.41, 2.0, 2.8, 4.0, 5.7, 8.0, and 11.3. This experiment was conducted at NTT using 13 naïve participants. 

Results and discussion

Using the results of each participant, we estimated the perceptual scale of perceived fineness as a function of the physical fineness index (X) by using the MLDS package of the R statistical software (Knoblauch & Maloney, 2008). Figure 4 shows the individual results of perceived fineness and results averaged across observers. They indicate a monotonic increase in perceived fineness with physical subresolution fineness, with some reduction in the rate of increase at higher fineness. 

Figure 4

 

Perceptual scale of subresolution fineness estimated using MLDS. Color symbols indicate the estimates of individual participants. Black circles indicate the average across participants, with error bars indicating 95% bootstrap confidence intervals. The results show that the perceived fineness monotonically increased with fineness X, with some reduction in the rate of increase at higher fineness.

Experiment 3: Pairwise fineness comparison using natural textures

Since we used simple artificial textures in Experiments 1 and 2, one could argue that texture fineness might not have been conceptually clear to some of the observers. However, verbal reports collected before the experiment indicated that some observers noticed by themselves differences in the subresolution fineness of the 1D random textures (3). In addition, to address concerns about the artificial textures, we carried out two further experiments using images showing natural textures (Experiments 3 and 4). In Experiment 3, we conducted a pairwise comparison of the subresolution fineness of natural textures. 

Methods

Stimuli and procedure:

We took photos of a brush, dark and light hair samples, and grass. The brush and hair samples were taken in a photography box illuminated with D65 fluorescent lamps (TOSHIBA FL20S-D-EDL-D65) using a standard digital camera (Nikon D5100) with a pixel resolution of 3264 × 4928. The photo of grass was taken outdoors on a sunny day using a standard digital camera (Olympus Stylus TG-2) with a pixel resolution of 2112 × 2886. From these original photos, we created rescaled digital images by applying a low-pass filter and down-sampling. We used three magnitudes of texture fineness (with the scales of 1/8, 1/16, and 1/32 of the original photo) for the brush and hair textures and two for the grass texture (1 and 1/2). Each texture image was cropped to 102 × 154 pixels. This experiment was conducted at NTT using 11 naïve participants. They viewed a pair of textures with different magnitudes of fineness at a distance of 43 cm and judged which of the presented pairs looked finer. Each participant made 10 judgments for each pair. The other procedures were the same as those used in Experiment 1. 

Results and discussion

Figure 5 shows the results of a pairwise comparison of subresolution fineness of textures made of a brush, dark/light wigs, and grass. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column), based on 10 judgments by 11 observers. As we found with the artificial textures, the stimulus discrimination was very accurate. It should be noted that the results with the 2D grass texture indicated that the subresolution fineness perception was not limited to 1D line textures. In addition, the finding that the observers could correctly perform the present task suggests that they did not interpret the image pairs as the same material at different image magnifications. 

Figure 5

 

Results of pairwise comparison of natural textures. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column) based on 10 judgments by 11 observers. The white numbers in each cell indicate 95% confidence intervals computed using bootstrap estimates. Even when the natural stimuli were used, the stimulus discrimination was very accurate.

Experiment 4: Pairwise fineness comparison using natural sewing threads with different line widths

The fineness of the stimuli used in Experiments 1, 2, and 3 was, more or less, defined on the basis of simulations, e.g., simulated subresolution elements or image scale differences. In Experiment 4, we asked if the observers could judge the subresolution fineness of natural textures with different line widths. We measured the performance of a pairwise comparison of natural sewing threads with different line widths. 

Methods

In Experiment 4, the observers compared the fineness of two textures (bundles of threads), one made of threads of ∼0.2 mm width and the other made of threads of ∼0.3 mm width. The image scale was reduced in three steps (1/8, 1/16, and 1/32 of the original photo where each thread subtended ∼15 or ∼20 pixels). We took photos of the threads with different line widths (about 0.2 and 0.3 mm/line) by using a digital camera (Nikon D5100) in a photography box illuminated with D65 fluorescent lamps (TOSHIBA FL20S-D-EDLD65). The pixel resolution of each photo was 3264 × 4928, and a line width of 0.2 and 0.3 mm/line was displayed on an image of about 15 and 20 pixels, respectively. To make supra- and subresolution textures, we changed the scale of the natural sewing threads by applying a low-pass filter and down-sampling to the photos. We used three scales: 1/8, 1/16, and 1/32 of the original photo. Each texture image was cropped to 102 × 154 pixels. This experiment was conducted at NTT using 10 naïve participants. They viewed a pair of textures with different magnitudes of fineness at a distance of 86 cm and judged which of the presented pairs looked finer. Each participant made 10 judgments for each pair. The other procedures were the same as in Experiment 3. 

Results and discussion

Figure 6 shows the results of the pairwise comparison of natural textures with different line widths. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column), based on 10 judgments by 10 observers. The results showed that the observers could discriminate the texture fineness fairly accurately and that their performance did not fall even when the width of single thread became smaller than the diameter of a single pixel. These findings suggest that perception of subresolution fineness is a real ability of the human visual system working in natural environments. 

Figure 6

 

Results of pairwise comparison of natural sewing threads with different line widths. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column) based on 10 judgments by 10 observers. The white numbers in each cell indicate 95% confidence intervals computed using bootstrap estimates. The observers could discriminate the texture fineness fairly accurately, and their performance did not deteriorate even when the width of a single thread became smaller than the diameter of a single pixel (i.e., textures with 1/32 scale).

Image statistics correlated with subresolution fineness

Although we showed in the last section that the visual system could estimate subresolution fineness, we still must ask, what is the underlying mechanism of this subresolution fineness perception? One possibility is that the human visual system reconstructs superresolution images using a certain interpolation technique (Baker & Kanade, 2002; Glasner, Bagon, & Irani, 2009). However, the 1D textures we used seem to exclude any potential cues for any currently known algorithm. 

Another possibility is that the visual system might make use of changes in image statistics correlated with the physical texture fineness in order to perceptually estimate the magnitude of image fineness beyond the resolution limit. The analysis of the spatial frequency and intensity histogram (Figure 7) suggests some image features an observer might use to judge the subresolution fineness. Concerning the spatial frequency power spectrum, while coarse textures (X < 1) are low-frequency dominant because thick lines occupying two or more adjacent pixels produce an intensity correlation across neighboring pixels, subpixel fine textures (X ≥ 1) have a flat spectrum because the pixel intensities are independent of one another. As subresolution fineness increases, the overall power, which corresponds to the luminance contrast, falls. 

Figure 7

 

(a) Power spectra of the random hair-like textures used in Experiment 1. (b) Intensity histogram of the textures used in Experiment 1. As the stimulus fineness (X) increases, the image contrast of a texture decreases, and the intensity histogram takes on a Gaussian shape.

With regard to the luminance histogram (Figure 7b), as long as a black/white line is thicker than a pixel (X < 1), the histogram has a binary distribution. As the line width becomes finer than the pixel width (X ≥ 1), the width of the intensity distribution, which again corresponds to the luminance contrast, is reduced, and the shape of the distribution approaches a Gaussian one. 

These image changes can be explained by the central limit theorem. For line textures with subpixel fineness, the pixel intensity is determined by the average intensity of independently colored lines falling within the pixel. As the texture becomes finer, and the number of lines per pixel increases, the mean pixel intensity does not change, but the variation of the pixel intensity, which determines the intensity contrast of the texture, decreases. In addition, the shape of the distribution of pixel intensities approaches that of a Gaussian distribution. 

Image contrast as a cue to subresolution fineness

Experiment 5: Fineness matching of textures modulating image features

The image analysis suggests that the intensity contrast of a texture may provide useful information for estimating subresolution fineness. To determine whether the visual system “knows” the image constraint, it is effective to check whether the appearance of a novel image can be changed by applying the constraint to the image (Fleming & Bülthoff, 2005; Motoyoshi et al., 2007; Sawayama & Kimura, 2015). We conducted Experiments 5 and 6 to see what kind of image cues the human visual system actually uses. In this experiment, we used artificial random-line textures and a matching method to estimate the apparent fineness with target images whose image features had been modified in various ways: overall contrast reduction, overall amplitude reduction, low-frequency reduction and enhancement, and high-frequency reduction and enhancement (Figure 8). 

Figure 8

 

The contrast of the texture is a determining factor of fineness perception. (a) Effect of reduction in overall contrast, reduction in low spatial-frequency (SF) contrast, and enhancement in high SF contrast. (b) Apparent fineness of image-modulated textures estimated using the matching method. The abscissa is X of an image-modulated texture, while the ordinate is X of the original texture having the matched apparent fineness. The error bars indicate 95% confidence intervals computed using bootstrap estimates. The low-high SF boundary is 64 cycle/image (7.11 cycle/°). The apparent fineness is affected by changes in the overall or low SF contrast, while it is little affected by changes in the overall amplitude or the high SF contrast. This experiment involved random hair-like textures (whose power spectra are shown in Figure A3), presented with an image resolution of 2.1 min/pixel.

Methods

Stimuli and procedure:

The following image modulations were given to the texture images: 

•  
  a. Overall luminance contrast modulation:

 

 

where Ī is the mean intensity and c is the contrast change factor. We tested with c = 0.6;  

•  
  b. Overall luminance amplitude modulation:

 

 

 

We tested with c = 0.6. Note that this modulation did not alter the contrast. 

•  
  c. Low spatial-frequency modulation: The low-frequency components of the texture were modulated in the Fourier domain.

 

 

 

 

 

 

•  
  d. High spatial-frequency modulation: Same as (c), except

 

 

 

The intensity beyond (0–1) was truncated. 

In each trial, an image-modulated texture with a given X was presented with a nonmodulated original texture. Participants adjusted the value of X of the nonmodulated texture until the fineness of both images looked most similar. A single key press increased or decreased the value of X of the nonmodulated texture by a factor of 2^0.5. Experiment 5 was conducted at Toho University using the same 10 participants as in Experiment 1 (2). Each participant made five adjustments for each image-modulated texture. The other methods were the same as those in Experiment 1 at Toho University. 

Results and discussion

Figure 8 shows the results of the matching experiment. The horizontal axis shows X of an image-modulated texture, while the vertical axis shows X of the original texture having the matched apparent fineness. Different panels indicate different modulations for textures. The results indicate that the texture looked finer when the overall luminance contrast was reduced, except at coarse resolutions, where the observers could directly see the width of the lines. A change in the luminance amplitude (which did not change the luminance contrast) had little effect. Reducing and enhancing the low-frequency components made the texture look correspondingly finer and coarser. In contrast, changes in the high-frequency components had little effect on the apparent fineness. In addition, image sharpening using high-frequency enhancement had little effect on subpixel fineness perception, despite that it is the standard way of increasing the visibility of image details and has recently been shown to be useful in increasing the apparent roughness of fabrics (Giesel & Zaidi, 2013). 

Experiment 6: Pairwise fineness comparison of natural textures with contrast modulations

Experiment 5 showed that reducing the intensity contrast of a texture tends to make the texture look finer. As in Experiments 3 and 4, we need to confirm whether this finding is specific to the simple artificial textures. To test the generality of the effect of texture contrast on apparent fineness, we thus reduced or enhanced the image contrast of the natural textures in Experiment 6. We conducted a pairwise fineness comparison task with these natural textures. 

Methods

Stimuli and procedure:

The textures in Experiment 3 were used in Experiment 6. Since the single hair width was 4–10 pixels in the original photos of the brush and hair textures, we rescaled the widths to 1/32 of the original ones to make subpixel fineness. In the original photo of the grass texture, the single element was already smaller than a single pixel. Each texture image was cropped to a square of 102 × 154 pixels. For each original texture, we created a contrast-down version and a contrast-up version by decreasing (× 3^−0.5) or increasing (× 3^0.5) the contrast of the texture. The experiment was conducted at NTT using 10 naïve participants. In each trial, the participants viewed a pair of texture images presented on the monitor. The pair was selected from the three versions of the image. The participants were asked to select the texture images that looked finer. Each participant made 10 judgments for each stimulus condition. In Experiment 6, a participant viewed the stimuli in a dark room at a viewing distance of 86 cm, where a single pixel subtended 1 min. The other methods were the same as those in Experiment 1. 

Results and discussion

Figure 9 shows the results of Experiment 6. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column), based on 10 judgments by 10 observers. The results indicate that the contrast manipulations for the natural textures with subresolution fineness made the textures look finer and coarser. Thus, our key findings can be generalized to natural textures. 

Figure 9

 

Contrast modulations change the apparent fineness of natural textures. The color and the black number in each cell indicate the proportion of trials in which a texture with X1 (row) was judged to be finer than another texture with X2 (column), based on 10 judgments by 10 observers. The white numbers in each cell indicate 95% confidence intervals computed using bootstrap estimates. For four natural fine textures, a contrast reduction made the texture look finer, and a contrast enhancement made it look coarser. This is supported by the results of a pairwise fineness comparison, shown on the left of each texture image.

Effect of intensity histogram on subresolution fineness perception

Experiment 7: Fineness and contrast matching of textures with different distributions

We have shown that a reduction in image contrast is an effective cue for human observers to evaluate subresolution fineness. On the other hand, contrast reductions can be caused by many other factors, including optical haze and inappropriate tone reproduction in photos. It is obvious that a texture contrast change should not always be ascribed to a change in fineness and that there should be image conditions where a contrast reduction can be reasonably interpreted as an increase in texture fineness rather than otherwise. As discussed above, the central limit theorem predicts two image changes that are produced by increasing the number of independently colored elements within a pixel. One prediction is a reduction of overall contrast, and the other is the pixel intensity histogram's shape approaching that of a Gaussian. This suggests that the human visual system may utilize the shape of the intensity histogram to judge whether the image is likely to be a superfine texture whose contrast is related to texture fineness. To investigate this possibility, Experiment 7 measured the perceived fineness of the artificial texture with several intensity distributions: uniform, binominal, Gaussian unimodal, and high-kurtosis unimodal. We used three levels of RMS (root mean square) contrast (0.1, 0.2, and 0.4) and three levels of line width (0.5, 1.0, and 2.0 lines/min). 

Methods

Stimuli and procedure:

Four types of intensity histogram were used in the experiment: uniform, binominal, Gaussian unimodal, and high-kurtosis unimodal. For the uniform condition, texture elements of an image of 256 × 256 pixels were randomly taken from a uniform histogram. For the binominal and Gaussian and high-kurtosis unimodal conditions, the kurtosis of each intensity histogram was modulated while the mean and the standard deviation of the histogram were kept constant. Specifically, we modulated the kurtosis values using the Matlab function “pearsrnd,” and the values were set to 1.5 for the binominal condition and 3.0, and 6.0 for the Gaussian and high-kurtosis unimodal conditions, respectively (Figure 10a). The mean luminance of each stimulus was set to 0.5. The RMS contrast of each condition was set to 0.1, 0.2, or 0.4. This experiment was conducted at a viewing distance of 86 cm, where a single pixel subtended 1 min. The line fineness X of each condition was 1 (2 lines/min), 0.5 (1 line/min), or 0.25 (0.5 lines/min). The RMS contrast of the matching stimulus in the fineness-matching task was set to 0.4, while the fineness X of the matching stimulus in the contrast-matching task was set to 0.5 (1 line/min). 

Figure 10

 

The contrast-modulation effect was changed with the intensity distribution of the texture. (a) Stimulus examples of the different intensity distributions: uniform, binominal, Gaussian unimodal (kurtosis = 3.0), and high-kurtosis unimodal (kurtosis = 6.0). (b) Perceived fineness of contrast-modulated textures estimated using the matching method. Left, center, and right panels indicate the line width of 2.0, 1.0, and 0.5 lines/min, respectively. The abscissa of each panel is the RMS contrast of a contrast-modulated texture, while the ordinate is X of the original texture having the matched apparent fineness. Different symbols in each panel indicate different intensity distributions as shown in the legend. The error bars indicate 95% confidence intervals computed using bootstrap estimates. For the suprathreshold line width, the interaction between the contrast condition and the distribution condition was statistically significant (p < 0.05), while the interactions were not (p > 0.05) significant for the near-threshold and subthreshold line widths. (c) Regression slope for the matched fineness of all participants. The asterisks in each panel indicate that the slopes were statistically significant; i.e., the perceived fineness increased as the contrast of the texture decreased. The error bars indicate 95% confidence intervals computed using bootstrap estimates. (d) Perceived contrast of contrast-modulated textures estimated using the matching method. The vertical axis indicates the matched RMS contrast of the matching stimulus. The other figure configurations are the same as in Figure 10b.

This experiment was conducted at NTT using 18 naïve participants. The rating tasks involved fineness and contrast judgments. A test stimulus was presented with a matching stimulus as in Experiment 5. For the fineness judgment, participants adjusted the value of X of the matching stimulus until the fineness of both images looked most similar. A single key press increased or decreased the value of X of the nonmodulated texture by a factor of 2^0.25. For the contrast judgment, participants adjusted the value of the contrast of the matching stimulus until the contrast of both images looked most similar. Each participant made six adjustments for each condition. The other methods were the same as in Experiment 5. 

Results and discussion

Figure 10b shows the perceived fineness of contrast-modulated textures estimated using the matching method. The left, center, and right panels indicate the line widths of 2.0, 1.0, and 0.5 lines/min, respectively. The horizontal axis of each panel shows the RMS contrast of a contrast-modulated texture, while the vertical axis shows X of the original texture having the matched apparent fineness. Different symbols in each panel indicate different intensity distributions. Figure 10c shows the regression slope for the matched fineness of all participants. The asterisks in each panel indicate that the slopes were statistically significant; i.e., the perceived fineness increased as the contrast of the texture decreased. 

The above results show that the fineness perception depended on the histogram shape (Figure 10b, right). Specifically, for the thickest line width (0.5 lines/min), there was a statistically significant interaction between the effect of contrast and the effect of the shape of the distribution, F(6, 102) = 4.05, p < 0.01. The negative slope of the contrast effect was statistically significant for the two unimodal conditions (p < 0.05), but not for the other two conditions (p > 0.05; Figure 10c). While for the thinnest line width (2.0 lines/min), the perceived fineness changed with the contrast modulation with little effect of the distribution shape (Figure10b, left and center), this result was expected because, regardless of the shape of the physical intensity distribution, spatial averaging by the visual system would make the perceptual intensity histogram unimodal and reduce the effect of the histogram's shape. 

One could argue that the shape of the histogram distribution might somehow change the apparent contrast, which in turn would alter the apparent fineness. However, when we estimated the perceived contrast for the same stimulus set, there was no effect of histogram shape at all (Figure 10d). 

These findings support the hypothesis that the human visual system uses the shape of the intensity histogram to judge whether the image is a superfine texture whose contrast is associated with texture fineness. 

General discussion

The present study explored the questions of whether human observers can discriminate the texture fineness of superfine structures whose individual elements are finer than the spatial resolution limit of the visual system or that of a digitized image, and if they can, what the critical image features are for the subresolution fineness perception. We found that human observers could correctly judge the order of magnitude of subresolution fineness (Figures 1 through 6). The image analysis indicated that as the fineness of a texture increases in the subresolution range, the intensity contrast of the texture monotonically decreases, and the intensity distribution approaches a unimodal Gaussian one, as predicted by the central limit theorem (Figure 7). We then found that the reduction in intensity contrast is actually a critical feature for judging the subresolution texture fineness (Figures 8 and 9). Finally, we showed that the intensity contrast affected the fineness perception only when the shape of the intensity distribution was unimodal, as should be the case for subresolution textures (Figure 10). 

The present findings indicate that the human visual system makes use of simple image statistics present in the luminance histogram to estimate image fineness with magnitudes higher than the spatial resolution limit, imposed either by the image pixel size or by the observer's visual acuity. Even under conditions in which a correct estimation of the physical parameters is computationally difficult, the visual system can make reasonable estimations using image cues that statistically correlate with the ground truth in typical natural environments. Just like binocular disparities contain information about depth, luminance contrasts contain information about subresolution fineness, although the link is less direct. Similar computational mechanisms using statistical image cues have been suggested for the perception of other real-world entities such as surface gloss (Nishida & Shinya, 1998; Adelson, 2001; Fleming, Dror, & Adelson, 2003; Motoyoshi et al., 2007; Marlow, Kim, & Anderson, 2012), translucency (Fleming & Bülthoff, 2005; Motoyoshi, 2010; Xiao et al., 2014; Kawabe, Maruya, & Nishida, 2015), and the gist of a natural scene (Oliva & Torralba, 2007). 

One might argue that the observers in our experiments did not judge the texture fineness, but just estimated low-level cues such as contrast. That is, such observers, rather than having actually perceived subresolution fineness, might have only made a cognitive decision to take the lower contrast texture as finer to fulfill the requirement of the pairwise comparison task. However, this is unlikely to provide a coherent explanation of the whole set of results for the following reasons: (a) Experiment 4 showed that the observers can veridically judge the actual fineness of physically different materials. (b) It is logically unclear why the confused observers associated lower contrast with higher fineness in the case of the perception of subresolution fineness, but not in other cases (Experiments 5 and 7). In Experiment 5, when individual elements of a texture are visible, the observers did not judge the reduced contrast texture to be a fine one. In Experiment 7, the observers did not judge the reduced contrast texture to be a fine one when the line width was thicker than the resolution limit and when the intensity histogram was not unimodal. (c) In all of our experiments, we instructed the observers only to judge texture fineness, without giving response feedback that would be necessary for them to develop a valid inference strategy to meet the task demands. (d) A significant number of naïve observers voluntarily noticed the difference in subresolution fineness when they first saw the texture stimuli (3). (e) In Experiment 2, the participants could not only qualitatively compare subresolution fineness between a texture pair, but also quantitatively evaluate the fineness difference magnitude. (f) The high-frequency modulations could be a strong low-level cue for cognitive inference of texture fineness, but they had little effect in Experiment 5. 

In addition, one might argue the possibility that some size illusions might explain the reduction in apparent line width with image contrast reduction. It has been suggested that image contrast reductions increase the apparent spatial frequency of sinusoidal gratings, though slightly (< 10%; Georgeson, 1985). Image contrast manipulations could also affect the irradiation illusion that alters the apparent width of bright figures (Galilei, 1632; Drake trans., 1967). 

These effects, however, are expected to occur even, or mainly, for textures with thick lines (Figure 8b), and there is no reason for them to diminish when the intensity histogram is not unimodal (Figure 10). Therefore, it seems hard to ascribe the effect of image contrast on subresolution fineness perception to the known effects of contrast on size. 

We found that reductions in intensity contrast increase the apparent texture fineness when the texture has a characteristic nonspatial image feature of subresolution textures, i.e., a unimodal intensity histogram. Other characteristic features, in particular those of spatial structures, may be also relevant. We showed that our subresolution random texture has a broad and flat power spectrum (Figure 7) as does white noise. This is approximately true for the natural fine textures we used (Figure 12). The broad and flat power spectrum could be suitable for super fine textures. If a texture has a regular repetitive structure and a narrow-band power spectrum, it will become invisible as soon as the scale enters the subresolution range, leaving few clues about the magnitude of fineness. On the other hand, if a random texture has a low-pass 1/f spectrum (like pink noise does), the texture appearance will change little with the spatial scale. Therefore, a broad and flat power spectrum may be an important texture feature for stable perception of the magnitude of subresolution fineness. In addition, the phase spectrum of our artificial hair textures was random. A quasirandom phase spectrum might be another critical texture feature for perception of subresolution fineness. A detailed evaluation of the effects of power and phase spectra, as well as their interactions with the histogram shape, remains open for future study. 

Figure 11

 

Effect of global or local contrast reduction on the perceived fineness. Global contrast reduction for a scene image (upper right) has little effect on fineness perception, although a contrast reduction applied to a local region-of-interest (lower left) tends to make a surface look finer (lower right).

Furthermore, texture uniformity may be also relevant for the effect. As in Figure 11, global contrast reduction for a scene image has little effect on fineness perception. The modulation just makes the scene look smoggy. On the other hand, when we applied the contrast reduction to a locally segregated region which had a uniform quasirandom texture, e.g., a road, the contrast-reduced region could be perceived to be finer. This suggests that a contrast reduction within a uniform texture relative to that of the entire image might have a role in fineness perception. 

Spatial distinction on a scale finer than visual acuity is known as hyperacuity, but known hyperacuity phenomena concern detection of displacements across positions (Westheimer & McKee, 1977), eyes (Schor & Badcock, 1985), and time (Nakayama, 1981), and the underlying mechanisms are totally different from the hyper resolution revealed in this study. It is also known that the images presented in the peripheral visual field, or those presented in motion, look sharper than would be expected from the degraded spatial resolutions under these viewing conditions (Bex, Edgar, & Smith, 1995; Galvin, O'Shea, Squire, & Govan, 1997). This perceptual sharpening is functionally similar to the subresolution fineness perception in that both indicate the visual system's function to overcome spatial resolution limits, although it makes the input image look sharper regardless of whether it is physically sharp or blurred. In another attempt to overcome the spatial resolution limit, the visual system uses the available fine high-contrast luminance edge information that is normally correlated with the color edge information in natural images to extrapolate fine color edge information lost by the low resolution of chromatic processing (Boynton, 1978). Considering the subresolution fineness perception as filling-out of missing high-frequency information from visible low-frequency information, one can find an example of the analogous effect in the opposite direction, i.e., the filling-out of missing low-frequency information from high-frequency information, in the Craik-O'Brien Cornsweet illusion (Cornsweet, 1970). Considering the perception of superfine surface structures from a broader perspective, while the present finding concerns the estimation of the surface meso-structure, the human visual system can estimate even a finer surface microstructure (e.g., polished specular surface, matte Lambertian surface, velvety surface) from the pattern of the surface angular dependence of light reflection (bidirectional reflectance distribution function; Dana, Van Ginneken, Nayar, & Koenderink, 1999). The visual system exploits a variety of strategies to overcome the limitations of spatial processing, and our finding adds a novel effect to the list. 

The reader should be reminded however that our study does not make a clear distinction between subresolution fineness perception and subresolution density perception. Although it remains to be studied how the texture density affects subresolution fineness perception, what is expected from the present finding is that the nature of the task changes for very sparse textures since the stimulus conditions necessary for subresolution fineness perception do not hold anymore. 

Concerning the neural basis, the response to local image features in the primary visual cortex (V1) might be able to represent the image statistics necessary for fineness perception, i.e., the intensity contrast and intensity histogram. In addition, a texture synthesis algorithm that preserves information to which the second visual cortical area (V2) has been suggested to be sensitive (Portilla & Simoncelli, 2000; Freeman & Simoncelli, 2011; Freeman, Ziemba, Heeger, Simoncelli, & Movshon, 2013) could perfectly reproduce the artificial hair-like textures and approximately reproduce the natural fine textures we used (Figure 12). 

Figure 12

 

Results of texture synthesis for a random hair-like texture and natural textures used in our experiments. We synthesized a new texture from low-level image statistics of a random hair-like texture and natural textures using the algorithm in Portilla and Simoncelli (2000). The results show that each synthesized texture is very similar in fineness appearance to the original texture. The power spectrum of each natural texture is shown at the bottom.

Therefore, early visual areas are likely to play major roles in the analysis of relevant texture features, but it remains unclear where and how the shape of the intensity histogram is evaluated. The image statistics extracted in the early visual areas might be associated with the notion of texture fineness in ventral areas reportedly showing sensitivity to surface texture and material properties (Cant & Goodale, 2007; Cavina-Pratesi, Kentridge, Heywood, & Milner, 2009; Hiramatsu, Goda, & Komatsu, 2011; Nishio, Goda, & Komatsu, 2012; Goda, Tachibana, Okazawa, & Komatsu, 2014; Okazawa, Tajima, & Komatsu, 2015). Further investigation is necessary as to whether the human brain acquires this elegant computational ability genetically through phylogenetic evolution or through personal multimodal interactions with the natural environment (Yamins & DiCarlo, 2016). Further research on how the brain represents invisible fine structures is also necessary. This question is related to long and unsettled debates as to whether the brain explicitly constructs representations for filled-on/filled-out perceptions (Anderson & Van Essen, 1987; Komatsu, 2006; Boyaci, Fang, Murray, & Kersten, 2007; Hsieh & Tse, 2009). Finally, with regard to practical applications, the present findings support the view that in computer graphics, accurate, but high-cost, simulations of detailed object structures are unnecessary for image rendering of superfine structures (Shinya & Nishida, 2014). 

Acknowledgments

This work was supported by Grants-in-Aid for Scientific Research on Innovative Areas (Numbers JP23135528, JP22135004, and JP15H05915) from MEXT/JSPS, Japan. The authors declare no competing financial interests. 

Commercial relationships: none. 

Corresponding author: Masataka Sawayama. 

Email: [email protected]. 

Address: NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Atsugi, Kanagawa, Japan. 

Appendix A: Visual acuity of participants

For all of the 29 naïve participants in the experiments conducted at NTT, we measured the visual acuity using the same experimental apparatus used for the following fineness judgment experiments. In each trial, a pair comprising a uniform field and horizontal grating pattern (100% contrast square wave) was presented on the monitor. The participants viewed the stimuli at a distance of 344 cm (where 1 pixel subtended 0.25 min) and judged which of the stimuli was the grating. The spatial frequency of the grating stimuli was 7.5, 15, 30, 60, or 120 cycles/° (16, 8, 4, 2, or 1 lines/pixel, or 4, 2, 1, 0.5, or 0.25 lines/min, respectively) and randomly changed from trial to trial. The mean luminance of each stimulus was approximately 75 cd/m², and it was slightly fluctuated by random Gaussian noise to prevent the participants from using the apparent mean luminance difference as a clue for discrimination. The total number of trials for each participant was 250 (50 for each frequency). The results are shown in Figure A1. All participants could perfectly resolve 15 c/° but could not resolve 60 cycles/°. Therefore, in the subsequent fineness judgment experiments, the visual resolution was primarily limited by the pixel resolution at the viewing distance of 43 cm, while it was primarily limited by the visual acuity of the participants at 344 cm (Figure 2). 

Figure A1

 

Visual acuity of participants. Each color symbol indicates the proportion of correct discrimination for each participant, and black diamonds indicate averages.

Appendix B: Pairwise fineness comparison of random hair-like textures—additional data

Figure A2 shows the results of Experiment 1 (a pairwise comparison experiment) that was carried out at Toho University using slightly different random line textures (whose power spectra are shown in Figure A3), slightly different apparatus, and a different group of 10 participants from those used at NTT (see “Methods” of Experiment 1). The participants showed nearly perfect performance in a texture fineness comparison task, as we observed in Figure 3. The results indicate that the subresolution fineness discrimination is a robust finding, and that the data collected at two labs were comparable despite minor differences in methodology. 

Figure A2

 

Results of pairwise fineness comparison for random hair-like textures. The data were collected at Toho University.

Appendix C: Free description of differences between random hair-like textures

The experiment was conducted in NTT using 17 naïve participants. We showed two pairs of random hair textures (a pair of X = 0.25 and X = 2, and a pair of X = 2 and X = 16) to the participants and asked them to freely report in what aspect the texture pairs differed from each other. This was the first time for them to see these stimuli, and we did not give them any further instructions. The order of the presentation of the pairs was counterbalanced across observers. The proportion of observers who mentioned fineness was 14/17 (82%) for the pair of X = 0.25 and X = 2, and 8/17 (47%) for the pair of X = 2 and X = 16. Among the eight observers who were shown the latter finer pair first, three mentioned fineness. Considering the nature of the task, these are considerably conservative estimates of the proportion of the observers who naturally perceived subresolution fineness in the texture stimuli; and the proportion was likely to be increased when the task directed their attention to this dimension. 

Figure A3

 

Power spectra of random hair-like textures used in the experiments conducted at Toho University. The pattern is very similar to Figure 7a (the power spectra of random hair-like textures used in the experiment conducted at NTT), except for small power drops at higher spatial frequencies for X = 1 and X = 2.