Abstract
The measurement of the true information content (i.e., entropy) of natural scenes is a computationally intractable problem. The true measure requires an estimate of the probability of all possible images: impossible even with small image patches. One can make estimates derived from conditional probabilities of some basis set, but that also can be shown to produce highly inaccurate measures unless all conditional probabilities are considered. Here we describe a new method that allows us to estimate the relative entropy of different data sets (e.g., natural scenes versus 1/f noise versus city scenes). An image patch is drawn at random from a large collection of possible patches. The image patch is then compared to patches in a comparison set to find the patch that is closest (Euclidean distance) as a function of the number of patches in the comparison set (e.g., 10 to 1 million). We find that the average minimum distance is typically a smooth monotonic function of the number of image patches in the comparison set. We also find that this function can be used to estimate entropy. For high contrast, 8x8 patches normalized for mean intensity and contrast, we find that natural scenes have approximately 1/5th the entropy of Gaussian white noise quantized to the same level. We discuss how this “projected entropy” measure behaves as a function of patch size and image structure. We also show that this measure allows us to estimate dependencies of the “independent components” as derived from ICA measures.
This works was supported by NGIA grant HM 1582-05-C-0007