December 2022
Volume 22, Issue 14
Open Access
Vision Sciences Society Annual Meeting Abstract  |   December 2022
A local probabilistic model of features and segmentation learned by optimizing prediction
Author Affiliations & Notes
  • Heiko Schütt
    New York University
    Columbia University
  • Wei Ji Ma
    New York University
  • Footnotes
    Acknowledgements  Partially funded by DFG Grant SCHU 3351/1-1.
Journal of Vision December 2022, Vol.22, 4178. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Heiko Schütt, Wei Ji Ma; A local probabilistic model of features and segmentation learned by optimizing prediction. Journal of Vision 2022;22(14):4178.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

One classical criterion for finding good features is that they should be predictable over space and time such that they explain a large part of the input, can be attributed to whole objects, and are distinguishable from noise. Natural images frequently contain object boundaries though, across which the predictions fail. Here, we optimize predictions that can take boundaries into account, such that we can extract where boundaries are in the image from the activations. We formalize the features to be optimized as the activations in a convolutional neural network. We then form a prediction as a product of predictions from neighboring locations. Each neighbor predicts a mixture of a uniform distribution and a Gaussian around its feature vector, corresponding to the cases with a boundary between the neighbor and the predicted location and without a boundary between them. We optimize the features and the prediction such that the predicted probability is higher for the feature vector at the predicted location than for feature vectors from randomly chosen locations (a contrastive predictive coding loss). We used unlabeled natural images from the MS COCO database to learn linear and deeper feature maps.The early linear features converge towards local averages of opponent colors and Gabor-like grating patterns that point to the neighboring locations for which they are predictive. This is superficially consistent with our knowledge about early visual processing. We also evaluate the boundaries extracted from our model on the Berkeley Segmentation Database of human contour annotations. To extract contours from our model we use a computer vision method called globalization. Our contours are reasonable without further adjustment (all models F>= 0.63). Thus, we present a probabilistic model of the feature maps in early visual processing that can take object boundaries into account and can be learned without supervision.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.