December 2022
Volume 22, Issue 14
Open Access
Vision Sciences Society Annual Meeting Abstract  |   December 2022
Optimizing Wisdom-of-the-Crowd by Measuring Idiosyncratic Perceptual Biases
Author Affiliations & Notes
  • Zixuan Wang
    University of California, Berkeley
  • Mauro Manassi
    University of Aberdeen, UK
  • Zhimin Chen
    University of California, Berkeley
  • David Whitney
    University of California, Berkeley
  • Footnotes
    Acknowledgements  This work was supported in part by the National Institutes of Health [grant numbers R01 CA236793].
Journal of Vision December 2022, Vol.22, 4336. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Zixuan Wang, Mauro Manassi, Zhimin Chen, David Whitney; Optimizing Wisdom-of-the-Crowd by Measuring Idiosyncratic Perceptual Biases. Journal of Vision 2022;22(14):4336.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Combining multiple observers can improve performance. This wisdom-of-the-crowd technique harnesses the independency of different observers by averaging out randomly-distributed noise. However, recent studies have shown that individuals have idiosyncratic perceptual biases, and some observers have biases that are more similar to each other. This presents the possibility that the choice of pairs when employing two observers in the same task could be optimized by measuring the perceptual biases for each observer and coupling the most independent pairs. Here we tested this hypothesis through a series of studies ranging from low-level visual localization to high-level emotion recognition tasks. In Experiment 1, observers adjusted a cursor to indicate the locations of briefly-presented noise patches. In Experiment 2, observers adjusted morphed shapes to match previously-seen shapes. In Experiment 3, observers used a cursor to dynamically track and report the emotion of a target actor in Hollywood movies. Stimulus-specific perceptual errors (the difference between perceptual report and corresponding ground truth) were measured for each observer. Emotional ground truth was defined as the consensus of an independent group of observers. We calculated the correlation between the perceptual errors for every pair of participants, within each task. To test our hypothesis, we combined the responses between each least-correlated pair of observers, using an independent set of data. This yielded significantly higher accuracies in all three studies compared to the performance of single observers or random pairs. Our results demonstrate that pairing observers according to their perceptual biases rather than sensitivity can also optimize the paired-observer performance, and this technique can be widely applied even to situations where ground truth is subjective rather than objective, such as emotion recognition. Our findings highlight the importance of measuring individual differences in perceptual biases, and they have implications for critical applications such as TSA and radiological screening.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.