Abstract
Combining multiple observers can improve performance. This wisdom-of-the-crowd technique harnesses the independency of different observers by averaging out randomly-distributed noise. However, recent studies have shown that individuals have idiosyncratic perceptual biases, and some observers have biases that are more similar to each other. This presents the possibility that the choice of pairs when employing two observers in the same task could be optimized by measuring the perceptual biases for each observer and coupling the most independent pairs. Here we tested this hypothesis through a series of studies ranging from low-level visual localization to high-level emotion recognition tasks. In Experiment 1, observers adjusted a cursor to indicate the locations of briefly-presented noise patches. In Experiment 2, observers adjusted morphed shapes to match previously-seen shapes. In Experiment 3, observers used a cursor to dynamically track and report the emotion of a target actor in Hollywood movies. Stimulus-specific perceptual errors (the difference between perceptual report and corresponding ground truth) were measured for each observer. Emotional ground truth was defined as the consensus of an independent group of observers. We calculated the correlation between the perceptual errors for every pair of participants, within each task. To test our hypothesis, we combined the responses between each least-correlated pair of observers, using an independent set of data. This yielded significantly higher accuracies in all three studies compared to the performance of single observers or random pairs. Our results demonstrate that pairing observers according to their perceptual biases rather than sensitivity can also optimize the paired-observer performance, and this technique can be widely applied even to situations where ground truth is subjective rather than objective, such as emotion recognition. Our findings highlight the importance of measuring individual differences in perceptual biases, and they have implications for critical applications such as TSA and radiological screening.