The graphical display of quantitative information is indispensible for conveying scientific data in a concise and compelling manner. However, little is known about whether our impressions of visualized data align with the actual statistical properties of the data. In other words, is the visual system a good statistician? In the current study, we presented observers with data from two distributions side-by-side, rendered with each data point visible in vertical bee-swarm plots. Observers completed only one trial in which they were asked to judge which of the two distributions (or "groups") seemed higher overall. To test sensitivity for statistical variables, we manipulated the distributions such that their comparison resulted in p-values of 0.5, 0.05, or 0.005 according to a t-test. We further manipulated whether, relative to a reference difference of p = 0.05, higher/lower p-values arose from smaller/bigger differences in the means of the distributions or more/less variance within each distribution. Finally, we evaluated the importance of domain-specific knowledge about statistics by manipulating whether the data were framed as scientific (i.e., from a biology experiment) or non-scientific (i.e., from a high-jump competition), and by assessing whether observers had received formal training in statistics. Initial results indicated that accuracy in choosing the higher distribution was above chance in all conditions (69% on average). However, accuracy was unaffected by the significance of the p-value of the difference between distributions, and this was true regardless of whether the means or variances were manipulated. Strikingly, observers who reported receiving statistical training had significantly lower accuracy than those who did not. Finally, there were hints that a non-scientific framing may boost accuracy under certain circumstances. These results demonstrate a general ability to compare distributions, but reveal a visual insensitivity to statistical variables over a dynamic range important to researchers for making scientific inferences.
Meeting abstract presented at VSS 2014