Purchase this article with an account.
Bhavin Sheth, Jasmine Patel; Human Perception of Statistical Significance and Effect Size. Journal of Vision 2015;15(12):337. doi: https://doi.org/10.1167/15.12.337.
Download citation file:
© ARVO (1962-2015); The Authors (2016-present)
Statistics are ubiquitous in the information age. While they are fundamental to scientific discovery, understanding the layperson’s intuitive grasp of statistics is necessary if scientific advances and policies are to garner popular support. We asked two questions: First, how tightly matched are our perceptions of statistical significance and statistics? Second, are perceptual judgments of discriminability driven by statistical significance or by effect size? To address the first question, we displayed scatter plots of differently colored points chosen from two gaussian distributions had different mean/variance; observers judged if the two patterns of dots were similar or different. Perceptual judgments were compared to the results of statistical tests. Observers perceived the two arrangements of dots as “different” when there was no true difference at all (Type I: false positives), but were more conservative than statistics when the level of true discriminability increased (Type II: misses). When distributions overlap in space, we seem to be, by and large, immune to statistical significance. To address the second question, we displayed two plots in which we varied the level of statistical significance and effect size in opposite directions. Each plot contained red and blue points chosen from two Gaussian distributions of different mean and variance and observers had to judge, in a binary choice task, which of the two plots had red and blue points that appeared more different. The plots were designed so that one had the larger effect size (effect size is defined as the difference in means over the pooled standard deviation: Cohen’s d), and the other the higher level of statistical significance. Over a range of effect sizes and p-values, observes chose the plot with the larger effect size as being more different. Our results suggest that effect size, rather than statistical significance, drives perceptual judgments.
Meeting abstract presented at VSS 2015
This PDF is available to Subscribers Only