Abstract
Many subareas of psychology face a replication crisis; perhaps because past empirical research used inappropriate sampling, analysis, or reporting methods. These questionable research practices impeded scientific progress by misleading researchers about effects and hindering theory development. To date, vision science was rarely mentioned in this context, but we show that many investigations of object-based attention seem to have similar problems. We applied the Test for Excess Success (TES) to each of 37 identified articles related to object-based attention with four or more experiments. Using the reported statistics and sample sizes, the TES estimates the probability that a replication of the studies in an article would produce the same degree of success as the original set of studies. For 19 of the articles (51%) this replication probability fell below 0.1, meaning that if the effects are real and of a similar magnitude as reported, it is unlikely that similar studies, with the same sample sizes, would reproduce the original results. Non-replicable results should not be trusted, so many of the articles do not provide adequate support for their conclusions. New studies will be required to properly examine the effects, and such studies will require much larger sample sizes or better designs. To investigate object-based attention with the classic two-rectangles paradigm, we ran an online study (n=274, estimated power of 0.97) and found a significant 14 millisecond reaction time advantage when a cue and target appear in a common rectangle compared to when they appear in different rectangles. Based on this finding, we suggest that studies of object-based attention require sample sizes nearly ten times larger than what are commonly used.