May 2008
Volume 8, Issue 6
Vision Sciences Society Annual Meeting Abstract  |   May 2008
Circular inference in neuroscience: The dangers of double dipping
Author Affiliations
  • Nikolaus Kriegeskorte
    Laboratory of Brain and Cognition, NIMH
  • William K. Simmons
    Laboratory of Brain and Cognition, NIMH
  • Pat S. Bellgowan
    Laboratory of Brain and Cognition, NIMH
  • Chris I. Baker
    Laboratory of Brain and Cognition, NIMH
Journal of Vision May 2008, Vol.8, 88. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Nikolaus Kriegeskorte, William K. Simmons, Pat S. Bellgowan, Chris I. Baker; Circular inference in neuroscience: The dangers of double dipping. Journal of Vision 2008;8(6):88. doi:

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is subjected to detailed analysis and presentation in a publication. This inevitable selection is a major determinant of the final conclusion, and selection among a set of noisy measurements can render circular an otherwise appropriate analysis, invalidating statistical tests. The issue of circularity is particularly important in both electrophysiological and neuroimaging experiments. Here we focus on neuroimaging and argue that the field needs to adjust some widespread practices to avoid the circularity that can arise from selection. Faced with even more parallel sites than electrophysiology (typically on the order of 100 000 voxels), neuroimaging has developed rigorous methods for statistical mapping. This powerful approach avoids selection altogether: by analyzing all locations equally, while accounting for the fact that multiple tests are performed. However, selective analysis is still commonly used to focus on a particular brain region and, in fact, statistical mapping can form the basis for defining a region of interest (ROI). Further analysis of such functionally defined regions must take the selection bias into account. This problem is well understood in theory and one solution is to use independent experimental data to analyze the ROI. In practice, however, the selection bias is often ignored and important claims rely on questionable circular analyses. In order to demonstrate the problem, we apply analyses widespread in the neuroimaging literature to data known not to contain any experimental effects: functional magnetic resonance imaging (fMRI) data acquired from a resting human subject performing no explicit task. This exercise shows that widespread analysis procedures can produce spurious effects in the context of both univariate activation analysis and multivariate pattern-information analysis. We conclude by suggesting some simple guidelines for better practice in research and reviewing.

Kriegeskorte, N. Simmons, W. K. Bellgowan, P. S. Baker, C. I. (2008). Circular inference in neuroscience: The dangers of double dipping [Abstract]. Journal of Vision, 8(6):88, 88a,, doi:10.1167/8.6.88. [CrossRef]

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.