August 2023
Volume 23, Issue 9
Open Access
Vision Sciences Society Annual Meeting Abstract  |   August 2023
Why you should lack confidence in signal-detection-based analyses of confidence
Author Affiliations & Notes
  • Derek Arnold
    Perception Lab, The University of Queensland
  • Alan Johnston
    School of Psychology, The University of Nottingham
  • Joshua Adie
    Research Institute for Sport & Exercise, University of Canberra
  • Kielan Yarrow
    Department of Psychology, City University London
  • Footnotes
    Acknowledgements  ARC Discovery Project Grant
Journal of Vision August 2023, Vol.23, 5852. doi:
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Derek Arnold, Alan Johnston, Joshua Adie, Kielan Yarrow; Why you should lack confidence in signal-detection-based analyses of confidence. Journal of Vision 2023;23(9):5852.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Signal-detection theory (SDT) is one of the most popular frameworks for analyzing data from studies of human behavior – including investigations of confidence. SDT-based analyses of confidence deliver both standard estimates of sensitivity (d’), and a second estimate that is based only on high-confidence decisions – meta d’. The extent to which meta d’ estimates fall short of d’ estimates is regarded as a measure of metacognitive inefficiency, quantifying the contamination of confidence by additional noise. These analyses rely on a key but questionable assumption – that repeated exposures to an input will evoke a normally-shaped distribution of perceptual experiences (the normality assumption). We have shown, via analyses inspired by an experiment and modelling, that when distributions of experiences do not conform to the normality assumption (due to skew or excess kurtosis), meta d’ can be systematically underestimated relative to d' – even though both statistics have been informed by a common set of data subject to a common source of noise. We show that this contingency results from trying to extrapolate the shape of an entire distribution of experiences from the subset of trials that have resulted in high-confidence target categorizations. These would reside at the right-side tail of a distribution of experiences, where the effects of any difference from a normal shape are disproportionate relative to the entirety of the distribution. Our data highlight that SDT-based analyses of confidence do not provide a ground truth measure of human metacognitive inefficiency.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.