Abstract
Humans have the metacognitive ability to judge the accuracy of their own decisions via confidence ratings. A substantial body of research has demonstrated that human metacognition is fallible but it remains unclear how meta-cognitive imperfection should be incorporated into a mechanistic model of confidence generation. In this study, we show that the psychometric properties of metacognitive measures along with empirical zROC functions can be used to reveal the mechanisms underlying metacognitive imperfection. We found that contrary to what is typically assumed, metacognitive imperfection depends on the level of confidence: across five different published data-sets and four different measures of metacognition (meta-d’, Mratio, phi, and Type-2 AUC), metacognitive ability decreased with higher confidence ratings. To understand the nature of this effect, we collected a large dataset of 20 subjects completing 2,800 trials each and providing confidence ratings on a continuous scale. Our findings demonstrated robustly concave zROC curves, despite a decades-old assumption of linearity. These results suggest that noise during confidence generation increases for higher confidence criteria, thereby making higher confidence ratings less reliable. We incorporated this mechanism into a new process model of confidence generation, which assumes the existence of lognormally-distributed metacognitive noise – that is, noise affecting the confidence ratings but not the perceptual decision – whose variance scales with the mean. The model outperformed competing models either lacking metacognitive noise altogether (average AIC difference = 29.70) or featuring Gaussian metacognitive noise (average AIC difference = 17.39). Finally, our model also yielded a measure of metacognitive ability that is independent of confidence levels unlike all currently popular measures such as meta-d’, Mratio, phi, Type-2 AUC, or Type-2 d’. These findings establish an empirically-validated model of confidence generation, have significant implications about measures of metacognitive ability, and begin to reveal the underlying nature of metacognitive imperfection.