On why we lack confidence in signal-detection-based analyses of confidence

Abstract

Signal-detection theory (SDT) is one of the most popular frameworks for analyzing data from studies of human behavior - including investigations of confidence. SDT-based analyses of confidence deliver both standard estimates of sensitivity (d), and a second estimate based only on high-confidence decisions - meta d. The extent to which meta d estimates fall short of d estimates is regarded as a measure of metacognitive inefficiency, quantifying the contamination of confidence by additional noise. These analyses rely on a key but questionable assumption - that repeated exposures to an input will evoke a normally-shaped distribution of perceptual experiences (the normality assumption). Here we show, via analyses inspired by an experiment and modelling, that when distributions of experiences do not conform with the normality assumption, meta d can be systematically underestimated relative to d. Our data therefore highlight that SDT-based analyses of confidence do not provide a ground truth measure of human metacognitive inefficiency. Public Significance StatementSignal-detection theory is one of the most popular frameworks for analysing data from experiments of human behaviour - including investigations of confidence. The authors show that the results of these analyses cannot be regarded as ground truth. If a key assumption of the framework is inadvertently violated, analyses can encourage conceptually flawed conclusions.