How Sure Should We Be? Understanding Confidence Calibration
A confident judgement may be accurate, mistaken or somewhere between the two. Calibration asks whether our degree of certainty is justified by how reliably our judgements turn out to be correct.
A commentator confidently announces that a proposed policy will succeed. Another thinks it will fail but expresses considerable uncertainty. Both predictions sound different. Yet neither person's confidence tells us, by itself, who is more likely to be correct.
To assess their judgement, we would need to know what each predicted, the evidence available at the time and what eventually happened. More importantly, we would need to look beyond a single prediction.
This is where confidence calibration becomes useful. The question is not whether someone sounds sufficiently doubtful, but whether their expressed certainty corresponds to their actual success over repeated judgements.
What researchers mean by calibration
In December 1977, Sarah Lichtenstein and Baruch Fischhoff reported experiments examining how well people's confidence estimates corresponded to their accuracy on general-knowledge questions.
They distinguished calibration from resolution. Calibration concerns whether predictions assigned a given probability are correct at approximately that frequency. Resolution concerns whether the judge can meaningfully distinguish more likely from less likely outcomes.
Their experiments found moderately good calibration alongside systematic biases. The findings challenge the idea that subjective certainty simply equals objective accuracy. They do not establish that people are uniformly overconfident about every subject, or that the same pattern applies directly to political predictions and digital media audiences.
The competing explanation: perhaps the task matters
One account emphasises the possibility of systematic errors in confidence judgement. A contrasting perspective asks whether some observed overconfidence reflects how researchers select questions and request judgements.
In 1991, Gerd Gigerenzer, Ulrich Hoffrage and Heinz Kleinbölting proposed a theory of probabilistic mental models. Their two experiments supported the view that apparent overconfidence can change, disappear or reverse depending on the questions and how people are asked to report uncertainty.
The accounts are not necessarily incompatible. People may make genuine judgement errors while particular experimental designs exaggerate or conceal them. Neither study justifies replacing one universal claim—“people are overconfident”—with another—“people are naturally well calibrated”.
What evidence could distinguish the accounts?
A useful comparison would ask participants to judge a broad set of questions while varying how those questions are selected and how confidence is recorded. Researchers could then compare stated probabilities with actual accuracy across sufficiently many judgements.
If miscalibration persists across representative questions and different response formats, that strengthens the case for a more general judgement problem. If the effect changes substantially with selection and format, the conditions of measurement deserve greater explanatory weight.
From psychological judgement to communication
News reports, public statements and social-media posts often communicate certainty through words such as “certain”, “likely” or “almost impossible”. Those expressions may be useful, but their meanings are not always consistent across speakers.
Media literacy asks how certainty is framed. Information literacy asks what evidence supports the claim. Digital literacy asks how a confident statement travels and whether its original qualifications remain visible. None of these questions requires dismissing confidence automatically.
One practical distinction
When someone expresses strong confidence in a consequential claim, ask what would allow that confidence to be checked. Is there a verifiable record, a clearly specified prediction or a body of previous judgements with known outcomes?
The aim is not to become less certain about everything. It is to become appropriately certain about each claim, for reasons that can withstand examination.
Key sources
Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know? Organizational Behavior and Human Performance, 20, 159–183.
Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98(4), 506–528.
Moore, D. A., & Healy, P. J. (2008). The trouble with overconfidence. Psychological Review, 115(2), 502–517.
Comments