Overconfidence and Optimism: The Three Kinds and What Each Costs

Overconfidence names three separate things. Overestimation is thinking one's own outcome will be better than it will. Overplacement is thinking oneself better than other people. Overprecision is being too sure of a number, so the range stated is too narrow. The three are barely related, they need different corrections, and it is overprecision that quietly does the damage.

The whole account rests on a split that most writing about the subject never draws. Treating overconfidence as one trait, a sort of loud personality that some people have and others do not, makes it impossible to do anything about. Split into three, it stops being a character flaw and becomes three separate errors, each with its own measurement and its own correction. Moore and Healy set the split out in The Trouble with Overconfidence, published in Psychological Review in 2008, and it is the single most useful thing anybody has done with the word. Everything below follows from taking it seriously.

What are the three different things the word overconfidence names?

The ordinary use of the word comes first, and that use is the one to unlearn. When somebody says a person is overconfident, they usually mean that person thinks too highly of themselves compared with everybody else. Thinking too highly of oneself is one of the three, and it happens to be the one that costs the least. The other two are quieter, they do not sound like arrogance at all, and one of them is where the money goes.

The three, stated as plainly as they can be stated, run as follows. Overestimation is about a person's own performance in absolute terms: the outcome expected is better than the evidence about people in that situation supports. Overplacement is about performance relative to others: the expectation is to come out ahead of the field. Overprecision is about a number: a range is stated as containing the true value, and the range is too narrow. The first two are judgements about a person and only the third is a judgement about a quantity. A quantity can be tested cheaply and repeatedly; a person cannot.

Take it out of finance for a moment. A cook preparing food for a wedding of two hundred guests can go wrong in three unrelated ways. Overestimation: he thinks the cooking will take four hours when work of that size normally takes six. Overplacement: he thinks he is a better cook than the other caterers in the same street. Overprecision: he says that between eighteen and twenty two kilograms of rice will be needed, when the honest range given what he actually knows runs from fourteen to twenty eight. Three different errors. The third one is the one that leaves the kitchen short at nine in the evening.

What is overestimation, and how is it caught?

Overestimation is the error of expecting one's own outcome to beat the base rateHow often something happens across all the comparable cases, before anything specific about the case at hand is taken into account.. Overestimation shows up wherever somebody predicts how long a job will take, how much a renovation will cost, or how much will be saved by the end of a year. The prediction is not random. The error runs in one direction: shorter, cheaper, better.

The correction is unglamorous and it works, a rare combination. The record of comparable cases comes first and becomes the starting point, and only then does an adjustment follow for whatever is genuinely different about the case at hand. Most people do the reverse. The estimate gets built from the inside, feature by feature, and then never checked against what actually happened to everybody else who did something similar.

Meera Sundaram, an invented investor, put Rs 25,000/- a month in by standing instruction against a stated goal of Rs 40,00,000/- for education in eleven years. Whether that plan runs short is an overestimation question and nothing else. The answer comes from the arithmetic of contributions and from what happened to comparable savers, not from how she feels about her chances. A plan either has enough in it or it does not, so overestimation is the cheapest of the three to detect: the arithmetic answers without needing anybody to be honest about themselves.

What is overplacement, and why is it the cheapest of the three?

Overplacement is the belief that one is better than other people at something. Overplacement is the finding everybody has heard: in a room asked how many are above average drivers, far more than half the hands go up. Svenson set this out in Acta Psychologica in 1981, asking drivers to rank themselves against others on skill and on safety, and finding the distribution of self rankings sitting well above where a distribution of rankings can actually sit. The arithmetic of a ranking will not permit that distribution, so the result is both beautiful and genuinely a bias.

Overplacement does very little work on its own. Acting on a belief of being better than others requires knowing who the others are and how many of them there are. Most decisions never make that comparison. When a household chooses between two schemes, nobody is competing with anybody. The belief that one is sharper than a neighbour has nowhere to attach itself, so it sits there costing nothing.

Overplacement also reverses on hard tasks. Most retellings drop that detail. On easy tasks people place themselves above others, and on genuinely difficult and unfamiliar ones they place themselves below. Moore and Healy showed that the direction flips with difficulty in exactly this way. A word whose supposed effect reverses depending on how hard the task is cannot be a stable trait, and that alone should stop anybody treating overconfidence as one thing.

What is overprecision, and why is it the expensive one?

Overprecision is being too sure of a number. Somebody states an intervalThe range within which somebody says the true value probably lies, given as a low figure and a high figure., says they are ninety per cent sure the true value lies inside it, and the interval turns out to be far too narrow to deserve that claim. Overprecision is the least dramatic of the three to describe and by far the most consequential, for one structural reason: almost every decision that involves money involves a number that somebody is more or less sure about.

Overprecision is also the only one of the three that can be measured on a person in an afternoon, and measured over and over. Ask a set of questions where a true answer exists. For each one, ask not for a guess but for a low figure and a high figure such that the person is ninety per cent sure the answer sits between them. Then count. If the person were well judged, ninety of every hundred intervals would contain the answer. The count has a name, the hit rateThe share of stated intervals that actually turn out to contain the true answer., and comparing it to the stated ninety is the whole of calibrationWhether stated confidence matches delivered accuracy. Ranges called ninety per cent certain should contain the answer ninety per cent of the time..

The repeated finding, across decades of studies of this exact form, is that intervals offered as ninety per cent certain contain the answer about half the time. That is not a small miss. The gap between what was claimed and what was delivered runs about forty points, and it does not close much with expertise, with education, or with being told about it beforehand.

Why are the three barely related, and can they run opposite in one person?

The three can run opposite in one person, and commonly do. The three components measure different things and they do not move together in the same person. The split is worth having for that reason, rather than merely tidy. One person can hold a well judged view of where they stand against others while stating absurdly narrow ranges. Another can be modest about every number and still expect their own outcome to beat what happened to comparable people.

The engineer who says the job will take five weeks when comparable jobs took nine, and who will freely say that half the engineers in the office are better than her, is high on overestimation and low on overplacement in the same sentence. The two are not in conflict. Each answers a different question: one asks about her outcome against a rate, the other about her rank against a field.

Treating overconfidence as one trait is what makes it untreatable: a correction aimed at the wrong component does nothing, and there is no single instrument that measures a person on all three at once. Correcting a plan that runs short means going to the base rate. Correcting a claim about ranking means counting the field. Correcting a number stated too surely means going to the width of the interval. Three targets, three tools, and nothing transfers between them.

Optimism Bias: how does it differ from all three?

Optimism bias is the expectation that good things will happen to oneself more often, and bad things less often, than they happen to comparable people. Weinstein set it out in the Journal of Personality and Social Psychology in 1980, asking people to rate how likely various events were to happen to them compared with others of the same age. The pattern was one sided: pleasant events rated more likely for oneself, unpleasant events less likely. Not a random spread of self flattery, a tilt.

The difference between optimism bias and the three is one of kind, not of degree. The three components of overconfidence are all statements about how sure a person is, or about how good a person is. The three are judgements about one's own judgement. Optimism bias is a statement about the world: about which events will occur, quite apart from anything the person does or decides. Somebody can state perfectly well judged ranges, hold an accurate view of their own rank, and still expect the world to be kind to them specifically. Optimism therefore sits beside the three rather than inside them.

Optimism bias overlaps most with overestimation, and separating those two is where people get stuck. Overestimation is about performance: the expectation of doing the work faster than it will actually be done. Optimism bias is about circumstances: the expectation that the delivery will arrive on time, the tenant will pay, the illness will pass over. One is a claim about a person's own output, the other is a claim about events that happen to that person. The correction differs too. Overestimation is corrected by finding the ordinary rate for work of that kind. Optimism bias is corrected by asking what happened to comparable people in comparable circumstances. The two corrections read different tables entirely.

How narrow is a narrow range, and what does it actually capture?

Saying that half of ninety per cent intervals contain the answer is true but hard to feel. Turn it into width instead. Width is something a person can picture. Suppose there is some width of interval that would genuinely capture the answer ninety per cent of the time. Call that the required width, and set it at 1.0. Now ask what a narrower interval captures.

Working it through on the assumption of an ordinary bell shaped spread, an interval at 1.0 times the required width captures 90.0 per cent, the figure claimed. At 0.8 times it captures 81.2 per cent. At 0.6 times, 67.6 per cent. At 0.4 times, 48.9 per cent. At 0.2 times, only 25.8 per cent. Go the other way and at 1.5 times it captures 98.6 per cent. Somebody delivering about half is setting intervals roughly four tenths as wide as they should be. The phrase too sure of a number suggests a far smaller error of width than that.

Read down the shortfall column and notice where the damage arrives. Going from 1.0 to 0.8, a narrowing of a fifth, costs 8.8 points. Going from 0.6 to 0.4, another narrowing of the same absolute size, costs 18.7 points. The curve is steep at the left and flat at the right, so the last part of the narrowing is where most of the loss lives. The steepness on the left is the reason a person who feels only slightly too sure can still be delivering half of what they claim.

How Overconfidence Can Affect Investment Decisions: what did the log measure?

Now attach it to something. The Palash decision log is an invented record of 240 decisions taken by 60 investors over eight quarters at Palash Advisory Services Private Limited, and the 60 investors in it were sorted into five groups of twelve by how much they traded. Annual turnoverHow much of a holding is bought and sold in a year, stated as a percentage of the holding. Turnover of 100 per cent means the whole thing changed over once. ran at 9 per cent in the quietest quintileOne fifth of a group, once the group has been sorted and cut into five equal parts. and at 210 per cent in the busiest, a difference of more than twenty times.

Here is the part worth slowing down for. Their gross returnThe return before dealing charges, spread and tax are taken out. The figure measures what the choices produced, before the cost of making them. figures were 11.2, 11.0, 11.1, 10.9 and 11.0 per cent across the five groups. The whole spread is 0.3 points, from 10.9 to 11.2, close enough to nothing that no story about better selection survives it. Their costs ran 0.3, 0.6, 1.5, 2.5 and 4.1 points, a spread of 3.8 points. Net of those costs the five groups returned 10.9, 10.4, 9.6, 8.4 and 6.9 per cent, a spread of 4.0 points. The gross figures sit inside 0.3 points of each other while the net figures run 4.0 points apart, so what separated the groups was what the deciding cost, not what was picked.

Connect that back to overprecision rather than to the other two. Deciding to act on a number requires being sure enough of it that acting seems worth the cost of acting. A narrow range makes a difference look real, so somebody whose stated ranges are four tenths as wide as they should be will find far more numbers that look settled enough to act on. Barber and Odean connected overconfidence to trading volume in Boys Will Be Boys, published in the Quarterly Journal of Economics in 2001. The 3.8 points of extra cost in this invented log is what that certainty was charged, and the 0.3 point gross spread is the evidence that it was not earned.

Sixty investors over eight quarters is enough to show a mechanism and nowhere near enough to size it. The log measures what the extra deciding cost these particular sixty, and no price appears anywhere in it.

Why does gathering more information make overprecision worse?

Here is the part that catches careful people, and it catches them precisely because they are careful. Somebody is challenged on a number. Their instinct is to go and find out more: read the second document, pull a longer history, ask a third person. The instinct is admirable and it makes the problem worse.

The reason is that added information raises confidence much faster than it raises accuracy. Each new item is one more thing known, so it feels like a reason to narrow the range. But most new items are correlated with what was already known: the third article repeats the second, the longer history contains the shorter one, the third person read the same document. So the interval tightens while the answer does not improve. The range narrows because the reader feels better informed, and the true uncertainty narrows hardly at all, so the gap between claimed and delivered widens with every hour of extra work.

What correction works, and what only appears to?

The correction that works runs the other way and feels wrong while it is being done. There are three steps and the order is the whole of it.

Step one is to state the range before gathering, not after. A range stated first is a record of what was actually known at the start, and it is much harder to slide into a narrow one before the reading has produced the feeling of being informed. Step two is to widen it on purpose. Not by a token amount: if the repeated finding is that intervals land at about four tenths of the width they needed, then roughly doubling the width first written down is a correction of the right size rather than a gesture. Step three, the one that does the real work, is to write down what result would put the answer outside the range.

Somebody who cannot name what would fall outside their range has not stated a range at all, they have stated a number with decoration around it. That single question is the cheapest test available and it takes about twenty seconds. The question forces the person to describe, in concrete terms, a world where they are wrong. Describing that world is the one thing a narrow interval never requires.

Now the correction that only appears to work: being told about the effect. Being warned that people state ranges too narrowly, and then being asked for a range, produces almost no improvement. Neither does confidence in one's own carefulness, nor experience in the subject. The reason is that the mechanism is not ignorance of the finding, it is that a narrow range feels correct from the inside, and knowing about the finding does not change how it feels. Only changing the procedure changes the output.

What does this account not explain?

Two limits are worth naming plainly. The first is that overprecision explains why a number felt settled enough to act on. Overprecision does not explain which way the action went, and it does not by itself explain why some holdings get sold and others kept. Direction and selling are separate mechanisms with separate measurements.

The second is that a measurement of calibration is made on ranges, on questions with true answers, under conditions where the answer can be checked afterwards. No such measurement is made of any reader here. The three components are properties of stated judgements, not verdicts about people, and any account that tells a reader they are overconfident has stopped teaching and started guessing.

Meera Sundaram, Devika Rao, Palash Advisory Services Private Limited and the Palash decision log are invented.
Educational material. Not advice on any investment, tax, budget or market position.