Decision Quality: Judging the Decision, Not the Outcome

Decision quality asks one thing only: was the reasoning sound on the information that existed when the choice was made. An outcome is one draw from a range nobody can see, so judging by it means judging on a sample of one. The two come apart in both directions, and the direction that does the damage is a poor process that happened to work.

There is a sentence almost everybody agrees with and almost nobody lives by: a good decision and a good result are two different things. The agreement lasts until a result actually arrives. Then the result is sitting there, large and specific and impossible to ignore, and the reasoning that produced it has gone quiet. The distance between agreeing with that sentence and behaving as though it were true is where the whole difficulty lives. Closing that distance takes something more definite than good intentions. A short set of questions, all of them answerable before anything has happened at all, is one way of doing it.

What is decision quality, and how is it different from a good result?

Decision qualityWhether the thinking behind a choice holds up on the information that existed when it was made. asks one question and only one: given what could be known when the choice was made, was the reasoning sound? Not was it clever, not was it bold, not did it feel right. Was it sound, and could somebody else pick it up afterwards and check it. Outcome qualityHow the decision actually turned out. asks something completely different: how did it turn out. Both are real questions. Only one of them is about the decider.

The pattern is clearer away from money, where nothing is at stake for the reader. A guest leaves a wedding at one in the morning, having slept two hours the night before, and drives ninety minutes home. The drive ends safely. Because the ending was fine, almost nobody telling that story afterwards describes it as a bad decision. Yet nothing about the ending changed the state of the driver at the moment the key turned. The decision was taken with a tired person, a dark road and no plan for what happens if concentration goes. Arriving home safely was one of the things that could have happened, and it happened. Arriving home safely is not a report on the driving.

Turn it round and it is just as clear. A household saves for eleven years towards a stated goal, chooses a boring monthly instruction they will not be tempted to interrupt, keeps two months of outgo aside so a bad month never forces a sale, and writes down why. Then something outside all of it goes wrong and the goal is missed. The reasoning did not become poor when the result arrived. The reasoning was the same reasoning the whole time.

So a decision, at the moment it is taken, is a choice made across a set of futures nobody can see. Some of those futures are kind to it and some are not. One of those futures arrives. The reasoning is the thing the decider was in charge of; the draw is the thing the decider was not. A judgement that scores the decider on the draw is scoring the decider on somebody else's work.

The phrase repays being precise about. Decision quality is not a softer, kinder version of outcome quality for people who did badly. Decision quality is a separate measurement of a separate thing, taken at a separate time, and it is the only one of the two that exists on the day it is needed.

What is resulting, and why does explaining it not stop it?

ResultingJudging a decision by its outcome rather than by its reasoning. is the name for judging a decision by how it came out. The word is Annie Duke's, from Thinking in Bets, 2018, and it is a useful word precisely because the thing it names is so ordinary that without a word for it nobody notices it happening. A trade worked, so it was a good call. The dinner was excellent, so the booking was inspired. The candidate turned out badly, so hiring them was a mistake. Each of those sentences has skipped a step so smoothly that the tape has to be slowed down to see it.

Resulting is hard to shift because resulting is not a piece of ignorance, and being told about it does not cure it. Jonathan Baron and John Hershey showed as much in Outcome Bias in Decision Evaluation, published in the Journal of Personality and Social Psychology in 1988. Baron and Hershey gave people descriptions of decisions, stated plainly the information that had been available when each decision was taken, and then attached different results to the same description. The ratings people gave separated according to the result. The information at the time was identical and it was written down in front of them, and the ratings moved anyway.

Read the finding again, slowly. The result is stronger than it first sounds. The experiment is not about people forgetting what was known at the time. The people rating those decisions had the information in their hands and were told outright that it was identical in both versions, and they still rated the decision differently once a result was attached. Resulting survives being explained, so an explanation of it is not a cure and was never going to be one. What works is not knowing about it; what works is scoring the decision on something the outcome cannot reach.

The gap between the two sets of ratings has a direction and a size, and only the direction carries over from one experiment to the next. How far apart the ratings went depends on the wording of the particular scenarios, so a number put on the size would be an invented measurement rather than a finding.

Why does the result get to be so loud? Partly because it is definite. The reasoning is a paragraph of qualified sentences about what might happen; the result is a single number that has already happened. Partly because a result arrives with feelings attached, and a paragraph does not. And partly because reviewing a result costs nothing while reconstructing what somebody knew four months ago is work, and the work is skipped unless it was written down at the time.

Why is one outcome a sample of one?

The arithmetic underneath all of this is not difficult arithmetic. When a decision is taken, there is a spread of things that could happen. One item from that spread arrives. To judge the decision by what arrived is to estimate the whole spread from a sample of oneA single observation, from which no general conclusion follows.. How wide the band of pure chance results grows when the number of observations is small is set out under representativeness. One is the smallest number there is, so the band is at its widest, and a judgement resting on it has no evidential content at all. Not weak content. None.

An everyday version. A street vendor outside an office building takes one afternoon off and misses the busiest day of the month. Was taking the afternoon off a poor decision? One afternoon cannot settle it. Thirty afternoons might, and even then only if something is written down about which afternoons were chosen and why. The single afternoon is loud because it is the one that happened, and it is silent because it is one.

The invented Palash 100 index shows the same thing in a shape that is easy to look at. The index opens at 100.0, and its eight quarter ends run 118.0, 131.0, 112.0, 104.0, 116.0, 124.0, 121.0 and 127.0. The peak is 131.0 at the second quarter end and the low is 104.0 at the fourth. Somebody who happened to read the number at the second quarter end saw a rise of 31.0 per cent from the open. Somebody reading at the fourth saw a rise of only 4.0 per cent, and from the peak a fall of 20.6 per cent. The path is one path. The reading depends entirely on which single moment supplied it.

Take four consecutive readings of the same invented index and the point stops being abstract. From the open to the first quarter end the Palash 100 rose 18.0 per cent, from there to the second a further 11.0 per cent, then it fell 14.5 per cent to the third and a further 7.1 per cent to the fourth. Four moves, one path, and any single one of them read on its own describes a rise or a fall nobody actually lived through.

The same trap sits inside the logged record. Across the eight quarters, 84 of the 240 logged decisions were sales, and 61 of those sales, or 72.6 per cent, were of positions standing in gain. Left there, the figure sounds like a finding about a preference for winners. The figure is nothing of the sort, because it ignores what was there to be sold. Of the 240 positions open across the period, 108 stood in gain and 132 in loss. Gains were realised at 61 of 108, or 56.5 per cent, and losses at 23 of 132, or 17.4 per cent, a ratio of 3.2 to 1. The ratio is the measurement and the 72.6 per cent is the trap, and the reason the trap works is the same reason resulting works: a single loud number, read on its own, with nothing underneath it.

What does Process Quality vs Outcome Quality look like, set side by side?

Most of the confusion between the two comes from only ever meeting them as a pair, so each one has to be defined on its own before they can be contrasted. So take them one at a time.

Process quality is a property of how a decision was made. Process quality is made of four things that could be photographed on the day: whether a reason was recorded, whether the reason committed to something that could later be found false, whether the standard being used was a fact about the thing being decided or a fact about the decider, and whether anybody was given the job of arguing the other side. None of those four needs the future. All four are finished by the time the decision is taken. Process quality is therefore complete on the day, and it is complete whether the decider is a committee of nine or a person at a kitchen table with nobody to consult.

Outcome quality is a property of what turned up. Outcome quality is made of the result itself, the costs that were taken out of it on the way through, and the stretch of time it happens to be measured over. Two of those three have nothing to do with the decider either. A result measured to one date and the same result measured to a date three months later are two different outcome qualities from one decision, and nobody decided the date. Outcome quality is real, it matters, and it is what the money actually does. Outcome quality is simply not a report on anybody’s thinking.

Now the two can be put side by side, and the contrast is sharper than most readers expect. Process quality is knowable at the time and outcome quality is not, and that single difference is what decides which of the two can be steered by. Everything else in the comparison follows from it.

A short warning about the table above. Reading it, it is tempting to conclude that outcome quality does not matter. Outcome quality matters enormously; it is the only thing that ever pays for anything. The narrower claim is worth stating exactly. Everything that happened after the reasoning stopped is mixed into the outcome, so the outcome cannot serve as a measurement of the reasoning.

Which of the four combinations does the damage?

Sound and poor reasoning set against good and poor results give four combinations. The four combinations are not equally common and, far more importantly, they are not equally dangerous. Most treatments stop at drawing the grid. The grid is not the teaching; the asymmetry inside it is.

Sound reasoning with a good result is the pleasant cell. There is nothing to correct, so it teaches almost nothing. Poor reasoning with a poor result is the obvious cell: the result hurts, somebody goes back to the entry, the fault is visible and it gets fixed. Sound reasoning with a poor result is the painful cell, and its risk is over-correction, a working method thrown away because one draw went against it. All three of those cells get looked at.

Poor reasoning with a good result is the cell that does the damage, and it does the damage precisely because it is never examined. The cell arrives looking like a success and is described as a good call. Nobody changes something that worked, so the reasoning gets repeated, and each repetition collects more evidence for a method that has none. Nobody in the history of reviews has opened a file and said, that one made money, so find out what went wrong.

There is a second, quieter cost. Because the rewarded cell is never opened, the person in it learns the wrong lesson with total confidence. The person now believes something that is not true about how the world works, and believes it on the strength of evidence lived through personally. Lived evidence is the most persuasive kind there is and the hardest to argue with.

How is a decision scored before the outcome exists?

Everything so far has been diagnosis. Here is the method, and the test it has to pass is severe: every question in it must be answerable on the day, by somebody who has no idea what is going to happen. A scoring method that needs the outcome is not a scoring method. Such a method is the outcome wearing a different hat.

Four questions, offered as one workable set rather than as a standard issued by anybody, and short enough to be answered in the time it takes to make a cup of tea.

Question one, the written reason, is the one everything else depends on. Russo and Schoemaker made the case for the written record in Decision Traps, 1989, and the argument is simply that memory of a reason is reconstructed after the fact and reconstructed in the direction of what happened. If it was not written, there is nothing to score. Atul Gawande, in The Checklist Manifesto, 2009, makes the related point about why a fixed short list beats a good intention: the list runs the same way on a tired day as on a sharp one.

Question two, the date, converts a hope into something that can expire. Without a date, a position can be held indefinitely on a reason that stopped applying months ago, and nobody notices because nothing ever falls due.

Question three, the ending observation, is falsification wearing working clothes. Karl Popper argued in The Logic of Scientific Discovery, 1934, that what separates a claim worth holding from one that is not is whether anything could show it to be false. Applied to a decision, the question is: what would have to be seen to conclude that the reasoning was wrong, and would it actually be seen? A reason with no such observation attached cannot be wrong. Being unable to be wrong sounds like a strength and is the opposite of one.

Question four, the reference pointThe level a change is measured against, which may be a fact about the decider., catches the most common single fault. Daniel Kahneman and Amos Tversky showed in Econometrica in 1979 that people evaluate outcomes as gains and losses against a reference level rather than as final positions, and that the reference level is often just where they happened to start. A purchase cost is exactly that. A purchase cost is a fact about the decider’s own history and carries no information whatsoever about the thing being held. If the standard being applied is a number out of the decider's own past, question four scores zero.

Two things stand out about the set. The four questions carry no opinion about what should have been decided: the moment a scoring method starts marking the conclusion, it has become the reviewer’s own view of the world with a number attached. The four questions also say nothing about money. The same four questions run on a decision to change jobs. Was a reason written down? Was a date set by which the new work should have felt like the right move? Was an observation named that would say it was not working? Is the standard a fact about the job, or is it the salary held before, a number out of the decider’s own past? The method does not know it is being applied to a career, and that is the test it has to pass to be worth anything.

What does the scoring do to the decision of 12 October?

Now run the four questions on one real entry, using nothing that was not on the table that day. The invented Palash decision log records that on 12 October, Meera Sundaram sold Suvarna Chemicals Limited whole at Rs 4,60,000/- against a cost of Rs 4,00,000/-, booking Rs 60,000/-, or 15.0 per cent. In the same sitting she kept Kesari Logistics Limited, which had cost Rs 3,00,000/- and stood at Rs 1,95,000/-, down Rs 1,05,000/-. She recorded that she would sell it when it got back to Rs 3,00,000/-.

The rest of the holding shows where that decision sat. On 30 September the holding stood at four positions against a cost of Rs 13,00,000/-. The Vindhya index scheme had gone from Rs 3,00,000/- to Rs 3,36,000/-, up 12.0 per cent, and Suvarna Chemicals Limited from Rs 4,00,000/- to Rs 4,60,000/-, up 15.0 per cent. The Nilgiri mid-cap scheme had fallen from Rs 3,00,000/- to Rs 2,55,000/-, down 15.0 per cent, and Kesari Logistics Limited from Rs 3,00,000/- to Rs 1,95,000/-, down 35.0 per cent. Two positions stood in gain, worth Rs 7,96,000/- between them, and two stood in loss, worth Rs 4,50,000/-. The four values total Rs 12,46,000/- against the cost of Rs 13,00,000/-, so the holding was down Rs 54,000/-, or 4.2 per cent.

That pairing has two named mechanisms in it. Realising the position standing in gain while holding the position standing in loss is the disposition effect, set out under the disposition effect and measured by Terrance Odean in the Journal of Finance in 1998. Using the purchase cost as the level everything is measured against is the reference point at work, set out under prospect theory and drawn from Kahneman and Tversky in Econometrica in 1979. Neither of those is what is being scored here. The scoring does not care which holding she picked; it cares whether the reasoning behind the pick could be checked.

One of four, or 25.0 per cent. Now the part that matters more than the number. Every one of those four verdicts could have been reached on 12 October, before a single thing had happened. Availability on the day is the entire practical difference between the two measurements, and it is why the scoring is worth the four minutes it takes.

A scoring method gets misused at exactly this point. State the limit of the score plainly. A score of 1 of 4 does not say the decision was wrong. The score says the reasoning was not in a condition to be checked. Those are separate claims and only the second one is being made. Keeping Kesari Logistics Limited on 12 October might have been entirely sensible. On this record there is no way to find out, because the entry contains no date at which anybody would look again and no observation that would have said the reason had failed. A 1 of 4 decision can be followed by an excellent result, and often is. The mismatch is not a defect in the scoring. The mismatch is the thing the scoring is built to survive.

The record does show what happened next, and it is worth stating so long as it is stated with its limits. By 31 March following, Suvarna Chemicals Limited had risen 8.0 per cent after the sale, so the Rs 4,60,000/- released would have been Rs 4,96,800/-, a forgone Rs 36,800/-. Kesari Logistics Limited had fallen a further 20.0 per cent, from Rs 1,95,000/- to Rs 1,56,000/-, a further Rs 39,000/- lost. The pair costs Rs 75,800/-. One case is not evidence that any rule works, and this one is offered as an illustration of the arithmetic and nothing more.

Notice what would have happened in the second panel. Had Kesari Logistics Limited recovered, the entry would have been remembered as patience rewarded, filed as a success, and used as the model for the next one. Same missing date. Same missing observation. Same purchase cost doing the work of a standard. Poor reasoning with a good result is the dangerous cell from earlier, arriving in a specific record on a specific day.

What is a Devil’s Advocate, and what makes the role fail?

A devil's advocateSomebody assigned to argue against a case within the group. is somebody given the job of arguing against a case from inside the group considering it. The phrase comes from a formal office that existed to put the case against a candidate before a decision was confirmed, and the idea it carries is a good one: a case that has never been argued against has not been tested, it has only been agreed with. Applied to the four questions above, the role is what tends to produce an answer to question three. The person arguing the other way is exactly the person who will say out loud what would have to be seen for the case to be wrong.

The role is one of the standard corrections, and Baruch Fischhoff set out the general shape of such corrections in his chapter on debiasing in Judgment under Uncertainty, 1982: a correction has to target a specific failure, and it has to run at the moment the failure happens rather than afterwards. A devil’s advocate targets one specific failure, a case that goes unchallenged because everybody in the room already agrees.

The role fails when everybody in the room knows it is a role. The objection arrives already labelled. The objection is heard not as somebody’s view but as somebody performing an assigned duty, and an argument heard that way is discounted before it has been weighed on its content. The person is thanked, the point is noted, and the case proceeds exactly as it would have done. Worse, the group now feels it has tested the case, so the appearance of a challenge has replaced the challenge. Nobody has measured how often an announced role fails this way. Treat the pattern as a regularity people report from inside such rooms rather than as a finding.

So what does work? Four adjustments, and each one attacks the labelling rather than the argument. Objections are asked for in writing before any discussion, so nobody has heard the room first. The job rotates, so it is never one person's known position. The request is for the opposite conclusion rather than for an attack. Charles Lord, Mark Lepper and Elizabeth Preston showed in the Journal of Personality and Social Psychology in 1984 that instructing people to consider the opposite reduced the pull of what they already believed, and considering the opposite is a different instruction from telling somebody to object. And for a person deciding alone, with no group and nobody to assign, the case against is written out by hand before deciding, since a sheet of paper does not know it is playing a part.

If outcomes cannot be used, what is left to judge by?

The heading is the fair objection to everything above, and it deserves a concrete answer rather than a principle. Take away the result and a surprising amount is left, but only if somebody wrote things down at the time. Everything on the list below is a fact about the record rather than about the world.

Was there an entry at all. Did it contain a reason in words. Did the reason commit to something that could later be found false. Was a date set. Was the standard a fact about the thing decided, or about the decider. Was anybody, including yourself, given the job of arguing the other way. And did the review actually happen on the date it was supposed to. Seven checkable things, and the result is not among them.

The logged record shows how thin most entries are on exactly these points. A written reason exists on 84 of the 240 decisions, or 35.0 per cent. And 71 of the 240, or 29.6 per cent, were taken within 48 hours of a news item. Of the 96 buys, 41 followed a media mention within three days, or 42.7 per cent, against 11.0 per cent of the eligible list being mentioned at all in a given week. Every one of those counts measures the record and the timing, and not one of them measures a return.

Something did move after 4 November, when 20 of the 60 adopted a written checklist. Across the fifth to eighth quarters those 20 recorded a written reason on 34 of 41 decisions, or 82.9 per cent, against 19 of 63, or 30.2 per cent, for the other 40. Their realisation ratio, gains realised against losses realised, fell from 3.2 to 1.6. Both of those are changes in behaviour and in what the record contains, and neither of them is a change in return. The distinction is not a technicality; it is the difference between what the record can support and what somebody would like it to say.

What is not claimed about results?

One limit belongs at the end, stated plainly. A record of this shape cannot show that a better way of deciding produces a better return. The claim is not measured by such a record, not implied by it, and not quietly assemblable from it either.

The reasons are specific rather than a general shrug about evidence. Look at each of them. The checklist was adopted on 4 November and the log was read back on 31 March following, one stretch of time with no comparison group running alongside it. The 240 decisions are spread across 60 people, four decisions each. There is no measurement of return before adoption and none after, so there is nothing on either side of a comparison that has not been made. And the 20 who adopted the checklist chose to adopt it, so they were already the kind of people who adopt a checklist. That willingness is a difference between the two groups that has nothing to do with the checklist.

So what is the process scoreA fixed set of questions answerable before any outcome exists. actually for? One thing only. The score makes the reasoning inspectable afterwards. Inspectable reasoning is worth having because a decision whose reasoning cannot be examined cannot be improved, and improving the next decision is the only use anybody has ever had for reviewing the last one. Whether examinable reasoning also earns more is a separate question needing a completely different kind of evidence, and this record is not that evidence. Concluding otherwise would be the exact error the scoring exists to correct: treating a result as though it settled a question about reasoning.

Meera Sundaram, Devika Rao, Palash Advisory Services Private Limited, the Palash decision log, the Palash 100 index, the Vindhya index scheme, the Nilgiri mid-cap scheme, Suvarna Chemicals Limited and Kesari Logistics Limited are invented.
Educational material. Not advice on any investment, tax, budget or market position.