Confidence Scores: What the Number Means and How to Threshold It

A confidence value is the component's own reading of how sure it is about one output, on its own scale. A confidence value is not the probability the answer is right, and it is not comparable across components or across field types. A threshold turns that value into a decision to accept or to route, and moving it trades wrong answers accepted against work sent to a person.

The only thing that connects a confidence value to correctness is somebody checking a sample of accepted answers by hand. Without that check the number is an ordering and nothing more: genuinely useful for deciding which answers to look at first, and worthless for saying how many of the accepted ones are wrong. Everything difficult about thresholds follows from that one gap between a number a component produces about itself and a number a person produced by looking.

What is a confidence value actually reporting?

An ordinary case makes the point. A clerk hands a photocopied form to a colleague across the desk and asks her to read out the account number. She reads it out, and then she adds something she was not asked for: "I am fairly sure, though that third digit could be a five or a six." That second sentence is a confidence value. Her sureness is a reading of her own reading. It is not evidence about the account number. Nobody has looked at the original. If she is the sort of person who is fairly sure about everything, the sentence carries almost no information at all, and that could only be found out by pulling the original and checking.

A component behaves the same way. At Sumeru Bank Limited, an invented mid-sized Indian bank running a retail loan intake chain, the field reading step returns a value between 0.00 and 1.00 beside every field it produces. The month holds 8,600 completed files at 14 fields each, so 120,400 field values arrive with 120,400 confidence valuesA component's own reading of how sure it is about one output, on its own scale. attached. The figures are the bank's own and describe one deployment in one month.

What is the number honestly good for? Ordering. Sorted by confidence value, the month's fields at the bottom of the list really are the ones most worth a person's attention first. The ordering survives almost every complaint that can be made about the numbers themselves, and it is not a small thing: an ordering is what makes a queue of 120,400 items workable at all. A confidence value is an ordering before it is anything else, and almost everything a chain does well with one is an ordering rather than a measurement.

The trouble starts the moment somebody draws a line across that ordering and calls everything above it correct. Drawing the line is unavoidable and completely reasonable. Calling what sits above it correct is the error, and it is an error that survives review meetings because nothing in the paperwork looks wrong.

What are the four things a confidence value is not?

Four denials do most of the teaching here, and they are worth learning as a numbered list because each one is a different mistake made by a different person in a different meeting.

1. A confidence value is not the probability the answer is right. A value of 0.95 is a point on a scale the component made up while producing the answer. Nothing in that arithmetic forces it to correspond to being right 95 times in 100.
2. The value is not comparable across components. The document classifier and the reading step both emit numbers between 0.00 and 1.00. The two numbers sit on separate scales that happen to share a range, in the way that two different examiners both mark out of ten.
3. The value is not comparable across field types. A six character amount and a twenty character employer name are not the same reading problem, and a value of 0.93 on each does not describe the same situation.
4. The value is not stable across a refit. Refit the component on newer documents and the values move. The acceptance barThe confidence level above which an output is taken without a person looking at it. written into the configuration does not move, because a bar is a number in a file and nothing refits it.

Every one of the four denials says the same thing in a different costume: the value is internal to one component at one moment, and every use anybody wants to make of it is external. Somebody wants to compare it to a target, to another team's figure, to last quarter, to a supplier's claim. None of those comparisons is supported by the number itself.

The second denial looks like a saving, and that is exactly why it is the one that costs money. A team that has already done the work of choosing a bar for one component has a number in hand, and carrying that number to the next component looks like reuse. Carrying the number across is not reuse. It is a fresh decision taken without any of the evidence, wearing the decimal places of an old decision that had some.

Why does a value of 0.95 not mean right 95 times in 100?

Two people mark the same pile of examination scripts. Both are asked to write a percentage sureness beside each mark. One is temperamentally cautious and writes 70 when he is nearly certain. The other writes 95 for everything he has ever got right. Their judgement might be identical. Their numbers are not. The numbers were never anchored to anything outside the marker's head, so no arithmetic on the numbers alone will show which of the two to trust.

A component is in exactly that position, and there is nothing negligent about it. The value came out of the same machinery that produced the answer. Whether a value of 0.95 corresponds to being right about 95 times in 100 is a separate property with its own name, calibrationWhether a value of 0.9 corresponds to being right nine times in ten, which is a separate subject in its own right., and how a component is calibrated, fitted or evaluated is covered separately. The plain consequence is what matters: nothing in the arithmetic of a confidence value forces 0.95 to mean right 95 times in 100, and at this bank nobody ever claimed it did.

A great deal still works without that property. The ordering holds. The queue of fields most worth checking is still the right queue. One kind of sentence does not work at all: fields are accepted above 0.92, so at most 8 per cent of accepted fields can be wrong. The sentence is arithmetic performed on a quantity that does not support arithmetic, and it appears in more threshold papers than anybody would like.

What connects a confidence value to whether the answer is right?

One thing only, and it is unglamorous. A person takes a sample of the outputs the chain accepted, opens the underlying documents, and checks each one. The exercise is called a hand checkA person checking a sample of accepted outputs against the source, which is what connects the value to correctness., and it is the only bridge between a component's opinion of itself and a fact about the world.

The word doing the work in that paragraph is accepted. Sampling the routed fields instead is tempting: they are already in front of people and the sampling costs almost nothing. The routed sample answers a question nobody needs answered. Routed fields already go to a person. Nobody knows what is quietly going wrong inside the 95 per cent that never stopped, and nothing except a hand check on accepted output will say. A hand check on routed output measures the work already being done, and a hand check on accepted output measures the risk already being carried.

A sweet shop keeps the same discipline when it opens one box from a batch of two hundred rather than tasting the ones the packer already set aside as doubtful. Checking what was rejected describes the rejecting. Checking what passed describes what the customers are about to eat.

What did one bank find when it checked 400 accepted fields?

Sumeru Bank Limited set its own acceptance bar at 0.92. The bar of 0.92 is this bank's own choice, taken for this bank's own chain, and a bar chosen that way is not a standard, a benchmark or anything anybody else is expected to match. At that bar, of the month's 120,400 fields, 114,380 are accepted without a person, being 95.0 per cent, and 6,020 are routed to a person, being 5.0 per cent.

Then the bank did the only thing that makes those numbers mean anything. The bank pulled 400 accepted fields, opened the documents behind them, and checked. Seven were wrong. Seven in 400 is 1.75 per cent. Projected across 114,380 accepted fields, that rate implies about 2,002 wrong field values sitting inside the month's accepted work, untouched, unflagged and already used.

Every one of the 400 checked fields had cleared a bar of 0.92 and 7 of them were wrong, which is a fact no reading of the confidence values on their own would ever have produced. The two sentences set next to each other hold the whole subject. The chain reports 95.0 per cent accepted. The chain does not report 1.75 per cent wrong, and cannot. The 1.75 per cent came from a person with a document open beside a screen.

Two honest caveats belong beside that finding, and the bank stated both. Seven is a small count, and how a sample size is chosen is covered separately. And 1.75 per cent is a reading of this component, on these documents, at this bar, in this month. The rate is not a property of reading steps in general.

How a bar is chosen, and who chooses it

Five steps, in this order, and the order is the teaching. Most threshold papers start at step two and skip step one entirely. Skipping step one is why most threshold papers carry a wrong-field figure that nobody measured.

1. Hand check a sample of accepted outputs. Not routed outputs. The hand check is the step that produces a wrong-field rate at all, and without it every later figure is an assumption with a decimal point.
2. Sweep the bar. Run the month's fields against several candidate bars and record, at each one, how many are accepted, how many of those are wrong on the checked rate, and how many are routed.
3. Price the extra routing in desk minutes. A routed fieldA field sent to a person because its confidence sat below the bar. is not free. A routed field becomes a stopped file, a case, a person's minutes and eventually a post somebody has to fund.
4. Price the wrong outputs in what they cause downstream. A wrong field is not free either, and what it costs depends entirely on which field it is and what reads it next.
5. Choose, and record who chose it. A bar with no name against it is a setting rather than a decision, and settings drift.

Who chooses? Not the team that built the component, whose evidence is essential and whose incentive is to accept more. The choice belongs with the people who carry the two consequences. At this bank those are Revathi Balan, head of retail credit, who carries what a wrong accepted field does to a lending decision, and Ismail Sheikh, head of the exception desk, who carries every routed field as minutes on a queue. A threshold is the point where two different people's problems are traded against each other, so it cannot sensibly be set by either of them alone or by anybody who carries neither.

What does the bank's own sweep across four bars show?

A threshold sweepMeasuring what is accepted, what is wrong and what is routed at several candidate bars. is the second step, and it is mechanical once the first is done. Run the same month of 120,400 fields against four candidate bars, apply the hand-checked wrong rate measured at each, and write down three columns. Sumeru Bank Limited did exactly that, and every row of the result sums back to 120,400. Summing back to the month's total is the first thing to check on anybody's sweep.

Read the two right-hand columns as a pair rather than one after the other, because that is where the decision lives. As the bar rises the wrong accepted fields fall from 3,415 to 256, a fall of 3,159. Over the same range the routed fields rise from 2,649 to 34,916, a rise of 32,267. The wrong column falls by about three thousand fields across the whole range and the routed column rises by more than thirty two thousand, so the two sides of a threshold are not two views of one quantity, they are two quantities moving at completely different speeds.

What is the exchange rate between a wrong field prevented and work sent to a person?

Both sides of a threshold are now measured, so they can be put in one number. Between the bank's own bar of 0.92 and the bar of 0.96 above it, wrong accepted fields fall from 2,002 to 954, so 1,048 wrong fields are prevented. Routed fields rise from 6,020 to 14,448, so 8,428 more fields go to a person. Dividing the second by the first gives 8.04.

The figure 8.04 is an exchange rateExtra work sent to a person for each wrong output prevented by raising the bar., and it is the single most useful number on a threshold paper. An exchange rate turns an argument into a question anybody can answer. Moving from 0.92 to 0.96 costs 8.04 extra fields for a person for every wrong field it prevents, and a person who cannot say whether that is worth paying has not been given a decision to take. The virtue of the number is that it is arguable. Somebody can say yes, a wrong income figure in a lending file is worth eight fields of somebody's attention. Somebody else can say no. Both are having the right conversation. Neither could have had it from a slide reporting only that errors fall by half.

Why does the exchange rate get worse as the bar rises?

Because the rate is not constant, and treating it as one is the second big error on this subject. Work the same division on each pair of adjacent bars in the sweep and three very different numbers come out. From 0.85 to 0.92 the rate is 2.39. From 0.92 to 0.96 it is 8.04. From 0.96 to 0.99 it is 29.3. The price of preventing one wrong field rises more than twelve times across the usable range, so where a bar already sits changes what moving it means far more than how far it moves.

The reason is worth building an intuition for rather than memorising. Picture the gate of a housing society where the guard is told to telephone a flat whenever he is not fully sure the person at the gate lives there. At a loose rule he phones only about the genuinely unfamiliar faces, and a fair share of those calls catch somebody who should not be walking in. Tighten the rule and he starts phoning about residents he has seen a hundred times but cannot place today. He is still catching the occasional outsider, but now he makes forty calls to do it. Nothing about the outsiders changed. The composition of everybody he is now stopping is what changed.

The same composition effect runs the sweep. Fields sitting just under a low bar are a mixed population with plenty of genuinely bad readings in them. Almost everything the component was uncertain about has already been routed at the lower bars, so fields sitting just under a high bar are overwhelmingly correct. Each step up the scale therefore sweeps in many more right answers for each additional wrong one it catches, and the ratio deteriorates accordingly.

What does the choice of bar cost the desk, in people?

Nobody funds a field, so fields are the wrong unit for the second half of the trade. Convert. The month's routed fields do not spread evenly across files; they sit together, at an observed clusteringFailures sitting together in a few files rather than spreading evenly across many. of 4.76 routed fields to each stopped file. At 0.92 the clustering puts the 6,020 routed fields inside 1,264 stopped files, an exception cause the desk already knows. At 0.96 the 14,448 routed fields sit in about 3,035 files instead.

Hold the other five exception causes at their measured 1,746 files and the desk's month rises from 3,010 exceptions to 4,781. At the desk's handling time of 19 minutes a file, that is 90,839 minutes against 57,190. At an assumed working month of 8,400 minutes a person, that is 10.81 posts against 6.81. Moving the bar from 0.92 to 0.96 is a request for four more people, and stating it that way is the only version of the sentence anybody can actually approve or refuse.

What has to be recorded about a threshold, and when is it revisited?

The record is short enough to fit on one screen, and being that short is exactly why nobody writes it. Five lines: the bar itself, who chose it, the hand check it rested on with its date and its sample size, the sweep it was chosen from, and the date it is next revisited. The five lines are the whole artefact.

The date and sample size are not bureaucratic decoration, and they are the two lines most often dropped. The bar means nothing without the check behind it, and the check ages. Refit the component and the values shift underneath a bar that did not move. Start receiving a new kind of document and the population the check was drawn from is no longer the population running through the chain. A threshold with no recorded check behind it is a setting, and a setting nobody revisits will eventually be defended in a review as though somebody had chosen it.

Recording the sample size does one more thing, and it is the honest one. Seven wrong in 400 is a small count. A reader of the record needs to be able to see that for themselves rather than meeting 1.75 per cent as though it arrived with three decimal places of authority. Write the 7 and the 400 down, not just the percentage.

What does a person handed one of these numbers actually do with it?

Three people meet a confidence figure in the ordinary course of a week, and each of them wants a different thing from it.

The head of an operations desk uses it as a staffing forecast. Ismail Sheikh does not care what 0.92 means philosophically; he cares that the bar and the clustering together produce 1,264 stopped files, 3,010 exceptions and 6.81 posts, and that a change to the bar arrives on his desk as a change to his roster with a lag. He uses one arithmetic chain: routed fields, over clustering, into files, times handling time, over the working month, into posts. The last number in that chain is the number being asked for, so anybody proposing a threshold change to an operations head should arrive with it already worked out.

A reviewer in a risk function uses it as a question. Neelima Rao, reviewing this chain, would ask one thing first: show me the hand check on accepted output, its date and its sample size. If the answer is that there isn't one, then every wrong-field figure in the paper is an assumption, and the review is about assumptions rather than about a threshold. The question about the hand check separates a measured threshold from a decorated one faster than any other.

Somebody buying a reading service from outside uses it as a translation test. A supplier's claim of high accuracy is an accuracy on the supplier's documents at the supplier's bar. The buyer's question is not whether the claim is true but what it becomes on their own documents at their own bar, and the only way to find out is to run their own hand check on accepted output after go-live. Ajay Agrawal, Joshua Gans and Avi Goldfarb's framing in Prediction Machines, 2018, is useful here: a prediction is only worth what somebody does differently because of it, and a confidence value that changes nobody's routing and nobody's checking has cost money and bought nothing.

All three uses share one habit worth taking away whole. Each person converts the number out of the component's units into their own before arguing about it: into posts, into a question about evidence, into what happens on their own documents. The confidence value itself never left the component, and it was never going to.