The Decision Threshold: Where the Cut-Off Sits and What It Costs

A decision cut-off is the number at which a system stops asking and acts. The cut-off is chosen by the deployer, not produced by the component. At Sumeru Bank Limited, invented, moving the accept cut-off from 720 to 760 would remove 39 expected bad accounts from the month's 4,902 accepts and move 760 applicants into a queue for a person.

A cut-off is the only place in a decisioning arrangement where a continuous number turns into something that happens to somebody. Everything before it is degrees. Everything after it is an answer in four minutes, or a wait of two working days, or a refusal. Because the scores are not spread evenly along the scale, an equal move in the cut-off moves wildly unequal numbers of people depending on where it already sits, and that is why the trade has to be computed rather than argued from principle. One month at one invented bank is enough to compute the trade end to end. Where the line belongs is a decision for the people accountable for it.

What is a decision threshold, and what does it convert?

A college admissions office with one seat left and a pile of marksheets makes the point. Marks run from 0 to 100 and arrive as a smooth spread: 71, 71.5, 72, 72.5. Somewhere in that spread a person writes a line and says that at or above this figure a candidate has a seat. The moment the line exists, a student on 72.5 and a student on 72.4 stop being nearly identical and become admitted and not admitted. Nothing about either student changed. A number was written on a wall.

A decision thresholdThe value at which a system stops asking and acts. Above it one thing happens to the file and below it another. is that line, running inside a bank instead of a college. At Sumeru Bank Limited, invented, component 6 of the intake chain reads an application and returns a value from 0 to 1000 on the bank's own invented scale, where higher means the component placed the application further from the outcome it was fitted to. The returned value is continuous, and a continuous value is a matter of degree. The cut-off is the one step that turns a matter of degree into a thing that happens to a person, and no other step in the arrangement does that. Agrawal, Gans and Goldfarb, in Prediction Machines, published in 2018, put the same split in general terms: a fitted component supplies a prediction, and somebody still has to decide what to do when the prediction comes back. The cut-off is where that deciding was written down in advance.

A great deal of confused argument comes from mixing the two sides of the conversion, so be exact about what sits on each. On the input side is a number the component produced from the fields on an application. On the output side is a category: accepted, referred, declined. A number and a category are not the same kind of thing at all, and the arithmetic that connects them belongs entirely to the bank.

Who chose this cut-off, and is the choice part of the model?

No. Revathi Balan, head of retail credit at Sumeru Bank Limited and the named accountable person for component 6, set both of this bank's cut-offs. Both cut-offs are her choices, and neither is a standard, a norm or a requirement of anybody. Component 6 returns exactly the same 5,981 scores whatever the cut-offs are, so every accept, every decline and every referral in the month is produced by the step after it rather than by the component itself.

The split between the component and the cut-off sounds pedantic until it is set against a real argument. If somebody says the model declined 688 people last month, the natural next move is to talk about the model: how it was fitted, what data it saw, whether it should be rebuilt. But changing the cut-off from 720 to 760 changes 760 outcomes without touching a single fitted number, and rebuilding the model changes nothing at all unless somebody also decides where the new line goes. Two different levers, two different people, two different approval routes, and a conversation that mixes them will pull whichever one is nearer to hand.

What are the two cut-offs at this bank, and what do they make?

Sumeru Bank Limited runs two lines rather than one. The accept cut-offThe score at or above which a file is accepted with no person involved. sits at 720: at or above it the file is accepted and the applicant has an answer in about four minutes. The decline cut-offThe score below which a file is declined with no person involved. sits at 580: below it the file is declined, also with nobody looking at it. Everything between the two falls into the referral bandThe scores between the two cut-offs, where the component declines to decide and the file goes to a person., where the component has effectively declined to decide and a person decides instead.

Two lines make three outcomes, and the count matters more than it looks. The middle outcome is the only place in this whole arrangement where a person makes a credit decision on a loan application, and it holds 391 files a month. The other 5,590 are settled by the position of a number relative to a line. The honest answer to how much human judgment sits inside this bank's lending is 391 files out of 5,981, and both of the lines that produce that figure were chosen rather than found.

How are the month's 5,981 scores actually spread?

One fact decides everything else in this guide. Applicants do not arrange themselves evenly along a score scale, and this month's did not. The queue outside a busy ration shop at nine in the morning has the same shape: it is not a smooth line of people spaced two feet apart, it is a dense knot at the front and a thin trail at the back. A rule that says the first fifty get served cuts through the knot. A rule that says the first three hundred get served cuts through the trail. Same rule, same shop, entirely different consequence, and the only thing that changed was where the line fell relative to where the people were standing.

The score distributionHow the month's scores are spread across the scale. The spread decides how many files a small move in the cut-off actually moves. at Sumeru Bank Limited has exactly that shape. 3,012 of the 5,981 scored 800 or above. Half the month, 50.4 per cent of it, sits in a single stretch at the top. Immediately above the accept cut-off, the twenty points from 720 to 739 hold 340 files. Immediately below it, the twenty points from 700 to 719 hold 53. The scores thin out sharply approaching the accept cut-off from above and thin out again below it, so the same twenty point move buys six times more or six times less depending on which way it points.

Which bad rate is this, and how does it sit beside the 3.4 per cent?

Two figures look like the same number and answer completely different questions. Pull them apart before a single expected bad account appears. Sumeru Bank Limited has a locked figure of 3.4 per cent, being 8,160 accounts carrying a bad label out of the 2,40,000 accepted applications in the past window component 6 was fitted on. The past window had no cut-off applied to it at all. The 3.4 per cent describes what the bank's accepted book did, under the bank's own four label choices, before any of this arrangement existed.

The 2.02 per cent that appears from here onward is a different animal entirely. The 2.02 per cent is what the deployed cut-off is expected to produce on this one month's accepts, using the bank's own fitted reading applied band by band. Different population, different question, different vintage. The two rates are not a before and an after, and treating them as one figure would make a fall from 3.4 to 2.02 look like a result the arrangement produced, when in fact nobody has observed anything yet. Under this bank's own label a credit outcome is not known until twelve months of observation have run and an account has reached ninety days past due inside that window, so the earliest the month 6 book can be scored at all is month 21.

The wait until month 21 governs every bad account figure that follows. Every number about bad accounts in this guide is an expected bad accountAn account the bank's own fitted reading expects to reach its bad definition. The figure is an expectation applied to a month, not an outcome anybody has observed., meaning an expectation the bank has applied to a month, and not an outcome anybody has seen. A cut-off argument conducted in expectations is still worth having. An expectation is just not a result, and the record has to say which of the two an argument was conducted in.

What does tightening the accept cut-off buy?

Now the sweep. Hold the decline cut-off at 580 so only one thing moves, walk the accept cut-off from 640 up to 800 in steps of twenty, and read off what happens to the same 5,981 scored files. No file leaves and none arrives, so every row below sums to 5,981. All that changes is which of the three outcomes a file lands in. The expected bad accounts come from the bank's own fitted reading applied band by band, as whole accounts.

Read the deployed row and the 760 row against each other and the purchase is plain. Tightening the accept cut-off from 720 to 760 takes the expected bad accounts among the accepts from 99 down to 60, so it buys the removal of 39 expected bad accounts, and the rate on accepts falls from 2.02 per cent to 1.45. Notice also what the sweep leaves alone. The declines sit at 688 in every single row. The accept cut-off has nothing whatever to say about who is refused outright, and any report that discusses tightening it in the language of turning people down has confused the two lines.

What does tightening it cost, and who pays that?

The 39 came from somewhere. Look at the same two rows again and the referrals go from 391 to 1,151. 760 applicants who would have had an answer in about four minutes are now waiting for a person, and at this bank that wait is two working days. The 760 are not a rounding adjustment and not a worse class of applicant. Every one of them is a person whose file landed on one side of a line last month and on the other side of a differently placed line this month, with nothing about the application changed.

Then follow the 760 to the desk that receives them. The exception desk at Sumeru Bank Limited runs at 7 people and the locked handling time is 19 minutes a case, so 760 extra cases is 14,440 desk minutes a month, being 1.72 posts at the assumed working month of 8,400 minutes a person. Put that in money at the bank's assumed fully loaded Rs 9,00,000/- a post and it is about Rs 15,48,000/- a year of desk capacity that has to exist before the cut-off moves. The Rs 15,48,000/- is arithmetic on the locked figures rather than a measured cost, and the second reviewer that every referral band file carries adds to it: at the locked 7 minutes for the re-reading and the deciding, 760 more files is a further 5,320 minutes, taking the whole to 19,760 minutes, being 2.35 posts.

And there is a queue underneath all of that. The desk closes about 154.7 cases a day against 150.5 arriving, so its marginHow much spare capacity a desk has before its queue starts growing instead of staying flat. is 4.2 cases a day, being 2.7 per cent of capacity. Spread 760 extra referrals over the month's 20 working days and the desk takes 38 more cases a day, a little over nine times the whole margin. Without hiring, the move from 720 to 760 does not make people wait two days. The queue grows every single day instead, and a queue that grows is a different and much worse thing.

There is a second way to see the same asymmetry, and it is the one that catches people out in a management pack. Draw the expected bad rate across the whole sweep and it falls, smoothly and reassuringly. Draw the referrals on the same picture and they climb, and they climb faster than the rate falls. Every setting that improves the number a report is likely to lead with makes both of the numbers it is unlikely to mention worse, and it does so in the same direction every time.

How can the trade be stated so somebody can argue with it?

Divide one side by the other and put both in units a person can hold. 760 applicants moved for 39 expected bad accounts prevented is 19.5 extra referrals for every expected bad account prevented. The exchange rate of 19.5 to one is the whole trade, and a credit officer and a desk head can genuinely disagree about 19.5 in a way neither of them can disagree about a fall from 2.02 per cent to 1.45.

Watch how differently the two versions behave in a room. Say the rate falls from 2.02 to 1.45 and nobody has anything to push against; a smaller number is a smaller number, and the meeting moves on. Say that 19.5 people wait two working days for every expected bad account prevented, and both sides have something to hold. A bad account costs the bank far more than nineteen delays, so the head of retail credit can argue that 19.5 is cheap. The exception desk's margin is 4.2 cases a day and the move needs 38, so its head can argue that 19.5 is impossible. Both of those are real positions. The percentage version does not let either of them be stated.

What happens if it is loosened instead?

Run the control the other way and the arithmetic refuses to mirror. Moving the accept cut-off down from 720 to 700 adds 53 accepts and 4 expected bad accounts, or 13.25 extra accepts for each extra expected bad account. Tightening by forty points moved 760 files for 39 accounts, being 19.5 each. The same lever, moved in opposite directions from the same starting point, does not cost the same thing per unit, and it is not even close.

The reason is the shape of the month, not anything about the lever. Just above 720 sit 340 files in twenty points. Just below it sit 53. Loosening reaches into the thin part of the spread and picks up very few people; tightening reaches into the dense part and picks up a great many. Assuming a cut-off behaves symmetrically is the commonest mistake anybody makes with one, and it survives so well because it is the natural thing to assume about any dial.

Is the exchange rate the same anywhere else on the scale?

No, and this is the thing worth carrying away. Each twenty point step in the sweep costs something different when taken on its own. Down at the bottom, moving from 640 to 660 removes 7 expected bad accounts and moves 60 files, being 8.57 files for each account. Up at the top, moving from 780 to 800 removes 13 accounts and moves 610 files, being 46.92 files for each account. The price of preventing one expected bad account rises more than fivefold across the sweep, so any sentence beginning with the words moving the cut-off costs is meaningless until it says from where and to where.

One honest caveat on those step figures. The expected bad accounts in each band are whole accounts from the bank's own rounded reading, so a band holding 53 files at 7.5 per cent is recorded as 4 accounts rather than 3.975. On the small bands at the bottom of the sweep that rounding moves the step price by a little. The rounding does not move the direction of travel. The file counts set the direction, and the file counts are exact.

Why is the decline cut-off a different question?

Everything so far has held the decline cut-off still at 580. Move the decline cut-off instead and a completely different set of people is affected. Two lines reaching two different sets of people cannot be set by one argument. Drop it from 580 to 560 and the 236 files sitting in that twenty point band stop being declined with no person involved and start being decided by a person. The declines fall from 688 to 452 and the referrals rise from 391 to 627. The accept cut-off never moved, so the accepts do not move by a single file.

Put the two questions side by side and the difference is obvious. Moving the accept cut-off decides how many people wait longer for an answer. Moving the decline cut-off decides how many people are refused credit without anybody looking at their file. A longer wait and an unread refusal are different kinds of harm to different people, they are argued by different parts of the bank, and a single number describing the arrangement's quality cannot possibly speak to both.

Why is a bar on a document reading step not the same kind of number?

The same word gets used for both and the two are not close relatives. Elsewhere in this same intake chain, component 4 reads fields off document images and returns a confidence value for each field, and the bank accepts a field above its own chosen bar and routes the field to a person below it. The confidence bar is also a line somebody chose, and moving it also has a cost. But look at what each line is about. A confidence bar decides whether a machine-read field is trusted; a decision cut-off decides whether a person gets a loan, and no amount of accuracy on the first says anything at all about the second.

The practical damage from mixing them is specific. A field read wrongly and accepted produces a wrong entry that some later check may catch at no cost to anybody. An applicant moved across a decision cut-off gets a different life outcome and no later check restores it. When a meeting settles an argument about the second with evidence about the first, and it happens, the bank has answered a question about people with a measurement about paper.

What does the record have to hold about a threshold?

Four short fields, and they are not onerous. The value. Who set it. The date it was set. And the reading of the tradeWhat tightening a cut-off buys set against what it costs, stated in the same units so the two can be compared. at the setting chosen, with the previous value written beside it so anybody reading later can see what was given up and what was bought. Four lines in a single record, and with them the arrangement can be examined by somebody who was not in the room.

At Sumeru Bank Limited this is item 4 of the nine numbered items of the credit decision record, and at the month 12 independent validation carried out by Neelima Rao it was one of four items found missing. The pattern in what was and was not documented is the whole point of the count. The five items that existed described how the component was made, and the four that did not described what it does to a file and who can stop it. A cut-off with no record is a decision nobody can be shown to have taken, a strange thing to sit underneath 5,590 outcomes a month.

How does somebody reviewing this arrangement actually use the sweep?

A person doing this work for real is not usually choosing the cut-off. A reviewer is asked whether the choice already made can be defended, and the sweep is the instrument that answers the question. A lender's own second line of defence reads the sweep to find out whether the deployed setting sits on a flat stretch or a cliff. At 720 the next twenty points in either direction move 340 files one way and 53 the other. The deployed setting therefore sits right at the edge of the dense part, and a small drafting change would have very large consequences. The finding stands on its own, and it needs no view about where the line belongs.

An analyst outside the bank cannot see the sweep, and knowing that it exists changes what an approval rate means when one is published. A bank reporting that its approval rate fell from 57.0 per cent to 55.6 has said something, but not whether the cause was the applicants, the data feeding the component or somebody moving a line. The three causes have entirely different implications, and the published figure separates none of them. The right question to a bank is not what its approval rate is; it is what its cut-off is, who set it and when it last moved.

And a household on the other side of a cut-off should know one thing only. The answer arriving in four minutes came from where a number fell relative to a line, and the line is somebody's choice at that lender rather than a property of the applicant. O'Neil, in Weapons of Math Destruction, published in 2016, makes the wider point that a model's errors rarely fall evenly across a population. A cut-off does not create that unevenness, but it does convert it into outcomes, all at once, on the day it is set.