Model Testing: What Is Checked Before a Model Goes Live

Testing before go-live asks whether the thing behaves as intended on data it has not seen, on every population it will meet, and when something expected does not arrive. One invented bank wrote down seven tests. The bank ran four before go-live and three afterwards, and the missing one is precisely the test that would have caught the fault that arrived in month 8.

A test is a question asked before anybody is harmed by the answer. The advantage a test holds over every other control a firm has is also, exactly, its limit. A test can only ask about a condition somebody has thought of, using data somebody already holds. So the tests that get postponed under a deadline are the ones about conditions nobody in the building has met yet, and those are exactly the conditions that arrive later.

What is testing before go-live actually asking?

Consider hiring a cook for a wedding. Not a small one, the full thing, four hundred people. Nobody hires on a conversation. The first step is to taste a few dishes the cook has already made. The cook chose those dishes, so they tell the family something real and nothing it did not already suspect. So the family names a dish itself, on the day, from ingredients it supplies. The first kind of question is about performance on something the person being tested did not choose.

Then comes a second kind of question, about scale and about spread. Cooking beautifully for six is not the same skill as cooking acceptably for four hundred, and the four hundred will not eat the same things: there will be children, elderly relatives, people who eat no onion. A cook who is wonderful on one of those groups and poor on another has an average that looks fine and a wedding that does not. The second kind of question is about populations.

And then there is a third question that almost nobody asks. How does the cook handle something going wrong that was never part of the plan? The vegetable delivery does not arrive. The gas runs out at seven in the evening. A tray of something arrives spoiled. None of those is a test of cooking. Each is a test of what happens at the edge, when the arrangement's own assumption simply is not there. Most people hiring a cook ask the first two questions thoroughly and skip the third entirely, and the reason is always the same: the delivery has always turned up before.

A pre-deployment testA question asked of a system before anybody is affected by the answer. asks those same three questions of a system. Does it behave as intended on data it has not seen? Does it behave as intended on every population it will actually meet, rather than on the average of them? And what does it do when something it expects does not arrive, or arrives in a shape nobody wrote down? Every one of the seven tests below is one of those three, plus one more about whether the whole assembly works when the parts are joined together.

What are the seven tests, and what is each one for?

Sumeru Bank Limited, invented, runs a retail loan intake chain of nine numbered components, five of which are fitted to data. Before it put that chain in front of customers, it wrote down seven tests. The seven are the bank's own set, not anybody's list, not a standard and not a requirement of any authority: one firm's working practice rather than a template. Their value is not that they are the right seven but that they are a real seven, with a real record of which were run and which were not.

Two of the seven are about data the chain has not seen. Test 1 runs it on held-back dataData deliberately kept out of the fitting so it can be used as something the model has not seen., a slice deliberately kept out of the fitting so it can stand in for the unfamiliar. Test 2 is an out-of-period testA test on a stretch of time after the window the model was fitted on. and runs it on a stretch of time after the fitting window closed. How that slice is chosen and how a reading on it is computed are questions about fitting and evaluation, covered separately. The question each test asks, and the price each test charges, are the two things to hold on to.

Three of the seven are about populations. A chain fitted mostly on one channel need not behave the same way on another, so test 3 reads the chain on each channel separately. Test 4 reads it on each kind of documentA category of document, distinct from the channel a document arrived through. separately, a different cut entirely: a channel is where something came from, a kind is what it is, and the two must never be merged. Test 6 reads the spread of outcomes across the groups the bank can actually construct from what it records. Whether those outcomes are fair, whether the chain repeats itself and whether a stated reason holds up are set out under bias, fairness and explainability testing, and are not settled by test 6 alone.

One of the seven is about things arriving wrong. Test 5, an absent field testA check of what the chain does when something it expects does not arrive, or arrives in a different shape., asks how the chain behaves when a field it expects is absent, or arrives in a format nobody wrote down. And one is about the assembly rather than the parts. Test 7 runs the whole chain end to end on files people had already decided, and at this bank test 7 was the pilot. Six of the seven can only ask about data that already exists somewhere, and exactly one of them, test 5, needs no data at all. The whole argument turns on that one test.

Why is the order they are run in not arbitrary?

Order matters because each test is only readable once the earlier ones have passed. Suppose the work starts at the other end, with test 7 first, the whole chain on real files, and the answers come back disappointing. Only one thing has been learned: something in a chain of nine components is wrong. Every component contributed to every answer, so which one cannot be said, and the only way on is to work backwards through the whole chain to find where the loss sits. The chain test is an excellent last test and a useless first one.

Read the four boxes as a ladder of shrinking information. A failure at step 1 says the fitting itself is wrong: bad news arriving early and cheaply. A failure at step 2 gives the weakness an address: this channel, that kind of document, this group. A failure at step 3 says the chain has no written behaviour for something arriving wrong, not a performance finding at all but a missing decision. Only at step 4 does a failure stop naming anything, and that is why the chain test belongs last rather than first, however tempting it is to start with the realistic one.

Which four did this bank run before go-live, and which three waited?

Sumeru Bank Limited ran tests 1, 2, 3 and 7 before go-live and ran tests 4, 5 and 6 only afterwards. Four run plus three not run is seven. The interesting thing is not the count but which three fell on the wrong side of the line.

Postponing tests is completely ordinary and it is not the failure in this episode. Every deployment in every industry ships with a list of things scheduled for after go-live, and a team that refuses to ship until all seven are done is usually a team that never ships. The decision worth recording is never how many tests were postponed but which ones. The three that waited at this bank contained the cheapest test on the board and the only one of the seven that asks about something arriving wrong.

What did the pilot buy, and what did its shadow decisions show?

Test 7 at this bank was a pilotA limited run beside the existing process, on real files, with the outcomes not acted on.. Months 1 to 3, on one channel, at about 400 files a month, the chain ran beside the process that was actually deciding. Every file went to a person as it always had, and the chain recorded what it would have done, without doing it. Three months at about 400 a month is 1,200 shadow decisionsA decision the system records without acting on it, beside the process that is actually deciding.: real files, real customers, and outcomes that touched nobody.

A pilot buys something no other test can buy. The pilot runs on the real thing, in the real order, with the real documents arriving in the real mess they arrive in. A shadow decision by construction cannot harm anybody, so the arrangement costs the applicant nothing. Back at the wedding, it is the equivalent of the new cook working three smaller functions alongside the cook the family already trusts, plating everything, and nobody serving any of it.

Now watch what happens to those two numbers in a meeting. The 1,092 is the number that goes on the slide. A reading of 91.0 per cent sounds like a pass and reads as reassurance. The 108 is the number people skim past. The order is exactly the wrong way round. Agreement shows only that the chain is not obviously broken, and every question the pilot actually raised sits inside the 108 files where the chain and a person reached different outcomes. A pilot that produces no disagreements has shown nothing at all; it has shown only that the chain repeats what people were doing anyway.

What is a disagreement evidence of, and what is it not?

A disagreementA file where the system and the person reached different outcomes, which is a question rather than a verdict. is a file where the chain and the person landed differently. A disagreement is a question, not a verdict, and the reason is simple: nobody in the room knows who was right. The person is not an oracle. On a credit file, finding out who was right means waiting to see whether the borrower repays. The wait runs as long as the outcome takes to arrive, and that is well over a year on this bank's own definitions.

The split is the useful part. Of the 108, seventy two were files the chain accepted and a person had referred, and thirty six were files the chain referred and a person had accepted. Seventy two plus thirty six is 108. The two groups are not two halves of one finding; they point in opposite directions. The first says the chain is looser than the people were, and if the chain is wrong on those, the cost lands on the bank in bad lending. The second says the chain is tighter, and if the chain is wrong on those, the cost lands on the applicant, who waits and may be refused something they should have had.

The errors, if they are errors, fall on two different people and cost two different things, and a single disagreement rate of 9.0 per cent averages away exactly that. O'Neil, in Weapons of Math Destruction, 2016, makes the general point that a model's errors do not fall evenly across the people it decides about, and a two direction split is the cheapest instrument a firm has for seeing that at all.

Why could a pilot of 1,200 files never have found what arrived in month 8?

The pilot was a good test. The pilot was real, it was three months long, and it produced a split worth arguing about. The pilot also could not possibly have found the fault that arrived later. The obvious answer to why is wrong, so the reason is worth being precise about.

The obvious answer is size. The pilot ran 1,200 files. The chain later decided 8,600 files every month, so 1,200 over 8,600 is 13.95 per cent, being 14.0 per cent of a single month. Put that next to a fault that occurs in one file in nine hundred and the arithmetic is stark: 1,200 divided by 900 is 1.33, so a set that size holds one such file, sometimes two, sometimes none. One file inside 1,200 is not a reading, it is an anecdote, and it is indistinguishable from a keying slip. Push the rarity out past one file in 1,200 and the expected count falls below one, so the set is expected to hold none at all. The honest floor on what any test set of a given size can reach is arithmetic rather than an opinion about diligence.

Now check that arithmetic against the actual fault, and watch the size argument collapse. Once the income field started arriving in the new format, about 12,900 files were decided across 30 working days and 176 of them moved from accept into the referral band. The rate is one file in 73, and a set of 1,200 files at that rate holds about sixteen of them. The fault did not exist while the pilot was running, so the pilot was more than large enough to see this fault and still could not have seen it. The pilot's last file was decided at the end of month 3 and the field changed shape in month 8, week 2, more than four months later.

The same collapse happens on the other fault the bank later met. Had the fifth kind of document been arriving during the pilot at the 8 per cent of monthly documents it later represented, a 1,200 file pilot carrying four documents a file holds 4,800 documents, of which 384 would have been of that kind. Three hundred and eighty four is plenty to read. So size was not the constraint there either. The constraint was what the pilot recorded, agreement with a person file by file, and never a reading taken separately for each kind of document.

Put the two together and the conclusion is uncomfortable but exact. A test set can fail in three quite different ways: it can be too small, it can be big enough but ask the wrong question, or it can be perfectly designed and simply predate the condition. Only the first of those is fixed by running more files, and it is the only one anybody ever budgets for.

Which single missing test would have caught the fault that arrived?

Test 5 asks how the chain behaves when a field is absent or arrives in a format nobody wrote down. In month 8, week 2, an income field arriving from one channel arrived in a format nobody had written down. The mapping is not rhetorical. The test and the fault are the same sentence twice, once as a question and once as an event.

The week that followed is the part worth sitting with. Nothing stopped. Nothing errored. Nothing was flagged. The chain read the field, did something with it, and carried on producing decisions that looked exactly like the decisions of the week before. Over the next 30 working days about 12,900 files were decided and 176 of them moved from accept into the referral band. The failure left no error message anywhere. Nobody had ever written down what that field should do when it arrived wrong, so there was no rule that could be broken.

The same absence shows up in the bank's own paperwork before it ever showed up in a decision. For every field the chain consumes, the bank keeps eight numbered items, and item 6 is what happens when the field is absent. Of the fourteen fields the reading step extracts, item 6 is written down for 3, being 21.4 per cent, and missing for 11, being 78.6 per cent. Three plus eleven is fourteen. The field record existed, in that state, for months before month 8. The fault did not appear out of nowhere in month 8; it had been sitting in a spreadsheet as eleven blank cells since before the chain went live, and nobody read it as a test result because it did not look like one.

The ranking is the sting. Test 7, the pilot, took three months of calendar time and a working version of the whole chain. Test 2 needed a later stretch of time to have happened and outcomes to exist for it. Test 5 needed a person, a list of fourteen fields and an afternoon of deciding what each should do when it arrives wrong. The chain itself cost Sumeru Bank Limited Rs 2,40,00,000/- to build and Rs 65,00,000/- a year to run. At the bank's own assumed fully loaded Rs 9,00,000/- a year a post, spread over the 240 working days its own arithmetic uses, one working day of one person is Rs 3,750/-.

So why was it skipped? For the same reason nobody in a household ever asks what happens if the milk does not arrive on Tuesday. The milk always arrives. There is no incident, no complaint and no near miss pointing at it, and a test with no story behind it loses every argument about what fits in the remaining fortnight. Nothing had ever gone wrong with those fields, and test 5 was postponed for the exact reason cheap tests are always postponed: a test that has never had anything to find looks indistinguishable from a test that will never find anything.

What did test 4 find when it was finally run?

Test 4 was one of the three that waited, and when it was eventually run it produced a reading nobody had seen before. The document classifier reads the four kinds of document it was fitted on correctly 96.2 per cent of the time. A fifth kind had begun arriving from one channel, and on that fifth kind it reads correctly 71.0 per cent of the time. Ninety six point two less seventy one point zero is a gap of 25.2 points.

Hold that gap next to what the reported figure did. The fifth kind is 8 per cent of monthly documents, so the overall reading becomes 92 per cent of 96.2 plus 8 per cent of 71.0, or 94.2. The headline moved from 96.2 to 94.2, a movement of 2.0 points. A population reading 25.2 points worse than the rest moved the number anybody actually looks at by 2.0 points. Tests 3 and 4 exist as separate tests rather than as a footnote to the overall figure for exactly that reason.

One separation matters and it is easy to fumble. The 71.0 per cent is a reading on a kind of document. The channel reading is a different measurement, where the chain reads 94.1 per cent on the channel that supplied 62 per cent of the fitting examples and 87.9 per cent on the others, a gap of 6.2 points. A channel is where something came from. A kind is what it is. Test 3 asks the first question and test 4 asks the second, and merging the two readings into one story about the classifier would be wrong in both directions.

What can no amount of testing show?

Suppose every one of the seven tests had been run before go-live and every one had passed. The finding would be real and narrow: the chain behaves as intended on conditions somebody thought of, using data somebody already held. A finding like that is genuinely worth having. And it is, precisely, everything seven passes can establish.

A test is a question, and a question nobody asked has no answer. There is no reading, no percentage and no pass mark that covers the conditions nobody in the building has met yet. More testing means more questions from the same people with the same experience of the same past, so more testing does not close that gap. A passing test set is a starting position rather than a conclusion, and monitoring after go-live exists for exactly that reason.

There is a second limit, quieter and easier to miss. A test that is run is not a test that is acted on. Sumeru Bank Limited ran test 3 before go-live, so the 6.2 point difference between one channel and the others was measured, written down and known while the chain was still deciding nothing. The chain went live anyway. Going live may well have been the right call. Either way, the one fault of the three the bank's own test set caught is also a fault the bank went live carrying, so a count of tests run overstates what a test set actually protects anybody from.

How to Test an AI System Before Production Deployment: what does a run of the seven actually look like?

Everything above collapses into a working sequence, and a firm that has been through it has a test set rather than a set of reassurances. The sequence runs in five moves, and none of them is statistical.

First, every condition the thing will meet is written down, not every condition it was fitted on. The two lists are different and the second is always shorter. Channels, kinds of document, groups that can be constructed from the firm's own records, and the ways an expected field can fail to arrive. Second, each condition becomes a question with a name and a number, so a postponed test is a visible row rather than a silence. Third, they are ordered so that a failure at each step names something: parts before assembly, always. Fourth, the tests that need no data are run first, as only those can be finished before anything else exists. Fifth, whichever are postponed are recorded as postponed, with a name against the decision. The postponement is the part of the record that will matter later.

The fourth move is the one this bank inverted. Test 5 required nothing and was scheduled last; test 7 required the whole chain and three months and was run first. The tests that need nothing are the tests that can be completed while everything else is still being built, and they are therefore the only ones a deadline cannot legitimately squeeze.

What does a person actually do with this on a Monday morning?

Take three people looking at the same seven rows. Revathi Balan, the named accountable person for the scoring model, is not reading the readings; she is reading the fourth column, the one that says what each test needed before it could be run, and asking which rows are empty for a reason and which are empty because nobody argued for them. Ashok Pillai, in technology risk, is doing something narrower and more useful: he is putting the seven tests next to the fourteen fields and noticing that eleven of them have no written behaviour when absent. Eleven blank cells is a test result sitting in a spreadsheet rather than in a test report.

Ismail Sheikh, who runs the exception desk, is the one with the sharpest instrument and the least formal standing. He sees the disagreements. He knows which files the chain and his people read differently, and the desk works the cases, so he knows it within a week rather than within a year. A firm that treats the desk's disagreement as an operations statistic rather than as the cheapest live test it has is throwing away the one signal in the building that keeps producing after go-live.

And for anybody outside a bank reading a claim about an automated arrangement, the useful question is not what its overall reading is. The useful question is much shorter: on which populations was that reading taken separately, and what does the arrangement do when an input arrives in a shape nobody wrote down? An overall figure with no per population readings behind it, and no written behaviour for a missing field, is a claim about the average of a thing rather than about the thing.

How is this different from monitoring and from independent challenge?

Three arrangements are easy to confuse because all three involve somebody checking something. The three ask different questions at different times and none substitutes for another.

The last column is the one firms most often think they can economise on, and the reasoning sounds solid: with thorough testing and thorough monitoring, what is a third party going to add? The answer is that both of the first two check behaviour, and neither checks the choices. The definition of a bad outcome, the window it is measured over, the population it is measured on and the cases excluded are decisions somebody made before any of this was built. A chain can pass every test and every alert rule while resting on a definition of a bad outcome that nobody outside the team building it has ever read. Examining those choices is the work of independent challenge, covered separately.