Model Drift: What Model Monitoring Watches, and When

The data arriving changes, so a deployed model can stop being right without anything about it changing. Monitoring is the standing watch that notices. Monitoring watches inputs, outputs and outcomes, and the three differ enormously in how fast they can speak. At one invented bank an upstream field changed format. Every signal being watched sat downstream of the fault, and six weeks passed before anybody knew.

Nothing inside a fitted component knows anything about the world outside it. Its numbers were set once, against one window of data, and they stay exactly where somebody put them. So every change has to arrive from outside the component. Either the inputs stop looking like the data the component was fitted on, or the meaning of the answer has moved. Monitoring is the only arrangement that turns either of those into something a person finds out about, and if it is not built, nobody finds out.

Why does a model that has not changed stop being right?

A tailor took a customer's measurements two years ago and wrote them in a notebook. He is a very good tailor. He stitches exactly to the numbers in the book, every time, with no variation at all. The customer is not the same shape he was two years ago. The tailor's reliability is the whole of his value and also the whole of the problem. Nothing about the tailor's method has slipped. The shirt still comes back wrong.

A fitted component is that notebook. Somebody chose a window of past data, fitted the numbers to it, and deployed the result. From that moment the component is a rule of arithmetic: give it the same input tomorrow and it returns the same answer it returned today. DriftA deployed model becoming less right over time while nothing about the model itself changes. is the name for the gap opening between a set of numbers that will not move and a world that will not stop moving. The component does not decay, does not tire and does not develop a fault; it simply keeps answering the question it was fitted to answer, long after that stopped being the question in front of it.

Drift is why looking at the model is the wrong instinct. At Sumeru Bank Limited, invented, the retail loan intake chain runs nine numbered components, five of them fitted to data. In month 8 the chain began producing worse decisions. An engineer could have opened component 6, the scoring model, on any day of those six weeks and found every fitted number precisely where it had been put. There was nothing to find. Agrawal, Gans and Goldfarb, in Prediction Machines, 2018, frame a learned component as something that supplies a prediction a person still has to act on. The corollary matters here. A prediction can go bad without the predictor changing at all.

What do Model and Data Drift each mean on a running system?

Two different things can go wrong. Each is found by completely different means, so the two are worth separating carefully. The first is data driftWhat arrives stops looking like what the model was fitted on, while the meaning of the right answer is unchanged.: what arrives stops looking like what the component was fitted on. The right answer has not moved anywhere. The inputs have. A field that used to arrive as a plain number starts arriving with a currency prefix, or a channel that used to supply a tenth of the volume starts supplying half of it, and the component keeps applying numbers fitted to a population it is no longer being shown.

The second is concept driftWhat the answer is supposed to mean moves, so the same input should now get a different outcome from the one it used to get.: the meaning of the answer has moved. The inputs look completely normal. The relationship between the profile and the thing being predicted has changed underneath everybody. A person with the same profile, the same income and the same record should now get a different outcome from the one they would have got a year ago.

People often speak of both together as model drift, and that phrase is doing something slightly unhelpful. Model and Data Drift are not two settings of one dial: the first names an effect, the deployed thing getting worse, and the second names one of the two causes, so a firm that has diagnosed model drift has not yet diagnosed anything. The useful question is never whether the model has drifted. The useful question is which of the two moved. One of the two is visible on the day it happens and the other is not visible for well over a year.

Why is Concept Drift so much slower to see than the other kind?

Picture a landlord in a large city who has decided over twenty years what a reliable tenant looks like. Salaried, an office to go to, a letter from an employer. The test served him well for two decades. Then the work itself changes shape, and a great many entirely reliable tenants now work from a laptop with three clients and no letter from anybody. Every input on his form still arrives looking exactly as it always did. The form is fine. The form has not moved. The relationship between the form and the one thing the landlord cares about, the rent turning up, is the part that has moved.

Concept drift is that shift in the relationship, and here is the cruel part. The inputs are behaving impeccably, so no signal on the inputs can see it. The only way to see it is to wait and find out what happened to the people already decided about, and finding that out takes as long as the outcome takes to arrive. At Sumeru Bank Limited the outcome signal is the share of accepted files reaching the bank's own definition of a bad account inside its own observation window. Both definitions are the bank's own, at ninety days past due and twelve months, and are nobody's standard. Add them together and the signal cannot speak for about fifteen months.

So a firm can be roughly a year and a quarter into a genuine change in what its own answer means before the only signal capable of seeing it is even able to open its mouth. The wait is not a failure of diligence and no amount of effort shortens it. The wait is arithmetic on a definition the firm chose itself. The same arithmetic is why the six weeks at Sumeru Bank Limited were, in one narrow sense, lucky: what happened there was the other cause, the fast one, and the bank still took six weeks. Had it been the slow one, month 9 would have come and gone with every watched number looking ordinary.

What is there to watch, and how many signals does a firm actually watch?

A monitoring signalOne thing watched on a running system, with a stated source and a stated lag before it can say anything. is one thing watched on a running arrangement, with a stated source and a stated lag. Sumeru Bank Limited wrote down nine of them for the intake chain, in three groups of three, and the group a signal belongs to decides almost everything about what it can tell and when. Three watch what arrives. Three watch what the chain produces. Three watch what happened to the people it decided about.

Three of nine sounds like a coverage problem that more effort would fix. It is not. The three the bank watched were signals 4, 5 and 9, and the thing they have in common is not that there are only three of them but that every one of them sits downstream of anything that could go wrong upstream. An approval rate cannot move until decisions have been made. A referral count cannot move until files have been referred. A complaint cannot arrive until a customer has had something to complain about. Each of the three is a perfectly good signal and each of them, by construction, speaks only after the damage.

How fast can each signal speak, and which one takes over a year?

Every signal carries a lagHow long a signal takes to be capable of saying anything at all, before anybody has even read it., meaning the time it takes to be capable of saying anything at all, before any question of who reads it or how often. Signals 1 to 6 are same day: the data to compute them exists the moment the files pass through. Signal 8 needs about a week. The desk has to work the cases before its answers can be compared with the chain. Signal 9 speaks whenever a customer decides to complain, and no lag there is under anybody's control. And signal 7 needs the twelve month observation window plus the ninety day arrears definition, being about fifteen months.

Notice what this does to the shape of a monitoring arrangement. The cheap signals are the fast ones and the fast ones are the input signals. Input signals cost almost nothing. The data is already passing through the door. The expensive signal, in patience rather than in money, is the one that answers the question everybody actually cares about. The fifteen months cannot be bought out of, so a sensible arrangement stops trying and instead covers the fast end properly, on the reasoning that a fault visible on day one should never be left to a signal that reports next year.

Which three signals did this bank watch, and what did the three have in common?

In month 8, week 2, an upstream income field arriving from one channel changed format. Not a fault in any component, not a change anybody in the bank made, and not a thing any person in the chain noticed. A field that had arrived one way began arriving another way, and component 6 carried on scoring it with numbers fitted to the old shape. Signal 1, the share of each input field arriving in the expected format, is available on the day and would have caught it on the day. Nobody was watching signal 1.

The geometry of the episode comes down to position. Ismail Sheikh, who runs the exception desk, could see his queue growing. Revathi Balan, the named accountable person for component 6, could see her approval rate. The cause had happened three boxes to the left, and nothing was standing there. Neither Ismail Sheikh nor Revathi Balan was looking at anything capable of telling them why. A monitoring arrangement is not measured by how many numbers it produces but by where in the chain those numbers sit relative to where a fault can land.

How does a real fault slip under an alert rule set in good faith?

Watching a signal does not tell anybody anything by itself. Somebody has to write down what movement is worth mentioning, and that written trigger is an alert ruleA firm's own written trigger on a signal, saying what size of movement is worth telling somebody about.. Sumeru Bank Limited wrote two, both its own invention and neither of them anybody's standard. On signal 4, a movement of more than 2.0 percentage points in the monthly approval rate. On signal 5, a rise of more than 10 per cent in the monthly referral count. Both were set by sensible people who had thought about it.

Now watch the same fault hit both rules. In month 8, half affected, the approval rate fell 0.7 percentage points, well inside the 2.0 point rule, so signal 4 said nothing. Referrals rose from 391 to 450, a rise of 59, being 15.1 per cent, comfortably over the 10 per cent rule, so signal 5 spoke. The identical 117 files that eventually moved read as 2.4 per cent of the 4,902 accepts and 29.9 per cent of the 391 referrals. Exactly the same event, measured against two different bases, produces one number that would be ignored and one number that would be acted on, and the difference is entirely in the size of the denominator rather than in the size of the problem.

Now push it to the uncomfortable conclusion. In month 9, with the fault fully worked through, the approval rate fell 1.4 percentage points from 57.0 to 55.6. By then the whole population had arrived through the changed channel and there was nothing left to affect. A fall of 1.4 points is the largest movement this fault could ever produce. The rule wanted more than 2.0. The alert rule on the approval rate was set above the ceiling of the fault rather than merely slower than it, so it could not have fired at any lag whatever. Waiting longer, reading more carefully or reporting more often would each have changed precisely nothing about signal 4.

How long does it take to notice, and what decides that?

Here is the part most firms get wrong when they describe their own arrangement. The bank produced a monitoring pack every month, on time, and the right people read it. So how did six weeks pass? Because a monthly cadence does not build in a wait of a month. A monthly cadence builds in a wait of about one and a half months, and the two halves of that wait have completely different causes.

The fault landed in month 8, week 2, roughly halfway through the month. Ten working days then passed simply waiting for month 8 to end. Nothing measured monthly can see inside its own month. A further twenty working days passed while the month 8 pack was produced and then read, in month 9, week 3. Ten plus twenty is thirty working days, and the month holds twenty working days, so thirty working days is six weeks. Neither of those two waits is anybody working slowly; both are properties of the reporting arrangement, decided long before the fault existed.

What did six weeks of not noticing cost, in files and in people?

Six weeks is thirty working days, and at the chain's rate of 430 files a working day that is about 12,900 files decided while nobody knew. Of those, 176 moved from accept into the referral bandThe range of scores where the chain sends the file to a person rather than deciding it itself., at 5.85 files a working day. Report that carefully. The number is easy to make sound like a catastrophe, and a catastrophe it was not. Every one of the 176 was decided by a person: 152 were accepted and 24 were not, being 13.6 per cent of the 176 and 0.19 per cent of the 12,900. The decline cut-off never moved and the 688 auto-declines never changed at all, so not one file was declined by the chain that would not otherwise have been declined.

The interesting cost is not in the files at all. The cost is on the desk. Ismail Sheikh runs seven people who close about 154.7 cases a working day against arrivals of 150.5. The margin is 4.2 cases a day, being 2.7 per cent of capacity. A margin of 2.7 per cent sounds like slack. Now add 117 more referrals a month, or 5.85 more a working day. Arrivals go to 156.35 against a capacity of 154.7, the open case count grows by 1.65 a day, and across the thirty working days that is 49.5 more cases, so the queue moved from about 285 open cases to about 335. The spare capacity was only 2.7 per cent wide to begin with, so a drift narrow enough to move the approval rate by 1.4 percentage points consumed all of it. The 117 extra files also cost the desk 117 times 19 minutes, being 2,223 minutes a month.

What did a change in the data do that a change in the setting would have done?

Sumeru Bank Limited keeps its own sweep of the accept cut-off in the monitoring pack, showing what the month's counts would look like at settings above and below the one it actually uses. The steps are equal effect steps of about 59 files each, labelled by the movement each produces rather than by a position on the score scale, and the rows below the deployed setting are equal effect steps rather than score points because the bands there are sparse. The measured band by band sweep of the score itself belongs with the credit material.

Read the two shaded rows again. Month 8 landed on exactly the plus 3 row and month 9 on exactly the plus 6 row, and nobody moved the cut-off. A change in the data did precisely what a deliberate change in the setting would have done. The outputs of a system can therefore never say which of the two has occurred. The coincidence is not a curiosity. One of the two is a decision somebody took, wrote down and can defend to a supervisor, and the other is a fault nobody in the building knows about. Only an input signal or a change record separates them, and this bank had neither pointed at the right place.

How to Monitor an AI System for Drift: what does the arrangement actually need?

Everything above collapses into four decisions, and a firm that has made all four has a monitoring arrangement rather than a set of numbers. The first is coverage across the three groups rather than within one of them: at least one input signal. An input signal is the only thing that speaks on day one, and it costs almost nothing when the data is already passing the door. The second is the base each rule sits on, chosen so that the fault of concern is large against it rather than small. The third is the cadence. As the control above shows, a monthly pack builds in a wait of about one and a half months whatever anybody does. The fourth is who is accountable for reading it and what that person is empowered to do next.

What does a person actually do with this on a Monday morning?

Take three people and the same arrangement. Ismail Sheikh, who runs the exception desk, is not reading a monitoring pack at all. He is looking at his open case count, a genuine drift signal wearing the clothes of an operations number. The queue moves before any pack does. Revathi Balan, the named accountable person for the scoring model, is reading the approval rate, and the thing she should be asking is not whether it moved but what the smallest movement is that her rule would catch, and whether any plausible fault produces one that large. Neelima Rao, in the risk function, is reading none of it week to week, and her question is different again: what is the longest a fault can run before any of these signals speaks? At this bank the answer was thirty working days before the correction, and it would have been nothing at all if signal 1 had existed.

The cost side is small, and worth stating plainly. The intake chain cost Sumeru Bank Limited Rs 2,40,00,000/- to build once and Rs 65,00,000/- a year to run, and the exception desk runs at an assumed fully loaded Rs 9,00,000/- a year a post. Against that, the six same day signals compute themselves out of data already flowing through the chain. The input signals were not skipped because they were expensive; they were skipped because nobody had asked what would speak first if something upstream moved.

Where does Model Risk Management sit, and what does monitoring report into?

Model risk workThe standing discipline in which a firm holds an inventory, a named accountable person and a periodic re approval for anything producing a value a decision uses. is the standing discipline a bank already has for anything producing a value that a decision uses. Model risk work holds an inventory of such things, a named accountable person for each, a periodic re approval, and an independent challenge by somebody who did not build the thing. Model Risk Management long predates every part of this chain. At Sumeru Bank Limited it held exactly one of the nine components, component 6, the one producing a value a credit decision uses. Monitoring reports into that discipline; it is not a separate hobby run by the technology side.

Reporting into that discipline is not the same as being detected by it. At month 12, before the annual re approval, Neelima Rao spent eleven working days on the chain, being 77 hours at the bank's seven hour working day, and wrote seven findings, two of them on the monitoring arrangement. Writing findings on the monitoring arrangement is exactly what independent challenge is for. But there is something it cannot be. An episode thirty working days long covers thirty of every 240 working days, so a once a year validation happens to sit inside such an episode in 12.5 per cent of the ways it could fall. A quarterly one, at a standing cost of 44 working days a year rather than 11, reaches 50 per cent. Catching every episode by validation would mean validating continuously, and continuous validation is a monitoring arrangement with a much more expensive name. Validation is a challenge to the design and must never be funded as a detector.