Model Risk vs Data Risk: Two Failures, Two Different Fixes

No. A data failure is a wrong input into a correct rule. A model failure is a wrong rule applied to correct inputs. The two failures arrive in different places, they are found by different people, and they need different fixes. And the output cannot separate them: at the invented Vindhya Commercial Bank Limited, a wrong valuation and a wrong haircut produce exactly the same expected loss of Rs 2.62 crore.

Any number a bank reports is a rule applied to inputs. So if the number is wrong, exactly one of three things has happened: the inputs were wrong, the rule was wrong, or both. The number itself is a single figure, and it carries no record of which. The missing record is not a weakness of banking systems, and no better system fixes it. A single figure that carries no account of its own making is a property of arithmetic.

If a reported number is wrong, how many ways are there for that to have happened?

Three, and it is worth being slow about them. Most confused conversations in a risk meeting are people arguing about two of the three without saying which they mean. The inputs can be wrong while the rule is right. The rule can be wrong while the inputs are right. Or both can be wrong at once. A reported number is nothing except a rule applied to inputs, and a rule applied to inputs has exactly two things in it that can break. There is no fourth possibility.

Start away from banking altogether. A scooter owner works out how much petrol the machine uses per kilometre, taking the litres put in and dividing by the kilometres on the odometer. Two things can go wrong. The odometer can be broken, so the kilometres figure is not the distance actually ridden: the arithmetic is perfect and the answer is nonsense. Or the odometer can be fine while the division runs the wrong way round, litres into kilometres instead of kilometres into litres: the reading is perfect and the answer is nonsense. Both routes produce a confident number with a decimal point in it, and the number looks exactly the same either way. Nothing about the answer reveals which of the two happened, and no amount of staring at the answer will help.

The scooter is the whole argument in miniature. A bank is a much bigger version of the same shape: many more inputs, many more rules, many more answers, and the same complete silence from the answer about which of its two parents let it down.

What is a data failure, and what is a model failure?

Both definitions first, in full, before either of them is compared with anything. The two are so often collapsed into one phrase in a meeting, and the collapsing is where the harm starts.

A data failureA wrong input fed into a correct rule. The answer is wrong and the method is not. is a wrong input into a correct rule. Every part of the method is sound. The computation does exactly what it was designed to do, in the right order, with the right logic, and it would have produced the right answer if it had been handed the right number. The input it was handed was wrong: absent, or out of date, or belonging to a different thing, or simply mistyped. The method never misbehaved. In a data failure there is nothing to fix in it. What has to change is what reaches it and what happens when nothing reaches it.

A model failureA wrong rule applied to correct inputs. The answer is wrong and the data is not. is the mirror image: a wrong rule applied to correct inputs. Every value arrived on the day it should have, from the place it should have come from, carrying the figure it claimed to carry. Nothing in the plumbing misbehaved either. The wrong part sits in the middle: the rule that turns those correct inputs into an answer rests on a choice nobody ever tested, or a specification that says the wrong thing, or a step that was right for the world of five years ago. The data was never wrong. In a model failure there is nothing to fix in it. What has to change is the rule, and changing a rule is a slower and more political act than repairing a feed.

The household version helps here too. A house runs on a monthly budget: take what comes in, subtract what usually goes out, and treat the remainder as what can be committed. A data failure is the electricity bill being read off last month's slip because this month's never came. A model failure is the phrase what usually goes out being defined, years ago, in a year with no school fees, no wedding contribution and no ageing parent in it. The first is a wrong number in a good rule. The second is a good number in a rule that stopped describing the house. The two feel like the same complaint, that the budget was wrong, and they are repaired by two completely different acts.

Vindhya Commercial Bank supplies one clean instance of each in the same twelve months, and both are worth naming now and then leaving alone for a moment. The data failure is incident I10: a collateral valuation feed stopped arriving in month 10, the last value it had sent was used for 11 working days, 340 loans were wrongly marked, and no customer lost money. No model was wrong anywhere in that. The model failure is V1, the behavioural deposit life model, one of the three in this bank that have never been validated. Every input it needs arrives correctly. The rule those inputs are fed into has never been tested.

What did the data failure look like while it was actually happening?

The data failure looked like nothing, and the invisibility is the reason it ran for as long as it did. In month 10 of this invented bank's twelve month year, the feed that delivers collateral valuations stopped arriving. The feed did not send a blank, it did not send an error, it did not fall over in a way that anybody could see. The feed simply sent nothing, and downstream the last value it had sent stayed exactly where it was and carried on being used, day after day, looking entirely normal on every screen it appeared on.

A stale valueA number that is present, looks normal and is older than it should be. A data failure can run for days on one. is the most patient kind of wrong number there is. A missing value announces itself: something breaks, a report has a gap, somebody telephones somebody. A stale value is present, it is formatted correctly, it sits in the right cell, and it is wrong only in the one dimension nobody was looking at, which is age. So a stale value announces nothing. The whole of incident I10 is the difference between a number being absent and a number being old, and only one of those two states is visible.

Here is what the record says, and every figure of it belongs to the invented bank. The feed was stale for 11 working days. In that time 340 loans were wrongly marked, and the collateral valuation that runs behind Rs 8,640 crore of secured advances was running on a figure that had stopped moving. No customer lost money. The gross loss was Rs 1.4 crore, there was no recovery, and the net loss was therefore also Rs 1.4 crore. The year's net operational loss was Rs 43.8 crore, so that Rs 1.4 crore is 3.2 per cent of the year. The 3.2 there is a percentage share and not a rupee amount, and the distinction matters because two other incidents in the same year each cost Rs 3.2 crore net.

And the reason it ran for 11 days rather than one is dull, specific and completely fixable. The data dictionary entry for that feed described what the value was, where it came from, who was accountable for it, who maintained it, what format it took, how often it should refresh, and how it reached the report line it fed. The entry did not describe attribute T7, the instruction for what should happen when the value does not arrive at all. Nothing was defined to happen, so nothing happened. A dictionary that describes a value and not its absence describes only the days on which everything works. Those attributes and how a data dictionary is built belong with the data governance material and are named here rather than taught again.

Two more facts from the record sharpen it. The same feed had been stale before, for 2 working days in month 6, and a data quality check caught it and nobody raised it as an issue: that is near miss N3, four months ahead of the incident, carrying exactly the same information at no cost at all. And this bank's assurance map showed second line assurance over process PR3, collateral management and valuation, as one of only two cells out of twenty seven with no assurance from anybody. The failure landed in the one place the map had already marked as unwatched.

What did the model failure look like, and why did nothing happen at all?

Now the other side, and the honest answer to the second half of the question is that it looked like nothing too, but for the opposite reason. Incident I10 looked like nothing because a stale value is invisible. Model V1 looks like nothing because there is genuinely nothing to see: no day on which it broke, no feed that stopped, no report with a gap in it, no customer affected, no rupee out of the door.

Model V1 is this invented bank's behavioural deposit life model. Current and savings balances of Rs 36,000 crore are contractually repayable on demand, so somebody has to decide how long, on average, that money actually stays. V1 decides. V1 gives those balances an average behavioural life of 0.5 years, and that figure is then used in the computation of what a rate move does to the economic value of the bank. How that computation works belongs with the market risk material and is not rebuilt here.

The behavioural life figure decides the sign. V1 sets a 0.5 year life, and on that life this invented bank's own 200 basis point scenario takes minus Rs 840 crore off economic value. The Rs 840 crore is the economic value change and not counterparty C7's funded exposure, a different Rs 840 crore in the same bank. Set a 2.0 year life instead, the midpoint of the bank's own repricing bucket RB5 where the very same Rs 36,000 crore is slotted, and the answer is plus Rs 240 crore. The rule that has never been tested is the rule that decides whether the bank's headline interest rate risk number is negative or positive.

And it rests on an assumptionA choice made where evidence does not settle the question. Most model failures live there.. Model failures overwhelmingly live in assumptions. An assumption is not a mistake and it is not laziness. An assumption is what gets written down when the evidence genuinely does not settle the question, and somebody still has to put a number in the cell so the report can be produced on Tuesday. Committee G4 sets this one. The trouble is not that a choice was made. The trouble is that a choice made because the evidence was thin becomes, four reports later, a figure that everybody treats as measured.

There is one more property of V1 that decides how it has to be handled, and it is the reason it cannot simply be checked against outcomes. The thing V1 predicts is how long a deposit stays, and that is only observable over years. There is no set of 250 days the model can be laid against to count how often it was wrong. Whether a model can be tested against realised outcomes, and how that differs from asking whether it is fit for the use it is put to, is covered separately. The outcome test is unavailable for V1, and the rule itself is left as the only thing anybody can examine.

If the answer is wrong, does the answer reveal which of the two went wrong?

No. The demonstration is worth working through slowly with real figures, and it is not easily unseen afterwards. Take one number this invented bank actually reports: the expected loss on counterparty C1, Nirjhar Industries Limited, also invented. The expected loss runs through eligible collateralA collateral value after a haircut, being the product of a data figure and a model figure. The product cannot report which term moved., and eligible collateral is a valuation multiplied by one less a haircut. Look at what those two terms are. The valuation is data, produced by a feed. The haircut is a model, and in this bank it is V2, the collateral haircut model, another of the three that have never been validated. One term of the product on each side of the divide.

The correct working first, and every figure in it belongs to the invented bank. A charge over inventory and receivables is valued at Rs 1,440 crore. At the bank's own 35.0 per cent haircut that gives eligible collateral of Rs 936 crore, and that Rs 936 crore is C1's eligible collateral and not any other Rs 936 crore in this case. The Rs 936 crore covers 936 over 2,160, being 43.33 per cent of the Rs 2,160 crore exposure at default. Coverage takes loss given default from the bank's own 40.0 per cent down to 22.67 per cent. Expected loss is 2,160 times the 0.45 per cent grade 4 probability of default times 22.67 per cent, giving Rs 2.20 crore. The Rs 2.20 crore is the answer the bank believes it has. How collateral, coverage and the expected loss product are built belongs with the credit and counterparty risk material, and the number is borrowed here rather than derived. There is a second route to the same figure that reduces exposure instead of loss given default, and it lands on Rs 2.20 crore as well: the loss given default route is used throughout, and the two are one answer and not two.

Now suppose the figure comes out at Rs 2.62 crore instead. There are two entirely different ways to arrive there.

Route one is a data failure. The valuation feed delivers Rs 1,080 crore rather than Rs 1,440 crore, an illustrative error and not something this bank recorded. The Rs 1,080 crore here is that illustrative revaluation and not counterparty C4's funded exposure and not the tolerance ceiling, both a different Rs 1,080 crore in the same case. The haircut is untouched at 35.0 per cent. Eligible collateral is 1,080 times 0.65, being Rs 702 crore. Coverage falls to 32.5 per cent, loss given default rises to 27.00 per cent, and expected loss is Rs 2.62 crore.

Route two is a model failure. The valuation is entirely correct at Rs 1,440 crore. The haircut model applies 51.25 per cent instead of 35.0, again an illustrative error and not a figure this bank uses. Eligible collateral is 1,440 times 0.4875, being Rs 702 crore, and the expected loss is therefore the same Rs 2.62 crore. Identical output. Two causes sitting in different parts of the bank, found by different people, fixed by different acts.

And a rounder wrong haircut lands almost on top of it. At 50.0 per cent, also an illustrative error, eligible collateral is Rs 720 crore, coverage 33.33 per cent, loss given default 26.67 per cent and expected loss Rs 2.59 crore. The Rs 2.59 crore sits Rs 0.03 crore away from the data failure answer. Expected loss here depends on eligible collateral alone, eligible collateral is one number multiplied by another, and a product carries no record of which of its two terms moved. So nothing downstream of eligible collateral can separate any of these.

Is there a haircut at which the two produce exactly the same answer?

Yes, and it is not an accident or a coincidence worth marvelling at. The equality is forced by the shape of the arithmetic. Eligible collateral is one number multiplied by another. Anything that reduces the product by a given proportion has the same effect on the product, whichever of the two terms it acts on. A valuation cut by a quarter and a haircut widened enough to remove that same quarter are, from the product's point of view, the same event.

Work the exact figure. The data failure delivers a valuation of Rs 1,080 crore against the correct Rs 1,440 crore, a 25 per cent understatement, and at the correct 35.0 per cent haircut gives Rs 702 crore of eligible collateral. To land on Rs 702 crore from the correct Rs 1,440 crore valuation, the haircut model must retain 702 over 1,440, being 48.75 per cent, so it must remove 51.25 per cent. And 1,440 times 0.4875 is 702, exactly as 1,080 times 0.65 is 702. At a haircut of 51.25 per cent the model failure and the data failure are not similar, not close and not approximately equal: they are the same number, to the rupee, and everything computed from that number afterwards is identical too.

The crossing is not a point, so widen the idea. Every pair of a valuation and a haircut whose product equals Rs 702 crore gives eligible collateral of Rs 702 crore and therefore an expected loss of Rs 2.62 crore. Rs 1,080 crore at 35.0 per cent, Rs 1,440 crore at 51.25 per cent, Rs 900 crore at 22.0 per cent, Rs 2,340 crore at 70.0 per cent: all of them one answer. The crossing is a whole line of points, and every one of them is a different story about what went wrong. The reported figure identifies the line and never the point on it.

Why do the two show up in completely different places in a bank's records?

Because the records were built to answer different questions, and only one of those questions has an answer when the failure is a rule. Vindhya Commercial Bank keeps five records that could in principle carry either of these two failures, and setting the two against the five is where the distinction becomes sharpest.

Incident I10 is in four of the five. The incident is in the loss logThe record of operational loss events. The log captures things that happened and structurally cannot capture a number that was always wrong. with a gross of Rs 1.4 crore, no recovery and a net of Rs 1.4 crore. I10 is in the incident register with a month and a duration of 11 working days. I10 is in the control findings as the year's one material weaknessThe most severe control finding rating, used where a deficiency creates a reasonable possibility that a material misstatement goes undetected., rated D4 on this bank's own four point scale, affecting the valuation of Rs 8,640 crore of secured advances. And it is entry RR3 on the risk register, one of that register's four reds. Four separate records, kept by different teams for different purposes, all caught the same thing.

Model V1 is in two. V1 is entry RR4 on the risk register, the behavioural deposit assumption, another of the four reds. And it is on the model inventory, as one of the three that have never been validated. V1 is in nothing else. There is nothing else for it to be in: it produced no incident, no loss, no control finding and no exception all year. Three of the five records are structurally blind to it, and the one record that caught both failures is the only one that describes states rather than events.

What is different about a register, and why did that one catch both?

A risk registerThe record of risks an institution has identified. A register can carry a model failure precisely because it records states rather than events. is built to answer a question no other record on that list asks. A loss log asks what happened and what did it cost. An incident register asks what happened, when, and for how long. Control findings ask which controls did not work when somebody tested them. All three are questions about events, and an event has a date. A register asks something else entirely: what is true about this institution right now that could hurt it. A state does not need to have occurred on a Tuesday in order to be recorded.

A register records states, and that is why this bank's register carries both. Entry RR3 is the collateral valuation control, the same object as the material weakness and the same object as incident I10. Entry RR4 is the behavioural deposit assumption, the same object as unvalidated model V1. Both are red, out of 46 entries carrying 4 reds, 12 ambers and 30 greens. Neither is a new discovery: every one of the four reds is already somewhere else in this bank's records, and that is the point rather than a criticism. A register that agrees with the breach log, the loss log and the model inventory is working, and a register that disagrees with them is the usual finding. This one agrees because it was rebuilt in month 6, and rebuilding it is the reason RR4 exists at all.

The difference between a state and an event has a consequence for anybody trying to see their own model risk. Failures that never happened on a date can be found only in the one record designed to hold things that are simply true, and only if they were put there. Nothing else on the list will surface them.

Which one does a loss log find, and which one does it structurally miss?

Do the two need different fixes, and who does each one?

The two do need different fixes, and the fixes have almost nothing in common except that both begin with somebody admitting which of the two they are looking at. The admission is the whole point of keeping the two apart. The diagnosis has to happen upstream of the answer, and the answer will not supply it.

Fixing incident I10 is small, concrete and quick. One attribute goes onto one feed's dictionary entry: what happens when the value does not arrive. Somebody has to decide what should happen, write it down, and make sure something visible occurs when it does. The person who can do that is the named data owner for that element, a business person accountable for the element being right, and this bank has one. The work is days rather than months, and once it is done the same failure cannot run for 11 days again. A rule for absence does not decay, so the fix stays fixed.

Fixing model V1 is none of those things. Nothing in the plumbing needs repairing. Somebody independent has to examine the rule itself: what it is for, what it rests on, what it cannot support, and what should be written down about its limits. The person who can do that is the independent validator in the risk function, who built none of what she checks, and after her the committee that set the assumption in the first place has to look at it again. The committee is G4, the setter of this bank's behavioural assumptions. And there is one more wrinkle already noted. How long a deposit stays is only observable over years, so the outcome test is not available and the rule is the only thing there is to examine. A data owner cannot validate a model, and a validator cannot write a data dictionary entry, and neither of them should try.

Which is why the failure mode in a meeting is so specific. Both of these produced a number somebody did not trust, so both get described as a data quality problem. The phrase is short and everybody nods. Both then go to whoever holds data quality, and one of them cannot be fixed there at all. The request does not come back as a refusal, it comes back as a delay, and eight months later the assumption that decides the sign of a headline figure is still exactly where it was. Root causeWhy a gap exists. A finding closed without it comes back. gets skipped when a label is applied too fast, and a finding closed without it comes back.

Can both be present at once, and what does that look like?

Both can be present at once, and the pair defeats every shortcut, including the ones set out above. Suppose the valuation is stale and the haircut is wrong in the same month. What can be said about the expected loss figure?

Only that it is wrong. Not by how much, not in which direction, and not which of the two errors is the bigger. The two can push the same way, in which case the figure is badly wrong and at least somebody is likely to query it. Or they can push opposite ways, in which case they partly cancel and the figure looks a little off. Or they can cancel exactly, in which case the figure is correct and there is nothing to notice at all. An offsetting pair is the hardest case in this whole subject. The output looks right while nothing underneath it is.

The arithmetic makes that concrete. Take a valuation of Rs 1,800 crore, far too high, and a haircut of 48.0 per cent, far too wide. Both are illustrative errors and neither is a figure this bank uses. The Rs 1,800 crore here is that illustrative valuation and not the share of non-maturity deposits slotted in the first liquidity bucket, a different Rs 1,800 crore in the same case. The product is 1,800 times 0.52, or Rs 936 crore, exactly the correct eligible collateral. Coverage comes out at 43.33 per cent, loss given default at 22.67 per cent, and expected loss at Rs 2.20 crore, precisely the number this bank believes it should be reporting. Two errors, one correct answer, and no committee on earth would ask a question about it. That is point D on the controls above, and it is worth landing on.

So the honest closing position is uncomfortable and worth stating plainly. Checking outputs finds some wrong numbers and misses others, and it never tells why. The only place the distinction can be made is upstream: at the feed, where the questions are when this value last changed and what happens when it does not arrive, and at the rule, where the questions are who chose this and what tested it. Neither question can be answered by looking at the answer.

Who actually uses this distinction, and what do they do with it?

Four people pick this up for four different reasons, and watching what each does with it is the quickest way to see why the distinction is worth the trouble.

A credit analyst at another institution, looking at this bank from the outside, uses it to decide which questions are worth asking. A published expected loss figure tells her something about the borrower and nothing at all about which of its two parents is sound. So she asks about the feed, when the collateral valuation last refreshed and what happens when it does not, and she asks about the haircut, who set it and what tested it. Neither question is answerable from the number, and both are answerable on a call. An analyst who only interrogates outputs is interrogating the one thing that cannot answer.

A lender inside the bank uses it to decide whether a limit is really being respected. If the eligible collateral behind a secured exposure is the product of a feed and an unvalidated rule, then the coverage figure on the credit paper is a claim about two things and it should be read as two claims and not one. Reading it as two claims does not stop anybody lending. The reading changes what the person asks before signing.

An internal auditor uses it to decide where to look. Reading the loss log shows where money went, and that is exactly the wrong map for finding rules nobody tested. The assurance map here already showed second line assurance over collateral management as uncovered before anything happened. And the head of operational risk uses it to keep two queues apart. A queue that mixes feeds and rules gets sorted by whichever is easier, and feeds are always easier.

The mechanism does not change with scale, so take the household version. Take one number a household actually relies on: what it believes it can afford as a monthly instalment. Two separate questions apply to it. Are the numbers going in still true, meaning the income, the rent, the school fee, the amount that actually leaves the account. Then, separately, is the rule still right, meaning the thing done to those numbers to get an answer. Most people check the first and almost nobody checks the second, and the second is the one that quietly stopped describing the house three years ago. The bank version needs a validator and a committee. The household version needs one honest evening.

What sits outside this distinction?