How to run Model Validation: Seven Steps and a Written Report

Validation runs in seven steps. Scope and tier the model, test the data, challenge the assumptions, replicate the implementation, analyse outcomes where any exist, write the limitations as instructions, then produce a report carrying rated findings, a use restriction and a signature. The fifth step is often empty, and a model with no observable outcome still gets the other six.

The seven step sequence is the shape the work takes, and it is not a standard handed down from anywhere. An institution's actual obligation to check its models is set by its own supervisor, and the requirement lives in that supervisor's text rather than in any method. The seven steps are one workable structure that a single person can hold in their head, and what actually binds a given institution is settled against the supervisors named below.

Everything below rests on one idea about method, and it is worth stating before the first step arrives. A validation is not a test, it is a sequence of tests on different objects, and each object has a different owner. The data belongs to a source system. The assumptions belong to a committee. The implementation belongs to whoever built it. The use belongs to whoever runs the model and acts on what comes out. A method that treats a model as one thing produces one verdict, and there is no route to a fix inside a verdict. A method that separates the objects produces findings somebody can pick up. Each one lands on a desk that can do something about it.

The household version comes first, so the shape is familiar before the vocabulary arrives. A household works out what it can afford as a monthly loan instalment: take-home pay in, usual spending out, keep half of what is left. Now suppose the answer looks wrong. Four separate things could be wrong with it and they are fixed by four different people. The pay figure could be stale, and only whoever reads the payslip can settle it. Keeping half could be the wrong rule, and only whoever chose the rule can settle that. The subtraction could have been done wrongly on the paper, and neither of those two people would know. Or the answer could be perfectly correct for a two year loan and be getting used for a seven year one, and use is a question all of its own. Asking whether the number is right produces an argument. Asking the four questions separately produces four answers, three of which are probably fine.

What is a validation for, and what may the person running it never have done?

A validation answers one question: is this model fit for the use it is actually being put to? Not is it clever, not is it the best available, not does it agree with last year. Until somebody writes the use down, the question has no answer at all. The missing statement of use is why the first step produces a document rather than a test result, and why a validation that starts by opening the spreadsheet has started in the wrong place.

The second thing that defines a validation is who runs it. The person doing the checking must have built none of what they check, and that is a condition on the person rather than on the method. At the invented Vindhya Commercial Bank Limited, that person is Kanaka Murthy, the independent validator sitting in the risk function, and the bank's rule is the plain one: she validates nothing she built and nothing she specified. The rule is not a comment on anybody's honesty. Somebody who chose an assumption has already settled once whether it was worth choosing, and will not ask again with an open mind.

The method below has seven steps, numbered MV1 to MV7, worked end to end on one model that has never been checked at all. Each step takes something in, does one thing to it, and hands something on. Nothing in the sequence is clever. The value is in the fact that it is a sequence: running the assumption challenge before the data test costs an afternoon of argument about a choice that was being fed rubbish anyway.

How is a validation scoped and tiered, and what does the tier change?

The validation scopeThe written statement of which model is being checked, for which use, and how deeply, agreed before any work starts. is a short document and it is agreed before anything is opened. The scope names the model, it names every use the model is put to, and it states the depth. Agreeing the scope first stops the argument at the end. The only thing anybody can then dispute is whether the work was done rather than whether it was the right work. A scope written after the testing is a scope written to fit the testing.

The tier is the second field, and the tier decides depth. The tier is a materiality rating the institution sets for itself: how much money moves if this rule is wrong, how many decisions rest on it, whether it feeds a limit, a capital number or a committee paper. Every model gets the same seven steps. The tier decides how far each step goes, how often the whole sequence is repeated, and how senior the person signing at the end has to be. Vindhya Commercial Bank Limited runs three tiers on its own test, and its register at month 12 splits 28 models into 6 in tier 1, 13 in tier 2 and 9 in tier 3, and six plus thirteen plus nine is twenty eight.

Now the part of MV1 that is easy to skip, and it matters on the model this guide works through. The tier on the record is an input to the scope rather than a fact about the world, and MV1 is allowed to challenge it in writing. A depth taken straight off a register is a depth chosen by whoever filled the register in. A validation exists to test exactly that kind of choice. Hold that thought: it comes back the moment the worked instance reaches model V1.

What is actually tested about the data a model runs on?

Step MV2 is the one people expect to be dull, and it is where a surprising share of the findings come from. MV2 is not a data quality review of the whole institution, and it is not the governance of the data. Data governance is a separate subject with its own owner and its own list of attributes. MV2 asks four questions about the specific inputs this specific model consumes, and each one gets a verdict written down beside it.

The fourth test is the one that catches people, so it is worth holding apart from the other three. Tests one to three ask whether the data is any good. Test four asks whether there is enough of it to answer the question the model is being asked, and a feed can be flawless and still fail. Two years of balance history is a perfectly good history. Two years is simply not long enough to see how balances behave through a full turn of the rate cycle, and a rule about behaviour through the cycle that has never seen one is a rule fitted to a fragment.

Two more things MV2 hands on rather than settles. MV2 records the source system against every input, and it records how the model behaves when a value does not arrive. Recording the source is what makes the next person's job possible at all. Vindhya Commercial Bank Limited has an instructive gap there. Across the 147 data elements feeding its monthly risk report, the attribute most often missing is the one saying what happens when a value is absent, and it is missing on 105 of them. A data dictionary that describes the value and never its absence describes only the days when everything works. Whose job it is to close that gap is a data governance question, covered separately.

How is an assumption listed, and who does each one belong to?

An assumptionA choice made because evidence does not settle the question, which is why an assumption always has an owner and a date. is a choice made because the evidence does not settle the question. The definition is short and it carries a consequence: somebody chose it, so somebody can be named, and it was chosen on a day, so it can be dated and revisited. An assumption with no owner is not an assumption, it is a habit. Nobody is answerable for a habit, so nobody reviews it. Listing them is most of the work in MV3, and the list is longer than the model's authors expect, because a choice made once and never argued about stops looking like a choice.

At the invented Vindhya Commercial Bank Limited the assumption followed here is a single number. Model V1, the behavioural deposit life model, gives the Rs 36,000 crore of current and savings balances an average behavioural life of 0.5 years. The current and savings balances are contractually repayable on demand, so no contract fixes the answer and something had to be chosen. Committee G4, the asset liability management committee, chose it. Vindhya Commercial Bank Limited makes that committee responsible for behavioural assumptions. So the entry on the list reads: assumption, 0.5 years; owner, committee G4; evidence, the bank's own review. MV3 writes that down and moves to the challenge.

How is an assumption challenged when there is nothing to compare it with?

An assumption with nothing to compare it against is the hard case, and it is the normal case rather than the exception. How long a deposit stays is only visible over years and the record holds no such measurement, so the 0.5 year figure cannot be checked against what actually happened. There is nothing to compare it with. The temptation at this point is to write that the assumption looks aggressive. An opinion is all that is, and an opinion loses an argument with the committee that chose it.

The way out is to stop asking whether the assumption is right and start asking how much it matters. The second question has a name, the range testRecomputing an answer across a range of an assumption to see how far the answer moves and whether it changes sign.: recompute the answer across a plausible range of the assumption and report where the answer goes. The range test converts an argument about a number nobody can settle into a measurement of consequence. Anybody can settle a measurement. And on model V1 it produces something much sharper than a range.

The arithmetic is the bank's own and it is short. Extending the assumed life by one year moves the economic value result by Rs 36,000 crore times the bank's own 2.0 per cent scenario. Rs 36,000 crore at 2.0 per cent is Rs 720 crore. Start from the locked point: at 0.5 years the economic value change is minus Rs 840 crore. Walk it forward. At 1.0 year it is minus Rs 480 crore, at 1.5 years minus Rs 120 crore, at about 1.67 years it is zero, at 2.0 years it is plus Rs 240 crore, at 2.5 years plus Rs 600 crore and at 3.0 years plus Rs 960 crore. The assumption is not adjusting the size of the answer, it is deciding which way the answer points, and that is a finding rather than an opinion.

A reader who skips the picture still needs two things from it. First, the answer stays comfortably inside the relevant limit at every point drawn. Limit L8 caps the economic value sensitivity at Rs 990 crore. The widest reading drawn is plus Rs 960 crore, or 97.0 per cent of the cap. Nothing on any report would look unusual at either end of the range, and a range test is worth running precisely for that reason: the failure mode is not a breach, it is a sign.

Second, and this is the part that makes it a validation finding rather than a curiosity, the range is not invented for the occasion. Vindhya Commercial Bank Limited's own repricing ladder slots the same Rs 36,000 crore at one to three years in bucket RB5. So the bank holds two views of one balance at the same time. Across bucket RB5 alone, from 1.0 year to 3.0 years, the answer runs from minus Rs 480 crore to plus Rs 960 crore and flips sign at about 1.67 years. The minus Rs 840 crore figure is the reading at the 0.5 year life the model itself assumes, and a 0.5 year life sits outside bucket RB5 entirely. The two must not be blurred into one range. The strongest form of a range test is a range the institution has already committed to somewhere else in its own reporting.

What does replicating an implementation mean, and why is it separate from checking the rule?

ReplicationRebuilding a computation independently from the same inputs, to separate a wrong rule from a rule wrongly coded. means rebuilding the computation from the written specification, using the same inputs, without looking at how the original was built, and then comparing. Replication is slow and it feels redundant right up until the moment it is not. The reason it earns a step of its own is that steps MV3 and MV4 catch two faults that look identical from the outside and are repaired by two completely different people.

Think of a household again. Two people work out the monthly instalment they can afford and get different answers. One possibility is that they disagree about the rule, one keeping half of what is left and the other keeping a third. The other possibility is that they agree completely about the rule and one of them added a column wrongly. From the outside both look the same: two numbers that do not match. Inside, the first needs a conversation and the second needs two minutes with a calculator. A wrong rule and a right rule wrongly built produce the same wrong number. One step therefore asks about the rule and a separate step asks about the build.

MV4 is where the independence condition does real work, and that has one practical consequence. A rebuild that starts by reading the original code will reproduce the original code's mistakes with great fidelity, so the rebuild is made from the specification and not from the original workings. If the specification is too thin to rebuild from, that is itself the finding, and it is a common one. Model V3 at this bank is a spreadsheet scorecard maintained by one person with no written specification, and MV4 cannot be run on it at all. There is nothing to rebuild from and nothing to compare against.

What is done when the model cannot be tested against any outcome?

Outcome analysisComparing what a model said with what happened, which is only possible where the outcome can actually be observed. is the step everybody pictures on hearing the word validation, and it is the step most likely to be empty. Outcome analysis needs two things: the model must have said something on a past date, and the thing it said must have been observable afterwards. Plenty of models fail the second condition through no fault of anybody's.

Model V1 fails it completely. Model V1 predicts how long a balance stays, and that is only visible over a run of years. The bank's record holds no realised deposit life at all, so there is nothing to place beside the model's output. Note carefully what that does and does not mean. Nothing in an empty section says the model has been checked and found sound on outcomes. The absence means the check cannot be performed, and a check that cannot be performed is a different statement altogether. The difference is exactly what the report has to carry.

There is a second reason to write the absence down rather than skip the section, and it decides how the whole report reads. Outcome analysis is the safety net, and a model whose output cannot be tested against any outcome has none. A model with no safety net most needs the other six steps. Where an outcome exists, a weak assumption eventually shows up as a run of misses and somebody notices. Where no outcome exists, nothing ever shows up, and the only thing standing between a bad assumption and a committee paper is the data test, the assumption challenge and the rebuild. Writing that in section 5 is what tells a reader to weigh sections 2, 3 and 4 more heavily than they otherwise would.

How is a limitation written so that somebody downstream can act on it?

A limitationA statement of what a model cannot do, written so that a user knows what to do about it rather than merely that it exists. is a statement of what the model cannot do. Almost everybody writes them as caveats, and a caveat is a sentence everybody nods at and nobody acts on. The test is simple and slightly brutal: read the sentence and ask what a person would do differently on Monday morning because of it. If the answer is nothing, it is a caveat.

Take the one from model V1. The caveat version reads that the economic value figure is sensitive to the assumed deposit life. Perfectly true, entirely inert. Nobody's behaviour changes. The instruction version reads that the economic value figure must be reported with the assumed deposit life printed beside it, every time it appears. An instruction can be followed or not followed, and so it can also be checked. A caveat can be neither. Same fact underneath, two completely different documents.

The pattern generalises. A limitation written as an instruction names who does what and when. The output must always be reported alongside a stated figure. The model must not be used beyond a stated boundary. The answer must be recomputed when a stated input changes. Each of those is checkable a month later by somebody who was not in the room. Checkability is the whole trick, and it is the difference between a report that describes a model and a report that changes how the model is used.

What goes in the validation report, and what changes because of it?

The report is the output of the whole sequence and it has seven parts. Five of them describe. Two of them act. A report that stops after the findings has described a model, and a report carrying a use restriction has changed what somebody is allowed to do on Monday. That is the only real test of whether MV7 was done or merely written.

The use restrictionThe written statement of what the model may and may not be used for, which is what a validation report exists to produce. is the part most often left out. Writing one feels like overreach. It is not. MV7 closes the sequence MV1 opened: MV1 asked which use the model serves, and MV7 answers whether the model is fit for exactly that use and for nothing wider. On model V1, a restriction that follows from the work would say that the economic value output may be used for internal reporting only when it is accompanied by the assumed deposit life, and may not be used to support a statement about the direction of the institution's rate exposure until the assumption has been tested.

What makes a finding usable rather than merely true?

A findingA gap between what was found and what should have been, carrying a cause, an effect and an agreed action with an owner and a date. is a gap between what was found and what should have been. Written properly it has four parts, and this invented bank uses the same four in its audit work, numbered AF1 to AF4, with the agreed action carried as AF5 alongside. The condition is what was found. The criteria is what should have been. The cause is why the gap exists. The effect is what it could lead to.

The cause decides whether a finding can be fixed or only patched, and the cause is the part most often missing. A finding with a condition and an action but no cause can always be closed, because somebody does the thing once and ticks the box, and then it comes back, because nothing changed about why it happened in the first place. In this invented bank, 9 of the 42 control testing findings carry no cause at all.

What happens when a finding cannot be closed on time?

Some findings cannot be closed inside the date agreed, and a method that has no answer for that leaves its own output rotting on a list. The answer is an escalation, and an escalation is three things written down in advance: who is told, on what timetable, and what they are being asked to decide. Not who is blamed. The decision is usually whether to accept the position for a further period with a dated plan, or to restrict the use of the model until the work is done.

Where the escalation goes is a question about this institution and not about the method, and it is worth naming because the answer is often surprising. In this invented bank, the risk taxonomy places model risk inside operational risk, so the model inventory and its validation status are reported to committee G6, the operational risk management committee. Meanwhile committee G2, the board risk management committee, sets every limit L1 to L12 and reads the numbers those models produce. The body that receives the validation status is not the body that relies on the models, and that placement is a choice the institution made rather than a rule it followed. How committees are chartered and how a decision is minuted belong to the governance material and are covered separately.

One more thing to keep an eye on. Good methods quietly fail here. An open finding has two different measures on it and they are not the same: how old it is, and whether it is past the date agreed for fixing it. An issue can be old and entirely on track, or raised last week and already late. The bank carries 92 open issues aged in five buckets, and 31 of them are past their agreed remediation date, being 33.7 per cent. A function reporting the first measure and managing on the second is doing it in the right order.

What does the whole method look like run end to end on one model?

The sequence now runs on the model the earlier steps have been circling. Vindhya Commercial Bank Limited runs MV1 to MV7 on model V1, the behavioural deposit life model, and model V1 has never been validated at all.

Two things in that run deserve pulling out, and the first is the one MV1 produced. The register does not place model V1 in tier 1, and the arithmetic makes that certain rather than likely. This bank's register carries 6 models in tier 1, and of those 6, 5 are validated and current and 1 is overdue. Five plus one is six, so the whole of tier 1 is accounted for, and none of it is a model that has never been validated. Model V1 has never been validated. Model V1 is therefore in tier 2 or tier 3, and the record does not say which. The model that decides the sign of this bank's headline interest rate risk number is not carried at its top materiality tier. MV1 exists to notice exactly that and write it down before any depth is accepted from the record.

The second is what MV3 found, and it is worth restating in one line because it is the payoff of the whole run. The range test moved the answer from minus Rs 840 crore at the model's own 0.5 year life, through zero at about 1.67 years, to plus Rs 960 crore at 3.0 years, and across the bank's own repricing bucket RB5 of one to three years alone it runs from minus Rs 480 crore to plus Rs 960 crore. Every one of those readings sits inside limit L8's Rs 990 crore cap. Nothing breaches. The finding is not about size, it is that one untested choice decides whether this institution reports its value exposure as a loss or a gain under the same scenario.

What does holding the cycle actually cost, and what happens to the queue?

Who reads a validation report, and what do they actually do with it?

A method is easier to hold on to once the audience for the document is clear, and it is read by four different people for four different reasons. Each one reads a different part, so they are taken in turn.

The person running the model reads the limitations and the use restriction, and nothing else. The model runner wants to know what changes on Monday: which figure now has to be published with something printed beside it, and which question may no longer be answered with this output. Parts 3 and 4 of the report are written as instructions rather than as observations for exactly that reason.

The head of internal audit, who at this invented bank is Rustom Batliwala reporting to the audit committee, reads the scope and the findings. Audit is not repeating the validation. Audit is testing whether the validation was done. Testing that means reading the scope to see what was promised and the findings to see what came back, and then checking whether the agreed actions are being closed on time rather than merely being closed.

The committee that relies on the number reads the use restriction and one line of the range. A committee does not want the method. The committee wants to know whether the figure in front of it may be used for the decision in front of it. A report whose fourth part is missing sends a committee a very careful document that answers a question nobody at the table asked.

And the fourth reader is the next validator, two years later. The next validator reads the whole thing. The fastest way to run a validation is to start from the last one and test what has changed. The quiet argument for writing an absence down rather than leaving a section blank sits here: the next person needs to know whether the outcome test was skipped or was impossible, and only one of those two is worth attempting again.

The household version of the same document is a note stuck to the fridge. Not a note that says the affordability sum is uncertain. Such a note changes nothing. A note that says the instalment figure was worked out on a year with no wedding in it and must be redone before anybody commits to a seven year loan. Same facts, and one of the two versions actually stops a decision.