Model Validation vs Backtesting: Two Different Questions

Validation and backtesting are not two names for one job. A backtest compares one model's output with what actually happened over a window, and can only say that the output looks wrong. Validation asks whether a model is fit for the use it is put to, reaching its data, assumptions, implementation and limitations. Validation can fail a model that backtests perfectly, and at the invented Vindhya Commercial Bank Limited one model cannot be backtested at all.

Two words get run together in almost every conversation about checking a model, and the merge is expensive rather than merely untidy. A bank that thinks it has checked a model because the exception count came back looks at the wrong thing for a year. Worse, a bank that aims its checking where the checking is easy ends up with its most consequential rule untouched. Vindhya Commercial Bank Limited is in exactly that state. The two come apart properly only when each one is examined for what it is handed before what it concludes.

Are validation and backtesting the same activity under two names?

No, and the fastest way to see it is to sit outside the room and watch what gets carried in. An activity can only ever conclude something about what it was given. The claim sounds like a truism, and every difference between the two activities follows from it.

A backtestA comparison of a model's outputs with what actually happened, over a stated window, one observation at a time. is handed two things and only two things. First, a stream of what the model said, one figure per period. Second, a stream of what then happened, one figure per matching period, lined up in time so that each pair belongs to the same day. The two streams are the entire input. From those pairs it can count how often the second number fell outside what the first number said it would, and it can note when those occasions arrived. A count of occasions and their dates is the entire output. A backtest cannot look at the code, never sees the specification, and has no idea what anybody is doing with the answer.

ValidationAn assessment of whether a model is fit for the use it is put to, covering its data, assumptions, implementation, limitations and purpose. is handed the model. Not the outputs, the model: the written specification of the rule, the data feeding it and where that data comes from, the assumptions chosen and who chose them, the code or the spreadsheet that actually runs the rule, and a statement of the use the answer is put to. Validation can therefore conclude a very different class of thing: that the data is stale, that an assumption was inherited rather than chosen, that the code does something the specification does not say, or that the model is fine and is being used to settle a question it was never built to settle.

Every difference between the two activities falls out of that one difference in what they are given, and no amount of skill on either side crosses it. A brilliant backtester with a hundred years of paired data still cannot establish whether the rule was written for a different portfolio. Paired data does not contain that fact. A specification is exactly where that fact lives, and a validator can find it in an afternoon by reading one.

Here is the household version, and it is exact rather than a loose analogy. Suppose a household keeps a rule for how much cash to carry to the vegetable market: last month's average spend plus a fifth. A backtest of that rule is one line in a notebook, kept for thirty days: what was carried, what was spent, and how often the cash ran short. The notebook will show that the cash ran short four times. The notebook will never show that the average it started from was measured in a month when a daughter was away at a hostel, or that a sister has started using the same figure to decide what to carry to the wholesale market where prices work completely differently. Neither of those two facts is in the notebook. Both sit in the rule and in the use, and somebody has to go and look at them.

What is a backtest actually doing, one day at a time?

Strip a backtest to its mechanism. The mechanism is small, and people assume it is large. On the morning of a trading day the model produces a figure: on this portfolio, over the next one day, a loss should not exceed this much except on rare occasions. The morning figure is written down before anything happens. At the close of the same day the actual profit or loss on that same portfolio is known. Two numbers, same day, one produced in advance and one produced afterwards. Compare them. If the realised outcomeWhat actually happened, as opposed to what was estimated in advance. fell outside what the morning figure said, that day is recorded as an exceptionAn observation where the realised outcome fell outside what the model said it would, such as a loss larger than the measure.. If it did not, the day is recorded and nothing more is said about it.

Repeat that for every day in a stated observation windowThe number of periods a backtest runs over, which fixes how many chances the model had to be wrong. and the result is a backtest. At the invented Vindhya Commercial Bank Limited the window is 250 observation days and the figure being tested is the bank's own one day value at risk measure at a 99 per cent level, run on its own historical simulation over the held for trading book. Every one of those choices, the 250 days, the 99 per cent and the one day holding period, is this invented bank's own decision and none of them is anybody's requirement. Changing any of them changes what a count of exceptions even means. Value at risk, how it is computed and how expected shortfall differs from it are covered with the market risk material. Both measures appear in this comparison only as the outputs being tested.

Two things follow immediately and both matter later. First, the test needs the pair. No morning figure, no test. No observed outcome, no test. Second, the test is blind to everything except the pair. The test does not know whether the portfolio changed shape halfway through the window, whether the historical window the model draws on contains a period like the one being lived through, or whether anybody has read the model documentation since it was written.

What did this bank's own backtest actually find?

At a 99 per cent level over 250 observation days, the bank expects about 2.5 exceptions across the window. About 2.5 is not a target and not a permitted number. The figure is simply what the bank's own choice of level implies over its own choice of window. Seven were observed. The seven are numbered X1 to X7. A backtest produces nothing else, so the whole record is seven rows.

All figures are Rs crore and all of them belong to the invented Vindhya Commercial Bank Limited. Now read the table three ways. A report that carries only the first reading has thrown away the other two, and the other two are where the finding lives.

The first reading is the count, and on the count seven against about 2.5 is 2.8 times the expectation. Put as a rate it is 2.8 per cent of days against the 1.0 per cent the level implies, and if the two figures look suspiciously alike that is because they are the same division wearing two labels: with 250 days and a 99 per cent level, the number of exceptions divided by 2.5 and the percentage of days both come out at the same figure. The match is arithmetic rather than a coincidence, and noticing it stops the two being reported as though they were separate evidence.

The count is also what moved the bank's own machinery. Its policy escalates a backtest to committee G7, the market risk committee, at five exceptions in 250 days, and forces a model review at seven. A count of seven passes five on the way, so seven crossed both. Both of those trigger levels are this invented bank's own policy choices and neither is anybody's standard.

Why is the clustering a bigger finding than the count?

Look at the dates rather than the total, and something changes shape. X1 sits alone in month 2. X4 sits alone in month 6. X7 sits alone in month 11. The other four do not sit alone at all: X2 and X3 fall on month 3 days 14 and 15, and X5 and X6 fall on month 9 days 2 and 3. Four of the seven arrived in two consecutive pairs.

Why does that matter more than the number seven? Because of what the model assumes about time. A one day measure of this kind treats each day as a fresh draw, independent of the day before it, in the way that a coin does not remember its last toss. If that assumption held, a rare event on Tuesday would say nothing about Wednesday, and rare events would scatter across the window rather than arrive in twos. The count says the model is set too low. The clusteringExceptions falling close together in time, which contradicts a model that assumes each period is independent of the last. says the model has the wrong shape, and wrong shape is the harder thing to fix.

The everyday version is a leaking roof. Four buckets filled in a year sounds like a maintenance number. Four buckets filled where two of them filled on consecutive days in one week reveals something the annual count cannot: whatever is going on has a duration, it persists, and the day after a bad day is not a fresh start. A rule built on the assumption that each day is unrelated to the last will keep underestimating exactly when accurate estimation matters most.

Is a limit breach the same thing as a backtesting exception?

A limit breach and a backtesting exception are not the same thing, and the confusion is worth stamping out before it starts. Both involve a number going over another number, and both happened in month 3 of this same year at this same bank. The two are different comparisons between different pairs of figures on different days.

A limit breach is one internal figure going past a ceiling somebody set for it. Limit L5 caps this bank's measured one day value at risk at Rs 18.0 crore. On month 3 days 21 and 22 the measurement came in at Rs 19.2 crore and Rs 18.6 crore, past the ceiling, and that is breach B5. Notice that the outside world plays no part in it. Two internal figures are being held against each other: what the model said that morning, and what committee G2 had decided the model was allowed to say.

An exception is what the world actually did going past what the model said it would. On month 3 days 14 and 15 the losses that arrived were Rs 24.6 crore and Rs 21.0 crore against measurements of Rs 15.2 crore and Rs 15.4 crore, and those two days are exceptions X2 and X3. Here it is the world doing the testing, and nobody's ceiling is involved. The two days month 3 lost most against its own measurement are not the two days limit L5 was crossed, and a pack that runs them together has manufactured an event that never occurred.

What does a count of exceptions throw away?

A count is a very cheap instrument and it treats every exception identically. One event, one tally mark, whether the realised loss cleared the measure by a whisker or by a mile. Put the sizes back in and the year reads differently.

Take each of the seven and subtract the measure written that morning from the loss that actually arrived. The excesses run Rs 4.6 crore, Rs 9.4 crore, Rs 5.6 crore, Rs 2.8 crore, Rs 12.6 crore, Rs 4.2 crore and Rs 2.8 crore. The excesses sum to Rs 42.0 crore across the seven and average Rs 6.0 crore each. The largest of them, on X5 in month 9, is four and a half times the smallest, on X4 and X7. Two exceptions counted as one tally mark each can differ by a factor of four and a half in what they actually cost, and the tally records neither.

Name the object here. Three different things in this bank wear the number 42. The Rs 42.0 crore just computed is the total excess of realised loss over measure across exceptions X1 to X7. The total excess is not the Rs 42.0 crore gross loss of operational incident I2, where a settlement instruction went out twice, and it is not the 42 findings raised by control testing in the same year. Three unrelated objects, one number, and a sentence that does not say which one it means has said nothing.

What do the same seven days say about the two measures?

The same seven rows test two different measures at once, and the two came out very differently. The bank runs a one day value at risk measure at 99 per cent, and it also computes an expected shortfall estimate. Expected shortfall answers a different question: not how often a bad day happens, but how bad the bad days are on average once they arrive. The estimate stood at Rs 21.9 crore.

Now take the seven realised losses. The seven sum to Rs 150.0 crore, an average of Rs 21.43 crore. Against an estimate of Rs 21.9 crore that is a miss of about Rs 0.5 crore, roughly 2 per cent. On the identical seven days, the count against the value at risk measure was out by a factor of 2.8. One measure was wrong by nearly three times and the other was right to within about 2 per cent, on the same portfolio, on the same days, in the same year. How either measure is built belongs with the market risk material; all that is reported here is how each of them did when the outcomes arrived.

Two things are worth taking away from moving that control. The first is that the expected 1.0 per cent sits between two attainable readings and is never reached: two exceptions is 0.8 per cent of days and three is 1.2 per cent, so the observed rate steps straight over the line it is being compared against. A count is a lumpy instrument being used to test a smooth claim. Nobody sensible judges a model on one year's count on its own. The second is that the bank's own trigger levels sit on top of that lumpy instrument, so the difference between an ordinary year and an escalated one can be two events in 250 days.

What does validation cover that a backtest never reaches?

Four things, and then a fifth that practitioners forget. Take them one at a time. Each one is a place a model can be wrong while its outputs look perfectly reasonable.

The data. Where does the input come from, is it current, and is it actually the population the model claims to describe? A model fed a feed that went stale eleven working days ago produces numbers on time, every day, in the right format. Operational incident I10 at this bank was exactly that: a collateral valuation feed sat stale for eleven working days and 340 loans were wrongly marked while every report still printed. No backtest catches a stale feed. The outputs and the outcomes can still line up while the inputs are wrong.

The assumptions. An assumption is a choice made because evidence did not settle the question. Every assumption therefore has an owner and a date. A validator asks which choices were made this way, who made them, on what evidence, and what the answer looks like if the choice moves. The last question, recomputing the answer across a range of an assumption, is the single most productive hour in most validations, and no outcome stream contains it.

The implementationThe code, spreadsheet or system that actually runs the rule, which can be wrong even where the rule is right.. The rule and the code that runs the rule are two objects, and only one of them is written in English. A specification can be immaculate and the spreadsheet that implements it can carry a range that stops one row short. Testing for that means rebuilding the computation independently from the same inputs and seeing whether the two answers agree. The rebuild is possible only where both the specification and the code are in front of the validator.

The limitations. Not a disclaimer, a set of instructions. A limitation written as a caveat says the model performs poorly in stressed markets, and tells a user nothing to do. A limitation written as an instruction says that in a month where a particular spread widens, this figure should be read alongside a stress result rather than on its own. The first is decoration and the second changes behaviour, and a backtest produces neither.

Can a model backtest perfectly and still be unfit?

Yes, and this is the fifth thing, the one that gets forgotten. FitnessWhether a model is suitable, which is never a property of the rule alone but of the rule together with the particular use it is put to. is not a property of a rule sitting on its own. Fitness is a property of a rule together with a use. Change the use and the fitness changes while nothing whatever changes about the model.

Picture a model that has backtested cleanly for three years on the book it was built for. Somebody then picks it up to price a new product. Ask what changed about the model, and the honest answer is nothing at all: same specification, same code, same data, same three years of clean results. Ask what changed about its fitness, and the answer is everything. The clean record was earned on a different question. A backtest can only ever establish how a model performed at the job it was doing, and it carries no information whatsoever about a job it has not done yet.

A validation report closes with a use restrictionA written statement naming the questions a model is cleared to settle, produced by validation and never by a backtest., a paragraph naming the questions this rule is cleared to settle and the ones it is not. A backtest closes with a count and a date. The two documents do not compete. The two answer different questions and a bank needs both, but only one of them is capable of stopping a model from wandering into a use nobody assessed.

What happens when a model cannot be backtested at all?

Here is the case that ends the argument, and it is not about the value at risk model. Model V1 at this invented bank is the behavioural deposit life model. Current and savings balances are repayable on demand, so contractually they could all leave tomorrow, and everybody knows they will not. Some rule has to say how long they actually stay. V1 is that rule, and it says an average life of 0.5 years on Rs 36,000 crore of current and savings balances.

An attempt to backtest it needs a stream of model outputs and a matching stream of realised outcomes lined up in time. The output is a life in years. The realised outcome is how long a given balance actually stayed, and that becomes known when the balance leaves. Balances that have not left yet produce no observation at all, not a small one. So across the same 250 observation days that gave the value at risk model 250 testable pairs, model V1 produced none. Not a weak test. No test.

Fractions repeat in this bank, so the objects in those two figures are worth naming. The minus Rs 840 crore economic value change runs against the Rs 990 crore cap on limit L8, giving a utilisation of 84.8 per cent. The 84.8 per cent is limit L8's utilisation and not the completeness of the model register, a different quantity that happens to sit at the identical fraction in the identical bank. Move the assumed deposit life to two years and the change becomes plus Rs 240 crore, or 24.2 per cent of the same Rs 990 crore cap. The 24.2 per cent is limit L8 again, and the share of the models actually running that have never been validated is also 24.2 per cent. Four numbers, two fractions, and a sentence that does not name its object conveys nothing. The register itself, the sweep behind it and what those completeness figures mean are covered where the model inventory is treated.

Which one runs every day, and which one runs on a cycle?

The last difference is a rhythm, and it explains why a bank can be quick about one kind of problem and very slow about the other.

The difference in rhythm is why this bank knew within a fortnight that its exception count had moved and still does not know, a year on, whether the assumption in model V1 was ever chosen deliberately. A continuous check surfaces a fast-moving problem fast. A slow problem produces no daily signal to notice, so a continuous check cannot surface one at all.

Who actually uses this distinction, and what do they do with it?

Four readers, four uses, and the difference between them is the fastest way to see why the distinction is worth holding.

The chief risk officer, Sunanda Ravikumar at this invented bank, uses it to ask a question that sounds rude and is not. Shown a pack in which one model has 250 tests behind it and a review under way, her useful question is not about that model at all. The question worth asking is which of the bank's models produce no test at all, and what those ones decide. The most exposed rule in an institution is rarely the one generating findings; it is the one generating nothing.

Somebody underwriting a line to this bank, reading it from the outside with no access to its papers, uses the distinction to hear what a disclosure did not say. A sentence promising that models are monitored every day is a sentence about backtesting: real, and narrow. The same sentence leaves entirely open whether anybody independent has ever read the assumptions sitting under the capital figure and the interest rate risk figure. Two claims, one of them made and one of them merely implied, and the reader has to keep them apart without help.

The person who built the model uses the restriction as cover. People new to the work find that use surprising. Outputs travel. Eighteen months after a model is delivered, a figure it produced turns up in somebody else's paper settling somebody else's question, and by then no one recalls what was said at the handover. A restriction agreed in advance and written down is the only thing in the room that still remembers.

None of this requires a balance sheet to bite. Suppose a household keeps a rule for how much of a salary can safely go to a monthly instalment. Backtesting it takes a notebook: in how many months this year did the instalment actually fall short? Worth doing, and quick. The notebook can never surface that the figure was worked out when two people were earning, or that a cousin has started quoting it for a loan running three times as long. Both facts live in the rule and in the use, and not one number in the notebook is wrong.

Where do the neighbouring subjects sit?