Case 095Model risk and validationCore
An expected credit loss model links default rates to GDP growth. A year of GDP falling 7% and a year of 9% growth send its predictions to 9% and 1%, while actual defaults were 4% and 5%. Diagnose the failure and design a management overlay with a governance trail.
1The situation
Samvika Finance, a non-bank lender with an Rs 8,000 crore loan book, uses a one-factor model for its expected credit loss provisions: the one-year default rate is 5.5% less 0.5 times GDP growth, fitted on years when growth ran between about 5% and 8%. Loss given default is 40%.
In two quiet years the model worked: it predicted 2.25% and 2.5% against actuals of 2.4% and 2.6%. Then GDP fell 7% and the model predicted a 9.0% default rate against an actual 4.0%, during a period of loan repayment relief. The next year GDP grew 9% and it predicted 1.0% against an actual 5.0%. It is now year 4 and the forecast is again 9% growth.
2Your task
Why did the model fail in both directions, what overlay would you put on this year's provision, and what governance must go with it?
Quick check
What is the most likely main reason the model failed in years 3 and 4?
Worked solution
Try it on paper, then open one step at a time.
30-second answerThe answer to give first
The model failed because GDP left the range it was fitted on and because defaults lagged the shock; an overlay using two-year average growth adds about Rs 128 crore to this year's provision. A linear rule turned minus 7% into 9% defaults and a 9% bounce into 1%, while repayment relief pushed defaults into the rebound year. Averaging two years of growth gives 5.0% against the model's 1.0%. The overlay needs a written rationale, independent review, committee approval and a release trigger.
Step 1Why did the model break in both directions?
A rule learned from ordinary days can say absurd things about extraordinary ones. A driving app that learned traffic from weekdays will badly misjudge a festival day. The model was fitted on growth of about 5% to 8% and then asked about minus 7% and plus 9%, so a straight line fitted inside that range was stretched far outside it. Three things compounded: defaults lag a downturn by a year or more, repayment relief in the crash year delayed defaults further, and the 9% bounce was a rebound from a low base, not a return to health.
Step 2What overlay would you put on this year's provision?
Keep the model's structure but feed it an input closer to what borrowers are living through. Using the average of the last two years' growth, minus 7% and plus 9%, gives 1.0%, and the model then predicts 5.0%, close to the 5.0% that followed; the same fix gives 5.75% for the crash year against an actual 4.0%. On an Rs 8,000 crore book at 40% loss given default, the model's 1.0% implies Rs 32 crore of provision and the overlay's 5.0% implies Rs 160 crore, a management overlayAn adjustment to a model output, made by management with documented reasons, for a risk the model cannot capture. of about Rs 128 crore.
| Year | GDP growth | Model default rate | Two-year average input | Actual |
|---|---|---|---|---|
| 1 | +6.5% | 2.25% | n/a | 2.4% |
| 2 | +6.0% | 2.50% | 2.38% | 2.6% |
| 3 | -7.0% | 9.00% | 5.75% | 4.0% |
| 4 | +9.0% | 1.00% | 5.00% | 5.0% |
| 5 | +6.8% | 2.10% | 1.55% | 3.1% |
Step 3What governance must go with the overlay?
An overlay is a judgement that moves profit, so it needs the same discipline as the model it corrects. Write down why the model fails, how the overlay was sized and what else was considered; have model validation review it independently; get it approved by the risk committee; and set a trigger for release, such as two quarters of actuals within half a point of the model, or a rebuilt model. Track the overlay against actuals each quarter and disclose it to the auditors. An overlay without a release trigger tends to become a permanent, unexplained cushion.
Then fix the model itself. Rebuild it with lagged and cumulative GDP, an unemployment or household income measure, and a flag for relief periods, and test it on the shock years it failed. Until the rebuild passes validation, the overlay stands.
Where candidates lose it
The frequent miss is proposing to refit the same line including the two shock years, which bends the coefficient without fixing the lag or the out-of-range problem. Name why it failed before proposing how to fix it.
The second is an overlay with no governance: a number chosen in a meeting, with no rationale, reviewer or release trigger. Interviewers ask about the governance trail because that is where overlays go wrong in practice.
What the interviewer asks next
- The overlay would have reduced the crash-year provision. Would you have allowed a negative overlay then?
- How would you validate a rebuilt model when only two shock years exist?
- Which disclosures would an auditor expect about the overlay?
Company names and figures are illustrative.
