Case 058Model risk and validationWarm up
A bank has 40 models: 8 complex capital and pricing models, 20 moderate scoring and reporting models and 12 simple tools. Tier them, set validation cycles and estimate the validation team's annual workload.
1The situation
Candrel Bank's new head of model risk inherits an inventory of 40 models and a small validation team. Eight are complex models used for regulatory capital and for pricing derivatives. Twenty are moderate impact models: application scorecards, provisioning inputs and management reporting models. Twelve are simple spreadsheet tools with limited use.
A full validation takes about 30 validator days for a complex model, 12 for a moderate one and 4 for a simple tool. Nobody has set review cycles yet, and the team has been validating whatever the business asks for first.
2Your task
Assign a tier and a validation cycle of one, two or three years to each group, estimate the annual workload, and say what the tiering protects against.
Quick check
Roughly how many validator days a year does a one, two and three year cycle need?
Worked solution
Try it on paper, then open one step at a time.
30-second answerThe answer to give first
Put the 8 capital and pricing models in tier 1 on a yearly cycle, the 20 moderate models in tier 2 every two years, and the 12 tools in tier 3 every three years: about 376 validator days a year. That is 240 plus 120 plus 16. At about 200 working days a validator, it needs roughly two people for periodic reviews alone, before new models and changes.
Step 1What decides a model's tier?
A hospital does not give every patient the same scan every week; it triages by how serious the condition is and how fast it could turn. A model's tier comes from what a mistake in it would cost and how likely a mistake is: its materiality, its complexity and how it is used. Capital and pricing models are all three: they drive large numbers, involve heavy maths and feed regulatory filings, so an error is both likely and expensive. A spreadsheet that formats a monthly report has none of those. Write the criteria down, so a model's tier is a rule anyone can apply, not a negotiation with its owner.
Step 2How does the workload add up?
Annualise each tier by dividing its effort by its cycle. Tier 1 costs 8 x 30 = 240 days every year, tier 2 costs 20 x 12 / 2 = 120, and tier 3 costs 12 x 4 / 3 = 16, about 376 days a year. Two thirds of the effort goes on one fifth of the models, which is the point. Stagger the tier 2 and tier 3 reviews across the cycle; if all twenty tier 2 models fell due in the same year, that year would need 528 days rather than 376.
| Tier | Models | Days per review | Cycle, years | Days a year |
|---|---|---|---|---|
| Tier 1 | 8 | 30 | 1 | 240 |
| Tier 2 | 20 | 12 | 2 | 120 |
| Tier 3 | 12 | 4 | 3 | 16 |
| Total | 40 | 376 |
Step 3What does the tiering not cover, and what does the team really need?
The 376 days cover periodic reviews only. New models, material changes to existing ones and annual performance monitoring for tier 1 models all come on top, and in many banks they take as much time again. So a team of two covers the cycle but not the bank; three or four validators is a more honest estimate. Tiering is also not fixed: a tier 3 tool that starts feeding a capital number moves to tier 1 the day it does, which is why the inventory needs an owner who reviews tiers at least yearly.
Say what the tiering protects against. Without it, a team validates whatever the business asks for first, and the busiest model owners get the most attention while quiet high-impact models go years without review. With it, the bank can show a supervisor that effort follows risk. Check the current supervisory guidance on model risk management for any minimum review frequencies before fixing the cycles.
Where candidates lose it
Candidates often give every model the same annual review because it sounds safe. It needs 528 days at each tier's effort, or 1,200 if every review is done at tier 1 depth, and the team does neither well.
The other miss is forgetting that periodic review is only part of the load. New models and changes often double it, so a workload estimate that stops at the cycle understates the team needed.
What the interviewer asks next
- A tier 2 scorecard is now used to set loan pricing. What happens to its tier and the workload?
- How would you decide the review depth for a vendor model you cannot see inside?
- What would you monitor between full validations for a tier 1 model?
Company names and figures are illustrative.
