AI Governance and the AI Policy: Approval and Accountability

AI governance decides three things about a deployed component: who approved it, who is accountable for it while it runs, and what happens when it has to be switched off. Governance is a thin layer sitting on top of the risk work a firm already does, not a replacement for it. At Sumeru Bank Limited, invented, that layer was ten written sections, one register and one named person against each use case.

Governance of this kind is not a body of knowledge. Governance of this kind is a set of answers to questions somebody will ask after something goes wrong, written down before it does. Every section of a real policy exists because one of those questions would otherwise have nowhere to go, and a section answering a question nobody would ever ask is furniture. So the test of a policy is not its length but whether the answers it held were true on the day it was signed. Sumeru Bank Limited is measured against that test across one deployed chain, and the bank comes out looking neither heroic nor negligent. The bank comes out looking like a bank doing this for the first time.

What does AI governance actually decide?

Start somewhere ordinary. A household has one car and three people who can drive it. No manual stops the arguments in that household. Three answers do: who said the keys could be lent out tonight, whose problem it is if the car comes back with a scraped wing, and what everybody does on the morning it will not start. Nobody in that household calls this governance. Three answers of that kind are exactly what governance is.

AI governanceThe arrangement that records who approved a deployed component, who is accountable for it, and what happens when it stops. answers the same three questions about something a firm has put into live use. Who approved it, who is accountable for it while it runs, and what happens when it has to be switched off: that is the whole of it, and every section of a real policy attaches to one of those three. At Sumeru Bank Limited the thing being governed is the retail personal loan intake and decision chain. The chain turns an application started on a handset into a decision, and in one steady month it took 10,000 applications, carried 8,600 of them to a decision and sent 3,010 of those to a person. The intake chain is one entry in the bank's register. Nine components sit inside it.

What does this layer leave to work that already exists?

This is the block that keeps a policy from swelling into something nobody reads. AI governance does not decide whether a fitted component is any good, and a policy that tries to decide it has quietly annexed a job that already has people, tests and a record of its own. Whether the scoring model inside the intake chain holds up on a fresh window of applications is answered by the risk function at Sumeru Bank Limited, by Neelima Rao, who built no part of the chain, and it is answered by model validation rather than by governance. Governance asks a narrower thing: is there a record of who approved that component, is there a name against it, and is there a way to stop it.

Three more things sit outside the layer and are worth naming plainly. Governance does not decide credit policy: who the bank lends to is a commercial judgement. Governance does not decide what the bank sells. And governance does not decide how a component is built. Building one is an engineering question with engineering answers. A good policy is short precisely because governance sits above all three and touches none of them. At Sumeru Bank Limited the whole thing is ten numbered sections, and it can be read in the time it takes to drink a coffee.

What is an AI Policy, and what has to be inside one?

An AI policyThe written document holding the answers to those three questions, section by section, before anything is deployed. is the written document that holds those answers section by section. Sumeru Bank Limited's runs to ten numbered sections. The numbering matters: every later record at that bank cites a section by its number rather than by a quotation. Section 1 says what the policy covers. Section 2 says who approves a use case before it is built. Section 3 describes the register and what an entry carries. Section 4 names the accountable person and what that accountability consists of. Section 5 says what must be documented before deployment. Section 6 says what must be monitored after deployment and who sees it. Section 7 sets the route back to a manual process and how often it is rehearsed. Section 8 says what may be bought rather than built and how a supplier is assessed. Section 9 covers access to the data a use case consumes and the least it may collect. Section 10 covers review, re-approval and retirement.

Look at how little of that list is about the technology. Not one of the ten sections says anything about how a component should be built, what it should be fitted on or how accurate it should be. People, dates and records are the parts that do not change when the technique does, so a policy that reads as a list of them rather than as a list of techniques will still be usable in three years. Writing it that way is not a stylistic preference. A section naming a technique dates the moment the technique moves, and then the whole document has to be reopened for a reason that has nothing to do with governance.

Why does the wording of section 1 decide everything after it?

Section 1 is the scoping testThe wording that decides which things a policy applies to. Changing it changes how many things must be approved, registered and monitored.: the wording that decides what the policy covers. Section 1 is the only section whose wording changes what every other section applies to, and it can be rewritten without touching a word of any other section. Sumeru Bank Limited tried three wordings on the same nine components and on the wider bank, and the counts came out very differently.

Every one of those figures is the invented bank's own: 3 working days of first-time documentation to bring one item into scope, so 5 items is 15 days, 19 items is 57, and 47 items is 141, which at 20 working days a month is about seven months of one person doing nothing else. Scoping by how a thing was built leaves component 9, a written rule that decided where every one of the month's 8,600 files went, outside the policy entirely, so the cheapest wording is the worst one. The cost is not hypothetical. Under the earlier wording, that component's waiting time before escalation was changed in month 7 with no approval and no record, and nobody found it until a sweep in month 10. The bank chose the third wording, the one asking about consequence rather than construction, and 19 items came into scope.

Who approves a use case, and what does approving it commit them to?

Section 2 answers the first of the three questions, and it answers it with a name and a date rather than with a committee. At Sumeru Bank Limited the intake chain was defined and approved at month 0, before the pilot, and the approval carried Revathi Balan's name as head of retail credit. Approving a use case is not approving a technique; it is accepting that a stated thing will run, that its outputs will do a stated thing to stated people, and that the approver will be the person asked about it. Revathi Balan built no part of the scoring model and could not have. She approved that it would exist, what it would decide, and who it would decide about.

Approving a consequence is not approving a technique, and the distinction is the one most often collapsed. Think of the person who signs off a new gas connection for a wedding caterer. The official is not certifying the burner. The official is accepting that a burner will be lit in this yard, on this date, near these people, and that when somebody asks who allowed it, the answer has a name. Approval is a decision about consequence and about who answers. Approval is not a technical endorsement. Asking an approver to be a technical endorser is how firms end up with approvals granted by whoever understood the maths, and that is exactly the wrong person to hold accountable for consequences.

What does the register say about whether any of this is real?

Section 3 covers the use case registerThe list of the uses a firm has, as distinct from a list of the components inside them. One use case can hold many components., and the register is where a governance arrangement stops being a document and becomes checkable. A register is a list of the uses a firm has. A register is not a list of components, and confusing the two is the commonest mistake in the room. At Sumeru Bank Limited the whole intake chain is one entry in the register, and the nine components sit inside that single entry.

The register was signed off in month 6 holding 9 entries. In month 10, Ashok Pillai in technology risk ran a sweep and found 14 uses actually running in the bank. All five of the ones the sweep added were already in use on sign-off day, so the register was 64.3 per cent complete on the day it was signed rather than growing incomplete over four months. The distinction between a day one shortfall and four months of growth is the whole point of the number. Read as growth, it points the fix at discovery. Read correctly, it points the fix at how the register was assembled in the first place, and that is where the fix actually belonged.

A reader arriving cold will otherwise reach for the wrong reaction, so two things about this finding deserve to be said out loud. First, not one of the five was concealed. Every one was set up by somebody solving a real problem, and four of the five had told somebody about it; the policy simply did not say who to tell. Second, that test does not catch three of the fourteen, so measured against the test the bank had itself chosen the register was 9 of 11 and 81.8 per cent complete. Both readings are true and the honest position is to state them together. Sumeru Bank Limited is a bank building an inventory for the first time and getting most of it, not a bank hiding anything.

Who is accountable once it is running, and accountable for what exactly?

Section 4 answers the second question with a name. A named accountable personOne person, named in the record, who answers for a use case being what the record says it is. is one person, in the record, who answers for a use case being what the record says it is. At Sumeru Bank Limited, 6 of the 14 uses carried such a name, being 42.9 per cent, and those 6 were all among the 9 registered entries, so two thirds of the register had a name and none of the five unregistered uses did. Revathi Balan is the named person for the scoring model.

Accountability is not authorship, and every one of the six duties attached to the name is something a person can check without being able to build the thing. The six are: that the use case is registered and its entry is true; that somebody can state in plain words what a component does and what it may not be used for; that what it consumes is documented and permitted; that its behaviour is monitored and that they see the monitoring; that a route back exists and has been rehearsed; and that it is re-approved or retired on a stated date. Read them again and notice that none of them requires knowing how a model is fitted. The six duties require reading a record, asking a question and receiving a report.

At the month 12 validation Revathi Balan could evidence 4 of the 6, being 66.7 per cent. Duty 4 she could not. The monitoring existed, but its output went to the team that had built the chain rather than to her. Duty 6 she could not. No re-approval date had been set when the component was approved at month 0. Neither of those is a personal failure. Both are design failures, and that is what the validation recorded. There is also a span problem worth naming. One person was named on three of the 6 uses carrying a name, and 3 plus 1 plus 1 plus 1 is 6, so those 6 uses carried only four names. At six duties each, that person carried 18 duties against another's 6.

What is a Manual Fallback, and what is it not?

A manual fallbackA written and rehearsed route back to doing the work without the component. A route that has never been run is a document. is the route back to doing the work without the component. Section 7 of Sumeru Bank Limited's policy required one for every use case, and required that it be rehearsed. Here is the ordinary version. The card machine at a shop goes down and the shopkeeper has a carbon pad in the drawer. Whether that pad is a control depends entirely on whether anybody has ever filled one in. If nobody has, the shop discovers at the counter, with a queue, that nobody remembers what goes in which box.

A route back that has been written down and never run is a document, and calling it a control is the single most common self-deception in this whole arrangement. The word that matters in section 7 is rehearsedActually run at least once while nothing was wrong, as distinct from written down and approved.: actually run at least once while nothing was wrong. Across the nine components of the intake chain, a written route back existed for 3 of them, being components 4, 6 and 9, and had been rehearsed for exactly 1, being component 6, and only because manual underwriting existed before the chain did. The other 6 had no written route at all.

What did exercising a route back actually cost, in people?

The chain fell back once. In month 9 week 3 an upstream income field had changed format on one channel and monitoring had finally flagged it. The channel fell back to manual decisioning for 4 working days, being 1,720 files at 430 files a working day. A route back is where the whole subject stops being a document and becomes a rota. At the pre-chain hands-on time of 11 minutes a file, 430 files is 4,730 minutes of desk time a day, and 4,730 over the assumed 420 minutes a person is 11.26 posts. The automation had removed exactly 11.26 posts. The two numbers are identical, and the identity is not a coincidence. A route back to manual is a promise to reinstate, for as long as the fallback lasts, precisely the headcount the deployment took out.

Now find the crossing, because it is the number a firm can actually plan against. The desk's own 2,940 minutes covers 2,940 over 11, being 267.3 files, or 62.2 per cent of the daily 430. A fallback covering more than 62.2 per cent of the book needs people the bank no longer has, and there is no arrangement of the remaining seven that changes it. Below that share the desk absorbs the work and merely falls behind on its own queue. Above it, somebody has to be borrowed, trained on a process they have not run and supervised, all on the day the thing is already broken.

What is checked after go-live, by whom, and how often?

Sections 6 and 10 answer this, and they answer it as a loop rather than as a line. Section 6 says what is monitored after deployment and who sees it. Section 10 sets re-approvalA stated date on which the decision to run something is taken again rather than assumed to still hold. and retirement: the date on which the decision to run the thing is taken again rather than assumed. Nothing ever forces anybody to look at whether a use case should still exist, so a use case with no stated re-approval date never leaves the middle of that loop. At Sumeru Bank Limited the scoring model was approved at month 0 with no re-approval date set, and the month 12 validation was the first occasion on which anybody asked the question again. The month 12 validation set a date.

The register itself needs the same treatment, and this is where the bank's own arithmetic is unusually useful. Its observed rate is about one new use a month. At a sweep interval of m months the register carries on average m over 2 unregistered entries, so it stands at 14 over 14 plus m over 2. Sweep every month and it reads 96.6 per cent. Every three months, 90.3. Every six, 82.4. Every twelve, 70.0. Every twenty four, 53.8. Completeness is not a property a register has; it is a property of how often somebody goes and looks. Two more uses had begun in the two months since the sweep, so at the month 12 validation the register stood at 14 of 16, being 87.5 per cent.

Why does a policy written before any use case exists govern nothing?

Sumeru Bank Limited's ten sections were written in month 0, before a single component existed. Writing the policy first is the normal order and not the wrong order; nothing can be approved under a policy nobody has written. But it has a consequence that is rarely warned about. Three of the ten sections described a system nobody had built, so three of the ten could not be complied with as written. Section 5 required a fitting population documented for every component, and four of the nine turned out to be rule sets nobody had fitted anything to. Section 7 required a rehearsed route back for a chain that had no manual route on six of its nine components. Section 9 required a data inventory that was not produced until month 11.

None of those three is a drafting failure in the ordinary sense. Each is what happens when a document describes a system nobody has built yet. The only language available at the time is the language of an imagined one. The month 12 rewrite changed 4 of the 10 sections: 1, 5, 7 and 9. Section 1 changed because the scoping wording had left the highest-volume component outside; the other three changed because reality had arrived. So the answer is not to delay the policy until the system exists. The answer is to write a rewrite date into it on the day it is signed, exactly as section 10 does for a use case.

How does a board member, a lender or an auditor actually read any of this?

Here is what people actually do with a governance arrangement, as distinct from what a policy says they do. A board member with twenty minutes does not read the policy. A board member opens the register and asks three questions of it: how many entries, how many carry a name, and when was the last sweep. At Sumeru Bank Limited in month 10 those answers were 14, six of them with a name, and the sweep was happening as they asked. Three answers are enough to know whether an arrangement is alive, and getting them takes less time than reading one section.

An internal auditor does something narrower and harder. An auditor picks one entry and tries to trace it end to end: the approval and its date, what the entry claims the thing does, what the monitoring says it has been doing, and the route back. At the month 12 validation Neelima Rao did exactly that across the nine registered entries and found that all twelve fields were complete for 2 of them, being 22.2 per cent, and that field 11, the route back and the date it was last rehearsed, was absent for 7 of the 9. The field that goes missing first is always the one that costs money to fill.

And somebody has to price it. Five posts at Sumeru Bank Limited's own assumed fully loaded Rs 9,00,000/- a year is Rs 45,00,000/- of annual cost sitting behind a control the policy had costed at nothing. The bank did not have to hold those five people permanently, and that is not the point. The point is that section 7 committed the bank to producing them inside a morning, and nobody had ever said out loud where they would come from. A control with no line item is a control somebody will have to improvise, and improvisation on the day something breaks is the most expensive kind there is.