Shadow AI: Tools in Use That Nobody Approved

An unapproved use is a use case running with no entry in the register, whatever it is made of and whoever set it up. Concealment almost never explains one. An unapproved use is what happens when somebody solves a real problem faster than the approval route works, and a register's completeness on the day it was signed measures how much of it a firm already has.

The whole subject turns on one asymmetry. Buying something takes a payment, and a payment leaves a line in a ledger that somebody can search. Building something inside software the firm already had takes nothing at all, and leaves nothing at all. The difference between leaving a line and leaving nothing is why a search of one kind finds a particular kind of thing, why a firm that searches one way concludes its problem is small, and why the honest answer to how much of this a firm has is always a little larger than the first search suggests.

What counts as an unapproved use, and what does not?

Start outside a bank. A household keeps a list of its standing commitments so that anybody in the house can see what money goes out every month. Somebody sets up a small monthly payment for something useful, forgets to add it to the list, and the list is now wrong. Nothing was hidden. Nobody was deceived. The list is simply not a description of the household any more, and the moment somebody needs it to be true, it will not be.

An unapproved useA use case that is running with no entry in the register, whatever it is made of. is that, at a firm. The test is one question and it has nothing to do with how the use was built: is it running, and is there an entry for it in the register. A calculation somebody assembled inside a spreadsheet and a service the firm pays a supplier to run every month are equally unapproved if neither appears in the record. So is a step that has been running quietly for two years. So is one that three people know about and nobody wrote down.

Being just as clear about what does not count is worth the same care. Four things get called an unapproved use in a meeting, and not one of them is one. Something registered but badly described is a bad entry, not an unapproved use. Something built and never switched on is not in use. Something a person does by hand once a quarter with no software involved at all is not a use case in the register's sense. And something sitting outside the firm's own scoping test is a scoping question rather than a discovery. At Sumeru Bank Limited, an invented bank, that last distinction turned out to be the whole story.

How does something start running without anybody approving it?

Nobody sets out to run something unapproved. The sequence that produces one is so ordinary that it can be watched happening in any office in an afternoon, and every step in it is a step a reasonable person would take.

Somebody has a real problem: a queue that keeps growing, a set of letters that all have to be written by Thursday, a pile of documents that has to be sorted before anyone can start work on it. Something already within reach solves it, either a service costing a few thousand rupees a month or a calculation put together in a spreadsheet in an afternoon. The approval routeThe path by which a use case is meant to be permitted before it starts running. exists, but at Sumeru Bank Limited first-time documentation on one item ran at about three working days, and the person with the growing queue does not have three working days. And at the end of it there is no named place to tell, so the person mentions it to whoever seems closest and gets on with the week.

Every one of those four steps is defensible on its own, and the four of them together produce a use case nobody approved. The four steps together make this a design finding about the approval route rather than a conduct finding about people, and a firm that responds with a warning changes nothing at all.

Is this concealment, and what does treating it as concealment cost?

Most firms read the finding wrong at exactly this point, and the evidence that they are wrong is sitting inside the finding itself.

At Sumeru Bank Limited, not one of the five unregistered uses was concealed. Every one had been set up by somebody solving a real problem in front of them, and four of the five, being 80.0 per cent, had told somebody about it before it started running. The policy did not say who to tell, so telling somebody was not the same as telling anybody in particular, and four honest disclosures landed nowhere. The fifth was not hidden either. Nobody had ever thought to raise it.

Now watch what follows if the firm reads that as indiscipline. Three things happen, and all three make the next sweep worse. The four people who told somebody learn that telling somebody was the mistake. Quiet has just become the cheaper option, so the next five uses are set up more quietly. And the firm loses the only signal it had about where its own approval route is too slow to use. The slow route, not the count of five, was the actual finding.

What does route 1, the payments ledger, actually find?

In month 10, Ashok Pillai in technology risk ran a deliberate sweepA search for uses already running, done on purpose rather than waiting for one to surface on its own. at Sumeru Bank Limited. He did not wait for anything to surface. He picked four places where a running use might have left a mark and searched each of them in turn. Every route is blind in a way that is entirely predictable from what it searches, so the four are worth walking through slowly.

The first search routeOne method of finding a use that is already running. Each method finds a particular kind and misses the rest. is the payments ledger, read with one question in mind: is the bank paying anybody, every month, for software services. The ledger found 3 of the 5, being every bought serviceA component the firm pays somebody else to run, which therefore turns up in a payments record. among them and not one built one. Uses 10, 12 and 13 all sat in that ledger as small recurring charges. A calculation assembled in a spreadsheet raises no invoice, so uses 11 and 14 were invisible to the ledger and would have stayed invisible however carefully it was read.

The artefact itself is strikingly ordinary. A bought unapproved use does not announce itself in the ledger. The charge appears as a modest recurring line with a flat description, small enough to sit under the bank's own approval limit for a standing charge, sitting between the courier account and the stationery account. Nobody reading that ledger for any other purpose would stop at it.

What do the other three routes find?

Route 2 was a question, put in person to the head of every business area: what is running in this area that decides something or writes something. Route 2 found 3 of the 5, being uses 10, 11 and 14, and that set contains both of the ones built inside an existing toolA component assembled in software the firm already had, so it turns up in no payments record at all.. The person answering a question does not care whether a component raised an invoice, so a question reaches what no record holds. Asking takes somebody sitting with a dozen people and listening properly. Route 2 is therefore the slowest of the four to run and the one most likely to be skipped.

Route 3 was the network record of which outside services the bank's own systems had connected to. Route 3 found 3 of the 5, and they were exactly uses 10, 12 and 13: the same three the ledger found, arrived at by a completely different search. Both routes can only see something that was bought, so two routes that look convincingly different turn out to be structurally identical. A firm that runs those two and calls it a thorough search has done twice the work for none of the coverage.

Route 4 was the change record of systems the bank already had, read for changes nobody could explain. Route 4 found 1 of the 5, being use 14. Route 4 is the thinnest by count, and it was also the route that turned up the unrecorded change to the workflow router set out below.

Why does no single route find them all, and which two do?

Put the four results side by side and the shape of the answer is immediate. Route 1 finds 3, route 2 finds 3, route 3 finds 3 and route 4 finds 1. No single route found more than 3 of the 5, and the reason is not that any route was run badly: each one can only see the kind of thing it searches for. A ledger sees payments. A network record sees connections out. A change record sees changes to systems the firm already had. A question sees whatever the person answering it knows about. Only that fourth kind of search is not restricted by construction.

So the useful question is not how many routes to run but which two. The payments ledger comes first, being the cheapest and fastest route and the one that clears every bought use in an afternoon. The question comes second, as the only route that reaches what was built. Together the two cover uses 10, 11, 12, 13 and 14, all five of them. Adding routes 3 and 4 after those two finds nothing that is not already found. Running the pair the other way round gives the same answer. The question paired with either record of buying closes the same gap.

The wrong pair is instructive too. Routes 1 and 3 together still find 3. Both of them see only what was bought. Adding route 4 to those two reaches 4. Only when the question arrives fourth does the count reach 5, four searches spent to arrive where two would have arrived.

What did the bank find, and where were the finds concentrated?

The headline is a single line: the register held 9 entries and the sweep found 14 uses in fact running, so 5 were unregistered. All five had been running on the day the register was signed off in month 6. The gap is a shortfall against day one rather than growth since. Sumeru Bank Limited's register was 64.3 per cent complete on the day somebody signed it, and that is an ordinary finding rather than a scandal. Any firm that has never swept has a number like it and simply does not know what it is.

The more useful part of the finding is where the five sat. Across all fourteen uses, 4 were bought from a supplier rather than built inside the bank, being 28.6 per cent. Among the five unregistered ones, 3 of 5 were bought, being 60.0 per cent. Unapproved use was more than twice as concentrated in bought services as the overall mix would suggest, and the reason is the asymmetry set out at the start. Buying is easy, and it leaves a trace. Building inside a tool the firm already had is easier still, and leaves none. So a firm sees the bought ones first, fixes those, and has to go looking on purpose for the rest.

How much of the shortfall matters under the bank's own scoping test?

Two numbers are now available and they are both true. Confusing one for the other is the commonest way a completeness figure gets misread. Register completenessEntries held against uses found, measured on a stated day. measured against everything running is 9 of 14, being 64.3 per cent. But this bank had settled on a consequence testScoping by what an output does to somebody rather than by how the thing was built. for what its policy covers: an output reaching a customer or a reported figure without a person deciding. On that test, 11 of the 14 uses are in scope. The register held 9 of those 11, being 81.8 per cent.

Three of the fourteen fall outside the test, and the reasons are worth stating because none of them is a loophole. Use 10 produces a draft that a person signs every time. Use 13 produces something a person reads before deciding anything. Use 14 produces an output that never leaves the system it runs in. The two numbers answer different questions: 64.3 per cent measures the register against everything running, and 81.8 per cent measures it against everything the bank said it would govern. A firm that reports only the second is flattering itself, and a firm that reports only the first is measuring against a promise it never made. Report both, in that order, and say which test produced the second.

What else did the same sweep find, and why was it the more serious of the two?

Route 4, the change record, had found only one unregistered use and looked like the weakest of the four. Route 4 also found something that was not an unregistered use at all, and that find matters more than all five put together.

Component 9 of the intake chain is the workflow router: a set of rules somebody wrote that decides where every file goes. The router touched every one of the month's 8,600 decided files, more than any other component in the chain. In month 7 its waiting time before escalation was changed. There was no approval for that change and no record of it anywhere. And here is the part that matters most: the change broke no policy at all. The policy's scoping test at that time was written around what a thing is made of rather than what its output does, and a set of rules somebody wrote was not in scope.

Read that carefully. Nobody in the story was careless. The person who changed the waiting time was changing a written rule inside a system, exactly as they were entitled to do, in a component the policy had never claimed. The five unregistered uses sat outside the chain and were plainly things somebody should have listed. The router sat inside the chain, inside a registered entry, in the component with the widest reach in the whole system, and it was outside the policy because of a wording. The bank later settled on the consequence test set out above, under which that router is in scope. How a firm chooses between those wordings, and what each costs to run, is set out under the AI governance framework.

What follows a find, and in what order?

The instinct on finding an unapproved use is to switch it off. The instinct is wrong twice over: switching off is the wrong first step, and at this bank it would have been the wrong step entirely for three of the five.

Start with one question about the output, the same question the bank's own test asks: does what this thing produces reach a customer or a reported figure without a person deciding. At Sumeru Bank Limited, 2 of the 5 answered yes, being uses 11 and 12. Uses 11 and 12 needed an entry, a named person and a decision about whether they should continue at all. The other three answered no, and needed an entry and a named person just the same. A register that lists only the frightening things is not a register.

The two branches give the order, and the order is the opposite of the instinct. Register it, name somebody against it, apply the test, decide whether it should continue, and only then stop anything. Stopping comes last for a practical reason as well as a cultural one: something that has been running for months is load bearing, and switching it off before anybody has written down what it does creates a second problem on top of the first. Stopping comes last for a cultural reason too. A firm that switches off whatever it finds is a firm nobody tells about the next one.

How are the next five stopped from arriving?

The next five are not stopped, entirely. Sumeru Bank Limited observed about one new use case a month, and a firm that is solving problems will keep producing them. The number of them that arrive unregistered can be changed, and three things do that work while a warning does none of it.

The first is a named destination. Four of the five had told somebody and the telling landed nowhere, so name the person or the mailbox in the policy itself and make the act of telling take five minutes rather than three working days. The second is an approval route people can afford to use. If the route costs three working days of writing for a small use, people will keep going around it, and a lighter first pass for small things is not a weakening of the control but the change that makes it reachable. The third is a sweep on a stated interval. The arrival rate is not zero, and any register drifts from the day it is signed. The rate at which unregistered uses appear is set by the cost of doing it properly, so lowering that cost is the only intervention that touches the rate at all.

Who runs a sweep in practice, and what does it cost them?

Whether any of that ever happens is decided by what a sweep costs the person who has to run it. A sweep is not a project. At Sumeru Bank Limited it was one person in technology risk with access to the payments ledger and permission to ask questions, and the two routes that mattered are a ledger extract read in an afternoon and a dozen conversations of twenty minutes each.

The searching is the cheap half. The expensive half is what follows a find, and it is worth pricing before the sweep starts so that nobody is surprised into abandoning the exercise. At this bank first-time documentation ran at about three working days an item, and 2 of the 5 finds fell inside the scoping test, so bringing those two properly into scope cost about 6 working days of writing. Against that sits the chain the register was built around, at Rs 2,40,00,000/- to build once and Rs 65,00,000/- a year to run. The governance work that closes the gap is small money next to the thing it governs, and it is nearly always the half that gets cut.

Two other readers use the same output for different purposes. Somebody doing an independent review reads the sweep as evidence of whether the firm can state what it is running. The question is about the search rather than about the count. And an outside reader with a legitimate interest, a supervisor or an auditor, is generally not asking for the number to be perfect. The three things asked for are when it was last checked, by whom, and by which routes. A firm that can answer those three has a process, and a firm quoting a flawless count without them has a document.