Incident Management: Detect, Contain, Recover, Learn

Incident management is the standing process that runs on every disruption, large or small, in four stages. Detect: something is wrong and somebody records it with a severity. Contain: stop it getting bigger, a different thing from fixing it. Recover: get the service back to the people who use it. Learn: find out what the event shows and change something.

Three of those four stages have somebody waiting for them. The fourth has nobody. The asymmetry between the three stages somebody waits for and the one nobody waits for explains why an institution can run a process correctly thirteen times in a year and still be surprised by an event it had already been warned about. The learn stage is skipped not because anybody decides to skip it but because nothing goes wrong on the day it is skipped. Everything below is worked on Vindhya Commercial Bank Limited, an invented bank, and on the figures its own record carries.

Incident management demands no acquaintance with a data centre, a server room or any banking system. Every technical failure below is described the way a person outside experiences it: a screen that will not load, a counter that cannot pay out cash, a payment that leaves twice. Reading a failure from where the affected person stands is not a simplification of the method. Standing there is the method.

What is incident management, and what does it actually run on?

An incidentAny event that disrupts a service or could have, recorded and worked through in four stages. is any event that disrupts a service or could have. The definition is deliberately wide. The definition catches the four hour outage and it also catches the twenty minute wobble that three people noticed, the file that went out twice, the report that arrived with yesterday's numbers in it.

Incident management is not a plan for a bad day. Incident management is the ordinary, unglamorous machinery that runs on all of them. The test of whether an institution has an incident process is not whether it coped with its worst event; it is whether the same four stages ran on its smallest one. At the invented Vindhya Commercial Bank Limited the process ran on all thirteen operational loss events of the year, numbered I1 to I13, from the card fraud in month 1 to the complaints that were closed without a reply in month 12. Two of those thirteen, incidents I3 and I13, were also declared crises. Eleven were not, and the process still ran on them.

Think about a household for a moment. The shape is identical and the stakes are smaller. The tap in the kitchen starts leaking. Detect is noticing it and saying out loud how bad it is. Contain is putting a bucket under it and shutting the valve. The floor is saved and nothing is repaired. Recover is getting the water back on so that dinner can be cooked. Learn is the conversation nobody has, the one where somebody asks whether the other three taps in the house are the same age and were fitted by the same person on the same afternoon. The bucket gets used every time. The conversation almost never happens.

How does something become an incident in the first place?

An event becomes an incident when somebody writes it down. Writing it down sounds trivial, and it is the single most fragile joint in the whole process. Everything downstream, the containment decision, the recovery, the review, the annual count, the picture the board eventually sees, rests on one person deciding that what they are looking at is worth recording.

Notice that the record has to start before anybody knows what happened. Detection is the discovery that something is wrong, not the discovery of why, and waiting for the why before opening a record is how an hour of the response disappears before it begins. At the invented bank, incident I2 in month 2 is the plainest example: a settlement instruction was sent twice and Rs 42 crore left the bank twice. At the moment somebody saw the second debit, nobody knew whether it was a system fault, a keying error, a duplicate approval or something worse. None of that was needed to open the record and none of it was needed to start chasing the money. Rs 41.4 crore came back and Rs 0.6 crore did not.

Detection arrives by four ordinary routes. An institution that has only one of the four is blind in three directions. A monitor notices, meaning a machine watching a number and complaining when it moves. A person inside notices, meaning a clerk seeing a total that cannot be right. A customer complains, the slowest route of the four and the one that has already cost the institution something by the time it arrives. Or a third party reports it, as happened with incident I13: the forgery ran for fourteen months and surfaced only when a beneficiary bank made a claim. A loss found by an outsider was invisible to every layer the institution had built, so the route by which an event is detected is itself information.

How is severity graded, and what does the grade read off?

SeverityHow serious an incident is, graded from what the service impact is rather than from what failed. is the label attached at detection that decides how much of the institution wakes up. The grade decides who is told, how fast, whether anybody is called at home, and whether the event gets a committee's attention or a line in a monthly list.

There is exactly one rule and it is broken constantly. Severity is read off the service impact, meaning which of the named services somebody outside could not use and for how long, and never off the technology impact, meaning how large the broken system was. Vindhya Commercial Bank Limited has named seven important business services, numbered S1 to S7, each with the board's own impact tolerance, and the grade is read against those.

Graded the other way, the answer comes out exactly backwards. A large piece of equipment failing at two in the morning while every service it sits under keeps running on the others is a big technology event and a small incident. Now turn it round. A supplier's equipment fails on a Saturday and tens of thousands of payments do not go through. The institution neither owns nor built that equipment, and the event is a small technology event and a very large incident. Incident I9 at this bank is the second of those: a vendor-hosted payment gateway failed for 9 hours and 48,000 transactions failed. The bank's own machinery was untouched.

The two columns of that drawing are not a matter of taste. One of them is answering the question the institution exists to answer and the other one is answering a question about equipment. Note also the gap in the bank's own record: it locks a tolerance measurement for incident I3 and for no other event, so which of the seven services incident I9 touched, and by how much, is nowhere on the record. The measurement was simply not made, and saying so is more honest than producing a number that nobody at the bank ever computed.

What does containing an incident mean, and why is it not fixing it?

ContainmentStopping an incident getting bigger, which can be done before anybody knows the cause. is stopping the thing getting bigger. Stopping the growth is the entire definition, and everything useful about containment follows from one property: containment is available before the cause is known, and repair is not.

Consider the plainest possible case. A payment file has gone out with the wrong account details on it, and nobody yet has any idea why. What is available this minute? The same file can be stopped from going out again. Whatever sits downstream can be stopped from acting on the entries already sent. The process that produced the file can be frozen so that it does not produce a second one while the fault is being looked for. Not one of those three actions requires anybody to understand the fault. All three of them reduce how large this event finally becomes.

Repair needs a diagnosis, and a diagnosis cannot be had this minute. Somebody has to find the fault, change something, and satisfy themselves that the change is right. Finding the fault and proving the change is real work on a real clock, and the whole of it happens after the cause is understood. An institution that waits for the diagnosis before doing anything at all has decided, without ever saying so, to let the incident keep growing for the length of the investigation.

The street version, again, has the same shape. A food stall's gas cylinder starts leaking. Containment is turning the valve off and moving the flame away. Anyone can do that in four seconds without knowing whether the fault is the washer, the pipe or the regulator. Repair is the mechanic, and the mechanic is not here yet. The people who get hurt in that situation are almost never the ones who could not diagnose the leak. The people who get hurt are the ones who waited to find out what was wrong before turning anything off.

What does the recover stage do, and what does it hand off to?

Recovery is getting the service back to the people who use it. Note the wording. The whole boundary of the recover stage is inside it. The unit is the service, not the machinery, and the finish line is a customer being able to do the thing again rather than a piece of equipment showing a healthy light.

The recover stage of incident management does not itself perform the recovery; it decides which arrangement performs it and it holds the clock while that arrangement works. Two arrangements sit underneath it and they are different things, each covered on its own. Keeping the service running by another route while the usual one is broken is a business arrangement: the counter takes a manual voucher, the payment goes through a second channel, the branch does the thing by hand. Restoring the technology so the usual route works again is a technology arrangement. One keeps the promise while the fault stands; the other removes the fault.

Both of those have designed target times at this invented bank, service by service, and both were tested in month 12. The month 12 test ran on a planned date, with the recovery team on standby, and the failure chosen in advance, so it tested the arrangement and not the surprise. Reading the result without that sentence beside it makes the arrangement look better than the record supports. Incident management adds the sequencing on top: which route is chosen, when, by whom, and at what point the institution says the service is back and starts the clock on everything that follows.

How to build an Incident Management Playbook

A playbookThe written procedure that says who does what at each stage, with the names filled in. is the written procedure that says who does what at each stage. Most institutions have a document with that title. Far fewer have one that would survive contact with a Saturday morning, and the difference between the two is not length or polish. The separation is that every step of a real playbook ends in a name or a number, and every step of a bad one ends in a principle.

"Incidents will be escalated promptly to the appropriate authority" is a sentence that has passed a hundred committee reviews and it tells nobody at three in the morning what to do. "Purnima Ganeshan, head of operational risk, or the duty officer named on the current roster, is reached on the number in section four and may declare" is a sentence that works. Build the second kind, in eight steps.

What does a playbook usually leave out, and where does it get tested first?

The missing step is almost always PB4. Plans are generous with structure and stingy with authority. Plans name the team, the meeting and the reporting line, and then leave the one sentence that starts everything unwritten. Writing it means choosing a person, and choosing a person is uncomfortable.

Vindhya Commercial Bank Limited had exactly this hole. Its plan named a crisis team of 7 roles and did not name who could convene it. Nobody noticed. In office hours the question never arises: a senior person is present, somebody takes charge, the machinery starts. A missing declaration rule is invisible for as long as somebody senior happens to be in the building.

Then incident I9 arrived on a Saturday. A vendor-hosted payment gateway failed, 48,000 transactions failed with it, and the disruption ran 9 hours. No crisis was declared, and the reason was not capability, not equipment, not budget and not skill. The failure began on a Saturday and no single named person held the authority to declare a crisis outside working hours. The plan named a crisis team of 7 roles and did not name who could convene it. One unwritten sentence left the longest disruption of the year running without the escalation the bank had already designed for it.

Declaring a crisis, running one, and deciding under pressure with incomplete information are covered separately. The joint between the two is narrower: incident management runs on every one of the thirteen events, and PB4 is the step where it says, in advance and in writing, at what point one of those thirteen stops being handled by the standing process alone. An institution can hold a complete crisis capability and still never use it, if nothing in the standing process is allowed to switch it on.

What has to be in the incident record while the clock is still running?

Step PB7 asks for a record kept during the event rather than after it, and the reason is not administrative tidiness. Four of the five things that record has to carry cannot be recovered afterwards at all, and the fifth would have survived anyway.

Here is what is in it. What was seen, and when. What was believed at each decision point. What was done, and by whom. Which services were affected, and for how long. And what was told to whom. The five fields deserve a second reading with one question in mind: which of them could honestly be written down a week later?

The second one is the killer. A week after an incident everybody knows the cause, and a record written then describes every decision as though the cause was known at the time. The distortion is enough on its own to destroy the learn stage before it starts. A decision that looks obviously wrong in hindsight was often entirely reasonable given what the person actually had in front of them, and a decision that looks fine may have been luck. Neither judgement can be made without knowing what was believed at the moment, and nobody can remember what they believed once they know what was true.

What happens in the learn stage, and why is it the one that gets skipped?

The learn stage is a meeting, and the meeting has a name: the post incident reviewThe meeting after the recovery where the learn stage actually happens, with an owner and a date attached to what it decides.. The review happens after the service is back, and nobody is waiting for it. Nobody waiting is the whole problem, and it is worth sitting with rather than rushing past.

Every other stage has a person on the other side of it. Detect has whoever is affected. Contain has the growing loss. Recover has the customer, the desk, the counter, the phone line. Skip any of the three and the consequence lands the same day, loudly, on somebody who will say so. Skip the learn stage and absolutely nothing happens. No part of a process can hold a more dangerous property than that one. The cost arrives months later, attached to an event that looks entirely unrelated, and by then nobody connects the two.

So what does the review actually do, beyond producing a document? Two things. The review decides what changes, with a name and a date against each change. Step PB8 is written the way it is for exactly that reason. And it searches. The search is the part almost nobody does, and it is the part that carries all the value. The search asks a question that has nothing to do with the incident in front of it: has this same failure appeared before, somewhere in the record, in a form that did not cost anything?

What is a near miss worth, and what makes it worth anything?

A near missAn event that would have been an incident if one more thing had not held. is an event that would have been an incident if one more thing had not held. Something went wrong, something else caught it, and no loss was booked. Most institutions record them, and the register is usually short and tidy.

Here is the uncomfortable claim. Recording a near miss is worth nothing at all on its own; the entire value sits in the linkingSearching the record for the same failure appearing before, which is the only thing that makes a near miss worth recording., meaning the search for the same failure appearing before or elsewhere. A register nobody searches is a filing cabinet. A near miss carries almost the same information as a loss and costs nothing to collect, so a register searched at every post incident review is the cheapest source of information an institution has.

Vindhya Commercial Bank Limited recorded five near misses in the year, numbered N1 to N5. In month 2, near miss N1: a second duplicate settlement instruction, this one for Rs 68 crore, was stopped by the four eyes check before release. In month 4, near miss N2: a payment file of 1,240 salary credits was queued against the wrong account and caught at reconciliation before value date. In month 6, near miss N3: the collateral valuation feed was stale for 2 working days and a data quality check caught it. In month 7, near miss N4: a trade finance document set carrying a forgery pattern was refused by a checker. In month 10, near miss N5: a privileged access account belonging to a leaver stayed live for 46 days and was found by the quarterly access review before it was used.

Read that list once more and notice that in every single case something worked. Something working is what a near miss is. And then notice the two that matter: two of the five, near misses N3 and N4, turned out to be the same failure as an incident that followed, and in both cases the bank had the warning in writing before the loss arrived.

Why does a near miss carry the same information as a loss?

There is a model that makes this obvious, and it is worth having because it changes what a near miss looks like. James Reason, in Human Error, 1990, described the protection an institution builds as a stack of layers, each one imperfect and each one with holes in it. A design rule, an automated check, a reconciliation, a supervisor's review, a person who refuses to sign. On any given day the holes sit in different places, and an event reaches the outside world only when the holes in every layer happen to line up.

Layered defencesThe idea that protection is a stack of imperfect layers and an event reaching the outside only when the gaps line up. reframe the near miss completely. A near miss is not a smaller event than a loss; it is the same event, and it found a hole in every layer except one. The near miss has already mapped the holes. Finding out where the gaps are is the expensive part of the work, and the near miss has done it without anybody paying for the finding out. The only thing it did not do was cost money, and money is the one part of a loss event that teaches nothing about the failure.

Which is why the closure of near miss N4 is such a costly piece of tidiness. A document set carrying a forgery pattern reached a checker's desk, having already passed everything in front of the checker. The checker was the last layer, and the last layer held. Every layer before it had a hole in exactly the right place. A refusal form is designed to record a refusal, so the form recorded none of that.

How far back should the learn stage look?

If the search is the valuable part, then the obvious next question is how far back it goes, and this is where the argument stops being a matter of principle and becomes a matter of arithmetic. A review that searches nothing finds nothing, and that is what this bank did. A review that searches a month finds one of this year's two links. A review that searches four months finds both.

Two numbers frame it. Incident I13, at a net loss of Rs 15.4 crore, is 35.2 per cent of the year's Rs 43.8 crore on its own. Incident I10 adds Rs 1.4 crore, taking the pair to Rs 16.8 crore and 38.4 per cent. A one month lookback reaches 35.2 of those 38.4 points, so the short search is not a proportionally small answer; it is nearly the whole of it. The control below sets how far back the learn stage searches.

Read the shape for a moment rather than the numbers. There are two risers and they are nothing like the same size. The first sits between a lookback of nothing at all and a lookback of one month, and it carries 35.2 percentage points of the year with it. The near miss N4 and incident I13 link is only one month wide, and the net loss behind incident I13 is Rs 15.4 crore. The second riser waits until four months and then adds 3.2 points. The near miss N3 and incident I10 link is four months wide, and the net loss behind incident I10 is Rs 1.4 crore. Almost the whole of what there was to reach in this record is reachable by a search that goes back a single month. Most people guess the opposite when they are asked.

The shape matters because the objection to a linking rule is always cost. Searching a register is somebody's afternoon, and the longer the window the more afternoons it takes, so a rule that demanded five years of searching would lose the argument in the room and deserve to. Here the shape says something else. The cheapest version of the rule, one month, already reaches most of what was there. The limit of that is plain: this is one invented bank's twelve months and not a general law about how far apart a warning and a loss sit. The method travels rather than the answer: measure the gaps in an institution's own record before arguing about the size of the window.

A step chart is persuasive in a way that a paragraph is not, so the caveat has to be repeated beside it. Every rupee on that chart is loss the learn stage would have been looking at, and none of it is loss that would automatically have been avoided. A review in month 8 that had pulled up the month 7 refusal would have had a question in front of it, not a conclusion. The review might have found the fourteen month pattern quickly, or it might have looked at one refused document set, agreed that the checker had done the right thing, and moved on. Linking changes who gets asked, and how early. Linking does not change what they then decide, and letting the two blur sells a comfort rather than a method.

There is also a practical reason the search fails even when somebody actually does it, and near miss N4 shows it exactly. The near miss was filed as a routine refusal. Incident I13 was booked as letters of credit issued against forged shipping documents. The two records describe the same failure and share almost no words, so a search run on the words would have returned nothing and the reviewer would have concluded, correctly by their own method, that there was no prior warning. A register becomes searchable only when the failure is described rather than the outcome. The refusal form has to have somewhere to say what was wrong with the documents and not only that they were refused. Adding that field is a design decision about a form, taken in a quiet month, and it separates a register that can be linked from a register that merely exists.

How do incident, crisis, continuity and recovery fit together?

Four words get used as though they were interchangeable, and they are not. The four words answer four different questions, they run on different numbers of events, and different people are waiting for each of them. Setting them side by side one final time is the fastest way to place any real event, including one already under way.

Incident management is the standing process and it runs on everything. Its question is: something is wrong, what is done about it, in four stages. At this invented bank it ran on all thirteen recorded events, incident I1 through to incident I13. Crisis management asks a narrower and harder question: the decisions have outrun the facts, so who decides, on what, and with what authority. Crisis management ran twice in the year, on incident I3 and on incident I13, and it sits on top of the standing process rather than replacing it. Business continuity asks how the customer keeps getting served while the usual route is broken: a business arrangement about counters, channels and manual routes. Disaster recovery asks how the usual route itself comes back: a technology arrangement about systems. Each of the last three is covered separately. The joint is this: the recover stage of incident management chooses between the last two and holds the clock while they work, and step PB4 is where the standing process is allowed to switch crisis management on.

Now put one morning through the grid, month 9. Incident I9: a payment gateway run by an outside supplier failed for 9 hours and 48,000 transactions failed with it. Incident management runs on every disruption, so it was running from the first minute. Continuity had a live question: how did somebody who wanted to pay for something still pay for it while the gateway was down? Recovery had a live question: how did the gateway itself come back, and by when? And crisis management had a live question too: who was going to hold the decisions that could not wait? Three of the four questions had somebody attached to them and the fourth did not. No crisis was declared, and no sentence in the plan said who could declare one on a Saturday. One morning, four questions, and the one that went unanswered was the one that only needed a line of writing.