Disaster Recovery: Restoring Technology After Failure

Disaster recovery is the job of restoring the technology underneath a service once that technology has stopped working. Two figures, both chosen by the business, define what a good recovery looks like. One fixes how soon the technology must be usable once more. The other fixes the largest quantity of already recorded work the institution will accept losing. The second figure is a volume of data and never a stretch of downtime.

One separation carries this whole guide, and almost every reader arrives having missed it. Time lost asks how long the service was down. Data lost asks how far back the institution has to go to find a usable copy of the record, and therefore how much of what people already did has disappeared. A service can be back in fifteen minutes having lost four hours of records, and a service can take a full day to come back having lost nothing at all. Every worked figure below belongs to Vindhya Commercial Bank Limited, an invented bank.

These decisions need no knowledge of technology. The hard part is not any product, supplier, design or technique, but a business question about the outside world.

What is disaster recovery actually restoring?

Start with a shop. The shape is identical there, and nobody has to know anything to see it. A cloth shop on a busy street sells on credit, and every sale goes into a ruled notebook: name, date, amount. Two things have to work for that shop to trade. The shutter has to open, and the notebook has to be there.

Now break each one. If the shutter jams, the shop is shut. Customers stand outside, nothing is sold, and the moment somebody prises the shutter open the shop is exactly as it was. If instead the shutter opens fine but the notebook was left out in the rain overnight, the shop is open and trading and something else is gone: three weeks of who owes what. Nobody is standing outside. The loss is invisible from the street and it is far worse.

Disaster recoveryPutting technology back into working order after it has broken, so the service riding on it can be delivered once more. is the work of putting both of those back after a failure, and the reason it needs two separate objectives is that they are two separate losses. Restoring the machinery decides how long nobody could use the service, and restoring the record decides how much of what people already did still exists. An institution that plans only for the first has planned for the shutter and not for the notebook.

Somewhere behind every service there is a place the technology can be run from when the usual place is unavailable. A standby siteA second place the technology can be run from, which the people outside never see or hear about while it is working. is that arrangement. The only thing worth holding on to is that the outside world is never supposed to know it exists. None of the decisions here are made at such a site, so how one is built is a separate subject.

What does the recovery time objective say, and why is it not enough on its own?

The recovery time objectiveHow quickly the technology has to be usable again after a failure, set as a design target inside the promise the board made. is the shorter half of the story and it is settled separately. The objective is the figure the recovery arrangement is built to deliver: how quickly the technology has to be usable again. At Vindhya Commercial Bank Limited it runs from 1 hour on S1 payments and remittances and S7 treasury settlement, through 2 hours on S3 internet and mobile banking and 3 hours on S2 branch counter service and cash withdrawal, out to 12 hours on S5 deposit account opening. Every one of those figures was set by the bank's own board.

The recovery time objective cannot say the thing the shopkeeper cared about. A recovery time objective is entirely silent on how much of what people had already done comes back with the service. Bringing the shutter up in one hour meets a one hour objective, whether the notebook is intact, three weeks short, or gone. So the same failure gets a second number, and the second number is the recovery point objective.

What does a recovery point objective measure, and why is it not a time?

The recovery point objectiveA ceiling on vanished work, set as the volume of record lying between the last usable copy and the moment everything stopped. answers a different question: when the record is put back, how far back does it go? Somewhere there is a copy of the record that is usable, and it was made at some moment. Everything that happened between that moment and the failure is not in the copy. The recovery point objective is the institution stating, in advance, the largest amount of that lost work it is willing to accept.

Here is the part that trips people, and it is worth sitting with. The objective is written down as a duration, five minutes or one hour or four hours, and that duration is not a period of downtime. The duration measures the record. Five minutes means the record may be missing up to five minutes worth of what people did. The objective says nothing whatever about how long the service was unavailable. The outage might have been ten seconds or ten hours.

To restoreTo put a copy of the record back into use so the service can run on it again, at the state that copy was in when it was made. a record is therefore to accept a specific, known, quantified hole in it. A known hole is not a defect in the method. The hole is the method. The only institution with no hole is one that arranged, in advance and at real cost, for the record never to exist in a single place, and that arrangement gets its own block further down.

Why are data lost and time lost two different measurements?

Take one service and hold it still. S3 internet and mobile banking at Vindhya Commercial Bank Limited carries a recovery time objective of 2 hours and a recovery point objective of 5 minutes, both of them set by the bank's own board. Both figures are written in minutes. Neither is on the same scale as the other, and the reason is simple: the two are not counting the same thing.

The 2 hours counts unavailability. The unavailability is the stretch during which a customer opens the app, gets nothing, and tries again later. The 5 minutes counts record. The 5 minutes is the largest gap the bank is willing to have between the last usable copy and the failure. At worst a customer who moved money four minutes before everything stopped may find, when the service comes back, that the transfer is not there. One number is about waiting and the other is about work that has vanished, and no amount of hurrying fixes the second.

Read the two figures against each other and the shape of the service appears. 120 minutes divided by 5 minutes is 24, so S3 may be unusable for twenty-four times as long as it may be short of records. The ratio of twenty-four is not a rule, and it is not the same for any other service here. The ratio is a statement about internet and mobile banking specifically: people will wait a couple of hours, and they will not accept that a payment they watched succeed has quietly stopped existing.

What do the seven services at this bank actually carry?

Vindhya Commercial Bank Limited has named seven important business services, numbered S1 to S7, and set a recovery point objective on each one. A guess ventured on one familiar service before the table is read is what makes the table stick.

Here are all seven. Every entry below was decided by the bank's own board.

Read down the two number columns and notice that they do not move together. S3 has the tightest time objective after S1 and S7, and the third tightest data objective behind the same two. S5 has the loosest of both, but not in the same proportion. Its time objective is 12 hours against S3 at 2, a ratio of six. Its data objective is 4 hours against S3 at 5 minutes, a ratio of forty-eight. The two columns are set by two different conversations and a reader who assumes one predicts the other will get every service wrong.

What does a recovery point objective of zero actually ask for?

Two of the seven carry a zero: S1 payments and remittances, and S7 treasury settlement. A reader running down the column reads 4 hours, 1 hour, 15 minutes, 5 minutes, zero, and takes zero as the last rung of a ladder. Zero is not a rung. Zero is not the small end of this scale, it is a different arrangement standing on the other side of it, and the price behaves accordingly.

Go back to the shop. Suppose the shopkeeper wants to lose less when the notebook is ruined. She can copy the day's credit entries into a second notebook every evening, and at worst she loses one day. She can copy them at lunch as well, and lose half a day. She can copy after every tenth customer, and lose nine sales. Every one of those is the same arrangement, done more often, and each step costs her more of her own time.

Now suppose she is asked to lose nothing. There is always a gap between one copy and the next, and the fire may land inside it. Copying more often does not get there, no matter how often. The only way to lose nothing is to stop copying and start duplicating: carbon paper under every leaf of the notebook, and the entry exists twice at the instant the pen moves. Carbon paper is a different way of running the shop. She buys different stationery, writes differently, and the cost arrives once and stays.

Duplication is the whole content of a zero. A zero asks for the record to exist in more than one place at the moment it is created, rather than for the record to be copied faster. How an institution arranges that is a separate subject, but the shape of the decision is exactly the shopkeeper's, and so is the shape of the bill.

Why those two services and not the others? Because of what the lost work is. An instruction to move money that the bank accepted, acknowledged and then lost cannot be reconstructed by anybody, at any price. The customer does not have it. The counterparty does not have it. Nobody in the branch remembers it. The problem is not that losing it would be expensive. There is no procedure at all for getting it back. Zero is what an institution writes when the honest answer to where else does this exist is nowhere.

How does the interval between copies set the worst case?

An objective is what the business asked for. Whether it is actually met is decided by something duller and more physical: how much time passes between one usable copy of the record and the next. Call that the copy intervalThe gap between one usable copy of the record and the next, which sets the largest amount of work a failure can take away.. The copy interval is the only quantity here that an institution controls directly.

Now think about when the failure lands. If copies are taken every fifteen minutes and the failure lands one minute after a copy, one minute of work is gone. If it lands fourteen minutes after a copy, fourteen minutes are gone. Planning is done on the worst case. The worst case is the failure landing immediately before the next copy, so the work lost is taken as the whole interval.

Why the worst case and not the average? Because the objective is a promise about what the institution will accept, and a promise tested against an average is not a promise. The shopkeeper who copies her notebook every evening does not tell a customer she loses half a day on average. She knows that a fire at nine at night takes the whole day, and she plans for the nine o'clock fire.

Put the seven objectives in a row against a single copy interval and something ugly appears. The thresholds are 0 for S1, 0 for S7, 5 minutes for S3, 15 minutes for S2, 1 hour for S4, 1 hour for S6 and 4 hours for S5. A single interval is either above or below each of those, so the count of services that would lose more than the board said they could does not slide upward as the interval grows. The count jumps, and the first jump is in a place nobody looks.

Who sets a recovery point objective, and what are they answering?

The figure has to come from the business, and the reason is not a governance nicety. The question behind the figure can only be answered by somebody who knows what the work is. The single question is whether the work that disappeared exists anywhere else, and can therefore be put back.

Everybody has done this arithmetic without naming it. Consider a household for a moment. An electricity bill receipt goes missing. Annoying, and entirely recoverable. The electricity office has its own record and a phone call fixes it. Cash handed to a cousin last Diwali, with nothing written down anywhere by anybody, is a different case: if both sides forget the amount, it is gone, and no amount of searching produces it. Same household, same week, two completely different exposures, and the difference is not about how valuable the item was. The difference is about where else the item exists.

Run that question across the seven services and the objectives fall out of it. S5 deposit account opening carries four hours because the application forms are physically sitting on the counter, so four hours of lost entries costs some re-keying by the branch staff and nothing else. S3 internet and mobile banking carries five minutes because a customer who moved money and watched it confirm cannot sensibly be asked to remember what they did and do it again. S1 payments and remittances and S7 treasury settlement carry zero because an instruction the bank accepted and then lost exists nowhere at all.

Why is the technology being back not the same as the service being back?

Here is the stage that goes missing from almost every plan, and it is the stage that decides what the customer experiences. The record has been put back. The record is complete up to the moment of the last usable copy and contains nothing after that. The service is technically available. And every bit of work that happened inside the gap is still gone.

Getting that work back is reconciliationWorking out, after a restore, what is missing from the record and having it re-entered or re-checked by people., and it is done by people, not by technology. Somebody has to establish what fell into the gap. Establishing it usually means comparing whatever partial evidence exists: paper at the counter, a queue of instructions that never completed, a customer on the phone insisting a transfer was made. Then somebody has to key it back in, and somebody else has to check it. Reconciliation is the stage nobody is assigned by default, and it starts after the technology report says the job is finished.

Notice how the shopkeeper's evening goes after the notebook is ruined. Getting the shutter open again takes an hour. Walking the street asking eleven customers what they bought last week takes three days, and two of them remember it differently. The first job is the restore; the second job is the reconciliation; and only the second one decides whether the shop actually knows what it is owed.

There is a second event on the other side of all this, and it deserves a name because plans forget it. FailbackReturning from the standby arrangement to the normal one once the failure is fixed, which is its own event rather than the absence of one. is the move back from the standby arrangement to the normal one. Failback is a change like any other change, with its own chance of going wrong. A plan that treats the return journey as automatic has planned half a round trip.

What does a restore test on a planned date actually establish?

In month 12 Vindhya Commercial Bank Limited ran a continuity test across all seven important business services. The test produced seven results. S1 came back in 52 minutes against a 60 minute objective and met it. S7 came back in 48 minutes against 60 and met it. S4 came back in 6 hours against 8 and met it, S5 in 9 hours against 12 and met it. Three missed: S2 in 3 hours 40 minutes against 3 hours, S3 in 3 hours 10 minutes against 2 hours, and S6 in 14 hours against 8 hours. Four met and three missed.

Now look at what that list is made of. Every one of the seven results is a time. The month 12 test measured the recovery time objective on all seven services and measured nothing whatever about data, so the recovery point objectives at this bank have never been tested at all. No plausible figure fills that gap. The absence is the finding: half the design was examined, the report was headed with the result of a continuity test, and nobody reading it would know which half.

There are two more things the test could not establish, and they matter as much. No work was actually lost, so none had to be put back, and the test could not show how long the reconciliation would have taken. And it could not show what happens on a day nobody chose. The month 12 test ran on a planned date, with the recovery team on standby, and the failure chosen in advance, so it tested the arrangement and not the surprise. That limitation belongs beside the result every time the result is quoted, or the result quietly claims more than it earned.

What changes when somebody else runs the technology?

Plenty of what a bank runs on is run by somebody else, and the moment that is true a reader tends to assume the objectives move across with it. They do not. Handing the technology to a third party relocates the arrangement. The promise stays put on the bank's own side of the table.

Incident I9 at Vindhya Commercial Bank Limited is the clean instance. A payment gateway run by an outside supplier failed for 9 hours and 48,000 transactions failed with it. Three questions have three answers, and only one of them changes. Who does the restoring? The supplier. Who is measured against the recovery point objective? The bank. Who does the customer deal with, before, during and after? The bank, throughout. The customer never had any relationship with the supplier and in most cases did not know one existed.

Money did move, and it is worth being precise about what that means. Incident I9 booked a gross loss of Rs 4.4 crore and a recovery of Rs 1.2 crore came from the supplier, leaving Rs 3.2 crore net. So the contract did some work. No contract takes the promise off the bank. Whether the supplier can actually deliver what the bank told its customers is a question about the contract and the monitoring behind it, and that whole subject is covered separately.