How to Document Data Requirements for a Financial AI Use Case

Documenting data requirements means writing down, for every value a system reads, eight things: the name of the value; the system that supplies it; who is accountable for it there; the shape it may take; how often it moves; the behaviour the chain follows when the value does not arrive; what may be kept and for how long; and the ground for using it. The sixth is the one nobody writes and the one that breaks things.

A data requirement documentA record of what every value a system reads must be, where it comes from, and what to do when it does not arrive. looks like the dullest artefact anybody produces on a project of this kind, and it is the one that decides how the system behaves on the worst day of its life. The document is not an inventory of what is already held; it is a set of commitments about values that other people control. The distinction between an inventory and a commitment decides everything that follows. An inventory is finished when every row is listed. A set of commitments is finished only when somebody on the other side has agreed to each one. The rows where nobody agreed to anything are the rows that will decide something on their own later.

Start with a household. The shape is identical, and nobody has to learn anything new to see it. A house runs on a milk delivery that arrives at six every morning. Everyone knows what arrives, everyone knows roughly what it costs, and nobody has ever written down what happens on the morning it does not arrive. Nothing goes wrong for two years. Then one morning it does not arrive, and three people in that house each do something sensible and different: one buys a packet on the way to work, one skips the tea, and one waits until eight and is late. Nothing broke. Nobody was careless. There was simply no written behaviour for the absence, so three behaviours were improvised, and none of them was recorded anywhere either.

Sumeru Bank Limited, an invented mid-sized Indian bank, runs a retail personal loan intake chain that reads documents customers upload and pulls values out of them. The chain pulls 14 of those values from every file. Across a steady month of 8,600 files that is 120,400 values read. The number is large enough that the reading step is worth documenting properly, and the variety is small enough that documenting it properly is an afternoon of work rather than a project.

Why write any of this down, when the data already exists?

Because the data existing is not the same as the data arriving, and a system consumes arrivals rather than existences. A fieldOne named value the chain reads, such as a declared monthly income or an employer name. that exists in a source somewhere, is correct there, and is read by three other teams for three other purposes, can still turn up at the reading step in a shape nobody has seen before, or not turn up at all. The document is where the team records which of those possibilities has been thought about.

There is a second reason and it is the one that gets missed. The people who control a field almost never know it is being read. A source systemThe place a field comes from, usually run by somebody other than the team building the chain that reads it. is run by a team with its own plans, its own release calendar and its own idea of what that field is for. Nothing about a field being read downstream is visible to the team that produces it unless somebody wrote it down and told them. A document that lives only on the reading side is half a document. Two of the eight items are about people rather than about values for that reason.

Think about what happens without one. The chain runs, files are decided, and every unstated assumption behaves exactly as assumed for as long as reality cooperates. The document costs an afternoon and buys nothing at all in the months when nothing changes. An artefact that buys nothing in a quiet month is the artefact that gets deferred. The document pays out entirely on the day something changes, and on that day it pays out in files.

What are the eight items recorded for every field?

Eight, and they are always the same eight. Being always the same eight is what makes the document worth having. Written as a grid, the 14 fields are rows and the eight items are columns, so the whole thing is 14 times 8, being 112 cells. A cell nobody can fill is a behaviour the system will choose for itself the first time reality supplies something unexpected. The value of the grid sits in those cells rather than in the ones somebody could fill. A full cell is a fact somebody already knew. An empty cell is a decision that has been left to whichever line of code happens to run first, and nobody in the building will find out which line that was until a customer is affected by it.

The items come in order. Each one is a line written down, and each one produces something concrete that did not exist before it was written. An item that produces nothing anybody can point at is a label rather than a requirement.

1. The fieldThe name of the value, written once and used by everybody afterwards. Declared monthly income, not income, and not the four different names four teams use for it.
   Produces: one row, one name, and an end to the argument about whether two teams are discussing the same value.
2. Where it comes fromThe named system the value arrives from, and the point at which it enters the chain. Not the team, not the report, the system.
   Produces: an address for the change. When the value goes strange, the release calendar to look at is known.
3. Who is accountable for it thereA named person on the producing side, not a mailbox and not a department.
   Produces: somebody to ask, and somebody to notify. A system cannot give warning that it is about to change and a person can.
4. Its format and permitted valuesThe shape and the set: a whole rupee amount with no decimal part, or one of four named states, or a date in one written order.
   Produces: a check that can actually run on arrival. A value can be tested against a set and cannot be tested against an intention.
5. How often it changesEvery application, every month, almost never. The item is about the value moving, not about the format moving.
   Produces: the rhythm at which a held copy goes stale. The rhythm is the difference between reading a value once and reading it every time.
6. What happens when it is absentThe written behaviour for the case where the value does not arrive, arrives empty, or arrives in a shape the chain cannot read.
   Produces: a branch that exists on purpose. Without it the chain still branches, but it branches wherever the code happens to go.
7. What may be retained and for how longRecorded against the field, not against the system as a whole. Different values carry different answers.
   Produces: an instruction somebody can carry out, and a date on which a stored copy stops being somebody else's problem and starts being the reading team's.
8. The consent basis where one appliesThe ground on which the customer's value is being used, recorded beside the field it applies to.
   Produces: the ability to answer, field by field, why this value was used at all, without reconstructing the answer two years later.

Seven of those describe the field. Only one of them describes a behaviour of the reading chain. The asymmetry is not a curiosity. The asymmetry is the whole reason the sixth item behaves differently from the other seven, and it is worth holding on to.

Why is item 6 the one that goes missing?

Because the other seven can be answered by asking somebody, and the sixth cannot. Item 2 has an answer sitting in a system diagram. Item 4 has an answer sitting in a schema. Item 3 has an answer sitting in an organisation chart. Item 6 has no answer anywhere. Item 6 is not a fact about the field at all, but a decision about the reading chain that nobody has taken yet. Seven lookups and one decision, in a row of eight cells, and it is entirely predictable which one is still empty when the document is filed.

Watch what that produced at Sumeru. The document was written before go-live and it covered the eight items. Of the 14 fields, 3 carry all eight items, being 21.4 per cent; 5 carry every item other than item 6, being 35.7 per cent, and the 3 complete fields sit inside that 5; and 11 of the 14 have no item 6 at all, being 78.6 per cent. The three figures stand on the same 14 and reconcile in one line: 14 less 11 is 3, exactly the count of fields carrying a written absent behaviourWhat the chain does when an expected value does not arrive, which is item 6 of the eight.. Anything that says all eight items were documented for 5 fields is counting the other seven columns and calling it eight.

Now put the gap in proportion. The proportion is the surprising part. Eleven blank cells out of 112 is 9.8 per cent of the grid. The document was, on any reasonable reading, more than nine tenths complete, and the tenth that was missing was the only tenth that described what to do when something went wrong. A completeness percentage across a document like this one is close to meaningless, and a completeness percentage down the sixth column is the whole story.

What did the blank column cost? One episode is enough to fix the scale. An upstream income field changed format on one channel, and nothing in the chain had been told what to do with a value it could not read, so files were decided as though nothing had changed. Six weeks passed before monitoring flagged it. Six weeks is 30 working days, and at 430 files a working day that is 12,900 files decided on an assumption that had stopped being true.

The format change matters less as an episode than as a class of failure. A drafting component elsewhere in the same bank silently shortened an input rather than refusing it, and the shortening belongs to the same class. Both are the same failure as the blank sixth column: nobody had written down what to do when something expected did not arrive, so something plausible happened instead of something visible. A missing item 6 does not announce itself. The chain produces an answer, and an answer is the one thing nobody investigates.

How is the document produced, and in what order?

In seven steps, and the order matters because the last three cannot be done by one team alone. The first four can be finished in a room with the building team's own people. The fifth and the sixth need somebody on the other side of a boundary to agree to something, and that is why they slip. The seventh turns a document into a living record instead of a file with a date on it.

Step one is a list of fields and the point at which each one enters, and it is the only step where being exhaustive matters more than being right. The list follows the chain rather than the schema. Every value that is read, compared, scored or printed goes on it, including the ones arriving from inside the same institution. A boundary between two teams behaves exactly like a boundary between two companies once a release calendar is involved. At Sumeru the answer to step one was 14 fields, a list small enough that nobody could argue the exercise was too big to finish.

Step two is filling the eight items, and the discipline here is specificity. A cell reading numeric is not an answer. A cell reading a whole rupee amount with no decimal part, inside bounds the bank chose for itself, is an answer. A value can be tested against that cell the moment it arrives. Permitted valuesThe set a value is allowed to take, against which an arriving value can be checked automatically. are worth more than any other cell in the row for exactly that reason: they are the only cell that becomes a running check without anybody writing anything new. A declared value arriving as Rs 45,000/- can be tested against that cell on the spot, and the chain can then go and corroborate it against a statement showing Rs 38,000/-. A value arriving as the words not stated fails the same test. The test exists precisely to catch it.

Read the sixth line of that row again. The sixth line is doing something the other seven cannot. Refused and routed to a person on the same working day is not a description. The line is an instruction that changes the chain's behaviour, and somebody will handle the file it produces. The sixth item always makes the same trade: it buys visibility with work, and every other item on the row is free. An item that costs nothing gets written. An item that costs a routed file gets discussed, deferred and left blank.

What goes in a cell nobody in the building can answer?

Step three is marking those cells, and it is worth a separate pass rather than a glance in passing. The two kinds of empty cell look identical and behave completely differently. One kind is an unknown: somebody knows the answer and has not yet been asked. The name is taken, the question sent, the cell filled next week. The other kind is a blank: nobody anywhere knows the answer, and nobody anywhere has ever decided it. An unknown is a task and a blank is a decision, and treating a blank as a task is how it stays empty for eighteen months.

The two are told apart by one question, asked out loud in the room: who would be asked? If a name comes back, it is an unknown. If the room goes quiet, or three people answer differently, it is a blank, and chasing it produces nothing because there is nothing to chase. A blank has to be settled by whoever is accountable for the chain, in the room, that afternoon.

Which absent behaviour is chosen, and what does each one cost?

Step four is the decision itself, and there are only three plausible answers, which is what makes it a decision anybody can take in an afternoon. Treat the missing value as a zero. Substitute a standing default. Or refuse the file and route it to a person. Two of those three decide the file wrongly and silently. The third costs a routed file and buys the only thing worth having, somebody noticing on the same working day.

The first two feel harmless because they keep the chain moving, and keeping the chain moving is what everybody is being measured on. The measurement is exactly why the first two get chosen by default when nobody chooses. A zero for a missing declared income is a perfectly reasonable local decision if nobody has thought about what a zero income does to a scoring step. A standing default is even more reasonable, and somebody at least thought about it once. Neither produces a single line anywhere saying that something was missing.

A missing value cannot produce a right answer whatever is done, so the comparison is not between a right answer and a wrong one. The comparison is between a wrong answer nobody sees and an hour of work somebody does, and the work is cheaper at almost any volume worth naming. A refused file goes to the exception desk, where this bank measures handling in minutes and staffs accordingly, and that desk is covered on its own elsewhere. A silently wrong decision goes to a customer.

That leaves one question: what is the sixth item actually worth, across a whole document? The answer is a delay, and the delay is where the files pile up. A field that has a written absent behaviour is caught on the working day it stops arriving. The chain refuses, and somebody sees the refusal. A field with no written behaviour is caught only when monitoring notices the drift, and at this bank the drift took 30 working days to show.

Who is accountable for a field at its source, and how is that found out?

Step five is a name, and people resist it harder than any other cell in the document. The objections are always the same two. People move, so the name goes stale. And it feels like blame. Writing somebody down beside a value looks like preparing to point at them later. Both objections are real and neither survives the alternative.

A stale name is enormously more useful than no name. A stale name gives a team, a floor, a predecessor who can say who took it over, and a starting point that takes four minutes rather than four days. A system cannot give notice that it is about to change and a person can. Item 3 records a person rather than an application for that reason alone. As for blame, the name is not there to carry fault. The name is there because a change notification has to be addressed to somebody, and an unaddressed notification is not a notification at all.

Finding the name is the easy part once it is accepted as necessary. The value is followed backwards to the system that produces it, that team is asked who is accountable for the value rather than for the system, and the answer is written down along with the date it was given. If the answer is a mailbox, the question is asked again. If the answer is a department, the question is asked again. At Sumeru the answer for the income field was a person on the channel team, and the person who eventually wrote that name down was Ashok Pillai in technology risk, working from the other end during a later sweep.

How does a change at the source arrive before it reaches the files?

Step six is agreeing the change notificationThe agreement that whoever produces a value tells whoever reads it before the value changes., and it is the only step in the seven that another team has to sign up to. Step six is also what explains why the month 8 episode ran as long as it did. The upstream team that changed the income field format did nothing careless. The upstream team changed a value they were accountable for, in a system they ran, and they held no record that anything downstream was reading it.

The two documents on either side of that boundary each had a hole shaped exactly like the other document. The reading side knew where the value came from and had no name on the producing side. The producing side knew what the value was for and had no list of who consumed it. Neither team was missing information the other team was hiding; both were missing information nobody had ever been asked to write down.

The agreement at step six is small: a named person on the producing side tells a named person on the reading side before the value changes, and a change made without that notice is treated as an incident rather than as routine. The agreement goes in both documents. A commitment recorded on one side is a hope.

When does the document come back for another pass?

Step seven is the one that separates a document from a folder, and it is four triggers rather than a calendar. Re-run it when the chain starts consuming a new field. Re-run it when a source system changes, whether or not anybody believes the fields in question are affected. Re-run it when a component is refitted. A refit can change which values matter and how much. And re-run it after any incident that traced back to a field. An incident is proof that a row was wrong.

A yearly review is not a trigger, it is a diary entry, and it will always run at a moment when nothing has just changed. Triggers fire when the risk is real. At Sumeru the fourth trigger fired late and by accident: the format change was found by monitoring rather than by anybody rereading a row, and by then the files were already decided. Retention and consent move when a rule outside the bank moves, so they are the two items most likely to shift without anybody telling the build team.

What did the whole thing actually miss at one bank?

Sumeru did the work. The document existed before go-live, it covered all eight items, and 5 of the 14 fields were complete on the seven items that describe the value. On any dashboard it was a green row. Three of the 14 carried a written absent behaviour and eleven did not, and every expensive thing that happened afterwards came out of that one column.

How does somebody outside the build actually use this?

By reading one column and ignoring the rest. Anybody handed a data requirement document, in any role, can do something useful with it in about five minutes: count the rows, go to the sixth item, and count how many rows have a real behaviour written in that cell rather than a dash or a phrase like handled by the system. The ratio of real behaviours to rows is the only completeness figure on the document that predicts anything.

Sumeru Bank Limited and Ashok Pillai are invented.
Educational material. Not advice on any investment, tax, budget or market position.