Intelligent Document Processing: Classification and Extraction

Intelligent document processing means deciding what a document is, pulling named values out of it, and passing both to something that acts on them. The processing is a chain of separate steps rather than one capability, and its reliability is set mostly by how structured the documents are rather than by anything about the components. A fixed template read reliably and a free-form letter read unreliably are the same chain on different inputs.

Everything awkward about this subject comes out of that last claim. Every reading below moves when the documents change and barely moves when the components change. The supply of documents decides whether a document chain works, and not the processing. The supply feels like somebody else's problem and the processing feels like the work, so the effort lands in the wrong place: teams spend on the part they can see themselves building and leave untouched the part that actually sets the answer.

Everything below runs on one invented case. Sumeru Bank Limited is a mid-sized Indian bank, invented, and so is every number attached to it. The bank built one thing, an intake chain for retail personal loans running from an application on a handset through to a decision. The reading step sits in the middle of that chain, and every figure below comes from one steady month of it.

What does a document processing chain actually do, step by step?

Five steps, in a fixed order, and each one hands something narrower to the next. First an image arrives, usually a photograph taken on a handset rather than a scan. Second, the characters are recovered from that image, so the picture becomes text with positions attached. Third, the whole document is classified: this is a pay slip, that is a bank statement. Fourth, the fields belonging to that kind of document are pulled out of it. Fifth, the values are passed onward to something that acts on them. At Sumeru that something is a written rule comparing declared income against a statement.

Nothing in that list is one capability, and no single component does more than one of the five. The split matters immediately. A supplier or a team can be excellent at step two and ordinary at step three, and the file experiences only the weakest of the five. Intelligent document processingDeciding what a document is, pulling values out of it, and passing both onward to something that acts on them. is the name for the assembled chain, not for any one part of it.

What are the document processing tasks, and what does each one assert?

A chain like this can be asked to perform four separate tasks, and they are worth separating because each one makes a different kind of statement about the document. Document classificationDeciding which kind of document a stretch of text or an image is. asserts which kind a document is. Extraction asserts that a particular value sits in it. Matching asserts that two records describe the same thing, such as a name on a pay slip and a name on an application form. Validation asserts that a value is of the permitted shape, such as a date that exists or an amount that is not negative.

The four are not degrees of the same skill, so a chain that is reliable at one of these tasks is not thereby reliable at another. Sumeru measured the difference directly. Its classifier reads 96.2 per cent of documents correctly. The figure states something about task one and nothing else. Its reading step accepts 95.0 per cent of the month's fields above the bank's own chosen bar, a second figure about task two on a different unit. Neither number says anything at all about the matching step, and nobody at the bank had ever measured that one separately.

The units in the third column are the quiet source of most confusion in this subject. One number is per document, one is per field, one is per pair of records and one is per value. A month at Sumeru holds 34,400 documents, 120,400 fields and 8,600 files, so the same underlying event can be reported as a small percentage or a large one depending purely on which of those is the divisor. Nobody is playing a trick. The spread between a small percentage and a large one is a consequence of four tasks sitting in one chain.

What does entity extraction assert, and why does it carry the risk?

Entity extractionPulling a named value, such as an employer or an amount, out of a document. is the second task above, looked at closely. Extraction takes a document and returns named values: the employer is this, the net pay is that, the account number is this other thing. The word entity here just means a named thing asked for by name, and the asking matters as much as the finding. A chain does not discover that a document contains an employer. Somebody wrote down that a pay slip carries an employer field, and the chain then goes and gets it.

At Sumeru the extraction step pulls 14 fields from each file. Across 8,600 files that is 120,400 fields in a month. Every later figure about the month sits on that arithmetic, so it is worth doing slowly. Four documents arrive per file, so 34,400 documents. Fourteen fields come off each file, so 120,400 fields. Of those, 114,380 cleared the bank's own chosen acceptance bar, being 95.0 per cent, and 6,020 did not, being 5.0 per cent.

Extraction carries more risk than the other three tasks because its output is the only one that travels onward as a number somebody then acts on. The fields that come back look wrong for the kind of document the chain thought it had, so a misclassification is usually caught. A wrong extracted value looks exactly like a right one. The wrong value is a plausible employer name or a plausible salary figure, and it flows into a written income rule that has no way of knowing where it came from.

Why does the classification decide what every later step can even look for?

Because the field list is attached to the kind of document, not to the sheet it is printed on. Classify a document as a pay slip and the chain goes looking for employer, gross pay, net pay and period. Classify the same document as a bank statement and it goes looking for account number, opening balance and credits. The second search runs perfectly well. The chain simply hunts for things that were never on the document.

Sumeru measured what that costs. Its classifier reads 96.2 per cent of documents correctly, so in a month of 34,400 documents, 1,307 are filed as the wrong kind. Of those, 1,224 come back with fields that are empty or absurd, the chain notices, and a later check catches them at no real cost. The other 83 reach a person. The 83 sit inside the 1,264 files that stopped for an unreadable field, so they are not an extra category of trouble but a slice of an existing one.

The everyday version is a wedding invitation card sorted into the utility bills tray at home. Nothing about the sorting was violent; the card is intact, the handwriting is fine. But whoever works the bills tray next month is looking for an amount and a due date, and will find neither, and the first assumption will be that the document was faulty rather than that the tray was wrong.

What is a document template, and what does having one buy?

A document templateA fixed layout in which the same value always appears in the same place. is a fixed layout in which the same value always sits in the same place. A pay slip issued by a payroll system has one. The employer name is in the box at the top left, the net pay is in the box at the bottom right, and that is true of the first pay slip and the ten-thousandth. A letter written by a manager confirming somebody's employment has no template at all. The employer name is in there somewhere, in a sentence, in a position that depends on how the person writing it chose to begin.

A template turns finding a value into looking in a known place, rather than working out what a document means. Looking and working out are not two difficulties of the same kind. One is a question about coordinates. The other is a question about language, and a component answering it has to hold far more, get far more right, and can go wrong in far more ways. A structured documentOne whose layout is fixed enough that a value's position is predictable. converts the hard problem into the easy one before any component is chosen at all.

Why does structure set reliability more than anything about the components?

Here is the claim this whole subject rests on, stated as plainly as it can be. Take one chain, change nothing whatsoever about its components, and feed it documents that are progressively more structured. The readings move a long way. Take the same chain, hold the documents fixed, and change the components, and the readings move much less. Reliability tracks the documents.

Sumeru has four measured points, all its own, all invented, and all from one deployment. With every document arriving on a fixed template, 98.6 per cent of fields clear the acceptance bar and 380 files reach the exception desk in a month. At three quarters templated, the bank's actual mix by type, 95.0 per cent clear and 1,264 files reach the desk. At half, 91.0 per cent and 2,320 files. With nothing templated at all, 82.0 per cent and 4,510 files, more than half the month landing on somebody's desk.

Between the best and worst of those four points, the files needing a person move by a factor of nearly twelve, and not one component was touched. The factor of nearly twelve is the whole argument for spending on the supply of documents rather than on the processing of them. The supply belongs to somebody else, so it is an argument almost nobody makes in the room where the money is decided.

What happens to the same chain on structured and unstructured input?

Now the number that changes how a desk is sized. In the month, 6,020 fields were routed for a person to look at, being 5.0 per cent of all fields. If those 6,020 had been sprinkled evenly across the month, one to a file, they would have touched up to 6,020 separate files, being 70.0 per cent of everything. The 6,020 touched 1,264 files, being 14.7 per cent, at an average of 4.76 routed fields each.

Failures cluster in files rather than spreading across them, and that single fact is why an exception desk is far smaller than a field-level error rate suggests. The clustering is not luck. The cause is structure again: a file whose employer letter is free form tends to be a file where several fields came off that same letter, so one badly structured document produces four or five routed fields at once. The badly structured documents also tend to arrive on the files that were complicated for other reasons, and the pile is denser still.

How a document use case is judged worth doing

Three questions, and the one usually asked first turns out to be the least decisive of the three. Volume is asked first because it is the easiest to obtain: how many documents a month, how many minutes each. Structure is asked second or not at all, and it moves the answer more than volume does. Consequence is asked last, and it decides not whether to build but how much of the output a person has to see.

How to Evaluate an Intelligent Document Processing Use Case

Score the three in this order and write the score down before anybody demonstrates anything. A use case with high volume and low structure is the one that looks best in a proposal and disappoints most reliably in production.

What is drawn before building any of it

A map, and a document map is not the same shape as a general process map. A process map has steps in a row with times against them. The document type is the unit that decides everything downstream, so a document map has one row per document type, and each row carries five columns.

How to Map a Document-Processing Automation Workflow

Column three is the one that goes missing, so draw the rows first and fill them in this order. For every field the chain consumes, Sumeru was meant to document eight numbered items: the field, where it comes from, who is accountable for it there, its format and permitted values, how often it changes, what happens when it is absent, what may be retained and for how long, and the consent basis where one applies. Item six, what happens when the field is absent, was missing for 11 of the 14 fields, being 78.6 per cent. Every item other than item six was documented for 5 of the 14 fields, and all eight for only 3. Nobody had written down what the chain should do when something expected did not arrive.

The map has a second job once the chain is running. When a file stops, somebody has to pick it up, and what that person receives is a case with seven numbered fields: the file, the cause from the six the bank recognises, what the component read, what it produced, the decision or the missing item the person is being asked for, the age of the case, and who holds it. If the map cannot fill those seven for a stopped file, the desk will fill them by hand, and that is where the handling time goes.

How is this different from simply recovering the characters from an image?

OCR vs Intelligent Document Processing

Optical character recovery (OCR) is step two of the five. It turns marks on an image into text with positions attached, and that is the entire job. Intelligent document processing is the assembled chain. The chain also settles the kind of document, the list of named values to hunt for, and what happens next. Character recovery can be flawless on a document the chain then files as the wrong kind, and a correct classification cannot rescue characters that were never legible in the first place. Two separate problems, two separate failures, and neither one covers for the other.

There is a second difference that catches people out, and it is arithmetic rather than judgement. Sumeru measured its reading step as right on 99.2 per cent of individual characters, and 99.2 per cent sounds excellent. But a field is only right if every character in it is right. On that basis a six character field comes back right 95.3 per cent of the time, a twelve character field 90.8 per cent, and a twenty character field 85.2 per cent. Nothing has changed except how long the field is.

One caution before those numbers are carried anywhere. The 95.0 per cent in the month is the share of fields accepted above the bank's own chosen bar, and that is a different question from whether a field is right. Sumeru hand-checked 400 accepted fields and found 7 wrong, being 1.75 per cent, its own measurement on its own sample. Acceptance and accuracy answer different questions and should not be treated as one number.

Which arrangement does each part of the chain need?

One chain at Sumeru runs three different processing arrangements at once, and each part uses the one its consequence demands rather than the one that sounds most current. Getting this wrong is expensive in a quiet way: the spend goes on making something faster in a place where nothing downstream is waiting.

Real-Time: the reading step, because somebody is waiting

Real-timeHandling each item as it arrives, while somebody waits for the result. means handling each item as it arrives while a person is on the other end of it. The reading step has to run this way because an applicant is sitting with a handset waiting for an answer, and the straight-through path returns a decision in about four minutes. There is no batch to wait for. The customer's attention is the only thing being waited on.

A daily batch: the reconciliation of 430 items

Disbursals are reconciled against the core banking ledger once every working day: 430 items, of which 421 match automatically, being 97.9 per cent, and 9 break. The nine break into 5 timing differences, 3 amount differences and 1 duplicate reference. Nobody is waiting on this during the day, so a batch is the right shape.

Event-Stream Processing: the monitoring, and what faster detection was worth

Event-stream processingReacting to each item as an event, as it arrives, without waiting for a batch to close. means reacting to each item as an event rather than waiting for a batch to close. The fraud monitoring on the servicing book runs this way. Moving the daily reconciliation onto the same arrangement was costed at 40 working days and would have cut break detection from up to 24 hours to under a minute. A break is investigated on the next working day regardless, so faster detection would have changed nothing a customer or a controller ever experiences. The bank did not do it.

There is a sharper version of the same lesson in the reconciliation itself. Before automation, one person spent 180 minutes working through the list and 45 minutes investigating the 9 breaks, being 225 minutes. After automation the matching is instant, but each break now takes 25 minutes instead of 5. The person no longer builds up any sense of the day by going through the list. Nine breaks at 25 minutes is 225 minutes. The total is identical. The automation saved no time at all at this volume; what it bought was elapsed time and the ability to run more often. The saving only appeared when the design changed to show the person what the automation had compared. A break then came down to 12 minutes, so 108 minutes in all and 117 minutes a day saved.

What does an automated document decision owe an audit trail?

A short, fixed list, and the surprise in it is that two of the items are about the image rather than about the decision. An audit trailThe record of what acted, on what, producing what, and who could have intervened. here holds the image exactly as it was received, its quality barA chosen minimum for an image, below which it is treated as unusable. reading, the classification the chain settled on with its confidence value, each of the 14 fields with the value read and the confidence attached to it, and finally what a person changed and when.

The image and its quality reading are on the list because most of what goes wrong downstream started there, and without them nobody can tell a reading fault from a supply fault. The images and their readings had been kept, so Sumeru could discover that 812 of its 1,264 stopped files carried an image below its own quality bar, being 64.2 per cent. Had the trail held values only, the same 1,264 would have looked like a component problem, and the money would have gone to the component.

How does a lender, an operations head or a reviewer actually use this?