Document Classification vs Entity Extraction

Document classification decides what a document is. Entity extraction decides what values sit inside it. Classification runs first and its answer sets the field list extraction then goes looking for, so a wrong classification does not produce a small error downstream, it sends the extraction after fields that were never on the document. The two steps fail differently, and each needs its own measure.

Classification and extraction sit next to each other in one chain, they are usually built by the same people in the same quarter, and they are almost always reported on one slide with one number each. Reporting one number each is where the trouble starts. A chain is not the average of its parts. Money should not go to whichever step carries the lower accuracy figure. Money should go to whichever step's errors survive long enough to reach a person. At the bank in this case the more accurate of the two steps is also the one whose mistakes are almost entirely caught for nothing. The less accurate step has no free catch behind it at all. The two accuracy figures rank the steps one way and the free catch reverses that ranking.

What does each of the two steps actually assert?

Consider a pile of paper on a table, a familiar enough scene. Somebody hands over an envelope from a tenant applying for a flat. The first act is not to read anything carefully but to sort: this is a salary slip, this is a bank statement, this is an electricity bill, this is a photocopy of an identity card. Only after the sorting does the reading of values begin, and what gets read off each one depends entirely on which pile it went in. The salary slip supplies an employer and a net pay figure. The electricity bill supplies an address. Nobody looks for an employer on an electricity bill, and the reason nobody does is that the type of the document has already been decided.

Sorting before reading is the whole shape of it. Document classificationDeciding which kind of document a piece of text or an image is. asserts a type for the whole document: this is a pay slip. One document, one answer. Entity extractionPulling a named value, such as an employer or an amount, out of a document. asserts a value for a named place: the net pay on this document is Rs 38,000/-. One document, many answers, one for each field it was asked to find. The two steps do not differ in difficulty or in cleverness, they differ in what kind of sentence they produce, and every other difference in this guide follows from that one.

At Sumeru Bank Limited, invented for this guide, the retail loan intake chain receives 4 documents on each of the month's 8,600 completed files. Four documents on each of 8,600 files makes 34,400 documents. The classifier produces one answer for each of those 34,400. The reading step then produces 14 field values for each file, and 14 values on 8,600 files makes 120,400 values in the month. Two steps, one after the other, and their outputs are not even counted in the same unit.

Why does classification have to run first?

Back to the envelope on the table. Suppose the values had to be read off the documents before any sorting was done. There would be nothing to look for. A net pay figure cannot be searched for until the thing in hand has been decided to be the sort of document that carries one. The sorting is not a tidy-up performed for the convenience of the reader. Sorting is the step that supplies the question.

In the chain the same thing happens mechanically. The classification answer picks a field listThe set of values a chain expects to find, chosen by the classification., and that field list is what the reading step is handed as its instructions. Answer pay slip and the reading step goes looking for an employer name, a pay period, a gross figure and a net figure. Answer bank statement and it goes looking for an account number, a statement period and three months of salary credits. Same reading step, same images, different question. The first step chose the question.

The ordering is therefore not an implementation detail that a different design could reverse. The ordering is the mechanism. The second step has no question of its own until the first one gives it one. This is worth holding on to, because it is what turns a modest-sounding error rate on the first step into something structurally different from an error on the second. An error on the second step is a wrong answer to the right question. An error on the first step is the right machinery pointed at the wrong question entirely.

What happens downstream when the classification is wrong?

Here is the sentence that most people get wrong on first meeting, and it is worth slowing down for. A wrong classification does not make the extraction worse. A wrong classification makes the extraction irrelevant, and leaves the extraction looking perfectly healthy.

Take one file. A bank statement arrives and is filed as a pay slip. The reading step is now handed the pay slip field list and told to find an employer name, a pay period and a net pay figure. The reading step does exactly what it was built to do. The step scans the document, finds text that sits where an employer name usually sits, and returns something. On a statement that something might be the name of the bank branch printed at the top. A statement carries a period, so the step finds a period. A statement is full of numbers, so the step finds one that looks like a net figure. Every one of those readings can come back with a high reading of its own. Nobody asked the reading step whether the document it was given was the right document.

A degraded answer and a redirected answer are not the same failure. A degraded answer is blurry and announces itself: the value comes back partial, or unreadable, or below whatever bar the bank set. A redirected answer is crisp and wrong, and there is nothing on the face of it to say so. A misclassificationA document filed as the wrong kind, redirecting everything after it. is therefore a harder failure to spot than a reading failure, and it is far rarer.

How is each one measured, and why are the two figures not comparable?

Look at what the chain writes down for one file. The record settles this argument faster than any argument does. For each of the four documents on the file there is a single line: what the classifier said this document was, and one reading attached to that answer. Then, for the file as a whole, there are fourteen lines: a field name, a value, and a reading attached to each value. One answer against fourteen, in two different units, is why an accuracy figure taken from one of them cannot be set beside an accuracy figure taken from the other.

The classifier at this bank reads 96.2 per cent of documents correctly. The 96.2 is a share of 34,400 things. The reading step clears 95.0 per cent of fields. The 95.0 is a share of 120,400 things. The two percentages sit within about one point of each other on a slide and they answer questions that have no relation. One says how often a whole document is filed correctly. The other says how often a single value clears a bar. Neither is a statement about the other, and no arithmetic converts between them.

There is a second trap sitting inside that 95.0, and this sequence is where it does the most damage, so take it slowly. The 95.0 per cent is the share of fields that cleared the bank's own chosen confidence bar; it is not a statement that 95.0 per cent of the accepted values are correct. Those are separate facts. The bank hand-checked a sample of 400 accepted fields and found 7 of them wrong, a rate of 1.75 per cent. Applied to the month's 114,380 accepted fields, 1.75 per cent puts about 2,002 wrong values through without anybody looking. The meaning of a confidence value, and how a bar gets chosen, are set out under confidence scores. The 95.0 is an acceptance share, and reading the 95.0 as an accuracy is the fastest way to reach a wrong conclusion about this chain.

How are the two kinds of error caught, and what does each one cost?

Now the fact that changes everything, and it has nothing to do with either step's cleverness. The fact has to do with what sits after each of the two steps.

A wrong classification usually contradicts itself within a few seconds. File a statement as a pay slip, and the reading step comes back with an employer name that is a bank branch, a pay period in a format pay slips do not use, and a net pay that does not sit where a net pay sits. The chain has a downstream checkA later test that catches an earlier error before anybody has to look at it. that notices these contradictions and sends the document back to be classified again, and that check costs nobody anything. The check runs on its own, in the same few seconds, and no person is told about it.

Put the month through that. At 96.2 per cent on 34,400 documents, 1,307 documents are misclassified. Of those, 1,224 are caught by a downstream check at no cost. 83 survive to reach a person. 1,224 and 83 make 1,307, and the survival rate is about 6.4 per cent. Ninety-four out of every hundred classification errors are corrected before anybody knows they happened. A free correction rate of ninety-four in every hundred is the single fact that decides where improvement money should go on this chain.

The reading step has nothing equivalent behind it. When the reading step cannot read a field, no second machine knows what the field should have said. A field value is precisely what the chain did not previously have. So a field that fails goes to the exception deskThe people who handle files the chain could not finish., every time. In the month, 6,020 of the 120,400 fields did not clear the bar. Failures cluster in files rather than spreading across them, so those 6,020 sat in only 1,264 files, an average of 4.76 fields to a file. The 1,264 files are exception cause 1, and they are 42.0 per cent of the desk's month.

How does one desk's work actually split between the two steps?

The 83 do not arrive at the desk wearing a label saying classification error. Each of the 83 arrives the same way every other stopped file arrives, as a file with a field the chain could not settle. So the 83 sit inside the 1,264, not beside it. Of the 1,264 files stopped for an unreadable field, at most 83 come from a classification error and at least 1,181 come from a reading error. The split is 6.6 per cent against 93.4 per cent.

The at most and the at least are doing real work in that sentence and are not hedging. A file can carry both faults at once: a misfiled document and, independently, a genuinely unreadable value. Every such file has already been counted in the 83, so 83 is a ceiling on the classification contribution and 1,181 is a floor under the reading one. The whole argument turns on which of the two numbers is bigger, so the direction of the uncertainty matters. The uncertainty runs the wrong way for the classifier rather than in its favour.

Now put the two facts side by side. The two accuracy figures, 96.2 and 95.0, look like near neighbours. The two contributions to the desk, 83 and 1,181, differ by more than fourteen times. Same two steps, same month, same bank.

Why does the more accurate step put less work on the desk?

Because accuracy is not the same quantity as consequence. The chain's next move after an error is what separates accuracy from consequence. Two steps can produce errors at similar rates and hand a person completely different amounts of work, purely because of what sits behind each of them.

The gap between accuracy and consequence gives improvement work a ceiling that can be computed before a rupee is spent, and the computation takes about a minute. Suppose the classifier were made perfect. Not better, perfect, at 100 per cent. Nothing is. The 1,307 misclassifications go to nil, so the 83 that reached the desk go to nil, and the desk goes from 3,010 files to 2,927. A perfect classifier removes 83 files a month from a desk handling 3,010, or 2.8 per cent of its work. The 2.8 per cent is the absolute best case rather than a realistic one.

Said out loud in a room, that ceiling changes the character of the conversation. Nobody is arguing about whether the classifier could be improved. It could. The question is whether the best imaginable outcome is worth the programme, and 2.8 per cent of one desk's month is a small answer to be holding once the money has already been spent.

What does a new kind of document do to each of them?

Everything above assumes the mail keeps looking like the mail. The assumption breaks the moment something new arrives, and two different new arrivals get constantly merged into one.

The first is a new kind of documentA type such as a pay slip or a statement, distinct from the channel it arrived on.. At this bank a fifth kind began arriving through one route, and the classifier reads it correctly 71.0 per cent of the time, against 96.2 per cent on the four kinds it was fitted on. The gap is 25.2 points, and it is a gap on a question the classifier was never asked before. At 8 per cent of the month's documents, being 2,752 of the 34,400, the headline reading falls to 94.2 per cent, so the number on the slide moved 2.0 points while one part of the incoming mail was being read a quarter of the time wrongly.

The second is a new channelThe route a document arrived by, changing its images without changing its type.. Take the same four kinds of document and let them arrive through a route that was not in the fitting set. The images are different: different camera, different lighting, different crop. The reading falls from 96.2 per cent to 93.1, a gap of 3.1 points. The same question is being asked on worse pictures.

The two readings answer different questions, and merging them destroys both. The fix for each is different and the effort each deserves is different. A new kind of document needs the classifier taught a type it has never seen, and until that happens the safe behaviour is to stop those documents rather than file them as the nearest match. A new channel needs the images improved or the fitting set widened, and the type answer was never in doubt. Confusing the two rebuilds a classifier to solve a photography problem, or buys better cameras to solve a problem about what a document is.

Where should the effort go when both could be improved?

Not to the lower figure. To the larger surviving error count, and those are different quantities that happen to be reported in the same place. The calculation has three lines and needs nothing that is not already to hand.

Count the errors each step makes in a month. Subtract the ones a later check catches without a person. The remainder is the work that step is actually sending to somebody, and the remainder is the only column a budget can act on.

The reading step's 6,020 are field failures and the 1,181 is the count of files those failures sit in. Routed fields cluster, so 6,020 of them sit in 1,264 files at 4.76 fields to a file rather than spreading across 6,020 separate ones. The last row is in files for both columns. Files are the unit the desk works in and the only unit in which the two steps are comparable at all. Every row above the last one invites the wrong comparison, and the last row is the only row a spending decision should be made from.

How would somebody outside the project use this?

Building a chain is not a precondition for this being useful. Three people read it differently, and all three are reading the same two rows.

A lender's own reviewer, sitting where Neelima Rao sits in the risk function of this bank, wants one thing from a document processing chain: the count of errors that reached a person, per step, per month, and the count that was caught without one. Ismail Sheikh runs the exception desk and can supply the first count from his own queue. The second count is the one nobody keeps. A check that fixes something silently produces no record unless somebody decided in advance that it should. A chain that cannot say how many errors it corrects invisibly cannot say where its next rupee should go, and that gap is a design choice made much earlier by somebody who did not think anyone would ask.

An analyst reading a lender who says it has automated its document handling should ask what the accuracy figure is a share of. If the answer is a percentage with no denominator attached, the figure is not yet information. Two lenders quoting 96 per cent may be counting documents and fields respectively, and a difference of that size sits inside a rounding error on the way the figure is written.

And anybody approving spend on either step should ask for the ceiling first, in the form used above: if this step became perfect, what leaves the desk? The ceiling is a one minute calculation, it is available before any vendor conversation, and on this chain it would have returned 2.8 per cent. Ajay Agrawal, Joshua Gans and Avi Goldfarb, in Prediction Machines, 2018, make the general form of this point: a component of this sort produces a prediction, and the value of a better prediction is set entirely by what somebody does differently as a result of it. Here, at the ceiling, somebody does 83 fewer files a month differently. Eighty-three files a month is the whole prize, and the prize was knowable on day one.