Computer Vision in Finance: Reading Images and Scanned Documents

Computer vision means a component that takes an image as pixels and returns a statement about it. In finance that statement is almost always one of three: what kind of thing is in the picture, where in the picture a thing sits, and what the characters say. Each statement covers the image and the conditions it was taken in, a narrower thing than the claim a deployer acts on.

The fact the rest of this guide turns on is not a fact about pixels. The image is produced once, by the customer, on the customer's device, in the customer's light, and every step after that inherits whatever that one moment produced. No component further along the chain can put back information the camera never captured. The inheritance of one moment is why vision failures land on identifiable groups of people rather than scattering across a population, and why the same component can read as accurate in aggregate and be unusable for somebody specific.

What does it mean for a component to read an image?

Start with what an image is to a machine. An image is a grid of numbers. A photograph of a salary slip is a few million small readings of brightness and colour arranged in rows and columns, and nothing in that grid says salary slip, or says anything at all. Computer visionA component that takes an image as pixels and returns a statement about it. is the name for a component that takes that grid in one end and returns a statement out of the other end. Pixels in, a statement out. The component does nothing else.

The statement always comes with a number attached, a confidence, and that pairing matters more than it looks. The component does not hand over a fact. The component hands over a proposition and a strength. The decision about what to do at each strength belongs to the bank and not to the component, and confusing the two is the beginning of most of the trouble that follows.

The everyday version runs like this. Somebody is handed a photocopy of a handwritten note in a badly lit corridor and asked what it says. An answer comes, and it comes with a feeling attached, somewhere between certain and guessing. If the corridor were brighter, the same note would be read with more certainty. Nothing about the note changed. The conditions it was read in changed, and any honest account of the answer has to mention them. A vision component is in exactly that corridor every single time, and it never gets to ask for the light to be turned on.

So the honest form of every vision output is longer than the form a system usually stores. Not this is a salary slip, but this image, taken in these conditions, at this size and this sharpness, matches what I was shown of salary slips more closely than anything else I was shown. The system stores the short form. The person reading the record later sees the short form. The long form is where the truth lives, and by the time anybody needs it, it has usually been thrown away.

What are the three statements a vision component makes about a picture?

Almost everything a bank asks a vision component for is one of three statements. The three fail in different ways and need different evidence, so it is worth being able to say which one is in play.

Statement one, what kind of thing is in the picture

The component returns a label and a confidence. A salary slip. A bank statement. An identity document. At Sumeru Bank Limited, invented, that job belongs to component 3, the document classifier, sitting inside the retail personal loan intake chain. The label is a sorting decision, not a reading decision: nothing has been read yet, and no field has been pulled out of anything. A component that sorts documents well answers which pile, and which pile is a far smaller question than what does this say.

The characteristic failure of statement one is a confident label on a kind of thing the component was never shown. The component does not shrug. Instead it returns the nearest label it holds, with a confidence that looks like every other confidence, and the file moves on.

Statement two, where in the picture a given thing sits

The component returns a position: a rectangle on the image, given in pixels, said to contain the field that was asked for. The employer name is in this box. The account number is in this box. Position produces no headline and no customer-facing sentence, so this quiet middle statement almost never gets discussed. Position is nonetheless the step that decides what the next step is even looking at.

The characteristic failure of statement two is a box in the correct place on a printed sheet that is skewed, cropped, or photographed at an angle so that half the intended content sits outside the frame. The box is right about the layout it was given. The layout it was given is not the layout of the document.

Statement three, what the characters in the picture say

The component returns characters, and a confidence for them. Reading the characters is the statement most people mean when they say a machine read a document. At Sumeru the job belongs to component 4, the field reading and extraction step, pulling 14 fields from each file. How that reading is actually done, and how such a step is measured, is covered under character reading and field extraction. The kind of claim matters more here: a claim about strokes of ink in a particular photograph, in particular light, at a particular sharpness.

Statement three fails fluently, and the fluency makes it the most dangerous of the three. A reading step asked to read a smudge returns a plausible set of characters rather than an admission that the strokes were not captured. A component that is reliable at sorting documents is not thereby reliable at reading them, and no amount of accuracy on the first statement establishes anything about the third.

Where does vision sit inside one deployed chain?

Sumeru Bank Limited runs its retail personal loan intake chain from a customer starting an application on a handset through to a decision and a disbursal. Three of the chain's nine components take an image as their input. Component 2 is the liveness check on the selfie image. Component 3 is the document classifier. Component 4 is the field reading and extraction step. Everything else in the chain works on what those three produced, or on data that never passed through a camera at all.

Notice the ordering. The sequence is the whole architecture of the problem. The camera fires once. After that there are three components in a row, each consuming what the previous one produced, and not one of them has any way to go back and ask for a better photograph. The chain is a sequence of readers, and the only writer in it is a customer standing in whatever light they happen to be standing in.

Think about a wedding photographer for a moment. Everything printed, framed, put in the album and posted to relatives comes out of a few hundred moments that either were or were not captured on the day. The album designer is skilled. The printer is excellent. Neither of them can produce a photograph of the one relative who was standing in shadow. Every craft downstreamAny step that consumes what an earlier step produced. of the shutter is a craft of arrangement, not a craft of creation. A financial intake chain is the same shape, and it is worth holding that picture when somebody proposes fixing an exception problem by improving a reading component.

What does a liveness check actually assert, and what does it not?

Component 2 is a liveness checkA component that judges whether an image is a photograph of a live person rather than of a picture.. Its job is to separate a photograph of a live person in front of a camera from a photograph of a photograph, a screen, or a printed sheet. Over one month the check rejected 620 of the 10,000 applications that were started, 6.2 per cent of them, and it did so before any of those files reached the decision engine.

Now say precisely what a rejection asserts. A rejection asserts that the pixels in one image did not match what the component was fitted to expect from a photograph of a live person in reasonable light. The output holds nothing beyond that. The assertion covers one image and the conditions that image was taken in. The assertion covers no person, no document, and not whether an applicant is who they say they are.

Somewhere between the component and the exception desk, that statement about pixels turns into a statement about a person, and nothing in the system marks the change. No field flips. No warning fires. A record that reads a numerical outcome on one image is read by a person, summarised into a queue, and comes out of a telephone call as a sentence about the applicant. The transformation is invisible because it happens in ordinary language rather than in software.

Agrawal, Gans and Goldfarb make this point in Prediction Machines, and it is worth naming because it is easy to nod at and hard to hold. A learned component produces a prediction. A prediction is not a decision, and it is certainly not a judgement about a person. Somebody still has to decide what to do with the prediction, and at Sumeru that somebody was a desk with a queue and a script, working at speed, with no field in front of them recording what the component had actually asserted.

Set the two claims side by side and the correction is not complicated. The correction is not even technical. Changing one sentence is the whole of it, and that costs nothing except somebody deciding it matters.

Why does image quality decide so much of what happens afterwards?

Because it is the one input in the whole chain that the deployer does not control, and every later step is a function of it. Sumeru sets an internal image quality barA chosen minimum for sharpness, lighting and size, below which an image is treated as unusable.: a chosen minimum for sharpness, lighting and size, below which an image is treated as unusable. The bar is the bank's own construction. So is the index it scores images on. The index runs from 0 to 100 and is not comparable with anybody else's index of anything.

The bank has measured, on its own files, how the index relates to how many of the 14 fields on a file come back readable. The share of fields returned above the acceptance bar is 0.50 plus 0.0062 times the index, capped at 1.00, and the field count is that share times 14 rounded down. At the bank's median index of 74 the share is 0.9588, or 13 of the 14 fields. At an index of 40 the share is 0.748, or 10 fields. The count falls away faster than the index does, so a modest slip in image quality produces a disproportionate jump in work at the desk.

One thing to hold before the control is moved. The relationship is an average measured across files, and the rounded-down count is what it predicts for one file at a given index. Quality is not the same on every file, so the month itself does not look like 8,600 copies of that one file. Across the whole month 6,020 of the 120,400 fields were routed, being 5.0 per cent, close to what the relationship implies at the median. The 6,020 fields sat in only 1,264 files rather than one field in each of 6,020 files. The gap between the average and the actual is the whole subject of concentrated failure.

What happens between an image below the bar and a field nobody can read?

An unreadable fieldA field the reading step could not return with enough confidence to accept. is the name the exception desk sees. The field arrives at the desk as a work item with a reason code saying the reading step could not return it with enough confidence, and the desk handles it as a reading problem: somebody opens the image, reads the field with their own eyes, types it in, and moves on. Ismail Sheikh, who runs the desk, watched 1,264 files arrive that way in the month, 14.7 per cent of the 8,600 files that reached the decision engine. An unreadable field was the single largest reason a file needed a person.

The bank then did the useful thing and looked one step back. Of those 1,264 files, 812 carried at least one document image below the bank's own chosen quality bar. The 812 are 64.2 per cent of them. The remaining 452 had acceptable images and an unreadable field for some other reason. Nearly two thirds of what the desk was handling as a reading problem had arrived as a capture problem wearing a different name.

The reframing changes where the money goes. If two thirds of the exceptions arrive with the information already absent from the image, then every rupee spent making the reading step cleverer is spent on the remaining third, and every rupee spent on the moment of capture, on guidance, on lighting prompts, on letting somebody upload from a different device, works on the two thirds. The chain cost Rs 2,40,00,000/- to build once and Rs 65,00,000/- a year to run, and none of that spend touched the corridor the photographs were being taken in.

Why are the errors never spread evenly across applicants?

One habit here is worth breaking. A rate looks like a property of a population. Six point two per cent of applications rejected at the liveness check sounds like a small tax levied evenly on everybody, a bit of friction that lands lightly on all applicants at once. A rate is almost never that. A rate is the average of groups whose experiences are nothing like each other, and averaging is exactly the operation that hides which group any particular person is in.

A concentrated failureErrors that fall on one identifiable group rather than scattering across a population. is the ordinary case rather than the unusual one, and the reason is mechanical. A learned component behaves according to what it was shown. If the images it was shown were mostly taken in good light on capable handsets, then its behaviour on an image taken in poor light on a modest handset is not a random slip. The same people keep having the same handset and the same corridor, so the behaviour is systematic and it repeats on them.

Cathy O'Neil makes this argument in Weapons of Math Destruction: a headline accuracy figure can be entirely true and still say nothing about the people the errors land on, and the people the errors land on are rarely a random draw. Sumeru's month is one instance of that pattern, an instance rather than a general claim about what any component does anywhere else.

What did the review of 200 rejections find?

Neelima Rao, in the risk function, who did not build any part of the chain, pulled 200 of the month's 620 liveness rejections and had every one of them looked at by hand. Thirty-one of the 200 were genuine applicants: real people, applying for a real loan, who had been turned away by component 2. The thirty-one are 15.5 per cent of the sample.

Then the arithmetic that makes the finding usable. ExtrapolationApplying a rate measured on a sample to the whole group it was drawn from. means applying a rate measured on a sample to the whole group it came from, so 15.5 per cent across all 620 rejections is about 96 people. A rate is an argument and a count is a queue, and the count is what turns a review finding into something anybody will act on. Nobody schedules work against 15.5 per cent. People schedule work against ninety-six people a month.

Then the finding that made the review worth doing. All 31 shared one condition. Every single one had submitted an image taken in low light on a low-specification handset. Not most of them. Not a tendency. All 31, in one cell of the grid, with none anywhere else.

The finding has a wrong reading, and the wrong reading costs people their loan applications. The finding does not say those applicants submitted a bad photograph. The finding says the bank built a step that worked on one kind of image and made it the only way in. An applicant photographing themselves in the light they have, on the handset they have, has done everything the process asked of them, and the failure belongs to the deployment that offered them one route and no other. The image was as good as the process made possible. A second way through was missing, and providing one is exactly what the review recommended. At month 6 there was none.

What follows once a failure is known to be concentrated?

The instinct is to improve the component, and the instinct is wrong here in a way worth being precise about. Improving the component moves the aggregate. A better component raises the number in the monthly pack. But the people the errors are landing on are, by construction, not average cases. Their images carry less information than an average image, and that shortfall is precisely why the component struggles with them. A component that gets better on average gets better fastest on the cases it already handled well. The concentrated group can sit exactly where it was while the headline improves.

The answer is a second routeAn alternative way to complete a step for somebody the first way failed.: another way to complete the same step that does not depend on the photograph that failed. Somebody the image route cannot serve completes the check another way, through a video call, at a branch, through an assisted channel, through any path that does not need the photograph the first path needed. Once the failures are known to fall on one identifiable condition, a second route is worth more than any improvement that raises the aggregate, and that is a design decision rather than a technical one.

The household version is a doorway. If a shop has one step at the entrance and finds that a small, identifiable group of customers cannot get in, it can spend on a slightly shallower step and raise the share of people who manage it, or it can put in a ramp. The shallower step is an average improvement. The ramp is a route. Only one of the two is aimed at the people who were actually being kept out, and only one of them can be pointed at afterwards as something the shop decided to do.

How does a lender, a desk or a reviewer actually use this?