Explainability and Interpretability Compared

Interpretability means the component itself can be read, so somebody can say what it will do before any case arrives. Explainability means one output can be accounted for afterwards, without the component being readable at all. Interpretability and explainability are not two strengths of one property. One is settled when the thing is built and can be missing entirely; the other is a service laid on top.

The two words get used as though the second were a weaker version of the first, and that mistake is worth taking apart. The two words answer different questions, they are bought at different moments, and a firm can hold a great deal of one and none of the other. At Sumeru Bank Limited, invented, the retail loan intake chain holds exactly one component of each kind, and in one steady month the difference between them decided what 1,290 refused applicants could be told. The practical test is not whether an account of a decision reads well, but whether acting on it would have changed the decision. The bank's own wording does not come through that test intact.

What actually separates the two, in one sentence?

The distinction is easiest to see in a kitchen. A recipe card is readable: it can be picked up before anybody has cooked anything and read for what will come out, and if the result is too salty the line that says two spoons of salt can be pointed at. A cook who has been making the same dish for thirty years without a card is not readable at all, and asking why today's version came out saltier draws an account made afterwards, from memory and from tasting: probably the stock, possibly the cheese. Both answers may be true. Only one of them existed before the dish did.

InterpretabilityThe property of a component that can be read, so somebody can say what it will do before it does it. is the recipe card. Interpretability is a property of the component itself, available before a single case has arrived, and it is settled at the moment the component is built. ExplainabilityThe ability to account for one output after it has been produced, whether or not the component can be read. is the cook's answer. Explainability is a service produced on top of one output that already exists, it can be arranged at any time, and it makes no claim at all about the next dish. The one sentence that separates them is about when the looking happens, not about how hard it is. Calling a component partly interpretable is therefore usually a category error. People who say it mean that somebody has arranged an account of the component's outputs, and an account of outputs is the second property doing the work of the first.

Which of the two does a person who has been refused need?

An applicant whose loan has been declined is not asking to read anything. The applicant is asking about one output, their own, and that output already exists. Answering that question needs the second property and only the second property. A firm can therefore answer a declined applicant even when the component behind the answer cannot be read by anybody. The reverse is also worth stating plainly. A supervisor, a validator or the person accountable for a component is usually asking the first question. Their concern is not one case but what the component will do next month, and no quantity of accounts of past outputs adds up to that.

So the two properties serve two different audiences, and a firm that supplies only one of them has left an entire audience unanswered. Sumeru Bank Limited supplies an account of a single output to every declined applicant, and that is the right property for that audience. The bank supplies almost nothing of the first property on the same component. Neelima Rao's independent validation of component 6 therefore took eleven working days, and reading the whole of component 5 took twenty five minutes. The gap between those two figures is not a comment on anybody's diligence. The gap is the price of a component that cannot be read.

What is an attributed reason, exactly?

Component 6, the scoring model, returns something with every decline: the three inputs that moved its value most for that applicant. Those three inputs are an attributed reasonThe inputs that moved a component's value most for one case, returned alongside the output., and an attributed reason is a real measurement rather than a story. Being precise about what an attributed reason is not matters here. Pointing at a layer inside a component and naming it as the cause is a fabricated explanation, not a small one. A layer is part of the machinery, not a thing that happened to the applicant. An attributed reason avoids that entirely. The inputs it names are things about the applicant, and the reason reports how much each of them moved the value.

The reason states a description of what the component weighed, and every word of that sentence is doing work. Weighed, not decided. For this case, not in general. Description, not promise. The record produced for one decline at the bank holds a file reference, the component, the outcome and three named inputs, and it holds nothing else at all. Everything a reader would like it to say next has to come from somewhere else or from nowhere.

What does an attributed reason not say?

Here is the sentence that does the damage, and it is never written down anywhere: if these are the three things that counted against me, fixing them will fix the answer. The unwritten sentence turns a description into advice. The reading is entirely natural, it is the reading almost every human gives a list of reasons, and it is a counterfactualWhat would have happened had an input been different, which an attributed reason does not state. claim that no attributed reason makes.

Neelima Rao tested the claim rather than arguing about it. Testing is the only respectable way to settle a question of this shape. She took 18 of the 100 sampled declines, altered only the inputs the reason had named, moved them to values the reason implied would have helped, and put each file through again. Eleven of the eighteen were declined a second time. Acting on the reason would have failed 11 times in 18, being 61.1 per cent, and this is a property of attributed reasons rather than a defect anybody at the bank introduced. Nothing was broken. The value the component produces is not built out of three inputs acting alone, so it weighed those three most heavily and still landed in the same place.

How can an explanation be tested for whether it is worth anything?

The instinct is to test whether the account reads well. Is it in plain words, does it name things the applicant recognises, would a person outside the team understand it. Every one of those questions is about the writing. The only test that measures the explanation rather than the prose is the re-run testAltering the inputs a reason named and putting the case through again to see whether the answer moves.: alter what the reason named, change nothing else, and put the case through again. If the answer moves, the account carried something actionable. If it does not, the account was true and useless in the same breath, and no amount of rewriting will change that.

Three details make the test honest and each of them matters. Alter only what the reason named. Moving anything else measures a different question. Move each input to a value the reason implied would have helped rather than to an extreme. An extreme will move almost any answer and prove nothing. And run enough cases to see a share rather than an anecdote: eighteen is a small number, and the bank reported it as a reading on eighteen files rather than as a rate for the component. The test is cheap. It needs no access to the inside of the component, and that is exactly the point. On a bought component there is no inside to reach.

How distinctive is a reason, and what does naming more inputs buy?

There is a second way an account can be true and useless, and it has nothing to do with counterfactuals. A reason can be perfectly intelligible and still be the reason nearly everybody got. Over the same sample of 100 declines, Sumeru Bank Limited counted how many distinct reasonA reason that differs from the reasons given to other cases rather than repeating them. triples came back. The answer was 30. One of those thirty came back for 71 of the 100 applicants.

Sit with the arithmetic for a moment. The count forces something the bank never had to state. Thirty distinct reasons cover one hundred declines. One of them covers 71. The other 29 declines spread across the remaining 29 reasons, and since each of those reasons was produced at least once, each was produced exactly once. So the thirty distinct reasons are one crowded triple shared by 71 people and twenty nine reasons given to a single applicant each, and 71 plus 29 is 100. A count of thirty sounds like variety. The shape underneath it is not variety at all.

Why is a longer reason both more distinctive and less usable?

The obvious repair is to name more inputs. If three inputs produce thirty distinct reasons, eight will produce more, and the bank swept exactly that. Naming one input gives 5 distinct reasons across the hundred declines, two gives 13, three gives 30, four gives 48, five gives 64, six gives 75 and eight gives 89. The climb is steep and it does not stop. There is a ceiling working underneath those counts that is pure arithmetic: with d distinct reasons spread across 100 declines and every reason produced at least once, the commonest reason can cover at most 101 less d applicants. At three inputs that ceiling is 71, exactly what the bank measured, and at eight inputs it falls to 12.

So lengthening the reason genuinely buys what it looks like it buys. Lengthening spends the readability that made the reason useful in the first place: the bank chose three because at three the reason fits one sentence a person can act on, and at eight it is a list nobody reads. The curve therefore has a middle rather than an end, and a firm choosing a length is choosing between a reason that is shared and a reason that is ignored. Neither end of that control is a good place to sit.

What does interpretability cost on a task where it can be had?

Losing readability leaves a firm with the account after the fact and nothing more. The price of keeping readability is worth seeing too, on a task at the same bank where keeping it was actually on the table. Component 3, the document classifier, was built twice. The readable build reads 91.5 per cent of documents correctly and has a nameable driver behind each answer. The unreadable build reads 96.2 per cent correctly and has none. The gap of 4.7 percentage points is the whole price of the explanation on that task, and it is not a small number multiplied by 34,400 documents a month.

Sumeru Bank Limited deployed the unreadable build, and on that task it was the right call for a reason that has nothing to do with performance. Component 3's output never leaves the system. The classifier feeds the reading step, and nobody outside the bank is ever told anything by it, so there is no audience for readability to serve. Buying the readable build there would have been paying 4.7 points for an explanation nobody would ever ask for. The same firm, in the same programme, refused that trade somewhere else.

Where did the same committee refuse that trade, and why?

Component 6, the scoring model, decides whether an applicant is accepted or declined, and 688 applicants a month are declined by it outright. Something has to be said to each of them. The committee that had cheerfully accepted an unreadable classifier looked at the same trade here and would not take it, and the difference between the two decisions is not technical at all. The difference is that on component 3 nobody has to be answered and on component 6 somebody does.

Notice what that refusal could and could not achieve. The two properties come apart for good here. The committee could not buy readability back. Component 6 was not readable by construction and there is no route from an unreadable component to a readable one that does not mean building a different component. The committee could insist on an account of every output, so it did, and the attributed reason exists because of that insistence rather than because anybody thought an account was equivalent to reading the component. The limits on what an attributed reason can carry are therefore not a criticism of the decision but a description of what was still available to buy once the first property had gone.

What do two refusals in one chain actually look like side by side?

Now put the two components in the same month and look at what an applicant receives. In one steady month 1,290 files at Sumeru Bank Limited reach an adverse outcomeAn outcome the person it was about would describe as a refusal.. Six hundred and two of them are routed by component 5, the income corroboration rule, and that rule is 34 lines a person can read end to end in twenty five minutes. Six hundred and eighty eight are declined outright by component 6, and nobody can read component 6 at all. 602 plus 688 is 1,290.

The difference in what can be said is enormous and it is entirely a difference of property rather than of effort. Behind each of the 602 there is a printed procedureThe written rule behind an outcome, which exists only where the component is readable.: the worked case declares Rs 45,000/- of monthly income against a corroborated Rs 38,000/-, a gap of Rs 7,000/- being 15.6 per cent of the declared figure, above the bank's own chosen tolerance of 10 per cent, so the file is routed. The printed procedure states what happened, why, and what a different figure would have produced. Behind each of the 688 there is a reason naming three inputs, and no procedure at all. So 53.3 per cent of this chain's refusals cannot be answered by quoting anything, and a complaints desk working from one script has the wrong script for the majority of the people who write in.

Which of the two can be bought after the fact, and which cannot?

The practical difference decides a build rather than a letter. An account of one output can be arranged on a component that is already running, already bought, already deployed and entirely closed. An account needs no access to the inside, and it is therefore available on a vendor-hosted component where there is no inside to reach. Readability is settled at the moment the component is built, by how it was built, and there is no later purchase that recovers it.

So only one of the two has a deadline, and it passes long before anybody complains. A firm that leaves the question until an applicant writes in has already answered it, by default, in favour of the property it can still buy. Answering by default is not always the wrong answer, as component 3 shows. The default is only wrong where somebody was always going to have to be told why, and that is knowable when the component is approved rather than when the first complaint arrives. Section 5 of this bank's policy, on what must be documented before deployment, is where the question belongs, and asking it there costs one meeting.

How does a complaints desk, a reviewer or an applicant use this?

The complaints desk comes first, because it is where all of this lands. Ismail Sheikh's people receive letters from applicants who were refused, and the first thing worth knowing about any such letter is which of the two piles it came from. If the file was routed by component 5 the desk can quote the line, state the two figures, and say what a different figure would have produced. If it was declined by component 6 the desk can state the three inputs and must not go further. The moment it says improving these would have helped, it has made a claim the re-run test says fails about three times in five. Two piles, two scripts, and knowing which pile a letter came from is a lookup rather than a judgement.

A reviewer inside the firm uses the pair differently. Faced with a component, the useful first question is not how accurate is it but which of the two properties does it have. The answer decides how the review itself is conducted. On a readable component the reviewer reads it, and the whole of component 5 took twenty five minutes. On an unreadable one that is not possible, so the review becomes a matter of testing behaviour from the outside, and component 6 took eleven working days. Anybody planning a review calendar who has not asked that question first has budgeted the wrong number of days.

And an applicant, or anybody advising one, gets the most useful point here for free: a reason is a description, to be read as information about what was weighed and not as a list of repairs. If the sender cannot say what would have been enough, that is not evasion and it is usually not a badly written letter. The silence is exactly what an account after the fact amounts to, and knowing that saves a great deal of wasted effort aimed at a door that may not open. Agrawal, Gans and Goldfarb make that point in Prediction Machines: a fitted component produces a prediction somebody still has to act on. O'Neil makes a second one in Weapons of Math Destruction: a component's misses fall unevenly across the people it decides about.