Rule Engines: Encoding Decisions That Must Be Explainable

A rule engine holds the rules of a decision as readable statements, separately from the software that runs them, so the rules can be read, changed and argued with by people who do not write software. Holding the rules apart from the software is the whole point. A decision somebody will have to explain belongs in a rule engine for that reason alone, and the cost of one is keeping every statement in it true at the same time.

Notice what is not in that answer. Nothing about speed, nothing about how many decisions a second, nothing about whether a machine follows the statements better than a person would. Anything can follow rules. The value of a rule engine is not that the rules get followed, it is that the rules exist as a document somebody can point at. A rule engine is a filing arrangement rather than a technology. Keeping the arrangement honest has a cost, and the cost is the part nobody budgets for.

What is a rule engine, and how is it different from rules written into software?

Consider the schedule of charges pinned behind the counter at a small co-operative society office. Three things are going on at once and they are easy to confuse. There is the schedule itself, a printed sheet with numbered items and a dated amendment slip stuck below it. There is the clerk, who reads the sheet and applies it to whoever is standing at the counter. And there is the register in the drawer recording that item four was amended on such a date, by whom, and what it said before. Take away the sheet and the clerk still gives an answer, but now the schedule is whatever the clerk remembers, and there is nothing to argue with.

A rule engineAn arrangement where the rules of a decision are held as readable statements apart from the software that applies them. is that arrangement, built in software. The statements sit in one place as ordinary readable sentences. The software that reads them and applies them to a case sits somewhere else and is deliberately general, knowing nothing about lending in particular. The change recordThe log of who altered a statement, when, what it said before, and who approved the alteration. sits in a third place. Rules written into software are not less correct and not slower; they are simply not separable, so there is no sheet to hand anybody.

Why is that separation the whole of the value?

Because of what it does to an argument. When the rules are inside the software, a disagreement about an outcome has nowhere to land: the person who is unhappy talks to somebody at a counter, who talks to somebody in an office, who eventually asks somebody technical how the software behaves, and the answer comes back as a description rather than as a text. Every step of that chain is a retelling. When the rules are a document, the disagreement lands on a numbered line, and everybody in the argument is looking at the same twenty words.

Three separate people need that document, and none of them writes software. The person answering a complaint needs to say which line acted on this file and what it said. The person reviewing the arrangement needs to read the whole rule setThe collection of statements governing one decision, read from the first line to the last. and form a view on whether it says what the institution meant. The supervisor of a regulated lender needs to be shown, on request, how an outcome was arrived at. A rule engine hands all three of them the same object.

There is a second consequence and it is more practical. Changing a rule stops being a software change. Editing line seventeen is an afternoon, with a review and a record around it, and nothing else in the arrangement moves. The separation is why an explainable decisionOne where somebody can be shown the reasoning that produced their particular outcome, and can check it. and a quickly changeable decision tend to arrive together: they are two effects of the same filing choice.

Where does a decision that must be explained belong?

Two questions place it, and neither of them is about how difficult or how clever the decision is. The first: can the correct outcome be stated in advance? Not guessed at, not usually right, but written down before the case arrives, in a form that lets anybody holding the statement predict the answer. The second: will somebody have to explain one outcome to the person it was about? Two yeses put the decision in a rule engine, and the second yes is the one that settles it.

Run the pair over a lending chain and the answers separate quickly. Whether a declared income is close enough to a corroborated one can be stated in advance, because it is a position the lender is taking rather than a fact waiting in data, and the applicant is owed a reason. Two yeses. Which kind of document an image is cannot be stated in advance in any usable way, because employers format their paperwork four hundred different ways, and nobody outside the system ever sees that answer on its own. Two noes.

What do four real rule sets look like, and how many lines is that?

Abstraction runs out quickly here, so take a real arrangement. The retail intake chain at Sumeru Bank Limited, invented, holds nine separately built components, and four of them are procedures somebody wrote: the identity match, the income corroboration step, the fraud rules that run on the servicing book, and the workflow router that decides where every file goes next. Between them those four hold 126 lines. Not a thousand, not ten. A hundred and twenty six numbered statements decide fifteen point eight per cent of the month's outcomes.

Across the rows, something awkward appears before maintenance has even entered the picture. The fraud rules are the largest set by a distance, at 61 lines, and they cost more to keep true than the other three put together. The fraud rules determined 452 outcomes. The identity match is nine lines, costs almost nothing to check, and determined 189. The size of a rule set has no particular relationship to how much of the month it decides. Anybody about to propose more lines should know that first.

What does keeping a rule set true actually cost?

Here is the thing people get wrong about maintaining written rules, and it is not laziness. Everybody understands that each line has to be right. A rule set sinks on a different point: every line has to be right at the same time as every other line. A statement that is perfectly sensible read on its own can be nonsense read against line forty. So the maintenance job is not reading 126 lines. The job is comparing them.

The comparison has a name and an arithmetic. A pair checkComparing every statement in a rule set against every other, to find two that could give different answers to one case. takes every line and holds it against every other line, asking whether there is any case both could catch and, if so, whether they agree. Each pair is looked at once and a line is not compared with itself, so for a set of n lines the count is n times n less one, divided by two. Nine lines make 36 pairs. Twenty two lines make 231. Thirty four make 561. Sixty one make 1,830.

The rules grow in a straight line and the checking grows as the square. A rule set everybody agreed was reasonable at thirty lines is unmaintainable at a hundred, and nobody noticed the moment it happened. The other way round is the version worth carrying: doubling the lines quadruples the checking. Not double. Four times.

Move it once and the shape of the problem is obvious in a way no table makes it. At the left the shaded triangle is almost invisible: nine lines is 36 pairs and half a morning. Push past sixty and the triangle starts eating the panel. The unpleasant part is that nothing announces itself on the way. Every individual line that got added was reasonable, small and well argued, and the total cost of holding the set together went up by roughly twice the current size each time somebody added one.

Which pairs does anybody actually check?

Now the arithmetic gets sharper, and this is the part that catches out people who have already understood the square. The four sets checked one at a time come to 36 plus 231 plus 561 plus 1,830 pairs, a total of 2,658, or about 6.6 working days. But the four sets are not four separate arrangements. All four act on the same file, one after another, in one chain. Treat all 126 lines as what they functionally are, one rule set governing one journey, and the pair count is 7,875.

The difference is 5,217 pairs, and those are the pairs where a line in one set is held against a line in another. Nobody is responsible for those 5,217. Each set has an author, a reviewer and a review date; the space between the sets has none of those things. Two thirds of the checking on this chain, 5,217 of 7,875 pairs, sits between the rule sets rather than inside any one of them, and between the sets is exactly where nobody is looking.

How do rules contradict each other?

Two shapes, and they look nothing alike. The first is a contradictionTwo statements that would give different answers to the same case.: two lines that both catch the same case and disagree about what to do with it. A file arrives holding a missing corroborated income and a long customer history, one line says route it and another says pass it, and which of them acts depends on which comes first in the document. Nobody wrote that ordering as a decision. The ordering is a decision all the same, and it is now the rule.

The second is an unreachable lineA statement no case can ever reach, because an earlier statement always catches the case first.: an earlier line always gets there first, so nothing ever reaches it. Somebody wrote a careful test for gaps above twice the allowed gap, meaning to mark those files for a closer look, and put it below the line that routes everything above the allowed gap. Every file that would have reached it was already caught. The statement is in the document, it reads perfectly, and it has never once run.

Why does a line nothing can reach survive for years?

At the month twelve validation of the intake chain, Neelima Rao in the risk function read all 126 lines and held them against each other. She found nine, being 7.1 per cent of the set, that either contradicted another line or could never be reached because something earlier always caught the case first. Not one of the nine had produced a wrong outcome that anybody could identify, and that is not a footnote, it is the reason they were still there.

Take the two shapes in turn and the survival makes complete sense. An unreachable line produces nothing: no outcome, no complaint, no alert, no entry in any report. There is no signal for anybody to miss, because there is no signal. A contradiction produces an outcome, but it produces a perfectly ordinary looking one, decided by whichever line happens to sit higher in the document. Somebody who is unhappy with it is unhappy with the outcome, not with the fact that a second line existed that would have said something else.

Nine lines in 126 on a rule set that has been edited by several people over a year is an ordinary finding, and reading it as a scandal is the wrong reading twice over. A pair check exists for exactly that, and a maintained rule set that had never been checked would be more likely to hide more. The uncomfortable version is not that the bank found nine, it is that a rule set which has never had a pair check has an unknown number and no way to say so.

What does the change record have to carry?

A statement on its own answers one question: what the rule says today. The statement does not answer why it says that, who decided it should, or what it said last year. A complaint, a review and a supervisor ask exactly those three questions, and none of them is answerable from the document. The answers come only from the record kept beside the statement, and the record has to carry four things: who changed the lineOne statement in a rule set, testable on its own, usually numbered so it can be cited., when, what it said before, and who approved that.

Take the tolerance line inside the income corroboration rule, the one holding the allowed gap of 10 per cent. The allowed gap decides whether an applicant declaring Rs 45,000/- a month against a corroborated Rs 38,000/- gets an answer in four minutes or waits two working days. The line itself states that the figure is 10 per cent. The record beside it states that 10 per cent is a choice the bank made, that it has not been changed since it was written, and which post signed it off. The line explains what happens; only the record explains why, and why is what the person affected is asking.

When does a rule set stop being the explainable option?

Long before it stops working, and that is the awkward finding. Neelima Rao read the 34 lines of the income corroboration rule end to end in 25 minutes. Hold that rate and 126 lines is about 93 minutes, an hour and a half of somebody's day. The set still works. Every line is still readable. But nobody is going to spend an hour and a half to answer one customer asking why their application was routed, and if the honest answer to a complaint is here are 126 lines, the arrangement has quietly stopped delivering the explanation it was chosen for.

Past a certain size a rule set is auditable but no longer readable, and the two are not the same duty. Auditable means somebody can, given time and a reason, read the whole thing and form a view. Readable, in the sense a complaints desk needs, means the reason a particular file got a particular outcome can be produced now, in a sentence. The second does not follow from the first, and it does not arrive by itself. Something has to be built that reports which line acted on this file, quotes it, and stops.

Where is a rule engine the wrong tool?

Go back to the first of the two placing questions and take a no seriously. Where nobody can state the correct outcome in advance, there is nothing to write down, and putting the step in a rule engine produces a document full of statements that are approximately true and gradually acquire exceptions until nobody can read them. Deciding which kind of document an image is, is the clean example: four hundred employers format their paperwork four hundred ways, and no set of written statements survives contact with the four hundred and first.

People get the next move wrong, not the first one. A no to the first question does not automatically hand the step to something fitted to data. A no may mean the step belongs with a person, and on the eleven step retail process at the invented bank exactly that happened: the step where the assessment is written was left with people, not because nobody had thought about it, but because nobody could say in advance what the right assessment was. Reaching for a component that learns from examples at that point answers a different question from the one that was asked.

There is a second wrong fit that is less obvious. A rule engine is the wrong tool where a decision genuinely changes every week. The change record and the pair check then cost more than the decision is worth. If the statements have to be rewritten faster than anybody can hold them against each other, the arrangement has stopped being a document and become a rolling argument, and the cost of the filing is being paid without the benefit arriving.

How does a reviewer, a complaints desk or a supervisor use this?

What each of them actually does with it

A reviewer uses the pair arithmetic to size the work before agreeing to it. Asked to give an opinion on whether four rule sets are internally consistent, the honest scoping answer is 2,658 pairs and about 6.6 working days if the sets are taken one at a time, and 7,875 pairs and about 19.7 working days if they are taken as the single set they functionally are. The two numbers are three times apart and they answer two different questions. Agreeing to the work without saying which one is being done is how a review ends up meaning less than everybody thought it did.

A complaints desk uses the change record rather than the rule. When somebody writes in about a routed file, the useful reply names the line that acted, quotes what it says, and states that the figure inside it is a choice the lender made. Naming the choice is what turns a refusal into something the applicant can argue with. Quoting the line without the record produces a reply that sounds like a law of nature, and Ismail Sheikh, who runs the exception desk, gets a second letter for every one of those.

Somebody proposing a new line uses the square. The right question to ask of any proposed addition is not whether the line is sensible. The line always is. The question is how much it adds to the total. Adding one line to the 34 line income rule adds 34 new pairs to check. Adding one to a 126 line set adds 126. A rule set gets expensive to hold together in exactly the way an address book gets expensive to keep free of duplicates: not line by line, but pair by pair.

The shape is identical at any scale, so here is a household version. A wedding caterer with four written rules about who gets which plate can hold all four in mind at once and spot instantly that two of them collide over the same guest. At forty rules, covering guests, dietary needs, the head table, children and staff, nobody can hold them all, and the contradictions do not announce themselves. The collisions just produce one confused guest at a time, and everybody assumes the confusion was that guest.