Embeddings: Turning Text Into Numbers That Carry Meaning

An embedding turns a passage of text into a list of numbers positioned so that passages about similar things sit near each other. Positioning text that way lets a system find material by what it is about rather than by which words it contains. Closeness establishes similarity of subject and nothing else: not that a passage answers the question, not that it is correct, and not that it is the current version of anything.

Everything useful about an embedding comes from one property, and everything painful comes from the same property running backwards. Two pieces of writing about the same thing end up in the same place, and that is exactly as true of a withdrawn procedure and the procedure that replaced it as it is of a question and its answer. Both halves of that sentence hold at once, and everything below follows from holding them together. How the numbers are produced is covered separately, and is not needed here.

What is an embedding, and what does turning text into numbers achieve?

A familiar starting point. Somebody walking into a vegetable market for the first time can still find coriander in about forty seconds, with no list of shop names in hand. The sellers have arranged themselves so that the leafy things are together, the roots are together, and the dry goods are at the far end. Nobody searches the market by name. A shopper walks to the part of it that is about the thing wanted, and looks at whatever is within reach.

An embeddingA piece of text represented as a position, so that pieces of text about similar subjects sit near each other. does that to writing. Take one passageOne piece of a document, small enough to be a useful answer on its own rather than a whole file to read. of text, a paragraph or a clause, and hand it to a component that returns a long list of numbers. The list of numbers is the passage's positionThe list of numbers standing for one passage, which is only ever useful in comparison with the positions of other passages.. On its own it means nothing at all and there is nothing to read in it. A position becomes useful the moment there is a second position to compare it with. The arrangement has one property built into it: passages about similar subjects come out near each other, and passages about unrelated subjects come out far apart.

Like sitting near like is the entire working content of an embedding, as far as anybody deploying one needs to care. How the numbers get produced, what is being counted, why the arrangement comes out the way it does: all of that is covered separately. A retrieval arrangement can be built, bought, reviewed and broken on the single fact that like sits near like, and a broken one cannot be saved by knowing more.

Now the step that makes it a search. A question is a piece of text too. Put the question through the same component, get its position, and ask which passages sit nearest. The nearest passages come back. Nearness is the whole operation, and it is worth saying plainly what has just happened: the system compared positions and returned whatever was nearby, and it took no view whatsoever on whether any of it was useful.

What does finding by position buy that finding by words does not?

One case makes the whole arrangement worth building. Ismail Sheikh runs the exception desk at Sumeru Bank Limited, an invented mid-sized Indian bank, and a desk officer working a stopped file asks the question the way a person asks it: the applicant says he earns more than his account shows, what am I supposed to do. The procedure that answers that question is headed corroboration of stated income against salary credits. Read the two side by side. The question and the procedure title have almost no words in common, and a search built on vocabulary overlapHow many words a question and a passage literally share, which is what an ordinary word search counts. returns nothing at all.

Position does not care. Both pieces of text are about the same thing, so both land in the same part of the arrangement, and the passage comes back first. The buy is a large one at a desk where nobody has memorised the title of a procedure and everybody has a question in their own words.

Now turn it round. The reverse case is the one people forget, and it is the reason nobody sensible throws the words away. A reviewer asks for clause 4.2 of the product terms. The string clause 4.2 sits in exactly one passage, and an ordinary word search returns that passage and nothing else. Clause 4.1 and clause 4.2 are about very nearly the same subject, and the arrangement was built to put things like that together. Position, asked the same thing, returns six passages that are all about product terms, several of them about clause 4.1. Each way of searching finds precisely what the other misses. A serious store keeps both and never treats one as a replacement for the other.

How was one bank's document store actually prepared?

Sumeru holds 11,400 documents that a desk officer might reasonably need: policies, procedures and product terms. The documents were cut into 47,000 passages, just over four passages to a document on average. Every passage was given a position. Every passage was also given a short list of fields sitting beside it. The fields carry what the position cannot, and they are the part everybody skips.

Four steps, then, and they are not equal. The cutting step decides more about how the arrangement behaves than every step after it put together. Everything after it works on whatever the cutting produced, and no later step can recover what the cutting destroyed. Cut too large and the answer to a question is one clause buried inside eight printed sides of other material, so the passage comes back and the reader still has to go hunting. Cut too small and the clause arrives without the heading that said which product it applied to, at which point it no longer says what it meant.

Think of somebody photocopying a recipe book for a cookery class. Copy whole chapters and every student gets the right chapter and still has to find the dish. Copy individual lines and somebody ends up with add two spoons with no way of knowing two spoons of what. The choice is made once, at the photocopier, and no amount of careful filing afterwards fixes it.

What has closeness established when two passages sit close together?

One thing. ClosenessHow near two positions sit, taken as a measure of how similar the two pieces of text are in subject. establishes that two passages are about similar subjects. Similarity of subject is the claim, it is the whole claim, and the arrangement was built to make exactly that claim and no other. The claim is a genuinely useful one, and an analyst who keeps it written on a card will spot most of the trouble on this subject before it costs anything.

The everyday version is a shelf in a chemist's shop. Everything on the cough and cold shelf is about coughs and colds. Being about coughs and colds is a real, load-bearing fact, and it is why a customer walks to that shelf. Being on that shelf does not mean that any particular box treats that customer's cough, that any of them works, or that any of them is still in date. The shelf sorted by subject, so subject is the only thing the shelf reports, and every other question is answered by reading the box.

What has closeness not established?

Three separate assumptions get attached to closeness in practice, usually without anybody noticing they have made one, and each of them fails in its own way.

The first is that the passage answers the question. A passage can be squarely about the corroboration tolerance and say only that a tolerance exists and is set out in the schedule, pointing somewhere else for the figure itself. The passage is a perfect match on subject and no use at all. Being about a question and answering that question are two different properties, and only the first one is what the arrangement was built to find.

The second is that the passage is correct. Consider a passage stating the position wrongly, whether through a drafting slip or a change somebody applied in one place and not another. Being wrong is not a change of subject, so the wrong passage sits precisely as close to the question as the one that states the position correctly. Correctness is the assumption that fails quietly. There is no moment at which anything looks unusual.

The third is that the passage is the version in force, and the third assumption is the expensive one. A withdrawn procedure and the procedure that replaced it are about the same thing, written in the same house style, frequently differing in one clause. The two versions sit almost on top of each other. Nothing about a position records which of them anybody is supposed to be following today.

What happens when the withdrawn version and its replacement sit on top of each other?

Take the procedure Sumeru uses to check declared income against a bank statement. The worked case on a stopped file is a declared monthly income of Rs 45,000/- against a corroborated figure of Rs 38,000/- from three months of salary credits, a gap of Rs 7,000/- which is 15.6 per cent of the declared figure and therefore above the bank's own chosen tolerance, so the file goes to a person. Version 5 of that procedure said refer the file. Version 6 replaced it, and version 6 says refer the file and record on the file the reason it was referred.

Seven words of difference. Everything else about the two passages is identical: same subject, same house style, same headings, same defined terms. The two passages therefore sit almost exactly on top of each other in the arrangement, and which of them comes back first is decided by wording so incidental that nobody could predict it in advance. When a question was phrased the way it used to be phrased, in the vocabulary people learned before the change, the withdrawn version came back first.

How often did an old version come back instead?

Sumeru measured it, on the same 200 questions it used for its second measurement of the drafting assistant. The passage that was actually needed was among the six returned for 178 of them, or 89.0 per cent. The remaining 22 questions are the interesting part, and they are not one problem.

Of the 22, fourteen were questions whose wording shared almost nothing with the passage that answered them, and eight were questions where a superseded version of the right document sat closer than the current one. Fourteen plus eight is 22. The two halves need completely different work, so a single improvement effort aimed at the average of the 22 would have fixed neither. Splitting the 22 is what makes the number actionable.

The fourteen are a cutting and wording problem. The fourteen get better by cutting passages differently, or by writing into the store the words a desk officer would actually use rather than only the words a policy drafter used. The work is patient and ordinary, and it has an obvious owner and an obvious method.

The eight are not that at all. Nothing about how those passages were cut or worded caused the miss. The right document was found; the wrong copy of it won. The eight cannot be improved away by doing the positioning better, and that is what makes them the half that matters.

What has to be kept beside the position?

Everything the last section pointed at lands here. The reason the eight cannot be repositioned away is that currencyWhether a document is the version in force today, which is a fact about the record rather than about the subject it discusses. is not part of what a passage is about. Version 5 and version 6 of that procedure are about identically the same subject. Being withdrawn is a fact about the record, not about the writing, so it never shows up in the writing and it never shows up in the position.

So it goes in a field. Beside each passage, Sumeru keeps the document it came from, the version, the date it took effect, the date it was withdrawn if it has been, and the person accountable for it. Five short fields. A store missing any one of them can never be filteredChecked against the fields sitting beside a passage, deciding whether that passage is allowed to be returned at all. on that thing later. The fact was never captured, and nothing in the text can recover it.

Every one of those five will eventually need to be filtered on, and not one of them is part of the subject a passage discusses. None of them can live in the position for exactly that reason.

Would a better way of positioning passages fix any of this?

The meeting that follows the failure goes the same way everywhere. Somebody proposes a better arrangement: position the passages differently, use a component that places them more carefully, put more work into how the store is built. A better arrangement sounds like the responsible answer and is the wrong one, and knowing why saves a quarter of somebody's year.

One question settles any property the retrieval has to respect: is the property part of the subject the passage discusses? If it is, the position can carry it. The position was built to carry subject. The clause covered, the product it applies to, the step of a process it describes: all of those are subject, and a better arrangement genuinely helps with every one of them.

If it is not part of the subject, no arrangement of positions will ever carry it, however good the arrangement gets. Whether a document is in force, when it took effect, who signed it, whether a customer is allowed to see it: none of those changes a single thing about what the passage discusses. Two documents can be identical in subject and opposite in every one of them. So the property has to be a field, and the filter on that field has to run before anything is returned rather than as a preference applied afterwards.

The before matters. Filtering afterwards means the withdrawn copies still take up places in the six that came back, so four useful passages come back instead of six and nobody can see why the answers got thinner. Filtering first and ranking what survives gives the six places to six passages worth acting on.

How does a lender, a reviewer or a desk officer actually use any of this?