Vector Databases: Storing Meaning for Retrieval

A vector store holds passages together with their positions and the fields describing them, and returns the passages nearest a question. The store is a lookup arranged by similarity of subject. Such a store holds no view about which passage answers a question, no view about which is correct, and no view about which is current, so anything depending on those has to be a field it filters on.

The word database makes people expect a thing that answers, and the expectation does a lot of quiet damage here. A store of this kind is extremely good at finding material on a given subject and carries no opinion whatsoever about whether that material is right, relevant or still in force. Every useful property of such a store, and every painful one, comes out of that one split.

How such a store is built, configured or tuned moves quickly and dates. Two duller and slower things decide whether a deployment works: what the store holds, and what it hands back when somebody asks it something.

What does one entry in the store actually hold?

A familiar object makes the point. A school library used to keep a drawer of cards, one card for each book. The card said what the book was about, so a reader knew whether the book was wanted. The card said where to walk, in the form of a shelf number that meant nothing on its own. And it carried a stamp on the back saying which edition this was and whether the library still had it. Three things on one card, doing three completely different jobs, and only the first of them was worth reading.

An entry in a vector storeA store that keeps passages along with a position for each one, and hands back the passages sitting nearest to a question. is that card. The entry holds the passage itself, roughly a paragraph of one document. The entry holds the passage's position, the long list of numbers that lets the store work out what sits near what. And it holds a set of fieldsDescribed properties recorded beside a passage, such as which document it came from and which version that document is.: which document this came from, which version of it, when that version took effect, when it was withdrawn, and who is accountable for the document.

Now the scale, and every number belongs to the invented Sumeru Bank Limited. Sumeru's store holds 11,400 documents. The documents are policies, procedures and product terms, and the filling job cut them into 47,000 passages, roughly four to a document. At roughly 300 words a passage that is about 1,41,00,000 words of material sitting in the store, and six passages, about 1,800 words, is what comes back to any one question. Six passages is roughly one part in 7,800 of everything the store holds. Most of the work is deciding what not to send.

Why do two decisions taken at filling outlive everything done later?

Two things about a store are settled on the day it is filled and cannot be reached afterwards: how the documents were cut into passages, and which fields were attached to each one. Everything else about the arrangement is an afternoon's work. Both of these need a rebuildFilling the store again from the documents, which is what changing the cutting or adding a field to every entry requires.. Every one of the 47,000 entries is made again from the documents.

The household version is the drawer where the bills live. On the day it is set up, the filing is settled: by year or by service, and whether each envelope gets a note on the front saying what it is. Later on, the choice of which drawer to open first and how many envelopes to pull out can both change. How the whole thing was filed cannot change without pulling every envelope out and doing it again. Nobody ever does.

Every later adjustment operates on the passages as they were cut and the fields as they were attached, so the two decisions nobody photographs for the committee pack are the two that survive all of them. A passage cut so large that it buries the sentence that was needed cannot be repaired by returning more passages. A store with no field for the version cannot be made to exclude an old version by any setting anybody moves.

What happens between a question arriving and passages coming back?

Four steps, and they always happen in this order. First the question is turned into a position by exactly the same treatment every passage was given, so the question and the material end up described in the same way. Second the store works out which entries sit nearest that position. Third the fields on those entries are read and anything failing a filterA check made against a field, deciding whether a passage is allowed to be returned at all. is dropped. Fourth what is left is returned, six passages, each with a pointer back to the document it came from.

The third step sits before the fourth, and that ordering is the entire reason anything can be kept out of an answer. A filter running after the return would be reading a list that has already been handed over. The ordering is the difference between a security check at the gate of a wedding hall and a security check at the buffet: same check, same list, and only one of them is doing anything.

What decides how many passages come back?

Somebody typed a number. A typed number is the entire answer, and it deserves saying plainly. The setting is usually discussed as though it fell out of the arrangement rather than out of a meeting. At this bank the number of passages returnedHow many results come back for one question, which is a setting somebody chooses rather than anything the store works out. is six, fixed, the same six for a question about a fee as for a question about a fraud referral.

Nothing in the store proposes six. The store will hand back one, or six, or two hundred, with equal willingness and no comment. The number of passages returned is a deployment decision made by a person, and a deployer moves it every day without ever calling the move a change to the arrangement. Measuring it beats arguing about it, for exactly that reason.

How often was the passage that was needed actually among them?

Sumeru Bank measured it, on 200 questions, alongside its second measurement of the drafting assistant. For each question somebody first decided which passage in the store actually answered it, then checked whether that passage was among the ones returned. The readings are the invented bank's own and describe this store only: at one passage the needed one was there for 61.0 per cent of questions, at three for 80.5 per cent, at six for 89.0 per cent, at ten for 93.0 per cent, and at twenty for 96.0 per cent.

Every one of those rates lands on a whole number of questions out of 200: 122, 161, 178, 186 and 192 questions respectively. A whole number in every case is a small check that the rates and the test agree. The deployed setting of six is the 178.

The shape is the finding. The first three passages buy 9.75 points each and the last ten buy 0.30 points each, so a deployer who triples the material returned is buying the thinnest part of the curve. The curve matters more than it looks. The same measurement found that where the needed passage came back the answer was right for 171 of 178 questions, and where it did not the component answered anyway and was right for 3 of 22. The share the retrieval finds is very close to the share the whole arrangement gets right.

What does returning more passages cost, and who pays each half?

Two currencies, and only one of them ever reaches a slide. The first is space. Everything supplied for one output shares a single holding limit. At this bank the limit runs to about 12,500 words in total, and the 340 word instruction is already inside it. Six passages take about 1,800 words of that and leave 10,360 words of room for the file itself. The file averages 4,800 words. Twenty passages take 6,000 and leave 6,160, still above the average file and now barely half of it clear.

The second currency is somebody's afternoon. Six passages a question across 3,010 exception notes a month is 18,060 passages handed over, about 54,18,000 words; at twenty it would be about 1,80,60,000. Verification of those notes was a measured 9 minutes each. Across 3,010 notes that is 27,090 minutes, being 451.5 hours a month, about 3.225 posts on the bank's assumed working month and about Rs 29,02,500/- a year at the assumed fully loaded Rs 9,00,000/- a post. The yearly amount is arithmetic on the bank's own figures rather than a separately counted amount.

Nobody budgets the second currency, and it behaves worse than the first: a checker handed six passages opens most of them, and a checker handed twenty opens fewer of them than they did when there were six. Space runs out honestly, with something visibly dropped. Reading time runs out quietly, and what it drops is the checking.

What has to be excluded before anything is returned?

Anything whose wrongness has nothing to do with the subject. A withdrawn version of a procedure is about precisely the same subject as its replacement, so nothing in the positions separates them and nothing ever will. The same goes for a document that has not taken effect yet, one belonging to a product this applicant did not apply for, and one nobody is allowed to quote to a customer. All of them are on subject and all of them are wrong to hand over.

A filter is exactly as current as the field it reads. The filtering step therefore has two ways of failing, and only one of them is ever designed for. The designed-for way is a missing field: if nothing on the entry records the version, no filter can act on the version, and no adjustment made later can invent one. The other way is quieter. The field is there, the filter runs, the filter works, and the field says something that stopped being true a week ago.

How does a store go stale while every component keeps working?

StalenessThe store holding a version of a document other than the one now in force, so answers come from material that has been replaced. is the failure this subject deserves to be known for, and the thing worth understanding about it is that there is nothing to find. Every part of the arrangement is working. The store is answering the question it was asked, on the material it was given, and the material it was given is the material somebody loaded.

Who is accountable for keeping it current, and why is that the hard part?

A question about who is accountable for the procedure gets a name. A question about who is accountable for the arrangement gets a name. A question about who is accountable for the eleven days between the two gets a pause, then a sentence with no subject in it, of the shape it gets loaded periodically. The gap lives in the space between two named jobs. The space between two named jobs is the one place an organisation chart cannot show anything.

The same shape appears in a household. The person who buys the medicines is named. The person who takes them is named. Nobody is named for noticing that the strip in the drawer expired, and so the strip in the drawer expires. Nothing about the buying went wrong and nothing about the taking went wrong.

The repair is dull, and worth naming for exactly that reason. Somebody is made accountable for the join, the loading runs on a stated rhythm that the desk knows, and the store can report the age of what it holds so a person can look. None of that is a change to the retrieval, and no amount of improving the retrieval substitutes for it.

What is a vector store not, however it is described?

Three things get attached to it in conversation, and each one costs a deployment something different. A vector store is not a search engine that knows the answer: it returns what is near, and deciding which of the six actually answers the question happens somewhere else entirely. The store is not a memory: nothing it holds arrived because anybody said it, and it changes only when somebody loads something. And it is not a record of what is in force: it is a record of what was loaded. The second claim wears the shape of the first and says something else.

Every one of those three mistakes ends in the same place. Somebody treats the first passage returned as an answer. The store put it first because it sat nearest. Nearest is the whole of what being first means, and being nearest is not nothing, but it is a very long way from being the passage somebody was looking for.

What do three different people do with any of this?