The Context Window: How Much the Model Can Hold at Once

The context window is how much material a component can be given for one output, instruction and supplied documents together. The behaviour past the limit is the important part: material is dropped rather than refused, the component produces a normal-looking output from what remains, and nothing in that output records that anything was missing.

The limit itself is an arrangement that keeps moving, and it is not worth learning. The behaviour at the limit does not move at all. Something has to be left out, whatever leaves it out does so without the component knowing it has done so, and the output is then produced from whatever arrived. Every deployment decision that follows comes from those three sentences and not one of them depends on the number. The number used below belongs to one invented bank and describes that bank's own arrangement.

What does the window actually limit, and what does it not?

Start with the thing it is easy to get wrong. The context windowHow much material a component can be given for one output, counting the instruction and the documents together. is not a measure of how much a component knows. The window is not a store, it is not a memory, and nothing put into it on one request is there on the next. The window is a limit on the size of one request, and the request is thrown away when the request is finished.

A photocopier with a feed tray works the same way. The tray does not know anything and does not remember the last job. The tray holds what can physically be stacked on it for this one copy run, and if sixty sheets arrive at a tray that takes forty, the tray will not say so. The window is the tray, not the filing cabinet, and confusing the two is what makes people expect the component to notice that something is missing.

The material that goes on the tray for one request has two parts. The first is the instruction. At Sumeru Bank Limited, an invented lender, the instruction runs to 340 words in six numbered parts and is supplied again with every single request. The second is the supplied materialThe documents provided for this particular request, as distinct from the standing instruction that comes with every request.. At this bank the supplied material is the file behind one exception noteThe written record a desk officer makes on a file that stopped, saying what stopped it and what the officer is being asked to decide.: the application, the documents, the statement lines, the identity and consent record, and whatever correspondence has since built up on it. Component 8 of the intake chain, the drafting assistant, produces the first draft of that note and a person reads, corrects and signs it. The drafting assistant runs on 3,010 of those notes a month.

Why do the instruction and the file compete for the same space?

Because there is one tray. Everything supplied for one output is counted together, so the instruction is not free and never was. At this bank the deployed arrangement held about 12,500 words in total, and the instruction takes 340 of them, so the room left for the file is 12,160 words. The accounting is that short, and it is worth writing down because almost nobody does.

Now watch what a longer instruction costs. Suppose the six numbered parts grow to 900 words. A 900 word instruction is not wild, and it is the sort of growth a year of small improvements produces. The limit has not moved. The room for the file has fallen by 560 words. Read off this bank's own measured distribution, the loss of 560 words moves the share of files fitting entirely from 95.5 per cent to about 94.9. A drop of 0.56 of a percentage point sounds like nothing until it is multiplied by the month: about 17 more notes a month drafted from a file the component never saw the whole of. Both of those are ordinary arithmetic on this bank's measured points rather than a second measurement.

What happens when the material does not fit?

Everything turns on one sentence. The request is not refused. The oversized part is dropped, the request proceeds on what is left, and a perfectly ordinary output comes back.

How strange that is deserves a moment. Almost every other size limit announces itself. A form field stops accepting characters. A parcel over the weight is sent back at the counter. An upload larger than the allowance fails with a message. Those refusals are all rude, and rudeness is the useful property: the failure becomes known. Silent truncationMaterial dropped because it did not fit, producing no error, no warning and no mark anywhere in the output. is the polite version, and politeness here is the whole problem. A limit that refuses is a problem visible on the day it happens, and a limit that drops is a problem somebody finds months later by accident.

The everyday version is a photocopier that runs out of paper at sheet forty of a sixty sheet file, quietly stops feeding, and delivers a neat stapled set. Nothing on the set says forty. The stapled set looks like a document. The missing twenty sheets turn up at the meeting, if they turn up at all.

Why does nothing in the output record that anything was missing?

Two separate reasons, and holding them apart is what makes the fix obvious later.

The first is that the component has nothing to report. The component was handed a quantity of material and it worked on the material it was handed. The component has no view of what was assembled before the cut, so it cannot say the file was shortened any more than a reader of the copy can say which sheets the photocopier declined to feed. Asking the component to flag it is asking the wrong party.

The second reason is the one that catches careful firms out, and it is worth slowing down for. The bank does check its drafts. Verification of an exception note is five numbered checks: every figure traced back to the supplied material, every assertion traced to a passage, nothing asserted that the material does not contain, the required format and required statements present, and the decline-to-answer case handled properly where it arises. Five checks is a serious list, and check 3 is the one that catches the unsupported statements. Every one of the five compares the draft against the material that was supplied, and the material that was supplied is already the shortened version, so not one of the five can see a drop. A truncated draft passes verification because it is, strictly, correct: everything in it is supported by everything that arrived.

The everyday version is a proofreader given a photocopy and asked to check it. Her whole job is defined against the copy rather than against the original. She will find every typographical error on the sheets in front of her, and she will never find the twenty sheets that did not come out of the machine.

Which part gets dropped, and why is the oldest part the worst one to lose?

Material is assembled in the order it arrived, oldest first, and when the pile stops fitting it is the oldest end that goes. Dropping from the oldest end is the ordinary arrangement, and it was the one deployed here. The arrangement sounds harmless. Recent things are more relevant, surely.

Recent things are not more relevant, on a loan file. Think about what is actually at the old end of an application. The application itself is there: the amount asked for, the tenor, the declared income, the stated purpose. The first set of documents is there, and so is the record of which of them was later replaced. Everything that came afterwards is a response to those things. Dropping the oldest part of a loan file removes what the customer asked for and keeps the correspondence about it. When the task is to write a note about the case, that is precisely the wrong half to keep.

The shopkeeper's version makes it plain. A shopkeeper sends the accountant only the last two sheets of the daybook because that is all that fits in the envelope. The accountant can reconcile those two sheets perfectly. Every entry is real, every total adds, and the work is genuinely correct. He has simply never seen the opening balance, so the statement he produces is arithmetically perfect and describes the wrong business.

How often did this happen on one bank's real files?

Once, somebody counted. In the first measurement of the drafting assistant at Sumeru Bank Limited, 200 exception notes were reviewed one by one. Nine of them, being 4.5 per cent, had been drafted from a file that had been quietly shortened. Nine in 200 is the only reading this bank has, and it describes one bank's own arrangement rather than a property of anything.

Scale it and it stops being a rounding error. At 3,010 exception notes a month, 4.5 per cent is about 135 notes a month written from a partial recordA file the component saw only part of, without knowing that a part was missing.. If each of those has to be written again properly by a person, at the 18 minutes a note took before the component existed, that is 2,430 minutes a month, being about 0.29 of a post on this bank's assumed 8,400 minute working month, or roughly Rs 2,60,000/- a year against its assumed fully loaded cost of Rs 9,00,000/- a year a post. Every one of those amounts is arithmetic on the bank's own figures, not a cost anybody at this bank invoiced.

The count is not the lesson. Now look at one of the nine. Take a file stopped by the income corroboration rule, where the declared monthly income is Rs 45,000/- and the median monthly salary credit over three months of statement corroborates Rs 38,000/-. The gap of Rs 7,000/- is 15.6 per cent, and the gap is what routed the file. The declared figure sits on the application, at the old end. The statement lines and the correspondence sit at the new end. A note drafted after the cut names the corroborated Rs 38,000/- accurately, never names the declared Rs 45,000/-, and therefore never names the gap that stopped the file at all.

The truncation was found in the end, and not by reading drafts. Drafts had been read all along. Somebody put one note next to the application it was supposed to be about. The reading of a note is exactly what has been made to look fine, so no amount of reading can reveal a truncation. One more uncomfortable fact sits beside that one: the second measurement, in month 10, counted unsupported statements, not truncated files. The bank has one reading of this rate and no second one, and none of the three changes made between the two measurements addressed the limit.

Were those files a random sample of the month?

No, and this is the part worth carrying away even if everything else is forgotten.

A file exceeds the limit for reasons, and the reasons are all the same reason. The file has been going on longer. The file carries more correspondence. The file has been escalated. The applicant has been back three times and each visit added something. Every one of those is a description of a complicated case, and a complicated case is exactly the one where a careful, complete note matters most. A limit that silently drops the largest cases is not sampling at random; it is selecting for complication and then failing precisely there. Cathy O'Neil, in Weapons of Math Destruction, makes the general version of this point about model errors falling unevenly across a population, and a size limit is a blunter instrument than a model.

The courier's version is exact. A van has a weight limit, so the heaviest consignments do not travel today. Nobody chose to delay the biggest customers. The scale chose, and the scale has no view about customers at all. Come back at the end of the month and the pattern of complaints is entirely explained by an arrangement nobody would have written down as a policy.

Which is why 4.5 per cent is not an acceptable residue once the people in it are named. About 135 people a month applied for money, spent longer on it than most, came back when they were asked to, and had a note written about them by something that never saw what they had asked for. The bank did not tabulate which of the six exception causes those nine fell into, so nothing more precise can be said about them, and that gap in the record is itself part of the finding.

Does a larger limit solve it?

Less than might be expected, and the shape of the answer is the useful part. The bank measured the share of its own files fitting entirely at nine different limits, and the readings rise steeply and then flatten hard.

Read the two doublings against each other. Going from 6,000 words to 12,000 moves the share from 70.0 per cent to 95.0, a gain of 25.0 points for 6,000 more words, being about 4.2 points for every thousand words added. Going from 12,000 to 20,000 moves it from 95.0 to 99.6, a gain of 4.6 points for 8,000 more words, being about 0.6 points a thousand. The same words bought about seven times as much on the first stretch as on the second. Because 100 is the ceiling, doubling again from 12,000 to 24,000 cannot possibly add more than 5 points.

So the honest reading is that a larger limit is genuinely worth having up to a point, that this bank was already past that point, and that no achievable limit takes the residue to nothing. The tail does not end; it thins. And while the wait goes on, the notes that get written from part of a file are the same complicated ones every month.

What does a deployer do instead of waiting for a larger limit?

Three things, in an order that matters, and none of them is waiting.

First, make an overflow raise an exceptionStop the request deliberately and tell somebody, instead of carrying on with whatever fitted. rather than pass quietly. Before the material is sent, its size is measured. If it will not fit, the request stops and produces a refusal that names the file rather than a note that does not. Making the overflow loud is the first move because everything else is easy once the failure is visible and nothing else is possible while it is not. The change also converts a silent, permanent, unmeasured problem into a loud, dated, countable one, and that is the trade every operational control on earth is making.

Second, measure what is actually being supplied. Somebody else sets the limit and the desk cannot change it, so the number to log is not the limit but the size of the material on every request. One logged number gives the distribution in the table above, and the distribution shows whether the desk is at 95.5 per cent or 70.0, and whether last quarter's growth in correspondence has quietly moved it.

Third, cut it deliberately. If something has to be left out, a person should choose what, once, in advance, and the output should say what was left out. Deciding that the middle section of correspondence goes and the application never does is a five minute conversation with a desk head. Letting the arrangement decide, by dropping whatever happens to be oldest, is the same decision made worse and made invisible.

What should be asked of whoever supplies the arrangement?

Three questions do the work outside a bank as well as inside one. Each is short and each answer is diagnostic.

The first question is what happens when the material does not fit. The limit itself will have moved by the time of deployment, so the question to ask is about the behaviour at the limit and not about the size of it. If the answer is that the request fails, that is a problem a desk can operate. If the answer is that it is handled automatically, the follow-up is what handled means, asked again and again until somebody says the word dropped.

Ask what is recorded when it happens. A size written to a log on every request is the whole of what is needed, and its absence is the reason this bank had one measurement and not a monitored number. An arrangement that cannot say how often it dropped material last month is not being monitored on the one thing that fails silently.

Ask who chose the order in which material is dropped, and on what basis. If the honest answer is that nobody chose and the oldest simply goes first, then a technical default is deciding what a note about a customer is allowed to be about. Choosing what gets dropped is a decision belonging to a desk head, and it takes five minutes to make properly.

Ismail Sheikh, who heads the exception desk, is the person these questions actually serve. His interest is not in the limit at all. His interest is in knowing that about 135 notes a month will need writing again from the beginning, so he can put the hours in a roster rather than discovering them as a quality problem eleven months later. Neelima Rao, doing the independent validation, wants the same number for a different reason: a sample of drafts drawn from a period when an unknown share came from partial records is a sample she cannot interpret. Revathi Balan, accountable for what goes into a decline letter an applicant reads, needs only one thing from all this: the applicants affected are systematically the ones with the most complicated cases.