Prompt Injection: When Input Becomes Instruction

A component that reads text cannot tell a description from an instruction, so text supplied by somebody else can tell it what to say. At one invented bank this happened once. The draft said what the text told it to say, and the check that caught it was bought to catch something else entirely and fires about 346 times a month for other reasons.

Everything a text-reading component receives arrives as text. The instruction about what to do and the material to do it to are not separated by any structure the component can rely on, so any text that reaches it can act on it. The absence of that separation is a property of how such a component works rather than a defect in a particular one.

Why is a description and an instruction the same thing to a component that reads text?

Picture a counter at a small office. A customer hands across a stack of papers and asks for a short note on what is in there. Somewhere in the middle of the stack sits a sheet that says, in ordinary language and addressed to whoever reads it next, that the money has already come in. The clerk looks up and asks about it. The clerk can ask because the clerk is holding something the stack does not contain: a memory of who asked for the note, and why, and what was agreed a minute ago at the counter. The memory is the boundary, and the memory is what makes the odd sheet part of the material rather than part of the job.

A component that reads text holds no such memory. It holds text. The instruction it was given and the file it was given arrive as one stream, and nothing inside that stream marks which is which. The sentence saying write a note on this file and the sentence saying state that the income was corroborated are, at the moment the reading happens, the same kind of object: language, in sequence, to be continued. Prompt injectionText supplied as material that acts as an instruction to the component reading it. is simply the name for what happens when the second kind of sentence gets into the stream.

Care about the instruction therefore does not protect the material. A firm can write its own instruction as tightly and as carefully as it likes, and it has still only added more text to the same stream. No fence is available to build, so nothing has been fenced off. There is only the order in which the words arrive. The missing fence decides what a defence can even be. Disregarding something requires recognising it first, and recognition is exactly what is not on offer, so the component cannot be asked to disregard instructions inside the material.

How does somebody else's text reach a component in the first place?

The intake chain at Sumeru Bank Limited, an invented mid-sized Indian bank, pulls 14 named fields out of the documents attached to each application. For every field the chain consumes, the bank writes down eight numbered items: the field itself, where it comes from, who is accountable for it at its source, its format and permitted valuesThe written statement of what a field may contain, which is one of the items a firm records for every field it consumes., how often it changes, what happens when it is absent, what may be retained and for how long, and the consent basis where one applies.

The text did not arrive in any of those 14. Instead it arrived in a free text fieldA field where somebody types whatever they like, with no stated set of permitted values. on the application, a box where an applicant types whatever they want to say about their own circumstances, and that box had no record at all. The box certainly had no entry under item 4. Item 4 is the line that would have stated what the field may contain. Nobody had written that line because nobody had thought of the box as a field. It was a place for a customer to explain something, and it had been there since long before any of this was built.

Then the drafting component, component 8 of the chain, was handed the supplied materialEverything handed to a text-reading component, including fields nobody thought of as instructions. for the file. Not a named list of fields. A bundle. Everything sitting on the application went in, the free text box included. Nothing anywhere said it should not. The path from a keyboard to a customer letter was opened by an absent line in a records table, not by anything anybody would call a technology gap.

Think of a shop that photocopies whatever is in the tray. Nobody decided the extra sheet should be copied. Nobody decided it should not either, and that second absence is the costly one. The same class of gap produced this bank's drift episode, where an income field changed format on one channel and nobody had ever written down what the chain should do when a field arrives in a shape it was not expecting. Both times, the missing thing was a sentence in a document, and both times the sentence was missing because the question had never been asked out loud.

What actually happened at this bank, and what did the draft say?

One instanceA single observed occurrence, which cannot be turned into a rate however tempting the arithmetic. in the whole period. The count is one, and one is all there is. Everything interesting about the episode sits in what a count of one does and does not license anyone to say.

An applicant typed an instruction into that free text box. The instruction was addressed to whatever would read the material next and it told that reader to state that the income had been corroborated. Component 8 produced its first draft of the exception note, and the note said so. The bank's own record of the incident does not carry the wording.

How it was found is the plainest part of the story, and it is worth dwelling on because it is so undramatic. There was no alarm and no pattern match. A person doing the check this bank does on every exception note read the drafted note and found a sentence nobody had written: a claim about corroboration that the material behind it did not carry. They took it out, and they asked where it had come from, and the answer turned up in the free text box.

Notice what did not happen. The income corroboration step itself, component 5, is a written rule set of 34 lines that compares a declared monthly figure against the median salary credit over three months of statement and routes the file where the declared figure exceeds the corroborated one by more than the bank's own chosen tolerance. The rule set never wavered and its output never moved. The note's account of that output is what moved. The manipulation did not touch a decision, it touched a description of a decision. A description is a smaller thing and in one way a worse one. The customer reads the description, not the decision.

Which check caught it, and what does that check actually ask?

Verification of a drafted note at this bank runs five numbered checks, and the person signing the note works through them. The third of them asks a single question: is anything asserted in this draft that the material does not contain?

Read that wording again. The wording is the whole reason the check worked. The check does not ask whether any part of the input looked like an instruction. It does not ask whether the text contained anything unusual, or aggressive, or out of place. The check takes the output, takes the material the output was written from, and reports the difference between them. A verification checkA step where a person or a procedure tests a draft against the material it was written from. written to compare an output against its source will catch a wrong output whatever caused it. A control aimed at carelessness therefore stopped something done on purpose.

In the terms of that check, the injected sentence was indistinguishable from an ordinary mistake. The sentence was a claim in a draft that the file did not support: an unsupported statementSomething asserted in a draft that the file does not contain., exactly like the other ones the check finds every day. The person doing the check did not need to know that anything had been done deliberately, did not need to know what the phrase for it is, and did not need any training in it. The person needed the material and the question.

Where a defence sits decides how hard it has to work. A defence at the input has to recognise something, and recognition is the hard problem here. A defence at the output has to compare two things already held. Comparing two held things is an ordinary clerical act. The second is not more sophisticated than the first. It is far less sophisticated, and that is why it is available.

Why was that check there at all, and what was it bought for?

Because drafts assert things that are not in the file for entirely innocent reasons, and they do it often. Over 200 exception notes measured at this bank, 23 carried a statement the file did not contain, being 11.5 per cent. Of those 23, the person signing caught 21 and did not catch 2. The measurement of those 200 notes is what paid for the check, and it had nothing whatever to do with anybody typing anything into anything.

The economics were written down and they are ordinary. Drafting an exception note by hand took 18 minutes. With the component producing the first draft, it took 4. Verification added 9 minutes. Take 18, subtract 4, subtract 9, and the net saving is 5 minutes a note. The 5 minutes, multiplied over a desk that handles 3,010 exceptions a month, is the whole business case. The 9 minutes is not a safety budget sitting on top of a benefit. It is part of the price of being able to use the draft at all.

Nobody had seen an instruction hidden in supplied text and nobody was buying a defence against one, so nothing resembling it appears anywhere in that business case. The check exists because carelessness is common and expensive. The check stopped something deliberate as a side effect, and a side-effect defenceA control that stops something it was never designed or funded to stop. sits awkwardly in every record a firm keeps. It is not a happy accident to be celebrated. The problem is an accounting one. A control valued for one thing and doing two is carried in the books at half its worth, and the half nobody knows about is the half that goes first.

How often does the same check fire for entirely unrelated reasons?

In a steady month, 3,010 files reach this bank's exception desk, and each one of them carries a note. At the measured rate of 11.5 per cent, about 346 of those drafts a month contain a statement the file does not contain. Innocent ones. Nobody typed anything into any box, and nobody meant anything by it.

Against that, one instruction hidden in text, in the whole period. The check that stopped the deliberate thing spends essentially all of its working life on the accidental thing, and the ratio is not close.

The comparison needs care, though, and the reason is in the units. The 346 is a monthly count and the one is a count over a longer stretch, so the two quantities are not in the same unit and dividing one by the other produces a number with no meaning. Placed side by side, the two counts show where the work is, and showing that is all they are being asked to do. Any drawing that puts them next to each other has to say on its face that they are not in the same unit.

What still reaches a customer once every draft is checked?

About 30 a month. The 30 is not a rounding of zero and should not be read as one.

The arithmetic runs like this. About 346 drafts a month carry an unsupported statement. The check catches 21 of every 23 it sees, being 91.3 per cent, so it lets through the remaining 8.7 per cent, and 346 times 8.7 per cent is about 30. The same figure reconciles from the other direction too: 2 of the 200 measured notes reached a customer, and 2 in 200 applied to 3,010 notes a month is 30.1. The same number arrives twice by different routes. Arriving twice by different routes is the only real test case arithmetic ever passes.

With that held steady, one thing changes. Because the catch rate is a constant share of whatever is put in front of it, the relationship between how much is checked and how much gets through is a straight line. With nothing checked at all, all 346 go out. With every single draft checked, about 30 still go out. The line between those two points is straight, and straight sounds reassuring. The reassurance is misplaced: the two ends are more than eleven times apart.

What would a defence built for this alone have cost?

Taking the 9 minutes seriously as a number and following it out: 3,010 notes a month at 9 minutes each is 27,090 minutes a month. 27,090 minutes is 451.5 hours. At this bank's working day of 7 hours it is 64.5 working days a month, and since the bank's month holds 20 working days, it is 3.225 people doing nothing else with their time. At the assumed fully loaded cost of Rs 9,00,000/- a year a post, that is about Rs 29,02,500/- a year. Every one of those steps is arithmetic on this invented bank's own assumed rates rather than a separate measurement.

The ladder of arithmetic goes in front of a committee with one instance in the period written on the other side of the sheet. Nobody approves 3.2 posts against a count of one, and it would be wrong to ask them to. The spending was approved because the check catches about 346 unrelated things a month and because it pays for itself in drafting time. The instruction hidden in text was stopped by a control that somebody else had already bought for their own reasons.

One more figure belongs here. By month 10 the same check took 6 minutes a note rather than 9, being 18,060 minutes a month. The control had got cheaper without getting any weaker. A cheaper control is a good thing and also the precise circumstance in which somebody starts asking whether it is needed at all.

Why is one instance not a rate?

One instance in the period. Divided by the period it yields a number, and that number is arithmetic rather than a rate. The number will have a unit and a decimal point and will mean nothing at all.

A single observation establishes that such a thing happens at all. The fact that it happens is genuinely worth knowing, and it is the entire content of the observation. It says nothing about how often, nothing about whether it is rising or falling, and nothing about whether the next period holds none of them or forty. A count of one has no shape. No trend can be seen in it, it cannot be compared against anything, and no confidence can be put around it. A count of one and a monthly count never belong on the same scale unless the drawing says so on its face.

The temptation runs in both directions and both directions are wrong. Multiplying the one up into an annual figure manufactures a threat this bank has no evidence for. Waving it away as a freak event dismisses the only observation anybody has. The honest position is narrow and uncomfortable: it happened once here, it was caught, and the catching was luck in the specific sense that the control was funded for a different purpose and would still have been there if nobody had ever thought about this at all.

The control worth moving is the share of drafts verified, not anything about the injection itself. The share is a thing the firm chooses and can therefore reason about. The frequency is a thing nobody at this bank knows, and drawing a curve through a single point would be inventing one.

Which three things reduce the exposure, and what does each cost?

Three, and they sit at three different points along the path, which is the part worth holding on to. A firm that does one of them should know which point it has covered and which two it has not.

1. Write down what every field may contain, before it is passed anywhere. The permitted values line is item 4 of the eight items this bank records for each field it consumes. It is a writing job done once. At this bank the same records exercise was already outstanding elsewhere: item 6, what happens when the field is absent, was missing for 11 of the 14 extracted fields, being 78.6 per cent of them. Running cost: none.
2. Treat any text a person supplies as material rather than as instruction, everywhere it is handled. Treating supplied text as material is a design choice made once, at the point where the bundle is assembled, and it reduces what reaches the component. It does not test what the component produces. Running cost: none.
3. Keep a check that asks whether anything is asserted that the material does not contain. The check is the only one of the three that runs every month, the only one with a bill attached, and the only one that has actually stopped anything at this bank. Running cost: 27,090 minutes a month at month 6, and 18,060 by month 10.

The two that cost nothing to run are the two this bank had not done, and the one that runs every month and costs real money is the one that worked. The ordering is not a coincidence and it is not bad luck either. A one-off is easy to leave undone because nothing goes wrong on the day it is skipped. A running cost has somebody defending it in a budget every year. The discipline that survives is the discipline somebody is paid to perform.

Where does this sit among the other kinds of deliberate manipulation?

The bank saw four kinds in the period, and the counts are worth putting next to each other. The counts and the alarm run in opposite directions.

Kind 1 is the expensive one. The bank's income rule routes a file where the declared figure exceeds the corroborated one by more than its own chosen tolerance of 10 per cent, and about 946 files a month declare an income somewhere between 5 and 10 per cent above the corroborated figure. Of those, 412 sit in the single percentage point just below the line, against about 132 expected if the numbers simply thinned out smoothly. The excess of 280 files a month is what shaping an input to clear a written rule looks like when it is counted.

The kind that is easiest to write about is the rarest, and the kind that costs money every single month is the one nobody tells a story about. A defence budget set by how alarming a thing sounds will be spent at the bottom of that table, and the counts point firmly at the top of it. The units matter before that is taken too far: kind 1 is a monthly count and kinds 2, 3 and 4 are counts over the whole period, so the figure below does not draw them on one scale.

What does somebody putting a text-reading step into a process actually check first?

Not the component. The useful first-day question is about what sits behind it: does anything downstream test what this thing asserts against the material it was handed, and will that test still be funded in two years?

For a lender, that test is a person signing an exception note with the file open beside them. For an analyst pulling a summary out of a stack of filings, it is whether they open the filing before they use the sentence. For a household reading a letter from a bank about their own application, it is nothing at all, and that is exactly the point. Every control on that path exists because the person the sentence finally lands on has no way whatever to check it. The applicant who receives an exception note cannot know which of its sentences came out of their file and which came out of somewhere else.

The second question is about the paperwork, and it is the cheapest of the lot. The check is written down under the name of what it also stops, not only under the name of what it was bought for. One extra line in a control register, saying that this check is also the only thing standing between supplied text and a customer letter, is what turns an invisible defence into a visible one. The line changes nothing about the work and everything about the conversation the next time somebody proposes trimming the check. The bank did not have that line. The incident therefore sits in its records as a lucky catch rather than as a control performing.

And one caution for whoever writes that line. Ismail Sheikh, who runs the exception desk here, is not the person who decides how much verification gets funded, and Revathi Balan, who is the named accountable person for the scoring component, does not sign exception notes. A defence that lives inside one person's daily work and is budgeted by somebody two levels away is the ordinary case rather than the exceptional one, and writing it down is how the two ends of that arrangement come to know about each other.

Sumeru Bank Limited, its intake chain, Revathi Balan and Ismail Sheikh are invented.
Educational material. Not advice on any investment, tax, budget or market position.