Simulation: Generating Outcomes You Cannot Observe

A simulation generates outcomes nobody observed, by applying a stated rule many times over and counting what comes out. Monte Carlo simulation draws each outcome from a stated shape. Historical simulation replays a record already held. The fifty month record of the Nakshatra unit contains four twelve month stretches and no more. Ten thousand drawn paths cost nothing. Neither answer is the truth.

Three ideas carry all of what follows, and nothing else is needed. The first is that a number between zero and one can carry a claim about how often something happens, and that a list of possible values with a number attached to each of them is a complete description of what can occur. The second is that a figure worked out from a sample sits some distance from the quantity it is estimating, and that the distance is a thing that can be given a size. The third is ordinary multiplication. A twelve month path is twelve multiplications and nothing else.

One invented traded unit runs through everything below: the Nakshatra unit, priced once a month, currently sitting at Rs 100/-. A traded unit here just means an object carrying a price that changes month by month, and nobody could buy this one. Alongside it sits the fifty month record: fifty months of actual monthly changes, already collected, already counted. The whole treatment turns on one question those two objects cannot answer between them, and on the two different ways of manufacturing an answer anyway.

What is a simulation, and what is it actually for?

Here is the situation a simulation exists for. Somebody wants to know how often something happens, the something takes a while to happen, and there is neither the time nor the patience to sit and watch it happen enough times to count.

Take a food stall outside one office building. On the evenings when the till runs out of small change before six, customers walk off. The person running it wants to know how often that happens. Forty evenings have been watched and the change ran out on six of them. Six out of forty is not nothing, but it is a thin number to reorganise a business around, and getting to four hundred evenings means waiting another four years. The stall owner does not have four years. A rule is available instead: roughly how many customers arrive, roughly what they pay with, roughly what the till starts with. Apply that rule to a made up evening and see whether the change runs out. Then do it again. Do it four hundred times over a cup of tea, and count the evenings that failed.

A made up evening counted four hundred times is a simulation, and the finance version is the same shape with different nouns. A simulation generates outcomes that were never observed, by applying a stated rule many times over and counting what comes out. One half of that sentence carries the answer, and the counting is the half. The drawing is only the means of producing something to count, and no single generated outcome is a forecast of anything at all. One made up evening tells the stall owner nothing. Four hundred made up evenings, totalled, tell the stall owner what share of evenings the rule says will fail.

The rule is doing all the work, and the rule is an assumption. The rule is where simulations actually fail. A simulation cannot manufacture information. A simulation takes whatever it was fed and works out the consequences of it far more thoroughly than any hand calculation could, and if what it was fed was wrong then it works out the consequences of something wrong, very thoroughly indeed, and returns a number with four decimal places on it.

Why can the fifty month record not answer the question directly?

The question carried all the way through is a small one. The Nakshatra unit is priced at Rs 100/- today. What is the chance it is below Rs 100/- twelve months from now?

The obvious move is to go and look. The fifty month record of the Nakshatra unit holds fifty months of monthly changes, so counting the twelve month stretches in it that finished lower and dividing looks like the whole of the work. The problem announces itself immediately. Twelve into fifty goes four times and leaves two months over. Fifty months hold four twelve month stretches that do not reuse a month, and four is not enough to put a number on anything. A count out of four can only ever come back as none of them, one of them, two of them, three of them or all four, and there is nothing in between.

Think about what that means for a household deciding anything on the strength of it. A household running on one salary that has seen four Januaries knows something about January, but it does not know the chance of a bad January to two decimal places, and no amount of staring at those four Januaries will produce one. The fifty month record has exactly the same shape of problem. The record is not bad. Fifty months is a perfectly respectable amount of history. Twelve month questions simply eat history twelve months at a time, and fifty months of it goes almost nowhere.

So the outcomes have to come from somewhere other than observation, and there are exactly two places they can come from. Monte Carlo simulation states a rule and generates outcomes from it. Historical simulation takes the record already held and replays it. Both are set out in full below and then held up against one another. Knowing which of the two produced a number is the difference between reading it correctly and reading it wrongly.

What is Monte Carlo Simulation, and where do its outcomes come from?

Monte Carlo simulation manufactures outcomes by drawing them, one at a time, from a shape somebody wrote down first. The shape for the Nakshatra unit is this: the monthly change takes exactly five values, minus 9.00 per cent, minus 4.00 per cent, 1.00 per cent, 6.00 per cent and 11.00 per cent, and each carries a weightThe number pinned to each possible value saying how large a share of the time it turns up, with all of them adding to one. saying how often it comes up: 0.08, 0.18, 0.48, 0.18 and 0.08. Adding the weights up in order gives the running totals 0.08, 0.26, 0.74, 0.92 and 1.00. Those five running totals are what turns the shape into something that can be drawn from.

Now the mechanical part, and it really is mechanical. A number between zero and one is obtained. Walking along the running totals until the first one the number is below gives the value sitting there. The walk is repeated twelve times. The twelve values are multiplied onto Rs 100/- one after another, which is compoundingEach change is applied to the price the month before finished on, so twelve of them multiply together instead of adding up., and that is one twelve month path. The whole thing then runs again, and again, ten thousand times over.

Where does the number between zero and one come from? From an arithmetic rule, stated in full so that every figure can be reproduced. The rule starts with a whole number, multiplies it by 16,807 and takes the remainder on 2,14,74,83,647. The remainder is the new whole number, and dividing it by 2,14,74,83,647 gives a value between zero and one. Then the step repeats. The whole number the rule starts from is called the seedThe number a generating rule is started from. Where a computer's numbers really come from, and what happens when the starting number changes, is covered separately., and every figure below was produced from the starting number 12345.

The shape is an assumption and the drawing is mechanical, and it matters enormously that those two are not the same kind of thing. Somebody chose five values and five weights. The choice of five values and five weights can be wrong, and if it is wrong then every path drawn from it is wrong in the same direction. The drawing, by contrast, cannot be wrong: it is multiplication and division, and it does exactly what it says.

Path one, written out so that every line of it can be checked

Here is the first path the rule produces from the starting number 12345, with nothing chosen by hand. The first value it gives is 0.096617, above 0.08 and below 0.26, so it lands in the second band and reads minus 4.00 per cent. Rs 100/- becomes Rs 96.0000/-. The next value is 0.833995, above 0.74 and below 0.92, so it reads 6.00 per cent, and Rs 96.0000/- becomes Rs 101.7600/-. And so on for ten more months.

The very first path this rule produces is one of the falling ones, and that is worth noticing precisely because it means nothing. Path one is not evidence about the Nakshatra unit. Path one is one line in a tally that is going to run to ten thousand lines, written out in full only so that the machinery can be confirmed to do what it claims. The starting number fixes the stream before the first path is drawn, so those twelve multiplications, run by hand, return Rs 99.2786/- today, tomorrow and in ten years.

Do it ten thousand times and 2,453 of the paths finish below Rs 100/-. Two thousand four hundred and fifty three out of ten thousand is 24.5300 per cent, and that is what Monte Carlo simulation says the answer is. Ten thousand paths took no longer than one path took to describe, and the ten thousandth cost exactly what the first one cost.

What is Historical Simulation, and how many paths does it give?

Historical simulation refuses to state a shape at all. Instead it takes the record already held, in the order it happened, and reads twelve month stretches out of it. Each stretch is a path. No assumption is made about how often anything happens. The months are just replayed.

Doing that on the fifty month record of the Nakshatra unit needs the record in an order, so here is the order, and it is not new information. The fifty monthly changes were already fixed: five months at minus 9.00 per cent, nine at minus 4.00 per cent, twenty five at 1.00 per cent, eight at 6.00 per cent and three at 11.00 per cent. The order they came in was never fixed. The order used here is produced by the same arithmetic rule from the starting number 4321, by walking from the last month back to the second and swapping each position with an earlier one chosen by the rule. The swapping rearranges the fifty values and alters not one of them, so the record still has the same mean, the same spread and the same counts it always had.

Reading the four stretches gives four ending prices: Rs 100.5078/-, Rs 129.3336/-, Rs 105.3002/- and Rs 96.0274/-. One of the four finished below Rs 100/-. One out of four is 25.0000 per cent, and that is what historical simulation says the answer is.

No assumption about shape was made anywhere in that, and the price of making no assumption is having four paths. That is the trade, stated as plainly as it can be. Monte Carlo simulation bought ten thousand paths by agreeing to accept somebody's stated shape. Historical simulation refused to accept anything and got four.

There is a tempting way to get more paths out of the same fifty months, and it is worth naming so it can be dismissed. Instead of cutting the record into four separate blocks, slide a twelve month window forward one month at a time. Months 1 to 12, then 2 to 13, then 3 to 14, all the way to months 39 to 50. The sliding window gives thirty nine stretches instead of four, of which eight finish below Rs 100/-, so the answer becomes 20.5128 per cent. But every one of those is an overlapping windowA stretch of months that shares most of its months with the stretch beside it, so the two are not separate evidence.: months 1 to 12 and months 2 to 13 have eleven months in common. Thirty nine of them are not thirty nine pieces of evidence, and the denominatorThe count sitting underneath a share, which decides what the share is a share of. of thirty nine is a promise the record cannot keep. The figure it produces is further from the exact answer than the four path one, not closer.

One more thing before leaving 25.0000 per cent. A reader who has been counting will have noticed it. Twenty five of the fifty months in the record hold 1.00 per cent, and the historical answer reads 25.0000 per cent. Nothing whatever connects those two numbers. One is a count of months and the other is a share of four paths, and the two twenty fives are a coincidence of this particular record.

Monte Carlo Simulation vs Historical Simulation: what is each one blind to?

Both methods now stand defined in full, and only then is a comparison honest. Four questions separate them, and the fourth is the one that matters most.

Start with where the outcomes come from. Monte Carlo simulation draws from a shape somebody stated: five values, five weights, written down before a single path exists. Historical simulation replays a record that already exists: fifty months that happened, read in the order they happened in. The first is an assumption, the second is a measurement, and they fail in completely different ways.

Then how many paths are available. Monte Carlo simulation gives as many as there is patience to wait for, and the ten thousandth costs what the first cost. Fifty months hold four twelve month stretches, so historical simulation gives four, and no amount of computing produces a fifth. Buying a faster machine buys no fifth stretch.

Then what a wrong input does. Monte Carlo simulation is wrong in exactly the way its stated shape is wrong, and historical simulation is wrong in exactly the way its record is unrepresentative. If the true chance of a bad month is higher than the stated 0.08, every one of the ten thousand paths understates the danger, in step, and running a hundred thousand instead makes the wrong answer more precise rather than less wrong. If the fifty months happened to be an unusually calm fifty months, the four stretches inherit that calm, and there is nothing to rerun.

And then the fourth question, the one about what each method cannot see. Monte Carlo simulation cannot see anything the stated shape does not contain. The drawing only ever reaches into the five bands that exist, so a sixth value that nobody wrote down can never be drawn, no matter how many paths are run. Historical simulation cannot see anything that has not already happened. Fifty months cannot show a month unlike all fifty of them, and the more unusual the month in question, the more certainly it is missing.

Here is the household version of the same opposition. Suppose the question is how often the monsoon floods a particular street. One approach is to write down a rule about rainfall and run it a thousand times, and that rule can be wrong in a way a thousand runs will never reveal. The other approach is to ask what actually happened in the last four monsoons. An answer built that way cannot be wrong about those four monsoons and cannot say anything about a fifth. Neither approach is the careful one and neither is the sloppy one. The two are blind to different things, and a person who knows which one produced a number knows what to distrust about it.

How would anybody know whether a simulation got the answer right?

In almost every real study nobody would, and that is not a failing of the study. A simulation is run because the answer is unavailable, so there is no answer sitting beside the estimate to check it against. Here there is, and it is the one real advantage of working on an invented subject.

The five values of the Nakshatra unit and their weights were set down on paper rather than measured off anything, and that single fact makes every twelve month path that can possibly happen known before a path is drawn. Twelve months across five values gives 1,820 different combinations of how many months took each value, each with a weight that can be worked out and a finishing price that can be worked out. Add the weights of the combinations that finish below Rs 100/-. The total is not an estimate of the chance. The total is the chance, and the chance is 24.6562 per cent.

So all three numbers can be put on one line. No real simulation study ever gets to do that. Monte Carlo simulation came in at 24.5300 per cent from ten thousand paths, a miss of 0.1262 below the exact 24.6562 per cent, and historical simulation came in at 25.0000 per cent from four, a miss of 0.3438 above it. Monte Carlo simulation was nearer, by a factor of nearly three.

Now the part that is easy to get backwards. Monte Carlo simulation being nearer is not the reason to prefer it, and if the two figures had come out the other way round the verdict would not change. Four paths admit no answer but 0.00, 25.00, 50.00, 75.00 or 100.00 per cent, so landing near the truth on a ladder that coarse is luck rather than accuracy. The rungs are 25.00 percentage points apart. Whatever the truth had turned out to be, one of those five rungs was going to sit reasonably close to it, and the one that did says nothing about whether the method worked.

The Monte Carlo figure, by contrast, comes with a size for its own error. With ten thousand paths the standard errorHow far a computed number typically sits from the quantity it was computed to estimate. on the estimate is 0.4310 per cent, and the miss of 0.1262 is about three tenths of that. An unremarkable run, then, rather than a lucky one. A run that had landed on 24.6562 per cent exactly would have been the surprising outcome. The real difference between the two answers sits there. One of them can state how wrong it probably is, and the other cannot.

The estimate does not march. From 500 paths it reads 24.8000 per cent, then 25.1000, then 25.2000, moving further from the truth for two settings in a row before turning back, and only at 2,56,000 paths does it settle at 24.6859 per cent. The band around the reading is what improves reliably, not the reading itself. The band shrinks from 1.9275 per cent at 500 paths to 0.0852 per cent at 2,56,000, while the historical answer sits frozen at 25.0000 per cent however long anybody waits. More paths buy a smaller wobble, not a straighter walk. And nothing done to the path count adds a single month to the fifty month record.

What should be asked of any simulated number that arrives without explanation?

Simulated numbers arrive on desks constantly, and they arrive stripped. A credit officer gets a chance of a shortfall. An analyst gets a range around a projection. A household gets a chance that a savings plan falls short of a target. In every case the figure is a point estimateA single figure put forward as the answer, with nothing said beside it about how wide of the mark it might be. that arrived without the machinery that produced it, and reading it well is mostly a matter of asking for the machinery back. Seven questions do it.

One, where did the shape come from, or was it a record rather than a shape? The answer settles which of the two blindnesses applies. Two, how many paths, and were they independent of each other? Four paths and forty overlapping windows are both small numbers wearing different clothes. Three, what rule generated the numbers and what starting number was it given? Four, would that same starting number give the same answer tomorrow? Five, what is the size of the error on the estimate, so the reader knows whether the last two decimal places mean anything. Six, is there anything in the problem that can be computed exactly and checked against, even a simplified corner of it. And seven, what could this method not have seen, whatever it says.

A simulated number quoted without its path count and its starting number is not a small omission, it is a number nobody can check. The omission is worth being blunt about, because it never looks serious. Leaving the apparatus out looks like tidiness. The figure reads more cleanly without the apparatus, and the apparatus is exactly what would let a second person disagree with it on the evidence rather than on instinct. A lender who cannot rerun a simulated chance of a borrower defaulting cannot argue with it either, and a number nobody can argue with is not a strong number, only an unexamined one.