How to Run a Monte Carlo Simulation Responsibly
Running a Monte Carlo simulation responsibly means six things done in order: state the shape, state the generator and the seed, fix the count of draws before any answer has been seen, run once, report the error alongside the estimate, and check against something known. Twenty seeds on one shape gave answers running from 22.0000 to 25.8000 per cent, and all twenty were correct.
The spread in those twenty answers is the uncomfortable part, so start there. Twenty people can run the same arithmetic, on the same assumption, with the same tool, and come back with twenty different numbers, none of which contains a mistake. There is no slip to find, no cell to correct and no argument to have about the method. The spread came out of the drawing itself, and the only thing standing between an honest report and a chosen one is a habit.
Six steps make up that habit. The method itself, and the building of a single run, are covered separately. The habit is a running order and a short list of things that go on a report, and the whole of its value is that it is boring enough to actually do.
What is already settled beforehand?
Three things arrive finished and are used without being explained a second time. First, the method itself: drawing a great many made up futures out of a written down assumption, then counting what fraction of them did the thing in question. Second, how one run is built, month by month, from a stream of drawn fractions. Third, what a standard errorThe typical distance between a computed figure and the quantity it was computed to estimate. More observations narrow it, and they narrow it slowly. is and why a point estimateOne number put forward as the answer, with nothing attached to it saying how far out it might be. offered on its own is an incomplete report rather than a confident one. All three are covered separately.
A question and its exact answer are also settled beforehand. An invented traded unit called the Nakshatra unit starts at Rs 100/-. Every month its price changes by one of five amounts, minus 9.00, minus 4.00, 1.00, 6.00 or 11.00 per cent, and the weightsThe numbers written against each value in a list, saying how often that value is meant to come up. Across a complete list they add to one. on those five are 0.08, 0.18, 0.48, 0.18 and 0.08. The question is simple: after twelve months of that compoundingApplying each change to the amount already standing, so a later change is measured against the running figure rather than against the amount begun with., what are the chances the price ends below the Rs 100/- it started at?
Because that shape was invented rather than measured, its exact answer can be worked out without drawing anything at all, and it is 24.6562 per cent. Twelve months across five possible values makes 1,820 distinct combinations, and weighing all 1,820 by hand is tedious but finite. An exact answer is a luxury no real study ever has. Every estimate below can therefore be printed beside the truth it is estimating, so how far each one landed is visible.
What does running a Monte Carlo simulation responsibly actually require?
Six steps, in this order, every time. Each one holds back a particular fault, and a step whose fault cannot be named is the step that gets skipped the first time somebody is pressed for time.
- State the shape. Write down the values and the weights before anything is drawn, and write down where they came from.
- State the generator and the seed. Name the arithmetic rule that produces the drawn fractions, and name the number it started from.
- Fix the count. The number of draws is decided before a single answer has been seen, and the record says that it was decided first.
- Run once. One run, the one declared. Not the best of several.
- Report the error. Publish the estimate with a figure attached saying how far out it might be.
- Check against something known. The method is pointed at a question whose answer is already known, before it is pointed at one nobody can check.
Of the six steps, which one holds back a number that nobody else can produce again?
Why must the number of draws be fixed before any answer has been seen?
Here is a thing that sounds harmless and is not. A run is going, the count of falling paths is ticking up, and the running estimate is on screen. Somebody watches it for a while. Then they stop it and write down what it says.
The trouble is that the running estimate does not walk steadily towards the answer. The estimate wanders towards the answer, and the wander is wide for a long time. Take the declared run on the Nakshatra unit, the one started from seed 12345, and read it at four points as it goes. At 500 paths it says 24.8000 per cent. At 1,000 it says 25.1000. At 2,000 it says 25.2000. At 5,000 it has come back down to 24.4600. Those four readings are not four runs. The four came out of one run, looked at four times.
A person who wanted any figure between 24.4600 and 25.2000 per cent could have got it from that single honest run simply by choosing the moment to look away. No number was altered. Nothing was rerun. The estimate was simply harvested at a chosen point on a line that was always going to pass through a range of values on its way to settling down.
The everyday version of this is a bathroom scale. A person steps on and reads 71, steps off because the reading was unwelcome, steps on again and reads 70, and writes down 70. Nothing about the scale is broken and neither reading is a lie. The stopping rule was allowed to depend on the reading, and once that is true the reading stops being about the person and starts being about what they were willing to accept.
One run read 24.8000 per cent at 500 paths and 25.2000 per cent at 2,000. A colleague reports the 500 path figure, explaining that the run was slow that afternoon. What has gone wrong, and what has not?
What is seed shopping, and how far can it move an answer?
Fixing the count does not close the door. Fixing the count closes one door. There is a second way into exactly the same room, and it is tidier, faster and much harder to see from outside.
The stream of drawn fractions is pseudorandomProduced by a fixed arithmetic rule that looks unpatterned but is not, so the same starting number always returns the same stream. How such a rule is built is covered separately.: it comes out of an arithmetic rule that always gives the same string of values in the same order from the same starting number. The starting number is the seed. Changing the seed gives a different stream, a different set of drawn futures, and a different answer to the same question, with the count of draws untouched and the assumption untouched. Nothing about that is a fault. Choosing the seed after the answers are visible turns it into a fault, and that choice is what seed shopping is.
Twenty seeds are about to be run at 1,000 paths each, same shape, same generator, nothing else different. How wide should the gap between the lowest answer and the highest be expected to be?
Here are the twenty, run at 1,000 paths each: 23.0000, 24.5000, 23.3000, 23.4000, 23.2000, 25.7000, 25.2000, 24.2000, 23.6000, 23.5000, 25.5000, 24.6000, 23.3000, 25.1000, 24.9000, 24.1000, 25.7000, 22.0000, 25.8000 and 23.4000 per cent. The lowest is 22.0000 from seed 18. The highest is 25.8000 from seed 19. The gap between them is 3.8000 percentage points, and the twenty average out at 24.2000.
A person who wanted a low answer could have reported 22.0000 per cent and a person who wanted a high one could have reported 25.8000, and neither of them would have computed a single thing incorrectly. That is what makes this the hardest of the six faults to catch. There is no wrong cell. There is no wrong formula. There is no wrong assumption. The seed that produced each report is right there in the write up, so both reports reproduce perfectly for anybody who repeats them.
Look at where the truth sits in all this. The exact answer is 24.6562 per cent, and it lies inside the range the twenty seeds produced, closer to the top than the bottom. Not one of the twenty landed on it, and none of them was ever going to. Twenty runs of a thousand paths is still only twenty thousand paths, and twenty thousand paths is not a great many, so even the average of all twenty comes in 0.4562 per cent below it.
Shopping for an answer, with the panel keeping count of how many seeds were tried.
One control moves: which of the twenty seeds is used. The shape never changes, the generator never changes, and the count stays pinned at 1,000 paths, so the seed is the only thing that differs between any two readings available here. The opening setting, seed 1, reads 23.0000 per cent against an exact answer of 24.6562 per cent, the first of the twenty readings printed above. The lower strip is the part that matters: it remembers every seed already looked at, so the shopping becomes visible from the inside rather than from a list.
Educational illustration. All twenty readings come from one shape, one generator and 1,000 paths apiece, so the seed is the only thing that differs between any two of them. Picking the seed whose reading looks best is precisely the fault described above, and the count of seeds tried is the only thing a reader downstream would ever have needed in order to see it.
An analyst reports 25.8000 per cent from seed 19. Every figure checks out, the run reproduces exactly, and the arithmetic is spotless. What question would expose the problem, and why does no check of the arithmetic ever find it?
What has to be reported alongside the estimate?
Four fields, and no fewer. The estimate. The count of draws. The generator and the seed. The error on the estimate. A reader handed all four can sit down and produce the number again; a reader handed three of them cannot, and that gap is the whole difference between a result and an assertion.
Consider a bill from a shop. A bill that says Rs 100/- and nothing else asks to be trusted. A bill that says four kilos at Rs 25/- the kilo, total Rs 100/-, invites checking, and the checking takes four seconds. Nobody thinks the second shop is being defensive. The four fields do the same work, and a report that carries them is not being cautious, it is being usable.
The declared run on the Nakshatra unit fills all four. The estimate is 24.5300 per cent. The count is 10,000 paths, of which 2,453 finished below Rs 100/-. The generator multiplies the running number by 16,807 and keeps the remainder on 2,14,74,83,647, and it was started from seed 12345. The error on the estimate is 0.4310 per cent, putting the honest reading somewhere between 24.0990 and 24.9610 per cent. The exact answer of 24.6562 sits comfortably inside that.
Two error figures read alike here and are not alike. The 0.4310 per cent is worked from the true share of 24.6562 per cent, known by enumeration. Worked instead from the counted share of 24.5300, the same formula returns 0.4303. Both round to 0.43 at two decimals, and they remain two quantities with two different inputs, so the one used above is named rather than the two being run together as one number.
Now set the declared run beside the twenty shopped ones. Twenty readings scattered across 3.8000 percentage points, every one of them a bare point estimate with nothing attached. Against that, one reading of 24.5300 with a stated width of 0.4310 either side. The single declared run carries more usable information than all twenty of the others put together, and it does so purely because it says how far out it might be.
A report gives the estimate and the count of draws but not the seed. Which of the four fields is missing, and what specifically can no reader now do?
The stated shape makes 26.00 per cent of single months fall, and that is known by construction rather than estimated. What should 1,00,000 drawn single months be expected to return, and what would a return of 31.00 per cent indicate?
How is a method checked for correct implementation?
Almost nobody does this and it costs almost nothing. Before a method is pointed at a question nobody can check, it is pointed at one that can be.
Such a question is already sitting in the assumption. Of the five monthly changes, two are falls, and their weights of 0.08 and 0.18 add to 0.26. So the shape says that 26.00 per cent of single months fall, exactly, by construction, with no drawing involved at all. The 26.00 per cent is a fact about the assumption rather than a fact about anything that happened, and that is precisely what makes it useful as a test.
So draw 1,00,000 single months from the shape under seed 12345, each one an independent drawA draw that says nothing about the draw before it and is told nothing by it., and count the falls. The count comes back at 25,975, or 25.9750 per cent. One check exercises the drawing, the mapping of each drawn fraction to a monthly change, and the counting, all in a single pass, against an answer already in hand. Had it come back at 31.00 per cent, the fault would be in the arithmetic rather than in the world, because the world has no say in what an invented assumption implies about itself.
The kitchen version is a scale and a sealed one kilo packet. The packet goes on before the vegetables are weighed. If the scale says 1.02 kilos the weighing carries on, and if it says 1.31 the scale is fixed rather than the vegetables argued with. Nobody thinks this is clever. Checking the scale first is simply what a person does who intends to believe the readings that follow.
A check comes back at 26.4000 per cent against a known 26.00 per cent. Is that a bug, and what should be computed before deciding?
What makes a result reproducible by somebody who was not there?
A result is reproducible when a stranger holding the shape, the generator, the seed and the count gets the same digits, not the same ballpark. Not a similar figure. The same figure.
The standard sounds severe until it is watched being met. The declared run starts at 12345. Multiplied by 16,807 and reduced to the remainder on 2,14,74,83,647, it gives 20,74,82,415. Divided by that same 2,14,74,83,647 it gives 0.096617, the first drawn fraction of the first path. Eleven more turns of the same handle and the twelfth fraction is 0.195113. The first path ends at Rs 99.2786/-, below the Rs 100/- it started from, and counts as one of the falling ones. Every step of that is a multiplication and a remainder, and a person with a calculator and a spare afternoon can produce all twelve.
Compare that to a recipe. Cook the rice till it looks right is not reproducible; two cups of rice to three and a half cups of water on a low flame for eighteen minutes is. Neither instruction is more honest than the other, but only one of them lets a second cook disagree about the result on some basis other than an opinion of whoever wrote it. A method whose figures cannot be regenerated leaves a reader nothing to push against. A figure nobody can push against has not thereby been made solid. The figure has been placed out of reach of examination instead, and from a distance the two look the same.
A stranger has the shape, the generator and the seed, but not the count of draws. Can they reproduce the figure, and what is the count doing in the four field report?
What does a clean run still not entitle anyone to say?
The limits on a clean run matter most to the person reading the work rather than doing it, and they are worth being blunt about. A run that follows all six steps perfectly still does not entitle anybody to three things.
A clean run does not entitle anyone to say the shape is right. Somebody sat down and chose those five values and those five weights. The drawing was never looking at anything except the shape, so every figure downstream inherits whatever that person got wrong, and no amount of drawing corrects it.
A clean run does not entitle anyone to say the answer describes anything outside the shape. The estimate of 24.5300 per cent answers one question: what fraction of paths drawn from this stated assumption ended below their starting price. No actual traded thing was ever consulted, so the estimate says nothing about any market and nothing about any actual traded thing.
And it does not entitle anyone to say that a narrow error means a reliable answer. The third limit catches careful people. The error of 0.4310 per cent measures one distance only: between the counted figure and whatever this identical assumption would settle on if the paths never stopped coming. The error is a measurement of the drawing. A perfectly run simulation of a wrong assumption is a wrong answer computed to four decimal places. The four decimal places are the dangerous part, reading as care when they are only arithmetic.
So what does a careful reader of somebody else's simulated number actually do with all this? Four things, and they take about a minute. The estimate is about the shape, so the shape comes first and the estimate second. Either one alone leaves a reader unable to reproduce anything, so the count and the seed both have to be there. Seed shopping leaves one trace only, a missing sentence saying the count was fixed before the first run, so that sentence is worth looking for. Then, if the answer is going to change what anybody does, ask for a sensitivity checkRunning the same work again with one input nudged, to see how much of the answer was resting on that input. on the assumption itself rather than on the drawing. The drawing is the part already covered by the error figure.
A run reports an error of 0.4310 per cent. What exactly is that number measuring the distance to, and what is it silent about?
Two correct numbers, one honest analyst, and a report nobody downstream can question
An analyst runs 1,000 paths and gets 22.0000 per cent. The reading looks low. Nothing is changed, nothing is fixed and no assumption is touched; the run is simply started again from a different seed, and this time it returns 25.8000 per cent. The second reading goes into the report, with the seed written beside it exactly as it should be.
Not one step of that was arithmetically wrong. Both readings are correct outputs of a correct method on a correctly stated shape. The data was not touched, the code was not touched, and the assumption was not touched. The write up is complete, it names its seed, and anybody who reruns it lands on 25.8000 per cent to the last digit. Perfect reproduction is precisely why the fault survives every check run on it.
The cost is not accuracy. The cost is that the reported figure has quietly stopped being a statement about the Nakshatra unit and started being a statement about which of two readings the analyst preferred, and nobody downstream can detect the difference. Of every fault here, seed shopping is the hardest to catch, surviving every check except the one that asks how many seeds were tried. The fix is a single sentence in the report: the seed and the count were fixed before the first run, and this is the only run performed. Anybody can write that sentence. The hard part is that the sentence has to be true, and the decision has to be made before there is anything to be tempted by.
What would somebody need in order to check every figure here?
An arithmetic result has no publication behind it and no maintained record, so there is no date on which anything was consulted. There is an arithmetic rule and a starting number instead, and a rule written out in full beats a document as a reference, since a doubtful reader settles such a rule with a calculator in an afternoon rather than by trusting whoever quoted it. The four rows below are the whole of what a stranger needs.
| What produced it | The exact setting | What it yields here |
|---|---|---|
| The stated shape of the monthly change | Five values, minus 9.00, minus 4.00, 1.00, 6.00 and 11.00 per cent, carrying weights 0.08, 0.18, 0.48, 0.18 and 0.08 | Every drawn month here, and the check on the falling share |
| The generator | Multiply the running number by 16,807, keep the remainder on 2,14,74,83,647, then divide by that same figure | The stream of drawn fractions, 0.096617 first |
| The seeds, each named where it is used | 12345 for the declared run and the validation check, and 1 through 20 for the twenty shopped runs | Which stream every reading above came out of |
| The exact answer, by enumerationListing every case that can occur and weighing each one, rather than drawing a sample of them. It gives an exact answer and is only available when the list is short enough to finish. | All 1,820 twelve month combinations weighed in full, with no drawing at all | 24.6562 per cent, the truth every estimate here is printed beside |
The Nakshatra unit is invented.
Educational material. Not advice on any investment, tax, budget or market position.