Backtest vs Live Performance: One Rule, Two Records

A backtest and a live stretch read the same rule and differ in what could be chosen. A backtest picks its threshold with the answers visible and can be recounted a second way; a live stretch cannot. Fitted on its first three years the Ashwin rule calls 26 of 29 scoreable months, or 89.66 per cent, and over the three years that followed it calls 18 of 36, or 50.00 per cent.

Two figures, one rule, and a gap of 39.6552 points between them. Most of what goes wrong with a comparison like this goes wrong in the next sentence somebody writes. The gap is real. The subtraction that produced it is correct. The meaning of the gap is a separate question. Two readings cannot answer it and three can at least be argued about, so the argument below builds a third.

Where these counts come from. The Nakshatra unit, an invented series, its six year record and the Ashwin rule that reads it were all built for teaching. Every count printed below was recomputed from the record's own seventy two monthly changes rather than lifted from a table, so there is no vintage to quote and no as of date to check. The recipe can be checked instead. Stating the record, the rule and the grid of thirteen settings brings every count printed below back. A count that refuses to reappear that way is simply wrong.

What makes a live stretch a different kind of record?

Start with the shopkeeper down the road. She wants to know her busiest hour, so she goes through last year's till roll, hour by hour, and finds that the hour after six in the evening beat every other hour in the day. The finding is real and it is about a real year. Now next Tuesday arrives. Whatever happens between six and seven next Tuesday is a number she did not pick, could not have picked, and had no hand in shaping. The number arrives on its own.

The distinction between the two stretches is nothing more than timing, and it is smaller than most people expect. A live stretch is simply the part of a record that arrived after the rule was written down, and what makes it different is not that the months are better but that they are later. Nobody chose them. There was nothing there to choose from at the moment the choosing happened. The months in the fitted stretch were already on the record while the thresholdA cut off figure. The rule compares its reading against this, and calls up on one side of it and down on the other. Where the cut off gets put is somebody's decision, never something the data hands over. was being decided, and the months in the live stretch were not.

The negative is what people get wrong, so say it back to yourself that way. The later months are not cleaner data. The later months are not harder, or fairer, or more honest. The monthly changes in the live stretch were produced by the same record as the earlier ones and carry no special property at all. The only thing separating the two heaps is a date, and the date matters because of what was happening on the day it fell: somebody was making a decision, and half the record was invisible to that decision.

What can a backtest do that a live stretch cannot?

Lists survive being repeated at a meeting in a way ideas do not, so here is the part worth memorising as a list. There are four things a backtest can do that a live stretch cannot do at all. The four are not subtle statistical wrinkles. The four are ordinary practical liberties, and every one of them is the liberty to choose.

One, a backtest can choose its threshold with the answers already in front of it. The Ashwin rule carries thirteen whole number settings, the lowest at minus 6.00 per cent and the highest at 6.00 per cent, and the fit walks all thirteen, scores each, and keeps the best. Every one of those thirteen scores was computed on months whose outcomes were already known. On the live stretch there is nothing to walk. The threshold was fixed in December 2021 and it stays fixed, whatever the months do afterwards.

Two, a backtest can be recounted under a different convention. Six months of this record did not move at all, and an unmoved month is excluded rather than scored. Excluding it is a conventionA counting rule that was agreed rather than discovered. Set it down beforehand and a second person working alone arrives at identical figures from identical rows., not a fact, and somebody could reasonably decide otherwise. Redo the backtest under the other decision and every figure moves. The live stretch was counted once, on the day, by whoever was counting, and there is no second pass to make.

Three, a backtest can be rerun on a different span of the same record. Start it in 2020 instead of 2019. End it in June instead of December. Each span is a different question with a different answer, and all of them are available at once because the whole record is on the table. A live stretch has exactly one span: the one that happened, from the day the rule was fixed to today, and it grows at the rate of one month per month.

Four, a backtest can be repeated after a fault is found. Somebody spots a mistake in the counting, fixes it, and runs the whole thing again in an afternoon. Repeatability is a genuine strength of a backtest and it should be said plainly. A live stretch cannot be repeated at all. If the months were counted wrongly while they were arriving, the fix is to count them again, but the months themselves are what they were.

Put the four together and the pattern is obvious. Every one of them is a form of choice, and a figure arrived at with choice available belongs to a different species from one arrived at with none. That is not a criticism of backtesting. Choice is what a backtest is for. A thing that can be rerun, recut and recounted is enormously more useful than a thing that cannot, and the price of that usefulness is that its headline figure has more of somebody's decisions baked into it.

What did the Ashwin rule read across the two stretches?

Now the counting. The fit takes the first three years, January 2019 to December 2021, walks the thirteen settings, and keeps the one that calls the most months correctly. The best setting is a threshold of minus 1.00 per cent. Across the 29 scoreable months in those three years it calls 26 of them the right way, or 89.66 per cent.

Then the rule is carried forward without a single change. Same signalWhatever a rule looks at before it commits itself. A signal is always built out of months that have already closed, never out of the month being called, and here it is just the change over the month that has finished., which is the Nakshatra unit's change over the month just ended. Same threshold of minus 1.00 per cent. Same call, up when the reading sits above the threshold and down otherwise. Across the 36 scoreable months of January 2022 to December 2024 it calls 18 of them the right way, or 50.00 per cent.

Notice what each side has to state about itself. Each reading carries its own denominatorThe bottom half of a fraction. The denominator settles what a percentage is a percentage of. Two results carrying the same top half can still be saying quite different things., 29 months on the left and 36 on the right, and the two differ because six months of the record did not move and every one of those six sits in the first three years. January 2019 drops out of the first stretch as well. The signal reads the month just ended, and January 2019 has no month before it, so 36 months come down to 29. A reader who sees only the two percentages has been handed the arithmetic with the denominators removed, and the denominators are half of what those two figures are saying.

One more thing about the second figure, and it needs saying before anybody builds a story on it. The live reading is 18 of 36, exactly one half. Reading something into that half is very tempting, and there is nothing in it to read. Eighteen of thirty six landing on exactly one half is a coincidence of this particular record, not a law that a failed backtest falls to a coin. The count could as easily have come out 17, which reads 47.22 per cent, or 19, which reads 52.78 per cent, and nothing anywhere in the arithmetic was pushing it toward the middle. The number is what it is because thirty six particular months did what they did.

Moving the split is worth settling first. December 2021 is not a sacred date. The date is a choice, like everything else on the backtest side, and the honest way to see how much rests on it is to make the same cut at seven different points and read both figures each time.

How much of the drop is the search, and how much is the record?

Here is the move that turns two figures into an argument. Put a third reading between them.

Take the same thirteen settings and fit them on the last three years, the very months the carried rule struggled on. Let the search have every liberty it had the first time round: all thirteen settings, all the outcomes visible, keep the best. The search picks a threshold of minus 6.00 per cent and calls 22 of the 36 scoreable months, or 61.11 per cent.

The 61.11 per cent is the hinge of the argument. A search with full choice over the later stretch reaches only 61.11 per cent there, so the later stretch is a harder stretch to call even for a rule fitted on it. Now the drop of 39.6552 points has somewhere to pass through. From 89.66 per cent down to 61.11 per cent is 28.5441 points, and that step changes the stretch while leaving the search its choice. The carried threshold of minus 1.00 per cent is not what the later stretch would have picked, so the step from 61.11 per cent down to 50.00 per cent, 11.1111 points, holds the stretch still while taking the choice away. The two steps add to 39.6552 points, the whole drop.

What kind of split is that, and what would make it an attribution?

The split is the most dangerous step in the argument, and it carries its own warning. The two parts add to the whole. Two parts will always add to the whole. Adding to the whole is not evidence of anything.

The arithmetic matters more than the story. The starting expression was 89.66 minus 50.00. Then 61.11 was subtracted and added straight back: 89.66 minus 61.11, plus 61.11 minus 50.00. The middle term cancels itself out. The addition is a consequence of how the expression was written and not a finding about the record, so any number at all put in that position would produce two parts adding to 39.6552. That is a telescoping split, and a telescoping split is not an attribution.

A plainly silly middle reading makes the point. With 70.00 per cent in the middle the two parts are 19.6552 points and 20.0000 points, adding to 39.6552. With 55.00 per cent there they are 34.6552 and 5.0000, adding to 39.6552. Nothing about the record changed between those three tellings. Only the number somebody chose to stand in the middle changed, and the two parts obediently rearranged themselves around it.

So what would make it an attribution? A middle reading earns the name on one condition. Somebody has to argue that it holds one thing genuinely fixed while the other genuinely moves, and has to say why that particular hold is the right one. The 61.11 per cent used above has a real argument behind it, which is why it is the one used here: it is what a full search reaches on the later months, so it separates "the later months are harder" from "the threshold was stale". But it is still one choice among possible choices, made by a person, and an account that prints 28.5441 and 11.1111 without saying so has quietly promoted its own arithmetic to a discovery.

The household version is easier to feel. A household electricity bill went up by nine hundred rupees this year. Somebody says four hundred was the tariff and five hundred was the new cooler. Checking that needs a bill for the same house with the new tariff and the old appliances, and if nobody has one, the four hundred and the five hundred are two numbers that add to nine hundred and nothing more. Two such numbers will always add to nine hundred. Whoever picked the middle figure picked the split.

Did the record itself get harder to call?

The record did get harder to call, and nobody chose that either. A record that changes on its own is what stops the drop from being a story about somebody's carelessness.

The first three years of the six year record carry a spreadA measure of how scattered a set of figures is around its own middle. Scatter them widely and individual entries land a long way out on both sides; scatter them narrowly and they crowd in tight. of 3.00 per cent. The last three years carry a spread of 6.4031 per cent, more than double. And through both stretches the mean sits at exactly 1.00 per cent. The centre of the record did not move at all; only the distance the months travel from it more than doubled, and a rule that calls direction has to work in whatever weather it is given.

Feel it with a street vendor. Two years running, the same juice stall. In the first year daily takings sit between three hundred and five hundred rupees, day after day, and the vendor can guess tomorrow's takings pretty well by looking at today's. A building site opened opposite and its shifts move around, so in the second year the average is exactly the same but the days swing from nothing to twelve hundred. Nothing about the vendor's guessing got worse. The days got wilder. Anyone judging the vendor by the hit rate alone would conclude the vendor lost their touch.

And now the uncomfortable part. The wide stretch had not happened yet, so the backtest could not possibly have separated a rule breaking down from a record becoming harder to call. In December 2021, when the threshold was fixed, every month available to look at came from the narrow stretch. There was no signal anywhere in that data saying the next thirty six months would swing twice as far. Missing it was not an oversight anybody could have avoided by being more careful. The limit is on what a backtest can know, and it is permanent.