Backtesting a Pricing Model: Testing the Fit Out of Sample

Backtesting a pricing model means fitting it on part of a set of target prices and then measuring it on the rest. The model is never shown that second part while the fitting runs. Every other check compares a model against numbers that shaped it, so a backtest is the only check on calibration that can fail and mean something.

Fitting a model to a set of prices is covered under calibration. Backtesting is the single arrangement that reveals whether the fitting went too far. The arrangement is a small one. A set of numbers is chosen to fit to, some of them are hidden before the fitting starts, the fit runs on what is left, and the finished model is then asked about the ones that were hidden. Nothing more elaborate than that, and nothing in it requires unusual software.

The mechanics are not what makes backtesting matter. A backtest is the first check on a fitted model with a genuine chance of coming out badly. Every other test that can be run on a fitted model asks it about numbers it was built to agree with, so agreement is the only answer available. Hide two of the five and the agreement stops being automatic, and whatever the model then says is information rather than an echo.

The quantity is the standard process, written S with a time subscript, starting at Rs 100/- with a rate of 5 per cent a year over a horizon of one year. The numbers it is fitted to are five invented call prices at strikes of Rs 80/-, Rs 90/-, Rs 100/-, Rs 110/- and Rs 120/-.

What is backtesting a pricing model?

BacktestingFitting a model on part of a set of numbers and then measuring it on the part held back. is the practice of holding some of the target numbers back from the fitting, and then using the held back numbers as a test rather than as targets. The model never gets a vote from them while it is being shaped. The model meets the held back numbers only afterwards, once it is finished and can no longer be adjusted.

The kitchen scale returns, the same object as before, and the extension to it is exact. A scale has one adjusting screw and five objects whose weights are printed on them. Turning the screw until all five readings sit as close as one screw allows to the five printed weights is calibration, and after it the scale can say nothing about those five objects that was not already known. Now one thing changes. Only three of the objects set the screw, the other two are put away where the screw cannot see them, and once the screw has stopped moving those two are weighed. The two withheld weighings are the first thing the scale has said all afternoon that could not have been written down in advance.

The withheld pair is the whole idea, and every complication built on top of it is bookkeeping. The three objects used to set the screw are the in-sample set. The two put away are the out-of-sample set. The difference between how badly the scale misses on the first group and how badly it misses on the second is what a backtest actually reports.

Notice what has changed about the question being asked. Before the split, the question was how close this instrument can be made to these five weights. A search always returns its best available answer, so the answer was always going to be some number. After the split, the question is whether an instrument shaped by three weights carries any of that shaping over to two it never touched. The first question is about the search; the second is about the model.

What does out of sample mean here?

In-Sample Fit vs Out-of-Sample Performance

Two phrases carry the whole argument and they are worth pinning down before any arithmetic. In sampleThe part of the target set the fitting was allowed to use. Three strikes in this guide. means inside the group the parameter was chosen from. Out of sampleThe part of the target set held back from the fitting. Two strikes in this guide. means outside it. Neither phrase says anything about time. In many other settings backtesting does mean testing on an earlier period, so the absence of time here is the first thing to get straight.

Here there is no earlier period. There is one horizon of one year and five contracts sitting side by side at five different strikes, all quoted at the same moment. The thing being withheld is not a stretch of time; it is a region of the strike scale. A model fitted only in the middle has been told nothing about the ends, and asking it about the ends is a fair question in exactly the way that asking it about the middle is not.

The two arrangements test different things, and the distinction matters more than it sounds. Withholding a later period tests whether a relationship that held then still holds now. Withholding a region tests whether a shape that fits in one place carries to another place. One number has to serve every strike at once, and the five targets carry five different implied volatilities, so the second question is the one a single volatility parameter is most exposed to.

The second everyday version is the one most readers have lived. A student is given a practice paper with five questions and the answers printed at the back. A student who learns all five answers will score full marks on that paper and will have learned nothing about whether the subject can be done at all. A student who instead works through three of them and then sits down cold with the other two collects, in the marks on those two, the first honest measurement of the afternoon. The difference between the score on the three and the score on the two is the only quantity in that afternoon that is about the student rather than about the paper.

One more caution about the word. Nothing about a set being out of sample makes it more real or more true. The two withheld prices are just as invented as the three fitted ones. Their usefulness comes from their position rather than their provenance: they were kept out of the room while the decision was made.

How is the test actually set up?

How to Test a Pricing Calculator Against Benchmarks

A benchmark here is simply a number the calculator is checked against that the calculator did not produce. A benchmark can be a value from a closed form solution where one exists, a value from a second implementation written independently, or a target price kept out of the fitting. The third kind tests the model rather than the arithmetic, and the third kind is the one used below. The first two kinds check whether the code is right; only the third checks whether the model is.

Running that search on the three middle strikes returns 0.199868. Running the same search on all five, the calibration case, returns 0.198202. The two answers sit 0.001666 apart. A reader handed only those two numbers and asked whether it mattered which set was used would honestly answer that it looks as though it barely matters at all. 0.001666 of volatility is a sixth of a tenth of one percentage point. The reading that it barely matters is wrong.

How much worse does the fit get outside?

Out-of-Sample Performance

The measure used on both groups is the root mean squared errorSquare each miss, average the squares, take the square root. The result is a figure in the same units as the thing being missed., which returns a figure in rupees of price and can therefore be compared directly between the two groups even though one has three members and the other has two. Squaring first is what stops a miss of minus Rs 0.64/- and a miss of plus Rs 0.50/- from cancelling into something that looks like agreement.

The two numbers are 0.394853 inside and 0.573319 outside. The second is 0.178466 larger than the first, a gap of 45.20 per cent. A parameter that moved by 0.001666 produced an error that moved by 0.178466, a hundred times further.

The aggregate hides where the trouble sits, so now the individual misses. At the frozen volatility of 0.199868 the five model prices sit against their five targets like this. The three fitted strikes are marked inside and the two withheld ones outside.

The two withheld misses are minus Rs 0.639963/- at Rs 80/- and plus Rs 0.497832/- at Rs 120/-. Read those two rows carefully. The shape of them is the finding. The larger of the two is minus Rs 0.639963/- at Rs 80/-, and it is the biggest single miss anywhere on the five. The smaller of them, plus Rs 0.497832/-, is not the second biggest: it sits below the miss of minus Rs 0.561716/- at Rs 90/-, which is inside the fitted group. Both withheld misses are larger than the in-sample root mean squared miss of 0.394853. Only one of the three fitted strikes is, and that contrast is what the aggregate figure reports.

The averages tell the same story more plainly. Ignoring signs, the three fitted strikes miss by 0.318923 on average and the two withheld ones by 0.568897. The model is nearly twice as far out where it was not looking, and it got there without anybody changing a thing about it.

A small parameter change does not always produce a large error change; one worked set establishes nothing that general. The narrower claim is the one that holds. On this set, with this split, the size of a parameter move says nothing whatever about the size of the error move, so reading the first as reassurance about the second is a mistake with a worked example against it.

What does the gap between the two indicate?

The quantity that carries meaning is the difference, and this is the single hardest idea in this guide because both of the other candidates feel more solid. The in-sample figure feels solid because it is what the search was aiming at. The out-of-sample figure feels solid because it is the honest measurement. Both figures mix together how good the model is with how much the fit absorbed, so neither one on its own says anything about the fitting. Only the difference separates the two.

The word for what a positive gap indicates is overfittingA fit that has absorbed something specific to the targets it was shown, which does not carry over to targets it was not.: the fitted parameter has taken on something about the three strikes it was shown that does not extend to the two it was not. In this worked set the mechanism is visible rather than mysterious. The three middle strikes carry implied volatilities of 0.220000, 0.200000 and 0.190000, a range of three points. The two withheld strikes sit at 0.240000 and 0.185000, outside that range on both sides. One constant fitted to the middle of a sloping set will always be too low for the low strikes and too high for the high ones, and the further out the strike sits the worse both get.

Read the low three row against the middle three row. That pair settles the argument. Fitting on Rs 80/-, Rs 90/- and Rs 100/- gives an inside miss of 0.394214. Fitting on Rs 90/-, Rs 100/- and Rs 110/- gives 0.394853. The two inside figures differ by 0.000639, a difference of nothing. Their outside figures are 0.837251 and 0.573319. Those two differ by 0.263932, and that is not nothing. Two fits that look identical from inside are very different from outside, and no amount of staring at the inside figure would ever have revealed it.

The high three row makes the point harder still. Fitting on Rs 100/-, Rs 110/- and Rs 120/- gives the smallest inside miss of any split shown here, 0.223625. The outside miss it produces is 0.756433, or 3.38 times the inside figure. A reader shown only the inside column would rank that split best. A reader shown only the outside column would rank it fourth. The gap is what shows which ranking is about the model.

What does a passed backtest not establish?

Suppose the gap had come out at nought, or close to it. The natural sentence to write next is that the model passed, and the natural thing to believe next is that the model generalises. The second of those does not follow from the first, and the distance between them is worth being precise about.

A passed backtest establishes exactly this: the model did not fail this particular split of this particular target set on this particular measure. Every one of those four qualifiers is doing work, and dropping any of them turns a narrow true statement into a wide false one.

The splitThe particular cut of the target set into a fitted part and a withheld part. Many different cuts of the same set are possible. is a choice, and four other cuts of the same five prices behave completely differently. The middle three split gives a ratio of 1.45. The low three gives 2.12. The high three gives 3.38. The outer two gives 0.83. Below one means the model does better on the strikes it never saw than on the ones it was fitted to. The model was identical throughout. Only the question changed.

The target set is a choice too. Five invented prices at one horizon is a small and particular thing to be tested against, and passing on them says nothing about any other horizon, any other five strikes, or any set of a different size. The measure is a choice as well: squared price error is used throughout here, and calibration records that the same five targets return 0.198202 under squared price error, 0.207000 under squared implied volatility error and 0.204400 when price error is weighted by vega. A test built on one of those is not a test of the other two.

The honest way to write up a passed backtest is therefore to write down the split, the set, the measure and the result together, as how to document a pricing model sets out, and to resist every temptation to compress the four into a verdict. A backtest is a question that was asked and answered, not a certificate the model now carries around.

How does a reviewer actually use this?

Everything above is a procedure with no institution attached, so here is what somebody reviewing a pricing calculator does with it in practice, whether they are validating a model somebody else built or checking their own before they trust it.

The first thing they do is refuse to accept a single error figure. Handed a validation note saying the model achieves a root mean squared price miss of 0.573319, the correct response is to ask which strikes that was measured on and what the figure was on the strikes the model was fitted to. A note that cannot answer both questions has not been reviewed, whatever it says at the top of it.

The second thing they do is check when the split was chosen. Whoever chooses a split after seeing the errors has already been told which answer they will get, so a split chosen that way is not a test. Timing is the one part of the procedure that cannot be verified from the numbers afterwards, and how to document a pricing model puts the split in the written record rather than in the results for exactly that reason.

The third thing they do is look at where the misses fall rather than only how big they are. In this worked set all three fitted strikes miss in a pattern, low at the low strikes and high at the high ones, and the two withheld strikes continue the same pattern further out. A pattern like that says the model has a shape problem the fit is absorbing rather than a noise problem the fit is smoothing. Misses scattered without a pattern would say the opposite. The aggregate figure destroys that information entirely, and the table of five individual misses sits beside the two summary numbers for that reason.

The fourth thing is the smallest and the most often skipped. The reviewer checks that the errors were computed at the parameter as it was recorded. The fit here returns a volatility whose full precision value carries many more places than six, and the recorded figure is 0.199868. Every miss in the table above is computed at 0.199868, the recorded value, and that is what makes the table reproducible by anybody holding only the written record. Errors computed at a precision the record does not carry cannot be checked by the person the record was written for, and a validation figure nobody can reproduce is a validation figure nobody should rely on.