Training, Validation and Test Sets, Compared in Full

Training months fit the model, validation months choose between models, and test months are opened once at the end so the figure reported was never tuned against. Cut the ten paired months of the Nakshatra unit and the Vasant unit six, two and two and one fitted line misses by 4.8774 on the months it learned from, 5.2169 on the validation pair and 5.8266 on the test pair.

Two words in that paragraph carry hidden weight, and both get settled before anything is built on them. A traded unitA label with a quoted price behind it and no story of any kind. Nobody issues it, nobody runs it, nothing is made or sold. All the arithmetic here needs is that the quote lands somewhere different each month. is a bare name with a quote against it. Its monthly change records the distance that quote covered across one month, stated as a share of where the month opened. When one reading sits at 2.00 per cent and the next at 18.50 per cent, what separates them is 16.50 percentage pointsWhat is left after one percentage reading is subtracted from a second, counted as plain units rather than as a share of anything. A reading that climbs from 2.00 to 18.50 has crossed 16.50 of them. Calling that a rise of 16.50 per cent would describe a completely different journey., and every miss below is measured in those.

Everything after this is one cut through one record, made three ways and then made every possible way. Where the cut falls decides more than most reported results admit.

What arrives already settled, and what does the split add?

Three things are taken as read here, and none of them gets explained a second time. The mechanics of laying a straight line through two columns of readings belong elsewhere. So does the idea of a miss, along with squaring the misses and totalling them. So does the range of regression forms on offer and what separates one from another. All three are treated in their own right elsewhere and are simply borrowed here.

The split adds a vocabulary and a warning: three named parts of a record with one job each, and the demonstration that a figure taken from any one cut is a single draw from a range wide enough to swallow the result. Somebody who already knows how to fit a line can still report a number that means nothing.

What are the three parts, and what is each one for?

The record comes first, and it is then cut into three named stretches. Each stretch gets one job. The contrast between the three is worthless until each side has been pinned down on its own, so all three are defined in full before any of them is put against the others.

The training months are where the numbers inside the model come from. A fitting routine reads those months and nothing else, and returns a set of coefficientsThe numbers a fitting routine settles on and writes into the model, such as the slope and the intercept of a straight line. They are chosen from the data rather than picked by a person, which is what makes the months they were chosen from a poor place to judge them.: on a straight line, a slope and an intercept. The coefficients are the model. Every month outside the training stretch is invisible to the routine while this happens, and that invisibility is the only thing that makes the rest of the exercise mean anything.

The validation months are where a choice between finished models is made. Nothing is fitted on them. Two or three or ten already-fitted candidates are each asked for their readings on these months, the misses are scored, and one candidate is kept. The validation months therefore do get used, repeatedly, and they leave their fingerprints on the model that survives. A candidate that wins on the validation months has been selected partly for suiting those particular months. A validation figure is therefore not a clean report either.

The test months are opened once, at the end, and report a figure that nothing was tunedAdjusted after seeing a result. Changing an input, a setting or a choice of model because a score came back disappointing is tuning, whether or not anybody calls it that, and whatever the score was measured on stops being an independent reading the moment it happens. against. Reporting is their whole job and their only job. The test months are not there to improve anything. The moment a test figure is looked at and something is changed because of it, those months have been tuned against, they have become validation months, and there is no test figure left on the desk at all.

Here is the everyday version, and it is worth holding on to because the three acts feel identical until they are named. A cook making a large pot for a wedding tastes it while cooking and adds salt, over and over. Tasting and salting is training: the pot is being changed by what the tasting says. Then two versions get made and one friend is asked which is better. Asking the friend is validation: nothing is being adjusted any more, one of two finished things is being picked. Then the pot goes out to the table. Serving the pot is the test, and it happens once. A cook who carries the pot back into the kitchen after the first guest pulls a face has not run a test, they have run a very expensive tasting.

Why can a model not be judged on the months it was fitted to?

The plainest point in the whole subject is regularly treated as a subtlety, and it needs stating without decoration. The coefficients were chosen to make the misses on the training months as small as they could be made. Making those misses small is not a side effect of fitting, it is the definition of fitting. Scoring the model on those same months therefore asks the arithmetic to mark its own work, using the very criterion it was told to optimise.

The circularity is not a delicate statistical caveat, and the trap is that the circular figure is the one a fitting routine prints without being asked for it. Run any fitting tool and a miss on the fitted rows appears in the output by default. Nobody requested it, nobody had to earn it, and it sits at the top of the report looking exactly like a result. A figure computed on months the model has never seen has to be asked for deliberately. The honest number is always the one somebody has to go out of their way to produce.

The household version: a candidate sets their own exam, writes the answer key, and then sits the exam. Scoring well tells the room that the candidate can copy their own key. A good score tells them nothing whatsoever about the paper somebody else will set next month. The fitting routine is doing exactly that, at speed, without malice, and it prints the score at the top.

What does the cut look like on these ten months?

The record runs to ten months, made up for teaching, and each one holds two readings. The Nakshatra unit's monthly change goes in and the Vasant unit's monthly change comes out. Both columns sit in time orderRows left exactly where the calendar put them, oldest at the top, never reshuffled and never ranked by size. Reordering tidies up a chart and throws away the one thing that lets anybody ask what followed what. and they stay in it. Months 1 to 6 train, months 7 and 8 validate, months 9 and 10 test.

Fitted on the training six alone, the line reads 2.6467 plus 1.5200 times the Nakshatra unit's monthly change. The line fitted to all ten months is worth setting beside it. The fuller line reads 0.5000 plus 1.5000 times the same input and leaves squared misses of 218 across the record. Notice which half moved. The slope barely shifted, from 1.5000 to 1.5200. The intercept more than quintupled, from 0.5000 to 2.6467. Dropping four months out of ten left the tilt of the line almost untouched and moved its height substantially, so a model can be badly wrong about the level while looking steady about the relationship.

And the honest note that belongs here rather than in a footnote. Six months is a very short record to fit anything on. The shortness is part of what makes the demonstration below as violent as it is. A longer record would make every figure here calmer without changing a single one of the arguments.

How much worse is the model on the months it has not seen?

One line, fitted once on the training six, then asked for its readings on all three stretches. The score used throughout is the root mean squared missSquare each miss so the sign stops mattering, average the squares across the readings, then take the square root to get back into the original units. It comes out in percentage points here, and how it compares with the other error measures is covered separately.. The score comes out in percentage points and can therefore be compared across stretches of different sizes.

The three figures rise in order, and the model is 19.46 per cent worse on the months it has never seen than on the months it learned from, both figures computed here rather than asserted. The rise is not dramatic and it is not supposed to be. The ordering matters, and so does the fact that it was measured at all. A report that quoted 4.8774 and stopped would have understated the miss by about a fifth without saying anything false.

Why must the cut follow the time order?

The ten months arrived in an order, and that order is part of the data rather than a presentational choice. Cut the record at random and month 10 can land in the training block while month 2 lands in the test block. The line is then fitted using something that happened in the tenth month and marked on something that happened in the second. No future record will ever be able to do that.

Shuffling the record before cutting it does not make the test fairer, it makes the test easier, and it does so silently. The output never records that the months were reordered. The score comes back looking like every other score. Think of a shopkeeper working out how much stock to hold on a Friday, using what the following Monday turned out to need. The rule that comes out will look wonderful on the record. Monday has not happened yet, so the rule cannot be run on a single future Friday at all.

Can a held-out score be perfect purely by luck?

A held-out score can be perfect purely by luck, and this record contains the proof rather than an argument for it. Change the exercise slightly: instead of one six, two and two cut, hold out just two months, fit on the remaining eight, and score the two that were kept back. There are 45 ways to pick two months out of ten, and all 45 were worked through.

Exactly one of them returns a miss of exactly nothing. Hold out months 4 and 6, and the line refitted on the other eight comes back reading 0.5000 plus 1.5000 times the input. The refitted line is the same line, to the last decimal place, as the one fitted on all ten. The two match because months 4 and 6 are the two the full line misses by nothing at all. Dropping a month the line already passes exactly through does not tug the line anywhere, so the refitted line arrives back at those two months and hits both of them dead on.

A held-out score of exactly nothing was produced here by which two months were chosen and by nothing else, and no step in the procedure raised a hand about it. The output would read: model tested on data it had never seen, miss of nought. Every word of that sentence is true and the number is worthless.

Does a model always do worse on months it has not seen?

No, and this is where a great many explanations quietly overstate the case. The claim that a model always scores worse on unseen data is comfortable, memorable and false, and the record here can settle it because every possibility was enumeratedEvery case worked through, one after another, rather than a handful tried and the rest assumed to follow. It is only available when the number of cases is small enough to finish, which is exactly why a ten month record is a good place to look. rather than sampled.

There are 1,260 ways to cut ten months into a six, a two and a two. All 1,260 were solved: the line refitted on each training six, then scored on that split's own training months and on its own test pair. The held-out months came out worse in 797 of the 1,260 and better in 463. The honest statement is that a model does worse on unseen months on average and 63.25 per cent of the time on this record, and that is a far weaker claim than always.

The spread is the part worth sitting with. Averaged across all 1,260 cuts the training miss is 4.1784 and the test miss is 5.7340. The general pattern shows up cleanly in those two averages. But the individual test figures run from 0.1865 all the way to 16.5680. Same ten months, same straight line, same procedure. The only thing that changed was which months landed in which block, and the reported quality of the model moved by a factor of about eighty-nine.

How is the cut chosen, and what has to be said about it?

Three instructions, and a lender, an analyst or anybody reading somebody else's model result can apply all three from the outside without touching the arithmetic.

1. Cut along the order the record arrived in, so nothing later ever helps fit anything earlier.
2. Fix the three sizes before a single result has been looked at, and write them down where they cannot be quietly revised.
3. Report which cut was used and why it was chosen. A cut selected after the answers were visible is not a test of anything.

The third instruction is the one that gets skipped, so it is kept here. The test pair used throughout is months 9 and 10. Months 9 and 10 were chosen because their miss of 4.7418 is exactly the medianThe middle value once everything is lined up in order, so half the readings sit below it and half above. It is used here rather than the average because a single extreme reading drags an average around and leaves the middle value where it was. of all 45 two-month holdouts. A median cut is a typical one rather than a flattering one. Had months 4 and 6 been quoted instead, the reported miss would have been exactly nothing. Had months 2 and 3 been quoted, it would have been 10.6419. All three of those figures come from the same model on the same ten months, so quoting any one of them without naming the cut is publishing a choice while calling it a measurement.

Anybody on the receiving end of a result is left with a single question. Before a held-out figure is taken seriously, the thing to establish is how far it would move if the cut moved. If nobody in the room knows, the figure is one draw from a range nobody has looked at, and it is worth precisely as much as the range is narrow.