Cross Validation and Regularisation: Testing Honestly, and Penalising Complexity on Purpose

Three things are taken as already settled: how a straight line gets fitted to a record, what cutting a record into parts is for, and why two columns saying almost the same thing hand back a wobbly pair of numbers. The ten paired months are the same ten months set out under fitting a straight line to a record, decimal for decimal.

What problem is cross validation actually solving?

Without the problem in front of you, cross validation looks like extra work for no reason. So the problem comes first. Holding two of the ten paired months of the Nakshatra unit and the Vasant unit back, fitting a straight line on the rest, and scoring the line on the two months it never saw gives one number. A different pair held back gives a different number. Across all 1,260 ways of cutting ten months into six for fitting, two for checking and two for scoring, the score on the held outKept aside and not shown to the fitting step, so the model has to produce an answer for it blind. What the parts of a split are called and what each is for is worked through separately. months lands anywhere between 0.1865 and 16.5680.

The range from 0.1865 to 16.5680 is not a sign that somebody chose badly, and no amount of care in choosing one cut will close it. The same ten months, the same shape of model, the same arithmetic every time. All that moved was which two months got held back. When a record is short, the answer from one split is mostly a statement about the split.

The everyday version: judging how good a cook somebody is by tasting one dish. If the dish is their best one the verdict is impressed; if it is the one they were making for the first time the verdict is unfairly unimpressed. The fix is not to choose the dish more carefully. Nobody is in a position to choose well. Which dishes are which is not yet known. The fix is to taste several, and then to look at the whole set rather than at any one of them.

Two more numbers from those 1,260 cuts stop a very common overstatement, and are worth holding on to. Across all of them, the average miss on the months used to fit was 4.1784 and the average miss on the held-out months was 5.7340. So held-out months are worse on average. But the held-out months were worse in only 797 of the 1,260 cuts, or 63.25 per cent, and better in the other 463. A model does not always do worse on data it has not seen, and on this record it does better nearly two times in five.

How does leaving one month out at a time work?

Leaving one out is the purest version of the idea, and it is easier to describe than almost anything else in modelling. Fit the straight line on nine of the ten paired months. Ask it for the tenth. Write down how far it missed. Then put that month back, take a different one out, and do the whole thing again. Ten months means ten fits and ten predictions.

Every month gets a turn at being the one nobody trained on, and no month ever helps predict itself. That second half is the property that makes the resulting figure worth anything at all. In each of the ten rounds the model was built by a fitting step that had never seen the month it was about to be asked about, so the miss it records is a genuine miss rather than a memory.

The ten squared misses come out as 1.2346, 121.4494, 95.9600, 0.0000, 30.8642, 0.0000, 13.4944, 11.1111, 5.9975 and 30.8642. Their average is a mean squared missTake how far the rule was wrong in each period, square each of those, and average them. Squaring is what makes a big miss count for much more than a small one. How this measure compares with three other ways of scoring the same misses is taken up separately. of 31.0975, whose square root is 5.5765 and which sits back in the units the Vasant unit is measured in.

Two of those ten are exactly nought, and they deserve a second look: they carry the sharpest warning cross validation has to offer. Month 4 and month 6 are the two months the fitted line misses by nothing at all. With either of them taken out the line does not move, so when it is asked about the month it just dropped it hits it perfectly. A perfect score arrived at that way says nothing whatsoever about the model, and anybody quoting month 4 alone as evidence has quoted an accident.

How far out is the figure a fitting routine prints by itself?

Fit the straight line on all ten paired months of the Nakshatra unit and the Vasant unit, then score it on those same ten months, and the mean squared miss is 21.8000. A fitting routine reports 21.8000 as the in sampleScored on the very records the fitting step was given. The opposite of scoring on records kept aside. What the parts of a split are called is worked through separately. figure, printed in the summary the moment the fit finishes, without anybody asking for it.

The honest figure, from leaving one month out, is 31.0975. The in-sample figure sits 29.90 per cent below the honest one, and the honest one sits 42.65 per cent above it. Those two percentages describe the same gap read from opposite ends, and both are worth quoting because people reach for whichever one flatters the model.

Why does the gap exist at all? Because the fitting step had those ten months in front of it while it chose the slope and the intercept, and it chose them to make the misses on exactly those ten months small. Some of what it fitted was the relationship. Some of it was the particular wobble of these particular months. When the model is then scored on the same months, it gets credit for both. Score it on a month it has not seen and only the first part still helps. An in-sample figure is optimisticBetter than the truth, and biased in one direction rather than simply uncertain. An optimistic figure is not a wrong guess that could have gone either way. It leans the same way every time. by construction, not by accident, and on a record this short it is optimistic by a lot.

Five folds of two months gives 34.2305, higher still. The reason is worth stating plainly. Each of those five fits gets only eight months to learn from. Each leave-one-out fit gets nine. Less to learn from means a worse model, and a worse model misses further. The trade between months to learn from and months to test on sits behind every choice of how many folds to use.

What do the individual folds look like when they are opened up?

Cut the ten paired months into five folds of two, keeping the time order intact, so fold one is months 1 and 2, fold two is months 3 and 4, and so on to fold five with months 9 and 10. Each fold is scored by a model fitted on the other eight months. Five folds, five fits, five numbers.

The five come out as 66.7652, 49.5921, 16.5556, 15.7551 and 22.4846. Average them and the answer is 34.2305, the figure a summary would report. Now look at what that average is standing in for. The worst fold is 4.24 times the best, on one record, one shape of model and one unchanged run of arithmetic.

The five fitted slopes are just as restless: 1.3115, 1.7036, 1.4667, 1.5690 and 1.4471, against 1.5000 when all ten months are used. Drop two months from a ten month record and the line tilts noticeably. The shortness of the record is showing up in a second place.

Averaging several folds is the method itself, not a tidying-up step at the end. One fold is one split, and one split has already been shown to be a lottery ticket. Five of them do not make the noise vanish, but they do stop any single unlucky pair of months from deciding the whole answer. An average of five folds is worth quoting; any one of the five is not.

What is regularisation, and how is it different from cross validation?

Cross validation and regularisation get run together constantly, and running them together is the single commonest confusion about both. The line between them comes before anything else. Cross validation measures. Cross validation leaves the model exactly as it found it and changes only how the model gets scored. Regularisation changes the model. Regularisation adds a charge to the fitting step that grows as the fitted coefficients grow, so the fitting can no longer chase the smallest possible miss without paying for the size of the numbers it uses to get there.

Only one of the two ever touches the model, and it is regularisation. A cross validated figure and an unvalidated figure describe the same model. A penalised fit and an unpenalised fit are two different models that happen to have been built from the same record.

The everyday version is a shop deciding tomorrow's prices. Cross validation is checking last month's price decisions against what actually sold, in five separate weeks rather than one, so the verdict does not rest on the week the weather was strange. Regularisation is a house rule, imposed before any pricing happens, that no single cost may swing the price by more than a certain amount. The shopkeeper has learned that letting one item dominate produces prices that look clever and behave badly.

So why do the two turn up together in nearly every description? Because a penalty has to have a size, and nothing in the record says what that size should be. Somebody has to choose it, and the sensible way to choose is to try several sizes and see which one holds up on months the fitting did not see. Cross validation is how the size of a penalty gets chosen, and that is the entire connection between the two ideas. Beyond it they have nothing to do with each other.

What does a penalty buy, and what does it cost?

A penalty described in words sounds either obviously good or obviously arbitrary, and it is neither. So put a number on it. Bring in a third invented column, the Chandana unit. The Chandana unit nudges each Nakshatra reading by five hundredths of a percentage point in eight of the ten months and leaves it untouched in the other two. The two columns correlate at 0.999967. The two columns are, to any practical purpose, one body of evidence entered twice, and their variance inflation factorA score saying how much less confidently a coefficient can be pinned down because another input carries almost the same information. A value near one is untroubled; a very large value means two columns are close to interchangeable. It was built and explained in earlier reading. is 15001.

Fit the Vasant unit on both of them at once with no penalty and the pair of coefficientsThe numbers a fitted rule multiplies each input by before adding everything up. Settled by the fitting step, never chosen by hand. comes out as minus 1.0000 and 2.5000. Look at what that says. One of two nearly identical columns is being subtracted. The other is being added at two and a half times its size. The two together still add to 1.5000, the slope the Nakshatra unit alone would have carried. The fit could barely tell the two columns apart, so it produced an enormous pair that happens to cancel out.

Now add a penalty of one on the size of the coefficients and refit. The pair reads 0.7314 and 0.7661. The two numbers have closed from 3.5000 apart to 0.0347 apart, and the fitting step is no longer claiming that one column matters negatively and the other enormously. That is what a penalty buys: stability, and a pair of numbers a reader can look at without being misled.

The cost is measurable too, and a lot of writing on the subject overstates it. The R squaredHow much of an outcome's movement a fitted rule managed to track, put on a scale from nought to one. Where the measure comes from is settled in earlier reading and simply used here. falls from 0.756019 to 0.755950. It falls. The fit really does move, by 0.000069, and any account calling it unchanged has said something slightly false. The proportion is the honest way to put it: the coefficient moved 1.7314 and the fit moved 0.000069, roughly 24,974 times as far, so the honest claim is that a penalty buys stability at almost no cost in fit rather than at no cost at all.

What is the leak that makes a cross validated figure lie?

Everything above assumes one thing that nobody states, and stating it is the difference between a tested figure and a number wearing a costume. The assumption is that each fold's model learned from the months outside that fold and nothing else. Break that assumption and every figure above becomes decoration.

Here is how it breaks, and it almost never breaks on purpose. Somebody looks at all ten months, notices that the Nakshatra unit tracks the Vasant unit better than the alternatives, and picks it as the input. Then they run five fold cross validation on the model built from that input and report 34.2305 as a tested figure. But the choice of input was made with all ten months in view, including the two months sitting inside every fold. Each fold was scored by a model whose most important decision had already been made using the very months it was about to be tested on.

The same thing happens with a penalty. Choose the penalty size once, by looking at how the whole record responds, then run cross validation at that setting, and the folds are testing a model that was shaped by them. The size of a penalty is a decision about the model exactly as much as the choice of input is.

What makes the leak dangerous rather than merely wrong is that nothing looks different. The arithmetic completes. A number comes out. The number has the same shape, the same decimals and the same air of having been checked as an honest one, and it is optimistic by an amount that cannot be worked out from the output. The only defence is to do the choosing inside each fold instead of once before the folds begin. Each fold's model then rests on that fold's own outside months and nothing else.

What does an honest tested figure actually say?

An analyst reading somebody else's model, a lender deciding how much to lean on a scoring rule, and a researcher checking a colleague's work all do the same four things with a tested figure, and none of them takes long.

First, it names the scheme. Leaving one out, or a stated number of folds, cut in a stated way. Without that the figure cannot be rebuilt and cannot be compared with anybody else's. Second, it gives the spread across the folds and not only the average. The figures 66.7652 and 15.7551 average to a number that describes neither of them. Third, it says what was chosen before the folds began and what was chosen inside them, decision by decision. Fourth, it puts the in-sample figure beside the tested one.

The fourth habit is the least common and among the most useful. The gap between 21.8000 and 31.0975 is not clutter. The gap is information about how short the record is and how much of the fit was the particular wobble of these particular months. A tiny gap on a long record and a huge gap on a short one are two very different situations, and quoting the tested figure alone hides which of them applies.

What is left for later reading?