Overfitting and Underfitting: Two Ways to Get It Wrong, and Why You Cannot Minimise Both

Underfitting is a rule too crude to catch the pattern; overfitting is a rule that has memorised the record instead of the pattern. Fitted to six training months of the Nakshatra unit read against the Vasant unit, a flat line misses by 10.9570 and a curve of degree four by 1.7321. On the two test months those same two rules give 9.6817 and 7.3824.

Read those four numbers in the order they arrived and something uncomfortable happens. Going from the flat line to the curve, the first column improved enormously, from 10.9570 down to 1.7321. The second column got worse, from 9.6817 up to 7.3824 against a straight line that managed 5.8266. One change, two columns, opposite verdicts. Which column can be believed is the whole question, and the answer is never the column that improved.

Think of a caterer working out how much rice to cook for a wedding. One caterer cooks the same quantity every time, whatever the guest list says, and is wrong at a quiet weekday lunch for twelve and wrong at a five hundred plate reception in the same careless way. Another has written down exactly what happened at the last six weddings, including the one where it rained and half the guests stayed home, and cooks to match that record detail by detail. The first has a rule too crude to be any use. The second has a rule that fits six evenings perfectly and the seventh not at all. Underfitting and overfitting are those two failures, and the arithmetic below is the same story with the guesswork taken out of it.

What arrives here already settled, and what does the diagnosis add?

Four things are taken as read and none is explained a second time. How the ten paired months of the record, the Nakshatra unit set against the Vasant unit, were cut into six training months, two validation months and two test months, in time order, was settled earlier in this reading order. So was the meaning of a miss on months a rule learned from, and a miss on months held outKept back from the fitting step on purpose, so that a rule never sees those readings while its numbers are being chosen. What the rule then says about them is the only reading that was not arranged in advance. from it. So was the counting of parametersEach number that gets pinned down by the fitting step and then lives inside the finished rule. One for a flat line, two for a straight line, and that count climbing is precisely what people mean when they say a rule has been handed more room to move. Where such numbers come from, and what else a rule carries besides them, is dealt with elsewhere., set out in its own right elsewhere. And so was the argument that 146 of the squared miss on the whole record is simply unreachable, whatever is built on that input alone.

The diagnosis has three parts: two named failures, the exact signature each one leaves in the two columns, and the demonstration that the column most people watch cannot tell them apart. A reader who can fit anything at all can still climb a ladder of rising flexibility, watch a number fall at every rung, and report the worst rule on the ladder as the best one found.

Everything here is done on one fixed cut of one record. The six training months are always months 1 to 6 and the two test months are always months 9 and 10. The pair was chosen for being ordinary rather than convenient: of the 45 ways to hold two of the ten months back, months 9 and 10 land exactly on the median. Only one thing moves from here on: how much freedom the rule is given.

What is underfitting, and what does it look like on these six months?

Start with the crudest rule there is. Ignore the Nakshatra reading completely, and answer every month with one number. Fitted on the six training months that number is 4.1667, the average of their outcomes, and it is the answer for a month that read 11.00 per cent and equally the answer for a month that read minus 9.00 per cent. Month 2 actually returned 18.50, so the rule is out by 14.3333. Month 6 returned minus 13.00, so it is out by 17.1667. Square all six misses, average them, take the root, and the training miss is 10.9570.

Now ask the same rule about the two months it has never seen. The rule answers 4.1667 again, and that single number is all it can say. Month 9 returned minus 7.50 and month 10 returned minus 3.00, so it is out by 11.6667 and 7.1667, and the test miss is 9.6817. The two figures 10.9570 and 9.6817 are the signature of underfitting: both large, and no meaningful gap between them. The rule is not confused by unfamiliar months. The flat line is simply too crude to say anything useful about any month, familiar or not.

Because the absence of a gap changes what should be done about the failure, the point is worth slowing down on. A rule that is bad on the months it studied is not suffering from a shortage of months. Given forty months instead of six, this flat line will still answer every one of them with a single number, and it will still be wrong by roughly the same amount. A longer record cures none of it. The only thing that helps is letting the rule respond to the input at all, and a rule that already responds to the input has left this failure behind. The caterer who cooks the same quantity every time does not need a longer diary. The caterer needs to start looking at the guest list.

What is overfitting, and why do the two columns point opposite ways?

Go to the other extreme. A curve of degree four learns five numbers from six months, very nearly one number per month, and with that much freedom it can bend wherever it likes. Fitted on the training six it interpolatesRuns precisely through a given reading rather than near it, so the miss on that reading is nothing at all. A rule with roughly as many numbers to set as there are readings to satisfy can usually arrange this, and arranging it is not the same as being right about anything else. four of them: month 2 at 18.50, month 3 at 2.50, month 4 at 17.00 and month 6 at minus 13.00, each hit dead on. The training miss collapses to 1.7321 and beats the straight line by 64.49 per cent.

Then the two test months arrive. Month 9 read minus 4.00 on the Nakshatra unit. Month 3 read minus 4.00 as well and returned 2.50, so the curve was built to answer 2.50 at that reading. Month 9 actually returned minus 7.50. The curve is out by 10.00 percentage points on a reading it has already met. Month 10 read 1.00 and the curve answers nothing, against an actual minus 3.00, so it is out by 3.00 more. The test miss is 7.3824 and loses to the straight line by 26.70 per cent.

The two columns disagree about the same rule, and that disagreement, not either figure on its own, is the reliable sign of overfitting. A curve that improves by 64.49 per cent on one column while giving up 26.70 per cent on the other has not found more of the pattern. The curve has learned the particular months in front of it, wobble and all, and the wobble does not repeat. The caterer who cooks to the record of the rainy evening will be wrong on every dry one.

What does the whole ladder from degree nought to degree four show?

Two cases could be two accidents, so the whole ladder is climbed. At each rung the rule is fitted on the six training months alone, then asked about the two validation months and the two test months without being allowed to adjust. The shape of the held-out column is the finding.

Read the third column top to bottom. 10.9570, then 4.8774, then 4.8559, then 4.8490, then 1.7321. The training miss falls at every rung without exception, and because it falls at every rung it distinguishes nothing: the same downward march covers the best rule on the ladder and the worst one. Some of those falls are enormous and some are minute. From degree one to degree two the training miss improves by 0.0215, and from two to three by 0.0069, a change in the third decimal place. Both are still falls, and a reader watching only that column has no way to know they were rounding crumbs.

Now read the last column. 9.6817, then 5.8266, then 6.1654, then 6.0327, then 7.3824. The test miss drops sharply once, at degree one, and never improves on that figure again. The turn is the information. The validation pair ranks the rungs differently again, preferring the flat line at 4.0087 to everything else. Two months are worth about that much as a jury, and the ranking is argued out under cross validation.

Why is a falling training miss not progress?

Because it cannot do anything else. Every extra number the rule is allowed to set is one more way for it to bend toward the months it can see, and the fitting routine will use that freedom only if using it helps. If the new freedom is worthless the routine sets the new number to nothing and the training miss stays exactly where it was. If the new freedom helps even slightly, the miss falls. There is no third possibility. A training miss can never rise when a knob is added, whatever the knob is, so a fall in it is not evidence about the knob at all.

The ratchet is why the argument from improvement sounds so convincing and means so little. Somebody adds a term, the training miss drops from 4.8774 to 4.8559, and reports an improvement. The improvement is real, in the sense that the arithmetic is correct. The improvement is also completely uninformative. A term chosen at random, a term with no relationship to anything, a term that is pure decoration, would have produced a fall too, or at worst a fall of nothing.

The everyday version is a student who marks their own homework with the answer sheet open. The score goes up every time and the score means nothing. The only thing being measured is how thoroughly the answers were copied. The examination is the held-out column, and it is the only reading anybody should quote.

What stops the training miss at 18, however flexible the curve gets?

Flexibility runs out, and it does not run out because the arithmetic gets tired. The limit comes from the record, and the record refuses to cooperate. Among the six training months there are only five distinct readingsDifferent values of the input, counted once each however many months carry them. Six months holding the readings 1, 6, minus 4, 11, 1 and minus 9 carry five distinct readings, because two of the months read the same thing. on the Nakshatra unit: 1, 6, minus 4, 11 and minus 9. Months 1 and 5 both read 1.00 per cent. Month 1 returned 3.00 and month 5 returned minus 3.00.

A rule that reads only the Nakshatra unit cannot tell those two months apart, so it must give them the same answer. The best single answer for them is nothing at all, halfway between, and it leaves each one out by 3.00, so 3 squared twice over is 18. So the squared miss on the six training months stops at 18, and no amount of extra flexibility can go under it. Even total freedom runs into the record rather than into the arithmetic. Averaged over six months and rooted, 18 becomes 1.7321, and that is exactly the figure degree four already reports. Degree four has not merely fitted well. Degree four has hit the wall.

The same fact ends the ladder. A curve of degree five needs six numbers, and six numbers cannot be settled from five distinct readings; the equations that would set them are singularA set of equations with no single answer, because two of the demands made on it are really the same demand written twice. Nothing has gone wrong with the calculation. There is simply not enough separate information to pin every number down.. Singularity is not a breakdown so much as the record saying there is nothing left to attach a sixth number to. Because it names the limit precisely, singularity is a far more informative kind of failure than a slightly worse score.

What are the two things being traded against each other in the bias-variance tradeoff?

Put plainly, and in the words used from here on. The first quantity is how far the rule's shape is from what is actually there. A flat line has a shape that cannot follow anything, so this quantity is enormous for it. A straight line follows a great deal more. A curve follows more still. The distance between the rule's shape and what is there falls as the rule gets richer, and it falls whether or not that is convenient.

The second quantity is how much the fitted numbers move from one draw of the record to another. A straight line fitted on four months has a slope that depends heavily on which four; fitted on nine, it depends much less. The movement in the fitted numbers rises as more of them are fitted out of the same fixed record. Each number is being pinned down by a smaller share of the evidence.

Both sides have already been measured on this record rather than asserted. On the falling side: the straight line leaves a squared miss of 218 on all ten months, of which 146 sits outside what anything built on that one input could ever recover, so exactly 72 is what the straight shape costs. Give the rule a better shape and that 72 is what is on offer, and no more. On the rising side: fit a single slope on every possible four months of the record and the slopes range over a spread of 0.6718; do the same on nine months and the spread is 0.0811. Fewer numbers pinned by more record move less.

One of those two falls as the other rises. No setting anywhere on the ladder makes both of them small, and the choice is therefore a judgement rather than a calculation. The sum of the two has a bottom somewhere in the middle, and finding roughly where that bottom sits is the entire job. The choice has the same shape as deciding how detailed a household budget should be. One line for everything tells nothing; a separate line for every packet of biscuits tells something different every month and never the same thing twice.

Is a knob that does nothing the same as a knob that hurts?

No, and treating them as the same thing leads to a superstition about complexity that the record here can settle. Take all ten months this time rather than the training six, and fit a cubicA curve carrying a term in the reading multiplied by itself three times, on top of the squared term and the plain one. It is simply the next rung of flexibility above a single bend, and it allows a second bend in the shape. to them. The fitting routine sets the cubic term to exactly nothing.

The consequence is worth stating carefully. A curve of degree two on all ten months leaves a squared miss of 211.33 and an R squared of 0.7633. A curve of degree three leaves a squared miss of 211.33 and an R squared of 0.7633. The same figures, not similar figures. The extra term was set to nothing, and a term of nothing changes nothing. Extra flexibility is not automatically damaging; here it was simply left unused, and it cost neither fit nor anything else.

Which makes the shape of the harm on this record worth noticing. The harm does not accumulate quietly, a little worse at each rung. It arrives. Degrees two and three are almost exactly where degree one is on the held-out column, within a few tenths. Degree four is a different animal, and that is where the damage lands. So the honest summary is not that every added term costs something. The honest summary is that added terms cost nothing until the rule has enough freedom to start chasing the particular months in front of it, and then they cost a great deal at once.

How is the stopping point decided?

Four habits, and they are what somebody working with a record like this actually does rather than what a textbook says.

Watch the held-out column and never the training column. Write both down by all means. The pair of them together is the diagnosis, but let only one of them decide anything. The training column on this ladder answers degree four, and degree four is the worst rule on offer.

The training figure never stops improving, so stop where the held-out figure stops improving and not where the training figure does. On this ladder the held-out figure improves once, from 9.6817 to 5.8266 at degree one, and then never again. Degree one is the stopping point, and it is chosen on evidence rather than on taste.

Prefer the simplest rule whose held-out figure is not clearly worse. Degree three reports 6.0327 against degree one's 5.8266, close enough that on a longer record the ranking might well swap. Degree one wins the tie anyway, having two numbers to be uncertain about rather than four, and every number left unfitted is a number that cannot move.

Say out loud what the record could not support. Six months carrying five distinct readings were never going to settle six numbers, and it is more useful to report that than to report a score. Fixing the ladder in advance and agreeing which column decides also stops a single lucky cut from being mistaken for a result, a job belonging to cross validationScoring a rule on several different cuts of the same record and averaging what comes back, instead of trusting one cut. It is treated in full elsewhere., covered separately.