Regression: Fitting a Line and What It Claims

Regression fits one straight line through paired observations so that the sum of the squared misses is as small as it can be. On ten invented months the Vasant unit moves 1.50 percentage points for every one point the Nakshatra unit moves, starting from 0.50. The line explains 75.59 per cent of the movement and leaves 24.41 per cent it cannot touch.

Two columns of numbers land on a desk, and there is really only one thing worth asking about them. When this one moves, what does that one do? Regression replies with a single straight line and the pair of numbers that fix where that line sits. Ten months are enough to build that line and read every figure it produces, and few enough to count on the fingers. A perfectly respectable looking number can still miss something those ten months contain, and what it misses is the harder half of the subject.

Two invented objects carry all of it. The Nakshatra unit and the Vasant unit are traded unitsAnything with a price on it that moves from one month to the next. Here the words label two invented objects, so nothing anybody could actually buy is meant.: things with a price attached that move about from one month to the next. Neither exists, neither stands in for anything that does, and both were built so the arithmetic would close without rounding. The record is ten months of each unit as a monthly changeHow much a price moved across one month, stated against where that price started. Ten of them, in a row, are the whole record in this guide.. A monthly change is the move across one month stated against where that month started. Set side by side, they form ten paired observationsTwo figures recorded about the same moment, so they can be set against each other. Break the pairing and every method here stops working.: the Nakshatra figure and the Vasant figure from the same month, tied together. Ten pairs is a very small sampleThe part of something actually observed, as against all of it. Ten months is a sample; every month there has ever been is not., and that smallness bounds everything that follows.

The ten months are printed in time orderThe order the months actually occurred in, oldest first. It is information in its own right, and sorting the rows destroys it for good. and they stay in it. The order is evidence, and sorting the rows destroys it. Keeping the order will look like fussiness for a long stretch. Almost every number below can be worked out with the rows shuffled into any arrangement at all. Then the order turns out to be carrying the one thing none of those numbers could report.

Where the figures came from, said once and up front. Nowhere. Ten pairs of numbers were chosen so that a line through them would land on figures a reader can check by hand, and every result below is arithmetic on those twenty numbers.

What does a fitted line actually claim?

The answer is much narrower than people assume. A fitted line makes exactly one claim, and it is this: across the record it was fitted on, for every one unit the input moved, the outcome moved by the slope, on average. Nine words of that sentence are doing work and the two that do the most are average and across the record.

Take an everyday version first. Ten shops sit in one shopping centre, and somebody fits a line through their opening hours and their takings. The line comes back saying that takings run about Rs 1,500/- higher for every extra hour a shop stays open. The sentence is a true description of those ten shops. The line does not say that the eleventh shop will earn Rs 1,500/- more if it stays open an hour longer, it does not say that staying open causes takings, and it says nothing about any shop in any other centre. A summary of ten shops is all it is, and the moment it is used for anything else a claim has been added that the arithmetic never made.

So the line here claims that across these ten months, on average, the Vasant unit moved 1.50 percentage pointsPercentages subtracted from one another rather than divided into one another. The distance between 4 and 6 on a percentage scale is two of them, whichever proportion of the first figure that happens to be. for every one point the Nakshatra unit moved, and it claims nothing else at all. Not that the eleventh month will behave that way. Not that the Nakshatra unit causes anything whatever. Causation is a separate question, covered under correlation and causation. Not that any particular one of the ten months behaved that way: two months below did and several did not. Average, across the record. The claim ends there.

Where do the two numbers in the line come from?

Before the two numbers, one word. Everything that follows is built from it. A residual is what the line missed by in a single month: the figure that actually turned up, less the figure the line said. Ten months means ten residuals. The word miss is used interchangeably with it below, and they mean exactly the same thing.

So which line? An infinity of straight lines can be drawn through a cloud of points, so a rule is needed to pick one, and the rule is this: take every residual, square it, add the ten squares up, and choose the line that makes that total as small as it can be. Squaring does two jobs at once. Squaring stops a miss of plus 5 cancelling a miss of minus 5, and it makes a large miss count for far more than a small one. Hold on to that second job. The second job comes back immediately below, and it is a choice somebody made.

On these ten months that rule gives a slope of exactly 1.5000 and a starting point of exactly 0.5000, and neither figure is a rounding of something untidy. The slope is the joint movement of the two columns, 450, divided by the movement of the input column on its own, 300, and 450 over 300 is 1.5 with nothing left over. The starting point is then the mean of the outcome less the slope times the mean of the input: 2.00 less 1.50 times 1.00 gives 0.50. Both numbers fell out of the same ten months. Nobody picked either of them, and nobody could have picked them differently without breaking the rule. How that arithmetic is derived, and the conditions it quietly rests on, are covered under ordinary least squares. Each of the two numbers makes its own claim, set out under regression coefficients.

Look at the scatterA drawing that puts one dot per pair, the input reading along the foot and the outcome up the side, so the shape of the relationship is visible. above and notice two things the summary numbers will hide. Months 4 and 6 sit exactly on the line: the fit missed those two by nothing at all. Month 2 sits 9.00 points above it, the worst the line does anywhere. And a line drawn to minimise squared misses is not a line drawn to pass near everybody. The least squares line is drawn to avoid one catastrophic miss, and it will happily accept several ordinary ones in exchange.

What is a miss, and why do Mean Squared Error and Mean Absolute Error disagree?

There are ten misses. Somebody wants one number that says how badly the line does. There are three ways in wide use, they are computed from the identical ten residuals, and they give three different answers.

The first is the Mean Squared Error. Square every residual, add the squares, divide by ten. Here that is 218 over 10, or 21.80. Now look at the units. The trouble starts there. The residuals are in per cent, so their squares are in per cent squared, and 21.80 per cent squared cannot be laid against a monthly change of 6.50 per cent and compared. The two figures are not the same kind of thing. The number is useful for ranking one line against another and it is close to meaningless read on its own.

So the second measure fixes exactly that. The root mean squared error is the square root of the first: the square root of 21.80 is 4.6690, and that figure is back in per cent, the outcome’s own units. The square root allows a sentence a person understands: on a typical month this line is out by something in the region of 4.6690 per cent.

The third takes a different road entirely. The Mean Absolute Error throws away the direction of each miss, keeps its size, and averages those. The ten sizes are 1, 9, 8, 0, 5, 0, 3, 3, 2 and 5. They add to 36, so the mean absolute error is 3.60 per cent. The mean absolute error is also in the outcome’s own units, and it is a full point smaller than the root mean squared error computed from the identical residuals.

The two are not two roads to one answer. The two measures are two answers to two different questions, and the gap between them is the price of squaring. Watch it on month 2. Month 2 misses by 9.00, the worst month there is. Of the mean squared error, month 2 carries 37.16 per cent, better than a third of the whole thing. Of the mean absolute error it carries 25.00 per cent, exactly a quarter. Same month, same miss, same ten residuals. One measure holds it responsible for half again as much as the other does.

Squaring is therefore a decision about how much a bad month ought to count, and it is not a neutral default just because it is the common one. If one dreadful month is genuinely far worse for the analyst than three mediocre ones, squaring says what is meant. If three mediocre months hurt in proportion, it does not. And there is a counterweight worth stating so nobody overclaims from this: on these particular ten months the line that would minimise the average absolute miss is the very same line, 0.50 plus 1.50 times the input. The two measures disagree about which month deserves the blame and they happen to agree here about which line wins.

What is R squared measuring, and against what?

R squared has a reputation for being a mysterious quality of the line, a kind of grade out of one. R squared is nothing of the sort. R squared is a comparison, and it is a comparison against the single laziest rule anybody could propose.

Here is that rule: ignore the Nakshatra unit entirely, ignore everything, and every month guess the mean of the Vasant unit, 2.00 per cent. The lazy rule misses too, and squaring its ten misses and adding them up gives 893. The 893 is the number to beat, and it is the only reason R squared means anything.

R squared is one less the fitted total divided by the baseline total, so here it is one less 218 over 893, or 0.7559, and every part of that sentence has to survive for the number to mean anything. Read it as a ratio and it becomes plain: 675 of the 893 squared units of miss were removed by using the line instead of the lazy rule, and 675 over 893 is 0.7559 to four places. So 75.59 per cent of the movement in the Vasant unit is accounted for by the Nakshatra unit across these ten months.

Which is why an R squared quoted with no baseline named is close to unreadable. The question is always: better than what? Here the answer is better than guessing 2.00 per cent every month, and that is a low bar somebody chose. A different baseline gives a different number from the identical line.

Two things are worth noticing about the slider. First, the bottom of that curve is flat, so the first tenth of a point of movement costs 3.00 and the sixth costs more than eight times as much: the penalty grows with the square of the distance strayed, and that is squaring showing up again in a new place. Second, no setting anywhere on the slider ever gets below 218.00. The floor is not a property of the slider. A floor at 218.00 is what being the least squares line means.

What is the part R squared leaves out, and why is it called Idiosyncratic Risk?

One finance word first, defined before it is used. The word is used loosely everywhere, and the loose meaning misleads here. Risk, in this guide, means how much something moves about. Nothing more. Not danger, not the chance of losing money, not anything anybody should feel. Just movement, measured as spreadHow widely a set of figures is scattered about its own average. Built and measured separately; used here without rebuilding it..

So the Vasant unit moves about. Across these ten months that movement adds up to 893 in squared terms. The Nakshatra unit accounts for 675 of it. The remaining 218 is 24.41 per cent of the whole, and it is the Idiosyncratic Risk: the part of the Vasant unit’s movement that is particular to the Vasant unit and about which the Nakshatra unit has nothing to say.

The leftover quarter is not error in the sense of a mistake, and reading it as one is the single commonest misuse of the word residual. Nobody made a slip. The line is the best line there is. The 24.41 per cent is simply everything one input was never going to be able to carry. The leftover would still be there if the arithmetic were done perfectly on a machine that never rounds, and that is roughly what happened here.

The household version makes it concrete. A household runs on one salary, and its monthly spending tracks that salary closely. Fit a line and most of the spending is accounted for. The rest is a wedding among relatives, a hospital week, a school fee that falls in one month and not the next. None of that is a failure of the salary to explain spending. The extra spending was never about the salary in the first place. The idiosyncratic part is exactly that, and here it is nearly a quarter of everything.

Say it in money and it stops being abstract. On an invented holdingAn amount of money a person has placed in one thing. Used here only to turn a percentage into rupees somebody can picture. of Rs 2,00,000/-, a typical monthly miss of 4.6690 per cent is about Rs 9,338/-, the average absolute miss of 3.60 per cent is Rs 7,200/-, and the worst single month, that 9.00 per cent in month 2, is Rs 18,000/-. A residual is worth that much in rupees only against the amount it is set beside, so the same 4.6690 per cent on half the holding is half the money.

Why can plain R squared never fall, and what does Adjusted R-Squared do about it?

Here is a fact about plain R squared that sounds like a technicality and is actually the whole problem. Whatever that input is, the worst a fit can do with a useless one is give it a weight of zero and carry on exactly as before, so adding an input can never lower plain R squared. A rise in R squared after adding something is therefore not evidence that the something helped. A rise was the only thing that could have happened.

Rather than assert that, watch it. A third invented object, the Chandana unit, is the Nakshatra unit nudged by five hundredths of a percentage point in eight of the ten months and left untouched in the other two. The Chandana unit carries almost no information that the Nakshatra unit was not already carrying. Add it as a second input and refit.

R-Squared vs Adjusted R-Squared, watched on one added input

Plain R squared goes from 0.7559 to 0.7560. The rise is one unit in the fourth decimal place, and the fit had no choice about it. Adjusted R squared goes from 0.7254 to 0.6863. The adjusted figure fell by 0.0391, and that fall is 391 times the size of the rise. The difference between the two measures is that the adjusted one charges the fit for every input it consumes: it divides the leftover by the number of observations less the number of things fitted, so an input that brings nothing costs more than it brings and the figure drops.

So the rise proved nothing at all and the fall is the honest signal, and any report that quotes the first without the second has quoted the one number that could not have gone the other way. A small gap between the two measures looks like this. On one input and ten months the gap here is 0.0305, the distance from 0.7559 down to 0.7254. The gap of 0.0305 is a property of this particular fit on this particular record and of nothing else whatsoever. A gap is not a rate, not a constant, and not related to any other quantity that happens to look like it. With a second input the charge for inputs goes up, so the gap widens to 0.0697.

One boundary, at exactly the place where the subject sprawls. Methods that penalise a fit for the number of inputs it uses, and methods that judge a fit by how it does on data it never saw, are a separate body of work with its own assumptions, and they are covered separately in the notes on building models. Adjusted R squared is not a stand in for one.

What does a respectable looking number still not show?

Every figure so far has one thing in common that nobody points out. Not one of them can see the order of the months. The slope, the starting point, the total squared miss, all three error measures, R squared, adjusted R squared: shuffled into any arrangement at all, the ten rows return every one of those numbers identical. Every one of those figures is built by adding things up, and addition does not care what came first.

Now put the misses back in the order they happened and look at them. Month 1 is above the line, month 2 is above it, month 3 is above it. Months 7, 8, 9 and 10 are all below it. Of the nine consecutive pairs in the record, five sit on the same side of the line as the pair before them.

The good looking number and the evidence against it come out of one dataset, and no summary figure above can hold both. Every one of them throws the order away. An R squared of 0.7559 reads as a solid fit. The same ten months, read as a sequence rather than a heap, say the misses are not landing independently of one another. When a line is missing in runs like that, the misses are carrying structure the line did not capture, and the usual figures quoted alongside the fit are computed as though that could not be happening.

Settling that needs a measure built for the purpose. The measure, and what it says about these exact ten misses, is covered separately under autocorrelation and partial autocorrelation. Both facts hold at once: 0.7559, and five runs out of nine.

What should be asked of any fitted line?

Fitted lines arrive in other people’s work constantly. A lender fits one through a borrower’s monthly receipts and something the lender can observe, so a shortfall in one month can be read against what was expected rather than against nothing. An analyst fits one to see how much of a thing’s movement is shared with everything else and how much is its own. A household planner fits one through a year of electricity bills and the temperature, so an unusual bill can be told apart from a hot month. In every case the person reading the output is not the person who fitted it, and six questions separate a reader who understands what arrived from one who does not.

Ask the units of the slope first, and say the sentence out loud with them in it. Percentage points of the Vasant unit for one point of the Nakshatra unit: that sounds like something. Rupees per hour, defaults per point of interest rate, units per degree. If the sentence does not survive being said out loud, the number is not being read, it is being quoted. Then ask how many observations. Ten, here, and ten is very few: a slope built on ten months would move a great deal if an eleventh arrived. Then ask the baseline, then the size of what is left over stated in units a person cares about, then how many inputs went in and what the adjusted figure did when the last one was added.

Then ask the sixth question, the one almost nobody asks: what do the misses look like in the order they happened? The sixth is the only one of the six that no summary number answers, it takes one drawing to check, and on these ten months it is the one with the uncomfortable answer.

Can every number be checked from one table?

Yes, and the table below is printed so that it can be done. Everything above came out of the twenty figures in the first three columns below. The rest of the table is worked from them, and the two totals rows are the arithmetic closing: the misses add to exactly zero, as they must for any least squares line with a starting point in it, and their squares add to 218.

And the summary figures, each with the single line of arithmetic that produced it, so nothing has to be taken on trust.