Ordinary Least Squares: The Method and Its Assumptions

Ordinary least squares picks the one line, out of every line that could be drawn, whose squared misses add to the smallest total. On ten invented months that total is 218 and the line is 0.50 plus 1.50 times the input. The answer is not searched for but computed, and the computed line always leaves misses that add to zero and carry no relationship to the input.

A fitted line has already been established: 0.5000 plus 1.5000 times the input, ten paired months behind it, a sum of squared misses of 218, and an R squaredThe single headline figure for how much of the outcome's movement a fitted line accounts for, running from nothing to one. of 0.7559. Left unsaid was why that particular line and not one of the infinitely many others that could have been drawn through the same ten dots. Somebody made a choice. The choice buys something, and it carries six conditions under which the resulting number means what it looks like it means.

Everything below runs on two made up objects. The Nakshatra unit is a traded unitA stand in for any object with a price attached, so that a stretch of its life can be written down as a single figure. whose monthly changeHow far a price travelled across one month, set against where it stood when the month opened. Rs 200/- climbing to Rs 206/- across a month gets written down as 3.00 per cent. is the input. The Vasant unit is a second one, and its monthly change is the outcome the line is trying to land on. Ten months, ten paired observationsOne row holding two readings taken from the same occasion, so the two can be lined up against each other. Ten months here give ten rows, and nothing may be shuffled between rows., kept in the order the months happened.

What does ordinary least squares actually do?

Ordinary least squares does one thing, and the whole method fits inside a single sentence. Out of every straight line anybody could draw across those ten dots, ordinary least squares hands back the one for which the misses, squared and added up, come to the smallest total available. The specification stops there. Not the truest line, not the most sensible line, not the line an expert would draw. The line with the smallest total of squared misses, and nothing about the arithmetic argues that this is the total worth making small.

Here is the everyday version. A caterer works twelve weddings a season and wants a single rule turning the guest count into kilograms of rice to cook. Whatever rule gets written down, it will be wrong at every wedding by some amount. Two rules cannot be compared until each of them has a score, so somebody has to decide how those twelve wrongnesses combine into one. Suppose the decision is that being out by four kilograms counts sixteen times as badly as being out by one. The sixteen to one ruling is a decision. A person took it, it could have gone the other way, and once taken it crowns a different rule than a gentler decision would have crowned. Least squares takes exactly that decision, and then never mentions it again.

So the honest way to read a fitted line is as the winner of a competition whose scoring rules were fixed in advance. Change the scoring and a different line wins. To see that the competition is real, hold the line's shape and slide the slope. At a slope of 1.30, refitting the starting value to 0.70 so the line still passes through the middle of the record, the ten squared misses add to 230.00. At 1.40 with a starting value of 0.60 they add to 221.00. At 1.50 with 0.50 they add to 218.00. Then the total climbs back: 221.00 at a slope of 1.60 and 230.00 at 1.70.

A slope of 1.50 is the bottom of that list, and moving one tenth in either direction costs exactly 3.00 in squared misses, the same 3.00 going up as going down. The symmetry is not a coincidence and it is not rounding. The total is a perfect bowl in the slope, and its floor sits at 1.50. The walls of the bowl climb with the square of the distance from its floor, so moving two tenths costs 12.00, four times three.

Why squared misses rather than plain ones?

The method never answers the question out loud, and it is worth asking directly. There are ten misses. Some are above the line and some below. The ten misses have to be turned into one number. Why square them?

There are three answers and they are not the same kind of answer. Two of them are judgements somebody made and only the third is mathematics.

The first reason is that squaring stops the misses cancelling. The ten misses as they stand add to zero, every single time, for reasons set out below. A total that is always zero cannot rank two lines. Squaring turns every miss into a positive contribution, so a line that is wrong in both directions is scored for being wrong in both directions. The cancelling reason, though, does not require squaring specifically. Dropping the sign and adding the sizes does the same job. So the first reason rules out the raw total and does not pick squaring out of the field.

The second reason is that squaring makes a big miss count more than proportionately, and that is a value judgement wearing a formula. A miss of 9.00 per cent is three times a miss of 3.00 per cent. Square them and the first contributes 81 while the second contributes 9, nine times as much. Whether one terrible month should count nine times a mediocre one, or three times, or twenty seven times, is a question about consequences in the world. The rent is due either way, so a household with one salary treats one month of no income as far worse than three months of income cut by a third, and it is right to. A vegetable seller with a stall outside one office building might feel the opposite. A series of thin weeks is what actually empties the cash box. Neither view is a fact about arithmetic. Squaring picks one of them and stops asking.

The third reason is the only one that is mathematics rather than opinion: squaring makes the answer come out in closed form, and the alternatives do not. Scored by squares, the winning line can be written down in two divisions, worked below. Scored by plain sizes, there is no such formula: the answer has to be hunted for, and on some records more than one line ties for the win. Closed form is why least squares became the method everybody learns first. Convenience of solution is a real reason and not a shameful one, though it is not a reason about the world.

The honest counterweight has to come next. On this particular record of ten months, the rule that makes the average size of a miss smallest lands on precisely the line least squares already chose, at an average miss size of 3.60 per cent. Two criteria that disagree in principle about which month matters have agreed here about which line wins. The two rules disagree about how to weigh the months and they happen to agree on this record about which line to pick, and both halves of that sentence have to be said. On a different ten months they can and do part company.

How does the method find the answer instead of hunting for it?

The bowl in the first figure invites a picture of somebody trying slopes one after another and keeping the best score. Trying slopes one after another is not what happens, and the difference matters more than it sounds. The winning line is computed in two divisions, from quantities read straight off the ten months, and no candidate line is ever tried.

Step one is the slope. Take how much the two units move together, measured as the paired departures from their own averages multiplied and added. On this record that comes to 450. Take how much the input moves on its own, measured as its departures squared and added. The input's own movement comes to 300. Divide: 450 over 300 is 1.5000, exactly. Step two is the starting value. The winning line always passes through the point made by the two averages, so put the input's average of 1.00 into the line and demand that it return the outcome's average of 2.00. The demand means 2.00 less 1.50 times 1.00, exactly 0.5000.

Both figures land on a clean decimal because the twenty numbers were composed to make them land there. On a measured record they would not, and the arithmetic would be identical. The exactness is worth taking one thing from: no estimateA figure worked out from the rows available that stands in for something that cannot be observed directly. What estimation costs, and how far an estimate can be trusted, are covered under sampling. on this record came out of a search that might have stopped in the wrong place. Every one came out of a division.

What two conditions does the answer always satisfy?

Take the fitted line back to the ten months and write down what it missed by each time. The fitted valueWhat the line says the outcome should have been at a given input, worked out before the actual reading is set beside it. for month one is 0.50 plus 1.50 times 1.00, or 2.00, and the outcome was 3.00, so the miss is 1.00. Do that ten times and the misses are 1.00, 9.00, 8.00, 0.00, minus 5.00, 0.00, minus 3.00, minus 3.00, minus 2.00 and minus 5.00.

Two things are now true about that list, and they are true of every ordinary least squares fit ever performed, on any data whatever.

First, the misses add to exactly zero. One plus nine plus eight plus nothing, less five, plus nothing, less three, less three, less two, less five. The positives come to eighteen and the negatives come to eighteen. Zero. Second, the misses carry no relationship to the input at all. Weight each miss by how far its month's input sat from the input's own average of 1.00 and add those up, and that also comes to zero. Neither of these is luck and neither is evidence. Both are consequences of how the line was picked: the two divisions in the previous section are precisely the arithmetic that forces them.

Now the trap that both conditions set. Both conditions are properties of the method rather than findings about the record, so both hold even when the straight line is completely the wrong shape for the data. Take seven readings whose scatterA plot with one dot per row, placed left to right by the first reading and up and down by the second, so the shape of a relationship can be looked at rather than summarised. forms an obvious U. Fit a straight line by least squares and it comes out perfectly flat, sitting at the average outcome of 4.00, a visibly wrong description of a U. Its misses are 5.00, 0.00, minus 3.00, minus 4.00, minus 3.00, 0.00 and 5.00. The seven misses add to zero. Weighted against the input they also add to zero. Both conditions pass with full marks on a fit that any eye can see is wrong.

What does the method assume about the data it is handed?

Six conditions. Not one of the six is needed for the arithmetic to run, and that is the source of nearly all the trouble. Hand ordinary least squares any two columns of numbers at all and it will return a slope and a starting value without complaint, so the assumptions are conditions for the answer to mean what it appears to mean, never conditions for getting one.

The middle column and the right column read as a pair. Each condition is a claim about the world that the numbers alone cannot settle, and each has an artefact that can actually be looked at. The last one only bites when there is more than one input, and it is demonstrated separately. The first four are checkable immediately on any record, and checking them is looking rather than computing.

Which of those assumptions do these ten months break?

The third one, and there is no need to hedge about it. Put the ten misses back into the order the months happened, the order they have been in all along, and they do not look like ten independent surprises. The ten misses look like a story. The first three months miss above the line, in a row. Then a miss below, then an exact hit, then four misses below the line, in a row. In five of the nine consecutive pairs, both months land on the same side.

The claim deserves a figure, so here is one. The ten misses carry a lag one autocorrelationOne figure for how strongly each entry in an ordered list resembles the entry immediately before it. How it is worked out, and what to do when it is large, are covered under autocorrelation. of 0.4862, and R squared for the same fit on the same ten months is 0.7559, a perfectly respectable looking result. The 0.4862 is computed under autocorrelation, and is quoted here so that the two numbers stand next to each other.

Why can R squared not see any of it? Because computing R squared throws the order away. R squared adds up squared misses, and addition does not care which month came first. Shuffled into any arrangement at all and recomputed, the ten months give 0.7559 every time. The only evidence against the fit lives entirely in the ordering, and the headline number is computed in a way that destroys the ordering before it starts. The blindness is not a flaw in R squared. R squared was never asked about the order. The flaw is in reading R squared as though it had been.

The record breaks a stated condition, and the fit's own headline figure is structurally incapable of noticing. How big a problem the dependence is, and what anybody does about it, is set out under autocorrelation.

What is ordinary least squares not?

Four denials, and the method is stronger for each of them.

Least squares is not a test of anything. Fitting a line returns two numbers and passes no verdict. Whether a slope of 1.5000 is distinguishable from no relationship at all is a separate question with separate machinery behind it, and the fit does not answer it by existing. Least squares is not a method for choosing which inputs to use. Hand it a column of anything and it will fit that column. The decision about which columns deserve to be in the room is made by a person, before the arithmetic runs, on grounds the arithmetic has no access to.

A fitted line is not a forecast. The line describes the ten months it was fitted on. Reading it as a statement about a month that has not happened adds a claim the fitting never made, and that claim needs its own defence. And it is not a method that shrinks a fit, penalises a fit, or resamples a record to pick among several fits. Shrinking, penalising and resampling all exist and are useful. All three are a different subject with their own conditions, covered under model building.

What should be checked before a fitted line is quoted?

Checking is the part a lender, an analyst or anybody with a spreadsheet actually does, and it takes about ten minutes. The cheapest checks are the least informative, and doing them first creates exactly the false comfort the two closing conditions invite, so the order matters.

The first check is that the misses add to zero, and what it establishes should be discounted immediately. The zero is a receipt that the arithmetic ran, in the way that a till receipt confirms the till worked rather than confirming the shopping was needed. Next comes plotting the misses against the input and looking at the shape. A formless cloud is what a satisfied straight line assumption looks like; an arch or a bowl means the shape is wrong and no amount of refitting a line will fix it. Then the misses are plotted against the clock, in the order the rows arrived, and examined for runs on one side. The clock plot is the check this record fails.

Then, if there is more than one input, check whether any two of them move together almost perfectly. A near copy destroys the individual coefficients while leaving the overall fit looking untouched. And last, the one people skip: record which condition went unchecked. The condition nobody checks is the one that fails. A sampleThe rows actually to hand, as against every row the world could have produced. The distinction, and what it costs, is covered under sampling. of ten months cannot be checked as thoroughly as a sample of ten thousand, and saying so out loud in the same paragraph as the slope is what separates a quoted figure from an oversold one.