Linear, Logistic, Ridge and Lasso Regression Compared

Four things arrive here already finished: a fitted line, the claim each of its two numbers makes, the way a fit gets scored, and the leftover when a rule misses. Earlier reading settled all four. The ten paired months worked through below arrive the same way, whole and unedited, down to the last decimal place.

What do all four of these forms have in common?

One thing is rarely said outright, and it makes the next thousand words much easier. All four of these forms do the same arithmetic. Every one of them takes the inputs it is given, multiplies each input by its own number, adds the results together, and then reads an answer off that total. The multiplying number attached to each input is called its coefficientThe number a fitted rule multiplies one input by. If the coefficient on rainfall is 1.5, the rule adds 1.5 for every extra unit of rain. It is settled by the fitting and is not chosen by hand., and fitting, in every one of these four forms, is the work of settling those numbers.

Picture three shops on one street, all selling the same thing, all setting tomorrow's price by adding up the same handful of costs: what they paid for the goods, what the rent works out to per item, what the electricity costs, what they think the shop next door will do. The three of them never argue about the arithmetic. Adding things up is adding things up. The three shopkeepers argue about how much weight each cost should carry, and about whether anybody is allowed to put an enormous weight on one item just because it made last month's numbers come out neatly. The argument over weights is the entire disagreement between these four forms, and it comes down to exactly two questions.

The first question is about what the answer describes. Linear regression, ridge and lasso all hand back a size: a number carrying the outcome's own units, whatever those happen to be. Logistic regression hands back something else entirely: a chance, a number obliged to sit between nought and one because it is a statement about how likely a yes is.

The second question is whether the size of the coefficients is punished. Plain linear regression and logistic regression do not care how large a coefficient grows, so long as the fit comes out well. Ridge and lasso both add a charge that grows as the coefficients grow, and they differ in how that charge is worked out. Between them the two questions place a form nobody has met before, and placing a form is worth more than memorising four recipes.

What does linear regression fit, and what does it read at the edges?

The worked record here is ten months long. Two invented columns of figures run down it, the Nakshatra unit and the Vasant unit, and every month carries one reading from each. Which column plays which part is settled already: readings of the first go in, and readings of the second are what a rule has to produce. Earlier reading ran a straight line through those ten pairs, and it came out as 0.5000, with a further 1.5000 added for each point the Nakshatra reading carries. Its interceptThe number a fitted rule returns when every input reads nought. It is where the line meets the vertical axis, and it is settled by the fitting alongside the coefficients. is 0.5000, its coefficient is 1.5000, and its R squaredA score between nought and one saying what share of the outcome's up and down movement the rule accounted for. It was built and explained in the earlier reading on fitting a line. is 0.7559.

The next question is what comes out of that line when a number goes in. A Nakshatra reading of 6.00 per cent gives back 9.50 per cent. Minus 9.00 gives back minus 13.00. And 400.00 gives back 600.50 without hesitation. Nothing anywhere in the arithmetic says that a Vasant change of 600.50 per cent is a strange thing to say. Linear regression answers a question about a size, and a size is allowed to be any number at all. Being allowed any number is exactly right when the question is how much, and exactly wrong when the question is whether.

The freedom to return any number at all is a feature and not a bug. If the question is how many litres of water a housing society tank will need tomorrow, the answer has to be free to come out at 4,200 or 11,600 or anything in between, and it has to be able to come out larger than anything that has happened before. A rule that quietly refused to say a number above 10,000 would be lying. The trouble starts only when the same freedom is pointed at a question that has a ceiling built into it.

What happens when that same straight line is handed a yes or no question?

Here is the experiment, and it changes only one thing. Keep the same ten months and the same Nakshatra readings, and throw away the size of the Vasant unit's monthly change, keeping only whether it rose. The month that moved 18.50 per cent is now simply an up month. So is the month that moved 2.50 per cent. The biggest month on the record and one that barely twitched have collapsed into a single category. Down swallows minus 1.00 per cent and minus 13.00 per cent with equal indifference. Counted out, the ten split five each way. The up or down reading is the month's labelA short word standing for which group something falls into, rather than a measurement of it. Here each month carries one of two labels, up or down, and there is nothing in between them..

Now a straight line is fitted to that, exactly the way a line was fitted to the size. The arithmetic does not object. With up as one and down as nought, the same fitting returns a line that starts at 0.4500 and climbs by 0.0500 for each Nakshatra point. The slope checks in one step: the ten Nakshatra readings sit 300 in squared distance from their own average, they cross with the labels at 15, and 15 divided by 300 is 0.0500.

Read that rule across the record and it looks almost suspiciously tidy. At the smallest month on the record, a Nakshatra change of minus 9.00 per cent, it returns exactly 0.0000. At the largest month, a change of 11.00 per cent, it returns exactly 1.0000. Everything in between lands somewhere inside, at 0.2500, 0.5000 and 0.7500. A reader glancing at that could easily conclude the form is well behaved and move on.

So push it one step further out, to a Nakshatra reading of 15.00 per cent, four points past the largest month anybody has recorded. The rule returns 1.2000. Offered as a chance, 1.2000 is meaningless, and not one step of the arithmetic paused over it, flinched, or warned anybody. Go the other way, to a reading of minus 15.00 per cent, and it returns minus 0.3000. Minus 0.3000 is not a chance either. Having a floor and a ceiling was never part of what a straight line is, so a straight line has no floor and no ceiling.

What does Logistic Regression fit instead, and what does it read at those same edges?

Logistic regression is the form built for exactly this problem. Logistic regression takes the same inputs, multiplies each by its own coefficient and adds them up in the same way, and then does one extra thing at the end: it pushes that total through a squashing step before reporting it. The squashing step takes any total at all, however wild, and returns one divided by one plus a positive number. Since the thing on the bottom is always larger than one, the answer is always smaller than one, and since the thing on the bottom is always positive, the answer is always larger than nought. The ceiling and the floor are not rules bolted on afterwards. Both fall out of the shape of the squashing step itself.

Fit that form to the same ten months, the same Nakshatra readings and the same up or down labels, and it settles on a curve that crosses one half at a Nakshatra change of exactly 1.00 per cent, the Nakshatra unit's own average across the ten months. The coefficient it settles on is 0.2839. Read the curve at the five distinct readings the record actually contains and it returns 0.0553 at minus 9.00, 0.1947 at minus 4.00, 0.5000 at 1.00, 0.8053 at 6.00 and 0.9447 at 11.00.

Now push the curve to the same place that broke the straight line, a reading of 15.00 per cent. The curve returns 0.9816. Push it much harder, to 44.00 per cent, four times the largest month on the record, and it returns 0.999995. Push it to 110.00 per cent, ten times the largest month, and the answer is so close to one that four decimal places cannot show the difference any more. But one divided by one plus a positive number is below one no matter how large that positive number grows, so the difference is still there, and it is there for every reading that could ever be named. The curve approaches one forever and arrives never, and that is the whole reason the form exists.

Do the two forms actually disagree on this record?

Most treatments skip the next question, and skipping it turns an honest comparison into a sales pitch. After the straight line embarrasses itself at 1.2000, the logistic curve might be expected to beat it on the ten months as well. It does not. It ties.

Call a month up whenever the fitted chance is one half or more. The half way cut is called a thresholdThe line somebody draws across a fitted chance to turn it into a call. Above the line the answer is yes, below it the answer is no. Where to draw it is a separate question taken up in later reading.. Apply it to the straight line and it calls seven of the ten months up. Apply it to the logistic curve and it calls the same seven months up. Not a similar seven. The same seven, month for month. Both shapes rise with the Nakshatra reading, and both cross one half at exactly the same place, a reading of 1.00 per cent.

So the two forms produce one identical scoreboard. Four months called up really rose, three called up really fell, one called down really rose and two called down really fell. Six calls out of ten came out right, an accuracyThe share of calls that came out right, counted across every case. Six right out of ten is 60.00 per cent. Two other views of the same set of calls are built in later reading. of 60.00 per cent for both, against 50.00 per cent for a rule that never glances at the Nakshatra reading and answers up to whatever is put in front of it. And because the logistic curve is a rising rearrangement of the very same total, it ranks the ten months in exactly the order the straight line ranks them, so every score built out of that ranking comes out the same too, including an area under the curve of 0.8000 that later reading takes apart properly.

One housekeeping note while three numbers are on the table together. The 0.8000 just quoted, the 60.00 per cent beside it, and a recall of 80.00 per cent that later reading will quote from this same set of calls are three different quantities on three different scales that happen to look alike, and the resemblance between them means nothing whatsoever.

The two forms differ where the record does not reach, and this record does not reach there, so a claim that the logistic form calls these ten months better would be inventing a difference that is not in the arithmetic. The reason to reach for the logistic form is not that it wins here. The reason is that the logistic form cannot produce 1.2000 and the straight line can, and the question being asked will not always fall inside the record it was fitted on.

What is a penalty, and what does ridge do with one?

Everything so far has been about the first of the two questions. Now for the second. A penalty is a charge added to the thing the fitting is trying to make small. Normally the fitting is trying to make the misses small and nothing else. With a penalty in place it is trying to make the misses small plus some charge that grows as the coefficients grow, so a coefficient of 40 has to earn its keep against a coefficient of 4 rather than simply arriving free.

Why would anyone want that? Because of a specific and very common mess. Bring in a third invented column, the Chandana unit: the Nakshatra unit nudged by one twentieth of a percentage point in eight of the ten months and left completely alone in the other two. The Chandana unit is not a copy of the Nakshatra unit, but it is very nearly one. Its correlationA number from minus one to one saying how tightly two columns move together. At one they move in perfect step. It was built and explained in the earlier reading on two moving columns. with the Nakshatra unit is 0.999967, and the standard warning score for this situation, the variance inflation factor, comes out at 15,001.

Fit the Vasant unit on both of those columns at once with no penalty and look at what falls out: a coefficient of minus 1.0000 on the Nakshatra unit and 2.5000 on the Chandana unit. Two columns that agree with each other to five decimal places have been handed a large negative weight and a large positive weight. The split is arithmetic nonsense dressed as a result. The nonsense happens because the two columns nearly cancel, so the fitting can add almost anything to one coefficient and take almost the same amount off the other without the misses noticing. Notice, though, that the two still add to 1.5000, exactly the coefficient the Nakshatra unit carried on its own.

Now switch on a penalty of one, charged on the squared size of the coefficients. A charge on squared size is ridge. The pair becomes 0.7314 and 0.7661. The two coefficients close from 3.5000 apart to 0.0347 apart. R squared moves only from 0.756019 to 0.755950, a distance of 0.000069, so the coefficient pair travels roughly 24,974 times as far as the fit does. The comparison is the whole argument for a penalty, and it is also easy to overstate. Read it carefully. The fit did move. The fit moved by one unit in the fourth decimal place, and anybody who says the fit was unchanged has said something slightly false. A fit which barely moved across a huge stretch of coefficient pairs was never pinning those coefficients down in the first place.

What does lasso do differently, and what does a coefficient of nought mean?

Lasso changes one thing about ridge, and the change looks tiny written down. Charge the coefficients by their plain size rather than by their squared size. That is it. But the behaviour that falls out is a different kind of thing altogether. A charge on plain size does not tail off as a coefficient approaches nought the way a charge on squared size does. The charge keeps pushing right up to the edge, and so lasso can push a coefficient all the way there and leave it.

Run it on the same two columns at a penalty of one. The Nakshatra coefficient comes out at exactly nought. Not 0.0004, not something that rounds to nothing. Nought. And the Chandana unit is left carrying 1.4967 entirely on its own, with an R squared of 0.7560, barely distinguishable from what the pair managed together. Ridge keeps both columns and splits the relationship between them; lasso refuses to split and hands the whole relationship to one column.

Two things are worth checking before the nought is believed. First, is it an accident of the order the arithmetic visited the two columns in? It is not. Settling the Chandana coefficient first instead produces the identical answer, so the result is a property of the problem rather than a quirk of the procedure. Second, and much more important: which column did it drop? Lasso dropped the Nakshatra unit, the original, and kept the Chandana unit, the near duplicateA column built out of another one with only a very small change, so that the two carry almost the same information as each other. Here one is the other nudged by one twentieth of a percentage point in eight of ten months. that was built by nudging the original. Nothing in the output says which of the two was kept, and nothing in it says why.

How would somebody choose between the four?

Two questions, asked in order, and they sort all four without any judgement calls.

First: is the answer wanted a size, or a yes or no? A size takes linear regression, or one of its penalised versions. A yes or no takes logistic regression. A housing society committee looking at its water tank faces both kinds. How many litres will be drawn tomorrow is a size question, and the honest answer might be larger than any day so far. Whether the tank will run dry before evening is a yes or no question, and the honest answer can never be more certain than certain. Handing the second question to a form built for the first is how a printed chance of 1.2000 ends up sitting unnoticed for a month.

Second: are two of the inputs carrying the same information as each other? If they genuinely differ, plain linear regression is doing nothing wrong and a penalty buys very little. If two or more of them are near duplicates, the coefficients will fly apart the way minus 1.0000 and 2.5000 flew apart, and a penalty is what stops that. Ridge keeps every column and shares the relationship out among the ones that agree. Lasso picks one and zeroes the rest. The choice between sharing and dropping changes what the output looks like, and which behaviour has been switched on is worth knowing before a coefficient is read out loud to anybody.

One warning to carry out of this, and it is the mistake that costs most often. Neither penalty is a way of finding out which input matters, and a lasso nought is not a vote. A workshop owner who tracks both the number of machines repaired and the number of machines collected, when almost every machine repaired is collected the same day, has two columns saying one thing. No penalty, and no procedure of any kind, can tell that owner which of the two drove the week. The record simply does not contain the answer. A penalty gives something else, and it is genuinely useful: a stable set of coefficients instead of a wild one.

What is left for later reading?