Regression vs Classification: Size or Label

The fitted line and its two numbers, the fitted value, the miss a line leaves behind and what R squared measures were all set up in the earlier notes on fitting a line, and what a chance claims was settled earlier still. Everything else, including every finance word, is built from the ground up below.

What is regression, on its own terms?

Regression is a rule that answers the question how much. Given an input, what comes back is a number on the same scale as the thing being predicted. If the thing being predicted is measured in per cent, the answer is in per cent. If it is measured in rupees, the answer is in rupees. If it is measured in kilograms of onions sold before noon, the answer is in kilograms of onions.

Think about a household deciding what the electricity bill will be next month. Everybody already knows there will be a bill. The question is how large it will be. Nine hundred rupees. Two thousand two hundred. Asking how large is a size question, and the answer to a size question can be close or far. Being out by fifty rupees and being out by four thousand are two very different evenings in that household. Regression cares about the size of every single miss, and a miss of 0.20 is genuinely better than a miss of 8.00.

The worked record is ten months of two invented traded unitsSomething that can be bought and sold at a price, so that its price can be recorded from one month to the next. Both of the ones here were made up for teaching and neither exists., the Nakshatra unit and the Vasant unit. For each of the ten months there is a monthly changeThe move in a price across one month, written as a percentage of where that price began. A figure of 6.00 per cent means it finished the month six per cent higher than it started. for each of them. The Nakshatra unit is the input and the Vasant unit is the outcome, and the notes on fitting a line already put a straight line through those ten pairs. The fitted line is 0.5000 plus 1.5000 for every point of the Nakshatra unit, its fit is 0.7559, and the ten misses it leaves behind are 1, 9, 8, 0, minus 5, 0, minus 3, minus 3, minus 2 and minus 5, all in per cent.

Read that fit out loud in the only way it can honestly be read: across these ten months, the line removed 75.59 per cent of the squared movement that guessing one single number for every month would have left. The fit is a share of movement, and a share of movement is not a count of anything. The difference between a share and a count matters enormously below.

What is classification, on its own terms?

Classification is a rule that answers a different question: which one. Given an input, what comes back is not a number on any scale but a label chosen from a fixed and usually very short list. Up or down. Sold out or not sold out. Paid on time, paid late, not paid at all.

Go back to a food stall outside one office building. On Monday the person running it asks how much they will take today, and the answer is a number: eleven thousand rupees, maybe twelve. On Tuesday they ask something else entirely. Will they sell out before three in the afternoon? If they will, they should cook more. The second question has exactly two answers and no middle. Selling the last plate at 2.59 and selling it at 3.01 feel almost identical standing at the stall, and yet one is a yes and the other is a no.

The character of a label question is exactly that. Classification cares about which side of a line the answer fell on and about nothing else at all, so being wrong by a hair costs exactly what being wrong by a mile costs. A rule that says the stall will sell out, on a day when the last plate goes at 3.01, is simply wrong, and it is wrong by the same amount as a rule that says it will sell out on a day when half the food goes home uneaten. There is no such thing as a near miss. There is a right call and a wrong call, and the scoreboard counts them.

What happens if one column of the same record is replaced?

Here is the move this guide turns on, and it is worth slowing down for. Almost every treatment of this subject uses one record for the size question and a completely different record for the label question, and the moment it does that a reader can no longer tell which differences came from the two questions and which came from the two records. So nothing changes here except one column.

Take the same ten months. Same Nakshatra unit, every figure of it, and the same ten paired observationsTwo readings taken on one and the same occasion, which therefore have to be kept together and can never be shuffled apart. Each month here contributes one input reading and one outcome reading. in the same time orderThe order the months actually happened in, first to tenth. Keeping it matters because some checks read the misses along that order, and sorting the rows for a tidier chart destroys them.. Now throw away the Vasant unit's monthly change and keep only its sign: did it rise, yes or no. A change of 18.50 per cent becomes the word up. A change of 2.50 per cent also becomes the word up, and so the largest month and a barely positive month are now the same thing. A change of minus 13.00 per cent becomes down, and so does minus 1.00 per cent.

Five of the ten months are up and five are down. Nothing else changed at all, so every difference from here on is a difference between the two questions rather than a difference between two records. The claim is stronger than it looks. There is only one record, so any gap that appears below cannot be blamed on one record being noisier, shorter, or collected differently.

Why do the two scoreboards refuse to compare?

Fit the label the same way the size was fitted, using the same input and the same method, and the result is a rule that reads 0.4500 plus 0.0500 for every point of the Nakshatra unit. Its fit is 0.3000. The figure of 0.3000 can be checked in two steps. The ten label gaps from their average are each one half in size, so their squares add to 2.5000. The part the input explains is 0.0500 times 15, or 0.7500. Dividing the second by the first gives 0.3000.

So one number went from 0.7559 to 0.3000 while nothing but the outcome column moved. The drop is already worth sitting with. The more dangerous comparison is with accuracy. The natural way to score a label rule is not a fit at all: accuracy asks how many of the ten months the rule called correctly. The answer is 60.00 per cent, and it is built cell by cell below.

An R squared of 0.7559 and an accuracy of 60.00 per cent are not two views of one quality, and there is no conversion between them. The first is a share of squared movement, computed against the spreadA measure of how far apart a group of figures sits around its own average. A record whose months all look much alike carries a small one, and a record full of violent months carries a large one. of the outcome, and it lives on a scale where zero means no better than a flat guess. The second is a count of months divided by ten, and it lives on a scale where the floor is not zero at all.

The floor is where people trip. Ask what the floor actually is here. Five of the ten months went up, so a rule that ignores the input completely and shouts up every single month is right five times out of ten. Five times out of ten is 50.00 per cent. A model reporting 60.00 per cent accuracy has bought ten percentage points over a rule that does not look at the data, and an accuracy quoted without that baseline beside it is worth nothing. If eight of the ten months had gone up, shouting up every month would have scored 80.00 per cent, and a proud announcement of 75.00 per cent accuracy would have been an announcement of something worse than nothing.

How is a confusion table read, cell by cell?

The rule for the label hands back a number between zero and one for each month, and on these ten months those numbers are 0.50, 0.75, 0.25, 1.00, 0.50, 0.00, 0.75, 0.50, 0.25 and 0.50. To turn a number into a call, somebody has to pick a cut. Take the obvious one for now and call a month up whenever its number is one half or more.

The cut at one half gives seven months called up and three called down. Now lay the calls against what actually happened, and there are only four things that can be in a cell. Called up and it rose: four months, numbers 1, 2, 4 and 7. Called up and it fell: three months, numbers 5, 8 and 10. Called down and it rose: one month, month 3, on its own. Called down and it fell: two months, 6 and 9. Four plus three plus one plus two is ten, and that is the first check to run on any such table: the four cells must add up to every month in the record, or something has been counted twice or dropped.

Three figures come off that table and each one uses a different denominator.

Readers usually go wrong here, so look at what precision and recall are doing. Both of them are counting exactly the same four months, 1, 2, 4 and 7, and they differ only in what they divide those four months by. Precision divides by the seven months the rule stuck its neck out on. Recall divides by the five months that actually rose. So the rule catches four of the five up months and sounds good. The same rule is right on only four of the seven months it called up and sounds much less good. Both statements are true at once, and both describe one and the same set of four months.

Who chose the threshold, and is it part of the model?

Go back to the sentence that decided the whole table above: call a month up whenever its number is one half or more. Nothing in the fitting produced that one half. The arithmetic handed back ten numbers between zero and one and then stopped, and a person walked in and drew a line. The line somebody drew is the threshold, and here is what makes it worth a section of its own.

Move it to three quarters, without touching a single thing about the rule. Now only three months clear the bar, months 2, 4 and 7, and all three of them really did rise. The table reads three, zero, two, five. Accuracy is eight months of ten, or 80.00 per cent. Precision is three of three, or 100.00 per cent. Months 1 and 3 rose and were not caught, so recall has fallen to three of five, or 60.00 per cent.

Accuracy on these ten months runs from 60.00 per cent to 80.00 per cent on one completely unchanged model, so a reader handed only the higher figure has been handed somebody's choice rather than a property of the model. Push it further and the point sharpens. At a threshold of zero every month clears the bar, and the rule becomes exactly the baseline that shouts up at everything, with accuracy at 50.00 per cent. At a threshold of 1.00 only month 4 clears it, precision is a perfect 100.00 per cent on a single call, and recall has collapsed to 20.00 per cent. Nothing about the fitted rule changed at any point in that paragraph.

Because the threshold moves all three readings at once, there are established ways of looking at every threshold in one picture rather than picking one and reporting it. The curve that does that, and the single number for the area underneath it, are covered separately in the notes on modelling. The discipline is simple: the threshold is quoted beside every confusion table, or only half a number has been quoted.

What breaks when a straight line predicts a chance?

There is one more thing to look at before leaving the arithmetic, and it is not a flaw in this particular record. Fitting the up-or-down label with a straight line, exactly as was done above, produces a rule that hands back a number for any input at all. On the ten months in the record, those numbers behave: the smallest is 0.00 and the largest is 1.00, and everything sits in between.

Both of those edges are worth noticing. The largest month, the one where the Nakshatra unit moved 11.00 per cent, gets 0.4500 plus 0.0500 times 11.00, or exactly 1.00. The lowest month, at minus 9.00 per cent, gets exactly 0.00. The line does not merely approach the ends of the scale, it lands on them, and it has nothing left to give. Now hand it a month the record has never seen. At an input of 15.00 per cent, four percentage points beyond the highest month on file, it returns 0.4500 plus 0.7500, or 1.20.

A chance of 1.20 is not a chance, and the arithmetic produced it without the slightest complaint. The silence is the point. Nothing gave a warning, nothing failed, no cell went red. Carrying on is the only thing a straight line knows how to do, so a straight line that reaches one must keep going past one. The overshoot is not a fault in these ten months, and no longer record would cure it. Methods built specifically for outcomes that are yes or no are shaped so this cannot happen, and they are covered separately.

Do the two questions even agree about which month went wrong?

The sharpest result of all falls out of the one record already built, and so costs nothing extra. Asked to name the month the model handled worst, the two scoreboards name different months.

Start with the size. The biggest residualWhat is left over in one month after the fitted value is taken away from what actually happened. It was built in the earlier notes on fitting a line and is simply the size of that month's miss. is month 2, where the line said 9.50 per cent and the Vasant unit delivered 18.50, a miss of 9.00 percentage pointsThe plain difference between two percentages. Going from 9.50 per cent to 18.50 per cent is a rise of nine percentage points, which avoids the confusion of calling it a nine per cent rise.. On the size question that is comfortably the worst month in the record. Now ask the label question about the same month. Month 2 has a fitted number of 0.75, it was called up, and it went up. The worst month for the size question is a month the label question gets right.

The mismatch runs the other way too. Month 3 is the only month that rose and was called down, the single most awkward kind of error a direction rule can make, and its regression miss of 8.00 is the second largest rather than the largest. The four months the direction rule gets wrong are 3, 5, 8 and 10, with misses of 8.00, minus 5.00, minus 3.00 and minus 5.00. Month 8 sits in that list with a miss smaller than months 2 and 3 both. Months 4 and 6, the two the line predicted perfectly with a miss of exactly zero, are called correctly as well.

A model can be at its very worst on the question nobody is asking, so choosing a scoreboard is choosing which failures ever get seen. Nobody in this worked case has done anything careless. One dataset, one input, one method, and two entirely reasonable ways of asking what went wrong, and they hand back different answers.

Which of the two does a given question need?

Here is what somebody deciding this in practice actually does, whether they are a lender sorting applications, an analyst writing a note, or a household working out next month. Nobody sensible starts from the arithmetic. The starting point is the use the answer will be put to, and the arithmetic follows.

The first question is the one that settles it most of the time: is a near miss nearly right, or is it simply wrong? Being out by two hundred rupees and being out by twenty thousand are not the same problem, so a lender estimating how much a borrower can repay each month wants a size. There is no partial approval, so a lender deciding whether to approve at all wants a label, and being just barely wrong about somebody is a full mistake. Same lender, same applicant, two different questions, and the second one has no near miss in it anywhere.

Then three checks that catch most of the damage. The balance of the labels has to be checked before an accuracy is believed. On a sampleThe particular set of cases in hand, as opposed to every case that exists. Ten months is a sample; the answer any check gives describes those ten and not all months everywhere. where nine cases out of ten carry the same label, a rule that shouts that label every time already scores 90.00 per cent and has learnt nothing. One model has already reported 60.00 and 80.00 per cent on one record, so the threshold and who chose it have to be checked. And what a wrong call costs in each direction has to be checked. No arithmetic settles that one: calling an up month down and calling a down month up are two different mistakes with two different consequences, and only somebody who understands the situation can weigh them.

Last of all, and it is the habit worth keeping longest: never convert one scoreboard into the other, not even loosely, not even in conversation. A fit of 0.7559 does not become three calls in four. An accuracy of 60.00 per cent does not become a fit of anything.

Where do these figures come from?