Precision and Recall: Two Views of the Same Classifier

Precision is the share of the months called up that really were up. Recall is the share of the real up months caught. On ten invented months relabelled up or down, a calling threshold of one half gives precision of 57.14 per cent and recall of 80.00 per cent, and every move of that threshold pushes one of them up and the other down.

Two numbers, one model, one set of ten months. The model is fitted once and never refitted, and no second record is built. Only one thing changes: which slice of one small table is being read. The slice chosen decides what can be said.

What arrives already settled, and what is left to do here?

Three things are carried in and none of them is explained a second time. The ten paired monthly readings of the Nakshatra unit and the Vasant unit are carried in already made, along with the relabelling of the outcome into up months and down months, five of each. The score attached to each month, a number between nought and one standing for how strongly the rule leans towards up, comes from the same place. And the thresholdThe cut-off a score has to reach before the rule commits to saying up. The threshold is set by a person after the model is built, not learned from the readings, and every score at or above it becomes an up call., together with the way a whole run of thresholds is stepped through in order, is covered under the threshold sweep.

One question is left, it has two answers, and most reports give only one of them. Given a table of calls against outcomes, how well did the rule do? There is no single honest number. There are two, they point in opposite directions, and a reader handed either one alone cannot tell a working model from a rule that says yes to everything.

Everything below reads one small table: the four cellsThe two by two count of calls against outcomes. Across the top, what the rule said; down the side, what actually happened. Every measure below is four whole numbers arranged in a square and nothing more. that record how many months were called up and turned out up, called up and turned out down, called down and turned out up, and called down and turned out down. At a threshold of one half those four counts are 4, 3, 1 and 2. Four counts of that kind are enough to rebuild every percentage below from scratch.

What is precision, on its own?

The question in its plainest form: of the months the rule stuck its neck out on, how often was it right? Precision answers exactly that, and its arithmetic is a single fraction. Count the months called upA month the rule committed to. Its score reached the cut-off, so the rule said up out loud. Whether the month then went up or down is a separate fact and lives in a different column of the same table.. Count how many of those really were up. Divide the second by the first.

At a threshold of one half, seven of the ten months were called up. Four of those seven really were up. Four divided by seven is 0.571428 and onwards. Cut to two places that reads 57.14 per cent. The calculation stops there, and it has a shape worth naming: the denominator is chosen by the model, not by the world. A rule that says up less often has a smaller denominator, and a smaller denominator is easier to keep clean.

Precision never looks at the months that were not called. Nothing it stayed silent about can embarrass it. Take that seriously, because it is the crack the measure can fall through. Push the threshold all the way to one, and this same rule calls exactly one month up, month 4, and month 4 really was up. One right out of one call. Precision reads 100.00 per cent. Nine months went by, four of them genuinely up months, and not one of them appears anywhere in that fraction.

There is a street version of this that lands faster than the arithmetic. Picture a vendor outside a station who only calls out to a passer by when he is completely certain of a sale. He is right every single time he opens his mouth. The people he let walk past do not appear in his hit rate at all, so his hit rate is perfect and his takings are almost nothing. Perfect precision and an empty till are entirely compatible, and no amount of staring at the hit rate will reveal it.

What is recall, on its own?

The table turned the other way answers the opposite question. Of the months that really were up, how many did the rule catch? Recall answers that one: the number of real up months the rule called up, divided by the number of real up months.

Five of the ten months really were up: months 1, 2, 3, 4 and 7. Month 3 scored 0.25 and the cut-off was 0.50, so at a threshold of one half the rule caught four of them and let month 3 slip past. Four divided by five is 80.00 per cent exactly. Recall never looks at how many wrong calls it took to catch what it caught, so it cannot be embarrassed by noise.

And that is its own crack, the mirror image of the first. A rule that simply calls every single month up, with no thinking involved whatsoever, catches all five real up months and scores a recall of 100.00 per cent. The same rule also makes five wrong callsA month the rule committed to that then went the other way. A wrong call costs nothing in recall. Recall counts only the real up months and never the calls, which is exactly why recall alone can be fooled. along the way, and recall does not know they happened.

The everyday version: a household that answers every unknown number that rings will never miss a call from the hospital. Recall of one hundred per cent, and eleven sales calls a day to pay for it. Nobody would describe that as a good system, and yet described only by recall it is flawless.

Put the two side by side and the shape of the thing appears. The two measures are cut from the same four numbers, but one reads across the calls and the other reads down the outcomes. Precision and recall share one cell, the four months that were called up and really were up, and everything else about them is different.

What happens to each as the calling threshold moves?

The model is finished. The scores are fixed. The only thing left to choose is where the cut-off sits, and that choice belongs to a person rather than to the fitting arithmetic. Step it through the five distinct scores this record contains and watch what the two measures do. A run of settings of that kind is the sweepRunning the same finished model at one cut-off after another and writing down the result each time. Nothing is refitted between steps, so every difference in the readings comes from the cut-off and from nothing else., and building one is covered under the threshold sweep.

Precision never falls as the threshold rises, recall never rises, and neither of those was assumed here: both were checked at every one of a hundred and one cut-off settings from nought to one in steps of a hundredth, and neither ever broke. That distinction matters. A claim that has been verified across the whole run is a different kind of statement from a claim that sounded right and was written down.

Read the two rows of numbers straight through. Precision climbs 50.00, 55.56, 57.14, 100.00, 100.00. Recall slides 100.00, 100.00, 80.00, 60.00, 20.00. Somewhere between the threshold of one half and the threshold of three quarters the two cross over, and after that the measure that looked weaker is the strong one. Nothing about the model changed anywhere in that story. Not one coefficient moved.

Why can the two never be raised together?

Because of what raising a threshold physically does, and it does only one thing. Lift the cut-off and some months that used to reach it no longer do. No month can newly reach a cut-off that has just been raised. So every step upward removes up calls and adds none, and that single fact settles the whole question.

Now think about what removal does to each measure. Removing a wrong call takes a month out of precision's denominator and leaves the numerator alone, so precision goes up or stays put. Recall's denominator is the count of real up months, and no threshold can change that. Removing a right call therefore takes a month out of recall's numerator and leaves the denominator where it was. So recall goes down or stays put. There is no third kind of removal available, and there is no move that only ever removes the wrong ones.

The step from one half to three quarters on this record shows it happening. At one half, seven months are called up: 1, 2, 4, 5, 7, 8 and 10. At three quarters, three are: 2, 4 and 7. Four calls were removed. Three of them, months 5, 8 and 10, were wrong calls, and losing them lifts precision from 57.14 per cent all the way to 100.00 per cent. One of them, month 1, was a right call, and losing it drops recall from 80.00 per cent to 60.00 per cent. Three good removals and one bad one in a single step, and the good ones cannot be ordered without the bad.

The argument leans on almost nothing. No sentence in it mentions the Nakshatra unit, the Vasant unit or any of the ten readings. Two facts carry the whole of it: raising a cut-off subtracts calls, and one measure counts calls in its denominator while the other counts outcomes. The trade-off is not a property of this record. The trade-off is a property of what the two measures are built out of, so it holds on any record at all.

What does F1 do with the two, and what does it hide?

The obvious wish, once there are two numbers that will not both go up, is for a single number that settles the argument. F1 is the usual answer. F1 combines precision and recall in a way that is dragged down hard by whichever of the two is worse, so a rule cannot buy a good F1 by being brilliant at one and hopeless at the other. On the counts themselves F1 is twice the right calls divided by twice the right calls plus the wrong calls plus the missed months. The arithmetic stays in whole numbers until the final division.

Across the sweep it reads 0.6667, 0.7143, 0.6667, 0.7500 and 0.3333. Look at the first and the third. The two readings are the same number, and they come from confusion tables that have nothing in common. At no threshold at all the four cells are 5, 5, 0 and 0: everything called up, every real up month caught, five wrong calls, nothing left alone. At one half the four cells are 4, 3, 1 and 2: a rule making an actual distinction, catching most of what mattered, wrong three times and correctly quiet twice. F1 puts both at 0.6667 and cannot say which of the two is in front of the reader.

The collision is not a defect anybody can fix. There are more pairs of measures than there are values to give them, so any single number built out of two will land on the same value from more than one place. The one available remedy is to refuse to let the single number travel alone.

Why is accuracy a poor summary on its own?

Accuracy is the friendliest of the four and the least informative. Accuracy counts everything the rule got right, of either kind, over all ten months. At a threshold of one half that is four right up calls plus two months correctly left alone, six out of ten, 60.00 per cent. Most readers hear sixty per cent and think of a pass mark.

Now put a baselineThe dullest rule available instead, written down so a result has something to be measured against. Here the baseline is the rule that calls every month up without looking at anything. Half the months went up, so that rule gets half of them right. next to it. The rule that calls every month up, having learned nothing from anything, gets five of the ten right and scores 50.00 per cent. So the model bought 10.00 per cent over doing nothing whatsoever, and 10.00 per cent is the real size of the claim, not 60.00 per cent.

Accuracy has a second silence that is worse than the first. Accuracy never says which kind of mistake was made. Three wrong up calls and one missed month score exactly the same as one wrong up call and three missed months, and on almost any record where the two mistakes cost different amounts, those are two different situations. And when one label is much rarer than the other, accuracy stops working altogether: if only one month in ten had been up, calling every month down would have scored 90.00 per cent while catching nothing at all.

Which of the two matters?

Here is the honest answer, and it is not the one a table of numbers can give. Nothing in the ten months decides it. Not the fit, not the scores, not the counts, not the sweep. The choice is made by whoever bears the cost of being wrong, and it depends entirely on which kind of wrong is expensive.

If a wrong up call is the expensive mistake, precision is what matters, and the cut-off is pushed up until the calls that are made are clean. A smoke alarm that empties an office building works that way: every false alarm costs an hour of everybody's day, so the alarm is set to be sure. If a missed up month is the expensive mistake, recall is what matters, and the cut-off drops until almost nothing gets past. A hospital screening test ahead of a serious one works the other way: a missed case is far worse than an extra check, so the test is tuned to be jumpy.

The data cannot make that call, and a threshold chosen without stating which mistake is more expensive has made the decision silently and hidden it inside a number. That is the real reason both measures belong in every report. Quoting one is quoting a conclusion about costs that was never argued.

What belongs in a report, and in what order?

Somebody reading a written result, an analyst checking a colleague, a lender looking at a scoring rule somebody else built, all face the same problem: they have to work out whether the numbers in front of them can be trusted, and they usually cannot rerun anything. The order the figures are given in decides whether that is possible.

Put the four cells first. Everything else in the report is derived from them, and nothing else lets a reader recompute anything. Four whole numbers, 4, 3, 1 and 2, and the reader can build precision, recall, accuracy and F1 without asking a single question. Then the threshold. Every measure below it depends on where the cut-off was set, and a measure quoted without its cut-off is a measure quoted without half its meaning. Then precision and recall together, never one without the other. Then the baseline, so the reader can see what doing nothing would have scored on the same months.

The ordering protects against a specific, common failure: a summary figure travelling on its own, arriving somewhere far from the table it came from, and being read as a description of a model when it is a description of a choice. A reader with four whole numbers can rebuild the argument. A reader with one percentage can only accept or reject it.

What is covered elsewhere?