Statistical Significance vs Economic Significance

Statistical significance asks whether a record this short could have thrown up a result this large by chance alone. Economic significance asks whether an effect of that size would matter to anybody. The two take different inputs and disagree routinely. The fifty month record of the Nakshatra unit cannot reject a mean of zero, at a p of about 0.4772. The same effect compounds to 12.68 per cent over twelve months.

Both of those sentences are true at the same time, about the same fifty months, and neither is a rounding or a softening of the other. Most people have been trained to treat one of the two as the verdict and the other as colour, so the pair is uncomfortable on first encounter. Each of the two is defined on its own below, in full, with its inputs listed. The contrast only makes sense once it is clear that the two lists are not the same list.

Two invented objects carry the arithmetic. The Nakshatra unit is a traded unitA stand in for any object whose price is quoted and moves about, which is all that is needed for one month to yield one figure. The arithmetic depends on the run of prices and not on the sort of object behind it. whose monthly changeOne month of price movement, expressed against the level the month started from. Something that opens at Rs 200/- and closes at Rs 212/- has changed by 6.00 per cent. is restricted to a list of exactly five outcomes. Minus 9.00 per cent comes up eight times in every hundred months, minus 4.00 per cent eighteen times, 1.00 per cent forty eight times, 6.00 per cent eighteen times and 11.00 per cent eight times. Draw fifty months from that generatorThe rule that produces the numbers, written down in advance. Here the rule is a short list of outcomes with a weight against each. Every month is one draw from that list. and the fifty month record appears. Every figure below is worked on that sample. In it, the five outcomes came up 5, 9, 25, 8 and 3 times respectively.

The unusual thing about these two objects is that the rule was written down before any month came out of it, so the true monthly effect of 1.00 per cent and the true spread of 5.00 per cent are known here rather than estimated. That almost never happens. With a real record there are the fifty months and nothing else, and what sits behind them can be argued over forever. Here the record and the truth can be set side by side and the distance between them measured exactly. Only a known truth lets anybody say with a straight face that a real effect went undetected.

What is statistical significance, taken on its own?

Statistical significance is the answer to one narrow question: if there were genuinely nothing there, how often would a record like this one throw up a result at least this far from nothing, purely by luck? If the answer is often, the record has shown nothing that shuffling could not have produced. If the answer is rarely, the record is hard to explain away as luck.

Start with a shopkeeper rather than a record of prices. The shopkeeper suspects Tuesdays are busier than Wednesdays, counts three of each, and Tuesdays come out ahead. Nobody would take that seriously, and everybody knows why without being taught: three days of anything wobbles enough to produce that gap by accident. Counting three hundred of each and finding the same size of gap is a different sentence entirely. Nothing about Tuesdays changed between the two counts. The amount of luck available to produce the gap changed.

Three things decide whether a result comes out statistically significant, and they are the size of the effect, the spreadHow far apart the individual cases sit from each other, boiled down to one number. How that number is built, and the several ways of building it, are covered separately. of the individual cases, and the number of cases. A larger effect is harder to fake. A tighter spread means less wobble to fake it with. Averages of many things settle down, so more cases mean the wobble in the average shrinks. The three inputs go into one arithmetic and one number comes out.

Now notice something about that list that almost nobody says out loud: two of those three inputs are properties of the record rather than of the world. The spread of the cases is partly the world and partly what was chosen for measurement. The number of cases is entirely the record keeper's own. Nothing about the Nakshatra unit changes when somebody keeps the ledger open for another two hundred months. The unit does what it always did. Only the record grows. So a statistical verdict is a joint statement about the world and about how much of it somebody bothered to write down, and it never separates the two.

What is economic significance, taken on its own?

Economic significance is the answer to a different narrow question: if a difference of this size is real, would it be worth anything in units somebody actually counts? Rupees, months, seconds at a counter, litres of milk. Not units of luck. Units of consequence.

Economic significance is the reading a household does instinctively and a report often forgets. One per cent a month sounds like nothing. Say it out loud and it sounds like a rounding error. But compoundingApplying a change to whatever the last change left behind, rather than to the starting amount every time. Each round works on a slightly larger base, so twelve rounds of one per cent do not add up to twelve per cent. is what turns one per cent a month into something visible. Rs 10,00,000/- with 1.00 per cent applied twelve times over, each time to whatever the last round left behind, finishes at Rs 11,26,825/-. The gap is Rs 1,26,825/- over one year, from a monthly figure most people would have called noise. In percentage terms it is 12.68 per cent, and that figure runs through this whole guide.

Two things decide whether a difference is economically significant: the size of the effect, and the setting it is applied in. The same 1.00 per cent a month is one figure when it runs on Rs 10,000/- and another when it runs on Rs 10,00,000/-, and a different one again over one month than over twelve. Size and setting. The list ends there.

The number of cases does not appear in that list at all, and the whole contrast rests on that absence. It is not there. How many months of record somebody happened to keep has no bearing whatsoever on what 1.00 per cent a month does to Rs 10,00,000/- over a year. The arithmetic of compounding does not consult the ledger. A hundred years of record and a single month of record leave 12.68 per cent exactly where it was.

Where exactly do the two part company?

One line does it. The two kinds of significance differ because one of them depends on how much record there is and the other does not. Everything else in this guide is a consequence of that single asymmetry.

Laid next to each other, the two lists overlap in exactly one place. Both consult the size of the effect. After that they go their separate ways. Statistical significance reaches for the spread and the count. Economic significance reaches for the setting. The two are not two opinions about the same calculation but two calculations that happen to share one ingredient.

The two can disagree without either of them being wrong, so asking which is the real verdict is asking the wrong question. There is no school of thought here to pick a side in. Different numbers fed into different arithmetic give different answers, and that is not a controversy, it is subtraction. A reader who demands to know which one counts is really asking somebody else to make the judgement for them, and no arithmetic in this guide or anywhere else will do that.

The practical version is simpler still. A statistical verdict can be changed by collecting more. An economic reading cannot. When a result flips from nothing to something and the only thing that changed was the length of the record, nothing about the world moved. When a result flips because the effect turned out to be twice the size, something did.

What does the fifty month record say on each of the two?

The record itself now goes through both, slowly.

On the statistical side, the fifty months average 0.50 per cent. The spread across those months is 4.9744 per cent. Divided by the square root of fifty, that spread gives a standard errorHow much the average of a record wobbles from record to record, purely because a record is a handful of cases and not the whole story. The standard error shrinks as the count grows, and how it is built is covered separately. of 0.7035 per cent. Divide 0.50 by 0.7035 and the statistic is 0.7107. Under the normal shapeThe single humped, symmetric shape that turns up whenever many small independent wobbles are averaged together. The normal shape judges how surprising a statistic is, and how it is built up is covered separately. used to judge it, a statistic that far from zero or further turns up about 47.72 times in every hundred records when nothing is there at all. So the record cannot reject a mean of zero. Not narrowly. By a distance.

On the economic side, the true monthly effect behind those months is 1.00 per cent, and that is known here only because the rule was written down first. As a share of the true spread of 5.00 per cent it is 0.20. Compounded through twelve months it is 12.68 per cent. On Rs 10,00,000/- that is Rs 1,26,825/- across a year. Nobody looking at that figure in rupees would call it nothing.

Both statements are true, and neither is a rounding of the other: the test says the record cannot see the effect, and the arithmetic says the effect is large. The honest one line summary is not that the effect is absent and not that the test failed. The summary is that fifty months is too short a record to see an effect of a size that would matter. Sitting between the two figures is a third one, computed separately, that reconciles them: resolving an effect this size reliably would take about 197 months, and the record holds 50.

Read the two columns downwards and count how many rows they share. One. The size of the effect, at the top. Every other row has one column answering and the other saying it never asked. The two columns are not two readings of one calculation but two calculations.

How can a real effect be statistically invisible?

A real effect going statistically invisible is the first of the two failure directions, and the fifty month record demonstrates it. There is a genuine 1.00 per cent a month sitting behind those months. The test cannot see it.

The third input is the reason. Statistical significance rises with the number of cases, and a short record fails to reject almost everything put to it. With fifty months the wobble in the average is 0.7035 per cent, which is larger than the whole effect being looked for. The signal is smaller than the noise around the measurement of it. Nothing is broken. The instrument simply is not fine enough for the thing it is pointed at.

A household version helps. The question is whether switching vegetable sellers saves money. One week of bills is compared against one week of bills, and the new seller comes out Rs 40/- cheaper. Weekly grocery bills swing by hundreds of rupees depending on what was in season and who came to dinner. One week against one week cannot separate Rs 40/- of saving from ordinary variation, and that is true even if the Rs 40/- is completely real and repeats every single week for a lifetime. Over a year that Rs 40/- is Rs 2,080/-. Real, worth having, and invisible to a one week comparison.

So the consequence for reading a report is sharp: a result of nothing found, taken from a short record, is not evidence that there is nothing there. It is a statement about what that record was capable of resolving. The fifty month record would have needed roughly 197 months to resolve an effect of 1.00 per cent a month reliably, and 381 months before its own reading would drop under the 0.05 line. Where those two figures come from is covered separately. The shape of the situation is what matters. The record was asked a question it did not have the length to answer. A test always answers, so the record answered anyway.

How can an effect too small to matter come out significant?

Now run the same machinery the other way. The fix for one direction quietly creates the other. A reader who learns to distrust null results will start treating every significant finding as important, and that is the second error wearing the first one's clothes.

Take a completely different record, invented for this block. A bank chain tries two counter layouts and measures how long each visit takes. Call it the Tarang counter record. Some people cash a cheque and some people open an account, so the average visit runs 360 seconds and the spread across visits is 210 seconds. Across 2,00,000 visits, one layout comes out 1.2 seconds faster than the other.

Put that through the arithmetic. The standard error is 210 divided by the square root of 2,00,000, or 0.4696 seconds. The statistic is 1.2 divided by 0.4696, or 2.5555, and the p value is 0.0106. Comfortably under 0.05. Statistically significant, and nobody could argue with the arithmetic.

The effect itself is 1.2 seconds off a 360 second visit. The difference is one third of one per cent of the wait. Nobody standing in that queue would notice it even after being told in advance to watch for it. Hold an effect tiny, raise the number of cases far enough, and any difference that is not exactly zero will eventually cross the threshold. The wobble in the estimate keeps shrinking while the effect does not. That same 1.2 seconds first crosses the 0.05 line at about 1,17,645 visits. Below that count it is not significant. Above it, it is. The seconds never changed.

On a very large record, statistical significance stops carrying information about whether anything matters. The verdict becomes a statement about how much data somebody collected. That is not a criticism of the arithmetic, which did exactly what it was asked. The warning is about what the word significant is doing in a sentence. Significant has never meant important. Significant means hard to explain as luck. On a record of two lakh cases, almost nothing is hard to explain as luck, so almost everything passes.

Who pays for each of the two errors?

The two errors are not symmetric in their consequences, so the two directions deserve separating rather than being folded into one warning about being careful.

The first error discards something real, and the person who pays is whoever needed to know it existed. A note says nothing was found. The finding goes in a drawer. Nobody looks again. Looking again costs money and somebody already looked. The 1.00 per cent a month carries on being 1.00 per cent a month, unrecorded by anybody, and the cost is silent: it is the thing nobody ever learned. Silent costs are the hardest to argue for a budget against, and the difficulty is exactly why this error survives so well.

The second error dresses up something trivial, and the person who pays is whoever acts on it. A significant result is louder than a null one. A significant result gets a slide. A significant result gets a rebuild of the counter layout, at real expense, to recover a third of one per cent of a wait. Here the cost is not silent at all, it is an invoice, and it still gets paid because the word significant was doing work in the room that it was never entitled to do.

A note that gives only one of the two figures leaves the reader unable to tell which of the two errors is being made. That is the practical reason to care. A p value on its own does not say whether the reader is in the corner where something real is being thrown away or the corner where something trivial is being polished. A p value names which half of the picture the reader is standing in and says nothing at all about the other half.

What goes into a note so a reader can judge both?

The distinction becomes usable inside a note. An analyst reading somebody else's note, a lender reading a portfolio review, a household reading a comparison of two plans, all face the same problem: the note tells them one thing and they need two. Four lines fix it, and three of them are usually already there.

Line one is the size of the effect in its own units. Here, 0.50 per cent a month measured, against a true 1.00 per cent a month behind it. Line two is the same size translated into units somebody counts. For a monthly figure that usually means a year of compounding: 12.68 per cent, or Rs 1,26,825/- on Rs 10,00,000/-. Line three is the number of cases, here 50 months. Line four is the one almost every note leaves blank, and it is the smallest effect that record could have detected reliably. For fifty months at this spread, that is about 1.97 per cent a month, or 26.39 per cent over twelve months.

Line four does the most work. Line four says that this particular record was only ever capable of reliably flagging an effect roughly twice the size of the one that was actually there. Once that line is in the note, a null result stops sounding like an answer and starts sounding like a description of the instrument. A p value on its own answers neither of the two questions, and a note that carries only a p value has told the reader almost nothing.