Data Surprise: Measuring the Gap Between Consensus and Print

A data surprise is the gap between what a release printed and what a gathered set of forecasts expected, quoted in points. The gap is a statement about the forecasters as much as about the economy. The benchmark it is measured against is never revised afterwards, so one print can be a miss today and a beat once the figure settles.

Two things already built are doing the work underneath that answer, and neither needs restating at length. The first is that one release carries more than one correct value: an early estimate, a restated one, and a settled one, all describing the same period, all published by the same body, and none of them wrong. The second is that the aggregates a release reports, output and its growth rate, prices and their movement, are each assembled from separately measured parts, and how that assembly works is set out under the output and price aggregates. The second number is the expectation. Every surprise ever quoted has a print on one side of it and an expectation on the other, and the expectation deserves at least as much attention as the print. Three things should be in place by the last block: computing a surprise in the unit that belongs to it, explaining why its sign is not a stable property of the print, and running a short check before one gap is allowed to change anybody's mind about anything.

What is a data surprise measured against, and is that thing a measurement?

Almost every careless reading of a surprise comes from treating the benchmark as though it were a second measurement of the world. Start with the benchmark rather than with the print.

Before a release is published, a number of forecasters produce an estimate of what it will show. Somebody collects those estimates and reduces them to one figure, usually the median forecastThe middle value once every forecast has been lined up in order, so half sit above it and half below. It is preferred to an average because one badly placed forecast cannot drag it far. of the set. The collected figure is the consensus. The Republic of Sankhya, an invented economy whose every figure was written for this lesson, carries a consensus of 6.40 points for the growth print in question. Its consensus was fixed before anything was published.

Now ask what kind of object that 6.40 is. Nobody counted anything to arrive at it. Nobody surveyed a household, opened a ledger or added up a volume. The consensus is the middle of a set of opinions about a number that did not yet exist. A data surprise is a gap between two numbers and one of them is not a measurement at all, so half of any surprise is a fact about what forecasters expected rather than a fact about the economy.

The household version lands this in about five seconds. Four relatives are asked what this month's electricity bill will be, and the middle of their four guesses is Rs 3,200/-. The bill arrives at Rs 2,900/-. There is a gap of Rs 300/-, and that gap can be described in two completely different ways. One description says the bill came in lower than expected, a fact about the bill. The other says the relatives guessed high, a fact about the relatives. The Rs 300/- gap is identical in both descriptions and it cannot settle which of the two sentences is the better one.

A consensus is a summary, and summarising throws information away. The set of forecasts behind 6.40 points might have been tightly clustered, with nearly everyone within a whisker of the middle, or it might have been spread wide with strong disagreement in both directions. The consensus figure is the same in both cases. The spread of the set has its own name, forecast dispersionHow far apart the individual forecasts in a set sit from one another. Wide dispersion means the forecasters disagreed among themselves; narrow dispersion means they did not. The published consensus figure carries no record of which it was., and a surprise quoted on its own carries no trace of it. A gap of 0.20 points against a set everybody agreed on is a different situation from a gap of 0.20 points against a set that was arguing with itself, and the published surprise looks the same either way.

How is the surprise worked out, and in what unit is it quoted?

The unit is where people come unstuck, not the subtraction. The arithmetic itself is trivial, and worth being slightly insulting about how trivial it is.

A data surprise is the print less the consensus. In Sankhya, at the first print, that is 6.20 less 6.40, a miss of 0.20. If the print had come in above the consensus the subtraction would have turned out positive and the same gap would be called a beat. Miss and beat are direction words for the sign of one subtraction, nothing more, and neither of them is a verdict on the economy or on anybody's work.

Now the unit, and this is the sentence to hold on to. Both 6.20 and 6.40 are already growth rates. Each of them is already expressed in per cent. The distance between two rates is therefore not itself a per cent of anything, but a number of percentage points, shortened by everybody to points. Both sides of the subtraction are already rates, and the gap between two rates is a distance rather than a proportion, so a data surprise is quoted in points and never in per cent.

The slip is the commonest one made with this figure. The gap of 0.20 divided by the consensus of 6.40 gives 3.13 per cent, rounded to two places from 3.125, and it is a real number computed from two published components. The ratio is also not the surprise, and nobody quotes it. The ratio answers a question nobody asked, namely how large the gap is relative to the size of the consensus. A proportion is not a distance. Quoted as the surprise, 3.13 per cent hands the listener a figure fifteen times larger than the one they were expecting to hear, in a unit that does not belong to it.

The everyday check is a thermostat. A room set to twenty six degrees that settles at twenty four is two degrees off. Nobody says the room missed by 7.69 per cent, even though the figure can be worked out. The gap between two readings on the same scale is quoted on that scale. Growth rates are a scale, and points are the notches on it.

Can the sign of a surprise change with nobody revising the consensus?

Everything before this block was setup and everything after it is consequence.

One Sankhya release carries three correct values, as established earlier. The first print reads 6.20 points. The first revision reads 6.50 points. A later vintage reads 6.80 points. All three describe the same period and none of them is an error by anybody. Nobody goes back and revises a consensus, so it stays exactly where it always was, at 6.40 points. A consensus is a set of forecasts gathered at one moment, and that moment has passed.

Run the subtraction three times against that one fixed benchmark. Against the first print of 6.20, the surprise is a miss of 0.20 points. Against the first revision of 6.50, the surprise is a beat of 0.10 points. Against the later vintage of 6.80, the surprise is a beat of 0.40 points. The surprise changed sign between the first two rows and then doubled in size against the first print, and the consensus never moved by a hundredth, so whether a print was a miss or a beat is a function of when the question is asked rather than a property of the print.

The strangeness is the whole point. Sit with it for a moment. Nothing about the forecasters changed. Nobody admitted an error, withdrew an estimate or gathered a new set. The only thing that moved was the measured side, and it moved by more than the gap anybody had been arguing about. A word that was correct on publication, miss, becomes wrong later without anybody having been wrong.

The shopkeeper version is worth having. The relatives guessed the electricity bill at Rs 3,200/-. The bill arrived at Rs 2,900/-, so they guessed high. Then the meter reading is corrected and the bill is reissued at Rs 3,500/-, so now they guessed low. Their guess of Rs 3,200/- never changed. The guess was not a good one and then a bad one. One guess sat still while the thing it was aimed at moved past it in one direction and then the other.

How to Interpret a Data Surprise Without Overreacting: what has to be established before reacting?

The previous block leaves an obvious question hanging. If the sign of a surprise is not stable, what is a reader supposed to do with one? The answer is not to ignore surprises. The answer is to establish four things first, and each of the four is a question with a findable answer rather than a matter of judgement.

A surprise smaller than the series' usual revision is not news yet, so the first and most important thing to establish is how far that particular series usually travels after its first print. In Sankhya the release travelled 0.60 points from the first print to the later vintage. The surprises against consensus were 0.20 points, then 0.10 points, then 0.40 points. Every single one of them is smaller than the distance the figure covered on its own, with nobody surprised by any of that movement. A gap of 0.20 points inside a series that habitually moves 0.60 points is well inside the noise, and reacting to it is reacting to the smaller of two things that both happened.

The second thing follows directly from the first: the surprise has to be compared, not merely noted. Once the usual travel is known, the surprise is set beside it and the question is whether the surprise is larger. If it is smaller, something real has been learned, but what has been learned is about the forecasters rather than about the economy. If it is larger, the gap is something the series' own habits cannot explain away, and only then does the surprise start to be about the thing being measured. The convenient name for the usual travel is the revision bandThe rough range within which a particular series is known to move between its early estimate and its settled one. How that range is established, and why some series carry a wider one than others, is worked through separately in these notes on how a figure keeps moving after publication., and it is a property of the series rather than of any one release.

The third thing to establish is about the benchmark rather than the print, and it is the one almost nobody checks. When was the consensus gathered, and what did the forecasters already know when they produced it? A set of forecasts assembled before several related releases had been published is answering a different question from a set assembled after them. If the forecasters were working without information that has since arrived, a gap between their middle value and the print is partly a measure of what they could not have known, and calling that a surprise about the economy stretches the word.

The fourth thing is to look inside the print rather than at its headline. Most published figures are aggregatesA headline figure assembled by adding up many separately measured parts, each contributing in proportion to its size. The construction of the output and price aggregates is worked through elsewhere in these notes and taken as given here., and an aggregate can be pushed around by one small componentOne of the separately measured parts that add up to a headline figure. A component can be a small share of the total and still move the headline, if it moved a long way. that happens to have moved a long way, even when its weightThe share of a headline figure that one part is allowed to account for, fixed in advance so the parts add up correctly. How weights are set for the output and price aggregates is worked through elsewhere in these notes. in the total is small. If the whole of a 0.20 point surprise sits in one part of the figure that is known for jumping about, the surprise is about that part rather than about the aggregate. A big move in a component with high volatilityA tendency to jump about a great deal from one period to the next, so a large move carries little information because large moves are ordinary for that item. was always likely, and a move that was always likely is precisely what makes a contribution uninformative.

None of the four asks whether the surprise is good or bad, and none of them asks what happens next. All four ask the same underlying question. Is this gap bigger than the ordinary machinery of the series and the ordinary limits of the forecasters? The question has an answer that can be looked up. The other questions do not.

Why does a surprise feel bigger than the arithmetic ever makes it?

Something is worth explaining that has nothing to do with the numbers. A surprise of 0.20 points is a small distance on a scale, and it does not feel small. The feeling is produced by the packaging rather than by the figure, and understanding that is a genuine defence against overreacting.

A release is reported as an event. A release happens at a moment, it has a headline, and the headline carries a direction word: missed, beat, came in below, came in above. Direction words are binary. There is no half a miss. So a continuous quantity that could have landed anywhere along a scale gets converted, at the moment of reporting, into one of two states. The conversion happens at a threshold, the consensus, and the consensus was itself a summary of opinions.

The underlying figure has no such structure. The figure is a quantity on a scale, and the scale has a known width: this series already moves 0.60 points on its own after publication. A print of 6.20 and a print of 6.50 are two nearby positions on one continuous line. The two prints are not two different states of the world. A binary direction word is being attached to a continuous quantity at a threshold nobody measured, so the reporting format supplies the drama and the arithmetic supplies none of it.

There is a homely version of this that everybody has felt. A student who needed forty marks to pass and scored thirty nine feels the whole weight of failing, and a student who scored forty one feels the whole relief of passing, and the two students are two marks apart. The two marks are real. The gulf between failing and passing is manufactured by a threshold. Nobody is being irrational; the format genuinely produces the feeling. The quantity underneath did not step. Two marks is the whole of the movement.

One further thing the format does. Because a release is an event, it arrives with the whole apparatus of an event: a moment of publication, a single number, and a word. All of that encourages the belief that the number is now settled, and the number is not settled and will move again. The event is over. The measurement is not.

What does a data surprise not establish?

A short block, and the most useful one to be able to recite. A surprise is a gap, and a gap has two sides, and that single structural fact is what limits everything the figure can say.

A data surprise does not establish that the economy changed. A gap of 0.20 points is equally consistent with the measured quantity coming in different from what was expected, and with the forecasters having placed their estimates in the wrong spot while the measured quantity did exactly what it was always going to do. A data surprise measures a gap, and a gap cannot report which of its two sides moved, so a surprise is never by itself evidence that anything in the economy is different from what was previously thought.

A data surprise does not establish that the figure will hold. The Sankhya print travelled from 6.20 to 6.50 to 6.80 points, and the surprise recomputed at each step gave a different answer, twice with a different sign. Whatever a surprise says, it says it about a value that has not finished moving.

A data surprise does not establish what anything else will do. A surprise is one gap on one release, and no gap between two numbers can say what follows from it in any market, on any instrument, at any horizon.

And a data surprise does not establish that it was worth reporting. The four questions set out above exist precisely because most surprises, once compared with how far the series routinely travels, turn out to be inside the ordinary machinery of the release. The finding is not a criticism of anybody. A small distance on a moving scale looks exactly like that.

How does somebody who reads releases for a living handle a surprise?

The order of operations is the whole method, and it is not the order most readers use. Watch the order before watching the arithmetic.

An analyst covering an economy does not start by asking how large the surprise is. The first move is to fetch how far the series typically travels between its first print and its settled value, and to do that before looking at the gap at all. A surprise read after that number is a different object from a surprise read before it: 0.20 points means one thing alongside the knowledge that the series moves 0.60 points on its own, and something else entirely without it. A gap looked at on its own always looks like the biggest thing in front of the reader, so reading the band first is a way of making sure the comparison happens.

A surprise inside the series' typical revision band is a statement about forecasters and nothing more, so an analyst compares the surprise with that band before comparing it with anything else. Only for a gap that survives that comparison does it become worth asking the other three questions: what the forecasters could have known, which component the gap sits in, and what the spread of the individual forecasts looked like.

A lender inside Sankhya uses the same material differently. A single release surprise tells a lender almost nothing about the borrower sitting in front of it, and the honest use is as one input into a slow view of conditions rather than as a trigger for anything. The temptation to treat the direction word as a signal is strongest where a decision is available and immediate, so an investor reading the same release has to be more careful still. Lender and investor alike, done properly, note the gap, note the band, and change nothing on a gap smaller than the band.

The household version is not a joke. A salaried person who checks one electricity bill against their expectation and concludes their consumption habits have changed is doing exactly what a reader who reacts to one small surprise is doing. Knowing the ordinary spread of the quantity, before one reading of it is treated as information, is what makes the difference in both cases. One bill is one reading. One print is one vintage of one release.

What does the Sankhya release look like measured at every vintage?

The whole argument compresses into one table, with every vintage named and every line laid out so it can be checked rather than believed.

Every line reconciles, and the middle column is the one to stare at. The consensus reads 6.40 points on all three rows because nobody revises a consensus. Subtracting it three times gives a miss of 0.20 points, then a beat of 0.10 points, then a beat of 0.40 points: 6.20 less 6.40 is a shortfall of 0.20, 6.50 less 6.40 is an excess of 0.10, and 6.80 less 6.40 is an excess of 0.40. The sign changes between the first row and the second. The size against the first print doubles by the third.

Now set those three surprises beside the movement the figure made on its own. The print went from 6.20 points to 6.80 points, a rise of 0.60 points, and that 0.60 is three times the 0.20 point surprise anybody reacted to at publication. Every one of the three surprises, at 0.20, 0.10 and 0.40 points, is smaller than the 0.60 points the figure travelled by itself, so the gap that got reported was in every case smaller than the movement that got no headline at all.