Differencing: Turning Levels Into Changes, and Why Returns Use Logs

Differencing replaces each observation with how much it changed from the one before. A level that wanders becomes a change that does not. The Nakshatra unit's price averages Rs 115.0525/- across three years and Rs 190.4591/- across the next three. Its monthly change averages 1.00 per cent across both. There are three ways to write that change, and only one of them adds up.

Where these numbers come from. The six year record is an arrangement of 72 monthly changes written for teaching. Every price, difference, total and coefficient printed below was worked out from that arrangement by the arithmetic set out here. No maintained series stands behind the record, so there is no as-of date and no vintage to look up. A recipe that can be rerun stands in place of both.

The six year record is 72 monthly observations of the Nakshatra unit, an invented object built for teaching, running from the month ending 31 January 2019 to the month ending 31 December 2024, measured out from an opening markThe figure a record is counted out from. The opening mark sits one step ahead of the first observation and is a starting line somebody drew, not a month that happened. On this record it is Rs 100.00/- dated to the end of December 2018. of Rs 100.00/- at the end of December 2018. The record closes at Rs 187.4539/-, quoted to the paisaOne hundredth of one rupee. Figures here run to four places after the point so a reader can rebuild them exactly. The extra places record how the arithmetic was carried. No claim is made about what anybody could pay. and past it. Arithmetic can be carried further out than any market quote.

Two things are settled elsewhere and used here without being rebuilt. The price reading of that record was separated from the return reading, and the price sits still for nobody while the monthly change does. One operation leads from the first reading to the second, and it has an awkward feature: it produces three different answers, depending on how the subtraction is written.

Here is the everyday version before any of the arithmetic. A vegetable stall outside one office building can report two completely different things at the end of a day. The stallholder can report what is in the cash box, a level: Rs 8,400/-. Or the stallholder can report what came in today, a change: Rs 900/-. Both are true. The cash box figure is mostly a record of every day that came before it, so only the change tells the stallholder whether the day went well. Differencing is the move from the first sentence to the second, done to a column instead of to a cash box.

What does differencing a series actually do?

Take a column of observations in date order. Differencing writes a new column beside it, in which each row holds the amount by which that observation moved away from the one directly before it. The subtraction is the whole operation. There is no fitting, no smoothing and no estimating in it, and nothing about the record is being modelled: it is a subtraction repeated 72 times.

Two things happen immediately, and both of them are worth naming. The first is small and mechanical. The very first observation has nothing sitting before it to be compared against, so the differenced column is one row shorter than the record it came from. The six year record holds 72 monthly changes precisely because it holds 72 months and one opening mark. Missed, that fact eventually leads to two columns lined up by row number and every month set against the wrong date.

The second is not mechanical at all. The new column answers a different question from the old one. The price column answers what one unit cost on a date. The differenced column answers how much it moved over a month. Differencing is not a tidier version of the same record, it is a change of question, and the answers to the two questions behave so differently that almost everything in these notes hangs on telling them apart. A price of Rs 187.4539/- in December 2024 carries the whole six years inside it. A change of 14.00 per cent in December 2024 carries only December.

What did differencing do to the six year record?

Both columns were measured in the notes on whether a series sits still, and their readings are taken as given here. Side by side, the two columns behave nothing alike.

The price column drifts. Averaged across the first three years it sits at Rs 115.0525/-, and averaged across the last three it sits at Rs 190.4591/-, a gap of Rs 75.4066/-. Its month to month autocorrelationOne number saying how closely a series matches a copy of itself shifted back by a fixed number of steps. The figure runs between minus one and one. A figure near one means this month looks very like last month. is 0.9650, which says that this month's price is very nearly last month's price. And the regressionFitting a straight line through pairs of numbers to describe how one moves with the other. The slope of that line is the coefficient quoted here. Earlier notes build the method in full. that asks whether the column is pulled back towards any level gives a coefficient of minus 0.026109, which is about nothing at all. A column that behaves like that is said to carry a unit rootThe property of a level column that has no home value to be tugged back to, so a shock to it never wears off and simply becomes part of where the column now sits. The unit root is settled in its own notes., and what that means is settled separately.

The differenced column does none of that. Its average across the first three years is 1.00 per cent and its average across the last three is 1.00 per cent, to the last decimal place. Its month to month autocorrelation is 0.2011. The same regression on the differenced column gives a pull back coefficient of minus 0.777838, a firm pull rather than nothing. Which is why the notes on whether a series sits still say the differenced column is stationarySaid of a column whose behaviour does not depend on where in the record it is read. The notes on whether a series sits still set out what the word does and does not promise, and their verdict is taken as given here. in the specific sense they define, and no more than that.

Differencing removed the wandering level and the drifting average, and it left two other things in this record exactly where they were: the pattern that repeats with the calendar month, and the spreadHow widely the values of a column are scattered around their own average, written in the same units as the column itself. Earlier notes build it and both of its denominators. that widens partway through the six years. Both of those are covered separately and neither is fixed by this operation. Taking the calendar pattern out is a seasonal adjustmentSubtracting the part of each observation that belongs to its position in the calendar year, so what is left shows the movement that was not simply the time of year. Seasonal adjustment has its own notes., which is its own subject and is not what differencing did here. Anybody who ends on the sentence that differencing has made a record well behaved has overclaimed on this very record. The differenced column still carries a January that averages 5.00 per cent against a June that averages minus 3.00 per cent, and a second stretch more than twice as wide as the first.

The autocorrelation figures above need one caution attached to them. There is more than one estimator in circulation and they do not agree. The figures above use a single mean for the whole record and the full sum of squared deviations in the denominator, the ordinary convention. Another estimator shrinks the denominator with the shift, and a third computes a fresh mean for each of the two shifted stretches. All three are called the autocorrelation and all three land on slightly different numbers, so quoting one without naming the convention hands the reader a figure they cannot reproduce.

Why does subtracting in rupees not work?

The first and most obvious way to difference a price column is to subtract last month's rupee figure from this month's. Call it the rupee difference. The rupee difference is a real quantity, it is easy to compute, and on this record it fails at the one job differencing was brought in to do.

Six of the 72 months of the six year record carry a change of exactly 6.00 per cent: January 2020, December 2020, October 2021, February 2022, July 2023 and September 2023. Identical months, by the only measure that describes the move itself. Now write them as rupee differences. In January 2020 the price went from Rs 103.8112/- to Rs 110.0399/-, a rupee difference of Rs 6.2287/-. In September 2023 it went from Rs 211.2546/- to Rs 223.9298/-, a rupee difference of Rs 12.6753/-, or 2.0350 times as much for the identical month.

Nothing about those two months differed. The price sitting underneath them differed, and 2.0350 is exactly the ratio of the two starting prices, Rs 211.2546/- over Rs 103.8112/-. The ratio of the starting prices is the whole diagnosis. A rupee difference is the size of the move multiplied by the level the move started from. It measures the level just as much as it measures the movement, and level contamination is precisely what differencing was supposed to clear out.

The distortion runs across a whole record. Across the first three years the rupee difference averages Rs 2.8836/- in size, and across the last three it averages Rs 9.8439/-. The rupee difference column, read on its own, suggests the Nakshatra unit became a wildly more eventful object. The unit did become more eventful, and the notes on a changing spread say so, but not by anything close to that multiple: most of the difference is simply the price having climbed.

What does the per cent change fix?

The second way to difference is to divide rather than subtract. Divide this month's price by last month's, subtract one, and write the result in per cent. The result is the per cent change. For January 2020 the sum is Rs 110.0399/- over Rs 103.8112/-. The division comes to 1.0600. Taking away one leaves 6.00 per cent. For September 2023 the sum is Rs 223.9298/- over Rs 211.2546/-. The division comes to 1.0600 as well, and so to 6.00 per cent as well.

The division is what does the work: rupees on the top and rupees on the bottom cancel each other, so the answer carries no level inside it at all, and two months at completely different prices land on the same number when the same thing happened. This is the reason the return reading of a record is the one that can be put beside anything else. Two objects, one priced in the tens and one in the thousands, produce per cent changes on a single common scale.

So the per cent change has fixed the failure of the rupee difference and it is the reading almost everybody quotes. The per cent change is also, for one particular job, the wrong instrument, and that job is adding a column of changes across many months.

What is a Log Difference, and how is it computed?

The third way to difference is to take the natural logarithm of this month's price, subtract the natural logarithm of last month's, and write the answer in per cent. The result is the log difference. In January 2020 it comes to 5.8269 per cent, and in September 2023, again, 5.8269 per cent. The log difference shares the good property of the per cent change: it is a ratio underneath, so the level cancels and two months at different prices agree.

The log difference has one property in addition, and that property is the entire reason anybody bothers with it. The logarithm turns multiplication into addition. The log of two numbers multiplied together is the log of the first plus the log of the second, a fact about logarithms rather than a fact about markets. And a price path is nothing but multiplication: each month multiplies whatever the month before finished at.

The two facts together give the property that matters: log differences add up. Add the 72 log differences of the six year record and they come to 62.8363 per cent. Now throw the 72 months away and compute the log difference of the whole six years in one step, from the opening mark of Rs 100.00/- to the close of Rs 187.4539/-. The single step comes to 62.8363 per cent as well.

The word exactly is doing real work in that sentence and is not a rounding claim. Carried further out, the sum of the 72 is 62.8362720980 and the single step is 62.8362720980. The sum of the logs of 72 ratios simply is the log of their product, so the two figures agree to every digit either one is carried to, on any record whatever. Nothing about the agreement is approximate and nothing about it depends on a well behaved record. The agreement is arithmetic.

Why does it matter that log differences add up?

Because the column everybody actually has in front of them, the per cent change, does not add up, and the shortfall on this record is not small.

Add the 72 per cent changes of the six year record. The 72 changes come to 72.00 per cent, and they come to exactly that because the record's monthly change averages exactly 1.00 per cent, 72 times over. Now ask what the record actually did between the end of December 2018 and the end of December 2024. The record went from Rs 100.00/- to Rs 187.4539/-, a total change of 87.4539 per cent. The addition falls short by 15.4539 percentage pointsThe plain subtraction between two figures that are already written in per cent. The step between 72.00 per cent and 87.4539 per cent measures 15.4539 percentage points. Reporting that step as 15.4539 per cent would mean something quite different. on a six year record, which is not a rounding matter and is not a fault in anybody's arithmetic.

The reason is compoundingA run of changes multiplying together instead of piling up, because every period works on the amount the period ahead of it ended with. Two rises of 10 per cent leave a figure 21 per cent higher and not 20., and the household version makes it obvious. A salary rising 10 per cent a year for two years is not up 20 per cent but up 21 per cent. The second rise lands on a salary that already carries the first. Over two years the extra one point is easy to ignore. Over 72 months of a record that mostly rose, the extras stack into 15.4539 percentage points.

Notice which way it went. Most of this record's months were rises, and each later rise landed on a bigger price than the addition was crediting it with, so the addition came out too low rather than too high. A record that mostly fell would break the other way, and a record with no trend at all would have the two figures much closer together. The habit survives long enough to do damage for exactly that reason.

What does using logs cost?

A log difference is not a per cent change written differently. The log difference is a different number, and on small months the difference is invisible while on large months it is not.

Work through the record's own months. A 1.00 per cent month reads as 0.9950 per cent as a log difference, a difference of half a hundredth of a point that nobody would ever notice. A 6.00 per cent month reads as 5.8269 per cent. October 2022, the record's largest month at 16.00 per cent, reads as 14.8420 per cent, more than a full point adrift. And July 2024, the record's smallest month at minus 13.00 per cent, reads as minus 13.9262 per cent, nearly a point adrift in the other direction.

The shape to remember is that the two readings sit on top of each other near zero and pull apart as the month gets bigger, and they do not pull apart symmetrically. A rise of 20.00 per cent reads as 18.2322 per cent, a gap of about 1.77 points. A fall of 20.00 per cent reads as minus 22.3144 per cent, a gap of about 2.31 points. The log difference always sits below the per cent change. For a fall that means further from zero rather than closer to it, and the gap on the falling side grows faster.

The practical rule follows without any theory. On monthly data of ordinary size the two are near enough to be confused, and that is exactly why they get mixed. A reader with no label has no way of telling from the numbers alone until a month gets large. Never put both in one column, never label one with the other's name, and always state which of the two a column holds.

How many times should a series be differenced?

Once, on this record, and the differenced column's behaviour shows it. The test is the one restated further up: the month on month movement of a column is regressed on its own previous value and the coefficient read off. On the price column it came out at minus 0.026109, no pull at all. On the differenced column it comes out at minus 0.777838, a firm pull. The pull has appeared, so the job is finished.

Differencing a column that already has the pull adds noise and destroys information worth keeping, so differencing runs until the pull appears and then stops. That is not a caution, it is a measurable cost on this record. Differencing the change column a second time raises its spread from 5.00 per cent to 6.1712 per cent, 1.2342 times as wide, making the record 23 per cent noisier for nothing. Worse, the second differencing manufactures memory that was never there: the month to month autocorrelation moves from 0.2011 to minus 0.4654, a strong alternating pattern that is a pure artefact of the operation. Anybody handed that column would set about explaining a swing that the record does not contain.

There is a third case worth naming. If a column still shows no pull after two differencings, the honest reading is that something about the record needs looking at rather than that a third differencing is due. Perhaps the record is two different stretches glued together, perhaps the observations are not evenly spaced, perhaps the column is not what its heading claims. Differencing is a subtraction and a subtraction cannot repair a record that was assembled wrongly.

What do the four columns look like against the same dates?

Here is the whole of it in one table. One record, four columns, the same 72 dates underneath all of them. The two rows that carry the argument are picked out: two months that both changed exactly 6.00 per cent, agreeing in three columns and disagreeing wildly in the fourth.

Read the two highlighted rows across. The price differs, the rupee difference differs by a factor of 2.0350, and the last two columns are identical to four decimal places. Two rows of one record make the case for dividing rather than subtracting.

Only one of those three rows was produced by adding a column, and it is the only one entitled to be. The other two are separated by 15.4539 percentage points, and the whole gap is compounding that the addition never accounted for.

How does anybody use this at a desk?

Anyone who hands a differenced column to another person, or to their own future self, should attach three statements to it. None of them is optional and all three are routinely missing.

State which of the three differences the column holds. A column headed simply change could be rupees, per cent or logs, and a reader who guesses wrong will be out by 15.4539 percentage points over six years or by nothing at all over one quiet month, with no way of telling which. A column differenced twice looks like an ordinary column and behaves like an artefact, and the alternating memory of minus 0.4654 computed above is what that artefact looks like from the inside. State how many times the column was differenced.

And state what the record started at. A differenced column cannot be turned back into a level without one starting value, and the level is not hiding inside the changes anywhere. Somebody given the 72 log differences of the six year record can say that the record grew 87.4539 per cent. Both records produce exactly the same 72 changes, so that person cannot say the record went from Rs 100.00/- to Rs 187.4539/- rather than from Rs 10,000/- to Rs 18,745.39/-. The change column simply never knew.

The everyday version is a household that keeps a note of every month's income and outgo but never writes down what was in the account on the first day. Every month is documented and the balance is unrecoverable. The loss is not a rare accident with data either: the usual way it happens is that somebody computes the differences, is pleased with the tidy new column, and deletes the price column to save space. The starting value goes with it, and nobody notices for a year.