Autocorrelation and Partial Autocorrelation: Direct and Indirect Relationships in a Series

Autocorrelation asks whether a column of readings follows itself: whether one period says anything about the next one. The misses left behind by a line scoring 0.7559 on ten invented months carry a lag one autocorrelation of 0.4862, so those misses are not independent of each other. Neither original column carries that pattern, so the fit produced it.

Two of the words in that paragraph get used loosely everywhere else, so they are pinned down here before anything is built on them. A traded unitA thing that carries a price, left without a name. Nothing computed here would come out differently if it stood for shares rather than for a basket, so the label is never filled in. All it has to do is carry one price on one date and a different price a month later. is a thing with a price on it, and its monthly change is the percentage by which that price ended the month away from where it started. A miss, here, is the gap between what a fitted line said a month would deliver and what the record says it delivered. Both of those were built earlier and are covered separately.

The clock is what is new. Sums and squares do not care which reading came first, so everything computed on this record so far would have come out identically if the ten months had been shuffled into any order at all. The two measures in this guide are the exception. Both read along the order the months arrived, and they are the only quantities anywhere in this material that a tidy-minded person can destroy by sorting the table.

Which numbers arrive already settled, and which one is finally computed here?

The record is ten paired observationsTwo figures belonging to the same case, which mean nothing apart from each other. A case here is a month, and it supplies one figure to each column. Shuffle either column on its own and the pairing breaks without anything on screen looking wrong., invented for teaching. Each month carries two figures. One records the Nakshatra unit's move for that month, and that column does the explaining. The other records the Vasant unit's move over the same month, and that column is the thing to be explained. A straight line was fitted through the pairs, and it came out as 0.5000 plus 1.5000 times the Nakshatra unit's move. Subtracting what that line said from what each month actually delivered gives the ten misses. All of that is settled and none of it is reopened here.

The last row of that table was pointed at three times earlier and left uncomputed. Fitting the line stated the figure and left it to be settled. The account of the method named independence of the misses as the one assumption this record breaks. The treatment of the misses themselves drew them against the clock and saw the shape without putting a number on it. The number is 0.4862 and the arithmetic behind it takes one line.

What does it mean for a column of readings to follow itself?

Start away from the record entirely. A tea stall sits outside one office building. Some days it sells out by four and some days it is still there at seven, and the owner wants to know whether a busy day tells him anything about tomorrow. Suppose busy days come in stretches while a conference is on or the weather has turned. Then today carries information about tomorrow, and knowing today is worth something. If busy days scatter with no pattern, today tells him nothing at all and he may as well plan every morning from scratch. The question is not whether the days differ. The real question is whether the days differ in a way that has memory.

Autocorrelation is that question turned into arithmetic, and the arithmetic is not new. Autocorrelation is the ordinary correlation, computed between a column and a copy of the same column moved along by a fixed number of periods. Nothing else changes. The readings are written down twice, side by side, the second copy slides down by one row, the rows that no longer have a partner are thrown away, and what is left is correlated. The number of rows slid by is called the lag. At a lag of one the question is whether this period says anything about the next one.

Sliding a ten row column down by one leaves nine rows with a partner. The first row of the shifted copy has nothing above it to pair with, and the last reading of the original slides off the bottom of the record. So a lag one reading on ten months rests on nine pairs, a lag two reading rests on eight, and a lag three reading rests on seven. That shrinkage matters more than it looks: it is the reason a reading at a long lag on a short record deserves very little confidence.

What is the lag one figure on this line's misses?

Written out along the clock, the ten misses run 1, 9, 8, 0, minus 5, 0, minus 3, minus 3, minus 2 and minus 5. The fitting method forces those ten misses to average out to nothing, so the arithmetic is unusually clean: multiply each miss by the miss immediately before it, add the nine products, and divide by the sum of all ten squared misses. The last of those totals is 218, and it is the same 218 that the fit statistic was built from.

The product column deserves attention before the answer does. Not one of the nine products is negative. Four of them are zero: three of the ten months sit exactly on the line or next to a month that does. Every product that is not zero is positive. A positive product means the pair sat on the same side of the line, and there is no pair anywhere in this record that sat on opposite sides. The single reading of 0.4862 is a compression of that fact, and the fact is more convincing than the reading.

The same point can be made without any multiplication at all. Across the nine consecutive pairs, the count of pairs with both misses on the same side of the line, positive with positive or negative with negative, is five. The other four all involve a month sitting exactly on the line, and a month on the line belongs to neither side. Five of the nine consecutive pairs have both misses on the same side of the line, and not one of the nine straddles it. A lag one reading of 0.4862 is what that arrangement looks like once it is compressed into a number.

Put one stretch of it into money so the size registers. Take an invented holdingA quantity of something a person actually has, priced in rupees. The round Rs 2,00,000/- below was picked for one reason only: percentages are hard to weigh in the head and rupee figures are not. of Rs 2,00,000/- whose value tracks the Vasant unit step for step. The first three months were all above the line, by 1, 9 and 8 percentage pointsWhat is left once one percentage has been subtracted from another. Climbing from 9.50 to 18.50 covers 9.00 of them. Describing the same climb as a rise of 9.00 per cent would be a claim about something else entirely, so the two get separate names throughout. in turn. On that holding those three gaps are Rs 2,000/-, then Rs 18,000/-, then Rs 16,000/- of movement the line did not account for. Rs 36,000/- of unexplained movement in one direction across three consecutive months. Every one of those figures measures the size of an error and nothing else. No one was holding this, no one made anything on it and no one was out of pocket.

How can a fit that explains three quarters of the movement fail this check?

Here are the two readings together, and neither one gets softened. The line accounts for 0.7559 of the Vasant unit's movement across these ten months. By any ordinary standard that reads as a decent fit. The misses that same line left behind carry a lag one autocorrelation of 0.4862, and a reading of that size says those misses follow one another. Both numbers come out of the same ten months, and they disagree about whether the model should be trusted.

The obvious question is why nothing computed earlier caught it, and the answer is structural rather than careless. The fit statistic is built from a sum of squared misses. The slope and the intercept are built from sums of products and sums of squares. The correlation, the typical size of a miss, the average size of a miss: every one of them is a total, and a total does not know what order its terms arrived in. Addition takes no interest in sequence, so rearranged into any order at all the ten months return the identical decimal for every one of those figures. The lag one reading pairs each reading with a specific neighbour, and it is the only quantity anywhere in this material that would change.

The point is worth stating as a rule rather than an observation. If a check would survive shuffling the record, it cannot detect anything about the order of the record. Every check performed earlier would survive shuffling. So the model passed a set of examinations none of which could have failed it on this particular point, and the one examination that could was the one nobody had run. The missing examination costs a single line of arithmetic: nine products, one sum, one division.

What is Partial Autocorrelation, and what does it take out?

Everything so far has been about lag one, where the question is simple: does this period say anything about the next one. Move to lag two and the question stops being simple. A period now sits in the middle, and that middle period has a relationship with both ends. The plain reading at lag two cannot separate the two things it is picking up. The plain reading reports whatever it finds between a period and the one two before it, whether that relationship arrived directly or travelled through the period in between.

Take the tea stall again and stretch it across three days. Suppose a busy Monday genuinely does make Tuesday busy: word gets round and stock runs low. And suppose a busy Tuesday genuinely does make Wednesday busy, for the same reasons. Now compare Mondays with Wednesdays across a whole year. Mondays and Wednesdays will look related, and the relationship will be real in the sense that the numbers show it. But Monday may have nothing whatever to do with Wednesday on its own: the entire apparent connection can be Monday reaching Wednesday through Tuesday, one step at a time.

The two measures draw exactly that distinction, and the distinction is the whole of the difference between them.

• Autocorrelation at lag two reports the total relationship between a period and the one two before it, including everything that travelled through the period in between.
• Partial autocorrelation at lag two reports the direct relationship only, with the route through the period in between taken out first.

The taking out is not a trick and it is not new arithmetic. Holding something fixed before looking at a relationship is the same idea. Every question about whether two things are related once a third thing has been accounted for needs that move. Here the third thing is the reading in between. Once the middle period's share of both ends is accounted for, the question becomes what relationship survives in what is left over. Whatever survives reaches across on its own.

At lag one there is no period in between, so there is nothing to take out, and the two measures are forced to agree. The total and the direct reading are always the same number at lag one, on every record that has ever been written down, and a reader who finds them differing there has made an arithmetic mistake rather than a discovery. The two measures only start to say different things from lag two onwards. Lag two is exactly where this record becomes interesting.

Where do the total and the direct reading disagree on this record?

Run both measures on the ten misses at three lags and lay them out together. The record only has ten readings, so lag three already rests on seven pairs and nothing beyond it is worth computing at all. The limitation belongs to the record rather than to the measure.

The first row is the forced agreement. At lag one there is no period in between and therefore nothing to remove, so the two columns hold the identical figure, 0.4862. Wherever those two columns match exactly at lag one, that is the arithmetic behaving, not the record saying anything.

The second row is why this measure exists. The total reading at lag two is 0.0183. On a scale that runs from minus one to one, that is nothing: it is the figure a reader would report as no relationship at all before moving on. The direct reading on the same eight pairs is minus 0.2856, a figure that is not nothing and points the other way.

Both are true and the reason is arithmetic rather than paradox. The misses lean positive at lag one, meaning a miss on the high side is usually trailed by a second one on the high side. The leaning carries forward. A month two steps back pulls the month in between up, and the month in between pulls the current month up, so the indirect route contributes something positive to the lag two total. Meanwhile the direct relationship, once that route is taken out, is negative. The positive route through the middle month and the negative direct relationship land almost exactly on top of each other and cancel, leaving a total of 0.0183 that reports nothing while something is plainly there.

The consequence is the practical half of this guide. A reader who computes only the total reading, sees 0.0183 at lag two, and concludes that there is nothing beyond lag one has been misled by their own check. The reader ran a real measure, read it correctly, and got an answer that was an artefact of which of the two questions they happened to ask. The direct reading at lag two is minus 0.2856 and it is invisible in the total.

Where did the pattern come from, if neither column has it?

Everything up to here has treated the misses as the thing to be examined. Turning the same measure on the two columns the misses were made from changes the character of the question. The arithmetic does not change at all: the column is written out twice, the copy slides down one row, and the nine surviving pairs are correlated. Only the column being fed in is different.

The Nakshatra unit's own monthly changes, in the order they arrived, give a lag one reading of minus 0.4167. Minus 0.4167 is negative, and a negative reading at lag one means the opposite of persistence: a move in one direction tends to be followed by a move back the other way. The Nakshatra unit alternates rather than repeats. The Vasant unit's own monthly changes give 0.0305, and on a scale running from minus one to one that is nothing whatever. The Vasant unit does not follow itself in either direction.

Now put all three readings on one line. Minus 0.4167 for the input column. 0.0305 for the outcome column. And 0.4862 for what was left over after a straight line was drawn through the two of them. Neither column that went into the fit carries persistence, and what the fit failed to explain carries it strongly. The pattern was manufactured by the model rather than inherited from the record.

The 0.4862 is a statement about the line and about nothing else. The reading says the straight line is the wrong description of how these two columns are related: it runs high for a stretch of months and then low for a stretch, and the leftovers keep a memory of which stretch the record is in because the shape of the relationship changes and the line does not. The reading says nothing at all about either traded unit having a rhythm somebody could lean on, and the readings above are the proof that neither does. A reader who comes away thinking the Vasant unit repeats itself has read the exact opposite of what the arithmetic says.

One warning about that 0.0305, and it is worth reading twice. A completely different quantity, arrived at by an unrelated calculation and covered separately, happens to come out at exactly the same four digits on this same record. The match is a coincidence of the arithmetic and not a connection: neither figure can be derived from the other, and lining them up would be a mistake this record makes unusually easy. Here 0.0305 means one thing only: the Vasant unit's own lag one reading.

What follows once the misses are not independent?

Three things follow, and one of them is an instruction. Take them in order: two of them are commonly overstated and the third is commonly skipped.

The first is about confidence rather than about the numbers. Every statement of how firmly a coefficient is pinned down rests on counting how much independent information the record contains, and that count assumed each month's miss carried its own separate news. Here they do not: a miss says something about the next one, so ten months contain less independent information than ten months appeared to. The slope of 1.5000 is still 1.5000. The arithmetic behind the standard error of 0.3014 counted information twice, so the claim that the slope is pinned down to within 0.3014 is now too confident. The degrees of freedomHow many readings remain free to move after a calculation has already claimed some of them. Fitting a straight line claims two, one for each figure the line is made of, so a record of ten leaves eight behind. that figure divides by are a count of readings, and they cannot tell that some of those readings are partly repeats of each other.

The second is what does not follow, and it needs saying because the first point frightens people into throwing the fit away. Nothing above says the line was calculated wrongly. The slope and the intercept came out of the arithmetic correctly, the fitted valueWhat the line says a month should have delivered, worked out by feeding that month's input through the line. It sits beside the figure the month actually delivered, and the gap between them is the miss. for each month is what the line genuinely says, and 0.7559 really is the share of movement the line accounts for. A claim about independence is what has failed, and the damage falls on every measure of certainty attached to the fit rather than on the fit itself.

The third is a lead rather than a verdict. Something systematic is sitting in the misses that the line did not pick up. In practice that means one of two things: an input is missing, or the shape is wrong. On this record it is the shape, and the shape is visible without any arithmetic: the misses run high for the first stretch of months and low for the rest. A single straight slope is being asked to describe a relationship that changed partway through the record, and the leftovers keep the memory of which stretch the record is in.

The instruction is one line long: report the figure beside the fit, in the same sentence, not in a footnote nobody reaches. Procedures that adjust a fit for misses that follow one another are covered separately. Modelling a column directly from its own past is covered separately too, and these two measures are usually the doorway into it.

How is the check actually run, in four lines?

The check is the part an analyst reviewing somebody else's model note does in under five minutes, and it is worth having as a habit rather than as knowledge. Think of a shopkeeper checking the till roll: the totals can be perfect and still hide the fact that every short day fell in the same week. Order is information, and it is the first thing a tidy spreadsheet destroys.

1. Keep the observations in the order they arrived and never sort them. Sorting by size makes a neater chart and turns every figure in this guide into a number about a record nobody ever observed. If the table has already been sorted and the original order is gone, the check cannot be run at all and the honest thing to write is that it could not be run.
2. Plot the misses against the clock and count the runs on one side. No arithmetic yet. On this record the eye finds it immediately: three months above the line and then six at or below it. Five of the nine consecutive pairs land on the same side.
3. Compute the lag one reading, one line of arithmetic. Each miss multiplied by the one before it, the nine products added to give 106, divided by the 218 already available from the fit, reads 0.4862.
4. Quote it beside the fit statistic, every time, in the same sentence. Two figures reported together are a report. Either one on its own is a selection, and the selection always flatters the model. The flattering figure is the one that gets computed by default.

The sampleThe handful of cases somebody managed to collect, doing duty for the far larger set nobody collected. Ten months is one of those handfuls, and the question it is answering is about every month there could ever be. here is ten months. Ten is very few, and the check is correspondingly rough: nine pairs at lag one, and fewer at every lag beyond it. Roughness is not a reason to skip it. A rough check that can fail is worth more than a precise one that cannot, and every other figure computed on this record belongs firmly in the second category. Hold the misses against the spreadA measure of how widely a set of readings scatters about its own middle. Large means the readings are thrown far apart; small means they sit close together. of the outcome itself and the same lesson appears twice: the line took a column wandering by about ten percentage points and left leftovers of about five. The improvement is real, and five points is still a great deal of movement to be following a pattern of its own.

The closing line of all of this is one sentence: the same ten months that produced the good looking number produced the evidence against it, and both belong in the same paragraph. A fit statistic of 0.7559 and a lag one reading of 0.4862 were computed from one record, by one person, on one afternoon. Whether a reader ever meets the second figure is a decision about what gets written down, and that decision is made by whoever writes the note rather than by the arithmetic.

What is covered elsewhere