Case 047Regression and model reviewHard

You track the hedge ratio between two cement stocks with a Kalman filter: prior beta 1.00 with variance 0.04, process noise 0.001 a day, and a new observation implying 1.20 with noise variance 0.02. Run two updates and explain the gain.

The situation

Ekatvam Quant runs a pairs book in two invented listed cement makers, Silvanta Cement and Oradi Cement. It holds Rs 5 crore of Silvanta and hedges with a short position in Oradi sized by the hedge ratio, the beta of Silvanta's daily return on Oradi's. Rather than re-running a rolling regression every day, the desk tracks beta with a Kalman filter.

The current estimate is beta 1.00 with a variance of 0.04, so a standard deviation of 0.2. The desk allows beta to drift by a random step with variance 0.001 each day. Today's pair of returns implies a beta of 1.20, and a single day's reading is noisy, with variance 0.02. Tomorrow's reading implies 1.10 with the same noise.

Your task

Run the predict and update steps for both days, give the gain, the new estimate and its variance each time, explain what the gain is doing, and say what it settles to and how the hedge changes.

Quick check

After today's update, where does the hedge ratio estimate land?

Worked solution

Try it on paper, then open one step at a time.

30-second answerThe answer to give first

Day one: gain 0.67, beta 1.134, variance 0.0134. Day two: gain 0.42, beta 1.120, variance 0.0084. The gain is the share of the surprise the filter believes: predicted variance over predicted plus reading variance. It is high while the estimate is uncertain and falls as confidence builds, settling near 0.20, where each day's drift balances each day's noise. The Oradi hedge moves from Rs 5.0 crore to about Rs 5.6 crore.

Step 1What are the two steps, in plain words?

Think of a ship's navigator. Between star sightings she moves her position estimate forward by dead reckoning and becomes a little less sure of it; when a sighting arrives she moves the estimate towards it, by more if the sighting is clear and her estimate is vague, by less the other way round. A Kalman filter does exactly two things each day: predict, which adds the process noise to the uncertainty, and update, which moves the estimate towards the reading by a share called the gain. Here beta is assumed to follow a random walk, so the prediction keeps the estimate at 1.00 and widens the variance from 0.04 to 0.041.

The relationship
P−=P+Q,K=P−P−+R,β^+=β^+K (z−β^),P+=(1−K) P−P^- = P + Q, \qquad K = \frac{P^-}{P^- + R}, \qquad \hat\beta^+ = \hat\beta + K\,(z - \hat\beta), \qquad P^+ = (1 - K)\,P^-
Pvariance of the current beta estimate, 0.04
Qprocess noise, how far beta may drift in a day, 0.001
Rnoise variance of one day's reading, 0.02
zthe beta the day's returns imply
Kthe Kalman gain, the share of the surprise accepted
What it says in wordsWiden the uncertainty for a day's drift, then move the estimate towards the reading by the share that reflects how much more you trust the reading than your own estimate, and shrink the uncertainty accordingly.
Step 2What happens on day one?

Predict: P is 0.04 + 0.001 = 0.041. Gain: 0.041 over 0.041 + 0.02 is 0.672. The surprise is 1.20 - 1.00 = 0.20, so the estimate moves by 0.672 x 0.20 to 1.1344. The filter believes two-thirds of the surprise because its own estimate is about twice as uncertain as the reading. The new variance is (1 - 0.672) x 0.041 = 0.0134, smaller than either the prior's 0.041 or the reading's 0.02: two independent pieces of evidence combined are sharper than either alone.

Day one: the estimate moves two-thirds of the way to the new reading0.60.81.01.21.41.6Hedge ratio, beta of Silvanta on Oradiprior after one day's drift1.00, variance 0.041the day's reading1.20, variance 0.020posterior 1.134variance 0.0134gain K = 0.67moves 67% of the gap from 1.00 to 1.20
On day one the predicted prior at 1.00 has variance 0.041 and the reading at 1.20 has variance 0.020, so the gain is 0.67, the estimate moves to 1.134, and the posterior variance falls to 0.0134, narrower than either input.
Step 3And on day two, why does the same-sized reading move it less?

Predict: P is 0.0134 + 0.001 = 0.0144. Gain: 0.0144 over 0.0144 + 0.02 is 0.419. The reading of 1.10 sits 0.034 below the estimate, so beta moves down by 0.419 x 0.034 to 1.1200, and the variance falls to 0.0084. The gain fell from 0.67 to 0.42 because the filter is now more confident than it was yesterday; nothing about the reading changed.

StepDay 1Day 2
Estimate before1.00001.1344
Predicted variance P-0.04100.0144
Reading z1.201.10
Gain K0.67210.4193
Estimate after1.13441.1200
Variance after P+0.013440.00839
Oradi hedge on Rs 5 crore, Rs crore5.675.60
The gain falls from 0.672 to 0.419 as the variance shrinks, beta goes from 1.00 to 1.134 and then 1.120, and the short Oradi hedge on Rs 5 crore of Silvanta moves from Rs 5.00 crore to Rs 5.67 crore and then Rs 5.60 crore.
Step 4Where does the gain settle, and what sets it?

Run the recursion forward and the predicted variance stops changing when what the update removes equals what the day's drift adds. Solving that gives P- of 0.0050 and a steady gain of 0.20. A steady gain of 0.20 makes the filter an exponentially weighted average of past readings with a half-life of about 3.1 trading days, and the ratio of Q to R alone decides that speed. Set Q to zero and the gain keeps falling towards zero: the filter becomes an ever-lengthening regression that eventually ignores new data, which is exactly wrong for a hedge ratio that does move.

The gain settles where daily drift balances daily noise0.20.40.60.8151015Update number, one a trading dayKalman gainsteady state 0.20day 1: 0.67day 2: 0.42no drift allowed, Q = 0:falls to 0.06 by day 15 and keeps falling
With process noise of 0.001 a day the gain falls from 0.67 on day one to 0.42 on day two and settles near 0.20, while with no process noise it keeps falling, to 0.06 by day fifteen, so the filter stops responding to a beta that has changed.

Say where the R of 0.02 comes from, because the interviewer will ask. One day's implied beta is Silvanta's return divided by Oradi's, and its noise is the regression's residual variance divided by Oradi's squared return. A quiet day for Oradi is a nearly useless reading, so a careful desk lets R change daily and the gain falls automatically on quiet days. The other limit is that a gain of 0.67 on day one rebalances the hedge by Rs 0.67 crore on a single day's evidence; most desks put a no-trade band around the hedge so that noise in beta does not become turnover.

Where candidates lose it

The usual loss is forgetting the predict step: computing the day-one gain as 0.04 over 0.06, 0.667, and then on day two reusing the posterior variance without adding Q. Without the 0.001 the gain collapses day after day, and the filter quietly turns into an expanding-window regression.

The second is describing the gain as a fixed smoothing constant chosen by taste. It is computed from the two variances each day, and the candidate who can say why it falls from 0.67 to 0.42 has shown they understand the filter rather than its formula.

What the interviewer asks next

  • What would you change if Oradi's return on day two had been only 0.1%?
  • How would you estimate Q and R from history rather than choosing them?
  • Compare this filter with a 60-day rolling regression: which reacts faster to a genuine change, and which is noisier?
  • Add an intercept to the hedge, so the state has two elements. What does the variance become?

Company names and figures are illustrative.