Case 063Strategy evaluation and backtestsCore

A small-cap backtest built on today's index members returns 22% a year over ten years. Over that period 30% of the original universe was delisted, with an average return of -60% in the year before delisting. Estimate the survivorship bias.

The situation

Trivenza Asset Managers wants to launch a small-cap strategy. An analyst's backtest takes the stocks in the small-cap index today, applies the strategy's rules to their price history over the last ten years, and reports a return of 22% a year.

A check of the exchange's records shows that 30% of the stocks that were in the small-cap universe ten years ago are no longer listed. In the year before they delisted, those stocks lost 60% on average. The chief investment officer asks how much of the 22% is real.

Your task

Estimate the return the strategy would have earned on the universe as it existed at each point, the size of the bias per year and over ten years, and what else could still be wrong.

Quick check

Roughly how much does leaving out the delisted stocks inflate the annual return?

Worked solution

Try it on paper, then open one step at a time.

30-second answerThe answer to give first

About 2.9 points a year: the strategy would have earned roughly 19.1% on the real universe, not 22%. Some 3.5% of names delist each year after losing 60%, and the survivors-only test simply deletes them. Over ten years Rs 1 grows to 5.76 instead of 7.30, so the backtest overstates final wealth by about 27%. Today's index also holds names that joined because they rose, so the true bias is larger still.

Step 1Why does testing on today's members bias the result?

Imagine judging how safe a mountain trek is by interviewing everyone who came back. Their stories are real, but the people who did not return are missing from the sample, and the trek looks safer than it is. A backtest run on today's index members does the same thing: every stock that collapsed and delisted along the way has been removed before the test starts, so the strategy is never asked to hold a loser. This is survivorship biasThe error of studying only the members of a group that survived to the end, which removes the failures and makes the group’s history look better than it was., and in small caps, where delistings are common, it is large.

Step 2How big is it here?

If 30% of the original universe delisted over ten years at a steady rate, the annual rate h satisfies (1 - h) to the power 10 = 0.70, so h = 3.50%. Assume, as the simplest case, that a stock earned the survivors' return in every year except its last. Each year, about 3.5% of the universe earns -60% instead of 22%, so the full-universe return is 0.965 x 22% + 0.035 x (-60%) = 19.13%, about 2.9 points a year below the backtest. Compounded, Rs 1 grows to 5.76 on the real universe against 7.30 in the survivors-only test, so the headline overstates ten-year wealth by about 27%.

The relationship

r_{full} \approx (1-h)\,r_{surv} + h\,r_{del} = 0.9650 \times 22\% + 0.0350 \times (-60\%) = 19.13\%

h	share of the universe delisting each year, 3.50%, from 30% over ten years
r_{surv}	annual return of the survivors, 22%
r_{del}	return of a stock in its final year before delisting, -60%

What it says in wordsPutting the delisted stocks back in, at the rate they actually disappeared, lowers the annual return from 22% to about 19.1%.

Over ten years Rs 1 grows to 7.30 in the survivors-only backtest but to 5.76 on the universe as it really existed, because deleting the stocks that delisted removes about 2.9 points of loss every year.

Step 3What else is still wrong with the 22%?

The delisting adjustment fixes only the names that left. Today's small-cap index also contains stocks that were too small to be in the universe ten years ago and joined because they rose, so the survivors' 22% is itself inflated by hindsight about who would grow. That effect cannot be estimated from the numbers given; it needs the historical index membership at each date. Delisting returns are also often missing from price databases, which record the last traded price before suspension rather than the near-zero value holders actually received. The right fix is not a haircut but a rebuild: run the strategy on point-in-time constituents, with delisting returns included, and report that number.

State the limitation of this estimate. It assumes delisted names matched the survivors until their final year, which is generous, since stocks heading for delisting usually lag for years before. It also treats all delistings as failures; a takeover at a premium is a delisting with a good return. Both points could move the estimate by a point or more in either direction, but neither makes the 22% believable.

Where candidates lose it

Candidates often say survivorship bias exists and stop, or produce the absurd 30% x 60% = 18 points. The interviewer wants an annual rate, the gap between the delisted return and the survivors' return, and a number.

The second miss is fixing only the delisted names. The index membership itself is chosen with hindsight, and that bias does not go away by adding back the failures; only a point-in-time universe removes it.

What the interviewer asks next

How would the estimate change if a third of the delistings were takeovers at a 30% premium?
Where would you get point-in-time index membership, and what would you check in it?
Does survivorship bias affect a long-short strategy more or less than a long-only one?

Company names and figures are illustrative.