Case 005Strategy evaluation and backtestsHard
A backtest shows 18% gross return at 9% volatility, turning the book over 60 times a year at an assumed 5 bps a side. Live, costs are 14 bps a side and only 70% of signals fill. Reconcile the backtest Sharpe with a live Sharpe near zero and decide what to fix first.
1The situation
Vetrana Systematic's short-horizon equity strategy backtested at 18% a year gross of costs with 9% annual volatility. It turns the whole book over 60 times a year, each turn a full sale and repurchase, and the backtest charged 5 bps a side.
Six months of live trading show two differences. Measured costs, including spread, impact and fees, average 14 bps a side. And only 70% of the signals actually fill, because the strategy uses limit orders that are often left behind. Live performance is roughly flat.
2Your task
Reconcile the backtest Sharpe with the live result, and say what you would fix first.
Quick check
How much annual return do the extra 9 bps a side cost?
Worked solution
Try it on paper, then open one step at a time.
30-second answerThe answer to give first
Costs explain almost all of it. At 120 sides a year, 5 bps a side costs 6%, leaving 12% net and a Sharpe of 1.33. At 14 bps, costs are 16.8% and the net falls to 1.2%. Filling only 70% shrinks gross and costs together, to about 0.84% on 6.3% volatility, a Sharpe near 0.1. Fix the cost model first, then cut turnover, because the strategy breaks even at 15 bps a side.
Step 1What did the backtest actually assume?
Convert the cost assumption into an annual number first. Sixty full turns means 120 sidesOne purchase or one sale. A round trip, buying and later selling, is two sides. a year. At 5 bps a side the backtest charged 6% a year, so its 18% gross became 12% net and a Sharpe of 1.33 on 9% volatility. A strategy that turns over this fast is a cost-sensitive machine: every basis point a side is 1.2% a year. It is like a delivery business that makes a small margin per parcel; a small rise in the fuel cost per trip can wipe out the whole profit.
Step 2How does the gap build up from 12% to nearly nothing?
Walk it one cause at a time. At the measured 14 bps a side, costs are 16.8% a year, and the net at full fill is only 1.2%. The extra 9 bps a side remove 10.8 points, nearly the entire backtest profit. The fill rate matters less than it looks: if missed trades are average trades, filling 70% scales gross return, costs and volatility alike, leaving 0.84% on 6.3% volatility. Sharpe drops from 1.33 to about 0.13.
| Annual, % | Backtest | Live costs, full fill | Live, 70% fill |
|---|---|---|---|
| Gross return | 18.0 | 18.0 | 12.6 |
| Costs | (6.0) | (16.8) | (11.76) |
| Net return | 12.0 | 1.2 | 0.84 |
| Volatility | 9.0 | 9.0 | 6.3 |
| Sharpe | 1.33 | 0.13 | 0.13 |
Step 3Why is the fill rate probably worse than it looks?
The 70% figure hides a selection problem. A resting limit buy order fills when the price comes down to it and is left behind when the price runs away. So the missed trades are disproportionately the ones where the signal was right and fast, and the filled trades are tilted towards the ones where it was wrong. If that is happening, the live gross is below 70% of 18%, and the true live result is negative. Compare the signal's forward return on filled and unfilled orders; if the unfilled ones did better, you have found the second leak.
Step 4What do you fix first?
Costs, for two reasons. They are the biggest term, and they are measurable today from the fills already in hand. At 60 turns the strategy breaks even at 15 bps a side, and live costs of 14 bps leave no room at all. Rebuild the backtest with the measured spread, impact that grows with order size, and fees and taxes, then attack turnover: a no-trade band that ignores small signal changes, or a slower signal. Halving turnover to 30 halves costs to 8.4% a year, and the strategy survives if the slower version keeps more than that of gross return.
Then run the usual checks in order, because the interviewer will ask. Look-ahead in the signal data, survivorship in the universe, and a backtest that was tuned on the same period it reports. Any of these can also produce a great backtest and a flat live record. Here the arithmetic already explains the gap, which is the point: reconcile with numbers before reaching for a theory.
Where candidates lose it
Candidates jump straight to overfitting and regime change. Those are real risks, but the interviewer has given you the cost numbers, and they alone take 12% to 1.2%. Reaching for a story before doing the arithmetic is the mark of someone who has not run a live book.
The second miss is counting turns instead of sides, which halves the cost drag and makes the gap look like a mystery.
What the interviewer asks next
- How would you measure market impact from your own fills?
- Would switching from limit orders to market orders help or hurt here?
- What turnover would give a Sharpe of 1 at 14 bps a side if gross return fell in proportion to turnover?
- How would you tell overfitting from cost underestimation using the live data?
Asked at Jump Trading, Quantitative Research, Chicago, 2018 (Wall Street Oasis): Suppose you backtested a trading strategy, it did very well. But in live trading, you keep losing money, what would you do?
Company names and figures are illustrative.
