Case 092Strategy evaluation and backtestsWarm up
Kadvari's signal uses each day's closing price to trigger a trade executed at that same close. Re-run with execution at the next open and the annual return falls from 15% to 4%. Where did the 11 points go, and how do you fix the timing?
1The situation
Kadvari Quant's junior researcher has a breakout signal: when a stock closes above its 20-day high, buy; sell when the close falls back below a trailing level. The backtest computes the signal from each day's closing price and fills the trade at that same closing price. It shows 15% a year from about 50 trades.
A reviewer changes one line so trades fill at the next day's open, and the return drops to 4%. The researcher thinks the open is a noisy price and wants to keep the close. You are asked to settle it.
2Your task
Explain where the 11 points went, show it with a three-day example, and say how to set the timing correctly.
Quick check
Why is filling at the same close the signal uses a problem?
Worked solution
Try it on paper, then open one step at a time.
30-second answerThe answer to give first
The 11 points are look-ahead: the backtest buys at the very close that triggered the signal, a price no order could have reached, and pockets the move from that close to the next open. Over 50 trades the overnight gap at entry is worth about 12.5 points, partly given back at exit. Fix the timing by trading at the next open, or by computing the signal from a price before the close and testing a fill in the closing auction, then add costs.
Step 1Why can't the backtest trade at the close it uses?
Imagine betting on a cricket match with a bookmaker who accepts the bet after the last ball. Anyone could win that way, and no bookmaker allows it. A closing price is known only when trading has finished, so a signal that needs the close cannot also be executed at the close. This is a look-ahead error, and it is easy to make because in a daily data file the close and the signal sit on the same row, one column apart. The backtest is, in effect, placing the order after seeing the result.
Step 2Where did the 11 points go?
Follow one typical trade. On Day 1 the stock closes at 100.00, above its 20-day high, and the signal fires. The backtest buys at 100.00. A live system learns of the signal after the close and buys at Day 2's open, 100.25. On Day 2 the close of 100.30 triggers the exit; the backtest sells there, the live system at Day 3's open, 100.33. The backtest earns 0.30%, the live trade 0.08%: the 0.25 move overnight after the signal is exactly the part of the trade no one could have caught. It is not random either: a strong close that breaks a 20-day high tends to be followed by a higher open, because other traders read the same close overnight.
Scale it up. With 50 trades a year, entry at the next open costs 50 x 0.25 = 12.5 points, and exiting at the next open instead of the close happens to give back 1.5, because the drift continues a little after the exit signal too. Net, 15% becomes 4%: most of the backtest's return was the overnight move the signal could not have traded.
| R bt | backtest return a year, filled at the signal close, 15% |
| n | trades a year, 50 |
| g entry, g exit | average move from the signal close to the next open at entry, 0.25%, and at exit, 0.03% |
Step 3How do you fix the timing?
There are two honest designs. Either compute the signal at the close and trade at the next open, which is what the reviewer did, or compute it from a price available before the close, say 20 minutes earlier, and test a fill in the closing auction at the real closing price. The second keeps more of the overnight move but changes the signal, since the early price is not the close, so it must be backtested as its own strategy. Either way, add trading costs, which for a 4% strategy with 50 round trips a year are a large share of what remains. Then sweep the rest of the code for the same mistake: adjusted prices that use later corporate actions, index membership as of today, and fundamentals stamped with their period end rather than their release date.
Where candidates lose it
The usual loss is defending the close as the 'cleaner' price and calling the open noisy. Noise is not the issue; availability is. A price you cannot trade at is not a price at all for a backtest.
The second is treating 4% as a disappointment caused by the reviewer. The 4% is the strategy; the 15% never existed. Reporting the honest figure early is what a research desk wants from a junior.
What the interviewer asks next
- How would you test whether the strategy can still capture part of the overnight move?
- What other look-ahead errors are common in daily backtests?
- How would trading costs of 10 basis points a side change the 4%?
- Why might a mean-reversion signal show the opposite gap at the open?
Company names and figures are illustrative.
