A balanced advantage fund's model sets equity at 80% when market price to book is below 2.5 and 30% above 4.0, sliding in between. The backtest shows a worst drawdown of 18% against 38% for the index over 15 years. How much of that is real, and how would you test it before launch?

Most of the 20-point gap is just holding less equity; the rule's own edge is about 3 points, and only in falls that start from expensive markets. A static 55/45 mix loses about 20.9% in the same fall. The backtest crash began at a price to book of 4.65; a shock from 2.9 would leave the fund at 67% equity and a drawdown near 31%. Test out of sample, shift the thresholds, add costs and count the independent events before quoting any number.

With the wrong thing. A cyclist who rides at half speed will fall less hard than a racer; that does not prove his braking is better. Any fund holding about 55% equity on average would lose far less than the index in a crash, so the rule's skill is measured against a static mix of the same average weight, not against 38%. A fixed 55/45 portfolio loses about 20.9% in a 38% fall. The rule's own contribution is the gap from 20.9% to 18%, about 3 points, not 20.

It helps when the fall starts from an expensive market, because that is when it holds little equity. Working back from the reported 18%, the backtest's big fall must have begun near a price to book of 4.65, where the rule held 30% equity, adding more as prices fell. A shock that starts from a fair valuation, a price to book of 2.9, finds the fund at 67% equity and costs about 30.8%, worse than the static mix. Pandemics, wars and credit events do not wait for markets to be expensive. A valuation rule protects against valuation-driven falls and nothing else.

Because the thresholds were chosen by looking at the answer. With two large falls in 15 years, a team can always find two numbers that sidestep both; that is overfittingChoosing a model or its parameters so that it fits the history it was built on, including the noise in that history, which makes it look better than it will perform on new data.. Two events are not a sample. Shift each threshold by 0.2 and the drawdown moves to 19.8% or 16.4% in the modelled fall, a reminder that small choices move the headline. There are quieter biases too. Price to book uses book values that are published weeks after the quarter ends; a backtest that uses them on the quarter-end date knows something the fund could not have known. And the index's own price to book has drifted up over decades as its mix shifted towards asset-light companies, so a fixed threshold means different things in different years.

Six, each designed to break the claim. First, walk forward: fit the thresholds on the first ten years and test on the last five, then repeat on rolling windows. Second, test on other data the team did not see: earlier decades, or other markets with similar indices. Third, perturb the thresholds and the slope and report the range of drawdowns, not the best one. Fourth, lag the book value to the date it was published. Fifth, add costs: trading, the cost of the arbitrage positions used to hedge, and the tax and category rules that set a minimum gross equity. Sixth, compare against the static mix with the same average weight on drawdown and on return, because the rule pays for protection by holding less equity in rallies.

Close with the view for the product committee: launch the fund if the walk-forward and perturbation tests hold, but never market the 18% against 38%. The honest claim is narrower: in falls that begin from expensive markets, the rule has historically lost a few points less than a static mix of the same average equity weight. The limit of this analysis is that the crash model here is stylised, a straight 38% decline with the rest of the portfolio flat; the real tests have to be run on actual monthly data.