Hedge Funds puzzles, solved step by step

Puzzles
100
Traced to a firm
38
Topics
14
Hard
30
Topic
All topicsBetting and sizing5Conditional probability and Bayes7Continuous probability and distributions7Counting and combinatorics7Estimation and mental maths4Expected value and dice games8Logic and brainteasers10Market making and trading games6Options and payoffs5Portfolio and risk maths8Random walks and Markov chains7Returns, compounding and fees7Statistics and estimation11Valuation, accounting and macro riddles8
Level
AnyWarm upCoreHard
Source
AnyReported at a firmStandard
Showing 1–1 of 1 · filtered from 100Clear filters
  1. 013A researcher tests 20 unrelated trading signals, each at a 5% significance level, and none of them truly works. What is the chance that at least one of them looks significant?Statistics and estimationWarm upQuant and systematic funds

    Try it first

    Your instinct: the chance of at least one false discovery?

    Show the worked solution

    About 64.2%. A useless signal clears a 5% bar by luck one time in twenty. The chance that all 20 stay insignificant is 0.95 to the twentieth, about 35.8%, so the chance that at least one looks like a discovery is 64.2%. On average the search turns up one false signal, as 20 x 5% suggests, but at least one appears in roughly two searches out of three.

    Why does testing more ideas manufacture a winner?

    Ask twenty friends to flip a coin five times each. Any one of them flips five heads only once in 32 tries, yet the chance that at least one of the twenty does is 47.0%, and that friend will look gifted. Each test is a lottery ticket for a false discovery, and buying twenty tickets makes a win likely even when nothing works. A signal chosen because it looked best among twenty has not passed a 5% test; it has passed a 64.2% one.

    Test enough useless signals and a false winner becomes likely25%50%75%5.0%122.6%540.1%1051.2%1564.2%205%What each single test promises: 5%What the search delivers at 20 tests: 64.2%Number of useless signals tested, each at 5%
    The chance of at least one false positive rises from 5% for one useless signal to 40.1% for ten and 64.2% for twenty, passing even odds at 14 tests, although every individual test is run at 5%.
    The relationship
    P(at least one false positive)=1−(1−α)m=1−0.9520≈0.642P(\text{at least one false positive}) = 1 - (1-\alpha)^m = 1 - 0.95^{20} \approx 0.642
    \alphathe significance level of each test, 5%
    mthe number of independent tests, 20
    What it says in wordsThe chance that every test stays quiet shrinks with each test added, so the chance of a false winner grows.

    How do you correct for it?

    Tighten the bar to match the number of tries. The Bonferroni correction tests each signal at 5% divided by 20, which is 0.25%, and that brings the chance of any false discovery back to 4.9%. The cost is power: a real but modest signal now struggles to get through. The other defence is data the search never touched: choose the best signal on one period, then test it once on another.

    What does a quant fund take from this?

    Research teams run thousands of tests, and the ones that get presented are the survivors. Count every test, including the ones you ran and forgot, because the significance of the survivor depends on how many were tried. That is why systematic funds keep research logs and hold data back, and why a backtest with a t-statistic of 2 means much less after a large search than after one planned test. Say the limitation: the 64% assumes independent tests; correlated signals give a lower figure, but rarely a comfortable one.

    Where candidates lose it

    The fast wrong answer adds the probabilities: 20 x 5% = 100%, a certainty. Adding only works for events that cannot happen together; here several false positives can appear at once, so go through the complement.

    The quieter error is answering 5%, treating the batch as one test. The interviewer wants you to see that the error rate of the search is not the error rate of each test inside it.

    What the interviewer asks next

    • How many tests at 5% before a false positive is more likely than not?
    • What significance level per test keeps the family-wide chance at 5% across 100 tests?
    • Why does out-of-sample testing help, and what can still go wrong with it?