We simulated 318,000 backtest searches in markets where nothing can be predicted, and counted how often each test said "edge". A t-test on the best rule: 10.9% to 78.9% of the time. Bonferroni: 8.0% to 10.3% when the market drifted, and 58.5% to 59.7% when stocks only differed in average return. A placebo test (shuffle the days, the same shuffle for every asset, and rerun the whole pipeline on 19 copies): 4.6% to 5.5%, against a promised 5%.

The placebo is now a tool in Canli Validation (placebo_test), next to a lookahead check that never sees your code (check_leakage). The study and its pre-registration: https://github.com/arhancanli/ca...

Has anyone here run a placebo on their own strategy or model pipeline? Curious what it found.

The question Canli Validation answers is simple to ask and easy to get wrong: if you tried many versions of a strategy and kept the best one, how much of its Sharpe ratio is the search rather than skill?

A concrete case. A strategy shows a Sharpe ratio of 1.50, and on its own that looks like a 98.1% chance the edge is real. But it was the best of 229 variants. Among 229 variants with no skill at all, the best one is expected to reach a Sharpe of about 1.60 by luck alone. Once you account for the search, the chance that the 1.50 is real drops from 98.1% to 44.4%: closer to a coin flip than a discovery.

That is the deflated Sharpe ratio (Bailey and L pez de Prado). Give the tool the Sharpe, the number of variants you tried and how much their Sharpes differed, and it returns this number plus the minimum track record you'd need. Give it every variant's returns and it also estimates the probability of backtest overfitting (CSCV).

What it doesn't do: it prices the search, not the backtest itself. Costs, fills, look-ahead and survivorship inside your backtest are still on you.