How often do backtest significance tests lie? We measured it

We simulated 318,000 backtest searches in markets where nothing can be predicted, and counted how often each test said "edge". A t-test on the best rule: 10.9% to 78.9% of the time. Bonferroni: 8.0% to 10.3% when the market drifted, and 58.5% to 59.7% when stocks only differed in average return. A placebo test (shuffle the days, the same shuffle for every asset, and rerun the whole pipeline on 19 copies): 4.6% to 5.5%, against a promised 5%.

The placebo is now a tool in Canli Validation (placebo_test), next to a lookahead check that never sees your code (check_leakage). The study and its pre-registration: https://github.com/arhancanli/canlicapital/tree/main/scripts/research/placebo/v1

Has anyone here run a placebo on their own strategy or model pipeline? Curious what it found.