What is data mining bias and why does it matter for backtests?

Question

AcadiFi · Accepted Answer

Data mining (or data snooping) bias arises when a researcher tests many hypotheses on the same dataset and reports only the significant ones. With a 5% significance level, you expect ~5% of random tests to appear 'significant' by chance. Test 1,000 random factors and you'll find ~50 with p<0.05 — none of which have true predictive power. Example: a quant team at Stellar Alpha Capital tests 500 technical indicators on S&P 500 data from 2000-2020. They find 'Indicator #347' — a combination of 14-day RSI threshold and 21-day moving average slope — generates 4.5% annualized alpha. Publishing without correction, this is meaningless: with 500 tests and a 5% false-discovery rate, ~25 indicators will appear profitable by chance. Proper corrections include the Bonferroni adjustment (divide significance threshold by number of tests: 0.05/500 = 0.0001), the Benjamini-Hochberg false-discovery rate, or Harvey-Liu's t-stat haircut (they argue factors need t-stats >3.0, not 2.0, to survive multiple-testing correction). Out-of-sample testing on a held-out period is the gold standard. Harvey, Liu, and Zhu estimate over half of published 'factor premia' are data-mined.

What is data mining bias and why does it matter for backtests?

Master Level II with our CFA Course

Related Questions

Practice Questions