What is the proper methodology for backtesting an equity strategy, and how can an analyst distinguish genuine alpha from data-mined results?

Question

AcadiFi · Accepted Answer

Backtesting simulates a strategy's historical performance using past market data. Done properly, it provides evidence of a factor's persistence; done poorly, it becomes an exercise in overfitting. Here is the disciplined framework.

**Proper Backtesting Methodology**

1. **Define the hypothesis first** — State the economic rationale before touching data. Example: "Stocks with improving earnings momentum and below-median EV/EBITDA outperform because the market underreacts to positive earnings revisions in undervalued names."

2. **Use point-in-time data** — Every data point must reflect what was actually available to investors on that date. Restated financials, adjusted figures, and delayed filings create look-ahead bias.

3. **Include dead stocks** — Use a survivorship-bias-free database. Stocks that delisted, merged, or went bankrupt must appear in the universe during the periods they traded.

4. **Realistic execution assumptions:**
   - Trade at next-day VWAP, not same-day close.
   - Include commissions, bid-ask spreads, and market impact.
   - Cap position sizes by average daily volume (e.g., no more than 10% of ADV).

5. **Out-of-sample testing** — Split data into in-sample (for calibration) and out-of-sample (for validation). A common split: 70% in-sample / 30% out-of-sample.

6. **Paper trading bridge** — After passing out-of-sample tests, run the strategy in paper (simulated live) trading for 3-6 months before committing capital.

**Distinguishing Alpha from Data Mining**

| Test | What It Checks | Threshold |
|---|---|---|
| t-statistic on alpha | Statistical significance | t > 3.0 (Harvey et al.) |
| Out-of-sample Sharpe | Performance persistence | > 50% of in-sample Sharpe |
| Haircut for multiple testing | Adjust for number of strategies tested | Use Bonferroni or BHY correction |
| Decay analysis | Does alpha degrade over time? | Stable or slow decay preferred |
| Economic rationale | Is there a behavioral or structural explanation? | Must exist independently of data |

**Worked Example — Kellworth Asset Management Backtest:**

Kellworth tests a 12-1 month momentum + low EV/EBITDA composite across US large caps from 2005-2023:

- In-sample (2005-2017): Annualized alpha +4.2%, t-stat = 3.8, Sharpe 0.85
- Out-of-sample (2018-2023): Annualized alpha +2.1%, t-stat = 2.1, Sharpe 0.52
- Out-of-sample Sharpe is 61% of in-sample — above the 50% threshold
- Harvey-adjusted critical t-value for 50 tested strategies: 3.4 — in-sample passes, out-of-sample does not alone but combined evidence is supportive

**Red Flags of a Data-Mined Strategy:**

1. No economic rationale — "it just works in the data."
2. Alpha disappears completely out-of-sample.
3. Extreme sensitivity to parameter choices (e.g., works with 11-month momentum but fails with 13-month).
4. Tested hundreds of factor combinations with no multiple-testing adjustment.

Our CFA Level III course covers backtesting rigor with hands-on case studies.

What is the proper methodology for backtesting an equity strategy, and how can an analyst distinguish genuine alpha from data-mined results?

Master Level III with our CFA Course

Related Questions

Practice Questions