How does best-subsets regression work and when is it feasible?

Question

AcadiFi · Accepted Answer

Best-subsets regression evaluates every possible combination of predictors and picks the best one by a chosen criterion (adjusted R², AIC, BIC, or Mallows' Cp). It's exhaustive rather than greedy, so it never misses a combination that stepwise might skip. With k candidate predictors, best-subsets fits 2^k models. Corbridge Analytics with 10 candidate factors fits 1,024 models. With 20 factors, that's over a million. With 40, about 10^12 — computationally infeasible. The hard ceiling is roughly k ≈ 25-30 for ordinary datasets on typical hardware. The algorithm (branch-and-bound, implemented in R's `leaps` or Python's `abess`) prunes dominated branches to accelerate. Output typically shows the best 1-variable model, best 2-variable model, etc., up to the full model. You then compare across model sizes using an information criterion. ```mermaid flowchart TD A[k candidate
variables] --> B[Fit all 2^k
combinations] B --> C[Best 1-var] B --> D[Best 2-var] B --> E[Best k-var] C --> F[Compare via
AIC or BIC] D --> F E --> F F --> G[Pick optimal] ``` Advantages over stepwise: you see multiple competitive models (not just the first greedy path) and can pick one that balances fit, parsimony, and interpretability. Disadvantages: still inflates Type I error from running many tests, still unstable across resamples, and still vulnerable to post-selection inference problems. The fitted coefficients are biased; standard errors are too small. For modern high-dimensional problems (k > 30), use LASSO with cross-validation instead. Best-subsets is a middle-ground tool for moderate problems where you want rigor without the computational cost of fully Bayesian alternatives.

How does best-subsets regression work and when is it feasible?

Master Level II with our CFA Course

Related Questions

Practice Questions