How does LASSO do automatic variable selection and what's the intuition?

Question

AcadiFi · Accepted Answer

LASSO (Least Absolute Shrinkage and Selection Operator) minimizes the sum of squared residuals PLUS a penalty proportional to the sum of absolute values of coefficients (L1 norm). Ridge uses the sum of squared coefficients (L2 norm). The difference in penalty shape — diamond vs. circle — is why LASSO produces exact zeros.

Consider Penmarket Advisors fitting a model with 50 predictors. LASSO minimizes RSS + λ × Σ|βⱼ|. The constraint region 'Σ|βⱼ| ≤ t' is a diamond in coefficient space with corners along the axes. The minimization is equivalent to finding the lowest RSS contour that touches this diamond. Because the diamond has pointed corners ON the axes, the touch point often lands on a corner — meaning at least one coefficient equals exactly zero.

Ridge's constraint region is a sphere, which has no corners. The touch point almost never lies exactly on an axis, so coefficients shrink toward zero but don't hit it.

As λ increases from 0:
- Many coefficients are non-zero (light selection)
- More coefficients get driven to zero
- Eventually all coefficients zero out (null model)

You choose λ via cross-validation: pick the value that minimizes out-of-sample prediction error (or, for a sparser model, the 'one-standard-error' rule that picks a slightly larger λ).

LASSO is exceptional when you suspect many predictors are irrelevant and you want an interpretable sparse model. Weakness: among highly correlated predictors, LASSO arbitrarily picks one. Elastic net (next question) addresses this by blending L1 and L2 penalties.

How does LASSO do automatic variable selection and what's the intuition?

Master Level II with our CFA Course

Related Questions

Practice Questions