What is boosting and how does gradient boosting differ from AdaBoost?

Question

AcadiFi · Accepted Answer

Boosting is an ensemble technique that builds a strong predictor by sequentially combining many weak learners (typically shallow decision trees), where each subsequent learner focuses on examples the previous ones got wrong. Unlike bagging/random forests that average independent models, boosting is inherently sequential and adaptive.

AdaBoost (Adaptive Boosting, 1996):

(1) Initialize equal weights for all training examples.

(2) Fit a weak classifier (e.g., decision stump).

(3) Increase weights on misclassified examples, decrease on correctly classified.

(4) Fit next weak classifier on reweighted data.

(5) Final prediction is weighted vote of all classifiers, with better-performing classifiers getting higher votes.

AdaBoost works well for binary classification but is sensitive to outliers (which get increasingly weighted).

Gradient Boosting (1999, Friedman):

(1) Start with initial prediction (e.g., mean of Y for regression).

(2) Compute residuals (errors from current prediction).

(3) Fit a weak learner to the residuals (predict the errors).

(4) Update prediction: f_new(X) = f_old(X) + η × f_residual(X), where η is learning rate.

(5) Repeat until convergence or maximum iterations.

The key insight: gradient boosting generalizes AdaBoost by treating the process as gradient descent in function space, with the loss function's negative gradient guiding what the next weak learner should predict. This allows arbitrary differentiable loss functions — squared error for regression, log-loss for classification, Huber loss for robustness, quantile loss for predicting quantiles.

Modern variants (XGBoost, LightGBM, CatBoost) add: (1) second-order gradients (Newton-like steps); (2) regularization penalties in the objective; (3) efficient implementations with parallel processing, histogram-based splits, handling categorical features natively; (4) column sampling similar to random forests.

Hyperparameters: number of trees (n_estimators), tree depth (controls interaction complexity), learning rate η (smaller = more trees needed but better generalization), subsample ratios, L1/L2 regularization. Cross-validation tunes these.

Gradient boosted trees dominate tabular ML competitions and are extensively used in credit scoring, fraud detection, and alpha generation. CFA Level II covers boosting fundamentals.

What is boosting and how does gradient boosting differ from AdaBoost?

Master Level II with our CFA Course

Related Questions

Practice Questions