What is stacking and how does it differ from other ensemble methods?

Question

AcadiFi · Accepted Answer

Stacking (stacked generalization) is an ensemble method that combines predictions from multiple base learners using another model (the meta-learner or blender) that learns how to optimally weight and combine them. Unlike bagging (averages same-type models) or boosting (sequentially corrects errors), stacking leverages heterogeneous models that may capture different aspects of the data. Algorithm: (1) Split training data into K folds. (2) Train each base learner on K−1 folds, predict on the held-out fold. Repeat for all folds to generate out-of-fold predictions for the full training set. (3) Assemble these out-of-fold predictions as new features — a matrix where each column is one base learner's predictions. (4) Train the meta-learner (often a simple model like logistic regression or ridge) on these predictions to learn the optimal combination. (5) For final prediction, each base learner (retrained on full data) predicts, and the meta-learner combines these predictions. ```mermaid flowchart TB D[Training
Data] --> B1[Random
Forest] D --> B2[XGBoost] D --> B3[Neural
Net] B1 --> ML[Meta
Learner] B2 --> ML B3 --> ML ML --> F[Final
Prediction] ``` Example: Nordvale Quant combines three heterogeneous base learners to predict monthly stock returns — a random forest capturing non-linear interactions, an XGBoost for refined tree ensemble, and a neural network capturing complex patterns. Each is trained with 5-fold cross-validation, generating out-of-fold predictions. A logistic regression meta-learner weighs these predictions, discovering that XGBoost gets 60% weight, random forest 25%, neural network 15%. The stacked model outperforms any individual base learner on validation data. Key principles: (1) base learners should be diverse — similar models gain little from stacking; (2) use out-of-fold predictions to avoid meta-learner overfitting to training predictions; (3) keep the meta-learner simple to prevent over-fitting; (4) more base learners help up to a point, then diminishing returns. Stacking routinely wins ML competitions (Kaggle) and is used in production for risk models, recommendation systems, and alpha generation. The CFA quant curriculum treats stacking as the most sophisticated ensemble technique, contrasting it with the simpler averaging in bagging and sequential error-correction in boosting.

What is stacking and how does it differ from other ensemble methods?

Master Level II with our CFA Course

Related Questions

Practice Questions