How does SMOTE create synthetic minority samples and what are its pitfalls?

Question

AcadiFi · Accepted Answer

SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic positive samples by interpolating between existing positives and their k-nearest neighbors in feature space.

Algorithm:
1. For each minority sample x_i, find k nearest minority neighbors (typically k=5).
2. Randomly pick one neighbor x_j.
3. Create synthetic sample: x_syn = x_i + λ(x_j - x_i), where λ ~ Uniform(0,1).
4. Repeat until desired balance is reached.

Example: Orinthorpe Investment Bank's default detection model has 180 defaults among 22,000 loans (0.82%). SMOTE to 20% positive generates 4,220 synthetic defaults by interpolating feature vectors.

Pitfalls:

1. Noise amplification: SMOTE interpolates with all minority samples including outliers, potentially creating synthetic points in wrong regions. Use Borderline-SMOTE or ADASYN which focus on the decision boundary.

2. Categorical features: Euclidean interpolation is meaningless for one-hot encoded categories. Use SMOTE-NC (Nominal-Continuous) which handles categoricals by majority vote.

3. Leakage into validation: CRITICAL — apply SMOTE only on training folds. If you SMOTE before splitting, synthetic samples from training leak to test, inflating validation metrics by 5-15 points of AUC.

4. Production drift: synthetic points don't exist in real data. Model learns boundaries along synthetic lines that may not hold when production distribution shifts. Orinthorpe saw 6-month AUC decay from 0.84 → 0.71 vs. only 0.82 → 0.79 for class-weighted model without SMOTE.

5. High-dimensional curse: in >50 features, nearest neighbors become unreliable. SMOTE helps less and may hurt.

Orinthorpe's final approach: XGBoost with scale_pos_weight=120 instead of SMOTE — stabler in production and avoids interpolation artifacts.

How does SMOTE create synthetic minority samples and what are its pitfalls?

Master Level II with our CFA Course

Related Questions

Practice Questions