Stacking & Blending

What is Model Ensembling?

No single machine learning algorithm is best for every problem (No Free Lunch Theorem).

Different algorithms develop different decision boundaries:

• Decision trees create axis aligned step boundaries.
• Support Vector Machines create smooth hyperplanes.
• Neural Networks create flexible non linear curves.

Ensembling combines predictions from multiple models to reduce error and build a more robust final system.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. VOTING / AVERAGING    │ 2. WEIGHTED AVERAGING    │ 3. STACKING (LEVEL 1)    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Takes simple mean or     │ Assigns higher weights   │ Trains a Meta-Model on   │
│ majority vote across all │ to models with better    │ predictions generated by │
│ base model predictions.  │ validation accuracy.     │ Level 0 base models.     │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Stacking (Stacked Generalization - Wolpert, 1992)

Stacking uses a two-level architecture:

  Input Features X ──┬──► [ Level 0 Model 1: XGBoost ]       ──► Prediction P1 ──┐
                     ├──► [ Level 0 Model 2: Neural Net ]    ──► Prediction P2 ──┼──► [ Level 1 Meta Learner: Ridge ] ──► Final Y
                     └──► [ Level 0 Model 3: Random Forest ] ──► Prediction P3 ──┘

1. Level 0 (Base Models): Diverse models trained directly on raw features.
2. Level 1 (Meta Learner): A simple model (like Ridge Regression or Logistic Regression) that takes predictions $[P_1, P_2, P_3]$ as inputs and predicts true target $y$.

Preventing Target Leakage via Out-of-Fold (OOF) Predictions

What happens if you feed Level 0 predictions on training data into the Meta Learner?

Base models overfit their training data, producing unnaturally perfect predictions. The Meta Learner learns to trust overfit outputs, leading to severe failure on test data (Target Leakage).

Fix: K-Fold Out-of-Fold (OOF) Prediction Pipeline:

  Data split into 5 Folds:
  - Fold 1: Train Base Models on Folds 2,3,4,5 ──► Predict on Fold 1 (OOF Prediction 1!)
  - Fold 2: Train Base Models on Folds 1,3,4,5 ──► Predict on Fold 2 (OOF Prediction 2!)
  ...
  Concat OOF Predictions [P1, P2, P3] ──► Train Level 1 Meta Learner cleanly!

Golden Rules of Ensembling

1. Prioritize Model Diversity: Combining XGBoost + LightGBM + CatBoost gives small gains because all three are tree boosting models. Combining XGBoost + Neural Network + Linear Model yields massive gains because their errors are uncorrelated.
2. Keep the Meta Model Simple: Use a simple regularized linear model (Logistic Regression or Lasso) for Level 1 to prevent meta-level overfitting.

Say this out loud

Ensembling combines predictions from diverse models to improve accuracy. Simple ensembling uses voting or weighted averaging. Stacking trains a Level 1 Meta Learner model on Out of Fold predictions generated by diverse Level 0 base models. Using K-fold Out of Fold predictions prevents target leakage when training the meta model.

Followups to expect

1. What is Blending? A simplified version of Stacking where instead of K-Fold OOF predictions, Level 0 models generate predictions on a single fixed holdout validation set to train the Level 1 meta learner.
2. Is Stacking suitable for low latency production systems? Stacking requires running forward passes through multiple models sequentially, increasing latency and maintenance overhead. It is used heavily in Kaggle competitions but less often in real time serving APIs.