Bagging vs Boosting

The Two Ensemble Paradigms

Ensemble methods combine multiple machine learning models to build a single predictor that outperforms any individual base model.

The two main ways to build ensembles are Bagging and Boosting:

  BAGGING (Parallel Variance Reduction):
  Data ──► Bootstrap Sample 1 ──► [ Deep Tree 1 ] ──┐
  Data ──► Bootstrap Sample 2 ──► [ Deep Tree 2 ] ──┼──► AVERAGE PREDICTIONS (Parallel!)
  Data ──► Bootstrap Sample 3 ──► [ Deep Tree 3 ] ──┘

  BOOSTING (Sequential Bias Reduction):
  Data ──► [ Shallow Tree 1 ] ──► Compute Residual Errors ──► [ Shallow Tree 2 ] ──► Compute Residuals ...
                                                                                   (Sequential!)

Detailed Architectural Comparison

┌──────────────────────────┬──────────────────────────┐
│ BAGGING (Bootstrap Agg.) │ BOOSTING (Gradient/Ada)  │
├──────────────────────────┼──────────────────────────┤
│ Base Models: Deep Trees  │ Base Models: Shallow     │
│ (High Variance, Low Bias)│ Stumps (High Bias, Low V)│
│ Built: PARALLEL.         │ Built: SEQUENTIAL.       │
│ Main Goal: Reduce Var.   │ Main Goal: Reduce Bias.  │
│ Overfitting Risk: Low.   │ Overfitting Risk: High.  │
└──────────────────────────┴──────────────────────────┘

1. Bagging (Bootstrap Aggregating)

• Concept: Train $B$ independent complex models on random bootstrap samples drawn with replacement.
• Aggregation: Average outputs (Regression) or take Majority Vote (Classification).
• Why it works: If $B$ models have variance $\sigma^2$ and correlation $\rho$:

$$\text{Ensemble Variance} = \rho \sigma^2 + \frac{1 - \rho}{B} \sigma^2$$

As $B \to \infty$, variance shrinks down to $\rho \sigma^2$. Decorrelating models (lowering $\rho$) via Random Forest feature subspacing cuts total variance dramatically.

2. Boosting (Sequential Error Correction)

• Concept: Train $B$ simple weak models sequentially. Model $k$ is trained specifically to fix errors made by Model $1 \dots k-1$.
• Aggregation: Weighted sum of all weak model outputs:

$$F(x) = \sum_{k=1}^B \gamma_k h_k(x)$$

• Why it works: Combines high bias weak learners (decision stumps with 1 split) into a single strong model with low bias and low variance.

Summary Matrix

Say this out loud

Bagging trains complex deep trees in parallel on random bootstrap samples to reduce model variance, making Random Forest fast and robust against overfitting. Boosting trains simple shallow trees sequentially, focusing each new tree on residual errors from previous steps to reduce model bias, producing state of the art tabular accuracy in XGBoost.

Followups to expect

1. What is Stacking (Stacked Generalization)? An ensemble method where diverse base models (e.g. SVM, Random Forest, Neural Net) train on raw features, and a meta model (e.g. Logistic Regression) trains on the predictions of base models.
2. Why does Boosting overfit if given too many trees? Because Boosting sequentially fits errors, adding thousands of trees will eventually force the ensemble to fit noise in individual training samples.