XGBoost vs LightGBM vs CatBoost

The Big Three Gradient Boosting Libraries

While classic Gradient Boosting was invented in 2001, modern production machine learning relies on three heavily optimized open source implementations:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. XGBOOST (2014)        │ 2. LIGHTGBM (2017)       │ 3. CATBOOST (2017)       │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Second-order Hessians.   │ Leaf-Wise tree growth.   │ Ordered Target Encoding. │
│ Exact & Approx Splits.   │ Histogram Binning.       │ Symmetric Balanced Trees.│
│ Gold standard stability. │ Ultra-fast, low RAM!     │ Best for Categorical data│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. XGBoost (Extreme Gradient Boosting - Chen & Guestrin, 2016)

XGBoost transformed Kaggle competitions by introducing:

1. Second Order Taylor Expansion: Uses both first order Gradients $g_i$ and second order Hessians $h_i$ to evaluate candidate splits:

$$\mathcal{L}^{(t)} \approx \sum_{i=1}^n \left[ g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) \right] + \Omega(f_t)$$

2. Built in Tree Regularization: Adds leaf count penalty $\gamma T$ and L2 weight penalty $\frac{1}{2} \lambda \sum w_j^2$ directly into the split scoring equation.
3. Sparsity-Aware Split Finding: Automatically assigns default split directions for missing data values.

2. LightGBM (Light Gradient Boosting Machine - Ke et al., 2017 / Microsoft)

LightGBM was designed for massive datasets where XGBoost ran too slowly:

  XGBOOST LEVEL-WISE GROWTH:                   LIGHTGBM LEAF-WISE GROWTH:
  Splits ALL nodes at depth d evenly           Splits ONLY the single leaf with max loss drop
     (O)                                          (O)
    /   \                                        /   \
  (O)   (O)                                    (O)   (O)
  / \   / \                                    / \
 (O)(O)(O)(O)                                (O) (O)  <-- Asymmetric Deep Branches!

Key innovations:

1. Histogram Binning: Discretizes continuous feature numbers into 256 integer bins, accelerating split evaluations by 10x while saving 80% RAM.
2. Gradient-Based One-Side Sampling (GOSS): Keeps all samples with large gradients (hard errors) while randomly sampling a small subset of samples with small gradients, maintaining accuracy with far fewer data rows.

3. CatBoost (Categorical Boosting - Prokhorenkova et al., 2018 / Yandex)

CatBoost specializes in tabular datasets dominated by Categorical Features (e.g. city names, product IDs, user categories):

1. Ordered Target Encoding: Replaces categorical text with target mean statistics calculated sequentially over random dataset permutations, completely eliminating Target Leakage.
2. Symmetric Oblivious Trees: Uses the exact same splitting feature and threshold across all nodes at the same tree depth, making execution sub-millisecond fast during inference.

Summary Comparison Matrix

Say this out loud

XGBoost introduced second order Hessian optimization and tree regularization. LightGBM introduced Leaf Wise tree growth, Histogram Binning, and GOSS to run 10 times faster with lower memory on massive datasets. CatBoost uses Ordered Target Encoding to process high cardinality categorical features natively without target leakage.

Followups to expect

1. Why does LightGBM overfit easily on small datasets (< 10,000 rows)? Leaf Wise tree growth can create deep asymmetric branches that overfit small data. Set max_depth or num_leaves to control depth.
2. When should you choose XGBoost over LightGBM today? When working with small to medium datasets where level wise depth growth provides stable, well regularized performance out of the box.