L1 vs L2 Regularization

What is Regularization?

When a machine learning model has too much capacity, it memorizes noise in the training data rather than learning real patterns (Overfitting).

Overfit models develop excessively large weight values to fit individual training data points.

Regularization adds a penalty term to the loss function that punishes large weights:

$$\text{Total Loss} = \text{Data Loss} + \lambda \cdot \text{Weight Penalty}$$

Hyperparameter $\lambda$ (lambda) controls penalty strength:

• $\lambda = 0$: Unregularized model (risk of severe overfitting).
• Large $\lambda$: High penalty, forcing weights to remain small (risk of underfitting).

┌──────────────────────────┬──────────────────────────┐
│ 1. L1 LASSO REGULARIZATION│ 2. L2 RIDGE REGULARIZATION│
├──────────────────────────┼──────────────────────────┤
│ Penalty = λ * sum( |w| ) │ Penalty = λ * sum( w^2 ) │
│ Drives uninformative     │ Shrinks all weights      │
│ weights to EXACT ZERO.   │ smoothly toward zero.    │
│ Sparse Feature Selection!│ Handles correlated inputs│
└──────────────────────────┴──────────────────────────┘

1. L1 Regularization (Lasso Regression)

$$\text{Loss}{\text{L1}} = \text{MSE} + \lambda \sum{i=1}^d |w_i|$$

Derivative for $w > 0$ is $+\lambda$, and for $w < 0$ is $-\lambda$.

Because the derivative is a constant step size regardless of how small weight $w$ becomes, gradient descent pushes small weights all the way to exact zero.

Feature Selection Superpower

If you have 10,000 input features but only 50 are useful, L1 Lasso sets the remaining 9,950 feature weights to exact zero, returning a sparse, easily interpretable model.

2. L2 Regularization (Ridge Regression)

$$\text{Loss}{\text{L2}} = \text{MSE} + \lambda \sum{i=1}^d w_i^2$$

Derivative is $2 \lambda w_i$.

Weight update step:

$$w_{\text{new}} = w_{\text{old}} - \eta \left( \text{Gradient} + 2\lambda w_{\text{old}} \right) = (1 - 2\eta\lambda) w_{\text{old}} - \eta \text{Gradient}$$

Notice the term $(1 - 2\eta\lambda) w_{\text{old}}$!

At every step, weight $w$ is multiplied by a fraction smaller than 1.0 (Weight Decay).

Large weights shrink rapidly, but as $w$ approaches zero, its penalty derivative shrinks to zero, so weights never become exact zero.

Geometric Intuition

  L1 CONSTRAINTS (Diamond Corners):            L2 CONSTRAINTS (Smooth Circular Sphere):
  Loss contours touch diamond corners on axes  Loss contours touch circular sphere boundary
  ──► Weights land on exact 0 axes!            ──► Weights shrink smoothly near 0!

Say this out loud

Regularization prevents overfitting by penalizing large parameter weights. L1 Lasso adds the absolute sum of weights, driving uninformative feature weights to exact zero for automatic feature selection. L2 Ridge adds the squared sum of weights, shrinking weights smoothly toward zero via weight decay without zeroing them out completely.

Followups to expect

1. How does L2 handle multi-collinearity (highly correlated features)? L2 splits weight magnitude evenly among correlated features. L1 randomly picks one feature from a correlated group and sets the rest to zero.
2. What is Elastic Net? Combining both L1 and L2 penalties into a single loss function to enjoy both sparse feature selection and correlated feature stability.