Elastic Net

What is Elastic Net?

L1 Lasso and L2 Ridge regularization each have unique strengths and weaknesses:

1. L1 Lasso: Performs sparse feature selection by zeroing out irrelevant features, but selects at random among strongly correlated features.
2. L2 Ridge: Keeps all features and handles correlated groups smoothly, but cannot zero out uninformative features.

Elastic Net (Zou & Hastie, 2005) combines both penalties into a single loss function to get the best of both worlds:

$$\text{Loss}{\text{ElasticNet}} = \text{MSE} + \lambda_1 \sum{j=1}^p |w_j| + \lambda_2 \sum_{j=1}^p w_j^2$$

In scikit-learn, this is parameterized using total penalty strength $\alpha$ (alpha) and ratio $l1_ratio$:

$$\text{Penalty} = \alpha \cdot l1_ratio \cdot \sum |w_j| + \frac{1}{2} \alpha \cdot (1 - l1_ratio) \cdot \sum w_j^2$$

• $l1_ratio = 1.0 \implies$ Pure L1 Lasso Regression.
• $l1_ratio = 0.0 \implies$ Pure L2 Ridge Regression.
• $0 < l1_ratio < 1 \implies$ Elastic Net Combination.

The Grouping Effect

Consider a genomics dataset where 10 different genes are strongly correlated because they belong to the same biological pathway.

  L1 LASSO:        Picks 1 gene at random, sets the other 9 to zero. (Instable across random seeds!)
  L2 RIDGE:        Keeps all 10 genes, but also keeps 10,000 non useful genes.
  ELASTIC NET:     Selects all 10 correlated genes as a GROUP, while zeroing out non useful genes!

Elastic Net creates a convex constraint shape (a rounded diamond) that enables Grouped Feature Selection.

When to Use Elastic Net

1. $p \gg n$ Datasets: When you have far more features $p$ than data samples $n$ (like 50,000 gene expressions on 200 patients). Standard L1 Lasso can select at most $n$ features. Elastic Net can select more than $n$ features.
2. Collinear Data: When features exhibit strong pairwise correlations (like sensor arrays or financial metrics).

Say this out loud

Elastic Net combines L1 Lasso and L2 Ridge penalties into a single objective. While pure L1 Lasso selects one feature at random from a correlated group, Elastic Net uses L2 regularization to select correlated feature groups together while using L1 regularization to zero out uninformative features.

Followups to expect

1. How do you tune l1_ratio in scikit-learn? Use ElasticNetCV, performing grid search cross validation over l1_ratio values like [0.1, 0.5, 0.7, 0.9, 0.99].
2. Is Elastic Net computationally slower than Lasso? Elastic Net adds a small matrix operation overhead, but uses coordinate descent optimization algorithms that converge quickly on sparse high dimensional data.