Multicollinearity & VIF

What is Multicollinearity?

Multicollinearity occurs when two or more input features in a linear regression model are highly correlated with one another.

Example

Predicting Home Price ($Y$) using two features:

• $X_1$: Square Footage in Feet.
• $X_2$: Square Footage in Meters.

Because $X_1$ and $X_2$ carry identical information ($X_1 = 10.76 \cdot X_2$), the model cannot determine whether $X_1$ or $X_2$ is driving home prices!

  Matrix Inversion in OLS:  beta = (X^T X)^-1 X^T Y
  When features are collinear ──► X^T X is NEAR SINGULAR (Determinant ~ 0!)
  ──► Inverting (X^T X) produces HUGE INFLATED VARIANCE on coefficients!

Symptoms of Multicollinearity

1. Inflated Standard Errors: Regression coefficients $\hat{\beta}_j$ have huge standard errors, producing wide confidence intervals and non-significant p-values ($p > 0.05$) despite a high overall $R^2$ score!
2. Unstable Coefficient Weights: Adding or removing a few data rows causes coefficient values to swing wildly or flip signs (e.g. positive correlation feature getting a negative $\beta$).
3. Loss of Interpretability: You cannot isolate the individual effect of feature $X_1$ holding $X_2$ constant.

Note: Multicollinearity does NOT reduce the overall predictive accuracy or $R^2$ of the model on the training data! It destroys feature interpretability and statistical inference.

Diagnosing Multicollinearity: Variance Inflation Factor (VIF)

The Variance Inflation Factor (VIF) measures how much the variance of coefficient $\hat{\beta}_j$ is inflated due to collinearity with other features:

$$\text{VIF}_j = \frac{1}{1 - R_j^2}$$

• $R_j^2$: $R^2$ score obtained by regressing feature $X_j$ against all other remaining predictor features.

┌──────────────────────────┬──────────────────────────┐
│ VIF VALUE                │ INTERPRETATION           │
├──────────────────────────┼──────────────────────────┤
│ VIF = 1.0                │ Zero Multicollinearity.  │
│ 1.0 < VIF < 5.0          │ Moderate Correlation.    │
│ VIF > 5.0 or 10.0        │ SEVERE MULTICOLLINEARITY!│
└──────────────────────────┴──────────────────────────┘

How to Fix Multicollinearity

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. DROP FEATURES         │ 2. L2 RIDGE REGRESSION   │ 3. PCA REDUCTION         │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Remove one of the highly │ Adds λI to X^T X matrix, │ Transforms correlated    │
│ correlated feature pairs │ making matrix inversion  │ features into orthogonal │
│ (e.g. drop SqMeters).    │ numerically stable.      │ principal components.    │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Say this out loud

Multicollinearity occurs when predictor features are highly correlated. It inflates coefficient standard errors and flips parameter signs without reducing overall R squared. Diagnosed using Variance Inflation Factor (VIF > 5), multicollinearity is resolved by dropping redundant features, performing PCA, or applying L2 Ridge Regularization.

Followups to expect

1. Does Multicollinearity impact Decision Trees or Random Forests? No! Tree based algorithms select single features sequentially at orthogonal split nodes, making them immune to multicollinearity during prediction.
2. What is Structural Multicollinearity vs Data Multicollinearity? Structural multicollinearity is created by adding polynomial features ($X$ and $X^2$). Fix by centering variables ($X - \bar{X}$) before squaring.