Assumptions of Linear Regression

The OLS Regression Assumptions

Ordinary Least Squares (OLS) Linear Regression is the workhorse of statistical modeling:

$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$$

However, the Gauss-Markov Theorem proves OLS is the Best Linear Unbiased Estimator (BLUE) ONLY when specific mathematical assumptions hold.

Violating these assumptions invalidates p-values, standard errors, and confidence intervals.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. LINEARITY             │ 2. INDEPENDENCE          │ 3. HOMOSCEDASTICITY      │ 4. NORMALITY OF ERRORS   │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Relationship between X   │ Errors ε_i are un-       │ Variance of errors ε_i   │ Residual errors ε_i follow│
│ and Y is linear.         │ correlated across rows.  │ is constant across X.    │ Normal Distribution N(0, σ2)│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Detailed Breakdown & Diagnostic Checks

1. Linearity

The expected value of $Y$ is a linear combination of parameters $\beta$.

• Diagnostic: Scatter plot of $Y$ vs $X$, or Residuals vs Fitted Values plot.
• Fix: Add non-linear features ($X^2, \log(X)$) or use polynomial regression.

2. Independence of Errors (No Autocorrelation)

Residual errors $\epsilon_i$ and $\epsilon_j$ must be independent.

• Violation: Common in Time Series data where today's error correlates with yesterday's error.
• Diagnostic: Durbin-Watson Test (values near $2.0$ indicate independence; $<1.5$ indicates positive autocorrelation).
• Fix: Use Time Series models (ARIMA) or include lagged target features.

3. Homoscedasticity (Constant Error Variance)

The variance of residual errors $\text{Var}(\epsilon_i \mid X) = \sigma^2$ must remain constant across all feature levels.

  HOMOSCEDASTICITY (GOOD):                 HETEROSCEDASTICITY (BAD - FUNNEL SHAPE!):
  Residuals                                Residuals
    │  •   •  •  •  •  •                     │                 •   •   •
  0 ┼───────────────────────               0 ┼───────── •  •  •  •   •   •   •
    │  •   •  •  •  •  •                     │                 •   •   •
    └───────────────────────► Fitted Values  └───────────────────────► Fitted Values

• Diagnostic: Breusch-Pagan Test or Residuals vs Fitted funnel inspection.
• Fix: Apply $\log(Y)$ transformation or use Huber-White Robust Standard Errors.

4. Normality of Residual Errors

Residual errors $\epsilon \sim \mathcal{N}(0, \sigma^2)$ must follow a Normal Distribution.

• Diagnostic: Q-Q Plot (Quantile-Quantile plot) or Shapiro-Wilk Test.
• Fix: Log transform skewed targets or increase sample size (Central Limit Theorem protects sample means for large $N$).

5. No Multicollinearity

Predictor features $X_i$ must not be perfectly linearly correlated with each other.

• Diagnostic: Variance Inflation Factor (VIF < 5.0).
• Fix: Drop redundant features or apply Ridge Regularization ($L2$).

Say this out loud

Ordinary Least Squares regression relies on Linearity, Independence of errors, Homoscedasticity, and Normality of residuals. Linearity requires straight relationships. Independence requires uncorrelated errors checked via Durbin Watson. Homoscedasticity requires constant residual variance, while Normality ensures valid p values and confidence intervals.

Followups to expect

1. What happens to OLS coefficients if the Normality assumption is violated for large N? According to the Central Limit Theorem, for large sample sizes ($N > 100$), non normal residuals do not invalidate coefficient point estimates or hypothesis tests.
2. Does Homoscedasticity violation cause coefficient bias? No! Heteroscedasticity does not bias point estimates $\hat{\beta}$, but it corrupts standard error calculations, producing wrong p-values and confidence intervals.