Maximum Likelihood Estimation

The Likelihood Function

Given i.i.d. observations $X = {x_1, x_2, \dots, x_N}$ from distribution $P(x | \theta)$:

$$L(\theta) = P(X | \theta) = \prod_{i=1}^N P(x_i | \theta)$$

$$\text{Log-Likelihood: } \ell(\theta) = \ln L(\theta) = \sum_{i=1}^N \ln P(x_i | \theta)$$

$$\hat{\theta}{\text{MLE}} = \arg\max\theta \ell(\theta)$$

To find $\hat{\theta}{\text{MLE}}$, solve score equation $\nabla\theta \ell(\theta) = 0$.

MLE Equivalence in ML Models

• Linear Regression (Gaussian Noise): $\arg\max_\theta \ell(\theta) \iff \arg\min_\theta \sum (y_i - w^T x_i)^2$ (MSE Loss).
• Logistic Regression (Bernoulli Noise): $\arg\max_\theta \ell(\theta) \iff \arg\min_\theta -\sum [y_i \log p_i + (1-y_i) \log(1-p_i)]$ (Binary Cross-Entropy).
• Multi-Class Networks (Multinomial Noise): $\arg\max_\theta \ell(\theta) \iff \arg\min_\theta -\sum y_{i,c} \log p_{i,c}$ (Categorical Cross-Entropy).

Asymptotic Properties of MLE

1. Consistency: As sample size $N \to \infty$, MLE $\hat{\theta}_{\text{MLE}} \to \theta^*$ (converges to true parameter).
2. Asymptotic Normality: As $N \to \infty$, $\hat{\theta} \sim \mathcal{N}\left(\theta^*, I(\theta)^{-1}\right)$, where $I(\theta)$ is the Fisher Information Matrix.
3. Efficiency: Achieves the Cramér-Rao lower bound for variance among unbiased estimators.

Say this out loud

"Maximum Likelihood Estimation finds parameter values that maximize the probability of observing the data. We maximize Log-Likelihood because log converts probability products into stable additions. Under Gaussian noise assumptions, maximizing MLE is mathematically identical to minimizing MSE loss in linear regression, while under Bernoulli assumptions it corresponds to Cross-Entropy."

Follow-ups to expect

• What is the difference between Likelihood and Probability? Probability $P(X|\theta)$ evaluates the chance of data $X$ for fixed parameter $\theta$. Likelihood $L(\theta|X)$ evaluates the plausibility of parameter $\theta$ for observed fixed data $X$.
• How does MLE handle small datasets? MLE overfits small datasets because it relies entirely on observed sample frequencies without incorporating prior domain knowledge (MAP/Bayesian estimation solves this).