MAP vs MLE
Frequentist likelihood maximization vs Bayesian posterior maximization with prior regularization.
The Derivation Bridge
Applying Bayes' Theorem to parameter estimation:
$$P(\theta | X) = \frac{P(X | \theta) P(\theta)}{P(X)}$$
$$\hat{\theta}{\text{MAP}} = \arg\max\theta \ln P(\theta | X) = \arg\max_\theta \left[ \ln P(X | \theta) + \ln P(\theta) \right]$$
MLE Objective: arg max_θ [ Log-Likelihood: ln P(X | θ) ]
MAP Objective: arg max_θ [ Log-Likelihood: ln P(X | θ) + Log-Prior: ln P(θ) ]
│
Acts as Regularizer!
How MAP Derives L1 and L2 Regularization
Given Linear Regression $y = X w + \epsilon$ with Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$:
1. Gaussian Prior $\implies$ L2 Regularization (Ridge)
Assume prior $w_j \sim \mathcal{N}(0, \tau^2)$:
$$\ln P(w) = -\frac{1}{2\tau^2} |w|_2^2 + \text{const}$$
$$\hat{w}{\text{MAP}} = \arg\min_w \left[ \sum{i=1}^N (y_i - w^T x_i)^2 + \lambda |w|_2^2 \right] \quad \left( \lambda = \frac{\sigma^2}{\tau^2} \right)$$
2. Laplace Prior $\implies$ L1 Regularization (Lasso)
Assume prior $P(w_j) = \frac{1}{2b} \exp\left(-\frac{|w_j|}{b}\right)$:
$$\ln P(w) = -\frac{1}{b} |w|_1 + \text{const}$$
$$\hat{w}{\text{MAP}} = \arg\min_w \left[ \sum{i=1}^N (y_i - w^T x_i)^2 + \lambda |w|_1 \right] \quad \left( \lambda = \frac{\sigma^2}{b} \right)$$
Comparison Matrix
| Property | MLE (Maximum Likelihood) | MAP (Maximum A Posteriori) |
|---|---|---|
| Formulation | $\arg\max_\theta P(X | \theta)$ | $\arg\max_\theta P(X | \theta) P(\theta)$ |
| Philosophical View | Frequentist (Parameters are fixed unknown constants) | Bayesian (Parameters have probability distributions) |
| Priors | No prior knowledge used | Incorporates explicit prior $P(\theta)$ |
| Overfitting Risk | High on small datasets | Low (Prior acts as explicit regularizer) |
| Large Data Limit ($N \to \infty$) | Identical | Converges to MLE (Data dominates prior) |
Say this out loud
"MLE maximizes data likelihood P(X|θ) without priors, making it prone to overfitting on small datasets. MAP incorporates a prior P(θ), maximizing log P(X|θ) + log P(θ). MAP bridges statistics and ML regularization: a Gaussian prior over weights derives L2 regularization (Ridge), while a Laplacian prior derives L1 regularization (Lasso). As sample size N grows, MAP converges to MLE."
Follow-ups to expect
- What is Full Bayesian Inference vs MAP? MAP outputs a single point estimate $\hat{\theta}_{\text{MAP}}$ (the mode of the posterior). Full Bayesian inference computes the full posterior distribution $P(\theta|X)$, integrating over all parameters for predictions: $P(y^|x^, X) = \int P(y^|x^, \theta) P(\theta|X) d\theta$.
- What is Conjugate Prior? A prior $P(\theta)$ is conjugate to likelihood $P(X|\theta)$ if the resulting posterior $P(\theta|X)$ belongs to the same probability distribution family as the prior (e.g. Beta prior + Binomial likelihood = Beta posterior).
Check yourself
What happens to MAP estimation when the prior P(θ) is chosen to be a uniform flat distribution (uninformative prior)?