Bayes’ Theorem

The formula and its components

Bayes' Theorem decomposes conditional probability into four distinct terms:

P(A | B) = [ P(B | A) · P(A) ] / P(B)

Posterior = [ Likelihood · Prior ] / Evidence

• Posterior P(A|B): Probability hypothesis A is true after observing evidence B.
• Prior P(A): Belief in hypothesis A before seeing evidence B (the base rate).
• Likelihood P(B|A): Probability of observing evidence B assuming hypothesis A is true.
• Evidence P(B): Marginal probability of seeing evidence B under all possible hypotheses: P(B) = P(B|A)P(A) + P(B|¬A)P(¬A).

Base rate fallacy: Why human intuition fails

When testing for rare events (fraud, rare disease, network intrusions), the prior base rate P(A) dominates the posterior calculation.

For 100,000 tested individuals:

• True Positives: 100 sick × 0.99 = 99
• False Positives: 99,900 healthy × 0.01 = 999

Posterior P(Disease | +) = 99 / (99 + 999) ≈ 9.0%. Even with 99% test accuracy, 91% of positive results are false alarms because the prior is so small.

Where Bayes appears in ML rounds

1. Naive Bayes Classifiers: Assumes feature independence given class: P(Y|X) ∝ P(Y) ∏ P(X_i|Y). Fast baseline for NLP and spam filtering.
2. MAP vs MLE: MAP adds a prior term to parameter estimation argmax_θ P(X|θ)P(θ). Adding a Gaussian prior over weights is mathematically equivalent to L2 regularization (Ridge).
3. Bayesian Optimization: Fits a Gaussian Process prior over an expensive objective function (hyperparameter search) to balance exploration and exploitation via acquisition functions (Expected Improvement, UCB).

Say this out loud

"Bayes’ Theorem updates prior beliefs with new evidence. In real-world systems, ignoring the base rate leads to massive overconfidence — a 99% accurate test on a 0.1% rare event still yields ~91% false positives. In ML, Bayes connects regularization to priors: MAP estimation with a Gaussian prior is equivalent to L2 regularization, while a Laplacian prior gives L1."

Follow-ups to expect

• Why is Naive Bayes called 'naive'? Because it assumes all features are conditionally independent given the class label, which rarely holds in practice but works remarkably well.
• How do you compute evidence P(B) in high dimensions? Direct integration is intractable, requiring approximations like Markov Chain Monte Carlo (MCMC) or Variational Inference.
• What happens when you have zero counts in Naive Bayes? Probability becomes zero (P(x|y)=0). Fix this using Laplace smoothing (additive smoothing).