Conjugate Priors

What is a Conjugate Prior?

Bayes' Theorem updates our belief about a parameter given data:

Posterior = (Likelihood * Prior) / Normalizing Constant

Calculating the normalizing constant often requires solving hard multidimensional integrals.

A prior is called conjugate to the likelihood if multiplying the prior by the likelihood produces a posterior in the exact same distribution family.

Common Conjugate Pairs

1. Beta - Binomial: Used for modeling conversion rates and coin flips. Prior: Beta(alpha, beta) Data: k successes, n - k failures Posterior: Beta(alpha + k, beta + n - k)

2. Dirichlet - Multinomial: Used for multi-class choices and topic modeling. Prior: Dirichlet(alpha_1, ..., alpha_K) Data: Counts (n_1, ..., n_K) Posterior: Dirichlet(alpha_1 + n_1, ..., alpha_K + n_K)

3. Normal - Normal: Used for measuring continuous physical quantities with Gaussian noise. Prior: Normal(mean_0, variance_0) Data: Sample mean x_bar from Normal noise Posterior: Normal(updated_mean, updated_variance)

Why This Matters in Practice

1. Exact Analytical Solution: You get an exact mathematical formula for the posterior rather than an approximation.
2. Instant Updates: Streaming systems can update beliefs in milliseconds by simply incrementing counts.
3. Thompson Sampling: Multi-Armed Bandit algorithms use Beta-Binomial conjugacy to sample success probabilities instantly for real-time recommendation engines.

Say this out loud

Conjugate priors make Bayesian statistics simple. When a prior and likelihood are conjugate, the posterior stays in the same family. Updating your beliefs after seeing new data is as simple as adding sample counts to your prior numbers. This allows real-time systems like Thompson Sampling to update recommendations instantly without running slow simulations.

Follow-ups to expect

• How do you choose prior parameters alpha and beta? You can base them on historical domain data, or set small equal values like Beta(1,1) for a uniform uninformative prior.
• What if your likelihood has no conjugate prior? You must use numerical approximation techniques like Markov Chain Monte Carlo (MCMC) or Variational Inference.