Distributions You Must Know
The probability distributions every AI/ML engineer is expected to recognize, parameterize, and apply.
Summary Matrix of Essential Distributions
Probability Distributions
┌──────────────────────────┴──────────────────────────┐
▼ ▼
Discrete Distributions Continuous Distributions
- Bernoulli(p): Single 0/1 flip - Gaussian(μ, σ²): Bell curve
- Binomial(N, p): k successes in N - Exponential(λ): Wait times
- Poisson(λ): Event counts per interval - Beta(α, β): Prior over [0,1]
- Categorical(p_1..p_K): K-class pick - Dirichlet(α_1..α_K): Prior over Simplex
| Distribution | Support | Key Parameters | Mean $\mathbb{E}[X]$ | Variance $\text{Var}(X)$ | Primary ML Application |
|---|---|---|---|---|---|
| Bernoulli | ${0, 1}$ | $p \in [0, 1]$ | $p$ | $p(1-p)$ | Binary Classification, Dropout |
| Binomial | ${0, \dots, N}$ | $N \in \mathbb{N}, p \in [0,1]$ | $Np$ | $Np(1-p)$ | A/B Testing conversion counts |
| Poisson | ${0, 1, 2, \dots}$ | $\lambda > 0$ (rate) | $\lambda$ | $\lambda$ | Call center arrivals, web traffic |
| Categorical | ${1, \dots, K}$ | $p_1, \dots, p_K$ | $\sum i p_i$ | Matrix form | Multi-class Softmax classification |
| Gaussian (Normal) | $(-\infty, +\infty)$ | $\mu, \sigma^2 > 0$ | $\mu$ | $\sigma^2$ | MSE Loss, VAE latent space, Noise |
| Exponential | $[0, +\infty)$ | $\lambda > 0$ | $1/\lambda$ | $1/\lambda^2$ | Time-to-failure, Survival analysis |
| Beta | $[0, 1]$ | $\alpha, \beta > 0$ | $\frac{\alpha}{\alpha + \beta}$ | Complex | Bayesian Prior over probabilities |
| Dirichlet | Probability Simplex | $\alpha_1, \dots, \alpha_K$ | $\frac{\alpha_i}{\sum \alpha_k}$ | Complex | Topic Modeling (LDA), Bayesian GMM |
Deep Dives on Critical Distributions
1. Gaussian / Normal Distribution $\mathcal{N}(\mu, \sigma^2)$
Probability Density Function (PDF):
$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(x - \mu)^2}{2\sigma^2} \right)$$
Log-likelihood maximization yields Mean Squared Error (MSE).
2. Beta Distribution $\text{Beta}(\alpha, \beta)$
Supported on $[0, 1]$. Shape depends on pseudo-counts $\alpha$ (successes) and $\beta$ (failures):
- $\alpha = 1, \beta = 1 \implies \text{Uniform}(0, 1)$.
- $\alpha = 10, \beta = 2 \implies$ Skewed toward $1.0$ (High confidence success).
Used in Multi-Armed Bandits (Thompson Sampling) and Bayesian A/B testing.
3. Dirichlet Distribution $\text{Dirichlet}(\boldsymbol{\alpha})$
Multivariate generalization of the Beta distribution over probability vectors $\mathbf{p} = [p_1, \dots, p_K]$ where $\sum p_k = 1.0$.
Core foundation for Latent Dirichlet Allocation (LDA) topic modeling.
Say this out loud
"Bernoulli models single 0/1 outcomes; Binomial models success counts in N trials; Poisson models event rates per time window with mean equal to variance λ. Gaussian N(μ, σ²) is the central limit baseline deriving MSE loss. Beta(α, β) models uncertainty over probabilities on [0,1] for Thompson Sampling, while Dirichlet generalizes Beta to multi-class probability vectors for topic modeling."
Follow-ups to expect
- What is the Memoryless Property of the Exponential Distribution? $P(T > s + t \mid T > s) = P(T > t)$. The probability of waiting an additional $t$ minutes does not depend on how long $s$ you have already been waiting.
- What is the Heavy-Tailed Cauchy distribution? Ratio of two independent standard Gaussians. Has undefined mean and infinite variance, violating the Central Limit Theorem.
Check yourself
Which continuous probability distribution is bounded strictly on [0, 1] and serves as the conjugate prior for Binomial / Bernoulli likelihoods in Bayesian modeling?