Law of Large Numbers

What the Law States

Let $X_1, X_2, \dots, X_N$ be i.i.d. random variables with finite expectation $\mathbb{E}[X] = \mu$.

Sample mean: $\bar{X}N = \frac{1}{N} \sum{i=1}^N X_i$.

• Weak Law of Large Numbers (WLLN): Sample mean converges in probability to $\mu$:

$$\lim_{N \to \infty} P\left( \left| \bar{X}_N - \mu \right| > \epsilon \right) = 0 \quad \forall \epsilon > 0$$

• Strong Law of Large Numbers (SLLN): Sample mean converges almost surely to $\mu$:

$$P\left( \lim_{N \to \infty} \bar{X}_N = \mu \right) = 1$$

   Sample Mean X̄_N
   1.0 ┤  /\  /\
   0.7 ┤ /  \/  \───/\───────────────────────────────────────► True μ = 0.50
   0.3 ┤              \──/¯¯\──/\──/¯\──/\──/¯\──
   0.0 ┴─────────────────────────────────────────────────────► Sample Size N
       N = 10         N = 100         N = 10,000           N = 1,000,000

LLN vs Central Limit Theorem (CLT)

• Law of Large Numbers: Describes WHERE the sample mean goes (it converges to constant $\mu$).
• Central Limit Theorem: Describes HOW the sample mean gets there (the distribution of errors around $\mu$ forms a Gaussian $\mathcal{N}(\mu, \sigma^2/N)$).

Key Applications in AI/ML

1. Empirical Risk Minimization (ERM): Justifies optimizing average training loss $\frac{1}{N} \sum L(f(x_i), y_i)$ as a proxy for true unobservable risk $\mathbb{E}_{(x,y)\sim P}[L(f(x), y)]$.
2. Monte Carlo Sampling: Estimates complex multidimensional integrals $\mathbb{E}_{P(x)}[f(x)]$ by drawing $N$ random samples and computing average $\frac{1}{N} \sum f(x_i)$.
3. Evaluation Metrics: Guarantees test set accuracy/F1 metrics approach true generalization performance as test set size $N \to \infty$.

Say this out loud

"The Law of Large Numbers proves that sample averages converge to true population expectations as sample size N approaches infinity. LLN guarantees WHERE the sample mean goes, while CLT describes the Gaussian distribution of errors around that mean. In ML, LLN underpins Empirical Risk Minimization, proving that training set average loss converges to true expected risk for large N."

Follow-ups to expect

• What happens to LLN if variance is infinite? For distributions with undefined or infinite means/variances (e.g. Cauchy distribution), sample means do not converge to a constant—averages fluctuate wildly no matter how large N becomes.
• How does LLN protect casinos and insurance companies? While individual coin flips or insurance claims are random, aggregating millions of independent policies/bets guarantees that average payouts converge predictably to expected profit margins.