Expectation & Variance

Expectation: The Probability-Weighted Mean

For discrete random variable X with probability mass function P(X = x_i):

E[X] = ∑ x_i P(X = x_i)

For continuous random variable X with probability density function f(x):

E[X] = ∫ x f(x) dx

Key Properties of Expectation

• Linearity of Expectation: E[aX + bY + c] = a E[X] + b E[Y] + c (Holds regardless of whether X and Y are independent!).
• Expectation of Product: E[XY] = E[X] E[Y] only if X and Y are uncorrelated (Cov(X,Y) = 0).

Variance & Standard Deviation

Variance measures how spread out values are around the mean:

Var(X) = E[ (X - E[X])² ] = E[X²] - (E[X])²

Standard Deviation is SD(X) = √Var(X) (in original measurement units).

Key Properties of Variance

• Non-negativity: Var(X) ≥ 0.
• Scaling: Var(aX + b) = a² Var(X).
• Sum of Variables: Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y).

Applications in ML

1. Batch Normalization: Subtracts sample expectation μ_B = E[X] and divides by sample standard deviation σ_B = √Var(X) to stabilize training.
2. Weight Initialization: Xavier and He initializations set initial weight variance Var(W) = 2 / d_in to preserve activation variance across layers.
3. Bias-Variance Decomposition: Test MSE decomposes into Bias[f̂]² + Var[f̂] + σ².

Say this out loud

"Expectation is the probability-weighted average E[X], while Variance Var(X) = E[X²] - (E[X])² measures expected squared spread around the mean. Linearity of expectation E[aX + bY] = aE[X] + bE[Y] holds always, but variance scaling squares constants: Var(aX + b) = a²Var(X). In ML, keeping expectation zero and controlling variance across layers is what prevents vanishing and exploding gradients."

Follow-ups to expect

• What is Jensen's Inequality? For a convex function g, E[g(X)] ≥ g(E[X]). For a concave function like log, E[log(X)] ≤ log(E[X]). Crucial in variational inference (ELBO).
• Sample Variance vs Population Variance denominator? Sample variance divides by N - 1 (Bessel's correction) instead of N to yield an unbiased estimator E[s²] = σ².
• What is Covariance? Cov(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y].