The mathematical recipe for updating beliefs given evidence — and why intuition fails on rare events.
Bayes’ Theorem updates a prior probability P(A) into a posterior P(A|B) upon observing evidence B: P(A|B) = P(B|A)P(A) / P(B). In machine learning, it underpins Naive Bayes, MAP estimation, Bayesian optimization, and VAEs. The key interview insight is base-rate neglect: when prior probability is low, even a test with 99% accuracy yields more false positives than true positives.
The magic theorem of statistics that guarantees sample means tend toward a Gaussian distribution regardless of original population shape.
The Central Limit Theorem (CLT) states that the sample mean of N independent, identically distributed (i.i.d.) random variables with finite variance approaches a Normal distribution N(μ, σ²/N) as N → ∞, regardless of the underlying population distribution. In ML and experimentation, CLT powers confidence intervals, hypothesis testing (z-test, t-test), A/B testing variance estimation, and mini-batch gradient descent stability.
Why sample averages inevitably converge to true population expectations as sample size approaches infinity.
The Law of Large Numbers (LLN) guarantees that the sample mean X̄_N of N independent, identically distributed (i.i.d.) random variables converges to the true population expectation μ as N → ∞. The Weak LLN proves convergence in probability; the Strong LLN proves almost sure convergence. LLN underpins Monte Carlo integration, empirical risk minimization in ML, A/B testing sample size stability, and casino profitability.
The fundamental operators for measuring central tendency and dispersion in probability and machine learning.
Expectation E[X] is the probability-weighted average of all possible values of random variable X. Variance Var(X) = E[(X - E[X])²] measures expected squared deviation from the mean, with properties Var(aX + b) = a²Var(X). In machine learning, E[X] and Var(X) form the foundation of loss functions, MSE bias-variance decomposition, batch normalization, and weight initialization.
Covariance tells you direction; correlation tells you direction and strength independent of scale.
Covariance Cov(X,Y) = E[(X - E[X])(Y - E[Y])] measures joint variability of two random variables, but its magnitude depends on measurement units. Pearson Correlation r = Cov(X,Y) / (σ_X σ_Y) normalizes covariance to [-1, +1], providing a scale-invariant measure of linear association. Key interview topics include Pearson vs Spearman correlation, covariance matrix construction, and correlation vs causation.
The probability distributions every AI/ML engineer is expected to recognize, parameterize, and apply.
Probability distributions model data generation processes. Discrete distributions include Bernoulli (binary trials), Binomial (k successes in N trials), Poisson (event rates in fixed intervals), and Categorical/Multinomial. Continuous distributions include Uniform, Gaussian/Normal (central limit theorem baseline), Exponential (time between events), and Beta/Dirichlet (priors over probabilities). Selecting the right parametric distribution dictates likelihood formulations and loss functions.
False Positives vs False Negatives: why minimizing one error always inflates the other.
Statistical decisions suffer from two distinct error modes: Type I Error (α - False Positive / False Alarm) occurs when we reject a true null hypothesis H₀. Type II Error (β - False Negative / Missed Detection) occurs when we fail to reject a false null hypothesis H₀. Statistical Power (1 - β) is the probability of correctly detecting a real effect. Balancing α and β is a cost trade-off driven by business risk.
Measuring vector magnitudes, distances, and regularization penalties across L1, L2, and Linf norms.
Vector Norms measure the size or length of a vector in vector space. The general L_p norm is defined as ||x||_p = ( ∑ |x_i|^p )^(1/p). The L1 Norm (Manhattan distance) sums absolute values, driving sparse feature selection in Lasso regression. The L2 Norm (Euclidean distance) measures straight-line distance, penalizing large outliers smoothly in Ridge regression. The L_infinity Norm (Chebyshev distance) measures the maximum absolute element.
Measuring vector alignment, magnitude projection, and semantic similarity in embedding spaces.
Dot Product and Cosine Similarity are the two core metrics for comparing vectors in machine learning. Dot product u · v = ∑ u_i v_i = ||u|| ||v|| cos(θ) combines both directional alignment and vector magnitude. Cosine similarity normalizes by vector lengths, measuring purely directional angle alignment cos(θ) on a [-1, 1] scale. When vectors are pre-normalized to unit length (||u|| = 1), Dot Product and Cosine Similarity become 100% mathematically identical.
Distinguishing statistical association from true cause-and-effect relationships.
Correlation vs Causation is a foundational distinction in data science and empirical inference. Correlation measures linear statistical association between variables P(Y | X), while Causation measures the intervention effect P(Y | do(X)) of changing X directly. Correlation does not imply causation due to Spurious Correlations, Reverse Causality, and Unobserved Confounders.
When two dependent variables become completely independent once a third confounding variable is observed.
Two random variables X and Y are conditionally independent given Z (denoted X ⊥ Y | Z) if P(X, Y | Z) = P(X | Z) P(Y | Z). Knowledge of Y provides zero additional information about X once Z is already known. Conditional independence simplifies joint probability distributions, enabling Naive Bayes classifiers, Bayesian Networks (DAGs), and Causal Inference d-separation algorithms.
Finding the parameter values that maximize the probability of observing your collected sample data.
Maximum Likelihood Estimation (MLE) estimates parameters θ of a probability distribution by maximizing the Likelihood function L(θ) = P(X | θ). For i.i.d. data, we maximize Log-Likelihood log L(θ) = ∑ log P(x_i | θ) because log turns products into computationally stable sums. MLE forms the statistical foundation for parameter fitting in OLS Linear Regression (under Gaussian noise), Logistic Regression, and Neural Networks (via Cross-Entropy).
Frequentist likelihood maximization vs Bayesian posterior maximization with prior regularization.
Maximum Likelihood Estimation (MLE) maximizes data likelihood P(X | θ) ignoring prior beliefs. Maximum A Posteriori (MAP) incorporates a prior distribution P(θ), maximizing posterior P(θ | X) ∝ P(X | θ) P(θ). MAP acts as regularized MLE: assuming a zero-mean Gaussian prior P(θ) yields L2 Regularization (Ridge), while a Laplacian prior yields L1 Regularization (Lasso). As dataset size N → ∞, MAP converges to MLE.
How to make statistical decisions under uncertainty without being tricked by random noise.
Hypothesis testing evaluates whether an observed effect is statistically significant or merely random chance. We set a Null Hypothesis H₀ (no effect) and Alternative Hypothesis H₁ (effect exists), compute a test statistic, and derive a p-value: P(Data | H₀). If p-value < significance level α (typically 0.05), we reject H₀. Key interview topics include Type I vs Type II errors, p-value misinterpretations, Z-test vs t-test, and effect size vs p-value.
Measuring your experiment's ability to detect real underlying business changes.
Statistical Power is the probability that a hypothesis test correctly detects a real effect when one truly exists (1 - Type II error rate beta). Standard experimental design targets 80% statistical power (power = 0.80). Power depends on four interconnected variables: significance level alpha, effect size, sample size N, and population variance. Running an underpowered A/B test leads to false negatives, wasting engineering effort by discarding winning product features.
Quantifying estimation uncertainty around point metrics with upper and lower statistical bounds.
A Confidence Interval (CI) provides a range of plausible values for an unknown population parameter based on sample data. A 95% Confidence Interval means that if you repeated the exact sampling process 100 times, 95 of those calculated intervals would contain the true population parameter. Confidence intervals combine sample mean, standard error, and critical z-score (or t-score). They provide far richer context than single point estimates or p-values alone.
The gold standard for product decision-making at tech companies.
A/B testing evaluates product changes by splitting users randomly into Control (A) and Treatment (B) variants. End-to-end execution requires hypothesis definition, metric selection (Guardrail vs Primary), sample size calculation based on Minimum Detectable Effect (MDE) and Power (80%), randomized unit assignment, AA validation, and statistical testing. Key interview topics include sample ratio mismatch (SRM), peeking pitfall, novelty effects, and network spillovers.
Contrasting fixed parameter assumptions with probability distributions over parameters.
Frequentist and Bayesian statistics represent two fundamentally different philosophical approaches to probability and inference. Frequentists view parameters as fixed unknown constants, measuring long-run frequency over hypothetical repeated samples (p-values, 95% confidence intervals). Bayesians view parameters as random variables with probability distributions, updating prior beliefs with data to compute posterior probability distributions (credible intervals, Bayes factors).
The special vectors that change only in magnitude—not direction—when transformed by a matrix.
For a square matrix A, an eigenvector v is a non-zero vector that satisfies Av = λv. The scalar λ is the corresponding eigenvalue, representing the factor by which v is stretched or shrunk. In machine learning, eigenvectors define principal component axes in PCA, stationary distributions in Markov Chains (PageRank), spectral graph embeddings, and singular values in SVD.
Understanding linear independence, full rank matrices, and why non-invertible matrices collapse linear regression computations.
Matrix Rank measures the maximum number of linearly independent column or row vectors in a matrix. A square n × n matrix is Full Rank (rank = n) if and only if its columns are linearly independent, its determinant is non-zero, its null space contains only the zero vector, and its matrix inverse A^-1 exists. When features in a dataset are linearly dependent (e.g. perfect multicollinearity), matrix X^T X becomes Rank Deficient (singular), causing (X^T X)^-1 to fail during OLS linear regression fitting.
Mastering first and second derivatives across scalar functions, vector outputs, and multi-dimensional loss surfaces.
Gradients, Jacobians, and Hessians represent the calculus foundation for optimizing machine learning models. The Gradient vector ∇f(x) contains first-order partial derivatives of a scalar loss function with respect to input vector x. The Jacobian matrix J contains first-order partial derivatives of a vector-valued function f: R^n -> R^m, used in backpropagation chain rule. The Hessian matrix H contains second-order partial derivatives of a scalar function, measuring local loss surface curvature.
Understanding the multivariable calculus chain rule that enables reverse-mode automatic differentiation.
Backpropagation (Backprop) is the algorithmic application of the multivariable calculus Chain Rule for computing exact loss gradients with respect to all trainable neural network weights. For composed functions y = f(g(x)), the chain rule multiplies local Jacobian matrices: dy/dx = (dy/dg) · (dg/dx). Reverse-Mode Automatic Differentiation traverses computation graphs backward from loss L, caching intermediate activation values to compute all N weight gradients in a single backward pass with O(N) compute complexity.
Understanding why convex optimization guarantees global minima while non-convex deep learning relies on local saddle point navigation.
Convexity is a fundamental property in optimization theory. A set S is convex if the line segment connecting any two points in S lies entirely within S. A function f(x) is convex if its epigraph is a convex set, satisfying f(λ x + (1-λ) y) ≤ λ f(x) + (1-λ) f(y) for λ ∈ [0, 1]. In convex optimization (Linear Regression, SVMs, Logistic Regression), any local minimum is guaranteed to be a global minimum. Deep Neural Networks are highly Non-Convex, possessing millions of saddle points, local minima, and plateau ravines.
Information theory's core trilogy: measuring uncertainty, cross-distribution surprise, and probabilistic distance.
Entropy H(P) measures the inherent uncertainty/impurity of a probability distribution P. Cross-Entropy H(P, Q) measures the expected code length when events from true distribution P are encoded using predicted model distribution Q. Kullback-Leibler (KL) Divergence D_KL(P || Q) = H(P, Q) - H(P) quantifies the extra surprise (relative entropy) from approximating P with Q. In ML, minimizing Cross-Entropy loss is identical to minimizing KL Divergence because true target entropy H(P) is constant.
Understanding why high-dimensional vector spaces become extremely sparse, causing distance metrics to fail.
The Curse of Dimensionality (coined by Richard Bellman) describes the severe challenges that arise when analyzing data in high-dimensional feature spaces (d >> 100). As dimensionality d increases, the volume of feature space grows exponentially (2^d), making training data extremely sparse. In high dimensions, Euclidean distances between points converge to nearly equal values (distance concentration), ruining distance-based algorithms like k-NN and k-means. Dimensionality reduction techniques (PCA, UMAP, Autoencoders) mitigate the curse by projecting data onto low-dimensional manifold subspaces.
How lurking confounders flip aggregate statistical trends when data is grouped into sub populations.
Simpson's Paradox occurs when a statistical trend or correlation observed in aggregated data reverses direction when the data is split into sub population subgroups. The paradox is driven by Confounding Variables that influence both group assignment and primary outcomes. Resolving Simpson's Paradox requires causal DAG modeling to determine whether to aggregate or disaggregate data based on real world causal relationships.
Modeling memoryless state transitions across discrete time steps using transition probability matrices.
A Markov Chain is a stochastic model describing a sequence of state transitions where future states depend exclusively on the current state. The Markov Property states that memoryless conditional probability P(X_t+1 | X_t, X_t-1...) = P(X_t+1 | X_t). Markov Chains are governed by a Transition Matrix P, converging over time to a unique Stationary Distribution pi under irreducible and aperiodic conditions.
Updating Bayesian beliefs in closed form without needing heavy numerical integrations.
In Bayesian inference, a prior distribution is conjugate to a likelihood function if the resulting posterior distribution belongs to the exact same probability family as the prior. Conjugacy simplifies Bayesian updating: instead of calculating intractable integrals, updating your belief requires simply adding sample observation counts directly to the prior parameters. Classic conjugate pairs include Beta-Binomial for success rates, Dirichlet-Multinomial for multi-class proportions, and Normal-Normal for continuous means.
Avoiding severe statistical traps like peeking, novelty effects, and sample ratio mismatches in A/B experiments.
A/B Testing pitfalls corrupt experimental validity and lead teams to ship bad product features. Continuous Peeking (checking p-values daily without sample size correction) inflates false positive rates from 5% up to 30%+. Novelty and Primacy Effects cause temporary metric spikes or drops that fade over time. Sample Ratio Mismatch (SRM) occurs when the actual user split ratio deviates from the expected control/treatment assignment, signaling severe technical logging bugs.
Controlling family-wise error rates when evaluating multiple metrics or testing dozens of product variants simultaneously.
The Multiple Testing Problem occurs when running many statistical hypothesis tests simultaneously. As the number of independent tests k increases, the probability of obtaining at least one false positive (Type I error) inflates exponentially: Family-Wise Error Rate (FWER) = 1 - (1 - alpha)^k. Corrections include the strict Bonferroni Correction (adjusting significance threshold alpha_new = alpha / k) and the False Discovery Rate (FDR - Benjamini-Hochberg procedure), which balances discovery power and false positives.
Factoring any rectangular matrix into rotation, scaling, and orthogonal basis matrices.
Singular Value Decomposition (SVD) is the foundational matrix factorization technique in linear algebra. SVD factorizes ANY real matrix A (m × n) into three matrices: A = U Σ V^T, where U is an m × m orthogonal matrix of left-singular vectors, Σ is an m × n diagonal matrix of non-negative singular values, and V^T is an n × n orthogonal matrix of right-singular vectors. Truncated SVD provides the mathematically optimal low-rank matrix approximation (Eckart-Young-Mirsky Theorem), forming the backbone of PCA, image compression, and Latent Semantic Analysis (LSA).
Comparing LU, QR, and Cholesky matrix factorizations for solving linear systems and optimizing ML algorithms.
Matrix Decompositions factorize complex matrices into products of simpler canonical matrices (triangular, orthogonal, or diagonal). LU Decomposition factors A = L U (Lower and Upper triangular) for fast linear system solving. QR Decomposition factors A = Q R (Orthogonal Q and Upper triangular R) for stable Gram-Schmidt orthogonalization and least-squares. Cholesky Decomposition factors symmetric positive-definite matrices A = L L^T, running 2x faster than LU for Gaussian processes and covariance modeling.
Understanding why positive definite matrices guarantee strict convexity, positive energy, and stable optimization.
A symmetric matrix A is Positive Definite (SPD) if x^T A x > 0 for every non-zero vector x. Equivalently, all eigenvalues of A are strictly positive (lambda_i > 0). Positive definite matrices represent bowl-shaped strictly convex quadratic loss surfaces, guaranteeing a single unique global minimum in optimization. Key machine learning applications include Covariance matrices, Hessian matrices at local minima, and Kernel matrices in SVMs (Mercer's Condition).
Approximating complex non-linear loss functions using local polynomial expansions around current parameter weights.
Taylor Series Expansion approximates smooth non-linear functions around a local expansion point x_0 using polynomial series of derivatives. First-order Taylor expansion f(x) ≈ f(x_0) + ∇f(x_0)^T (x - x_0) forms the local linear approximation underpinning Gradient Descent. Second-order Taylor expansion f(x) ≈ f(x_0) + ∇f(x_0)^T Δx + (1/2) Δx^T H Δx incorporates curvature (Hessian matrix H), deriving Newton's Method and natural gradient optimizers.
Solving constrained optimization problems by converting constraints into unconstrained Lagrangian scalar multiplier functions.
Lagrange Multipliers optimize a objective function f(x) subject to equality constraints g(x) = 0 or inequality constraints h(x) ≤ 0. The technique constructs the Lagrangian function L(x, λ) = f(x) + λ g(x). At constrained optimal points, the gradient of the objective function ∇f(x) must be parallel to the gradient of the constraint ∇g(x), yielding ∇f(x) + λ ∇g(x) = 0. For inequality constraints, the Karush-Kuhn-Tucker (KKT) conditions provide the necessary first-order optimality criteria, underpinning SVM dual formulations.
Comparing probability distance metrics: asymmetric KL divergence, symmetric JS divergence, and smooth Earth Mover Wasserstein distance.
Measuring distance between two probability distributions P and Q is a core problem in generative modeling (GANs, VAEs). Kullback-Leibler (KL) Divergence is asymmetric and explodes to infinity when distributions do not overlap. Jensen-Shannon (JS) Divergence symmetrizes KL divergence, producing a bounded metric [0, ln(2)]. Wasserstein Distance (Earth Mover's Distance) measures the minimal work required to transport probability mass from P to Q, providing smooth, non-vanishing gradients even when distributions have completely disjoint support.
Quantifying non-linear dependence between random variables via information theory.
Mutual Information I(X; Y) measures the amount of information obtained about random variable X by observing variable Y. Unlike Pearson Correlation (which captures only linear relationships), Mutual Information measures general non-linear statistical dependence. Mathematically, I(X; Y) = D_KL( P(X, Y) || P(X)P(Y) ) = H(X) - H(X|Y). Applications include non-linear feature selection, contrastive representation learning (InfoNCE loss), and disentangled representation learning.
Approximating complex distributions using Bootstrap resampling, Importance Sampling, and Markov Chain Monte Carlo.
Statistical Sampling Methods enable drawing samples and estimating expectations from complex probability distributions. Bootstrap Resampling draws repeated random samples with replacement to estimate confidence intervals without parametric assumptions. Importance Sampling shifts sampling toward high probability proposal distributions using likelihood ratio weights. Markov Chain Monte Carlo (MCMC - Metropolis Hastings, Gibbs Sampling) generates sample sequences whose stationary distribution matches complex un-normalized posterior distributions.
Mapping cause and effect relationships using Directed Acyclic Graphs and Structural Causal Models.
Causal Inference provides mathematical tools (Judea Pearl's Structural Causal Models and DAGs) to estimate true causal effects from observational data. Directed Acyclic Graphs (DAGs) represent causal assumptions visually with nodes (variables) and directed arrows (causal paths). Key graph structures include Confounders (common causes), Colliders (common effects), and Mediators (intermediate steps), determining which variables must be controlled or un-conditioned to block spurious backdoors.
Balancing treatment and control groups in observational studies using propensity score matching.
Propensity Score Matching (PSM - Rosenbaum & Rubin, 1983) is a statistical technique for estimating causal treatment effects from observational data. The Propensity Score e(X) = P(T = 1 | X) is the conditional probability that a subject receives treatment given observed baseline covariates X. Matching treatment subjects with control subjects sharing identical propensity scores balances baseline confounder distributions, simulating a Randomized Controlled Trial.