Gaussian Mixtures & EM

Model Formulation

Data $x_i \in \mathbb{R}^d$ is generated from a mixture of $K$ Gaussian components:

$$P(x) = \sum_{k=1}^K \pi_k \mathcal{N}(x \mid \mu_k, \Sigma_k)$$

• Mixing Coefficients ($\pi_k$): $\sum_{k=1}^K \pi_k = 1, \quad \pi_k \ge 0$.
• Component Means ($\mu_k$): Center of Gaussian component $k$.
• Covariance Matrices ($\Sigma_k$): Shape, orientation, and spread of component $k$.

       k-Means Hard Clusters                 GMM Soft Elliptical Clusters
     (Spherical Voronoi Cells)                (Overlapping Probabilities)
          ┌──────┬──────┐                         . .  .   . . .
          │  *   │  *   │                      (  *  )   (   *   )  γ_ik = 0.85
          │      │      │                     . .  .   . . .
          └──────┴──────┘

The Expectation-Maximization (EM) Algorithm

EM finds local maximum of log-likelihood $\sum_{i=1}^N \ln \left( \sum_{k=1}^K \pi_k \mathcal{N}(x_i \mid \mu_k, \Sigma_k) \right)$:

1. Expectation Step (E-step): Compute Responsibilities

Compute posterior probability $\gamma_{ik}$ that point $x_i$ belongs to component $k$:

$$\gamma_{ik} = \frac{\pi_k \mathcal{N}(x_i \mid \mu_k, \Sigma_k)}{\sum_{j=1}^K \pi_j \mathcal{N}(x_i \mid \mu_j, \Sigma_j)}$$

2. Maximization Step (M-step): Re-estimate Parameters

Update parameters using weighted responsibilities:

$$N_k = \sum_{i=1}^N \gamma_{ik} \quad \text{(Effective count of points in component } k\text{)}$$

$$\mu_k^{\text{new}} = \frac{1}{N_k} \sum_{i=1}^N \gamma_{ik} x_i, \quad \Sigma_k^{\text{new}} = \frac{1}{N_k} \sum_{i=1}^N \gamma_{ik} (x_i - \mu_k^{\text{new}})(x_i - \mu_k^{\text{new}})^T, \quad \pi_k^{\text{new}} = \frac{N_k}{N}$$

Repeat E and M steps until log-likelihood converges.

Covariance Matrix Constraints in Scikit-Learn

• covariance_type='full': Each component has its own general covariance matrix (Elliptical, arbitrary orientation).
• covariance_type='tied': All components share the exact same covariance matrix.
• covariance_type='diag': Axis-aligned elliptical clusters ($\Sigma_k$ is diagonal).
• covariance_type='spherical': Spherical clusters ($\Sigma_k = \sigma_k^2 I$) — equivalent to soft k-Means!

Say this out loud

"GMM is a probabilistic clustering model representing data as a sum of K Gaussians. Unlike k-Means hard assignments, GMM outputs soft posterior probabilities P(k|x) and fits full covariance matrices Σ_k for elliptical clusters. Parameters are optimized via Expectation-Maximization: the E-step calculates component responsibilities γ_ik, and the M-step updates means, covariances, and mixing weights."

Follow-ups to expect

• Does EM guarantee global convergence? No. EM guarantees monotonic log-likelihood improvement at every step ($\mathcal{L}^{(t+1)} \ge \mathcal{L}^{(t)}$), but can converge to local maxima depending on initialization (use k-Means++ to seed initial GMM means).
• What is Singular Covariance in GMM? If a Gaussian component collapses onto a single data point, its variance $\sigma^2 \to 0$, causing likelihood to approach $\infty$. Fix by adding small regularization value $\epsilon I$ to diagonal of $\Sigma_k$.