Principal Component Analysis

What is Principal Component Analysis (PCA)?

When working with high dimensional data (such as 1000 features), visualizations become impossible and models suffer from the Curse of Dimensionality.

Principal Component Analysis (PCA) compresses high dimensional data into a smaller number of uncorrelated linear features called Principal Components.

  2D High Dimensional Feature Space (x1, x2)
     ▲
     │    /  ◄── First Principal Component PC1 (Direction of Maximum Variance!)
     │   / .
     │  / . .
     │ / .
  ───┴─────────────► Second Principal Component PC2 (Orthogonal to PC1!)

1. PC1: Direction along which data points are most spread out (Maximizes Variance).
2. PC2: Direction perpendicular (orthogonal) to PC1 that captures the second highest variance.
3. PC3...PCk: Subsequent orthogonal directions capturing remaining variance.

Step-by-Step Mathematical Algorithm

Given data matrix $X$ of shape $[N \times d]$:

Step 1: Mean Centering

Subtract feature means so every feature has mean zero:

$$X_{\text{centered}} = X - \mu$$

Step 2: Compute Covariance Matrix

Calculate the $d \times d$ sample covariance matrix $\Sigma$:

$$\Sigma = \frac{1}{N-1} X_{\text{centered}}^T X_{\text{centered}}$$

Covariance matrix measures pairwise linear correlations between all feature pairs.

Step 3: Compute Eigenvectors and Eigenvalues

Solve the characteristic equation for covariance matrix $\Sigma$:

$$\Sigma v_i = \lambda_i v_i$$

• Eigenvectors ($v_i$): Represent the directional axes of Principal Components.
• Eigenvalues ($\lambda_i$): Represent the amount of variance along each eigenvector direction.

Step 4: Sort and Select Top K Components

Sort eigenvalues in descending order ($\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d$).

Select the top $k$ eigenvectors to form projection matrix $W_k \in \mathbb{R}^{d \times k}$.

Step 5: Project Data onto Lower Dimension

Multiply original centered data by projection matrix:

$$X_{\text{reduced}} = X_{\text{centered}} W_k \in \mathbb{R}^{N \times k}$$

Preserved Variance Ratio

The Explained Variance Ratio for component $i$ is:

$$\text{Explained Variance Ratio}i = \frac{\lambda_i}{\sum{j=1}^d \lambda_j}$$

Summing ratios for $k$ components tells you what percentage of original total information was retained (for example, keeping 95 percent of variance).

Say this out loud

PCA is an unsupervised linear dimensionality reduction algorithm that projects data onto orthogonal principal components to maximize preserved variance. It mean centers data, computes the covariance matrix, and calculates eigenvectors for direction and eigenvalues for variance magnitude. PCA decorrelates features and compresses dimensions while retaining maximum information.

Followups to expect

1. Why is SVD (Singular Value Decomposition) preferred over Eigenvalue decomposition in practice? Computing covariance matrix $\Sigma = X^T X$ takes time and risks numerical instability. Direct SVD on $X = U S V^T$ is faster and numerically stable (np.linalg.svd).
2. Can PCA capture non linear patterns? No, standard PCA finds linear combinations of features. Use Kernel PCA or non linear embedding algorithms (t-SNE / UMAP) for non linear manifolds.