Eigenvectors & Eigenvalues

The Core Geometric Intuition

Matrix multiplication A x generally rotates and rescales vector x.

An Eigenvector v is a special vector whose direction is unaffected by A:

A v = λ v

• Eigenvector v: Direction vector (non-zero v ≠ 0).
• Eigenvalue λ: Scalar scaling factor (stretch if |λ| > 1, shrink if |λ| < 1, flip if λ < 0).

  Standard Vector x:        Ax rotates and stretches x.
  Eigenvector v:            Av = λv stays on the same line span!

Solving for Eigenvalues & Eigenvectors

1. Characteristic Equation: (A - λI) v = 0.
2. Determinant Condition: For non-zero v to exist, (A - λI) must be non-invertible:

det( A - λ I ) = 0

Solving this polynomial yields d eigenvalues λ_1, ..., λ_d. Plugging each λ_i back into (A - λ_i I) v_i = 0 yields corresponding eigenvectors v_i.

Spectral Theorem for Symmetric Matrices

Real symmetric matrices (A = Aᵀ), common in ML (Covariance matrices, Hessian matrices, Laplacian matrices):

1. All eigenvalues λ_i are real numbers.
2. Eigenvectors v_i corresponding to distinct eigenvalues are strictly orthogonal (v_i · v_j = 0).
3. Matrix can be eigen-decomposed as A = V Λ Vᵀ, where V is an orthogonal matrix of eigenvectors (VᵀV = I) and Λ is a diagonal matrix of eigenvalues.

Applications across ML

• PCA: Eigenvectors of covariance matrix XᵀX define directions of maximum variance; eigenvalues measure variance magnitude.
• PageRank: Google's PageRank vector is the principal eigenvector (λ = 1) of the web transition probability matrix.
• Spectral Clustering: Graph Laplacian eigenvectors partition graph networks into dense communities.

Say this out loud

"An eigenvector v of square matrix A satisfies Av = λv—it maintains its line direction under matrix transformation and is scaled only by eigenvalue λ. For symmetric matrices like covariance matrices, eigenvectors are strictly orthogonal. In PCA, eigenvectors of the sample covariance matrix represent axes of maximum variance, and eigenvalues quantify the variance explained by each axis."

Follow-ups to expect

• What is a Positive Definite Matrix? A symmetric matrix where all eigenvalues are strictly positive (λ_i > 0). Ensures quadratic forms xᵀ A x > 0 are strictly convex with a single global minimum.
• What is Power Iteration? An iterative algorithm to find the dominant eigenvector (largest eigenvalue λ_max) by repeatedly multiplying a random vector by matrix A: v_{k+1} = A v_k / ||A v_k||.