t-SNE vs UMAP

Visualizing High Dimensional Data

Linear methods like PCA compress high dimensional data along straight variance axes.

However, complex datasets (like word embeddings or image features) live on curved non linear manifolds. PCA collapses these curves into overlapping visual clutter.

t-SNE and UMAP are non linear algorithms designed specifically to map complex high dimensional manifolds into 2D or 3D scatter plots for human inspection.

┌──────────────────────────┬──────────────────────────┐
│ 1. t-SNE (2008)          │ 2. UMAP (2018)           │
├──────────────────────────┼──────────────────────────┤
│ Converts distances to    │ Uses Riemannian geometry │
│ probability distributions│ & fuzzy simplicial sets. │
│ Excellent local clusters.│ Preserves LOCAL AND      │
│ Destroys global distances│ GLOBAL relationships!    │
│ Slow $O(N^2)$ compute.   │ Fast, supports transform!│
└──────────────────────────┴──────────────────────────┘

1. t-SNE (t-Distributed Stochastic Neighbor Embedding)

Introduced by van der Maaten & Hinton (2008).

How t-SNE Works

1. High Dimensional Space: Converts pairwise Euclidean distances between points into conditional probabilities $p_{j|i}$ using a Gaussian distribution. Perplexity controls effective neighbor window size.
2. Low Dimensional 2D Space: Maps points into 2D space using a heavy tailed Student t-distribution ($q_{ij}$).
3. Kullback-Leibler (KL) Divergence Minimization: Uses gradient descent to minimize KL divergence between high dimensional probabilities $P$ and low dimensional probabilities $Q$:

$$C = \text{KL}(P \parallel Q) = \sum_i \sum_j p_{ij} \log \frac{p_{ij}}{q_{ij}}$$

The Crowding Problem Fix

In high dimensional space ($d = 100$), there is immense volume surrounding a point. In 2D space, volume shrinks drastically.

If we used Gaussian distributions in 2D, moderate distances would collapse into a dense central crowd. The heavy-tailed Student t-distribution $1 / (1 + |y_i - y_j|^2)$ pushes moderate distances outward in 2D, creating clean, isolated cluster visual separation.

2. UMAP (Uniform Manifold Approximation and Projection)

Introduced by Leland McInnes et al. (2018).

UMAP builds a fuzzy simplicial set graph representation of the high-dimensional manifold, optimizing a cross-entropy objective against a low-dimensional layout.

Why UMAP Outperforms t-SNE

1. Preserves Global Structure: In t-SNE, distances between distant clusters are completely arbitrary. UMAP preserves both local neighborhoods AND global macro relationships (e.g. cluster A is closer to cluster B than cluster C).
2. 10x Faster Speed: Uses Approximate Nearest Neighbors (Nearest Neighbor Descent), running drastically faster on large datasets.
3. Supports transform(): Can project new unseen test samples into an existing 2D embedding space.

Say this out loud

t-SNE and UMAP are non linear dimensionality reduction techniques for 2D data visualization. t-SNE maps Gaussian probabilities to heavy-tailed Student t-distributions to solve the crowding problem, focusing on local clusters while destroying global distances. UMAP uses fuzzy simplicial sets to preserve both local and global cluster relationships, running faster and supporting transform on new test points.

Followups to expect

1. Why is Perplexity critical in t-SNE? Perplexity sets the effective number of nearest neighbors evaluated for each point (typically 5 to 50). Low perplexity focuses on tiny local details; high perplexity considers broader global structure.
2. Why should you never interpret t-SNE cluster sizes or distances literally? t-SNE expands dense clusters and contracts sparse clusters to equalize probability distributions. Cluster sizes and inter-cluster distances on a t-SNE plot do not reflect true Euclidean distances.