Choosing k: Elbow & Silhouette
Determining the optimal number of clusters k using Elbow inertia plots and Silhouette coefficient scores.
The Challenge of Selecting $k$
In unsupervised clustering algorithms like k-Means, the algorithm does not know how many real clusters exist in your data.
If you set $k = 1$, the model underfits.
If you set $k = N$ (where $N$ is total sample count), every data point becomes its own centroid, resulting in Inertia = 0.0, but producing a completely useless model!
We need quantitative methods to select the Optimal $k$ Value.
┌──────────────────────────┬──────────────────────────┐
│ 1. THE ELBOW METHOD │ 2. SILHOUETTE ANALYSIS │
├──────────────────────────┼──────────────────────────┤
│ Plots Inertia vs k. │ Measures cluster density │
│ Looks for the inflection │ and separation distance. │
│ "elbow" point where │ Score ranges from │
│ inertia gain flattens. │ -1.0 to +1.0. │
└──────────────────────────┴──────────────────────────┘
1. The Elbow Method
Plots Within-Cluster Sum of Squares (Inertia) against increasing values of $k$:
Inertia (WCSS)
│
100┤ \
│ \
50┤ \
│ \ ◄── ELBOW INFLECTION POINT (k = 3!)
20┤ └───┬───┬───┬───► Cluster Count k
0 ┴────1───2───3───4───5
- For $k < 3$: Adding clusters rapidly reduces inertia because points find nearby centroids.
- For $k > 3$: Adding clusters yields diminishing returns (flattening curve).
The Elbow Point ($k=3$) represents the optimal trade off between compactness and model complexity.
Limitation
The elbow is often smooth and ambiguous on real noisy data, making it hard to identify a clear bend.
2. Silhouette Analysis (The Gold Standard)
Silhouette Analysis measures how similar an object is to its own cluster compared to neighboring clusters.
For each sample $i$:
- $a(i)$: Mean distance to all other points in the same cluster (Intra cluster distance).
- $b(i)$: Mean distance to points in the nearest neighboring cluster (Nearest cluster distance).
$$s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$$
$$\text{Range: } -1.0 \le s(i) \le +1.0$$
┌─────────────────────────────────────────────────────────────┐
│ s(i) ≈ +1.0 ──► Sample is well clustered (Far from neighbors!)│
│ s(i) ≈ 0.0 ──► Sample lies directly on cluster boundary │
│ s(i) ≈ -1.0 ──► Sample is assigned to the WRONG cluster! │
└─────────────────────────────────────────────────────────────┘
The overall Silhouette Score for a dataset is the average $s(i)$ across all samples. Select the $k$ value that maximizes the average Silhouette Score.
Say this out loud
Choosing optimal k balances compactness against model complexity. The Elbow Method plots Inertia against k, searching for an inflection point where marginal inertia reduction flattens. Silhouette Analysis measures cluster separation by comparing mean intra cluster distance a(i) to nearest neighbor cluster distance b(i), selecting the k that maximizes average Silhouette Score between minus 1 and plus 1.
Followups to expect
- What is a Silhouette Plot? A visual plot sorting individual silhouette coefficients per cluster as horizontal bars, allowing you to check if all clusters have equal thickness and exceed the average score threshold.
- What is the Davies Bouldin Index? An alternative clustering metric measuring the ratio of intra cluster distance to inter cluster separation. Lower Davies Bouldin scores indicate better clustering performance.
Check yourself
What happens to Within Cluster Sum of Squares (Inertia) as cluster count k increases all the way to N (total sample count)?