Covariance vs Correlation
Covariance tells you direction; correlation tells you direction and strength independent of scale.
Defining Covariance vs Correlation
For samples X and Y:
Covariance: Cov(X, Y) = 1/(N-1) ∑ (x_i - x̄)(y_i - ȳ)
Correlation: r(X, Y) = Cov(X, Y) / ( s_X · s_Y )
Unbounded Covariance (-∞, +∞) ──Scale Normalization (s_X s_Y)──► Bounded Correlation [-1, +1]
- Covariance: Units are
Unit(X) × Unit(Y). Unbounded. Sensitive to metric scales. - Correlation (Pearson): Dimensionless number in
[-1.0, +1.0]. Invariant to linear scalingaX + b.
Pearson vs Spearman Rank Correlation
| Metric | Formula | Relationship Assessed | Outlier Sensitivity |
|---|---|---|---|
Pearson r | Cov(X,Y) / (σ_X σ_Y) | Pure Linear (y = ax + b) | High (Outliers pull the line) |
Spearman r_s | Pearson applied to Rank(X) and Rank(Y) | General Monotonic (y increases as x increases) | Low (Ranks bound outlier impact) |
The Covariance Matrix (Σ)
For a feature matrix X ∈ ℝᴺˣᵈ with zero-centered columns:
Σ = 1/(N-1) Xᵀ X = [ Var(X₁) Cov(X₁,X₂) ... Cov(X₁,X_d) ]
[ Cov(X₂,X₁) Var(X₂) ... Cov(X₂,X_d) ]
The diagonal entries are feature variances Var(X_i), and off-diagonal entries are pairwise covariances. The Covariance Matrix is always symmetric and positive semi-definite.
Say this out loud
"Covariance measures joint directional variation between two variables, but its magnitude depends on feature units. Pearson correlation normalizes covariance by dividing by standard deviations, yielding a scale-invariant measure of linear association from -1 to +1. Pearson misses non-linear relationships like Y = X², so for non-linear monotonic trends or heavy outliers, we use Spearman rank correlation."
Follow-ups to expect
- What is multicollinearity? High pairwise correlation between independent features (r > 0.8), causing high variance in OLS linear regression weight estimates.
- How does covariance matrix relate to Mahalanobis Distance? Mahalanobis distance
d = √((x - μ)ᵀ Σ⁻¹ (x - μ))measures distance in standard deviations, accounting for feature correlations and scale differences. - Is zero correlation equivalent to independence? Independence implies zero correlation, but zero correlation does NOT imply independence (e.g. non-linear relationship Y = X² has r = 0).
Check yourself
If feature X is height in meters and Y is weight in kg, what happens to Cov(X,Y) and Pearson Correlation r if height is converted to centimeters (multiplied by 100)?