Scaling & Normalization
Standardizing numerical feature scales to prevent large magnitude variables from dominating model gradient optimization.
Why Scale Numerical Features?
Consider a dataset predicting loan approval with two continuous numeric features:
- Age: Ranges from $18$ to $80$ years.
- Annual Income: Ranges from $$20,000$ to $$500,000$.
If you compute Euclidean Distance between two applicants:
$$d = \sqrt{(\text{Age}_1 - \text{Age}_2)^2 + (\text{Income}_1 - \text{Income}_2)^2}$$
The Income difference ($100,000^2 = 10,000,000,000$) completely swamps the Age difference ($30^2 = 900$). The model effectively ignores Age!
Additionally, in Gradient Descent, unscaled features create elongated oval loss contours, causing optimization paths to oscillate slowly back and forth.
Unscaled Loss Contours: Elongated ovals ──► Gradient descent oscillates slowly.
Scaled Loss Contours: Spherical circles ──► Gradient descent steps directly to minimum!
Common Scaling Methods
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. STANDARDIZATION │ 2. MIN-MAX SCALING │ 3. ROBUST SCALER │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Z-Score Normalization. │ Rescales features into │ Uses Median and IQR. │
│ Mean = 0, Std Dev = 1. │ fixed range [0.0, 1.0]. │ Robust to extreme data │
│ Best for Gaussian data! │ Preserves zero values! │ outliers! │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
1. Standardization (Z-Score)
$$x_{\text{std}} = \frac{x - \mu}{\sigma}$$
Subtracts the sample mean $\mu$ and divides by standard deviation $\sigma$. Resulting distribution has $\text{Mean} = 0$ and $\text{Std} = 1$.
2. Min-Max Normalization
$$x_{\text{norm}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$
Rescales data into a strict bounded interval $[0.0, 1.0]$. Sensitive to extreme outliers (a single huge outlier squashes all normal values into a tiny range).
3. Robust Scaling (IQR)
$$x_{\text{robust}} = \frac{x - \text{Q2}}{\text{Q3} - \text{Q1}}$$
Subtracts the median (Q2) and divides by Interquartile Range (IQR = Q3 - Q1). Ideal when features contain extreme outliers.
Algorithm Sensitivity Matrix
| Algorithm Family | Sensitive to Feature Scaling? | Reason |
|---|---|---|
| Distance-based (k-NN, k-Means, SVM) | YES (Critical) | Large features dominate distance math |
| Gradient-based (Neural Nets, Logistic Reg) | YES (Critical) | Fast gradient convergence |
| Tree-based (Decision Trees, XGBoost) | NO (Invariant) | Splits depend on feature rank order |
Critical Rule: Avoid Data Leakage During Scaling
Always fit scaling parameters ($\mu, \sigma, x_{\min}, x_{\max}$) ONLY on the Training Set:
$$\text{Correct: } \text{scaler.fit}(\text{X_train}) \to \text{scaler.transform}(\text{X_train}) \to \text{scaler.transform}(\text{X_test})$$
Fitting scaling parameters on the full dataset leaks test set statistics into training data!
Say this out loud
Feature scaling transforms numerical variables onto comparable scales to prevent large range features from dominating distance metrics or gradient optimization. Standardization transforms data to zero mean and unit variance. Min-Max scaling rescales data into a 0 to 1 interval. Tree based models are invariant to scaling, but neural networks and distance algorithms require proper scaling to function.
Followups to expect
- What is Log Transformation? Applying $\log(x + 1)$ to skewed positive distribution features (like income or transaction amounts) to convert exponential right skewed data into a symmetric Gaussian distribution.
- What is L2 Normalization (Unit Vector Scaling)? Rescaling individual sample feature vectors to have unit Euclidean norm $|x|_2 = 1$, commonly used in text TF-IDF and embedding vectors.
Check yourself
Why do distance based algorithms like k-Nearest Neighbors and SVM require feature scaling?