Vector Norms (L1, L2, L∞)

What is a Vector Norm?

A Vector Norm is a function that assigns a non-negative length or magnitude to a vector.

To be a valid mathematical norm $|x|$, three properties must hold:

1. Non-negativity: $|x| \ge 0$, and $|x| = 0$ if and only if $x = 0$.
2. Absolute Homogeneity: $|\alpha x| = |\alpha| \cdot |x|$.
3. Triangle Inequality: $|x + y| \le |x| + |y|$.

The L_p Norm Family

The general $L_p$ norm for vector $x = [x_1, x_2, \dots, x_d]^T$:

$$|x|p = \left( \sum{i=1}^d |x_i|^p \right)^{\frac{1}{p}}$$

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. L1 NORM (Manhattan)   │ 2. L2 NORM (Euclidean)   │ 3. L_INF NORM (Max)      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ ||x||_1 = ∑ |x_i|        │ ||x||_2 = √( ∑ x_i² )    │ ||x||_∞ = max |x_i|      │
│ Sum of absolute values.  │ Straight line distance.  │ Maximum absolute value.  │
│ Creates SPARSE weights.  │ Smooth, differentiable.  │ Worst-case error bound.  │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

L1 vs L2 Norm Geometry & Regularization

Consider vector $w = [3, -4]^T$:

• L1 Norm: $|w|_1 = |3| + |-4| = 7$.
• L2 Norm: $|w|_2 = \sqrt{3^2 + (-4)^2} = \sqrt{25} = 5$.
• L_inf Norm: $|w|_\infty = \max(|3|, |-4|) = 4$.

  L1 Constraint Ball (Diamond)                   L2 Constraint Ball (Circle)
           w2                                             w2
           ▲                                              ▲
           │  / \                                         │  .---.
     -1 ───┼─/───\─── 1  w1                         -1 ───┼─/───\─── 1  w1
           │  \ /                                         │  '---'
           ▼                                              ▼
  (Sharp corners on axes -> $w_1=0$!)            (Smooth circle -> $w_i \approx 0$ but not 0)

1. L1 Regularization (Lasso): The diamond-shaped constraint has sharp corners on the coordinate axes. Loss function level sets hit these corners first, forcing uninformative feature weights to exactly 0.0.
2. L2 Regularization (Ridge / Weight Decay): The smooth circular constraint shrinks all weights proportionally toward zero, but rarely sets weights to exactly 0.0.

Machine Learning Applications

1. L1 Norm: Lasso Feature Selection, Sparse Autoencoders, Robust Loss (MAE).
2. L2 Norm: Ridge Regression, Weight Decay in PyTorch, Cosine Similarity normalization.
3. L_inf Norm: Adversarial Machine Learning (FGSM attack budget $\epsilon$ bound).

Say this out loud

Vector norms measure vector length. The L1 norm sums absolute values, forming diamond constraints that drive feature weights to exact zeros for sparse feature selection. The L2 norm measures straight-line Euclidean distance, shrinking weights smoothly in Ridge regression. The L_infinity norm measures the maximum absolute element for worst-case adversarial bounds.

Follow-ups to expect

• What is the Frobenius Norm for matrices? The matrix equivalent of the L2 norm: $|A|F = \sqrt{\sum \sum a{ij}^2} = \sqrt{\text{Tr}(A^T A)}$.
• Is L0 "norm" a real norm? L0 counts non-zero elements. It violates absolute homogeneity ($|\alpha x|_0 = |x|_0 \neq |\alpha| |x|_0$), so it is not a true mathematical norm, though used informally in sparse coding.