Support Vector Machines

What is a Support Vector Machine?

For a linearly separable dataset, many possible decision boundary lines can separate two classes.

Which line is mathematically the best choice?

  Line A: Separates classes, but passes dangerously close to Class 1 points.
  Line B: Separates classes, but passes dangerously close to Class 2 points.
  SVM Hyperplane: Passes directly through the middle, MAXIMIZING MARGIN DISTANCE!

Support Vector Machines (SVM) find the unique hyperplane that maximizes the Margin Distance between the boundary and the closest data points of any class.

  Class 1 (+)
       +       +    ( Margin Boundary: w^T x + b = +1 )
       ─────o─────  ◄── Support Vector!
            │
            │   MAXIMUM MARGIN WIDTH = 2 / ||w||
            │
       ─────o─────  ◄── Support Vector!
       -       -    ( Margin Boundary: w^T x + b = -1 )
  Class 2 (-)

Support Vectors

The data points that lie directly on the margin boundaries are called Support Vectors.

If you delete all other training points far away from the boundary, the SVM decision boundary remains 100 percent identical! The entire model depends exclusively on Support Vectors.

Hard Margin vs Soft Margin SVM

┌──────────────────────────┬──────────────────────────┐
│ 1. HARD MARGIN SVM       │ 2. SOFT MARGIN SVM       │
├──────────────────────────┼──────────────────────────┤
│ Assumes data is 100%     │ Allows some samples to   │
│ linearly separable.      │ cross margin boundaries  │
│ Sensitive to noise/outliers. using Slack Variables ξ.│
│ Fails on overlapping data. Controlled by C hyperparam.│
└──────────────────────────┴──────────────────────────┘

Soft Margin Formulation

To handle real-world noisy data, we introduce Slack Variables ($\xi_i \ge 0$):

$$\min_{w, b, \xi} \frac{1}{2} |w|^2 + C \sum_{i=1}^N \xi_i$$

Subject to constraint:

$$y_i (w^T x_i + b) \ge 1 - \xi_i, \quad \xi_i \ge 0$$

• $\frac{1}{2} |w|^2$: Maximizes margin width (since margin $= \frac{2}{|w|}$).
• $C$: Hyperparameter controlling misclassification penalty strength.

The Role of Hyperparameter C

• Large C: High penalty for misclassifications. Model builds a narrow margin, risking overfitting.
• Small C: Low penalty for misclassifications. Model builds a wide margin, allowing some training errors for better generalization.

Say this out loud

Support Vector Machines find the optimal hyperplane that maximizes the geometric margin distance to the closest data points, called Support Vectors. Soft Margin SVM uses slack variables and hyperparameter C to balance margin width against classification errors. Large C forces narrow margins with zero training errors, while small C creates wider margins with greater error tolerance.

Followups to expect

1. Why does SVM require Feature Scaling (StandardScaler)? Margin distance is calculated using Euclidean distance $|x_1 - x_2|$. Features with large scales dominate distance calculations, distorting margin boundaries.
2. What is Hinge Loss? The loss function used in linear SVM: $L(y, f(x)) = \max(0, 1 - y \cdot f(x))$. If a point is correctly classified beyond margin boundary ($y f(x) \ge 1$), loss is exact zero.