Label Smoothing

What is Label Smoothing?

In standard classification tasks, target labels are represented as hard one hot vectors:

  Hard One Hot Target (Class 2):  [ 0.0,  1.0,  0.0,  0.0 ]

To output a probability of 1.0 for Class 2 using Softmax:

$$\text{Softmax}(z_2) = \frac{e^{z_2}}{\sum e^{z_j}} \to 1.0 \implies z_2 - z_j \to +\infty$$

To make probability reach 1.0, the network must push logit $z_2$ to positive infinity relative to other classes.

This forces weight magnitudes to grow endlessly large during training, making the network overconfident and prone to severe overfitting.

How Label Smoothing Works

Label Smoothing softens hard target vectors by blending them with a uniform probability distribution:

  Label Smoothed Target (epsilon = 0.1):  [ 0.025,  0.925,  0.025,  0.025 ]

Formula for smoothed target $y_k$:

$$y_k = (1 - \epsilon) \cdot y_k^{\text{hard}} + \frac{\epsilon}{K}$$

• $\epsilon$: Smoothing hyperparameter (typically $\epsilon = 0.1$).
• $K$: Total number of classes.

Now the network only needs to learn finite logit differences (for example, target probability 0.925 instead of 1.0), preventing weights from growing endlessly.

Key Benefits

1. Prevents Overconfidence: Keeps logits bounded, producing realistic confidence probabilities on test data.
2. Improves Model Calibration: Predictions better reflect true real world accuracy.
3. Better Generalization: Standard trick used when training Vision Transformers, EfficientNets, and Machine Translation models.

When to Avoid Label Smoothing

Do NOT use label smoothing if you plan to use the trained model as a Teacher for Knowledge Distillation.

Label smoothing flattens logit differences among non target classes, stripping away fine grained information about which incorrect classes are most similar to the target.

Say this out loud

Label Smoothing replaces hard one hot targets like 1.0 with softened targets like 0.9, distributing remaining probability uniformly across wrong classes. This prevents Softmax logits from growing infinitely large, stopping the model from becoming overconfident and improving model calibration on unseen test data.

Followups to expect

1. How does Label Smoothing impact clustering in hidden feature space? Muller et al. (2019) showed that label smoothing forces representations of samples in the same class to cluster tightly around equidistant cluster centers.
2. What is Temperature Scaling for Calibration? A post processing technique that adjusts logit scale using a validation temperature parameter T to calibrate confidence probabilities without retraining model weights.