Choosing a Loss Function

What is a Loss Function?

A Loss Function measures how far off your model predictions are from real ground truth labels.

Optimization algorithms use the loss value to compute gradients during backpropagation, adjusting weights to drive loss as close to zero as possible.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. REGRESSION LOSSES     │ 2. CLASSIFICATION LOSSES│ 3. HYBRID LOSSES         │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ MSE: Mean Squared Error  │ Binary Cross Entropy     │ Huber Loss               │
│ MAE: Mean Absolute Error │ Categorical Cross Entropy│ Focal Loss               │
│ Continuous price/height  │ Class probabilities      │ Robust to outliers       │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Regression Losses (Continuous Numbers)

Mean Squared Error (MSE / L2 Loss)

MSE = Mean( ( y_true - y_pred )^2 )

• Pros: Smooth and differentiable everywhere, making gradient descent easy.
• Cons: Squaring errors heavily penalizes large mistakes. A few extreme outliers can completely corrupt the model.

Mean Absolute Error (MAE / L1 Loss)

MAE = Mean( | y_true - y_pred | )

• Pros: Robust against outliers because error penalties grow linearly instead of quadratically.
• Cons: Gradient is constant everywhere, causing slow convergence near zero.

Huber Loss (Smooth L1)

Combines the best of MSE and MAE. It uses quadratic MSE for small errors, and switches to linear MAE for large errors beyond a threshold delta.

2. Classification Losses (Probabilities)

Binary Cross Entropy (BCE)

Used when predicting binary outcomes (Yes/No, 0/1):

Loss = - [ y * log(p) + (1 - y) * log(1 - p) ]

If true label is 1 and model predicts probability near 1, loss is near 0. If model predicts probability near 0, loss explodes to infinity.

Categorical Cross Entropy (CE)

Used for multi class decisions where each sample belongs to exactly one category:

Loss = - sum( y_true * log(y_pred) )

Directly measures the distance between predicted Softmax probability distributions and one hot true labels.

Say this out loud

Loss functions measure prediction errors to guide gradient descent updates. For continuous regression, MSE penalizes large outliers heavily while MAE is robust against outliers. For classification, Cross Entropy compares predicted probabilities against true labels, penalizing confident wrong predictions heavily. Huber loss combines MSE for small errors and MAE for large errors.

Followups to expect

1. What is Focal Loss? A classification loss that down weights easy background examples, focusing training attention on hard misclassified samples.
2. What is Contrastive Loss? A loss function used in metric learning that pulls positive pair embeddings closer together while pushing negative pair embeddings apart.