Overfitting vs Underfitting

The two failure modes

Underfitting — the model is too constrained to represent the pattern. High train error, validation error right next to it. The model has not even learned the training data.

Overfitting — the model has enough capacity to memorise idiosyncrasies of the sample. Train error near zero, validation error much higher. It learned the noise.

The gap between the curves is the tell. Absolute level tells you about bias; the gap tells you about variance.

Learning curves: the diagnostic tool

Plot error against training set size (not just epochs):

• Both curves converge to a high value → underfitting. More data is useless. Add capacity.
• Curves are converging but still far apart → overfitting, and more data will genuinely help.
• Validation curve has flattened well above train → more data has stopped paying; regularize or change the model.

Plot error against epochs to find the early-stopping point: the epoch where validation loss bottoms out before turning up.

Fixes that actually work, in order

For overfitting:

1. More/better data — including augmentation (crops, flips, paraphrase, noise). Highest leverage.
2. Early stopping — free, and always worth wiring up. Checkpoint on best validation metric.
3. Regularization — L2/weight decay, L1 for sparsity, dropout for nets.
4. Reduce capacity — fewer layers/units, shallower trees, fewer features.
5. Ensembling — bagging averages away variance.
6. Cross-validation — does not fix overfitting, but stops you fooling yourself about it.

For underfitting:

1. More capacity (depth/width/trees).
2. Better features — often bigger than model choice.
3. Train longer, raise learning rate, check the optimizer is actually converging.
4. Reduce regularization.

The subtle version interviewers probe

You can overfit the validation set too. Tune hyperparameters against the same validation split 200 times and its error estimate becomes optimistic — you have leaked information through your own decisions. The defence is a held-out test set you touch once, or nested cross-validation.

Similarly, "my model overfits" is sometimes actually leakage: a feature that encodes the target (e.g. account_closed_date for a churn model). The tell is validation performance that looks too good, then collapses in production. Always ask "could this feature have been created after the label?"

Say this out loud

"I'd plot learning curves against both epochs and dataset size. Curves converging high means bias, so I add capacity. A persistent gap means variance, so I add data, augmentation or regularization, and early-stop. I'd also keep a test set I only touch once, because repeatedly tuning on validation overfits it."