Bias–Variance Tradeoff

Understanding Model Errors

Every error a machine learning model makes can be split into three simple parts.

1. Bias: This is error caused by overly simple guesses. Imagine trying to fit a straight line through a curving set of points. The model is just too simple to see the curve.
2. Variance: This is error caused by being overly sensitive to specific training details. The model memorizes random quirks in the data instead of learning the real underlying rule.
3. Irreducible Noise: This is natural random noise in the real world. No model can remove this noise.

A helpful way to picture this is a target board. Bias means your shots are systematically off to one side. Variance means your shots are scattered wildly all over the board.

How to Read Your Errors in Real Life

You can easily diagnose your model by comparing training error with validation error.

1. High training error and high validation error: Your model is too simple. This is underfitting or high bias. You should try a bigger model, add better features, or train longer.
2. Low training error but much higher validation error: Your model memorized the training data. This is overfitting or high variance. You should collect more data, simplify the model, or use regularization.
3. Low training error and low validation error, but still missing business targets: The target might be unrealistically low, or the dataset contains too much irreducible noise.

How to Fix Bias and Variance

Ways to lower variance

Getting more training data is the best option because it lowers variance without harming bias. You can also simplify the model, add regularization like L1 or L2 penalties, or use early stopping.

Ways to lower bias

Make the model bigger, add more informative features, or reduce regularization penalties so the model has enough freedom to learn complex patterns.

What Candidates Need to Know

Classic textbooks teach that making a model bigger always increases variance. However, modern deep neural networks often break this rule. When you make a massive neural network very large, test error often goes down again in a pattern called double descent. Heavy neural networks trained with stochastic gradient descent often find simple, smooth solutions even when they are large enough to memorize everything.

Say this out loud

I always check the gap between training error and validation error first. A small gap with high error means high bias, so I would make the model bigger or add features. A large gap means high variance, so I would gather more data or add regularization. I also compare against human baseline performance so I know if I am fighting natural data noise.

Followups to expect

1. Does bagging reduce bias or variance? Bagging reduces variance by averaging predictions across multiple models. Boosting reduces bias by building models step by step to fix previous mistakes.
2. Why does cross validation give better error estimates? It splits data into multiple groups and averages the test results across all splits, reducing estimation noise.
3. Is high bias always bad? Not always. On very tiny datasets, a simple model with high bias often outperforms a flexible model that overfits quickly.