Dropout

What is Dropout?

Overfitting happens when a deep neural network memorizes training data by developing fragile dependencies between specific neurons. Neighboring neurons rely on each other to fix mistakes rather than learning independent features.

Dropout breaks these co dependencies by randomly turning off a fraction $p$ of hidden neurons during each forward training step.

  FULL NETWORK (Testing Mode)                  DROPOUT ACTIVE (Training Mode, p = 0.5)
  (All neurons active)                         (Random 50% of neurons zeroed out)

     (O)   (O)   (O)                              (X)   (O)   (X)
    / \   / \   / \                              /       |
  (O)   (O)   (O)   (O)                        (O)   (X)   (O)   (X)
    \ /   \ /   \ /                              \       |
     (O)   (O)   (O)                              (X)   (O)   (X)

By deactivating random neurons:

1. The network cannot rely on any single neuron being present.
2. Every neuron is forced to learn useful features on its own.
3. Training effectively samples and averages thousands of distinct smaller sub networks.

How Inverted Dropout Works

If you drop 50 percent of neurons ($p = 0.5$) during training, the sum of signals reaching the next layer drops by half.

If you turn all neurons back on during testing, the next layer receives double the total signal magnitude, ruining predictions.

To keep signal scale equal without modifying test code, modern frameworks use Inverted Dropout:

During training, whenever a neuron stays active, its output is scaled up by dividing by $(1 - p)$:

$$\text{output} = \begin{cases} \frac{x}{1 - p} & \text{with probability } 1 - p \ 0 & \text{with probability } p \end{cases}$$

Because training scales up active outputs, testing runs with all neurons active and zero extra math!

Common Dropout Rates

1. Hidden Dense Layers: Typically set $p = 0.2$ to $0.5$.
2. Input Layers: Kept low around $p = 0.1$ or $0.0$ to avoid discarding raw input details.
3. Convolutional Layers: Standard dropout is rarely used because adjacent pixels are strongly correlated. Use Spatial Dropout instead (dropping entire feature maps).

Say this out loud

Dropout randomly deactivates a fraction p of hidden neurons during each training step. This breaks neuron co adaptation and forces the network to learn robust redundant features. Inverted Dropout scales active outputs by 1 / (1 - p) during training so that evaluation uses all neurons with zero scaling adjustments.

Followups to expect

1. What is Spatial Dropout in Convolutional Networks? Instead of dropping random individual pixels, Spatial Dropout drops entire feature channels, forcing the network to avoid relying on specific feature maps.
2. What is Monte Carlo Dropout? Keeping dropout active during inference and running multiple forward passes to measure prediction variance, estimating model uncertainty.