Weight Initialization (Xavier/He)

Why Starting Weights Matter

Before a neural network learns anything from data, it must start with initial weight values.

If you choose starting weights poorly:

1. Activations shrink to zero (vanishing signals).
2. Activations explode to infinity (exploding signals).
3. Neurons learn duplicate features (symmetry problem).

Proper initialization keeps activation variance stable from layer to layer so information flows smoothly during step one of training.

The Zero Initialization Mistake

What if you set every starting weight to zero?

Every neuron in a hidden layer receives identical inputs, computes the exact same output, and gets the exact same gradient update during backpropagation.

Even after 1000 training steps, every neuron in that layer remains identical to its neighbors. The hidden layer acts like a single neuron, destroying the network capacity.

Rule: Always initialize weights randomly to break symmetry.

Two Classic Scaling Solutions

To keep layer signals from shrinking or exploding, random weights must scale inversely with the number of input connections (fan in) and output connections (fan out).

┌──────────────────────────┬──────────────────────────┐
│ 1. XAVIER / GLOROT       │ 2. HE / KAIMING          │
├──────────────────────────┼──────────────────────────┤
│ For Tanh and Sigmoid     │ For ReLU and Leaky ReLU  │
│ Variance = 2 / (in + out)│ Variance = 2 / in        │
│ Preserves variance when  │ Accounts for ReLU        │
│ inputs are symmetric     │ zeroing out half of      │
│ around zero              │ inputs                   │
└──────────────────────────┴──────────────────────────┘

1. Xavier Initialization (Glorot et al., 2010)

Designed for symmetric activations like Tanh and Sigmoid.

Variance = 2 / ( fan_in + fan_out )

This keeps activation variance constant as signals move forward and gradients move backward.

2. He Kaiming Initialization (He et al., 2015)

Designed specifically for ReLU activations.

Because ReLU sets all negative inputs to zero, it effectively drops half of the incoming signals at each layer.

Variance = 2 / fan_in

Multiplying by 2 compensates for the zeroed out half of the inputs, maintaining stable variance through deep ReLU networks.

Say this out loud

Weight initialization sets starting values before training. Initializing weights to zero fails because all neurons receive identical updates due to symmetry. Xavier initialization scales variance for Tanh and Sigmoid activations. He initialization doubles variance scaling to account for ReLU setting half of all inputs to zero, keeping signal variance stable through deep networks.

Followups to expect

1. How should bias vectors be initialized? Bias vectors are usually initialized to zero because random weights already break symmetry across neurons.
2. What is orthogonal initialization? Initializing weight matrices as orthogonal matrices so multiplying by weights preserves vector length, used commonly in Recurrent Networks.