Activation Functions

Why Do We Need Activation Functions?

If you multiply an input by a weight matrix and add a bias, you are performing linear algebra.

If you stack ten linear layers together:

Layer3( Layer2( Layer1( x ) ) ) = Combined Linear Transformation( x )

Without non linear activation functions, no matter how many layers you stack, the entire neural network acts like a single simple linear model.

Non linear activation functions bend and curve the output of each layer, enabling the network to learn complex real world shapes like images, speech, and natural text.

Popular Activation Functions

1. Sigmoid: Squashes numbers into a range between 0 and 1. Great for output layers predicting binary probability, but suffers from vanishing gradients in deep hidden layers because extreme positive or negative numbers produce tiny flat gradients.
2. Tanh: Squashes numbers between minus 1 and plus 1. It is zero centered, making optimization easier than Sigmoid, but still saturates at extreme values.
3. ReLU (Rectified Linear Unit): Returns 0 for negative numbers and returns the input value unchanged for positive numbers. It is extremely fast to compute and keeps gradients strong for positive inputs.
4. Leaky ReLU: Modifies ReLU by allowing a tiny non zero slope for negative numbers, preventing neurons from becoming permanently dead.
5. GELU (Gaussian Error Linear Unit): A smooth curve that weighs inputs by their probability under a normal distribution. Used heavily in modern transformer models like GPT and BERT.

Summary Comparison Table

1. Sigmoid: Output Range 0 to 1. Main use is Binary Classification Output.
2. Tanh: Output Range minus 1 to plus 1. Main use is Recurrent Neural Networks.
3. ReLU: Output Range 0 to infinity. Main use is Deep CNN Hidden Layers.
4. GELU: Output Range minus 0.17 to infinity. Main use is Transformer Language Models.

Say this out loud

Activation functions introduce non linearity, allowing neural networks to learn complex patterns instead of acting like a simple linear equation. Sigmoid outputs probabilities between 0 and 1. Tanh outputs zero centered values between minus 1 and plus 1. ReLU is fast and avoids vanishing gradients for positive inputs, while GELU provides a smooth curve used in modern transformers.

Followups to expect

1. What is the derivative of ReLU? For positive inputs x greater than 0, the derivative is exactly 1. For negative inputs x less than 0, the derivative is 0.
2. Why is zero centered output helpful in hidden layers? Zero centered outputs prevent gradients from systematically shifting in one direction during backpropagation, making optimization updates more balanced.