Implement Backprop for an MLP

Math Foundations

Consider a 2-Layer Multi-Layer Perceptron (MLP) for binary classification:

$$\text{Layer 1: } Z_1 = X W_1 + b_1, \quad A_1 = \text{ReLU}(Z_1)$$

$$\text{Layer 2: } Z_2 = A_1 W_2 + b_2, \quad A_2 = \sigma(Z_2) = \hat{y}$$

The objective is to compute loss gradients with respect to weights $W_1, b_1, W_2, b_2$ using the Chain Rule:

Forward Pass:  X ──► [ W1, b1 ] ──► Z1 ──► [ ReLU ] ──► A1 ──► [ W2, b2 ] ──► Z2 ──► [ Sigmoid ] ──► A2 (y_hat) ──► Loss
Backward Pass: dW1, db1 ◄── dZ1 ◄── dA1 ◄───────────────── dW2, db2 ◄── dZ2 ◄────────────────────── dLoss

Step by Step Derivative Equations

1. Output Error Gradient ($dZ_2$):

$$dZ_2 = A_2 - y$$

2. Layer 2 Parameter Gradients:

$$dW_2 = \frac{1}{N} A_1^T dZ_2, \quad db_2 = \frac{1}{N} \sum dZ_2$$

3. Propagate Error to Hidden Layer ($dA_1$ and $dZ_1$):

$$dA_1 = dZ_2 W_2^T, \quad dZ_1 = dA_1 \odot \text{ReLU}'(Z_1)$$

Where $\text{ReLU}'(Z_1) = 1$ if $Z_1 > 0$ else $0$.

4. Layer 1 Parameter Gradients:

$$dW_1 = \frac{1}{N} X^T dZ_1, \quad db_1 = \frac{1}{N} \sum dZ_1$$

NumPy Implementation from Scratch

import numpy as np


class MLPFromScratch:

  def __init__(self, input_dim, hidden_dim, output_dim=1, lr=0.1):
    self.lr = lr
    # He Weight Initialization for ReLU
    self.W1 = np.random.randn(input_dim, hidden_dim) * np.sqrt(2.0 / input_dim)
    self.b1 = np.zeros((1, hidden_dim))
    self.W2 = np.random.randn(hidden_dim, output_dim) * np.sqrt(
        2.0 / hidden_dim
    )
    self.b2 = np.zeros((1, output_dim))

  def _relu(self, z):
    return np.maximum(0, z)

  def _relu_derivative(self, z):
    return (z > 0).astype(float)

  def _sigmoid(self, z):
    return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))

  def forward(self, X):
    self.X = X
    self.Z1 = np.dot(X, self.W1) + self.b1
    self.A1 = self._relu(self.Z1)
    self.Z2 = np.dot(self.A1, self.W2) + self.b2
    self.A2 = self._sigmoid(self.Z2)
    return self.A2

  def backward(self, y):
    N = self.X.shape[0]
    y = y.reshape(-1, 1)

    # 1. Output Layer Gradient (Binary Cross Entropy + Sigmoid derivative)
    dZ2 = self.A2 - y
    dW2 = (1 / N) * np.dot(self.A1.T, dZ2)
    db2 = (1 / N) * np.sum(dZ2, axis=0, keepdims=True)

    # 2. Propagate Error Back to Hidden Layer
    dA1 = np.dot(dZ2, self.W2.T)
    dZ1 = dA1 * self._relu_derivative(self.Z1)
    dW1 = (1 / N) * np.dot(self.X.T, dZ1)
    db1 = (1 / N) * np.sum(dZ1, axis=0, keepdims=True)

    # 3. Update Parameters via Gradient Descent
    self.W2 -= self.lr * dW2
    self.b2 -= self.lr * db2
    self.W1 -= self.lr * dW1
    self.b1 -= self.lr * db1

Say this out loud

Backpropagation uses the chain rule to compute loss gradients backwards from output layers to early weights. Forward pass computes hidden activations and predictions. Backward pass calculates layer error terms, multiplies by activation derivatives, and updates weight matrices using matrix products of previous activations and incoming errors.

Followups to expect

1. What is Vanishing Gradient in deep MLPs? When using Sigmoid activations in deep networks, multiplying small derivatives repeatedly causes early layer gradients to shrink to near zero, preventing early layers from learning.
2. Why is He Weight Initialization preferred for ReLU networks? Scaling initial weights by $\sqrt{2 / D_{\text{in}}}$ preserves activation variance across layers, preventing exploding or vanishing activations during initial passes.