Implement Logistic Regression from Scratch

Math Foundations

Logistic Regression models binary classification probabilities $P(y = 1 \mid X)$ by passing linear logits $z = X w + b$ through the Sigmoid Activation Function $\sigma(z)$:

$$z = X w + b, \quad \hat{p} = \sigma(z) = \frac{1}{1 + e^{-z}}$$

The objective is to minimize Binary Cross-Entropy Loss (Log Loss):

$$\mathcal{L}(w, b) = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(\hat{p}_i) + (1 - y_i) \log(1 - \hat{p}_i) \right]$$

Remarkably, taking partial derivatives yields the exact same clean gradient update form as linear regression:

$$\frac{\partial \mathcal{L}}{\partial w} = \frac{1}{N} X^T (\hat{p} - y), \quad \frac{\partial \mathcal{L}}{\partial b} = \frac{1}{N} \sum_{i=1}^N (\hat{p}_i - y_i)$$

Gradient Descent updates parameters with learning rate $\alpha$:

$$w \leftarrow w - \alpha \frac{\partial \mathcal{L}}{\partial w}, \quad b \leftarrow b - \alpha \frac{\partial \mathcal{L}}{\partial b}$$

NumPy Implementation from Scratch

import numpy as np


class LogisticRegressionFromScratch:

  def __init__(self, lr=0.01, n_iters=1000):
    self.lr = lr
    self.n_iters = n_iters
    self.w = None
    self.b = None

  def _sigmoid(self, z):
    # Clip z to prevent numerical overflow in exp(-z)
    z_clipped = np.clip(z, -500, 500)
    return 1.0 / (1.0 + np.exp(-z_clipped))

  def fit(self, X, y):
    n_samples, n_features = X.shape
    self.w = np.zeros(n_features)
    self.b = 0.0

    for _ in range(self.n_iters):
      # 1. Forward Pass (Linear Logits -> Sigmoid Probability)
      linear_logits = np.dot(X, self.w) + self.b
      y_pred = self._sigmoid(linear_logits)

      # 2. Compute Gradients
      error = y_pred - y
      dw = (1 / n_samples) * np.dot(X.T, error)
      db = (1 / n_samples) * np.sum(error)

      # 3. Update Parameters
      self.w -= self.lr * dw
      self.b -= self.lr * db

  def predict_proba(self, X):
    linear_logits = np.dot(X, self.w) + self.b
    return self._sigmoid(linear_logits)

  def predict(self, X, threshold=0.5):
    probs = self.predict_proba(X)
    return (probs >= threshold).astype(int)

Numerical Stability Considerations

When computing np.exp(-z) or np.log(y_pred), large values cause floating point overflow (NaN errors):

1. Clip Logits: Use np.clip(z, -500, 500) before exponentiation.
2. Epsilon Offset for Log: Add tiny constant $\epsilon = 10^{-15}$ inside log calculations: np.log(np.clip(probs, 1e-15, 1 - 1e-15)) to prevent $\log(0)$ crashes.

Say this out loud

Implementing Logistic Regression from scratch combines linear matrix products with the Sigmoid activation function to output probabilities between 0 and 1. Minimizing Binary Cross Entropy loss yields gradient updates proportional to prediction error. Numerical stability techniques like logit clipping and log epsilon bounds prevent floating point overflow errors.

Followups to expect

1. How do you extend Logistic Regression to Multi-Class Classification? Use Softmax activation instead of Sigmoid, replacing Binary Cross Entropy with Categorical Cross Entropy loss (Multinomial Logistic Regression).
2. Why doesn't Logistic Regression have a closed form solution like Linear Regression? Because the non-linear Sigmoid function inside the Log Loss derivative makes setting $\frac{\partial \mathcal{L}}{\partial w} = 0$ non-linear, requiring iterative Gradient Descent or Newton Raphson optimization.