Logistic Regression

Log-odds and the Sigmoid function

Linear regression outputs unbounded continuous numbers (-∞, ∞). Logistic regression wraps the linear combination in the Sigmoid (logistic) function to constrain outputs to (0, 1):

σ(z) = 1 / (1 + e⁻ᶻ),   where z = wᵀx + b

Derivative of Sigmoid has a clean closed form: σ'(z) = σ(z)(1 - σ(z)).

The Log-Odds interpretation

The ratio of probability of occurrence to non-occurrence is the Odds:

Odds = P(y=1|x) / P(y=0|x) = P / (1 - P) = e^(wᵀx + b)

Taking the natural log gives Log-Odds (Logit):

ln( Odds ) = ln( P / (1 - P) ) = wᵀx + b

Logistic regression is simply linear regression on the log-odds of the positive class.

Loss function: Binary Cross-Entropy (Log Loss)

Under Maximum Likelihood Estimation (MLE), assuming independent Bernoulli trials:

L(w) = - 1/N ∑ [ y_i · log(p_i) + (1 - y_i) · log(1 - p_i) ]

where p_i = σ(wᵀx_i + b).

Gradient of Log Loss

Taking the partial derivative with respect to weight w_j:

∂L / ∂w_j = 1/N ∑ (p_i - y_i) x_ij

Notice how clean this gradient is — the error term (p_i - y_i) multiplied by feature x_ij. This is identical in form to linear regression MSE gradient, making gradient descent updates fast and intuitive.

Why no closed-form solution?

Equating ∂L / ∂w_j = 0 produces a system of transcendental non-linear equations due to σ(z). There is no closed-form algebraic matrix inverse like OLS β = (XᵀX)⁻¹Xᵀy.

Instead, we use iterative optimization algorithms:

• Gradient Descent / Adam: Suitable for large-scale datasets.
• L-BFGS / Newton-Raphson (IRLS): Second-order optimization methods that converge faster in few iterations on small to medium datasets.

Say this out loud

"Logistic regression models the log-odds of a positive outcome as a linear combination of features. We pass the linear logit through a Sigmoid activation to get calibrated probabilities. Loss is optimized via Binary Cross-Entropy, which is strictly convex, guaranteeing convergence to a global minimum via gradient descent. Weight w_i represents the additive change in log-odds (or multiplicative change in odds via e^w_i) per unit feature change."

Follow-ups to expect

• How do you extend Logistic Regression to multiclass? Use Multinomial Logistic Regression (Softmax regression) with Cross-Entropy loss, or One-vs-Rest (OvR) binary classifiers.
• What happens when features are perfectly linearly separable? Weights will diverge to infinity attempting to push predicted probabilities to exactly 0 and 1. Fix this by applying L2 regularization.
• Is Logistic Regression sensitive to outliers? Less sensitive than Linear Regression with MSE, but extreme outliers can distort the decision boundary. Robust scaling or clipping helps.