Gradients, Jacobians & Hessians

First and Second Derivatives in Multivariable Calculus

In single-variable calculus, we have $f'(x)$ (slope) and $f''(x)$ (curvature).

Multivariable calculus extends these concepts to vectors and matrices:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  1. GRADIENT ∇f(x)       │  2. JACOBIAN MATRIX J    │  3. HESSIAN MATRIX H     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Function: f: R^n -> R    │ Function: f: R^n -> R^m  │ Function: f: R^n -> R    │
│ First derivatives        │ First derivatives        │ Second derivatives       │
│ Dimensions: n × 1 Vector │ Dimensions: m × n Matrix │ Dimensions: n × n Matrix │
│ Direction of steepest ascent| Maps input to output change| Measures surface curvature|
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. The Gradient Vector $\nabla f(x)$

For a scalar loss function $f(x_1, x_2, \dots, x_n)$:

$$\nabla f(x) = \left[ \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \dots, \frac{\partial f}{\partial x_n} \right]^T$$

• Direction: Points in the direction of steepest local increase of loss $f$.
• Gradient Descent Update: Move in the opposite direction: $x \leftarrow x - \alpha \nabla f(x)$.

2. The Jacobian Matrix $J$

When a function takes $n$ inputs and outputs $m$ values ($f: \mathbb{R}^n \to \mathbb{R}^m$):

$$J_{ij} = \frac{\partial f_i}{\partial x_j} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}_{m \times n}$$

Backpropagation Chain Rule

For composed layers $y = f(g(x))$:

$$J_{y, x} = J_{y, g} \cdot J_{g, x} \quad \text{(Matrix Multiplication of Jacobians!)}$$

3. The Hessian Matrix $H$

Contains all pairwise second-order partial derivatives of scalar loss $f(x)$:

$$H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j} = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \dots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial^2 f}{\partial x_n \partial x_1} & \dots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix}_{n \times n}$$

By Schwarz's Theorem, if functions are smooth, $H$ is a Symmetric Matrix ($H_{ij} = H_{ji}$).

Curvature Interpretation

• Eigenvalues of $H$ measure curvature along principal directions.
• Large positive eigenvalue -> Steep upward ravine walls.
• Near-zero eigenvalue -> Flat plateau.
• Mixed positive and negative eigenvalues -> Saddle Point.

Say this out loud

The Gradient is an n x 1 vector of first derivatives pointing in the direction of steepest ascent for a scalar loss. The Jacobian is an m x n matrix of first derivatives for vector-valued layer outputs, underpinning the chain rule in backpropagation. The Hessian is an n x n symmetric matrix of second derivatives measuring local loss surface curvature and saddle points.

Follow-ups to expect

• What is the Condition Number of a Hessian matrix? The ratio of its maximum eigenvalue to minimum eigenvalue: condition number = lambda_max / lambda_min. A high condition number indicates ill-conditioned, elongated loss ravines where standard gradient descent bounces wildly.
• What are Quasi-Newton methods (BFGS / L-BFGS)? Algorithms that approximate inverse Hessian step H^-1 using past gradient updates without explicitly building or inverting the full n x n Hessian matrix.