Taylor Expansions in Optimization

What is a Taylor Series?

A Taylor Series approximates any smooth non-linear function near point $x_0$ using a sum of polynomial terms constructed from derivatives:

$$f(x_0 + \Delta x) = f(x_0) + f'(x_0) \Delta x + \frac{1}{2!} f''(x_0) \Delta x^2 + \frac{1}{3!} f'''(x_0) \Delta x^3 + \dots$$

   Non-linear Loss Function f(x) vs Taylor Approximations
   Loss
    High ┤            /  Actual Non-linear Loss f(x)
         │           /  / 2nd-order Quadratic Approx (Parabola)
         │          /  / /
         │         /  / /  1st-order Linear Approx (Tangent Line)
         │        /  / /  /
     Low ┴───────(  *  )─────────────────► Parameter x
                 x_0 (Expansion Point)

First-Order Approximation (Gradient Descent)

Truncating the series after the first derivative term:

$$f(x_0 + \Delta x) \approx f(x_0) + \nabla f(x_0)^T \Delta x$$

To make loss $f(x_0 + \Delta x) < f(x_0)$, choose displacement $\Delta x = -\alpha \nabla f(x_0)$:

$$f(x_0 - \alpha \nabla f) \approx f(x_0) - \alpha |\nabla f(x_0)|^2 < f(x_0) \quad (\alpha > 0)$$

This proves mathematically why moving in the opposite direction of the gradient guarantees local loss reduction!

Second-Order Approximation (Newton's Method)

Including the second derivative (Hessian matrix $H$):

$$f(x_0 + \Delta x) \approx f(x_0) + \nabla f(x_0)^T \Delta x + \frac{1}{2} \Delta x^T H(x_0) \Delta x$$

To find the step $\Delta x$ that minimizes this quadratic approximation, set derivative wrt $\Delta x$ to 0:

$$\frac{\partial}{\partial \Delta x} \left[ f(x_0) + \nabla f(x_0)^T \Delta x + \frac{1}{2} \Delta x^T H(x_0) \Delta x \right] = \nabla f(x_0) + H(x_0) \Delta x = 0$$

$$\mathbf{\Delta x_{\text{Newton}} = -H(x_0)^{-1} \nabla f(x_0)}$$

Newton's Method uses curvature $H$ to jump directly to the minimum of the quadratic approximation in a single step!

Why Deep Learning Prefers 1st-Order

1. 1st-Order (SGD / Adam): Requires $O(N)$ memory and compute per step.
2. 2nd-Order (Newton): Requires storing $O(N^2)$ Hessian matrix and computing $O(N^3)$ inverse $H^{-1}$. For $N = 100\text{M}$ weights, $H^{-1}$ is computationally impossible.

Say this out loud

Taylor series expands complex functions locally using derivatives. 1st-order expansion f(x) ≈ f(x_0) + ∇f^T Δx proves why setting Δx = -α ∇f guarantees local loss reduction in Gradient Descent. 2nd-order expansion adds Hessian curvature (1/2) Δx^T H Δx, deriving Newton's method step Δx = -H^-1 ∇f for single-step quadratic minimization.

Follow-ups to expect

• What is the Trust Region in 1st-order optimization? The local hyper-sphere radius ||Δx|| <= r where 1st-order linear Taylor approximation remains accurate. Learning rate alpha controls step size within the trust region.
• How does XGBoost use Taylor Expansion? XGBoost computes 2nd-order Taylor expansions of arbitrary loss functions, deriving optimal leaf split scores directly using gradient g_i and hessian h_i values per sample.