Computational Graphs & Autograd

What is a Computational Graph?

Deep neural networks consist of millions of nested mathematical operations:

$$L = \text{Loss}(f(W_2 \cdot \sigma(W_1 x + b_1) + b_2), y)$$

To compute gradient derivatives $\frac{\partial L}{\partial W}$ during backpropagation, deep learning frameworks construct a Computational Graph.

A Computational Graph is a Directed Acyclic Graph (DAG) where:

• Leaf Nodes: Input Tensors and Trainable Model Parameters ($x, W, b, y$).
• Interior Nodes: Mathematical Operator Nodes ($+, \times, \text{ReLU}, \text{CrossEntropy}$).
• Edges: Direction of data tensor flow.

       [ Input x ]    [ Weight W ]
            │              │
            └───────┬──────┘
                    ▼
              [ MatMul (*) ] ──► Node a
                    │
                    ▼
              [ Add (+) ] ◄── [ Bias b ]
                    │
                    ▼
              [ ReLU ] ──► Output Activation y_hat

Static vs Dynamic Computational Graphs

┌──────────────────────────┬──────────────────────────┐
│ 1. STATIC GRAPHS (TF 1.x)│ 2. DYNAMIC GRAPHS (PyTorch)
├──────────────────────────┼──────────────────────────┤
│ Define Graph ONCE ──►    │ Define-by-Run! Graph is  │
│ Compile ──► Execute.     │ constructed dynamically  │
│ Fast C++ optimization,   │ on the fly during every  │
│ but hard to debug!       │ forward pass. Pythonic!  │
└──────────────────────────┴──────────────────────────┘

PyTorch uses Dynamic Computational Graphs (Define-by-Run):

1. Every time a forward pass executes, PyTorch records created operations in a fresh DAG.
2. Supports native Python if statements, for loops, and dynamic sequence lengths seamlessly!

Reverse-Mode Automatic Differentiation (Autograd)

Why is Autograd superior to Symbolic or Numerical Differentiation?

• Numerical Differentiation (Finite Differences): $\frac{f(x+h) - f(x)}{h}$. Requires $N$ forward passes for $N$ parameters. Terribly slow!
• Symbolic Differentiation: Algebraic expansion. Causes formula explosion!
• Reverse-Mode Autograd: Computes partial derivatives for ALL $N$ parameters in a single backward pass!

How Reverse-Mode Autograd Works

During the forward pass, PyTorch saves intermediate activation values.

During loss.backward():

1. Starts at the scalar Loss $L$ ($\frac{\partial L}{\partial L} = 1.0$).
2. Traverses the graph backwards, applying the Chain Rule at each operator node:

$$\frac{\partial L}{\partial x_i} = \sum_{j \in \text{Children}(i)} \frac{\partial L}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_i}$$

3. Accumulates gradients directly into param.grad tensors.

  FORWARD PASS:   Inputs / Weights ──► Matrix Mult ──► ReLU ──► Loss  (Record Operations!)
  BACKWARD PASS:  Loss Derivative ◄── Matrix Mult ◄── ReLU ◄── Loss  (Compute Chain Rule!)

Managing Autograd Graph Memory in PyTorch

# 1. Disable graph construction during inference to save GPU memory:
with torch.no_grad():
    predictions = model(inputs)

# 2. Reset accumulated gradients before each optimizer step:
optimizer.zero_grad() # Prevents gradient accumulation across batches!

Say this out loud

Computational Graphs represent neural network operations as Directed Acyclic Graphs of tensors and operators. PyTorch builds Dynamic Computational Graphs on the fly during forward passes. Reverse Mode Autograd applies the Chain Rule backward from scalar Loss to compute gradient derivatives for all model parameters in a single backward pass.

Followups to expect

1. What is Forward Mode Automatic Differentiation? Computes directional derivatives along with forward evaluations. Efficient when output dimension is much larger than input dimension ($d_{\text{out}} \gg d_{\text{in}}$).
2. What is tensor.detach()? Returns a new tensor detached from the active computational graph, stopping backward gradient flow through that specific tensor branch.