Advantage Estimation & GAE

Balancing bias and variance in policy gradient advantage estimation via exponential temporal difference decay.

🔴 advanced5 min readrl

Generalized Advantage Estimation (GAE - Schulman et al., 2015) is the standard advantage estimator used in PPO and Actor-Critic RL. Estimating Advantage A(s,a) = Q(s,a) - V(s) involves a fundamental trade-off: 1-step TD targets have low variance but high bias, while full Monte Carlo returns have zero bias but high variance. GAE introduces decay parameter λ ∈ [0, 1] to compute an exponentially weighted average of k-step TD advantages, allowing precise tuning of the bias-variance trade-off.

The Bias-Variance Spectrum in Advantage Estimation

Given 1-step TD error $\delta_t^V = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$:

  1-Step Advantage (k=1):     A_t^{(1)} = δ_t^V                                 (High Bias, LOW VARIANCE)
  2-Step Advantage (k=2):     A_t^{(2)} = δ_t^V + γ δ_{t+1}^V
  k-Step Advantage:          A_t^{(k)} = ∑_{l=0}^{k-1} γ^l δ_{t+l}^V
  Monte Carlo (k=∞):         A_t^{(∞)} = ∑_{l=0}^∞ γ^l δ_{t+l}^V = G_t - V(s_t) (ZERO BIAS, High Variance)

The GAE Formulation

GAE defines the advantage estimator $A_t^{\text{GAE}(\gamma, \lambda)}$ as the exponentially weighted average of all $k$-step advantage estimators:

$$A_t^{\text{GAE}(\gamma, \lambda)} = (1 - \lambda) \sum_{k=1}^\infty \lambda^{k-1} A_t^{(k)} = \sum_{l=0}^\infty (\gamma \lambda)^l \delta_{t+l}^V$$

Where $\delta_{t+l}^V = r_{t+l+1} + \gamma V(s_{t+l+1}) - V(s_{t+l})$.

                    GAE Exponential Weighting (λ ∈ [0, 1])
  λ = 0 ───────────────► 1-Step TD (Fastest learning, low variance, high bias)
  λ = 0.95 ────────────► Standard PPO Setting (Optimal Bias-Variance Sweet Spot!)
  λ = 1.0 ────────────► Full Monte Carlo (Slow learning, high variance, zero bias)

Recursive Implementation

GAE can be computed backwards from the end of an episode in $O(T)$ time:

$$A_t^{\text{GAE}} = \delta_t^V + (\gamma \lambda) A_{t+1}^{\text{GAE}}$$

# Backward calculation of GAE in PyTorch
gae = 0
advantages = torch.zeros_like(rewards)
for t in reversed(range(len(rewards))):
    delta = rewards[t] + gamma * values[t+1] * (1 - dones[t]) - values[t]
    gae = delta + gamma * lam * (1 - dones[t]) * gae
    advantages[t] = gae

Say this out loud

"Generalized Advantage Estimation (GAE) calculates Advantage A_t by taking an exponentially weighted average of k-step TD errors using parameter λ ∈ [0, 1]. λ=0 gives 1-step TD (low variance, high bias); λ=1 gives full Monte Carlo (high variance, zero bias). PPO sets λ=0.95 as the optimal sweet spot for deep RL and alignment."

Follow-ups to expect

How does GAE interact with discount factor γ? $\gamma$ determines the effective time horizon of returns ($1/(1-\gamma)$ steps). $\lambda$ determines the effective horizon of bootstrapping trust in the Critic network ($1/(1-\gamma\lambda)$ steps).
What happens if the Critic V(s) is inaccurate? If $V(s)$ is poorly trained (high approximation error), low $\lambda$ transfers Critic estimation error directly into the Advantage, corrupting Policy updates. Train Critic $V(s)$ thoroughly.

Check yourself

Question 1 of 3

What fundamental trade-off in Advantage Estimation does GAE (Generalized Advantage Estimation) address?

More in Reinforcement Learning

See all →

Value-Based vs Policy-Based Methods5 min Multi-Armed Bandits4 min Proximal Policy Optimization5 min