Actor–Critic Methods

Actor-Critic Architecture

                               ENVIRONMENT (State s_t)
                                         │
                 ┌───────────────────────┴───────────────────────┐
                 ▼                                               ▼
     [ ACTOR NETWORK π_θ(a|s) ]                      [ CRITIC NETWORK V_ϕ(s) ]
     Generates Action Distribution                   Estimates State Value
                 │                                               │
                 ▼                                               ▼
          Action a_t ──► Environment Step ──► Reward r_{t+1}, Next State s_{t+1}
                                                                 │
                                                                 ▼
                                                    [ ADVANTAGE ESTIMATOR ]
                                                    TD Error δ_t = r + γ V_ϕ(s') - V_ϕ(s)
                                                                 │
                 ┌───────────────────────────────────────────────┘
                 ▼ (Updates BOTH networks!)
  - Actor Update:  θ ← θ + α_actor · ∇_θ ln π_θ(a_t|s_t) · δ_t
  - Critic Update: ϕ ← ϕ - α_critic · ∇_ϕ ( δ_t )²

The Advantage Function

$$\text{Advantage: } A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$$

Using 1-step TD Bootstrapping, $Q(s, a) \approx r + \gamma V_\phi(s')$:

$$A^\phi(s_t, a_t) \approx r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t) = \delta_t \quad \text{(TD Error!)}$$

Notice that TD Error $\delta_t$ acts as an unbiased sample estimate of Advantage $A(s, a)$!

Advantages over Pure Methods

• Vs Pure Policy Gradient (REINFORCE): Substantially lower gradient variance due to TD bootstrapping baseline $V(s)$.
• Vs Pure Value-Based (DQN): Handles continuous action spaces naturally and learns stochastic policies.

Asynchronous Advantage Actor-Critic (A3C / A2C)

• A3C (Asynchronous): Multiple CPU worker threads run parallel environments independently, executing asynchronous gradient updates to a shared central model.
• A2C (Synchronous): Wait for all parallel environment workers to finish step $t$, stacking mini-batches into a single GPU tensor for synchronous parallel forward/backward passes (faster on GPUs!).

Say this out loud

"Actor-Critic methods combine policy optimization and value estimation. The Actor π_θ(a|s) proposes actions; the Critic V_ϕ(s) evaluates state values to compute Advantage A(s,a) = r + γ V(s') - V(s). TD bootstrapping replaces noisy Monte Carlo returns with smooth value baselines, drastically reducing gradient variance while supporting continuous action spaces."

Follow-ups to expect

• What is Generalized Advantage Estimation (GAE)? A trade-off hyperparameter $\lambda \in [0, 1]$ that smoothly interpolates between 1-step TD Advantage ($\lambda = 0$, low variance, high bias) and full Monte Carlo Advantage ($\lambda = 1$, high variance, zero bias): $A^{\text{GAE}(\gamma, \lambda)}t = \sum{l=0}^\infty (\gamma \lambda)^l \delta_{t+l}^V$.
• How does A2C handle multi-task parallel environments? Uses $K$ parallel environment instances (e.g. 16 vectorized games), stepping all 16 environments simultaneously to generate a batch of 16 $(s, a, r, s')$ tuples per iteration for GPU matrix multiplication.