Value-Based vs Policy-Based Methods
Comparing the two core families of Reinforcement Learning algorithms: learning action values vs directly optimizing policy parameters.
Comparison Matrix
VALUE-BASED (DQN) POLICY-BASED (REINFORCE) ACTOR-CRITIC (PPO)
Learns Q(s, a) Learns π_θ(a|s) Learns BOTH!
Policy: a = argmax_a Q(s, a) Optimizes θ via ∇_θ J(θ) Actor π_θ(a|s) + Critic V_ϕ(s)
| Property | Value-Based (DQN, SARSA) | Policy-Based (REINFORCE) | Actor-Critic (PPO, SAC, A2C) |
|---|---|---|---|
| Core Target Learned | Action-Value $Q(s, a)$ | Policy Distribution $\pi_\theta(a \mid s)$ | Both Policy $\pi_\theta(a \mid s)$ & Value $V_\phi(s)$ |
| Action Space | Discrete (Buttons, Moves) | Continuous & Discrete | Continuous & Discrete |
| Policy Type | Deterministic (Greedy $\arg\max_a$) | Stochastic / Continuous Gaussian | Stochastic / Continuous |
| Sample Efficiency | High (Off-policy Replay Buffer) | Low (On-policy Monte Carlo) | Moderate to High |
| Variance | Low (Bootstrapping) | High (Monte Carlo Return $G_t$) | Low (Advantage $A = Q - V$) |
Mathematical Objectives
1. Value-Based Loss (DQN)
Minimizes Bellman Mean Squared Error over replay buffer $D$:
$$\mathcal{L}(\theta) = \mathbb{E}{(s,a,r,s') \sim D} \left[ \left( r + \gamma \max{a'} Q(s', a'; \theta^-) - Q(s, a; \theta) \right)^2 \right]$$
2. Policy-Based Objective (Policy Gradient Theorem)
Maximizes expected trajectory return $J(\theta) = \mathbb{E}{\pi\theta}[G_0]$ via gradient ascent:
$$\nabla_\theta J(\theta) = \mathbb{E}{\pi\theta} \left[ \nabla_\theta \ln \pi_\theta(a \mid s) \cdot G_t \right]$$
3. Actor-Critic Objective
Replaces Monte Carlo return $G_t$ with Advantage Function $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$:
$$\nabla_\theta J(\theta) = \mathbb{E}{\pi\theta} \left[ \nabla_\theta \ln \pi_\theta(a \mid s) \cdot A^\phi(s, a) \right]$$
Say this out loud
"Value-based methods (DQN) learn action values Q(s,a) for discrete actions, deriving greedy policies via argmax_a Q(s,a). Policy-based methods (REINFORCE) parameterize policy π_θ(a|s) directly using gradient ascent, handling continuous action spaces naturally. Actor-Critic methods (PPO) combine both: the Actor learns policy π_θ and the Critic learns value baseline V_ϕ(s) to reduce gradient variance."
Follow-ups to expect
- What is On-Policy vs Off-Policy learning? On-policy methods (PPO, REINFORCE) evaluate and update the exact policy currently generating environment samples. Off-policy methods (DQN, SAC) evaluate one policy while using samples generated by an older/different exploration policy stored in a Replay Buffer.
- What is Soft Actor-Critic (SAC)? Off-policy maximum-entropy actor-critic algorithm that adds entropy bonus $\mathcal{H}(\pi(\cdot|s))$ to the reward objective, encouraging exploration in continuous control tasks.
Check yourself
Why do Value-Based methods (Q-Learning / DQN) struggle with continuous action spaces (e.g. steering wheel angle from -180° to +180°)?