GRPO & Group-Relative Methods
Eliminating separate Critic value networks in LLM alignment by computing group-relative advantage across sampled responses.
The Paradigm Shift: PPO vs GRPO
TRADITIONAL PPO RLHF:
Prompt x ──► Policy LLM ──► Response y ──► Reward Model r(x,y)
│
▼
Requires CRITIC MODEL V_ϕ(x) to compute Advantage: A(x,y) = r(x,y) - V_ϕ(x)
(Massive VRAM Overhead: Actor + Critic + Ref + Reward = 4 Models!)
GROUP RELATIVE POLICY OPTIMIZATION (GRPO):
Prompt x ──► Policy LLM ──► Samples Group of G = 4 Responses: {y_1, y_2, y_3, y_4}
│
▼
Scalar Rewards: {r_1, r_2, r_3, r_4}
Group Mean: μ_r = mean(r)
Group Std: σ_r = std(r)
Group Advantage: A_i = (r_i - μ_r) / (σ_r + ε) ──► NO CRITIC MODEL NEEDED!
(50% Lower VRAM: Actor + Ref + Reward = 3 Models!)
The GRPO Mathematical Objective
For prompt $x$ and group of $G$ sampled outputs ${y_1, y_2, \dots, y_G}$:
$$\mathcal{L}{\text{GRPO}}(\theta) = \hat{\mathbb{E}} \left[ \frac{1}{G} \sum{i=1}^G \min\left( \frac{\pi_\theta(y_i \mid x)}{\pi_{\text{old}}(y_i \mid x)} A_i, \text{clip}\left( \frac{\pi_\theta(y_i \mid x)}{\pi_{\text{old}}(y_i \mid x)}, 1-\epsilon, 1+\epsilon \right) A_i \right) - \beta D_{KL}(\pi_\theta \parallel \pi_{\text{ref}}) \right]$$
Group Advantage $A_i$:
$$A_i = \frac{r_i - \text{mean}({r_1, \dots, r_G})}{\text{std}({r_1, \dots, r_G}) + \epsilon}$$
Memory & VRAM Breakdown
| Resource Component | Standard PPO RLHF | GRPO (Group Relative) |
|---|---|---|
| Policy Network ($\pi_\theta$) | Active (Trainable) | Active (Trainable) |
| Critic Network ($V_\phi$) | Active (Trainable - 100% Params) | ELIMINATED (0 Params!) |
| Reference Model ($\pi_{\text{ref}}$) | Frozen | Frozen |
| Reward Model ($r_\psi$) | Frozen | Frozen |
| VRAM Footprint Reduction | Baseline | ~50% Lower Memory Overhead |
Why GRPO Excels in Reasoning Models (DeepSeek-R1)
For math and coding tasks, binary rule-based verifiers provide exact execution rewards ($r_i = 1.0$ if code passes unit tests; $r_i = 0.0$ if it fails).
Sampling $G = 8$ candidate reasoning chains per prompt creates a natural intra-prompt tournament:
- Responses that solve the math problem receive positive normalized Advantage $A_i > 0$.
- Responses that fail receive negative normalized Advantage $A_i < 0$.
Grades update the Policy LLM without needing any neural value estimator!
Say this out loud
"GRPO eliminates the memory-heavy Critic Value Model in RLHF. For each prompt, GRPO samples a group of G responses, computing baseline value dynamically as group average reward. Advantage for response i is normalized against group mean and std: A_i = (r_i - μ_r) / σ_r. DeepSeek-R1 used GRPO to cut VRAM memory by 50% while scaling reasoning performance."
Follow-ups to expect
- What is the optimal Group Size G in GRPO? DeepSeek-Math empirically evaluated group sizes $G \in [4, 16]$, finding $G = 4\text{--}8$ provides optimal variance reduction without inflating generation rollout time.
- How does GRPO handle KL Divergence? Instead of adding KL penalty directly into reward $r_i$, GRPO adds an explicit KL divergence loss term $D_{KL}(\pi_\theta \parallel \pi_{\text{ref}})$ directly to the main loss function.
Check yourself
How does GRPO (Group Relative Policy Optimization) eliminate the need for training a separate Critic Value Model V_ϕ(s) in LLM RLHF?