Offline RL

Online RL vs Offline RL

  ONLINE REINFORCEMENT LEARNING:
  Policy π_θ ──► Environment Interaction ──► New (s, a, r, s') Data ──► Update Policy π_θ
  (Continuously collects real-time trial-and-error feedback)

  OFFLINE (BATCH) REINFORCEMENT LEARNING:
  Static Dataset D = {(s_i, a_i, r_i, s_i')} ──► [ OFFLINE RL ALGORITHM ] ──► Deploy Policy π_θ
  (ZERO real-time environment interaction allowed during training!)

The Out-of-Distribution (OOD) Extrapolation Problem

Consider the standard Off-Policy TD Target:

$$y = r + \gamma \max_{a'} Q(s', a'; \theta)$$

When evaluating $\max_{a'} Q(s', a')$, the maximization operator evaluates actions $a'$ that may never appear in the static dataset $D$.

  Q-Value Prediction Curve Q(s, a)
  Q-Value
   High ┤                       OOD Action Extrapolation Spike! (Falsely High!)
        │                            /\
        │  In-Distribution Data     /  \
   Low  ┴──────[ Dataset Actions ]─/────\─────────────────► Action Space a

1. Neural networks have non-zero generalization error over unseen inputs.
2. The $\max$ operator explicitly selects actions with the highest positive error spikes.
3. In Online RL, the agent tries action $a'$, observes real reward, and fixes the error.
4. In Offline RL, no real-time feedback exists, so the policy greedily selects catastrophic OOD actions!

Offline RL Solution Taxonomies

                                OFFLINE RL STRATEGIES
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  1. POLICY CONSTRAINTS   │  2. CONSERVATIVE Q (CQL) │  3. IMPLICIT Q (IQL)     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Add KL penalty to keep   │ Penalize Q-values on OOD │ Avoids max_a over OOD    │
│ policy near dataset:     │ actions; forces lower-   │ actions entirely using   │
│ D_KL(π || π_dataset).    │ bound Q-values.          │expectile regression on V.│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Conservative Q-Learning (CQL) Loss

Adds a regularizer to the standard Bellman error:

$$\mathcal{L}{\text{CQL}}(\theta) = \alpha \cdot \mathbb{E}{s \sim D} \left[ \log \sum_a \exp(Q(s, a; \theta)) - \mathbb{E}{a \sim D}[Q(s, a; \theta)] \right] + \mathcal{L}{\text{DQN}}(\theta)$$

Minimizing this loss pushes down Q-values on un-observed actions while pulling up Q-values on dataset actions, guaranteeing conservative lower-bound values.

Say this out loud

"Offline RL learns policies exclusively from static historical datasets without environment interaction. Standard algorithms fail due to Out-of-Distribution (OOD) action extrapolation error, where max_a Q(s,a) overestimates unseen actions. Algorithms like CQL and IQL constrain policy updates to dataset distributions or penalize OOD Q-values, enabling safe policy learning for healthcare and robotics."

Follow-ups to expect

• What is Decision Transformer (Chen et al., 2021)? Frames Offline RL as a sequence modeling problem: inputs sequence of (Target Return, State, Action) tokens into a Causal Transformer, predicting actions autoregressively without any Q-learning or Bellman updates.
• How do you evaluate an Offline RL policy before real-world deployment? Use Off-Policy Evaluation (OPE) techniques like Doubly Robust (DR) estimation or Marginalized Importance Sampling (MIS) to estimate policy performance from historical logs.