Imitation & Inverse RL

Imitation Learning Spectrum

  BEHAVIORAL CLONING (BC)        DAGGER (Dataset Aggregation)        INVERSE RL / GAIL
  Supervised Learning            Interactive Expert Labels          Adversarial Distribution Matching
  (Train on static expert data)  (Expert labels agent states)       (Learns implicit reward R(s,a))

1. Behavioral Cloning & Compounding Covariate Shift

Behavioral Cloning (BC) fits policy $\pi_\theta(a \mid s)$ directly on expert dataset $\mathcal{D}_{\text{expert}}$ using supervised cross-entropy or MSE loss:

$$\mathcal{L}{\text{BC}}(\theta) = \mathbb{E}{(s, a) \sim \mathcal{D}{\text{expert}}} \left[ -\log \pi\theta(a \mid s) \right]$$

The Failure Mode: Compounding Errors ($O(T^2)$)

  Expert Trajectory:  State s0 ──► State s1 ──► State s2 ──► Goal
                        │
  Agent BC Execution:   └──► State s1' (Small Error ε)
                               │
                               └──► State s2'' (Unseen State -> LARGER ERROR!) ──► CATASTROPHIC CRASH!

Because training data contains only perfect expert states, the agent never learns how to recover from minor mistakes!

2. DAgger (Dataset Aggregation - Ross et al., 2011)

DAgger trains policies on the distribution of states visited by the agent itself:

  1. Train initial policy π_1 on expert dataset D.
  2. For iteration i = 1 ... N:
     a. Execute policy π_i in environment to collect trajectories D_agent.
     b. Query Human Expert to provide ground-truth action labels a_expert for all states in D_agent.
     c. Aggregate datasets: D ← D ∪ D_agent.
     d. Retrain policy π_{i+1} on aggregated dataset D.

Error scales linearly $O(T)$ with trajectory length!

3. Generative Adversarial Imitation Learning (GAIL)

Instead of matching individual actions, GAIL matches the entire state-action occupancy distribution $\rho_\pi(s, a)$:

  Agent Policy π_θ (Generator) ──► (s, a) pairs ──┐
                                                  ├──► Discriminator D_ψ(s, a) ──► Reward Signal
  Human Expert Trajectories    ──► (s, a) pairs ──┘    (Predicts Expert vs Agent)

Min-Max Objective:

$$\min_\theta \max_\psi \mathbb{E}{\pi\theta} [\log D_\psi(s, a)] + \mathbb{E}{\text{expert}} [\log (1 - D\psi(s, a))]$$

The agent uses RL (PPO) to maximize reward $R(s, a) = -\log(1 - D_\psi(s, a))$.

Say this out loud

"Imitation Learning learns policies directly from expert demonstrations. Behavioral Cloning uses supervised learning but fails over long horizons due to compounding covariate shift (O(T²) error). DAgger fixes this by having experts label agent-visited states. GAIL uses adversarial training (GAN framework) to match expert state-action occupancy distributions directly."

Follow-ups to expect

• What is Inverse Reinforcement Learning (IRL)? Algorithms (Maximum Entropy IRL) that infer the underlying reward function $R(s,a)$ that makes expert behavior optimal, rather than copying actions directly.
• When to use Imitation Learning over Reinforcement Learning? When designing a reward function is extremely complex or prone to specification gaming (e.g. autonomous driving steering wheel control), but expert demonstration data is abundant.