Reinforcement Learningintermediatemust-know5 min

Value-Based vs Policy-Based Methods

Comparing the two core families of Reinforcement Learning algorithms: learning action values vs directly optimizing policy parameters.

Reinforcement Learning algorithms divide into Value-Based methods (Q-Learning, DQN) and Policy-Based methods (REINFORCE, PPO). Value-based methods learn action-value function Q(s, a), deriving implicit greedy policies a = argmax_a Q(s, a). Policy-based methods parameterize policy π_θ(a|s) directly, optimizing parameters θ via gradient ascent on expected return. Actor-Critic methods combine both: an Actor policy π_θ(a|s) and a Critic value function V_ϕ(s).

Read the deep dive →
Reinforcement Learningintermediatemust-know4 min

Multi-Armed Bandits

Optimizing online decision-making when actions yield stochastic rewards without full sequential state transitions.

A Multi-Armed Bandit (MAB) is a simplified Reinforcement Learning framework where an agent chooses among K independent actions ("arms") to maximize cumulative reward. Unlike full MDPs, bandits have no state transitions—each action choice is stateless and independent. MABs replace traditional static A/B testing in digital marketing, ad click optimization, and website UI layout experiments by dynamically routing traffic to high-performing variations while continuously testing alternatives.

Read the deep dive →
Reinforcement Learningadvancedmust-know5 min

Proximal Policy Optimization

Clipping probability ratio updates to achieve stable, sample-efficient policy gradient updates in RL and LLM alignment.

Proximal Policy Optimization (PPO - Schulman et al., 2017 / OpenAI) is the workhorse policy optimization algorithm for deep RL and LLM RLHF alignment. Standard policy gradient updates suffer from destructive large parameter steps that permanently collapse model performance. PPO introduces a Clipped Surrogate Objective function L_CLIP(θ) = E [ min( r_t(θ) A_t, clip(r_t(θ), 1-ε, 1+ε) A_t ) ], capping the probability ratio r_t(θ) = π_θ(a|s) / π_old(a|s) within [1-ε, 1+ε] to guarantee conservative, monotonic policy improvements.

Read the deep dive →
Reinforcement Learningintermediate4 min

Markov Decision Processes

The formal mathematical framework for modeling sequential decision making under uncertainty.

A Markov Decision Process (MDP) is the mathematical foundation of Reinforcement Learning. An MDP is defined as a 5-tuple (S, A, P, R, γ): States S, Actions A, Transition Probability P(s'|s,a), Reward function R(s,a,s'), and Discount factor γ ∈ [0, 1). The core assumption is the Markov Property: future state s_{t+1} depends ONLY on current state s_t and action a_t, independent of past history.

Read the deep dive →
Reinforcement Learningintermediate5 min

Bellman Equations

Decomposing cumulative value functions into immediate rewards plus discounted future values.

Bellman Equations form the recursive foundation of dynamic programming and Reinforcement Learning. They express the value of a state V(s) or state-action pair Q(s,a) as the immediate reward plus the discounted expected value of successor states. Bellman Expectation Equations evaluate a fixed policy π; Bellman Optimality Equations define the unique optimal value functions V*(s) and Q*(s,a). Dynamic Programming methods (Policy Iteration, Value Iteration) solve Bellman equations iteratively.

Read the deep dive →
Reinforcement Learningintermediate4 min

Q-Learning

Off-policy model-free temporal difference control for learning optimal action values.

Q-Learning (Watkins, 1989) is a foundational model-free, off-policy Temporal Difference (TD) Reinforcement Learning algorithm. It learns the optimal action-value function Q*(s, a) directly from environment experiences (s, a, r, s') without knowing environment transition dynamics P(s'|s,a). It is Off-Policy because it updates Q values using the greedy maximum target max_{a'} Q(s', a') regardless of the exploratory action taken by the behavior policy (e.g. ε-greedy).

Read the deep dive →
Reinforcement Learningintermediate5 min

ε-greedy, UCB & Thompson Sampling

Solving the classic Exploration vs Exploitation trade-off in decision systems, recommendation feeds, and multi-armed bandits.

Exploration vs Exploitation is the fundamental dilemma in Reinforcement Learning and recommendation systems: should the agent Exploit the current best-known action to maximize short-term reward, or Explore unknown actions to discover potentially higher long-term rewards? Key strategies include ε-greedy (random uniform exploration), Upper Confidence Bound (UCB - Optimism in the face of uncertainty), and Thompson Sampling (Bayesian posterior probability matching).

Read the deep dive →
Reinforcement Learningintermediate5 min

Why RL Is Sample-Inefficient

Analyzing why model-free Reinforcement Learning requires millions of environment steps and how model-based RL improves sample efficiency.

Reinforcement Learning is notoriously sample-inefficient, often requiring tens of millions of environment interaction steps (equivalent to weeks of real-world time) to learn basic control tasks. Sample inefficiency stems from high-variance credit assignment, sparse rewards, non-stationary policy data distributions, and model-free trial-and-error. Mitigation strategies include Off-Policy Replay Buffers (SAC / TD3), Model-Based RL (World Models, MBPO, DreamerV3), Pre-trained Representations, and Reward Shaping.

Read the deep dive →
Reinforcement Learningadvanced5 min

SARSA vs Q-Learning (On vs Off Policy)

Comparing On-Policy temporal difference control (SARSA) against Off-Policy greedy control (Q-Learning).

SARSA and Q-Learning are two foundational Temporal Difference (TD) Reinforcement Learning control algorithms. SARSA is On-Policy: its name derives from tuple (S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1}), updating Q(s,a) using the actual next action a_{t+1} selected by the exploratory behavior policy π. Q-Learning is Off-Policy: it updates Q(s,a) using the maximum greedy action max_{a'} Q(s',a') ignoring the actual exploratory action taken. SARSA learns safer policies when exploration risks (e.g. falling off a cliff) carry high negative penalties.

Read the deep dive →
Reinforcement Learningadvanced5 min

Deep Q-Networks & Replay Buffers

Scaling Q-Learning to high-dimensional continuous state spaces using neural network function approximators, Experience Replay, and Target Networks.

Deep Q-Networks (DQN - Mnih et al., 2015 / DeepMind) revolutionized Reinforcement Learning by using deep convolutional neural networks to approximate Q(s, a; θ) directly from raw Atari screen pixels. Combining non-linear neural network function approximators with Q-learning causes severe divergence. DQN stabilized training through two key innovations: Experience Replay (breaking temporal correlation between sequential training samples) and Target Networks Q(s, a; θ^-) (freezing target weights to prevent moving target feedback instabilities).

Read the deep dive →
Reinforcement Learningadvanced5 min

Policy Gradients & REINFORCE

Directly optimizing policy parameters θ via gradient ascent on expected trajectory returns.

Policy Gradient methods directly parameterize a policy π_θ(a|s) and update parameters θ using gradient ascent on expected return J(θ). The Policy Gradient Theorem proves that ∇_θ J(θ) = E [ ∇_θ ln π_θ(a|s) · Q^π(s,a) ] without requiring knowledge of environment transition derivatives. The REINFORCE algorithm (Williams, 1992) uses Monte Carlo trajectory returns G_t as unbiased estimators of Q^π(s,a), but suffers from high variance.

Read the deep dive →
Reinforcement Learningadvanced5 min

Actor–Critic Methods

Combining policy gradients and value bootstrapping to achieve low-variance, sample-efficient reinforcement learning.

Actor-Critic methods combine Policy-Based (Actor) and Value-Based (Critic) reinforcement learning. The Actor parameterizes policy π_θ(a|s), selecting actions; the Critic parameterizes value function V_ϕ(s) or Q_ϕ(s,a), evaluating action quality. By replacing high-variance Monte Carlo trajectory returns G_t with the Advantage function A(s,a) = Q(s,a) - V(s) estimated via 1-step TD bootstrapping r + γ V(s') - V(s), Actor-Critic architectures achieve significantly lower gradient variance and higher sample efficiency.

Read the deep dive →
Reinforcement Learningadvanced5 min

GRPO & Group-Relative Methods

Eliminating separate Critic value networks in LLM alignment by computing group-relative advantage across sampled responses.

Group Relative Policy Optimization (GRPO - Shao et al., 2024 / DeepSeek-Math, DeepSeek-R1) eliminates the memory-heavy Critic Value Model in RLHF. For each prompt x, GRPO samples a group of G outputs {y_1, y_2, ..., y_G} from the old policy. Instead of training a separate Critic model V_ϕ(s) to estimate baseline values, GRPO computes the Advantage A_i for candidate i by normalizing its scalar reward r_i against the group mean and standard deviation: A_i = (r_i - mean(r)) / std(r).

Read the deep dive →
Reinforcement Learningadvanced5 min

Advantage Estimation & GAE

Balancing bias and variance in policy gradient advantage estimation via exponential temporal difference decay.

Generalized Advantage Estimation (GAE - Schulman et al., 2015) is the standard advantage estimator used in PPO and Actor-Critic RL. Estimating Advantage A(s,a) = Q(s,a) - V(s) involves a fundamental trade-off: 1-step TD targets have low variance but high bias, while full Monte Carlo returns have zero bias but high variance. GAE introduces decay parameter λ ∈ [0, 1] to compute an exponentially weighted average of k-step TD advantages, allowing precise tuning of the bias-variance trade-off.

Read the deep dive →
Reinforcement Learningadvanced5 min

Contextual Bandits in Production

Personalizing online action selection based on real-time user context vectors.

Contextual Bandits extend Multi-Armed Bandits by observing a context vector x_t (user profile, location, device, time) before choosing an action a_t. Unlike full MDPs, action choices do not alter future state transitions. Contextual Bandits form the core foundation of production recommendation feeds (news ranking, ad targeting, dynamic push notifications). Algorithms include LinUCB (linear reward modeling with ridge confidence bounds) and Contextual Thompson Sampling (sampling weights from Bayesian ridge regression posteriors).

Read the deep dive →
Reinforcement Learningadvanced5 min

Reward Shaping & Specification Gaming

Designing dense intermediate rewards without introducing unintended policy exploits or specification gaming.

Reward Shaping modifies an environment's sparse reward signal by adding domain-specific intermediate rewards to accelerate RL learning. Naive reward shaping leads to **Specification Gaming** (Reward Hacking / Perverse Incentives)—where the agent discovers unintended shortcuts that maximize shaped rewards without fulfilling true goal intent (e.g., spinning in circles to collect speed points). Potential-Based Reward Shaping (PBRS - Ng et al., 1999) proves that defining shaping rewards as F(s, s') = γ Φ(s') - Φ(s) mathematically guarantees preserving optimal policy invariance.

Read the deep dive →
Reinforcement Learningadvanced5 min

Offline RL

Learning optimal policies from static pre-collected datasets without online environment interaction.

Offline Reinforcement Learning (Batch RL) trains policies exclusively on static, pre-collected datasets D = {(s, a, r, s')} without any real-time environment interaction. Standard off-policy algorithms (DQN, SAC) fail catastrophically in offline settings due to Out-of-Distribution (OOD) Action Extrapolation Error: the Q-function overestimates un-observed actions (argmax_a Q(s,a)), causing the policy to choose catastrophic OOD actions. Offline RL algorithms (CQL, IQL, TD3+BC) constrain policy updates to stay within the empirical data distribution.

Read the deep dive →
Reinforcement Learningadvanced5 min

Imitation & Inverse RL

Learning policy behaviors directly from expert human demonstrations without engineering explicit reward functions.

Imitation Learning trains an agent policy to mimic expert human behavior from demonstration trajectories D_{expert} = {(s_0, a_0), (s_1, a_1), ...}. Behavioral Cloning (BC) frames imitation as supervised regression/classification, but suffers from Compounding Covariate Shift errors. DAgger (Dataset Aggregation) resolves compounding errors by iteratively querying human experts on agent-visited states. Inverse Reinforcement Learning (IRL - GAIL) infers the implicit reward function R(s,a) optimized by the expert, using adversarial training to match expert state-action distributions.

Read the deep dive →
SCROLL · SAVE · TAP TO GO DEEPER