Inference Optimization & Batching
Maximizing GPU memory bandwidth and inference token throughput for high-concurrency LLM deployments.
LLM Inference is heavily Memory Bandwidth-Bound rather than Compute-Bound during autoregressive token generation. Key optimization strategies include Continuous Batching (vLLM iteration-level scheduling), PagedAttention (virtual memory management for KV cache), Quantization (FP16 → INT8/INT4/FP8), FlashAttention (fused GPU kernel tiling), and Tensor Parallelism. These techniques increase serving throughput by 5x–20x while reducing cost per token.
Compute-Bound vs Memory-Bound Regime
PREFILL PHASE (Prompt Processing) DECODE PHASE (Token Generation)
- Processes full input prompt in parallel - Generates 1 token at a time autoregressively
- Matrix-Matrix Multiplication (GEMM) - Matrix-Vector Multiplication (GEMV)
- HIGH Arithmetic Intensity (FLOPs / Byte) - LOW Arithmetic Intensity (~1 FLOP / Byte)
- COMPUTE-BOUND (GPU Cores Saturation) - MEMORY BANDWIDTH-BOUND (HBM Transfer Speed)
During the Decode Phase, GPU tensor cores sit idle waiting for weights to transfer from HBM to SRAM.
Key Inference Optimization Stack
INFERENCE OPTIMIZATION STACK
┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. CONTINUOUS BATCHING │ 2. PAGEDATTENTION │ 3. FLASHATTENTION-2 │ 4. QUANTIZATION │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Iteration-level request │ Virtual memory paging │ SRAM tiling; avoids │ FP16 ──► FP8 / INT4 │
│ scheduling; zero idle GPU│ for KV Cache; drops VRAM │ materializing N×N matrix │ 2x-4x lower VRAM & HBM │
│ bubbles. │ fragmentation to < 1%. │ in HBM. │ bandwidth transfer. │
└──────────────────────────┴──────────────────────────┴──────────────────────────┴──────────────────────────┘
Continuous Batching vs Static Batching
Static Batching (GPU idle waiting for Req 1 to finish 500 tokens):
Req 1 (500 tokens) ████████████████████████████████████████████████████
Req 2 (50 tokens) █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ (IDLE GPU VRAM!)
Continuous Batching (vLLM schedules Req 3 as soon as Req 2 completes):
Req 1 (500 tokens) ████████████████████████████████████████████████████
Req 2 (50 tokens) █████
Req 3 (New Ingest) ███████████████████████████████████████████████ (100% GPU UTILIZATION!)
Say this out loud
"LLM token generation is memory bandwidth-bound because loading model weights for 1 token operation has low arithmetic intensity. We optimize serving throughput using Continuous Batching to schedule requests at the token iteration level, PagedAttention to eliminate KV cache memory fragmentation, and FP8/INT4 Quantization to shrink VRAM bandwidth load by 2-4x."
Follow-ups to expect
- What is Tensor Parallelism (TP) vs Pipeline Parallelism (PP)? Tensor Parallelism splits individual linear layers (e.g. matrix W) across GPUs via Megatron-LM intra-node NVLink. Pipeline Parallelism splits layers 1..N across distinct GPU nodes sequentially.
- What is FP8 Quantization (E4M3 vs E5M2)? FP8 uses 8-bit floats: E4M3 (1 sign, 4 exponent, 3 mantissa) is preferred for forward weights and activations (higher precision); E5M2 (1 sign, 5 exponent, 2 mantissa) is used for gradients (wider dynamic range).
Check yourself
Question 1 of 3
Why is autoregressive LLM token generation memory bandwidth-bound rather than compute-bound?