Speculative Decoding

The Speculative Execution Paradigm

Step 1: Small Draft Model (1B) generates γ = 4 draft tokens rapidly:
        "The" ──► "capital" ──► "of" ──► "France"

Step 2: Large Target Model (70B) evaluates ALL 4 tokens IN 1 SINGLE PARALLEL FORWARD PASS:
        Matches: "The" (OK) ──► "capital" (OK) ──► "of" (OK) ──► "France" (OK)

Step 3: Target Model accepts all 4 tokens AND generates 5th token "is"!
        Result: 5 tokens produced in time of ~1 target forward pass! (5x speedup iteration)

                             Speculative Decoding Loop
┌───────────────────────────┐         γ Draft Tokens        ┌───────────────────────────┐
│ Draft Model (e.g. 1B)     │ ────────────────────────────► │ Target Model (e.g. 70B)   │
│ Fast, 1-by-1 generation   │                               │ 1 Parallel Verification   │
└───────────────────────────┘                               └─────────────┬─────────────┘
                                                                          │
                                                                          ▼
                                                             Accept n Tokens + 1 Extra
                                                             Rejection Sampling Check

Modified Rejection Sampling Scheme

To guarantee the output distribution matches the Target Model exactly:

For draft token $x$ with draft probability $q(x)$ and target probability $p(x)$:

• Acceptance Probability: $P_{\text{accept}}(x) = \min\left( 1, \frac{p(x)}{q(x)} \right)$.
• If Accepted: Keep token $x$ and continue checking next draft token.
• If Rejected at token $k$: Discard token $k$ and all subsequent draft tokens. Sample replacement token from adjusted target distribution:

$$P_{\text{sample}}(x) = \max\left( 0, \frac{p(x) - q(x)}{1 - \sum_y \min(p(y), q(y))} \right)$$

This mathematical trick guarantees zero quality degradation—the output text distribution is 100% identical to running the 70B target model alone!

Performance Factors

• Draft Acceptance Rate ($\alpha$): Higher acceptance rate $\implies$ Higher speedup (typically 60-80% acceptance when draft model is trained on target model outputs).
• Draft Length $\gamma$: Typically $\gamma = 3\text{--}5$ tokens. Too large $\gamma$ wastes target verification FLOPs if early tokens are rejected.

Say this out loud

"Speculative Decoding accelerates LLM inference by 2x-3x with zero loss in output quality. A fast, small draft model predicts gamma candidate tokens speculatively. The large target model verifies all candidate tokens in a single parallel forward pass. Using modified rejection sampling P_accept = min(1, p/q), accepted tokens are kept, maintaining exact mathematical equivalence to sampling from the target LLM alone."

Follow-ups to expect

• What is Prompt-Lookup Decoding? A simple non-neural draft strategy: search the prompt context for matching n-gram phrases to find candidate token continuations without using any draft model neural network.
• What is Eagle (Extrapolation Algorithm)? Speculative decoding that passes hidden state vectors rather than text tokens from the target model back to the draft heads, boosting draft acceptance rate to > 80%.