Structured Output & Constrained Decoding

The Two Approaches to Structured Output

  1. PROMPT-LEVEL INSTRUCTION (Probabilistic / Flaky)
  System: "Respond in JSON matching {'name': str, 'age': int}."
  Output: "Sure! Here is the JSON: {'name': 'Alice', 'age': 30,}"  ──► JSON.parse() CRASHES! (Trailing comma)

  2. GRAMMAR-CONSTRAINED DECODING (Deterministic / 100% Guaranteed)
  System + Pydantic Schema
  At every token step, Logit Masking sets invalid syntax tokens to -∞.
  Output: {"name": "Alice", "age": 30}                              ──► 100% VALID PARSE!

How Grammar-Constrained Decoding Works

Let $V$ be the LLM vocabulary ($32,000$ tokens).

Suppose the LLM has generated: {"name": "Alice",

The JSON schema specifies the next key MUST be "age".

  Logit Vector z [32,000]
  Token '"age"'   ──►  Logit z = 12.4  (VALID -> Retained)
  Token 'true'    ──►  Logit z = -∞    (INVALID -> Masked!)
  Token 'null'    ──►  Logit z = -∞    (INVALID -> Masked!)
  Token '}'       ──►  Logit z = -∞    (INVALID -> Masked!)

$$\text{Logit Masking: } z_i = \begin{cases} z_i & \text{if token } i \text{ matches Grammar State} \ -\infty & \text{otherwise} \end{cases}$$

After applying Softmax, probabilities for invalid tokens become exactly zero ($e^{-\infty} = 0$).

Popular Frameworks

• Outlines (Python): Uses FSM (Finite State Machines) compiled from Pydantic schemas and Regex patterns for ultra-fast logit masking.
• Guidance (Microsoft): Interleaves Python execution loops with constrained LLM token generation.
• OpenAI Structured Outputs (response_format={"type": "json_schema"}): Server-side grammar constraint guaranteeing 100% schema match on GPT-4.
• vLLM / SGLang: Native engine-level JSON schema logit masking.

Integration Example with Pydantic & Outlines

from pydantic import BaseModel
import outlines

class UserProfile(BaseModel):
    name: str
    age: int
    roles: list[str]

# Load model with outlines sampler
model = outlines.models.transformers("meta-llama/Meta-Llama-3-8B-Instruct")
generator = outlines.generate.json(model, UserProfile)

# Guarantees 100% valid UserProfile instance!
result: UserProfile = generator("Extract user profile from: Alice is a 30yo admin.")

Say this out loud

"Prompt instructions cannot guarantee valid JSON because stochastic sampling can pick invalid tokens like missing quotes or trailing commas. Grammar-Constrained Decoding enforces 100% schema compliance by using a Finite State Machine to mask invalid token logits to -∞ at every generation step. Tools like Outlines and OpenAI Structured Outputs enforce Pydantic schemas with zero syntax errors."

Follow-ups to expect

• Does Constrained Decoding limit model reasoning performance? If the model is forced into JSON format immediately without space to think, performance can drop. Allow the model to generate a Chain-of-Thought field first inside the schema ({"reasoning": "...", "final_answer": "..."}).
• How does Constrained Decoding handle regex patterns? Compiles regular expressions into Deterministic Finite Automata (DFA) state machines that track valid next-character transitions at the token level.