Test-Time Compute & Inference Scaling

Scaling inference-time FLOPs to solve complex math, code, and logic reasoning tasks without retraining pre-trained weights.

🔴 advanced5 min readllm

Test-Time Compute Scaling (Inference Scaling) shifts compute allocation from pre-training parameters to inference-time reasoning. Instead of relying on single-pass forward generation, systems trade extra GPU FLOPs at inference time for higher accuracy. Techniques include Search over Reasoning Trees (Monte Carlo Tree Search - MCTS, Process Reward Models), Majority Voting (Self-Consistency), and native reasoning models (OpenAI o1) trained via RL to execute adaptive internal thinking steps before returning answers.

The Compute Frontier: Pre-training vs Test-Time

Historically, AI capabilities scaled by increasing Pre-training Compute (more parameters $N$, more tokens $D$).

Test-Time Compute introduces a second dimension of capability scaling:

  TRADITIONAL INFERENCE (Fixed Single Pass)     TEST-TIME COMPUTE SCALING (System-2 Reasoning)
  User Query ──► 1 Single Forward Pass ──►     User Query ──► [ MCTS Search / CoT Rollouts ]
                 (Output in < 1 second)                      ├── Step 1 ──► PRM Check (0.95)
                                                             ├── Step 2 ──► PRM Check (0.12) -> BACKTRACK!
                                                             └── Step 3 ──► Final Verified Output
                                                             (Spends 30 seconds of GPU compute!)

$$\text{Total Intelligence} = f(\text{Pre-training Compute}, \text{Post-training Compute}, \mathbf{\text{Test-Time Compute}})$$

Test-Time Compute Scaling Mechanisms

                               TEST-TIME COMPUTE STRATEGIES
┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. SELF-CONSISTENCY      │ 2. REASONING TREE SEARCH │ 3. PROCESS REWARD (PRM)  │ 4. NATIVE REASONING (o1) │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Sample N CoT paths; take │ Monte Carlo Tree Search  │ Evaluate step-by-step    │ RL-trained internal      │
│ majority vote answer.    │ (MCTS) over thought nodes│ correctness to guide     │ thinking tokens generated│
│                          │ with beam search pruning.│ beam search expansion.   │ automatically before text│
└──────────────────────────┴──────────────────────────┴──────────────────────────┴──────────────────────────┘

Snell et al. (2024) Trade-off Curves

  Task Accuracy (%)
  100% ┤                                       ┌─── 7B Model with Test-Time Search!
       │                                 ┌─────┘
   70% ┤                           ┌─────┘
       │                    ┌──────┘
   40% ┼─────────────────── 70B Base Model (Single Pass)
       ┴──────────────────────────────────────────────────────────► Test-Time Compute (FLOPs)

Spending 100x more compute at inference time on an 8B parameter model using MCTS search outperforms a 70B parameter model evaluated in a single forward pass on competition math (AIME/MATH benchmarks)!

Say this out loud

"Test-Time Compute Scaling trades inference-time GPU FLOPs for higher accuracy on complex math, coding, and reasoning tasks. Instead of relying on a single forward pass, systems use tree search (MCTS) guided by Process Reward Models (PRMs), majority voting, or native RL-trained thinking tokens (OpenAI o1) to explore, verify, and backtrack before outputting final answers."

Follow-ups to expect

What is the difference between System-1 and System-2 Thinking in AI? System-1 is fast, intuitive, single-pass generation (standard LLM completion). System-2 is slow, deliberate, multi-step analytical reasoning and search (Test-Time Compute).
How does Test-Time Compute affect serving latency and pricing? Test-time compute increases latency (from < 1 second to 10–60 seconds) and increases per-query cost (charging for generated internal thinking tokens alongside output tokens).

Check yourself

Question 1 of 3

Why does spending additional compute FLOPs at inference time (Test-Time Compute) boost LLM accuracy on hard reasoning problems?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min