Temperature, Top-p & Sampling

The Logit-to-Token Pipeline

Raw Un-normalized Logits z [|V|]
            │
            ▼
   [ TEMPERATURE SCALING ]   ──► z_scaled = z / T
            │
            ▼
    [ SOFTMAX ACTIVATION ]   ──► P_i = exp(z_i / T) / ∑ exp(z_j / T)
            │
            ▼
  [ TOP-K / TOP-P FILTERING ]──► Zero out low-probability tails
            │
            ▼
    [ STOCHASTIC SAMPLING ]  ──► Sample token x_t ~ Filtered_P

1. Temperature ($T$)

Rescales logits prior to Softmax:

$$P(x_i \mid z, T) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$

• $T = 0.0$ (Greedy / Argmax): Purely deterministic. Use for Math, Coding, SQL, Fact Retrieval.
• $T = 0.7$ (Balanced): Default for Chatbots, Summarization, Q&A.
• $T = 1.2+$ (High Entropy): Creative writing, brainstorming. Warning: High temperatures increase hallucination rates.

2. Top-$k$ Sampling

Restricts sampling pool strictly to the top $k$ highest-probability tokens (e.g., $k = 40$).

Drawback: Fixed $k$ is rigid. If top token has $P = 0.99$, Top-$40$ still includes 39 low-quality noise tokens. If top token has $P = 0.05$, Top-$40$ truncates valid continuations.

3. Top-$p$ (Nucleus) Sampling (Holtzman et al., 2019)

Dynamically selects the smallest set of tokens $V^{(p)}$ whose cumulative probability mass reaches $p$ (e.g., $p = 0.90$):

$$\sum_{x \in V^{(p)}} P(x \mid x_{<t}) \ge p$$

Confident State ("Capital of France is...")    Uncertain State ("The story begins in...")
  Top 1 Token ('Paris') = 92%                   Token 1 = 15%, Token 2 = 12%, Token 3 = 10%...
  Pool Size = 1 Token!                          Pool Size = 25 Tokens! (Expanded dynamically)

Recommended Preset Settings

Say this out loud

"Temperature scales logits z/T before Softmax: T=0 gives deterministic argmax outputs for code and math, while higher T increases distribution entropy for creative text. Top-k filters a fixed number of tokens; Top-p (Nucleus) dynamically samples from the smallest pool of tokens reaching cumulative probability p, adapting dynamically when the model is confident vs uncertain."

Follow-ups to expect

• Can you use Temperature and Top-p together? Yes. Standard pipeline applies Temperature scaling first, computes Softmax, applies Top-p cumulative truncation, and samples from the remaining nucleus.
• What is Repetition Penalty / Presence Penalty? Subtracts a penalty from a token's logit if that token has already appeared in the generated text, preventing infinite repetition loops ("the the the...").