Causal Masking

Hiding future text tokens to enforce autoregressive next token prediction in language models.

🟡 intermediate4 min readtransformers

Causal Masking (Look Ahead Masking) ensures that autoregressive language models (GPT, LLaMA) only attend to past and current tokens during training. By setting upper triangular attention scores above the diagonal to minus infinity before computing Softmax, future tokens receive zero attention weight. This allows Transformer decoders to process entire training text sequences in parallel while preventing information leakage from future tokens.

What is Causal Masking?

Language models like GPT and LLaMA are trained on a simple objective: predict the next word.

During training, we feed an entire 1000 word document into the Transformer at once so the GPU can process all positions in parallel.

However, if token 5 can use self attention to look at token 6, 7, and 8, the model can simply cheat by looking ahead at the answer!

Causal Masking (also called Look Ahead Masking) blocks future attention links, ensuring token $t$ can only attend to past tokens $1 \dots t$.

  UNMASKED ATTENTION (BERT - Bidirectional):      CAUSAL MASKED ATTENTION (GPT - Autoregressive):
  Token 1 ──► Can see 1, 2, 3, 4                  Token 1 ──► Can see 1 ONLY
  Token 2 ──► Can see 1, 2, 3, 4                  Token 2 ──► Can see 1, 2
  Token 3 ──► Can see 1, 2, 3, 4                  Token 3 ──► Can see 1, 2, 3
  Token 4 ──► Can see 1, 2, 3, 4                  Token 4 ──► Can see 1, 2, 3, 4

How Causal Masking Works Mathematically

Before computing Softmax on raw attention dot products $Q K^T / \sqrt{d_k}$, we add a Mask Matrix $M$:

$$\text{Attention}(Q, K, V) = \text{Softmax}\left( \frac{Q K^T}{\sqrt{d_k}} + M \right) V$$

The Causal Mask Matrix $M$ contains 0 for valid past positions and $-\infty$ (Minus Infinity) for future positions:

  Causal Mask Matrix M (4 x 4 Sequence):
  [   0,  -inf,  -inf,  -inf ]   Token 1 can only see Token 1
  [   0,     0,  -inf,  -inf ]   Token 2 can see Token 1, 2
  [   0,     0,     0,  -inf ]   Token 3 can see Token 1, 2, 3
  [   0,     0,     0,     0 ]   Token 4 can see Token 1, 2, 3, 4

When Softmax exponentiates $e^{-\infty}$:

$$e^{-\infty} = 0.0$$

Future token positions receive an exact 0.0 attention weight, making it mathematically impossible for the model to gather information from future tokens!

Why Causal Masking Enables GPU Speed

Without causal masking, training an autoregressive model would require running 1000 separate sequential forward passes.

With causal masking, we pass the entire 1000 word sequence into the Transformer once. The lower triangular mask allows the GPU to compute loss for all 1000 next-token predictions simultaneously in a single parallel step!

Say this out loud

Causal Masking ensures autoregressive language models only attend to past and current tokens. Adding minus infinity to future positions in the attention score matrix forces Softmax to output exact zero weights for future tokens. This prevents models from looking ahead at future answers while enabling full parallel GPU training on long sequences.

Followups to expect

Why does BERT not use Causal Masking? BERT is a Masked Language Model trained to fill in blanks anywhere in a sentence (bidirectional context), so it needs to look at both past and future words simultaneously.
Is Causal Masking used during inference? During real world generation, the model generates tokens one by one step by step. Causal masking is maintained so the KV cache of past tokens remains valid as new tokens are appended.

Check yourself

Question 1 of 3

Why must Causal Masking be applied during parallel training of autoregressive decoder language models like GPT?

More in NLP & Transformers

See all →

The Attention Mechanism5 min Transformer Architecture5 min Tokenization & BPE5 min