Masked vs Causal Language Modelling

Comparing Pretraining Paradigms

Before supervised fine tuning, models learn language patterns from unlabelled text using Self Supervised Objectives:

┌──────────────────────────┬──────────────────────────┐
│ 1. MASKED LM (BERT)      │ 2. CAUSAL LM (GPT)       │
├──────────────────────────┼──────────────────────────┤
│ Hides 15% random words.  │ Hides all future words.  │
│ Uses Bidirectional       │ Uses Unidirectional      │
│ context (Past + Future). │ left to right context.   │
│ Task: Fill in blanks.    │ Task: Predict next word. │
└──────────────────────────┴──────────────────────────┘

1. Masked Language Modeling (MLM)

In MLM, 15 percent of input tokens are randomly replaced with a [MASK] token.

The model reads surrounding left and right context words to predict the missing original word:

Input: "Paris is the [MASK] of France."

Target: "capital"

  Left Context ("Paris is the")  ──┐
                                  ├──► [ BERT Encoder ] ──► Predicts "capital" (Score = 0.98!)
  Right Context ("of France")   ──┘

• Pros: Learns rich bidirectional representations of language.
• Cons: Pretraining relies on artificial [MASK] tokens that never appear in real downstream datasets. Cannot generate open ended continuous text.

2. Causal Language Modeling (CLM)

In CLM, the model receives a sequence of tokens and predicts the very next token:

Input: "Paris is the"

Target: "capital"

Using Causal Masking, the model calculates predictions for all sequence positions simultaneously during GPU training:

  Position 1: "Paris"       ──► Predicts "is"
  Position 2: "is"          ──► Predicts "the"
  Position 3: "the"         ──► Predicts "capital"
  Position 4: "capital"     ──► Predicts "of"

• Pros: Perfectly aligns with real world text generation, instruction following, and reasoning. Zero artificial mask token mismatch.
• Cons: Unidirectional context (cannot see future words when processing a token).

Say this out loud

Masked Language Modeling (BERT) hides random words and uses bidirectional context to fill in blanks, excelling at text understanding but failing at text generation. Causal Language Modeling (GPT) hides future words using causal masking and predicts the next token left to right, matching real world text generation and reasoning.

Followups to expect

1. What is Permutation Language Modeling (XLNet)? Trains on random permutations of input token order, combining the bidirectional advantages of MLM with the autoregressive advantages of CLM.
2. What is Prefix LM? A hybrid objective used in models like PaLM where early prompt tokens use bidirectional attention while generated response tokens use causal attention.