BERT vs GPT: Encoder vs Decoder

The Great Architectural Divide

The 2017 Transformer paper introduced both an Encoder and a Decoder.

Researchers quickly discovered that splitting the architecture into specialized halves yielded powerful model families:

  BERT (Encoder Only)                               GPT (Decoder Only)
  Bidirectional Context (Past + Future)             Causal Left-to-Right Context (Past Only)
  [ Word 1 ] ◄───► [ Word 2 ] ◄───► [ Word 3 ]      [ Word 1 ] ──► [ Word 2 ] ──► [ Word 3 ]
  Best for: Understanding, Search, NER             Best for: Text Generation, Chat, Code

1. BERT (Bidirectional Encoder Representations from Transformers)

Developed by Google in 2018.

Uses Bidirectional Self Attention where every token can attend to all other tokens in the sentence simultaneously (both left and right context).

Pretraining Objective: Masked Language Modeling (MLM)

Randomly masks 15 percent of input tokens with a [MASK] token, training the model to fill in the blanks:

Input: "The cat sat on the [MASK] because it was tired."

Output Target: "mat"

• Strengths: Deep bidirectional context understanding. Excellent for search ranking, sentiment classification, named entity recognition, and sentence embeddings.
• Weaknesses: Cannot generate open ended text efficiently.

2. GPT (Generative Pretrained Transformer)

Developed by OpenAI in 2018.

Uses Causal Masked Self Attention where token $t$ can only attend to past tokens $1 \dots t$.

Pretraining Objective: Next Token Prediction (Causal LM)

Predicts the next token given preceding text:

Input: "The cat sat on the"

Output Target: "mat"

• Strengths: Natural autoregressive text generation, chat conversations, code writing, and in context few shot learning.
• Weaknesses: Unidirectional (cannot see future context when encoding tokens).

Comparison Summary Table

1. BERT: Encoder Only. Bidirectional context. Pretraining task is Masked LM. Best for Search, Classification, NER.
2. GPT: Decoder Only. Unidirectional Causal context. Pretraining task is Next Token Prediction. Best for Generation, Chat, Reasoning.
3. T5: Encoder Decoder. Full seq2seq context. Pretraining task is Span Corruption. Best for Translation, Summarization.

Why Decoder Only Architecture Won

In 2018, researchers assumed Encoders were better for understanding and Decoders were only for generation.

By 2024, Decoder Only models (LLaMA 3, GPT 4, Claude 3) proved that scaling Next Token Prediction on Causal Decoders unlocks both deep understanding AND generation in a single unified architecture!

Say this out loud

BERT is an Encoder Only model trained with Masked Language Modeling to look at past and future context simultaneously, excelling at text classification and search embeddings. GPT is a Decoder Only model trained with Causal Next Token Prediction to process text left to right, excelling at text generation, chat, and reasoning.

Followups to expect

1. What is RoBERTa? An optimized version of BERT that removed the Next Sentence Prediction task, trained longer on larger batch sizes with dynamic masking, outperforming original BERT.
2. What is RoFormers / DeBERTa? Disentangled Attention models that separate word content embeddings from relative position embeddings, improving BERT style encoder performance.