Transformer Architecture

The Paradigm Shift: RNNs vs Transformers

Recurrent networks process text sequentially word 1, then word 2, then word 3. Because word 3 requires hidden state $h_2$, training cannot be parallelized across GPU cores.

Transformers (Vaswani et al., 2017) discarded recurrent loops entirely.

Instead of processing words step by step, Transformers feed the entire sequence into GPU memory at once, processing all token relationships simultaneously in parallel.

  RECURRENT PROCESSING (Sequential, Slow on GPUs):
  Word 1 ──► [ Cell ] ──► Word 2 ──► [ Cell ] ──► Word 3 ──► [ Cell ]

  TRANSFORMER PROCESSING (Fully Parallel Matrix Ops on GPUs):
  [ Word 1, Word 2, Word 3, Word 4, Word 5 ] ──► [ MULTI HEAD SELF ATTENTION ] ──► All Context Output!

Complete Transformer Architecture

                 ENCODER BLOCK STACK                            DECODER BLOCK STACK
                 Output Text / Embeddings                       Output Probabilities (Softmax)
                            ▲                                              ▲
                            │                                              │
                 ┌──────────────────────┐                       ┌──────────────────────┐
                 │  Feed Forward (FFN)  │                       │  Feed Forward (FFN)  │
                 ├──────────────────────┤                       ├──────────────────────┤
                 │ Multi Head Attention │                       │ Encoder Decoder Attn │ (Cross Attention!)
                 └──────────────────────┘                       ├──────────────────────┤
                            ▲                                   │ Causal Masked Attn   │ (Masks future words!)
                            │                                   └──────────────────────┘
                            └──────────────────────────────────────────────┘

The Core Building Blocks

1. Positional Encoding

Self attention treats input tokens as an unordered bag of words.

To inject word order awareness, Positional Encodings (sinusoidal curves or learned Rotary Position Encodings) are added directly to input token embeddings.

2. Multi Head Self Attention (MHSA)

Runs multiple parallel attention heads, allowing the network to simultaneously focus on different types of relationships (such as grammatical roles, pronoun references, and subject verb ties).

3. Layer Normalization & Residual Connections

Wrapped around every sub layer:

Output = LayerNorm( x + SubLayer(x) )

Residual connections prevent vanishing gradients, allowing Transformer stacks to scale safely to hundreds of layers.

4. Position Wise Feed Forward Network (FFN)

A two layer dense network applied to each token independently:

FFN(x) = GELU( x W_1 + b_1 ) W_2 + b_2

The FFN acts as a key value memory store holding factual knowledge learned during pretraining.

Encoder vs Decoder Family Split

1. Encoder Only (BERT): Uses bidirectional self attention to look at past and future context simultaneously. Best for text classification, search embeddings, and extraction.
2. Decoder Only (GPT, LLaMA, Claude): Uses Causal Masking to prevent looking at future words. Best for autoregressive text generation.
3. Encoder Decoder (T5, BART): Combines bidirectional encoder with causal decoder. Best for translation and summarization.

Say this out loud

Transformers replaced sequential RNN loops with parallel self attention mechanisms, enabling full GPU parallelization. An Encoder block uses Multi Head Self Attention and Feed Forward layers wrapped in residual connections and LayerNorm. Positional Encodings inject word order information. Decoder models use Causal Masking to generate text autoregressively.

Followups to expect

1. Why is Feed Forward layer inner dimension 4 times larger than model dimension? In standard Transformers with model dimension d_model = 768, the FFN hidden layer expands to d_ff = 3072 (4x expansion) to provide sufficient capacity for storing factual association memories.
2. What is SwiGLU activation in modern Transformer FFN blocks? Replaces standard GELU with Swish Gated Linear Units, improving representation quality in modern LLMs like LLaMA and PaLM.