The Attention Mechanism

Why Was Attention Invented?

Early Sequence to Sequence models forced an Encoder to squeeze an entire 50 word sentence into a single fixed size vector (for example, 512 numbers).

This created an information bottleneck. By the time the Decoder reached word 10 of its output, early details from the input sentence were lost.

Bahdanau et al. (2014) solved this by introducing Attention:

Instead of relying on a single static summary vector, the Decoder dynamically looks back at all hidden state vectors of the input sequence, assigning high attention weights to the specific words most relevant to the current output step.

  Decoder Generating Word "French":
  "The"   (Attention Weight = 0.05)
  "white" (Attention Weight = 0.10)
  "cat"   (Attention Weight = 0.85 ──► HIGH ATTENTION FOCUS!)

The Three Attention Vectors: Query, Key, Value

Vaswani et al. (2017) formalized attention using an intuitive library search analogy:

1. Query (Q): What am I searching for? (The current token looking for context).
2. Key (K): What index label do I hold? (Every sequence token advertising its identity).
3. Value (V): What actual content do I contain? (The information payload delivered if selected).

  1. Measure Similarity: Score = Query · Key^T
  2. Normalize Scores:   Weights = Softmax( Score / sqrt(d_k) )
  3. Blend Content:     Context Output = sum( Weights * Values )

Scaled Dot Product Attention Math

$$\text{Attention}(Q, K, V) = \text{Softmax}\left( \frac{Q K^T}{\sqrt{d_k}} \right) V$$

• $Q K^T$: Matrix of dot products measuring pairwise directional alignment between all queries and keys.
• $\sqrt{d_k}$: Scaling factor dividing dot products by the square root of key dimension size $d_k$.
• $\text{Softmax}(\cdot)$: Converts raw similarity scores into positive weights summing to 1.0.
• Multiply by $V$: Computes a weighted average sum of Value vectors.

Why Divide by $\sqrt{d_k}$?

When vector dimension $d_k$ is large (such as 128), dot products $Q K^T$ grow very large in magnitude.

Large inputs into Softmax push probability outputs into extreme 0.0 and 1.0 regions where derivatives shrink to near zero (Vanishing Gradients).

Dividing by $\sqrt{d_k}$ keeps variance stable at 1.0, ensuring Softmax gradients remain healthy during backpropagation.

Say this out loud

The Attention Mechanism allows models to dynamically focus on relevant context across a sequence. It projects inputs into Query, Key, and Value vectors. Scaled Dot Product Attention computes Softmax( Q K^T / sqrt(d_k) ) V. Dividing by sqrt(d_k) prevents dot products from growing excessively large, maintaining healthy Softmax gradients during training.

Followups to expect

1. What is the difference between Self Attention and Cross Attention? In Self Attention, Queries, Keys, and Values all come from the same sequence. In Cross Attention, Queries come from one sequence (Decoder) while Keys and Values come from another (Encoder).
2. What is the computational complexity of Self Attention? Self Attention compares every token to every other token, resulting in $O(N^2)$ quadratic computational complexity with respect to sequence length $N$.