Positional Encodings & RoPE

Why Transformers Need Positional Encodings

Self attention calculates pairwise dot products between token embeddings.

If you shuffle sentence word order, the set of pairwise dot products stays 100 percent identical.

To a raw Transformer without positional encodings:

"Dog bites man" AND "Man bites dog" are indistinguishable!

To make Transformers aware of word order, we must inject Position Signals into token representations.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. SINUSOIDAL (Vaswani)  │ 2. LEARNED ABSOLUTE      │ 3. ROTARY (RoPE)         │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Uses sine and cosine     │ Assigns a trainable      │ Rotates Query and Key    │
│ wave functions of        │ embedding vector to each │ vectors in 2D planes by  │
│ different frequencies.   │ fixed position index.    │ position angles.         │
│ Added to input vectors.  │ Fails on longer context. │ Modern LLM standard!     │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Absolute Sinusoidal Positional Encoding

Original Transformers added static sine and cosine wave values to token embeddings:

$$PE_{(pos, 2i)} = \sin\left( \frac{pos}{10000^{2i/d}} \right)$$

$$PE_{(pos, 2i+1)} = \cos\left( \frac{pos}{10000^{2i/d}} \right)$$

Higher dimensions oscillate slowly while lower dimensions oscillate rapidly, allowing the model to attend to relative positions via trigonometric identities.

Rotary Position Embedding (RoPE - Modern LLM Standard)

Modern LLMs (LLaMA 2, LLaMA 3, Mistral, PaLM) replace absolute additive encodings with Rotary Position Embedding (RoPE).

Instead of adding position vectors to input tokens at the bottom of the network, RoPE rotates Query and Key vectors inside every self attention layer!

  Query Vector q at Position m  ──► Rotate by Angle (m * theta) ──┐
                                                                 ├──► Dot Product q_m · k_n
  Key Vector k at Position n    ──► Rotate by Angle (n * theta) ──┘    Depends ONLY on Relative Distance (m - n)!

The Geometry of RoPE

RoPE pairs up adjacent 2D vector elements $(q_1, q_2)$ and rotates them in 2D planes:

$$R_{\Theta, m}^d q_m = \begin{pmatrix} \cos m\theta_1 & -\sin m\theta_1 & 0 & 0 \ \sin m\theta_1 & \cos m\theta_1 & 0 & 0 \ 0 & 0 & \cos m\theta_2 & -\sin m\theta_2 \ 0 & 0 & \sin m\theta_2 & \cos m\theta_2 \end{pmatrix} \begin{pmatrix} q_1 \ q_2 \ q_3 \ q_4 \end{pmatrix}$$

When computing Query Key dot product:

$$(R_{\Theta, m}^d q)^T (R_{\Theta, n}^d k) = q^T R_{\Theta, n - m}^d k$$

The inner product depends strictly on relative distance $n - m$!

Why RoPE Dominates

1. Relative Distance Awareness: Attention naturally decays as token distance $n - m$ increases.
2. Context Window Extension: Allows scaling context length from 4k to 128k tokens using Rotary Position Interpolation (YaRN) without retraining model weights from scratch.

Say this out loud

Positional Encodings inject word order awareness into self attention. Absolute sinusoidal encodings add sine and cosine waves to input embeddings. Rotary Position Embedding (RoPE) rotates Query and Key vectors in 2D planes by position angles inside attention layers. RoPE makes dot products depend strictly on relative token distance m - n, enabling long context window extensions in models like LLaMA.

Followups to expect

• What is ALiBi (Attention with Linear Biases)? A relative position method that subtracts a linear distance penalty |m - n| directly from raw attention scores before Softmax, offering extrapolation to arbitrary sequence lengths.
• What is RoPE Position Interpolation (YaRN)? Scaling down rotation angles theta by factor s to fit longer context sequences into original pretraining rotation ranges.