Self-Attention vs Cross-Attention

Comparing Attention Sources

Attention always calculates:

Attention( Q, K, V ) = Softmax( Q K^T / sqrt(d_k) ) V

The fundamental difference between Self Attention and Cross Attention is where Q, K, and V come from.

┌──────────────────────────┬──────────────────────────┐
│  1. SELF ATTENTION       │  2. CROSS ATTENTION      │
├──────────────────────────┼──────────────────────────┤
│ Q, K, V come from the    │ Q comes from Target seq. │
│ SAME input sequence.     │ K, V come from SOURCE    │
│ Measures internal token  │ external sequence.       │
│ relationships.           │ Pulls context from outside│
└──────────────────────────┴──────────────────────────┘

1. Self Attention (Internal Relationships)

In Self Attention, an input sequence $X = [x_1, x_2, \dots, x_N]$ is multiplied by three separate projection weight matrices ($W_Q, W_K, W_V$):

Q = X * W_Q, K = X * W_K, V = X * W_V

Every token in the sequence compares itself to every other token in the same sequence.

Example

In the sentence "The animal didn't cross the street because it was too tired":

Self Attention allows the word token "it" to compute high similarity with "animal", resolving pronoun coreference within the sentence.

2. Cross Attention (Inter Sequence Retrieval)

In Cross Attention, there are two distinct sequences:

1. Target Sequence ($Y$): Generates Query vectors $Q = Y W_Q$.
2. Source Sequence ($X$): Generates Key vectors $K = X W_K$ and Value vectors $V = X W_V$.

The Target sequence uses its Queries to look up and pull relevant information from the Source sequence Keys and Values.

Real World Applications

1. Seq2Seq Translation (T5): French Decoder tokens (Queries) look up English Encoder tokens (Keys and Values) to select translated word meanings.
2. Text to Image Generation (Stable Diffusion): Image feature map patches (Queries) look up CLIP text prompt embeddings (Keys and Values) to steer image generation toward prompt keywords.
3. Multimodal Vision LLMs (LLaVA): Language model text queries attend to Vision Transformer image features.

Say this out loud

In Self Attention, Queries, Keys, and Values all come from the same input sequence to learn internal token relationships. In Cross Attention, Queries come from a target sequence while Keys and Values come from a separate source sequence. Cross Attention is used in translation decoders to look up encoder words and in text to image models to guide image generation.

Followups to expect

1. Can Cross Attention be used when source and target have different sequence lengths? Yes. Queries have target length N_target, while Keys and Values have source length N_source. Output tensor shape matches target length N_target.
2. Why isn't Cross Attention used in Decoder Only LLMs like GPT 4? Decoder only models use single sequences where prompt text and generated text are concatenated together into a single stream using Causal Self Attention.