Static vs Contextual Embeddings
Why modern NLP replaced static dictionary lookup vectors with dynamic context aware transformer embeddings.
The Polysemy Problem in Static Embeddings
Consider the word "bank" in two different sentences:
- "I sat on the grassy bank of the river."
- "I deposited money into my bank account."
To a human, these two uses of "bank" mean completely different things (river shore vs financial institution).
Static Word Embeddings (Word2Vec, GloVe, FastText):
Maintain a single fixed vector dictionary table. Every occurrence of "bank" gets the exact same vector lookup:
Vector("bank") = [ 0.42, -0.19, 0.88, ..., 0.05 ] (Static Dictionary Lookup!)
Because static embeddings assign 1 vector per word string, homonyms and polysemous words get squeezed into a single averaged vector that blurs distinct meanings together.
The Contextual Embedding Revolution (ELMo, BERT)
Contextual Embeddings (BERT, RoBERTa, DeBERTa) generate dynamic vectors that depend directly on surrounding sentence context.
Sentence 1: "Grassy bank of the river" ──► [ BERT Transformer ] ──► Vector("bank") = [ 0.89, 0.12, -0.45 ] (Shore!)
Sentence 2: "Money in bank account" ──► [ BERT Transformer ] ──► Vector("bank") = [-0.12, 0.94, 0.88 ] (Finance!)
Inside BERT, initial static token embeddings pass through 12 to 24 Multi Head Self Attention layers.
As self attention layers process the sentence, surrounding context words ("river", "grassy") mix into the embedding of "bank", shifting its vector towards nature concepts.
Detailed Comparison Matrix
| Dimension | Static Embeddings (Word2Vec, GloVe) | Contextual Embeddings (BERT, RoBERTa) |
|---|---|---|
| Vector Mapping | 1 fixed vector per word | Dynamic vector per word per context |
| Polysemy Support | Fails (Blurs multiple word meanings) | Excellent (Adapts vector to context) |
| Inference Speed | Ultra-Fast $O(1)$ Hash Lookup | Requires Deep Neural Network forward pass |
| Memory Footprint | Small (Embedding matrix table) | Large (Full Transformer weights + VRAM) |
| Out of Vocabulary | Fails unless using subwords | Handled natively via BPE subwords |
Say this out loud
Static embeddings assign a single fixed vector to each word string regardless of context, blurring distinct meanings of polysemous words like bank or apple. Contextual embeddings use Transformer self attention layers to dynamically update word vectors based on surrounding sentence words, resolving polysemy and capturing subtle semantic context.
Followups to expect
- What was ELMo (Peters et al., 2018)? The first popular contextual embedding model, using a Bidirectional LSTM to combine left to right and right to left context representations before Transformers took over.
- When are Static Embeddings still useful today? Ultra low latency edge applications, fast keyword clustering, initial candidates retrieval in lightweight search, and resource constrained embedded hardware.
Check yourself
Why does a Static Embedding model like Word2Vec fail on words with multiple meanings (polysemy)?