Word2Vec, GloVe & Embeddings

From One Hot Vectors to Dense Embeddings

Imagine representing a 50,000 word vocabulary using One Hot Encoding:

• "cat" = [ 1, 0, 0, 0, ..., 0 ]
• "dog" = [ 0, 1, 0, 0, ..., 0 ]

Problems with One Hot Vectors:

1. Extremely Sparse and Large: Vector dimension equals vocabulary size ($N = 50,000$).
2. Zero Semantic Similarity: Dot product between any two distinct one hot vectors is always 0.0. The computer thinks "cat" and "dog" are as completely unrelated as "cat" and "airplane".

Word Embeddings project words into a dense $d$-dimensional space ($d = 100\text{--}300$), placing semantically related words close together:

  Vector Space Geometry:
  Vector("cat")   = [ 0.25, -0.81, 0.44, 0.12 ]
  Vector("dog")   = [ 0.28, -0.79, 0.41, 0.15 ]  ──► High Cosine Similarity!
  Vector("table") = [-0.91,  0.11, 0.05, 0.88 ]  ──► Low Cosine Similarity!

1. Word2Vec (Mikolov et al., 2013 - Google)

Word2Vec popularized the concept: "A word is characterized by the company it keeps."

It trains a shallow neural network on raw unlabelled text using two architectures:

┌──────────────────────────┬──────────────────────────┐
│ 1. CBOW (Bag of Words)   │ 2. SKIP GRAM             │
├──────────────────────────┼──────────────────────────┤
│ Predicts center word     │ Predicts surrounding     │
│ given context words.     │ context words given      │
│ Faster training.         │ center target word.      │
│ Great for frequent words.│ Great for rare words!    │
└──────────────────────────┴──────────────────────────┘

Famous Vector Analogy Property

Word2Vec embeddings capture linear directional semantic concepts:

$$\text{Vector("King") - Vector("Man") + Vector("Woman")} \approx \text{Vector("Queen")}$$

$$\text{Vector("Paris") - Vector("France") + Vector("Germany")} \approx \text{Vector("Berlin")}$$

2. GloVe (Global Vectors - Stanford 2014)

Word2Vec processes text using local sliding windows.

GloVe (Global Vectors) combines local window context with Global Word Co Occurrence Matrix Statistics across the entire corpus.

It minimizes a weighted least squares objective fitting log co occurrence counts:

$$J = \sum_{i,j=1}^V f(X_{ij}) \left( w_i^T \tilde{w}_j + b_i + \tilde{b}j - \log X{ij} \right)^2$$

Say this out loud

Word Embeddings map words into dense continuous vector spaces where geometric distance reflects semantic meaning. One hot vectors are sparse and treat all words as orthogonal. Word2Vec uses CBOW and Skip Gram context prediction to capture semantic relationships like King minus Man plus Woman equals Queen. GloVe combines local context with global matrix co occurrence statistics.

Followups to expect

1. What is FastText (Facebook 2016)? An extension of Word2Vec that represents words as bags of character n-grams (for example, "apple" = ["app", "ppl", "ple"]), enabling embeddings and out of vocabulary handling for rare or misspelled words.
2. What is the main limitation of Word2Vec and GloVe? They produce Static Embeddings. The word "bank" receives the exact same fixed vector whether used in "river bank" or "bank account". This led to Contextual Embeddings (BERT).