Machine Translation

Evolution of Machine Translation

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. STATISTICAL (SMT)     │ 2. RECURRENT NMT         │ 3. TRANSFORMER NMT       │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Phrase tables +          │ Encoder-Decoder LSTM     │ Encoder-Decoder          │
│ Language models +        │ with Bahdanau Attention. │ Transformer (NLLB-200).  │
│ Alignment pipelines.     │ End-to-end continuous    │ Parallel GPU training,   │
│ Rigid word translations. │ representation learning. │ state of the art fluency.│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Statistical Machine Translation (SMT)

Before deep learning, systems like early Google Translate used Noisy Channel Models:

$$\arg\max_Y P(Y \mid X) = \arg\max_Y P(X \mid Y) \cdot P(Y)$$

• $P(X \mid Y)$: Translation Model (phrase tables learned from parallel corpora).
• $P(Y)$: Language Model (ensured target language fluency).

SMT required complex pipelines: sentence alignment, tokenization, phrase extraction, and reordering rules. Translations were often robotic and grammatically awkward.

2. Neural Machine Translation (NMT)

Neural NMT replaced multi-step pipelines with a single end-to-end neural network.

The Encoder-Decoder Framework

• Encoder: Reads full source sentence (e.g. English) into continuous contextual representations.
• Decoder: Cross-attends to source representations while generating target sentence (e.g. French) token by token using Causal Masking and Beam Search Decoding.

  Source (English): "The cat sat on the mat." ──► [ ENCODER ]
                                                      │
                                                      ▼ Cross Attention
  Target (French):  [SOS] ──► [ DECODER ] ──► "Le" ──► "chat" ──► "s'est" ──► "assis" ──► [EOS]

Modern NMT: No Language Left Behind (NLLB-200)

Meta's NLLB-200 model translates across 200 different languages using a single massive 54B parameter Encoder-Decoder Transformer with Mixture of Experts (MoE).

Key innovations in modern NMT:

1. Multilingual Shared Vocabulary: A single subword BPE tokenizer covering 200+ languages.
2. Language Tokens: Prepending target language tags (__fra_Latn__) to instruct the decoder which language to generate.

Evaluating Translation Quality

1. BLEU Score: Measures exact n-gram precision overlap between model output and human reference translations.
2. chrF Score: Measures character n-gram overlap, performing better on morphologically rich languages.
3. COMET (Neural Metric): Uses a pre-trained cross-lingual Transformer (XLM-RoBERTa) to predict human quality ratings directly from source, target, and reference embeddings.

Say this out loud

Machine Translation evolved from phrase-based statistical pipelines to end-to-end Neural Machine Translation (NMT). Modern NMT uses Encoder-Decoder Transformers where the encoder reads source text and the decoder uses cross-attention to generate target translations token by token. Quality is evaluated using n-gram overlap metrics like BLEU and neural metrics like COMET.

Followups to expect

1. What is Back-Translation? A data augmentation technique for low-resource languages: take monolingual target text, translate it back to source language using a temporary model, and use the synthetic pair for training.
2. Why is Beam Search preferred over Greedy Decoding for MT? Greedy decoding picks the single highest probability word at step t, which can trap the decoder in bad sentence structures. Beam Search keeps top-K candidate translations alive in parallel.