Seq2Seq & Encoder–Decoder

What is Seq2Seq?

Standard classifiers map fixed input vectors to fixed output labels.

However, many real world NLP tasks require converting a variable length input sequence into a different variable length output sequence:

1. Machine Translation: Translating a 7 word English sentence into a 9 word French sentence.
2. Text Summarization: Compressing a 500 word news article into a 20 word summary.
3. Speech to Text: Converting a 5 second audio clip into a written transcript sentence.

Sequence to Sequence (Seq2Seq) handles this using two connected neural networks: an Encoder and a Decoder.

  ENCODER (Compresses Input Sequence)                 DECODER (Generates Output Sequence)
  "How" ──► [ Cell ]
              │
  "are" ──► [ Cell ]
              │
  "you" ──► [ Cell ] ──► Context Vector c ──► [ Cell ] ──► "Comment"
                                                │
                                              [ Cell ] ──► "allez"
                                                │
                                              [ Cell ] ──► "vous" ──► [EOS] Token!

1. The Encoder

The Encoder processes input tokens $(x_1, x_2, \dots, x_T)$ step by step.

Its goal is to condense the semantic meaning of the entire input sentence into a single summary vector called the Context Vector ($c$) (typically the final hidden state of an RNN, LSTM, or GRU).

2. The Decoder

The Decoder takes the Context Vector $c$ as its initial state and generates output tokens $(y_1, y_2, \dots, y_{T'})$ one by one.

At each step:

1. Predicts the next output token.
2. Feeds its own predicted token back as input for the next step (Autoregressive Generation).
3. Continues generating until it outputs a special End of Sequence [EOS] Token.

The Context Vector Bottleneck

Early Seq2Seq models forced the Encoder to compress an entire 60 word sentence into a single fixed size vector of 512 numbers.

This created an information bottleneck. As sentences grew longer, performance dropped sharply because early sentence details were squeezed out.

This exact bottleneck motivated Bahdanau in 2014 to invent the Attention Mechanism, allowing the Decoder to look back at all individual Encoder hidden states directly.

Say this out loud

Seq2Seq converts variable length input sequences into variable length output sequences using an Encoder and Decoder. The Encoder compresses input tokens into a Context Vector. The Decoder uses that vector to generate output tokens one by one until reaching an End of Sequence token. Attention was invented to solve the Context Vector bottleneck on long sentences.

Followups to expect

1. What is Auto-Regressive Decoding? Generating text token by token where each new output token is fed back into the model as input for predicting the subsequent token.
2. What is the difference between Greedy Decoding and Beam Search in Seq2Seq? Greedy decoding picks the single highest probability token at each step. Beam Search tracks multiple top candidate sentences in parallel to find globally higher probability output sentences.