Tokenization & BPE

Converting raw text strings into numerical subword tokens using Byte Pair Encoding.

🟡 intermediate5 min readnlpmust-know

Tokenization breaks raw text strings into numerical token IDs that neural networks can process. Character level tokenization creates tiny vocabularies but long sequences. Word level tokenization creates huge vocabularies with frequent out of vocabulary errors. Byte Pair Encoding (BPE) is a subword tokenization algorithm that iteratively merges the most frequent pairs of adjacent bytes or characters. BPE balances vocabulary size and sequence length, handling rare words and code efficiently without out of vocabulary errors.

What is Tokenization?

Neural networks cannot process raw text strings like "Hello world".

Tokenization breaks text into pieces called Tokens and maps each token to a unique integer ID:

  Raw Text: "Unstoppable innovation" ──► [ TOKENIZER ] ──► Token IDs: [ 4521, 8932, 19042 ]

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. CHARACTER LEVEL       │ 2. WORD LEVEL            │ 3. SUBWORD LEVEL (BPE)   │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Break into letters:      │ Break into full words:   │ Break into subwords:     │
│ ['u', 'n', 's', 't'...]  │ ['Unstoppable', 'in...'] │ ['Un', 'stoppable'...]   │
│ Tiny Vocab (~256)        │ Huge Vocab (1 Million+)  │ Optimal Vocab (32k-100k) │
│ Very Long Sequences      │ Out of Vocab Errors!     │ Zero OOV Errors!         │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

How Byte Pair Encoding (BPE) Works

Byte Pair Encoding (Sennrich et al., 2015; Radford et al., 2019) builds a subword vocabulary from data.

Step 1: Initialization

Start with a base vocabulary of individual characters or bytes (all 256 raw byte values).

Step 2: Iterative Merging

Count all adjacent token pairs in your training text corpus. Find the most frequent pair and merge them into a single new token.

  Text Corpus: "l o w e r", "l o w e s t", "l o w"

  Iteration 1: Most frequent pair is ('l', 'o') ──► Merge into new token 'lo'.
  Iteration 2: Most frequent pair is ('lo', 'w') ──► Merge into new token 'low'.
  Iteration 3: Most frequent pair is ('e', 'r')  ──► Merge into new token 'er'.

Repeat this merge step thousands of times until your vocabulary reaches target size (for example, 32,000 tokens for LLaMA 2, or 100,000 tokens for GPT 4).

Byte-Level BPE (tiktoken / GPT 4 Standard)

Instead of operating on Unicode text characters directly (which struggles with non English scripts and emojis), Byte-Level BPE operates on raw UTF 8 bytes.

Since there are only 256 possible byte values, the base vocabulary starts with 256 bytes.

Benefit: Zero Out of Vocabulary (OOV) Errors! Any string in any language, code script, or emoji sequence can be tokenized into bytes.

Say this out loud

Tokenization breaks text strings into numerical token IDs. Word tokenization suffers from huge vocabularies and unknown word errors. Character tokenization creates excessively long sequences. Byte Pair Encoding builds a subword vocabulary by iteratively merging the most frequent adjacent byte pairs, balancing sequence length and handling rare words cleanly.

Followups to expect

What is SentencePiece? A language independent subword tokenizer (using BPE or Unigram) that treats input text as a raw byte stream including spaces, eliminating language specific pre tokenizers.
Why do numbers take more tokens in older tokenizers? Older BPE tokenizers split multi digit numbers like 123456 into individual digit tokens ['1', '2', '3', '4', '5', '6']. Modern tokenizers group numbers into multi digit subwords to improve math performance.

Check yourself

Question 1 of 3

Why is Subword Tokenization (Byte Pair Encoding) preferred over simple word level or character level tokenization in Large Language Models?

More in NLP & Transformers

See all →

The Attention Mechanism5 min Transformer Architecture5 min BERT vs GPT: Encoder vs Decoder5 min