Tokenizer Pitfalls (Numbers, Code, Unicode)

The Invisible Boundary Problem

Developers often think Large Language Models read text character by character like humans.

They do not. LLMs see a sequence of numerical Token IDs.

Because tokenization happens before the neural network starts, tokenizer quirks create hard limits on model capabilities.

  Human Perception:   S - T - R - A - W - B - E - R - R - Y (10 Characters)
  LLM Perception:     Token ID 12450 ("straw") + Token ID 452 ("berry") (2 Tokens!)

Top 4 Tokenizer Pitfalls

┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. CHARACTER COUNTING    │ 2. ARITHMETIC SPLITTING  │ 3. MULTILINGUAL INFLATION│ 4. CODE INDENTATION      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ LLMs cannot count letters│ Numbers get split into   │ Non Latin scripts take   │ Leading spaces get merged│
│ inside subwords because  │ arbitrary digit chunks,  │ 3x to 4x more tokens,    │ inconsistently, altering │
│ letters are invisible.   │ ruining math alignment.  │ inflating API costs.     │ code generation logic.   │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Character & Spelling Blindness

Question: "How many R's are in strawberry?"

Why LLMs fail: The tokenizer groups the letters into ["straw", "berry"]. The model never sees the individual letters s-t-r-a-w-b-e-r-r-y unless forced to spell the word out letter by letter first.

2. Arithmetic & Number Splitting

In older tokenizers (like GPT 2 and GPT 3):

• 100 might be tokenized as ["100"] (1 token).
• 1000 might be tokenized as ["10", "00"] (2 tokens).
• 10000 might be tokenized as ["100", "00"] (2 tokens).

Because numbers were split into arbitrary non uniform chunks, addition and multiplication alignment broke.

Modern Fix: Tokenizers like LLaMA 3 and tiktoken use explicit regex rules to group digits into consistent 1 or 3 digit subwords.

3. Multilingual Token Inflation

English text averages 1.3 tokens per word.

Languages using non Latin scripts (Hindi, Arabic, Chinese, Japanese) often require 3.0 to 5.0 tokens per word because their UTF 8 bytes are split into tiny subword pieces.

Consequences:

1. Multilingual users pay 3x to 4x higher API costs for the exact same message length.
2. Effective context window size is cut by 70 percent for non English conversations.

4. Trailing Space Sensitivity

In BPE tokenizers, spaces are merged into tokens:

• Token ID 234: " hello" (Space before hello).
• Token ID 891: "hello" (No space).

If a prompt ends with a trailing space ("Tell me about cats "), the final token becomes a standalone space token (" ").

This prevents the model from generating natural word continuations that rely on leading space token merges!

Say this out loud

Tokenizers convert text into numerical subword tokens before the model sees them. This causes character blindness where LLMs struggle to count letters inside words. Inconsistent number splitting degrades math reasoning, while non Latin scripts suffer from token inflation that increases API costs. Ending prompts with trailing spaces alters token context.

Followups to expect

1. How do you fix character counting in LLM prompts? Ask the model to spell the word out with hyphens or spaces first (s - t - r - a - w - b - e - r - r - y), forcing the tokenizer to output individual character tokens.
2. What is Tokenizer Vocabulary Size Tradeoff? Larger vocabularies (128k tokens) shrink sequence length and lower API costs for non English text, but increase embedding layer memory parameters.