Chat Templates

What is a Chat Template?

Large Language Models do not natively understand python dictionaries or message roles (system, user, assistant).

At the hardware level, an LLM takes a single flat string of text tokens.

Chat Templates define the exact rules for converting structured conversation history into a single formatted text string:

# Raw Conversation Data
messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Write a Python hello world script."}
]

Converted to ChatML Formatted String:

<|im_start|>system
You are a helpful coding assistant.<|im_end|>
<|im_start|>user
Write a Python hello world script.<|im_end|>
<|im_start|>assistant

Why Chat Templates Matter

Different LLM families use completely different control token formats:

1. OpenAI ChatML: <|im_start|>user \n Hello <|im_end|>
2. LLaMA 3: <|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|>
3. Mistral / LLaMA 2: [INST] Hello [/INST]

If you fine tune a model on LLaMA 3 headers, but serve inference using ChatML headers:

The model will fail to recognize user turns, generating garbled gibberish or echoing inputs.

HuggingFace Jinja2 Template Integration

HuggingFace tokenizers store the model's exact Jinja2 prompt template inside tokenizer.chat_template.

Using tokenizer.apply_chat_template() guarantees your prompt formatting matches training 100 percent perfectly:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is 2+2?"}
]

# Formats string using model exact Jinja template!
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True # Appends assistant header to end!
)

print(prompt)
# Output: <|begin_of_text|><|start_header_id|>system<|end_header_id|>...

Key Jinja Parameters

1. tokenize = False: Returns a human readable formatted string.
2. tokenize = True: Returns pre tokenized PyTorch tensors ready for model forward passes.
3. add_generation_prompt = True: Appends the assistant turn header (e.g. <|start_header_id|>assistant<|end_header_id|>\n\n) to the end of the prompt, instructing the model to start generating its response.

Say this out loud

Chat Templates standardize how multi turn conversation messages are formatted into single raw prompt strings. Different LLM families use unique control tokens like ChatML or LLaMA headers to mark speaker turn boundaries. HuggingFace apply_chat_template formats message dictionaries using Jinja templates, preventing formatting mismatches during fine tuning and inference.

Followups to expect

1. What happens if special control tokens are not added to tokenizer special_tokens? If control tokens like <|im_start|> are not registered as special tokens, the tokenizer splits them into subword pieces (["<|", "im", "_", "start", "|>"]), breaking control signals.
2. How do chat templates prevent Prompt Injection? Explicit control token boundaries prevent malicious user text from faking assistant or system headers.