Special Tokens in LLMs Cheatsheet

What are Special Tokens?

Special tokens are reserved vocabulary entries that don’t represent ordinary words or subwords — they exist purely to carry structural, control, or metadata signals that the model needs to understand the shape of its input and output. A regular token like "cat" carries semantic meaning; a special token like <|endoftext|> carries no linguistic meaning at all, but tells the model (or the surrounding system) something about the structure of the sequence: where a document ends, where a user’s turn starts, which part of the input is a system instruction versus user content, or which part should be ignored during loss calculation.

Special tokens are added to the tokenizer’s vocabulary at training time and are reserved IDs that the BPE/WordPiece/SentencePiece merge process never produces by accident from ordinary text — they’re inserted deliberately by the code that prepares training data or chat prompts, not discovered through the normal tokenization of a string. This is also why prompt injection defenses often care about special tokens: if a tokenizer naively tokenizes raw user text and that text happens to contain the literal string <|endoftext|>, a poorly guarded pipeline might accidentally let a user inject what looks like a structural boundary. Well-built tokenizers either disallow this or escape it.

Why Models Need Them

Without special tokens, a language model only ever sees an undifferentiated stream of subword tokens — it has no built-in notion of “this is the beginning of a document,” “this is the system prompt versus what the user typed,” or “stop generating now.” Special tokens give the model and the surrounding application a shared, unambiguous vocabulary for:

• marking the boundaries of documents, sequences, or conversation turns
• separating distinct roles in a chat (system, user, assistant, tool)
• padding batches of different-length sequences to a uniform length
• marking a position whose true value is hidden, to train models to predict it
• signaling tool/function calls and their outputs
• telling the model (or the sampler) when generation should stop

Common Special Tokens Across Model Families

Exact names vary by model family and tokenizer library — the concepts above recur across almost all of them even when the literal strings differ.

Chat/Instruction Special Tokens

Modern chat-tuned LLMs rely heavily on special tokens to encode conversational structure, since the raw model only sees one long token stream and needs explicit markers to know “this part is the system prompt,” “this part is what the user said,” “this part is what I (the assistant) am supposed to generate.” This formatting is usually applied by a chat template (e.g. Hugging Face’s apply_chat_template), not typed by the user directly.

A ChatML-style example (used by OpenAI models and adopted by many open models):

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What's the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>

A Llama-style example (Llama 2/3 chat formatting, simplified):

<s>[INST] <<SYS>>
You are a helpful assistant.
<</SYS>>

What's the capital of France? [/INST] The capital of France is Paris. </s>

These role-boundary tokens let the model’s training data clearly distinguish instructions it should follow (system/user turns) from text it generated itself (assistant turns), and let inference-serving code know exactly where to stop generating and hand control back to the user.

Tool / Function-Calling Tokens

Newer models that support structured tool use often define dedicated special tokens to mark where a tool call begins, what the call looks like, and where the tool’s result is inserted back into the conversation — keeping this machine-readable structure clearly separated from natural-language text. Examples include tokens analogous to <|tool_call|> / <|tool_call_end|> or model-specific equivalents, though exact token names differ by provider and change frequently as function-calling formats evolve, so it’s worth checking a given model’s current documentation rather than assuming a fixed convention here.

Stop Tokens / End-of-Sequence Tokens

A stop token (often the same as the end-of-sequence token, e.g. <|endoftext|>, </s>, or <|im_end|>) tells the generation loop to halt. During inference, the sampler checks after each generated token whether it matches one of the configured stop tokens/strings, and if so, ends the response. APIs typically also let you supply custom stop sequences on top of the model’s built-in ones, which is useful for constraining output format (e.g. stopping as soon as the model emits a specific delimiter you defined).

Reserved / Unused Token Slots

Many tokenizer vocabularies deliberately include a block of unused/reserved token IDs (e.g. <extra_id_0> through <extra_id_99> in T5, or a range of <|reserved_special_token_N|> entries in some Llama tokenizer configs). These exist so that developers fine-tuning the model for a new structural purpose (a new role, a new delimiter, a new control signal) can repurpose an already-reserved ID rather than expanding the vocabulary and resizing the model’s embedding matrix, which is a more invasive change.

Special Tokens and Attention Masking

Padding tokens in particular interact with the attention mechanism: when sequences of different lengths are batched together, shorter sequences are padded out to match the longest one, and an accompanying attention mask tells the model which positions are real content versus padding, so the padding tokens don’t influence the real tokens’ computed representations or contribute to the loss.

Practical Notes

• Special tokens occupy real vocabulary slots and real embedding rows, so adding new ones (e.g. when fine-tuning a base model for a new chat format) requires resizing the model’s embedding and output layers, not just editing the tokenizer’s vocabulary file.
• Chat templates, not end users, are responsible for inserting role/turn special tokens correctly — getting this wrong (or letting user-supplied text contain literal special-token strings that get parsed as if they were structural) is a known class of prompt injection vulnerability.
• Not every tokenizer uses the same token names for the same concept — always check a specific model’s tokenizer config (e.g. tokenizer_config.json or special_tokens_map.json in a Hugging Face repo) rather than assuming the names above apply universally.

Quick Reference: Inspecting Special Tokens Yourself

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tok.special_tokens_map)
# e.g. {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]',
#       'cls_token': '[CLS]', 'mask_token': '[MASK]'}

print(tok.all_special_tokens)   # full list of special token strings
print(tok.all_special_ids)      # their corresponding token IDs

# OpenAI's tiktoken
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(enc.special_tokens_set)   # e.g. {'<|endoftext|>', ...}

Created using Anthropic Claude. Exact special token names, counts, and formats vary by model and change as new models are released — verify against the specific model’s current tokenizer documentation before relying on these details.