LLM Tokenizers Cheatsheet

What is a Tokenizer?

A tokenizer converts raw text into a sequence of discrete units — tokens — that a language model’s neural network can actually process, since neural networks operate on numbers, not characters or words. Each distinct token is mapped to an integer ID, which is then looked up in an embedding table to get the vector the model actually computes with. The tokenizer also does the reverse: converting the model’s output token IDs back into readable text.

The choice of tokenizer matters more than it might seem. It determines the model’s vocabulary size, how many tokens a given piece of text costs (which affects context window usage and inference price), how well the model handles rare words, typos, code, numbers, and non-English languages, and even subtle behaviors like how well a model can count letters in a word or do arithmetic. Almost all modern LLMs use some form of subword tokenization, which sits between two older extremes: splitting text into whole words, or splitting it into individual characters/bytes.

The Spectrum: Word, Subword, and Character/Byte Tokenization

Word-level tokenization

Splits text strictly on whitespace and punctuation, so each token is a complete word ("unbelievable" stays one token).

• Pros: tokens map directly onto intuitive linguistic units, short sequences for a given text.
• Cons: vocabulary explodes to handle every word form and typo; any word not seen during training becomes an unknown/out-of-vocabulary token, which loses information. Largely abandoned in modern LLMs.

Character-level tokenization

Splits text into individual characters ("cat" → c, a, t).

• Pros: tiny, fixed vocabulary; literally no out-of-vocabulary problem.
• Cons: sequences become extremely long for a given amount of text, which is expensive and makes it harder for the model to capture long-range structure. Rarely used on its own in production LLMs.

Byte-level tokenization

Splits text into raw bytes (UTF-8 encoded), so the vocabulary is at most 256 possible values before any merging.

• Pros: can represent absolutely any text, in any language or encoding, with zero out-of-vocabulary tokens — even emoji, malformed text, or unseen scripts.
• Cons: on its own, sequences are very long. In practice byte-level tokenization is almost never used raw — it’s the foundation underneath byte-level BPE (see below), which merges bytes into longer, more efficient tokens.

Subword tokenization

The approach essentially every modern LLM actually uses. Common words stay as a single token ("the"), while rarer or unseen words get broken into smaller, reusable pieces ("unbelievable" → "un", "believ", "able"). This gives the best of both worlds: a manageable, fixed-size vocabulary, no out-of-vocabulary problem (any word can be built from smaller pieces, down to individual characters/bytes if needed), and reasonably short sequences because frequent words/phrases get their own single token.

The rest of this cheatsheet covers the specific subword algorithms and tokenizer families used in practice.

Byte Pair Encoding (BPE)

The most widely used subword tokenization algorithm, originally a data compression technique adapted for NLP.

How it works: Start with a vocabulary of individual characters (or bytes). Count all adjacent pairs of symbols in the training corpus, find the most frequent pair, and merge it into a single new symbol. Repeat this merge step thousands of times, building up a vocabulary of increasingly long subword units, until you hit a target vocabulary size. The final vocabulary is this fixed set of merge rules plus the base symbols.

Byte-level BPE (the variant used by GPT-2, GPT-3, GPT-4, and most of the GPT family) runs this merging process over raw UTF-8 bytes rather than Unicode characters. This guarantees the tokenizer can represent literally any input text without an unknown-token fallback, since every possible byte sequence is reachable.

• Used by: GPT-2, GPT-3, GPT-3.5, GPT-4 (via tiktoken), and many open models.
• Characteristic behavior: common English words and short sequences become single tokens; numbers are often split in somewhat arbitrary multi-digit chunks (a known source of LLM arithmetic errors); leading spaces are typically folded into the following token (e.g. " world" as one token, distinct from "world").

WordPiece

Developed by Google, used in BERT and its descendants. Conceptually similar to BPE — it also builds a vocabulary of subword merges — but instead of merging purely by raw frequency, WordPiece merges the pair that maximizes the likelihood of the training data under a language model, which tends to produce slightly different (often more linguistically sensible) splits than BPE.

• Used by: BERT, DistilBERT, Electra, many encoder-only models.
• Characteristic behavior: subword continuations are marked with a ## prefix, e.g. "playing" → "play", "##ing", making it visually obvious which tokens start a new word versus continue one.

SentencePiece

Not itself a merging algorithm, but a tokenizer framework/library (from Google) that can implement either BPE or a unigram language model underneath it. Its key feature is that it treats the input as a raw stream of Unicode characters, including spaces, rather than pre-splitting on whitespace first — this makes it language-agnostic and particularly well suited to languages without clear word boundaries, like Japanese, Chinese, or Thai.

• Used by: T5, ALBERT, XLNet, Llama (1 and 2), Mistral, many multilingual models.
• Characteristic behavior: spaces are represented explicitly as a special meta-symbol (commonly ▁, an underscore-like character) rather than being implicit, which makes detokenization (reconstructing exact original spacing) fully reversible.

Unigram Language Model Tokenization

A probabilistic alternative to BPE’s greedy merging. Instead of building a vocabulary bottom-up by merging pairs, it starts from a large candidate vocabulary and iteratively prunes it down, keeping the subset of subwords that maximizes the likelihood of the training corpus under a unigram language model. Because it’s probabilistic, a given piece of text can in principle be tokenized multiple valid ways, with the model picking the highest-likelihood segmentation (or sampling among alternatives, used for a regularization trick called subword regularization).

• Used by: often paired with SentencePiece as one of its two supported algorithms (e.g. in T5, ALBERT).
• Characteristic behavior: tends to produce more linguistically natural subword boundaries than greedy BPE merging.

tiktoken (OpenAI’s BPE implementation)

Not a different algorithm so much as OpenAI’s specific, highly optimized byte-level BPE implementation and the specific vocabularies/merge tables (“encodings”) it ships with.

Each new encoding generally expands the vocabulary size (e.g. cl100k_base has ~100k tokens, o200k_base ~200k), which tends to improve token efficiency, especially for non-English text and code, at the cost of a larger embedding table.

Tokenizer Comparison Table

Why Tokenization Choice Matters in Practice

• Cost and context length: API pricing and context window limits are measured in tokens, not characters or words. A tokenizer that splits a given language or domain (e.g. source code, non-English text) less efficiently means more tokens for the same content, which costs more and eats into the available context.
• Multilingual fairness: Byte-level and SentencePiece approaches generally handle non-English languages better than word-level tokenizers, but vocabulary design still matters — a vocabulary trained mostly on English text will still tokenize other languages less efficiently (more tokens per word), which is a well-documented fairness and cost concern for non-English users.
• Numbers and arithmetic: Many BPE tokenizers split multi-digit numbers inconsistently (e.g. "1234" might become "123" + "4" in one context and "12" + "34" in another), which is a commonly cited contributor to LLMs’ struggles with arithmetic. Some newer tokenizers deliberately tokenize digits one at a time to make number handling more consistent.
• Whitespace and formatting sensitivity: Code and structured text (JSON, Markdown, YAML) are sensitive to exact whitespace; tokenizers that handle indentation and repeated spaces efficiently (dedicated tokens for common indentation patterns, as in cl100k_base/o200k_base) tend to perform better on code-heavy tasks.
• Character-level tasks: Because subword tokens often bundle several characters together, LLMs can struggle with tasks that require reasoning about individual characters, such as counting letters in a word or reversing a string — the model never actually “sees” the individual characters as separate tokens.

Quick Reference: Checking Tokenization Yourself

# OpenAI's tiktoken
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("Hello, world!")
print(tokens)                  # list of token IDs
print(enc.decode(tokens))      # back to text
print(len(tokens))             # token count

# Hugging Face tokenizers (works with BPE, WordPiece, SentencePiece models alike)
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tok.tokenize("unbelievable"))   # see the actual subword split
print(tok.encode("unbelievable"))     # token IDs

Created using Anthropic Claude. Verify specifics (exact vocabulary sizes, which encoding a given model version uses) against current model provider documentation, as these details change frequently with new model releases.