Glossary
Every term and acronym used in this book, in plain language. Where a term is the everyday name for an idea explained in a chapter, the chapter is noted.
Activated parameters — In a mixture-of-experts model, the small slice of the model actually switched on for a single word (the router plus the few chosen experts). Determines speed and running cost. Contrast with total parameters. (Chapter 7)
Alignment (of a model) — The work of making a model behave the way people want: helpful, honest, well-mannered. Done through the shaping in Chapter 10. (Not to be confused with the translation sense below.)
Alignment (in translation) — The soft matching, learned automatically by cross-attention, between each output word and the input words it draws from. (Chapter 6)
Attention — The mechanism by which each word surveys the whole text, decides which other words are relevant to it, and pulls in information from them. The core idea behind nearly every model in this book. (Chapter 3)
Autoregressive — Generating output one piece at a time, each piece conditioned on those before it. The way language models write. (Chapters 4, 8)
Backpropagation — The efficient method that computes, in a single backward sweep, which way to nudge every one of a model’s knobs to reduce its error. The engine of learning. (Chapter 9)
Base model — A freshly trained language model: fluent and knowledgeable but not yet shaped into a helpful assistant. (Chapter 10)
BART — An encoder–decoder model trained by repairing many kinds of deliberately damaged text; strong at summarization. (Chapter 6)
BERT — The original and most famous encoder (reader) model. (Chapter 5)
Bidirectional — Able to look in both directions at once, at the words before and after. The defining trait of a reader. Opposite of causal. (Chapter 5)
Causal — Restricted to looking only backward, at what came before. The rule a writer follows so it cannot peek at the answer it is trying to predict. (Chapter 4)
Chinchilla — The study that showed model size and training data should grow in balance — roughly, that earlier giant models had been under-fed with data. (Chapter 9)
Context window — The amount of text a model can take in and attend to at once. Limited largely by the cost of attention and the growing cache. (Chapters 3, 11)
Contextual representation — The rich, context-aware bundle of numbers an encoder produces for each word; a meaning-fingerprint shaped by the whole sentence. (Chapter 5)
Cross-attention — Attention where the questions come from the writer (the half-finished output) and the answers come from the reader (the memory of the input). The one new mechanism in encoder–decoder models, and the same mechanism that lets a diffusion model follow a text prompt. (Chapters 6, 8)
DALL·E — A well-known text-to-image diffusion model. (Chapter 8)
DDIM / DDPM — Two methods for generating images by running a diffusion model’s reverse process; DDPM is the original many-step method, DDIM a faster scheme that can skip most steps. (Chapter 8)
Decoder model — A writer: generates text left to right, one word at a time. GPT, Llama, Claude. (Chapter 4)
Dense model — An ordinary model that uses all of itself for every word. Contrast with mixture-of-experts. (Chapter 7)
Diffusion model — An image (or audio, video, etc.) generator that learns to predict the noise in a corrupted sample and generates by removing noise step by step, starting from pure static. (Chapter 8)
DPO — A simpler way to teach a model from human preferences, training directly on preferred-versus-rejected answer pairs without a separate reward model. (Chapter 10)
Embedding — The list of numbers that places a token at a location on the “map of meaning,” where nearness means similarity. Also, a single such vector summarizing a whole sentence or document, used for search. (Chapters 2, 5)
Encoder model — A reader: takes in text bidirectionally and produces understanding, but cannot generate. The BERT family. (Chapter 5)
Encoder–decoder model — A translator: a reader bolted to a writer, for turning one text into another. T5, BART. (Chapter 6)
Expert — In a mixture-of-experts model, one of many copies of the per-word processing stage, each with its own learned settings. (Chapter 7)
Expert collapse — The failure where the router lazily routes most words to a few favorite experts, leaving the rest as dead weight. Prevented by load balancing. (Chapter 7)
Fine-tuning — Taking a broadly trained model and adapting it, comparatively cheaply, to a specific task or behavior. (Chapters 5, 10)
GLUE / SuperGLUE — Standard collections of language-understanding tests used to grade reader models. (Chapter 5)
GPT — A prominent family of decoder (writer) models. (Chapter 4)
Gradient descent — Learning by repeatedly stepping “downhill” across a landscape of possible knob settings toward lower error. (Chapter 9)
Guidance (classifier-free guidance) — The control that exaggerates how strongly a diffusion model follows its prompt; the “prompt strength” slider in image tools. (Chapter 8)
Head — A small attachment bolted onto a model to turn its internal output into a specific kind of answer (a category, a label per word, a highlighted span). (Chapter 5)
Inference — Using a trained, frozen model to produce outputs. Contrast with training. (“Training writes; using reads.”) (Chapters 1, 11)
KV cache — The saved working memory that lets a model avoid re-reading the whole conversation for each new word. Grows with conversation length, and is the main reason long contexts are expensive. (Chapter 11)
Latent diffusion — Running diffusion in a compressed “essence” space rather than on full-resolution pixels, which is what made high-quality image generation affordable. (Chapter 8)
Learning rate — The size of each downhill step during training; the single most consequential training dial. (Chapter 9)
Llama — A widely used family of openly available decoder models, known for being relatively small and heavily trained. (Chapters 4, 9)
Load balancing — The extra pressure during mixture-of-experts training that pushes the router to spread work evenly across experts. (Chapter 7)
LoRA — A cheap fine-tuning shortcut that freezes the model and learns only a small add-on patch. (Chapter 10)
Loss — The single number scoring how wrong a model’s output was; what training tries to reduce. (Chapter 9)
Map of meaning — This book’s informal name for the high-dimensional space of embeddings, where location encodes meaning. (Chapter 2)
Masked language modeling — The fill-in-the-blank training game for readers: hide some words and have the model recover them from context. (Chapter 5)
Mixture-of-experts (MoE) — A model design with many experts and a router that activates only a few per word, decoupling total knowledge from per-word cost. Switch, Mixtral, DeepSeek. (Chapter 7)
Momentum — A refinement to gradient descent that builds up speed along consistent directions and cancels aimless jitter. (Chapter 9)
Normalization — Plumbing that rescales numbers at each layer to keep a deep model stable. (Chapters 4, 9)
Parameters — The learned numbers — the “knobs” — that define a model. (“Seventy billion parameters” counts these.) (Chapter 1)
Perplexity — A friendly restatement of the loss for language models: roughly, how many words the model is effectively choosing between at each step. (Chapter 9)
Prefix language model — A middle-ground design using one stack that lets the input portion look around freely while the output portion stays left-to-right. (Chapter 6)
Pretraining — The initial, broad, expensive training of a model on vast data, before any task-specific adaptation. (Chapters 5, 9)
Prompt — The input you give a model: the text for a language model, or the description for an image model. (Chapters 8, 10)
Reward model — A model trained to predict human preferences, used as an automated stand-in for human taste when shaping an assistant. (Chapter 10)
RLHF (reinforcement learning from human feedback) — Shaping a model using human comparisons of its answers, usually via a reward model, kept on a leash so it cannot game the score. (Chapter 10)
RoBERTa — A retrained BERT that showed the original had simply been undertrained; same design, much better results. (Chapter 5)
Router (gating network) — The small component in a mixture-of-experts model that decides which experts each word goes to. (Chapter 7)
Sampling — Choosing the next word by rolling dice weighted by the model’s odds, rather than always taking the most likely one. (Chapter 11)
Score (in diffusion) — The direction, at any point, that points toward more realistic images; predicting noise turns out to be equivalent to estimating this. (Chapter 8)
Self-attention — Attention in which a sequence asks questions of itself (as opposed to cross-attention). (Chapters 3, 6)
Sequence-to-sequence (seq2seq) — The “read one thing, write a different thing” task shape, e.g. translation and summarization. (Chapter 6)
SFT (supervised fine-tuning) — Teaching a base model the shape of being helpful by training it on curated example answers. (Chapter 10)
Stable Diffusion — A widely used, efficient text-to-image diffusion model, made practical by latent diffusion. (Chapter 8)
T5 — An encoder–decoder model built on the idea that every task is “text in, text out.” (Chapter 6)
Temperature — The master creativity dial during generation: low for conservative and predictable, high for adventurous and surprising. (Chapter 11)
Token — A chunk of text (usually a common word-fragment) — the unit a model actually processes. (Chapter 2)
Tokenization — The fixed, pre-built recipe that chops text into tokens. (Chapter 2)
Top-k / top-p — Guardrails that discard unlikely options before the model samples a word, keeping only the most plausible handful. (Chapter 11)
Total parameters — In a mixture-of-experts model, everything the model stores across all experts — its full capacity. Contrast with activated parameters. (Chapter 7)
Training — Building a model by adjusting its knobs on vast data; the only time the model changes. (“Training writes; using reads.”) (Chapters 1, 9)
Transformer — The architecture, built around attention and stacked blocks, that underlies essentially every model in this book. (Chapter 3)
U-Net — The hourglass-shaped network commonly used as a diffusion model’s noise-spotter: it squeezes the image to reason globally, then re-expands it, with shortcut wires preserving fine detail. (Chapter 8)
Variational lower bound — The intimidating-sounding theoretical training goal for diffusion that, once simplified, collapses to “predict the noise and minimize the error.” (Chapter 8)